The workbook implements NLP text search in Elasticsearch using a simple dataset consisting of Elastic blogs titles.
You will index blogs documents, and using ingest pipeline generate text embeddings. By using NLP model you will query the documents using natural language over the the blogs documents.
Prerequisities
Before we begin, create an elastic cloud deployment and autoscale to have least one machine learning (ML) node with enough (4GB) memory. Also ensure that the Elasticsearch cluster is running.
If you don't already have an Elastic deployment, you can sign up for a free Elastic Cloud trial.
Install packages and import modules
Before you start you need to install all required Python dependencies.
Deploy an NLP model
We are using the eland tool to install a text_embedding model. For our model, We have used all-MiniLM-L6-v2 to transform the search text into the dense vector.
The model will transfer your search query into vector which will be used for the search over the set of documents stored in Elasticsearch.
Install text embedding NLP model
Using the eland_import_hub_model script, download and install all-MiniLM-L6-v2 transformer model. Setting the NLP --task-type as text_embedding.
To get the cloud id, go to Elastic cloud and On the deployment overview page, copy down the Cloud ID.
To authenticate your request, You could use API key. Alternatively, you can use your cloud deployment username and password.
Connect to Elasticsearch cluster
Create a elasticsearch client instance with your deployment Cloud Id and API Key. In this example, we are using the API_KEY and CLOUD_ID value from previous step.
Alternately you could use your deployment Username and Password to authenticate your instance.
Create an Ingest pipeline
We need to create a text embedding ingest pipeline to generate vector (text) embeddings for title field.
The pipeline below is defining a processor for the inference to the NLP model.
Create Index with mappings
We will now create an elasticsearch index with correct mapping before we index documents.
We are adding text_embedding to include the model_id and predicted_value to store the embeddings.
Index data to elasticsearch index
Let's index sample blogs data using the ingest pipeline.
Note: Before we begin indexing, ensure you have started your trained model deployment.
Querying the dataset
The next step is to run a query to search for relevant blogs. The example query searches for "model_text": "how to track network connections" using the model we uploaded to Elasticsearch sentence-transformers__all-minilm-l6-v2.
The process is one query even though it internally consists of two tasks. First, the query will generate an vector for your search text using the NLP model and then pass this vector to search over the dataset.
As a result, the output shows the list of query documents sorted by their proximity to the search query.