Semantic search with ELSER in Elasticsearch

Learn how to use the ELSER for text expansion-powered semantic search.

Note: This notebook demonstrates how to use ELSER model .elser_model_2 model which offers an improved retrieval accuracy.

If you have set up an index with ELSER model .elser_model_1, and would like to upgrade to ELSER v2 model - .elser_model_2, Please follow instructions from the notebook on how to upgrade an index to use elser model

To get started, we'll need to connect to our Elastic deployment using the Python client. Because we're using an Elastic Cloud deployment, we'll use the Cloud ID to identify our deployment.

First we need to pip install the following packages:

elasticsearch

Next, we need to import the modules we need. 🔐 NOTE: getpass enables us to securely prompt the user for credentials without echoing them to the terminal, or storing it in memory.

Now we can instantiate the Python Elasticsearch client.

First we prompt the user for their password and Cloud ID. Then we create a client object that instantiates an instance of the Elasticsearch class.

Enable Telemetry

Knowing that you are using this notebook helps us decide where to invest our efforts to improve our products. We would like to ask you that you run the following code to let us gather anonymous usage statistics. See telemetry.py for details. Thank you!

Test the Client

Before you continue, confirm that the client has connected with this test.

{'name': 'instance-0000000011', 'cluster_name': 'd1bd36862ce54c7b903e2aacd4cd7f0a', 'cluster_uuid': 'tIkh0X_UQKmMFQKSfUw-VQ', 'version': {'number': '8.11.1', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': '6f9ff581fbcde658e6f69d6ce03050f060d1fd0c', 'build_date': '2023-11-11T10:05:59.421038163Z', 'build_snapshot': False, 'lucene_version': '9.8.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}

Refer to https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html#connect-self-managed-new to learn how to connect to a self-managed deployment.

Read https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/connecting.html#connect-self-managed-new to learn how to connect using API keys.

In this example, we are going to download and deploy the ELSER model in our ML node. Make sure you have an ML node in order to run the ELSER model.

The above command will download the ELSER model. This will take a few minutes to complete. Use the following command to check the status of the model download.

Once the model is downloaded, we can deploy the model in our ML node. Use the following command to deploy the model.

This also will take a few minutes to complete.

In order to use ELSER on our Elastic Cloud deployment we'll need to create an ingest pipeline that contains an inference processor that runs the ELSER model. Let's add that pipeline using the put_pipeline method.

ObjectApiResponse({'acknowledged': True})

Let's note a few important parameters from that API call:

inference: A processor that performs inference using a machine learning model.
model_id: Specifies the ID of the machine learning model to be used. In this example, the model ID is set to .elser_model_2.
input_output: Specifies input and output fields
input_field: Field name from which the sparse_vector representation are created.
output_field: Field name which contains inference results.

Create index

To use the ELSER model at index time, we'll need to create an index mapping that supports a text_expansion query. The mapping includes a field of type sparse_vector to work with our feature vectors of interest. This field contains the token-weight pairs the ELSER model created based on the input text.

Let's create an index named elser-example-movies with the mappings we need.

ObjectApiResponse({'acknowledged': True, 'shards_acknowledged': True, 'index': 'elser-example-movies'})

Insert Documents

Let's insert our example dataset of 12 movies.

If you get an error, check the model has been deployed and is available in the ML node. In newer versions of Elastic Cloud, ML node is autoscaled and the ML node may not be ready yet. Wait for a few minutes and try again.

Done indexing documents into `elser-example-movies` index!

Inspect a new document to confirm that it now has an plot_embedding field that contains a list of new, additional terms. These terms are the text expansion of the field(s) you targeted for ELSER inference in input_field while creating the pipeline. ELSER essentially creates a tree of expanded terms to improve the semantic searchability of your documents. We'll be able to search these documents using a text_expansion query.

But first let's start with a simple keyword search, to see how ELSER delivers semantically relevant results out of the box.

Let's test out semantic search using ELSER.

Score: 12.763346 Title: Fight Club Plot: An insomniac office worker and a devil-may-care soapmaker form an underground fight club that evolves into something much, much more. Score: 9.930427 Title: Pulp Fiction Plot: The lives of two mob hitmen, a boxer, a gangster and his wife, and a pair of diner bandits intertwine in four tales of violence and redemption. Score: 9.4883375 Title: The Matrix Plot: A computer hacker learns from mysterious rebels about the true nature of his reality and his role in the war against its controllers.

Next Steps

Now that we have a working example of semantic search using ELSER, you can try it out on your own data. Don't forget to scale down the ML node when you are done.