Using Cohere for RAG and Rerank in Elasticsearch

This tutorial shows you how to compute embeddings with Cohere using the inference API and store them for efficient vector or hybrid search in Elasticsearch. This tutorial uses the Python Elasticsearch client to perform the operations.

You'll learn how to:

create an inference endpoint for text embedding using the Cohere service,
create the necessary index mapping for the Elasticsearch index,
build an inference pipeline to ingest documents into the index together with the embeddings,
perform hybrid search on the data,
rerank search results by using Cohere's rerank model,
design a RAG system with Cohere's Chat API.

The tutorial uses the SciFact data set.

Refer to Cohere's tutorial for an example using a different data set.

🧰 Requirements

For this example, you will need:

An Elastic deployment with minimum 4GB machine learning node
- We'll be using Elastic Cloud for this example (available with a free trial)
A paid Cohere account is required to use the Inference API with the Cohere service as the Cohere free trial API usage is limited.
Python 3.7 or later.

Install and import required packages

Install Elasticsearch and Cohere:

Import the required packages:

Create an Elasticsearch client

Now you can instantiate the Python Elasticsearch client.

First provide your password and Cloud ID. Then create a client object that instantiates an instance of the Elasticsearch class.

Create the inference endpoint

Create the inference endpoint first. In this example, the inference endpoint uses Cohere's embed-english-v3.0 model and the embedding_type is set to byte.

You can find your API keys in your Cohere dashboard under the API keys section.

Create the index mapping

Create the index mapping for the index that will contain the embeddings.

Create the inference pipeline

Now you have an inference endpoint and an index ready to store embeddings. The next step is to create an ingest pipeline that creates the embeddings using the inference endpoint and stores them in the index.

Prepare data and insert documents

This example uses the SciFact data set that you can find on HuggingFace.

Your index is populated with the SciFact data and text embeddings for the text field.

Hybrid search

Let's start querying the index!

The code below performs a hybrid search. The kNN query computes the relevance of search results based on vector similarity using the text_embedding field. The lexical search query uses BM25 retrieval to compute keyword similarity on the title and text fields.

Rerank search results

To combine the results more effectively, use Cohere's Rerank v3 model through the inference API to provide a more precise semantic reranking of the results.

Create an inference endpoint with your Cohere API key and the used model name as the model_id (rerank-english-v3.0 in this example).

Rerank the results using the new inference endpoint.

Title: Interchangeability of Biosimilars: A European Perspective Text: Many of the best-selling ‘blockbuster’ biological medicinal products are, or will soon be, facing competition from similar biological medicinal products (biosimilars) in the EU. Biosimilarity is based on the comparability concept, which has been used successfully for several decades to ensure close similarity of a biological product before and after a manufacturing change. Over the last 10 years, experience with biosimilars has shown that even complex biotechnology-derived proteins can be copied successfully. Most best-selling biologicals are used for chronic treatment. This has triggered intensive discussion on the interchangeability of a biosimilar with its reference product, with the main concern being immunogenicity. We explore the theoretical basis of the presumed risks of switching between a biosimilar and its reference product and the available data on switches. Our conclusion is that a switch between comparable versions of the same active substance approved in accordance with EU legislation is not expected to trigger or enhance immunogenicity. On the basis of current knowledge, it is unlikely and very difficult to substantiate that two products, comparable on a population level, would have different safety or efficacy in individual patients upon a switch. Our conclusion is that biosimilars licensed in the EU are interchangeable. Title: siRNA specificity searching incorporating mismatch tolerance data. Text: UNLABELLED Artificially synthesized short interfering RNAs (siRNAs) are widely used in functional genomics to knock down specific target genes. One ongoing challenge is to guarantee that the siRNA does not elicit off-target effects. Initial reports suggested that siRNAs were highly sequence-specific; however, subsequent data indicates that this is not necessarily the case. It is still uncertain what level of similarity and other rules are required for an off-target effect to be observed, and scoring schemes have not been developed to look beyond simple measures such as the number of mismatches or the number of consecutive matching bases present. We created design rules for predicting the likelihood of a non-specific effect and present a web server that allows the user to check the specificity of a given siRNA in a flexible manner using a combination of methods. The server finds potential off-target matches in the corresponding RefSeq database and ranks them according to a scoring system based on experimental studies of specificity. AVAILABILITY The server is available at http://informatics-eskitis.griffith.edu.au/SpecificityServer. Title: ImmTACs for targeted cancer therapy: Why, what, how, and which. Text: Overcoming immunosuppression and activating a cytotoxic T cell response has the potential to halt the progression of cancer and, in some circumstances, eradicate it. Designing therapeutic interventions that achieve this goal has proven challenging, but now a greater understanding of the complexities of immune responses is beginning to produce some notable breakthroughs. ImmTACs (immune-mobilising monoclonal TCRs against cancer) are a new class of bispecific reagents, based on soluble monoclonal T cell receptors, which have been engineered to possess extremely high affinity for cognate tumour antigen. In this way, ImmTACs overcome the problem of low affinity tumour-specific T cells imposed by thymic selection and provide access to the large number of antigens presented as peptide-HLA complexes. Once bound to tumour cells the anti-CD3 effector end of the ImmTAC drives recruitment of polyclonal T cells to the tumour site, leading to a potent redirected T cell response and tumour cell destruction. Extensive in vitro testing coupled with promising early clinical data has provided an enhanced appreciation of ImmTAC function in vivo and indicates their potential therapeutic benefit in terms of a durable response and ultimately the breaking of T cell tolerance. This review introduces ImmTACs in the context of immunotherapy, and outlines their design, construction and mechanism of action, as well as examining target selection and aspects of preclinical safety testing. Title: Coming soon to a Wal-Mart near you Text: According to the Web site of the Association for Automatic Identification and Data Capture Technologies [http://www.aimglobal.org/technologies/rfid], "radio frequency identification (RFID) technology is an automatic way to collect product, place, and time or transaction data quickly and easily without human intervention or error. " With the ability to track everything from crates of disposable razors to individual peanut-butter jars on the store shelves, RFID technology offers the potential of "real-time supply chain visibility. " Promoters of RFID technology feel [C. Humer, 2003] that "RF tags are to this decade what the Internet was to the 1990's-a promise of radical change in the way business is done. " However, before the full potential of RFID technology can be realized, several hurdles need to be overcome: reliability, cost, lack of standards, and security. As these hurdles gradually diminish, Wal-Mart publicly embraces the technology. Title: Dynamic Helix Interactions in Transmembrane Signaling Text: Studying how protein transmembrane domains transmit signals across membranes is beset by unique challenges. Here, we discuss the circumstances that have led to success and reflect on what has been learned from these examples. Such efforts suggest that some of the most interesting properties of transmembrane helix interactions may be the least amenable to study by current techniques. Title: Ciliary Extracellular Vesicles: Txt Msg Organelles Text: Cilia are sensory organelles that protrude from cell surfaces to monitor the surrounding environment. In addition to its role as sensory receiver, the cilium also releases extracellular vesicles (EVs). The release of sub-micron sized EVs is a conserved form of intercellular communication used by all three kingdoms of life. These extracellular organelles play important roles in both short and long range signaling between donor and target cells and may coordinate systemic responses within an organism in normal and diseased states. EV shedding from ciliated cells and EV–cilia interactions are evolutionarily conserved phenomena, yet remarkably little is known about the relationship between the cilia and EVs and the fundamental biology of EVs. Studies in the model organisms Chlamydomonas and Caenorhabditis elegans have begun to shed light on ciliary EVs. Chlamydomonas EVs are shed from tips of flagella and are bioactive. Caenorhabditis elegans EVs are shed and released by ciliated sensory neurons in an intraflagellar transport-dependent manner. Caenorhabditis elegans EVs play a role in modulating animal-to-animal communication, and this EV bioactivity is dependent on EV cargo content. Some ciliary pathologies, or ciliopathies, are associated with abnormal EV shedding or with abnormal cilia–EV interactions. Until the 21st century, both cilia and EVs were ignored as vestigial or cellular junk. As research interest in these two organelles continues to gain momentum, we envision a new field of cell biology emerging. Here, we propose that the cilium is a dedicated organelle for EV biogenesis and EV reception. We will also discuss possible mechanisms by which EVs exert bioactivity and explain how what is learned in model organisms regarding EV biogenesis and function may provide insight to human ciliopathies. Title: How nascent phagosomes mature to become phagolysosomes. Text: Phagocytosis mediates the clearance of apoptotic bodies and also the elimination of microbial pathogens. The nascent phagocytic vacuole formed upon particle engulfment lacks microbicidal and degradative activity. These capabilities are acquired as the phagosome undergoes maturation; a progressive remodeling of its membrane and contents that culminates in the formation of phagolysosomes. Maturation entails orderly sequential fusion of the phagosomal vacuole with specialized endocytic and secretory compartments. Concomitantly, the phagosomal membrane undergoes both inward and outward vesiculation and tubulation followed by fission, thereby recycling components and maintaining its overall size. Here, we summarize what is known about the molecular machinery that governs this complex metamorphosis of phagosome maturation. Title: What influences government adoption of vaccines in developing countries? A policy process analysis. Text: This paper proposes a framework for examining the process by which government consideration and adoption of new vaccines takes place, with specific reference to developing country settings. The cases of early Hepatitis B vaccine adoption in Taiwan and Thailand are used to explore the relevance of explanatory factors identified in the literature as well as the need to go beyond a variable-centric focus by highlighting the role of policy context and process in determining the pace and extent of adoption. The cases suggest the feasibility and importance of modeling 'causal diversity'-the complex set of necessary and sufficient conditions leading to particular decisional outcomes-in a broad range of country contexts. A better understanding of the lenses through which government decision-makers filter information, and of the arenas in which critical decisions are shaped and taken, may assist both analysts (in predicting institutionalization of new vaccines) and advocates (in crafting targeted strategies to accelerate their diffusion). Title: Aire regulates negative selection of organ-specific T cells Text: Autoimmune polyendocrinopathy syndrome type 1 is a recessive Mendelian disorder resulting from mutations in a novel gene, AIRE, and is characterized by a spectrum of organ-specific autoimmune diseases. It is not known what tolerance mechanisms are defective as a result of AIRE mutation. By tracing the fate of autoreactive CD4+ T cells with high affinity for a pancreatic antigen in transgenic mice with an Aire mutation, we show here that Aire deficiency causes almost complete failure to delete the organ-specific cells in the thymus. These results indicate that autoimmune polyendocrinopathy syndrome 1 is caused by failure of a specialized mechanism for deleting forbidden T cell clones, establishing a central role for this tolerance mechanism. Title: Parallel processing in the mammalian retina Text: Our eyes send different 'images' of the outside world to the brain — an image of contours (line drawing), a colour image (watercolour painting) or an image of moving objects (movie). This is commonly referred to as parallel processing, and starts as early as the first synapse of the retina, the cone pedicle. Here, the molecular composition of the transmitter receptors of the postsynaptic neurons defines which images are transferred to the inner retina. Within the second synaptic layer — the inner plexiform layer — circuits that involve complex inhibitory and excitatory interactions represent filters that select 'what the eye tells the brain'.

Retrieval Augmented Generation (RAG) with Cohere and Elasticsearch

RAG is a method for generating text using additional information fetched from an external data source. With the ranked results, you can build a RAG system on top of what you created with Cohere's Chat API.

Pass in the retrieved documents and the query to receive a grounded response using Cohere's newest generative model Command R+.

Then pass in the query and the documents to the Chat API, and print out the response.

Query: What is biosimilarity? Response: Biosimilarity is based on the comparability concept, which has been used successfully for several decades to ensure close similarity of a biological product before and after a manufacturing change. Over the last 10 years, experience with biosimilars has shown that even complex biotechnology-derived proteins can be copied successfully. Sources: Interchangeability of Biosimilars: A European Perspective: Many of the best-selling ‘blockbuster’ biological medicinal products are, or will soon be, facing competition from similar biological medicinal products (biosimilars) in the EU. Biosimilarity is based on the comparability concept, which has been used successfully for several decades to ensure close similarity of a biological product before and after a manufacturing change. Over the last 10 years, experience with biosimilars has shown that even complex biotechnology-derived proteins can be copied successfully. Most best-selling biologicals are used for chronic treatment. This has triggered intensive discussion on the interchangeability of a biosimilar with its reference product, with the main concern being immunogenicity. We explore the theoretical basis of the presumed risks of switching between a biosimilar and its reference product and the available data on switches. Our conclusion is that a switch between comparable versions of the same active substance approved in accordance with EU legislation is not expected to trigger or enhance immunogenicity. On the basis of current knowledge, it is unlikely and very difficult to substantiate that two products, comparable on a population level, would have different safety or efficacy in individual patients upon a switch. Our conclusion is that biosimilars licensed in the EU are interchangeable.