Elasticsearch highlights

This list summarizes the most important enhancements in Elasticsearch 7.16. For the complete list, go to Elasticsearch release highlights.

Elasticsearch SQL now supports cross-cluster search (CCS) using the <remote_cluster>:<target> syntax, where <remote_cluster> maps to a SQL catalog (cluster) and <target> to a table (index or data stream).

If a search hits a large number of shards, the search operation includes a pre-filter phase called the can-match phase. During this phase, Elasticsearch checks if an impacted shard contains data that could potentially match the search query. If not, Elasticsearch doesn’t run the query on the shard.

Previously, the search’s coordinating node sent an individual request to each shard checked during the can-match phase. However, if the search needed to check thousands of shards, the coordinating node would need to handle thousands of requests, resulting in high overhead.

With 7.16, the coordinating node instead sends a single request to each data node during the can-match phase. This request covers can-match checks for all impacted shards on the node, significantly reducing the number of requests and related overhead.

Previously, field caps were gathered by sending a request per index to the appropriate nodes in the cluster. This resulted in high overhead when many indices were targeted. Now requests that target indices on a single node are grouped together, resulting in no more than a single request per node.

Requests and queries against indices in the frozen tier are now cached, which improves performance for subsequent requests and queries.

With 7.16, we added the range enrich policy type for the enrich processor. You can use a range policy to enrich incoming documents based on a number, date, or IP address that matches a range in the enrich index.

For example, if incoming documents contain an IP address, you can use a range policy to enrich the documents based on their IP range. For an in-depth example, see Example: Enrich your data by matching a value to a range.

For data streams, Elasticsearch now sorts a backing index’s segments by maximum @timestamp value in descending order. This speeds up searches sorted by @timestamp, which are common for data streams.

In 7.16, we added the with runs statement syntax to EQL sequence queries. Sometimes you want to find a sequence that contains an event multiple times in succession. Rather than type the same event criteria multiple times, you can use a with runs statement to declare the criteria once and run it successively. For more details, check out the EQL syntax documentation.