Elasticsearch highlights

This list summarizes the most important enhancements in Elasticsearch 7.9. For the complete list, go to Elasticsearch release highlights.

Cross-cluster replication now retries operations that failed due to a circuit breaker or a lost remote cluster connection.

When indexing data, Elasticsearch and Lucene use heap memory for buffering. To control memory usage, Elasticsearch moves data from the buffer to disk based on your indexing buffer settings. If ongoing indexing outpaces the relocation of data to disk, Elasticsearch will now throttle indexing. In previous Elasticsearch versions, this feature was broken and throttling was not activated.

EQL (Event Query Language) is a declarative language dedicated for identifying patterns and relationships between events.

Consider using EQL if you:

A good intro on EQL and its purpose is available in this blog post. See the EQL in Elasticsearch documentaton for an in-depth explanation, and also the language reference.

This release includes the following features:

An in-depth discussion of EQL in ES scope can be found at #49581.

A data stream is a convenient, scalable way to ingest, search, and manage continuously generated time series data. They provide a simpler way to split data across multiple indices and still query it via a single named resource.

See the Data streams documentation to get started.

Snapshot operations can now execute in a fully concurrent manner.

Before 7.9, many of our more complex aggregations made a simplifying assumption that required that they duplicate many data structures once per bucket that contained them. The most expensive of these weighed in at a couple of kilobytes each. So for an aggregation like:

POST _search
{
  "aggs": {
    "date": {
      "date_histogram": { "field": "timestamp", "calendar_interval": "day" },
      "aggs": {
        "ips": {
          "terms": { "field": "ip" }
        }
      }
    }
  }
}

When run over three years, this aggregation spends a couple of megabytes just on bucket accounting. More deeply nested aggregations spend even more on this overhead. Elasticsearch 7.9 removes all of this overhead, which should allow us to run better in lower memory environments.

As a bonus we wrote quite a few Rally benchmarks for aggregations to make sure that these tests didn’t slow down aggregations, so now we can think much more scientifically about aggregation performance. The benchmarks suggest that these changes don’t affect simple aggregation trees and speed up complex aggregation trees of similar or higher depth than the example above. Your actual performance changes will vary but this optimization should help!

Rounding dates on a shard that contains a daylight savings time transition is currently drastically slower than when a shard contains dates only on one side of the DST transition, and also generates a large number of short-lived objects in memory. Elasticsearch 7.9 has a revised and far more efficient implemention that adds only a comparatively small overhead to requests.

Elasticsearch now has mechansisms to safely resume peer recoveries when there is network disruption, which would previously have failed any in-progress peer recoveries.

Elasticsearch now supports a wildcard field type, which stores values optimised for wildcard grep-like queries. While such queries are possible with other field types, they suffer from constraints that limit their usefulness.

This field type is especially well suited for running grep-like queries on log lines. See the wildcard datatype documentation for more information.

Elasticsearch 7.9 now tracks metrics about the number of indexing request bytes that are outstanding at each point in the indexing process (coordinating, primary, and replication). These metrics are exposed in the node stats API. Additionally, the new setting indexing_pressure.memory.limit controls the maximum number of bytes that can be outstanding, which is 10% of the heap by default. Once this number of bytes from a node’s heap is consumed by outstanding indexing bytes, Elasticsearch will start rejecting new coordinating and primary requests.

Additionally, since a failed replication operation can fail a replica, Elasticsearch will assign 1.5X limit for the number of replication bytes. Only replication bytes can trigger this limit. If replication bytes increase to high levels, the node will stop accepting new coordinating and primary operations until the replication work load has dropped.

In 7.6, we introduced inference that enables you to make predictions on new data with your regression or classification models via a processor in an ingest pipeline. Now, in 7.9, inference is even more flexible! You can reference a pre-trained data frame analytics model in an aggregation to infer on the result field of the parent bucket aggregation. The aggregation uses the model on the results to provide a prediction. This addition enables you to run classification or regression analysis at search time. If you want to perform analysis on a small set of data, you can generate predictions without the need to set up a processor in the ingest pipeline.