What’s new in 7.12

Here are the highlights of what’s new and improved in Elasticsearch 7.12!

For detailed information about this release, see the Release notes and Migration guide.

Other versions: 7.11 | 7.10 | 7.9 | 7.8 | 7.7 | 7.6 | 7.5 | 7.4 | 7.3 | 7.2 | 7.1 | 7.0

With 7.12, we’re adding a frozen phase to ILM along with a corresponding frozen data tier as experimental features. You can use the new frozen phase and frozen tier to archive rarely searched data as searchable snapshots for long-term storage and maximum cost savings.

The key advantage of the frozen tier is an experimental new option for mounting searchable snapshots: shared snapshot caches. Rather than loading a full copy of a snapshotted index into your cluster, this option uses a small local cache containing only recently searched parts of the index’s data. If a search requires data that’s not in the cache, Elasticsearch fetches the missing data from the snapshot repository. These caches have a fixed size and can significantly reduce your storage costs. ILM uses shared snapshot caches by default in the frozen phase and corresponding frozen tier.

To add a node to the frozen tier in an on-premise cluster, assign it the data_frozen node role and set xpack.searchable.snapshot.shared_cache.size in elasticsearch.yml. See the Data tiers and Searchable snapshots documentation.

While there are many third-party storage systems, not all of them work well as snapshot repositories. Some systems perform poorly or behave incorrectly as a repository, especially when multiple Elasticsearch nodes access them concurrently.

To detect and avoid these problems, you can now use the repository analysis API to analyze a repository’s storage system before using it in production. The API performs a collection of read and write operations on the repository. These operations are designed to detect incorrect behavior and measure the performance characteristics of a system.

The in and not in lookup operators now support case-insensitive variants: in~ and not in~. You can also make an EQL function case-insensitive by adding ~ after the function name, such as endsWith~(process_name, ".exe").

In 7.12, we added the like and regex keywords to EQL. You can use the like keyword to match strings to wildcard patterns, such as foo* or ba?. You can use the regex keyword to match strings to regular expressions. You can use both like and regex to match a string against a list of patterns.

By default, both keywords are case-sensitive. For case-insensitive matching, use like~ or regex~. For more information, see the Pattern comparison keywords section of the EQL syntax documentation.

Continuous transforms add new data to the destination index as new entities are encountered. 7.12 introduces a data retention policy for transforms that enables you to delete old data. This is especially useful for the latest transform that has been added in 7.11. For example, if you collect the latest host information you might want to assume hosts have been decommissioned if they have not appeared in the logs for more than 30 days. At each checkpoint, transformed documents whose age is greater than a configured value are deleted.

Data frame analytics training is influenced by settings called hyperparameters. One of the training outputs is now hyperparameter importance. This value shows which hyperparameter had the most impact on the training, and what the optimum values were for each one. You can use this information to speed up the training process as the hyperparameter optimization process can be stopped when you see that gains in accuracy flatten.

You can now use runtime_mappings in transforms to support search-time runtime fields in searches of the source index. The syntax is the same as for a standard search. After specifying them, the runtime fields can be used in the rest of the transform configuration.