What’s new in 8.4
editWhat’s new in 8.4
editHere are the highlights of what’s new and improved in Elasticsearch 8.4!
Other versions:
Speed up filters/range/date_histogram aggs
editThis speeds up a few aggregations when they don’t have child aggregations.
That’s super common, for example, the histogram at the top of Kibana’s
discover tab is a date_histogram
without any child aggregations. That
particular aggregation is sped up by about 85% in our rally tests, dropping
from 250ms to 30ms.
Minimum conditions for the rollover API and ILM actions
editThe rollover API and ILM actions now support minimum conditions for rollover.
Minimum conditions prevent rollover from occurring until they are met. That is, an index will rollover once one or more max conditions are satisfied and all min conditions are satisfied.
As an example, the following ILM policy would roll an index over if it is at least 7 days old or at least 100 gigabytes, but only as long as the index is not empty.
PUT _ilm/policy/my_policy { "policy": { "phases": { "hot": { "actions": { "rollover" : { "max_age": "7d", "max_size": "100gb", "min_docs": 1 } } } } } }
Infinite and adaptive retries for transforms
editInfinite and adaptive retries – available in 8.4 – makes it possible for transforms to recover after a failure without any user intervention. Retries can be configured per transform. The transform retries become less frequent progressively. The interval between retries doubles after reaching a one-hour threshold. This is because the possibility that retries solve the problem is less likely after each failed retry.
In the Transforms page in Stack Management in Kibana, the number of retries can be configured when creating a new transform or editing an existing one.
Composite aggregations in datafeeds are Generally Available
editThe support for composite aggregations in datafeeds is now generally available.
Optimizing speed of data frame analytics
editData frame analytics is even faster in 8.4. The new function automatically stops the process of hyperparameter optimization early in case the accuracy gain for a different set of hyperparameter values would be insignificant. The early stopping of the optimization process results in a shorter runtime for the data frame analytics job.
Integrate ANN into _search
endpoint
editThis change adds a knn
option to the _search
API to support ANN
search. It’s powered by the same Lucene ANN capabilities as the old
_knn_search
endpoint. The knn
option can be combined with other
search features like queries and aggregations.
Synthetic _source
editThis functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
When Elasticsearch ingests documents, it creates several data structures that
enable querying, aggregating, and retrieving data. One of those data
structures is _source
, which is used for reindexing, updating documents, and
in Kibana’s Discover. Storing _source
enables fetching the original documents
exactly as they were indexed. However, fetching the exact original
documents isn’t required by many use cases, and storing _source
uses a lot
of space.
This release introduces a new feature in technical preview called synthetic
_source
. For specific data types, synthetic _source
can significantly reduce
the index size by rebuilding _source
from doc values. Doc values are data
structures that are typically stored already because they’re needed for
aggregations. While synthetic _source
doesn’t recreate the exact structure of
the original documents, it’s good enough for features like reindexing data.
Errors with the cross_fields
query type
editThe Elasticsearch 8.4.2 release introduced a bug that may fail queries using the
cross_fields
type at search time with an error message containing
totalTermFreq must be at least docFreq
. If you run queries that use
the
cross_fields
type,
we strongly recommend upgrading to 8.4.3 to avoid hitting this bug.