Elasticsearch 6.0.0-beta1 released

While synced flush has greatly improved shard recovery times for indices that are not being written to, recovery of active indices is still a slow and heavy operation. An active replica on a node that leaves the cluster for a brief period still needs to copy over all or most of the files in the primary shard in order to bring itself up to date.

The new Sequence Numbers infrastructure assigns an incremental operation ID to every index, update, or delete. This new infrastructure allows a replica to ask the primary for all operations from X onwards. If these operations are found in the primary’s translog an older replica can bring itself up to date by just replaying the transaction log and avoid the need to copy files.

This release features:

We also have the infrastructure we need to start developing the cross data centre replication (xDCR) X-Pack feature. We will continue to use the new infrastructure to tackle more complex correctness problems, specifically the rollback of unacknowledged operations in replicas and the usage of sequence numbers for optimistic locking.

A search against an index pattern like logstash-* can fan out to a huge number of shards. Usually, queries like these include a date range filter which means that the majority of shards won’t contain any matching documents. We already include an optimization to abort search requests on these shards early, before any real work is done, but this is not enough. Imagine a multi-search request containing 10 search requests, each of which target 2,000 shards. That’s 20,000 shard-level search requests which are added to the search threadpool queue. This could easily result in rejections, even though the majority of these requests are very quick.

Previously, Kibana used the _field_stats API with a date range filter to figure out which indices might contain matching documents, and then ran the search request against only those indices. We wanted to remove this API because it was much heavier than users expected and open to abuse. Instead, a search request now has a light shard prefiltering phase which is triggered if a search request targets at least 128 shards (by default). These prefilter requests are not added to the search queue and so cannot be rejected because the queue is full. The prefilter request rewrites the query at the shard level and determines whether the query has any chance of matching any documents at all. The full search request is then sent only to those shards which have a chance of matching.

But what if the user actually does want to search all 2,000 shards, or searches all indices by mistake? These wide-ranging requests should not overwhelm the cluster nor get in the way of search requests from other users. In order to solve this, we introduced the max_concurrent_shard_requests parameter whose default value depends on the number of nodes in the cluster, but which has a fixed upper limit of 256. This may make a single search request that targets many shards slower, but it makes for fairer concurrent searches by many users.

We have long had the cluster.routing.allocation.disk.watermark.low and cluster.routing.allocation.disk.watermark.high settings which prevent shards being assigned to full disks and actively move shards away from full disks. However, if all of the disks in your cluster are full, then there is nowhere to move shards to and eventually you will run out of disk space. Now, we have added the cluster.routing.allocation.disk.watermark.flood_stage setting. When a disk passes this level, indices that have shards on this node will be set to read only. No more writes will be accepted. Instead, you will need to either delete the index or add more space and set it back to read-write.

To prevent a persistent logged failure from filling up the disk, Elasticsearch is switching to the following out-of-the-box logging config:

In 5.x, X-Pack Security set the passwords of internal users to changeme by default, in order to make the getting-started experience easier, but it is never a good idea to ship with default passwords. In 6.0, we have added a bootstrap.password setting which can be added to the secure keystore before startup. When the cluster starts up, any node with this setting will try to set the password for the elastic user unless that user already has a password, so that the cluster will start in a secure state. On top of that, we’ve added a setup-passwords command line tool which will generate and set strong passwords for all of the internal users.