Elasticsearch 6.2.0 released

SAML Support

The ability to login to X-Pack has existed for quite some time. We’re seeing a trend in the industry where every application having its own login screen is not only annoying but less secure than using single sign on for authentication. With the release of 6.2.0 you now have the ability to login to Kibana using a SAML single sign on provider. This should simplify managing users in your elasticsearch cluster and provide a more secure environment. No more forgotten passwords or Kibana login screens. Just login to your SAML provider and when you visit Kibana you have access.

Audit Filtering

If you’ve used the audit logging capabilities of X-Pack in the past you know it has a very all or nothing feel to it. Logging too little may not provide the data you need and too much can fill up storage. As compliance requirements continue to become more strict good audit logs become a necessity. We have added to ability to filter audit logs in version 6.2.0. Now you don’t have to worry about logging too little data or filling up a disk because of a noisy client. Log what you need and only what you need.

Search Quality and the New Ranking Evaluation API

Let's say you've built a local search service, and a user has just searched for "Amsterdam", would he or she prefer a document on "Welcome to Amsterdam!" or "Sights and Sounds of Holland"? How do you ensure a user will always see the best results for the search term "Amsterdam", especially as your search index grows over time, new documents and fields are added, and other factors affect the score of various search results?

In Elasticsearch 6.2, we are excited to announce a new ranking evaluation api. The Rank Evaluation API is a new endpoint in Elasticsearch for tracking how the rankings of expected results are quantitatively measured against specific queries.  This allows you to create a set of metrics that can be measured programmatically over time. Imagine creating a set of unit tests that ensure "Welcome to Amsterdam!" always ranks highly for our search term above.

The Ranking Evaluation API can take into account Mean Reciprical Rank (MRR), Precision-at-k (P@k), and Discounted Cumulative Gain (DCG), as ways to measure quality in results. If you've never heard of these, have a look over the linked wiki articles.

Providing greater safeguards to user searches

Creating a search query containing 100,000 ids sounds like a bad idea, and any user performing a query like this could negatively impact Elasticsearch. In 6.2, we've provided improved safeguards to limit the number of bad queries that can be executed by Elasticsearch.

A few of the new safeguards include:

• Default limit for terms in a terms query: 65,536
• Limit on the number of nested objects: 10,000
• Limit to the number of characters to be analyzed for highlighting: 10,000
• Default max_shingle_size and min_shingle_size allowed difference: 3
• Default max_ngram and min_ngram allowed difference: 1

We’ve created these numbers as sane defaults for most Elasticsearch systems, however, any of the settings above can be modified by an administrator.

Well Known Text Support

Several geospatial systems use Well Known Text (WKT) as their preferred/only format for geospatial objects.  What if you wanted to use Elasticsearch for your geospatial data though?  Until 6.2, Elasticsearch has only provided the option of providing shapes in GeoJSON format.   To get your WKT data into Elasticsearch, you may have to go through a complicated export + conversion process.  No longer!  You can now index a shape in a WKT string directly to Elasticsearch.

Better Programmatic Access to the Task Manager

In Elasticsearch 2.3, we introduced a new Task Management API.  That API has been responsible for tracking ongoing long-running tasks like reindex operations.  However, because Elasticsearch may rewrite parts of tasks once it receives them, actioning ongoing tasks including cancelling them has required manual intervention to try to figure out which of the tasks matched up with a particular action.  For many types of tasks, that may be rare and easy, but doing this programmatically would be nearly impossible.  Now, in 6.2, we’ve added a new capability to this tasks API: you can now assign a task an ID to a new task using the X-Opaque-Id header and  then use that to look up and/or cancel tasks programmatically later.

Java 9 Support

Plugin developers and users that build at home may have already noticed some changes in Elasticsearch which now requires JDK 9 to build.  We’ve outlined many elements of this change in a blog post.  With the release of Elasticsearch 6.2, we’re happy to announce that we now support Java 9!  Before you go go through and replace every Java install in your company, you’ll want to be aware of a few things:

1. The majority of JDK vendors are treating version 9 and version 10 as short term releases.  With only a six month total version lifecycle, both JDK 9 and 10 are looking to hit an end of public updates before JDK 8.  In fact, at the time of this writing, it looks like JDK 9 will hit end of life as soon as March 2018 and JDK 10 is slated to EOL in September 2018.  Practically, all of this means: if you like changing things less often, you may want to stick to Java 8 for now.
2. JDK 9 hits end of updates before Elasticsearch 6.2 EOLs.  You should be extra aware before using it that JDK vendors will be unable to patch any security issues in JDK 9 after it reaches that stage.

As a result of these, while we will be supporting Java 9 per our support matrix, we strongly encourage users to stick to Java 8 unless you’re prepared for dealing with this rapidly evolving landscape.

Please download Elasticsearch 6.2.0, try it out, and let us know what you think on Twitter (@elastic) or in our forum. You can report any problems on the GitHub issues page.

Elasticsearch 6.2.0 is affected by a bug that impacts the ability to upgrade plugins. A fix will be released shortly. To workaround until a fix is available, remove plugins before upgrading from an older version or downgrade Elasticsearch to your previously-installed version, remove plugins, and then upgrade again.