Breaking changes in 6.0

This section discusses the changes that you need to be aware of when migrating your application to Elasticsearch 6.0.

Elasticsearch 6.0 can read indices created in version 5.0 or above. An Elasticsearch 6.0 node will not start in the presence of indices created in a version of Elasticsearch before 5.0.

The include and exclude options of terms aggregations used to accept a sub pattern object which has been removed. The pattern should now be directly put as a value of the include and exclude fields. For instance, the below terms aggregation:

POST /twitter/_search?size=0
{
    "aggs" : {
        "top_users" : {
            "terms" : {
                "field" : "user",
                "include": {
                  "pattern": "foo.*"
                },
                "exclude": {
                  "pattern": ".*bar"
                }
            }
        }
    }
}

should be replaced with:

POST /twitter/_search?size=0
{
    "aggs" : {
        "top_users" : {
            "terms" : {
                "field" : "user",
                "include": "foo.*",
                "exclude": ".*bar"
            }
        }
    }
}

Numeric to and from parameters in date_range aggregations used to always be interpreted as epoch_millis, making other numeric formats like epoch_seconds unusable for numeric input values. Now we interpret these parameters according to the format of the target field. If the format in the mappings is not compatible with the numeric input value, a compatible format (e.g. epoch_millis, epoch_second) must be specified in the date_range aggregation, otherwise an error is thrown.

The execution hints global_ordinals_hash and global_ordinals_low_cardinality are deprecated and should be replaced by global_ordinals which now internally choose whether it should remap global ordinals to dense ordinals or directly use the segment ordinals.

In 6.0, Synonym Token Filter tokenizes synonyms with whatever tokenizer and token filters appear before it in the chain.

The tokenizer and ignore_case parameters are deprecated and will be ignored when used in new indices. These parameters will continue to function as before when used in indices created in 5.x.

Highlighting a text that was indexed without offsets or term vectors, requires analysis of this text in memory real time during the search request. For large texts this analysis may take substantial amount of time and memory. To protect against this, the maximum number of characters that to be analyzed will be limited to 1000000 in the next major Elastic version. For this version, by default the limit is not set. A deprecation warning will be issued when an analyzed text exceeds 1000000. The limit can be set for a particular index with the index setting index.highlight.max_analyzed_offset.

Previously if a queue size backing a thread pool was unbounded, the cat thread pool API would output an empty string in the queue_size column. This has been changed to now output -1 so that the output is always present and always numeric.

Starting from version 5.6.0 a new Java client has been released: the Java High Level REST Client. This official high-level client (named like this to differentiate it from the existing low-level client) for Elasticsearch can be used to execute search, index, delete, update and bulk operations using the same Core Java classes as the TransportClient uses.

This Java High Level REST Client is designed to replace the TransportClient in a near future.

Previously the cluster name could be used in the path.data setting with a warning. This is now no longer allowed. For instance, in the previous version this was valid:

# Assuming path.data is /tmp/mydata
# No longer supported:
$ tree /tmp/mydata
/tmp/mydata
├── <cluster_name>
│   └── nodes
│       └── 0
│           └── <etc>

# Should be changed to:
$ tree /tmp/mydata
/tmp/mydata
├── nodes
│   └── 0
│       └── <etc>

Document modification operations may no longer specify the version_type of force to override any previous version checks.

Adding a version to an upsert request is no longer supported.

The created field has been removed in the Index API as in the index and create bulk operations. operation field should be used instead.

The found field has been removed in the Delete API as in the delete bulk operations. operation field should be used instead.

The ShapeBuilders class containing static methods to create shape specific builders (e.g., ShapeBuilders.newPoint) has been deprecated. One must now use the builder’s constructor instead (e.g., `new MultiPointBuilder()).

The ShapeBuilder.parse method has been deprecated. Use ShapeParser.parse instead.

Previously templates expressed the indices that they should match using a glob style pattern in the template field. They should now use the index_patterns field instead. As the name implies you can define multiple glob style patterns in an array but for convenience defining a single pattern as a bare string is also supported. So both of these examples are valid:

PUT _template/template_1
{
  "index_patterns": ["te*", "bar*"],
  "settings": {
    "number_of_shards": 1
  }
}
PUT _template/template_2
{
  "index_patterns": "te*",
  "settings": {
    "number_of_shards": 1
  }
}

Shadow replicas don’t see enough usage, and have been removed. This includes the following settings:

The default value of the allow_no_indices option for the Open/Close index API has been changed from false to true so it is aligned with the behaviour of the Delete index API. As a result, Open/Close index API don’t return an error by default when a provided wildcard expression doesn’t match any closed/open index.

Delete a document from non-existing index has been modified to not create the index. However if an external versioning is used the index will be created and the document will be marked for deletion.

The index parameter in the update-aliases, put-alias, and delete-alias APIs no longer accepts alias names. Instead, it accepts only index names (or wildcards which will expand to matching indices).

The index parameter in the delete index API no longer accepts alias names. Instead, it accepts only index names (or wildcards which will expand to matching indices).

Omitting the + has the same effect as specifying it, hence support for + has been removed in index expressions.

Translog files are now kept for up to 12 hours (by default), with a maximum size of 512mb (default), and are no longer deleted on flush. This is to increase the chance of doing an operation based recovery when bringing up replicas up to speed.

To safeguard against creating too many index terms, the difference between max_ngram and min_ngram in NGramTokenFilter and NGramTokenizer has been limited to 1. This default limit can be changed with the index setting index.max_ngram_diff. Note that if the limit is exceeded a deprecation warning is logged.

To safeguard against creating too many tokens, the difference between max_shingle_size and min_shingle_size in ShingleTokenFilter has been limited to 3. This default limit can be changed with the index setting index.max_shingle_diff. Note that if the limit is exceeded a deprecation warning will be logged.

The type of the "timestamp" meta-data field has changed from java.lang.String to java.util.Date.

Previously, since Elasticsearch 5.4.0, you needed to use ingest.new_date_format to have the ingest.timestamp metadata field be formatted in such a way that ES can coerce it to a field of type date without further transformation. This is not necessary anymore and this setting was removed. You can now simply set a field to {{ingest.timestamp}} in a pipeline, and have that field be of type date without any mapping errors.

Previously the setSource methods and other methods that accepted byte/string representations of an object source did not require the XContentType to be specified. The auto-detection of the content type is no longer used, so these methods now require the XContentType as an additional argument when providing the source in bytes or as a string.

In previous versions of Elasticsearch, delete by query requests without an explicit query were accepted, match_all was used as the default query and all documents were deleted as a result. From version 6.0.0, a DeleteByQueryRequest requires an explicit query be set.

The count value in the stats aggregation represents a doc count that shouldn’t require a formatted version. This method was deprecated in 5.4 in favour of just using String.valueOf(getCount()) if needed

When sending a request through the request builders e.g. client.prepareSearch().execute(), it used to be possible to call addListener against the returned ListenableActionFuture. With this change an ActionFuture is returned instead, which is consistent with what the Client methods return, hence it is not possible to associate the future with listeners. The execute method that accept a listener as an argument can be used instead.

The terms, histogram, and date_histogram aggregation code has been refactored to use common code for ordering buckets. The BucketOrder class must be used instead of Terms.Order and Histogram.Order. The static methods in the BucketOrder class must be called instead of directly accessing internal order instances, e.g. BucketOrder.count(boolean) and BucketOrder.aggregation(String, boolean). Use BucketOrder.key(boolean) to order the terms aggregation buckets by _term.

In BulkResponse, SearchResponse and TermVectorsResponse getTookInMiilis() method has been removed in favor of getTook method. getTookInMiilis() is easily replaced by getTook().getMillis().

As GetField and SearchHitField have the same members, they have been unified into DocumentField.

The classes for the variants of the range aggregation (geo_distance, date and ip) were moved into the org.elasticsearch.search.aggregations.bucket.range package.

The org.elasticsearch.search.aggregations.bucket.terms.support package was removed and the classes were moved to org.elasticsearch.search.aggregations.bucket.terms.

The filter aggregation classes were moved to org.elasticsearch.search.aggregations.bucket.filter

It is now required to include the desired ranks as a non-null, non-empty array of doubles to the builder’s constructor, rather than configuring them via a setter on the builder. The setter method values() has correspondingly been removed.

Previously, Elasticsearch recognized the strings true, false, on, off, yes, no, 0, 1 as booleans. Elasticsearch 6.0 recognizes only the strings true and false as booleans and will throw an error otherwise. For backwards compatibility purposes, during the 6.x series the previous coercion rules will continue to work on pre-6.0 indices. This means that you do not need to change affected existing mappings immediately. However, it is not possible to create new indices from existing index templates that violate the strict boolean coercion rules.

On new mappings, the _all meta field that contains a copy of the text from each field is now disabled by default. The query_string and simple_query_string queries that previously used _all to search will now check if _all is enabled/disabled and switch to executing the query across all fields if _all is disabled. _all can no longer be configured for indices created with Elasticsearch version 6.0 or later.

Since the _all field is now disabled by default and cannot be configured for indices created with Elasticsearch 6.0 or later, the include_in_all setting is now disallowed for these indices' mappings.

Previously Elasticsearch would silently ignore any dynamic templates that included a match_mapping_type type that was unrecognized. An exception is now thrown on an unrecognized type.

The locale option of date fields previously allowed almost any string values, and unrecognized values would simply be ignored. Unrecognized values are now rejected on indices created with Elasticsearch 6.0 or newer.

Previously someone could configure the $ES_USER and $ES_GROUP variables to change which user and group Elasticsearch was run as. This is no longer possible, the DEB and RPM packages now exclusively use the user and group elasticsearch. If a custom user or group is needed then a provisioning system should use the tarball distribution instead of the provided RPM and DEB packages.

Previous versions of Elasticsearch enabled setting path.conf as a setting. This was rather convoluted as it meant that you could start Elasticsearch with a config file that specified via path.conf that Elasticsearch should use another config file. Instead, to configure a custom config directory, use the ES_PATH_CONF environment variable.

Previous versions of Elasticsearch enabled using the CONF_DIR environment variable to specify a custom configuration directory for some configuration files and some scripts (it was used inconsistently). Starting in Elasticsearch 6.0.0, the usage of this environment variable has been superceded by ES_PATH_CONF, and this new environment variable is consistently used for all configuration files and scripts.

Previous versions of Elasticsearch enabled setting default.path.data and default.path.logs to set the default data path and default logs path if they were not otherwise set in the configuration file. These settings have been removed and now data paths and log paths can be configured via settings only. Related, this means that the environment variables DATA_DIR and LOG_DIR no longer have any effect as these were used to set default.path.data and default.path.logs in the packaging scripts.

Additionally, this means that if you were using the package distributions (i.e., you have installed Elasticsearch from the RPM or the DEB distributions), you had not previously explicitly configured path.data or path.logs, and you carry over your elasticsearch.yml file from 5.x, then you will need to add settings for path.data and path.logs. To use the defaults that you were implicitly using previously, you should add these lines to your elasticsearch.yml:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

(If you already had explicit values for either of these settings, you should of course preserve those). If you do not do this, Elasticsearch will refuse to start.

We previously attempted to ensure that Elasticsearch could be started on 32-bit JVM (although a bootstrap check prevented using a 32-bit JVM in production). We are no longer maintaining this attempt.

Related to the previous change that 32-bit is no longer supported, the default jvm.options file that ships with Elasticsearch no longer includes the -server flag. Previously, the installation of the Windows service would filter the -server flag because it is not compatible with the service. Since the default jvm.options file no longer includes -server, the filtering of this flag was removed from the installation of the Windows service. This means that if you have a jvm.options file that includes this flag you will need to remove it.

The environment variable ES_JVM_OPTIONS that enabled a custom location for the jvm.options file has been removed in favor of using the environment variable ES_PATH_CONF. This environment variable is already used in the packaging to support relocating the configuration files so this change merely aligns the other configuration files with the location of the jvm.options file.

The environment variable ES_INCLUDE could previously be used to establish the environment used to start Elasticsearch (and various supporting scripts). This legacy feature could be useful when there were several environment variables useful for configuring JVM options; this functionality had previously been replaced by Setting JVM options. Therefore, ES_INCLUDE has been removed.

Instead the percolate query should be used via either the search or msearch apis.

The percolator.map_unmapped_fields_as_text setting should be used instead.

Previous versions of Elasticsearch would skip hidden files and directories when scanning the plugins folder. This leniency has been removed.

The icu4j library has been upgraded to 59.1, Indices created in the previous major version will need to be reindexed in order to return correct (and correctly ordered) results, and to take advantage of new characters.

Previously, plugins could construct an Environment instance from Settings to discover the path to plugin-specific config files. This will no longer work in all situations as the Settings object does not carry the necessary information for the config path to be set correctly. Instead, plugins that need to know the config path should have a single constructor that accepts a pair of Settings and Path instances, and construct an Environment using the corresponding constructor on Environment.

The size parameter can no longer be explicitly set to -1. If all documents are required then the size parameter should not be set.

In previous versions of Elasticsearch, JSON documents were allowed to contain unquoted field names. This feature was removed in the 5.x series, but a backwards-compatibility layer was added via the system property elasticsearch.json.allow_unquoted_field_names. This backwards-compatibility layer has been removed in Elasticsearch 6.0.0.

In previous versions of Elasticsearch, documents were allowed to contain duplicate keys. Elasticsearch 6.0.0 enforces that all keys are unique. This applies to all content types: JSON, CBOR, Yaml and Smile.

In previous versions of Elasticsearch, having a proper Content-Type for the data in a request was not enforced. Elasticsearch 6.0.0 enforces that all requests with a body must have a supported Content-Type and this type will be used when parsing the data.

When using the source query string parameter, the source_content_type parameter must also be specified with the media type of the source.

All REST APIs parameters (both request parameters and JSON body) support providing boolean "false" as the value false and boolean "true" as the value true. All other values will raise an error.

The deprecated request parameters and plain text in request body has been removed. Define parameters in request body.

Analyze API can analyze normalizer and custom normalizer. In previous versions of Elasticsearch, Analyze API is requiring a tokenizer or analyzer parameter. In Elasticsearch 6.0.0, Analyze API can analyze a text as a keyword field with custom normalizer or if char_filter/filter is set and tokenizer/analyzer is not set.

timestamp and ttl are not accepted anymore as parameters of index/update requests.

Refresh requests that are broadcast to multiple shards that can have one or more shards fail during the request now return a 500 response instead of a 200 response in the event there is at least one failure.

In previous versions of Elasticsearch, delete by query requests without an explicit query were accepted, match_all was used as the default query and all documents were deleted as a result. From version 6.0.0, delete by query requests require an explicit query.

Running DELETE index/type/id now implicitly creates type with a default mapping if it did not exist yet.

Previously it was possible to execute GET /_aliases,_mappings or GET /myindex/_settings,_alias by separating multiple types of requests with commas in order to retrieve multiple types of information about one or more indices. This comma-separation for retrieving multiple pieces of information has been removed. GET /_all can be used to retrieve all aliases, settings, and mappings for all indices.

In order to retrieve only the mapping for an index use: GET /myindex/_mapping (or _alias for a list of aliases, or _settings for the settings).

Issuing a request to an endpoint that exists, but with an incorrect HTTP verb (such as a POST request to /myindex/_settings) now returns an HTTP 405 response instead of a 404. An Allow header is added to the 405 responses containing the allowed verbs. For example:

$ curl -v -XPOST 'localhost:9200/my_index/_settings'
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
> POST /my_index/_settings HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 405 Method Not Allowed
< Allow: PUT,GET
< content-type: application/json; charset=UTF-8
< content-length: 134
<
{
  "error" : "Incorrect HTTP method for uri [/my_index/_settings] and method [POST], allowed: [PUT, GET]",
  "status" : 405
}
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact

The _cache and _cache_key options in queries have been deprecated since version 2.0.0 and have been ignored since then, issuing a deprecation warning. These options have now been completely removed, so using them now will throw an error.

An IndexClosedException is returned whenever an api that doesn’t support closed indices (e.g. search) is called passing closed indices as parameters and ignore_unavailable is set to false. The response status code returned in such case changed from 403 to 400

The Groovy, JavaScript, and Python scripting languages were deprecated in elasticsearch 5.0 and have now been removed. Use painless instead.

Native scripts have been removed. Instead, implement a ScriptEngine.

doc.some_date_field.value now returns ReadableDateTimes instead of milliseconds since epoch as a long. The same is true for doc.some_date_field[some_number]. Use doc.some_date_field.value.millis to fetch the milliseconds since epoch if you need it.

The _index variable has been removed. If you used it for advanced scoring, consider writing a Similarity plugin.

All of the existing scripting security settings have been removed. Instead they are replaced with script.allowed_types and script.allowed_contexts.

The lang variable can no longer be specified as part of a request that uses a stored script otherwise an error will occur. Note that a request using a stored script is different from a request that puts a stored script. The language of the script has already been stored as part of the cluster state and an id is sufficient to access all of the information necessary to execute a stored script.

Stored scripts can no longer have the lang parameter specified as part of the url when performing PUT, GET, and DELETE actions on the _scripts/ path. All stored scripts must have a unique id as the namespace is only id now and no longer lang and id.

The PUT, GET and DELETE _search/template apis have been removed. Store search templates with the stored scripts apis instead.

For example, previously one might have stored a search template with the following:

PUT /_search/template/custom_template
{
  "query": {
    "match": {
      "f1": "{{f1}}"
    }
  }
}

And instead one would now use the following:

PUT /_scripts/custom_template
{
  "script": {
    "lang": "mustache",
    "source": {
      "query": {
        "match": {
          "f1": "{{f1}}"
        }
      }
    }
  }
}

The search shards API no longer accepts the type url parameter, which didn’t have any effect in previous versions.

The "time" field showing human readable timing output has been replaced by the "time_in_nanos" field which displays the elapsed time in nanoseconds. The "time" field can be turned on by adding "?human=true" to the request url. It will display a rounded, human readable time value.

Query normalization has been removed. This means that the TF-IDF similarity no longer tries to make scores comparable across queries and that boosts are now integrated into scores as simple multiplicative factors.

Other similarities are not affected as they did not normalize scores and already integrated boosts into scores as multiplicative factors.

See LUCENE-7347 for more information.

Coordination factors have been removed from the scoring formula. This means that boolean queries no longer score based on the number of matching clauses. Instead, they always return the sum of the scores of the matching clauses.

As a consequence, use of the TF-IDF similarity is now discouraged as this was an important component of the quality of the scores that this similarity produces. BM25 is recommended instead.

See LUCENE-7347 for more information.

Fielddata on _uid is deprecated. It is possible to switch to _id instead but the only reason why it has not been deprecated too is because it is used for the random_score function. If you really need access to the id of documents for sorting, aggregations or search scripts, the recommendation is to duplicate the id as a field in the document.

The unified highlighter is the new default choice for highlighter. The offset strategy for each field is picked internally by this highlighter depending on the type of the field (index_options). It is still possible to force the highlighter to fvh or plain types.

The postings highlighter has been removed from Lucene and Elasticsearch. The unified highlighter outputs the same highlighting when index_options is set to offsets.

The deprecated fielddata_fields have now been removed. docvalue_fields should be used instead.

docvalue_fields now have a default upper limit of 100 fields that can be requested. This limit can be overridden by using the index.max_docvalue_fields_search index setting.

script_fields now have a default upper limit of 32 script fields that can be requested. This limit can be overridden by using the index.max_script_fields index setting.

The source inside a hit of inner hits keeps its full path with respect to the entire source. In prior versions the source field names were relative to the inner hit.

The from parameter can no longer be used in the search request body when initiating a scroll. The parameter was already ignored in these situations, now in addition an error is thrown.

The maximum number of results (from + size) that is allowed to be retrieved via inner hits and top hits has been limited to 100. The limit can be controlled via the index.max_inner_result_window index setting.

Setting request_cache:true on a query that creates a scroll ('scroll=1m`) is deprecated and the request will not use the cache internally. In future versions we will return a 400 - Bad request instead of just ignoring the hint. Scroll queries are not meant to be cached.

Executing a Terms Query with a lot of terms may degrade the cluster performance, as each additional term demands extra processing and memory. To safeguard against this, the maximum number of terms that can be used in a Terms Query request has been limited to 65536. This default maximum can be changed for a particular index with the index setting index.max_terms_count.

For 6.x and starting in 6.3 a deprecation warning will be printed to warn against search requests that contain extra tokens after the main object. These extra tokens were ignored by the query parser before 6.3 but the next major version will not accept invalid body anymore.

The configuration file found in the Elasticsearch config directory could previously have a .yml, .yaml or .json extension. Only elasticsearch.yml is now supported.

In previous versions of Elasticsearch, the configuration file was allowed to contain duplicate keys. For example:

node:
  name: my-node

node
  attr:
    rack: my-rack

In Elasticsearch 6.0.0, this is no longer permitted. Instead, this must be specified in a single key as:

node:
  name: my-node
  attr:
    rack: my-rack

Previously, Elasticsearch recognized the strings true, false, on, off, yes, no, 0, 1 as booleans. Elasticsearch 6.0 recognizes only true and false as boolean and will throw an error otherwise. For backwards compatibility purposes, during the 6.x series index settings on pre-6.0 indices will continue to work. Note that this does not apply to node-level settings that are stored in elasticsearch.yml.

The internal setting cluster.routing.allocation.snapshot.relocation_enabled that allowed shards with running snapshots to be reallocated to different nodes has been removed. Enabling this setting could cause allocation issues if a shard got allocated off a node and then reallocated back to this node while a snapshot was running.

Store throttling has been removed. As a consequence, the indices.store.throttle.type and indices.store.throttle.max_bytes_per_sec cluster settings and the index.store.throttle.type and index.store.throttle.max_bytes_per_sec index settings are not recognized anymore.

The default index.store.type has been removed. If you were using it, we advise that you simply remove it from your index settings and Elasticsearch will use the best store implementation for your operating system.

The blocking TCP client, blocking TCP server, and blocking HTTP server have been removed. As a consequence, the network.tcp.blocking_server, network.tcp.blocking_client, network.tcp.blocking,transport.tcp.blocking_client, transport.tcp.blocking_server, and http.tcp.blocking_server settings are not recognized anymore.

The previously unused settings transport.netty.max_cumulation_buffer_capacity, transport.netty.max_composite_buffer_components and http.netty.max_cumulation_buffer_capacity have been removed.

The base similarity is now ignored as coords and query normalization have been removed. If provided, this setting will be ignored and issue a deprecation warning.

All of the existing scripting security settings have been removed. Instead they are replaced with script.allowed_types and script.allowed_contexts.

The discovery.type settings no longer supports the values gce, aws and ec2. Integration with these platforms should be done by setting the discovery.zen.hosts_provider setting to one of those values.

Given that store throttling has been removed, the store stats do not report throttle_time anymore.

Elasticsearch has defaulted to assuming that it is running on SSDs since the 2.x series of Elasticsearch. As such, Elasticsearch no longer needs to collect information from the operating system as to whether or not the underlying disks of each data path spin or not. While this functionality was no longer needed starting in the 2.x series of Elasticsearch, it was maintained in the filesystem section of the nodes stats APIs. This information has now been removed.