Migrating to 7.16

Details
Orchestrated environments such as Elasticsearch Service and Elastic Cloud Enterprise rely on the disk thresholds in Elasticsearch to operate the cluster correctly. For example the disk thresholds help determine how large an auto-scaled cluster should be. Disabling these disk thresholds prevents the orchestration system from working correctly, so starting in 7.16.0 the cluster.routing.allocation.disk.threshold_enabled setting is an operator only setting which cannot be changed by end-users.

Impact
Discontinue use of this setting in orchestrated environments such as Elasticsearch Service and Elastic Cloud Enterprise. Contact the environment administrator for help with disk space management if needed.

This change has no impact on users outside of orchestrated environments.

Details
The XContent library exported package is renamed from org.elasticsearch.common.xcontent to org.elasticsearch.xcontent. This change breaks imports when migrating to Elasticsearch 7.16.

Impact
If you’re maintaining a Java client application that uses the Java High Level Rest Client (HLRC), update any import statements in your Java source code that match:

import org.elasticsearch.common.xcontent.<class>

to instead use:

import org.elasticsearch.xcontent.<class>

This is the minimum required change. You must then recompile your source code to work with Elasticsearch 7.17.

A more permanent solution is to migrate from the (HLRC) entirely.

Details
In #77864 the High Level Rest Client was changed to only send indices options if the request differed from the default request options. However, in some cases the default options for the HLRC request object inadvertently differ from the effective options of the Elasticsearch APIs, meaning that it is possible for the API response to differ.

Impact
If you use the HLRC and have a differing response due to indices options, you can update the options using the indicesOptions(...) method, as shown below:

// Previously:
highLevelClient.indices().exists(request, RequestOptions.DEFAULT);

// With indices options (change boolean options as needed):
final var requestWithOptions = request.indicesOptions(IndicesOptions.fromOptions(false, false, true, false));
highLevelClient.indices().exists(requestWithOptions, RequestOptions.DEFAULT);

A more permanent solution is to migrate from the (HLRC) entirely.

Details
In SAML, Identity Providers (IdPs) can either be explicitly configured to release a NameID with a specific format, or configured to attempt to conform with the requirements of a Service Provider (SP). The SP declares its requirements in the NameIDPolicy element of a SAML Authentication Request. In Elasticsearch, the nameid_format SAML realm setting controls the NameIDPolicy value. Previously, the default value for nameid_format was urn:oasis:names:tc:SAML:2.0:nameid-format:transient. This setting created authentication requests that required the IdP to release NameID with a transient format. The default value has been removed, which means that Elasticsearch will create SAML Authentication Requests by default that don’t put this requirement on the IdP. If you want to retain the previous behavior, set nameid_format to urn:oasis:names:tc:SAML:2.0:nameid-format:transient.

Impact
If you currently don’t configure nameid_format explicitly, it’s possible that your IdP will reject authentication requests from Elasticsearch because the requests do not specify a NameID format (and your IdP is configured to expect one). This mismatch can result in a broken SAML configuration. If you’re unsure whether your IdP is explicitly configured to use a certain NameID format and you want to retain current behavior , try setting nameid_format to urn:oasis:names:tc:SAML:2.0:nameid-format:transient explicitly.

Details
Configuring any SSL settings for xpack.security.transport.ssl without also configuring xpack.security.transport.ssl.enabled generates warnings in the deprecation log. In 8.0, this configuration will result in errors.

Impact
To avoid deprecation warnings, either:

If you want to enable SSL, follow the instructions to encrypt internode communications with TLS. As part of this configuration, explicitly set xpack.security.transport.ssl.enabled as true.

For example:

xpack.security.transport.ssl.enabled: true 
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

Details
Configuring any SSL settings for xpack.security.http.ssl without also configuring xpack.security.http.ssl.enabled generates warnings in the deprecation log. In 8.0, this configuration will result in errors.

Impact
To avoid deprecation warnings, either:

If you want to enable SSL, follow the instructions to encrypt HTTP client communications for Elasticsearch. As part of this configuration, explicitly set xpack.security.http.ssl.enabled as true.

For example:

xpack.security.http.ssl.enabled: true 
xpack.security.http.ssl.certificate: elasticsearch.crt
xpack.security.http.ssl.key: elasticsearch.key
xpack.security.http.ssl.certificate_authorities: [ "corporate-ca.crt" ]

Details
Enabling SSL for the transport interface without also configuring a certificate and key through use of the xpack.security.transport.ssl.keystore.path setting or the xpack.security.transport.ssl.certificate and xpack.security.transport.ssl.key settings generates warnings in the deprecation log. In 8.0, this configuration will result in errors.

Impact
If xpack.security.transport.ssl.enabled is set to true, provide a certificate and key using the xpack.security.transport.ssl.keystore.path setting or the xpack.security.transport.ssl.certificate and xpack.security.transport.ssl.key settings. If a certificate and key is not provided, Elasticsearch will generate warnings in the deprecation log.

Details
Enabling SSL for the HTTP layer without also configuring a certificate and key through use of the xpack.security.http.ssl.keystore.path setting or the xpack.security.http.ssl.certificate and xpack.security.http.ssl.key settings generates warnings in the deprecation log. In 8.0, this configuration will result in errors.

Impact
If xpack.security.http.ssl.enabled is set to true, provide a certificate and key using the xpack.security.http.ssl.keystore.path setting or the xpack.security.http.ssl.certificate and xpack.security.http.ssl.key settings. If a certificate and key is not provided, Elasticsearch will generate warnings in the deprecation log.

Details
The ILM freeze action is now deprecated. This is because frozen indices provide no benefit given improvements in heap memory utilization. In 8.0 the freeze action will be a no-op and perform no action on the index, as the freeze API endpoint has been removed in 8.0.

Impact
Update your ILM policies to remove the freeze action from the cold phase.

Details
Using the Monitoring plugin to collect and ship monitoring data is deprecated. Metricbeat is the recommended method for collecting and shipping monitoring data to a monitoring cluster.

Impact
If you have previously configured legacy collection methods, you should migrate to using Metricbeat collection. Learn more about Collecting Elasticsearch monitoring data with Metricbeat.

The following settings are now deprecated:

Details
The default alerts from Monitoring plugin will no longer be installed by default in 8.0.0. The recommended method for alerting based on monitoring data is Kibana alerts.

Impact
If you have previously configured alerts from the Monitoring plugin, you should migrate to using Kibana alerts instead. Alerts installed from previous versions will continue to function until uninstalled. Learn more about Kibana alerts.

Details
The xpack.monitoring.exporters.*.use_ingest property has been deprecated in 7.16.0 and will be removed in a release after 8.0.0. This parameter controls the creation of pipelines for monitoring indices. These pipelines currently have no function.

Impact
Discontinue the use of the xpack.monitoring.exporters.*.use_ingest setting as it will have no functionality in 8.0.0 and will eventually be removed.

Details
The xpack.monitoring.exporters.*.index.pipeline.master_timeout property has been deprecated in 7.16.0. This parameter sets the timeout when waiting for the remote Monitoring cluster to create pipelines. These pipelines for monitoring indices currently have no function and will be removed in a release after 8.0.0.

Impact
Discontinue the use of the xpack.monitoring.exporters.*.index.pipeline.master_timeout setting as it will have no functionality in 8.0.0 and will eventually be removed.

Details
The xpack.monitoring.exporters.*.index.template.create_legacy_templates property has been deprecated in 7.16.0. This parameter instructs the exporter to install the previous version of monitoring templates on the monitoring cluster. These older templates were meant to assist in transitioning to the current monitoring data format. They are currently empty and are no longer of any use.

Impact
Discontinue the use of the xpack.monitoring.exporters.*.index.template.create_legacy_templates setting as it will have no functionality in 8.0.0 and will eventually be removed.

Details
The estimated_heap_memory_usage_bytes property in the create trained models API is deprecated in 7.16.

Impact
Use model_size_bytes instead.

Details
We no longer recommend using transient cluster settings. Use persistent cluster settings instead. If a cluster becomes unstable, transient settings can clear unexpectedly, resulting in an undesired cluster configuration.

Impact
Transient cluster settings are not yet deprecated, but we plan to deprecate them in a future release. For migration steps, see the Transient settings migration guide.

Details
Directly accessing system indices is deprecated. In Elasticsearch 8.0, you must add the allow_restricted_indices permission set to true on a user role to access system indices. Refer to indices privileges for information on adding this permission to an index privilege.

Impact
Accessing system indices directly results in warnings in the header of API responses and in the deprecation logs. Use Kibana or the associated feature’s Elasticsearch APIs to manage the data that you want to access. While it’s still possible to access system indices in Elasticsearch 7.16, they are reserved only for internal use by Elastic products and should not be accessed directly.

Details
Setting script.max_compilations_rate to use-context and configuring context-specific caches is deprecated.

Context-specific caches are no longer needed to prevent system scripts from triggering rate limits.

Configure the general cache to control the max compilation rate, cache size, and cache expiration for your scripts.

Impact
Remove script.max_compilations_rate: use-context and the context-specific cache settings: script.context.$CONTEXT.cache_max_size, script.context.$CONTEXT.max_compilations_rate, script.context.$CONTEXT.cache_expire.

The general cache defaults to a max size of 3000 with a rate limit of 150/5m, which allows 150 compilations per 5 minute period. By default, the entries in the cache do not expire.

To override the defaults, configure the script.cache.max_size, script.max_compilations_rate, and script.cache.expire settings.

Details
After 7.16.2, we’ll no longer release Windows MSI installer packages for Elasticsearch. These packages were previously released in beta and didn’t receive widespread adoption.

Impact
To install Elasticsearch on Windows, use the .zip archive package instead.