Cross-region and cross-provider support, warm and cold data tiers, and more

Added support for cross-cluster search (CCS) and replication (CCR) across-regions and cloud providers. Break down data silos by searching across remote clusters regardless of where they are deployed. Increase data redundancy by storing copies of your data with CCR to ensure, in the event of a regional failure, a secondary, remote cluster can be used as a hot backup. Reduce search latency by storing copies of your data in a location nearer to the user. Check Enable cross-cluster search and cross-cluster replication.

Add an optional warm and cold tier in all 3+1 deployments. All built-in deployment templates have been extended to include an optional warm and cold data tier. You can now enable a warm tier on any type of deployment, and for deployments on Elastic Stack version 7.10 and higher, you can take advantage of snapshot-backed indices in a dedicated cold data tier and store much more data at the same cost. Check Data tiers.

Support for the Elasticsearch node.roles configuration in the CRUD deployments API. When creating or updating a deployment on Elastic Stack version 7.10 and higher, users can now configure the Elasticsearch node roles by using the node_roles field instead of the node_type field in the ElasticsearchClusterTopologyElement.

Better logging on EnsureRepository. Improves the error reporting for failures that occur while creating snapshot repositories.

Prevent plans termination and creation of resources of the same kind. Adds a validation to check that resources of the same kind are not terminated and added to a single plan. This can happen if the refId of a resource is changed, which would cause issues.

Better explanation message when allocators are vacated. Improves the error message Moving nodes off allocator by adding additional context.

Respect kibana authc selector user prefs. When configuring Kibana in a deployment, users are now able to set xpack.security.authc.selector.enabled: false without being overwritten by internal Cloud SSO settings.

Do not remove zero-sized tiers from create request. Adds zero-sized tiers to the API Create Deployment request, to support having optional data tiers available in all hardware profiles.

Add Workplace Search thumbnail enable/disable configuration. Adds a workplace_search.content_source.sync.thumbnails.enabled configuration for Enterprise Search, which determines whether thumbnails are generated for synced documents.

Added support for new generation storage-optimized AWS D3 instances. The new generation instances offer better performance with a reduced price per GB of storage. The new hardware will replace the AWS D2 machine and is used for both warm and cold data nodes, allowing you to automatically move data that is less frequently searched to more cost-effective hardware. We currently offer D3 in the following regions: Ireland (EU West-1), North Virginia (US East-1), Ohio (US East-2), and Oregon (US West-2).

Handle terminated deployments and missing templates on Edit screen. Editing terminated or certain system deployments should no longer throw an error.

Apply correct timestamps to downloaded bundles. Fixes a bug that could cause instances to bootloop during rolling plans if a cluster is configured with user bundles.

Use recommended JVM heap allocation for dedicated masters. Fixes an issue where dedicated masters can go out of memory due to over-allocated heap size.

Use smaller Elasticsearch heap when Filebeat and Metricbeat are running. Fixes a bug where small, master-only instances and tiebreaker instances have memory swapping issues when logs and metrics are enabled on Elasticsearch clusters.

Make some Elasticsearch domain fields optional. Fixes a bug that might cause plans to fail during the "Migrating shard data" step.

Optimize ML cluster settings. Some plans could fail if an ML node is offline, even if the failsafe flag is set. This change lets plans continue if the ‘Optimize ML Cluster settings’ step fails and the failsafe flag is set.

Strip hop-by-hop headers. Removes the hop-by-hop headers from both request and response, and removes any header specified in the Connection header.

Log metrics UI errors, Kibana and 24 hour logs.

Hot-Warm Architecture deployment template is being deprecated. Now that all deployment templates include an optional warm tier on on any deployment, the dedicated Hot-Warm Architecture is deprecated and no longer available in the user console when creating new deployments. Existing deployments created with the Hot-Warm template are not affected and will also have access to the cold data tier when using a compatible 7.10+ version.

Add Workplace Search content source limit configurations. The following configurations are deprecated and will be removed in Elastic Stack version 8.0.0:

Use the following configurations instead:

The cross-cluster search deployment template is now deprecated. The cross-cluster search deployment template has been removed from the Elasticsearch Service user console. You no longer need to use the dedicated cross-cluster template to search across deployments. Instead, you can now use any template to configure remote clusters and search across them when using stack version 6.7 or above. Existing deployments created using this template are not affected, but they are required to migrate to another template before upgrading to the next 8.0 major version.

Remove basicAuth from the ESS API documentation. Updated the API documentation to remove basic authentication from supported authentication mechanisms for Elastic Cloud.

Add SAML to the ESS API documentation. Support for Elastic Stack version 7.11 includes a new SAML API.