Elastic Cloud Enterprise 3.0.0

Introducing Integrations Server. You can now use Integrations Server to add a combined Application Performance Monitoring (APM) Server and Fleet Server to your deployment. APM allows you to monitor software services and applications in real time, turning that data into documents stored in the Elasticsearch cluster. Fleet allows you to centrally manage Elastic Agents on many hosts. To get started using Integrations Server in Elastic Cloud Enterprise, check Manage your Integrations Server.

Implement per-target resource connection limits for the HTTP listener. Allow the user to setup per-target resource connection limits. Note that this is currently limited to HTTP(S) connections.

Make X-Strict-Request-Payload header toggle-on or inhibit strict deserialization. Previously, the X-Strict-Request-Payload API header was only able to inhibit configuration-enabled strict de-serialization. With this change, it permits enabling strict de-serialization in requests including it with value true, even when the global setting is off.

Add 8.0.0-SNAPSHOT to bundled ECE versions. This release of Elastic Cloud Enterprise comes with 8.0.0-SNAPSHOT.

Formatting and display adjustments to PDF invoices template. Fixed minor layout issues in PDF statements and invoices.

Speed up some deplyment changes. At the end of plan changes that do not use the RollingGrowShrink strategy, non-master instances will all be stopped at the same time instead of one at a time. This will make some plans a bit faster.

Add audit log ignore filter by actions setting to allowlist. Allow setting to ignore audit event based on the content of the audit log actions field. Check the audit logfile event ignore policies documentation for more details.

Add Deployment API endpoint for upgrade assistant Status. The deployments API can now be queried to get the upgrade status. The deployments upgrade assistant status endpoint will provide information about whether the deployment is ready to be upgraded to the next major version or not.

Remove the max limit for the APM setting bulk_max_size. Remove the upper limit from the APM output.elasticsearch.bulk_max_size configuration.

Show plan warnings on activity page. Plans which use "move_only: true" will no longer fail with a message about ignored/unapplied changes. Instead, those plans will succeed, but will show a warning that there are unapplied changes.

Reduce usage of transient settings by constructor. This release reduces the constructors reliance on transient section for a few frequently used settings by having them always be set to null in the transient section whenever they’re added in the persistent section.

Update the default sizes of instances in the ECE default template. The default template now includes machine learning, APM and Fleet, and Enterprise Search by default in ECE.

Prevent Route Server from failing if no routes exist. If an entire route is deleted from the route-server we will now mark the file as soft-deleted, and at a later point actually delete the route. This operates using the existing graveyard-expiration and graveyard-interval options. This has the minor side effects of routes that have recently been deleted returning a 502 instead of a 404 when requested via the proxy. Additionally this should resolve some amount of issues around flapping routes, which in turn get cached (due to being 404s) even after they are no longer flapping.

Add Kibana’s migrations.maxBatchSizeBytes config option. Adds support for Kibana’s migrations.maxBatchSizeBytes Kibana user setting.

Enable Node Shutdown everywhere. When executing plans against Elasticsearch clusters with version 7.15.2 or greater, if the plan stops nodes, a new step "Calling Elasticsearch node shutdown API" will replace the "Migrating shard data" step.

Include zkstat tool in the ECE image thus enabling the path to generate ECE diagnostics with zkstats. Make zkstat tool generally available through the elastic-cloud-enterprise image to help operators and customers when debugging ECE Zookeeper issues.

Remove xpack.searchable.snapshot.shared_cache.size for versions >= 7.13.0. Remove the setting xpack.searchable.snapshot.shared_cache.size for versions >= 7.13.0

Beats upgrade to 7.12. Filebeat and Metricbeat used to ingest logs and metrics into the logging-and-metrics system cluster are upgraded to version 7.12.

Collect proxy metrics. Ingest proxy metrics into logging-and-metrics on ECE.

Add cross-org trust flyout. Bidirectional trust can now be established between deployments in different Elastic Cloud organizations.

Update user settings on major version upgrades that will otherwise fail. Major version upgrades will now prompt for changes to incompatible user settings.

Enable the Kibana user setting execution_context.enabled: boolean. It might be hard, in the distributed environment, to understand which Kibana entity initiated a request to the Elasticsearch server. To overcome this problem Kibana introduced the execution_context service to attach domain-specific metadata to every request. execution_context propagates specified metadata from the Kibana browser app or Kibana server to the Elasticsearch server by way of the x-opaque-id header emitted to the Elasticsearch slow logs.

To enable execution_context propagation from Kibana to Elasticsearch logs, set execution_context.enabled: true. To disable the execution_context feature set execution_context.enabled: false.

Prevent users from deleting Elasticsearch Settings when updating other settings (using the Deployments API). The endpoint to update deployments supports updating the settings of the Elasticsearch resource (PUT "/api/v1/deployments/{deployment_id}").

PUT "api/v1/deployments/{deployment_id}" -X PUT --data '{
  "prune_orphans": false,
  "resources": {
    "elasticsearch": [
      {
        "ref_id": "main-elasticsearch",
        "region": "us-east-1",
        "plan": {
          "cluster_topology": [...],
          "elasticsearch": {
            "version": "7.16.0"
          },
          "deployment_template": {
            "id": "default"
          }
        },
        "settings": {
          "dedicated_masters_threshold": 2 <-- THESE SETTINGS
        }
      }
    ]
  }
}'

Previously, the settings passed in the body would override all the settings (even the ones not included in the request). Now, the settings not included in the request will be kept and only the ones included will be passed. In order to delete a setting, it has to be passed as null.

For example, this will update just the dedicated_masters_threshold and the snapshot settings, leaving the rest unmodified (trust, curation, monitoring…​ and so on)

"settings": {
          "dedicated_masters_threshold": 10,
          "snapshot": {
            "enabled": false
          }
        }

This would delete the snapshot setting:

"settings": {
          "snapshot": {
            "enabled": null
          }
        }

Update Zookeeper version. Internal service upgrade: Zookeeper has been upgraded from version 3.5 to 3.7.

Update the Elasticsearch deprecation logger. Incorporate version 7.16+ deprecation level (CRITICAL/WARN) changes https://github.com/elastic/elasticsearch/pull/77030

Naive implementation always clearing any archived settings. During upgrade to Elasticsearch 8.0 and later Elastic Cloud Enterprise will now clear any old discovery settings left over by version upgrades.

Run ConfigureAutoExpandIndexes less often. Step "Adjusting auto-expand indexes in Elasticsearch" will no longer run during plans on Elasticsearch clusters >= 7.13.0

Remove Enterprise Search standalone UI in version 8.0 and higher deployments. Removes the "Open" link for Enterprise Search on deployments >=8.0, and the "Copy endpoint" link results in a URL to / instead of /login.

Allow overlapping SLM schedules. Allows deployments running Elastic Stack version 7.12.1 or higher to change the SLM retention schedule. In previous versions, changes to this schedule are reverted due to a race condition within Elasticsearch.

Update the Elasticsearch deprecation log. Updates the Elasticsearch log4j config for version 7.17 to contain the additional field x-elastic-product-origin which can be used to filter deprecation log entries. https://github.com/elastic/elasticsearch/pull/81381

display customers' current hourly rate for trial converted customers. Fixed a bug where the hourly rate value displayed in Account & Billing > Usage was always zero for trial converted users.

Fix user access to API for 5.x clusters. Fixed the Elasticsearch API console for 5.x clusters. All requests were refused with Endpoint is unavailable right now. Please retry in 30s.. This was due to a new user for the API console that was not supported by 5.x clusters.

Retain autoscaling values on change instance configuration. Fixed a bug that removes autoscaling values when switching to a custom instance configuration when creating a deployment template.

Cannot reapply a plan through the UI on a deployment with a disabled Enterprise Search application. Fixed a bug that would prevent re-applying previous changes through the UI if the deployment had a disabled Enterprise Search application.

Fix bug in heap dump compression. Fixed an issue where OOM heap dump compression would take hours while an instance is unavailable.

Do not restart after update of disk quota. Fixed a bug where, when adding a disk quota override in the UI, the instance would restart. The behaviour is now that the instance does not restart.

Add rate limiting filter to log delivery deprecation logger. Added a rate limiting filter to log delivery deprecation logs to reduce duplicate deprecation logs.

Don’t start extra instances when upgrading stateless clusters. This fixes a bug where the upgrade process attempted to start unrequired resources. Now, Kibana, APM, and Enterprise Search upgrade plans will succeed more often in cases where previous failed plans left behind mixed instance versions.

Add missing OIDC setting to allowlist. All OIDC settings should be allowed on Elastic Cloud Enterprise. The rp.client_auth_method setting had been omitted and has now been added to the allowlist for OIDC realms.

Make a field optional for the sake of Elasticsearch 2.x. Fixed a bug that would cause plans for Elasticsearch versions older than 2.4.1 to fail with an error message that included "Null value for creator property cluster_uuid".

Stop overwriting templates from Filebeat. Changed log delivery Filebeat to disable setup.template.overwrite. This flag was causing many PUT _template/elastic-cloud-logs-* requests which could cause increased load on monitoring deployments when using log delivery.

Re-add supporting alias lookup by X-Found-Cluster. Allow requests to resolve hosts by alias passed in the X-Found-Cluster header.

Show operations when pending plan is running if it isn’t a create plan. Fixed a bug where the Operations page was not displayed when a plan is in progress.

Fix deserialization of slm.retention_duration. Fixed a bug that would cause most plans to fail for clusters which have a non-null value set for slm.retention_duration in the Elasticsearch cluster settings.

For the full list of features and API endpoints that are being removed with ECE 3.0, and migration options where appropriate, check Removals for ECE 3.0.

App Search is no longer supported. App Search is no longer supported. Upgrading to ECE version 3.0 requires first to migrate all deployments that have App Search to use Enterprise Search. Check Upgrading App Search to Enterprise Search for instructions on how to migrate.

Curator is removed. The logs and metrics curator component has been removed from ECE. The curator component is responsible for enforcing log and metrics retention on the logs and metrics cluster cluster (not to be confused with index curation or the curator ZooKeeper framework). The functionality of curator has been replaced by installing ILM policies and attaching them to the system logs and metrics index templates.

Remove legacy cross cluster search API. Cross cluster search is no longer supported on ECE for Elastic Stack versions below 6.7.0.

Delete old stackpacks during ECE upgrade. This release of ECE removes all Elastic Stack versions below 6.0. If there are live clusters currently on any of these versions, the upgrade is stopped.

Reject upload of stackpacks older than 6.0. Elastic Stack packs with versions below 6.0 are no longer supported and cannot be uploaded to ECE.

Remove deprecated CORS header handling functionality. The deprecated handling of Cross-Origin Resource Sharing (CORS) headers was removed from the proxy. These headers will be handled by the deployment components such as Elasticsearch, Kibana, and others.

Remove deprecated plan parameters. The following API parameters have been removed and are no longer supported in ECE:

Node attributes not included by default. New deployments created in the Cloud UI and based on Elastic Stack versions 8.0 and higher will not have node attributes defined on data tiers. If these attributes are required, they can be added directly using the API.

Some APM user settings have been removed. The following APM settings have been removed in Elastic Stack version 8.0. This change is only relevant to users upgrading a standalone (legacy) deployment of APM Server to Elastic Stack version 8.0. Check Add APM user settings for more details.

apm-server.api_key.enabled
apm-server.api_key.limit
apm-server.ilm.*
apm-server.frontend.*
apm-server.register.*
apm-server.rum.allow_service_names
apm-server.rum.event_rate.lru_size
apm-server.rum.event_rate.limit
apm-server.rum.rate_limit
output.elasticsearch.bulk_max_size
output.elasticsearch.index
output.elasticsearch.indices
output.elasticsearch.pipeline
output.elasticsearch.pipelines
output.elasticsearch.worker
setup.*
queue.*

Remove Enterprise Search login_assistance_message in 8.0+. The Enterprise Search login_assistance_message setting is removed in Elastic Stack versions 8.0 and higher.

Remove APM payload in deployments API. Beginning with Elastic Stack version 8.0, the Apm payload in the deployments API is deprecated. New deployments should be created using the integrations_server payload. After the Elastic Stack 8.0 release is available, refer to the Definitions section of the API documentation to see the expected payload format.

Avoid verbose SLM queries. To improve performance, deployments which use SLM (Snapshot Lifecycle Management) no longer display the basic snapshot list in the Elastic Cloud UI. The full snapshot interface is available in Kibana, linked to from the same Deployment > Snapshots page. Additionally, when restoring from a snapshot in the Elastic Cloud UI, if the source deployment uses SLM, the name of a snapshot must be entered directly instead of selected from a dropdown.

Deprecate Templates - Deployments API. The Templates - Deployments - Legacy API is now deprecated and the associated endpoints will be removed in the next major release. Instead, you should now use the Deployments - Templates API.

Update max versions for apm_oss settings to 8.0.0. Elastic Stack version 8.0 is the maximum supported version for the following APM user settings: