Upgrade to Elastic 9.0.0-beta1

Upgrading to the latest version provides you access to Elastic latest features, enhancements, performance improvements, and bug fixes, many of which enable you to save your organization money, respond faster to potential threats, and improve the tools you use to investigate and analyze your data. As new versions are released, older versions reach their end of life at a regular cadence, so it’s important to ensure that your deployment is fully maintained and supported. For more information, refer to Elastic’s Product End of Life Dates.

Before you upgrade, there are a number of things that you need to plan for before performing the actual upgrade, so create a test plan. Consider the following recommendations:

Check system requirements

Ensure the version you’re upgrading to for Elasticsearch, Kibana, and any ingest components supports your current operating system. Refer to the Product and Operating System support matrix.

JVM and FIPS compliance

By default, Elasticsearch includes a bundled Java Virtual Machine (JVM) supported by Elastic. While we strongly recommend using the bundled JVM in all installations of Elasticsearch, if you choose to use your own JVM, ensure it’s compatible by reviewing the Product and JVM support matrix. Elasticsearch 9.0+ requires Java 21 or later.

If you’re running Elasticsearch in FIPS 140-2 mode, Elasticsearch 9.0+ has been tested with Bouncy Castle's FIPS implementation and is the recommended Java security provider when running Elasticsearch.

Conduct a component inventory

It is very important to map all the components that are being used on the Elastic Stack. When you upgrade your deployment, you may also need to upgrade all the other components. You should record if each component is used, and if it is, you should record its current version. While this is not a comprehensive list, here are some components you should check:

The procedures you follow to upgrade depend on your infrastructure and deployment method. You’ve installed Elastic components using either Elastic-managed infrastructure or self-managed infrastructure.

Elastic-managed infrastructure includes Elastic Cloud – the umbrella term for Elastic Cloud Hosted (ECH) and Elastic Cloud Serverless. Elastic Cloud Serverless (“Serverless”) is the fully managed cloud offering and has three products: Elasticsearch Serverless, Elastic Observability Serverless, and Elastic Security Serverless. All serverless products are built on top of the Search AI Lake. Customers receive the latest features automatically when updates are published.

Elastic Cloud Hosted is Elastic’s cloud offering for managing Elastic Stack deployments, built on top of Elasticsearch. A single click in the Elastic Cloud console can upgrade a deployment to a newer version.

Self-managed infrastructure — either on-prem or on public cloud, includes:

For ECE and ECK, you need to ensure the orchestrator is running a compatible version with the Elastic Stack version you’re upgrading to. If not, you need to upgrade your orchestrator before you can upgrade your cluster.

If you’re running the Elastic Stack on your own self-managed infrastructure, you’ll need to upgrade each component individually.

Before you upgrade to 9.0.0-beta1, it’s important to take some preparation steps. Unless noted, these following recommendations are best practices regardless of deployment method.

To upgrade to 9.0.0-beta1 from 8.17 or earlier, you must first upgrade to the latest patch version of 8.18. This enables you to use the Upgrade Assistant to identify and resolve issues, reindex indices created before 8.0, and then perform a rolling upgrade.

Upgrading to 8.18 before upgrading to 9.0.0-beta1 is required even if you opt to do a full-cluster restart of your Elasticsearch cluster.

Alternatively, you can create a new 9.0.0-beta1 deployment and reindex from remote. For more information, refer to Reindex to upgrade.

Beats and Logstash 8.17 are compatible with Elasticsearch 9.0.0-beta1 to give you flexibility in scheduling the upgrade.

The anomaly detection result indices .ml-anomalies-* created in Elasticsearch 7.x must be either reindexed, marked read-only, or deleted before upgrading to 9.x.

Reindexing: While anomaly detection results are being reindexed, jobs continue to run and process new data. However, you cannot completely delete an anomaly detection job that stores results in this index until the reindexing is complete.

Marking indices as read-only: This is useful for large indexes that contain the results of only one or a few anomaly detection jobs. You need to update or delete all obsolete model snapshots before using this option. If you delete these jobs later, you will not be able to create a new job with the same name.

Deleting: Delete jobs that are no longer needed in the Machine Learning app in Kibana. The result index is deleted when all jobs that store results in it have been deleted.

GET /.ml-anomalies-*/_migration/deprecations

  "index_settings": {
    ".ml-anomalies-shared": [
      {
        "level": "critical",
        "message": "Index created before 8.0",
        "url": "https://ela.st/es-deprecation-8-reindex",
        "details": "This index was created with version 7.8.23 and is not compatible with 9.0. Reindex or remove the index before upgrading.",
        "resolve_during_rolling_upgrade": false
      }
    ]
  }

GET _cat/indices/.ml-anomalies-custom-example?v&h=index,store.size

PUT .ml-anomalies-custom-example/_block/write

GET .ml-anomalies-custom-example/_search
{
  "size": 0,
  "aggs": {
    "job_ids": {
      "terms": {
        "field": "job_id",
        "size": 100
      }
    }
  }
}

The transform destination indices created in Elasticsearch 7.x must be either reset, reindexed, or deleted before upgrading to 9.x.

Resetting: You can reset the transform to delete all state, checkpoints, and the destination index (if it was created by the transform). The next time you start the transform, it will reprocess all data from the source index, creating a new destination index in Elasticsearch 8.x compatible with 9.x. However, if data had been deleted from the source index, you will lose all previously computed results that had been stored in the destination index.

Reindexing: You can reindex the destination index and then update the transform to write to the new destination index. This is useful if there are results that you want to retain that may not exist in the source index. To prevent the transform and reindex tasks from conflicting with one another, you can either pause the transform while the reindex runs, or you can write to the new destination index while the reindex backfills old results.

Deleting: You can delete any transform that are no longer being used. Once the transform is deleted, you can either delete the destination index or make it read-only.

GET /_migration/deprecations

"index_settings": {
    "my-destination-index": [
      {
        "level": "critical",
        "message": "One or more Transforms write to this index with a compatibility version < 9.0",
        "url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/migrating-9.0.html#breaking_90_transform_destination_index",
        "details": "Transforms [my-transform] write to this index with version [7.8.23].",
        "resolve_during_rolling_upgrade": false
      }
    ]
  }

POST _transform/my-transform/_reset

GET _cat/indices/.transform-destination-example?v&h=index,store.size

DELETE _transform/my-transform

DELETE _transform/my-transform?delete_dest_index