Upgrading the Elastic Stack

edit

Upgrading the Elastic Stack

edit

Before upgrading any component of the Elastic Stack, you should read through this guide to ensure that you upgrade in the right order and in the right way. When upgrading any component of the Elastic Stack, such as Beats, you should refer to the instructions for that component, including the breaking changes section.

Each component serves a particular role in the Elastic Stack. For some use cases, it’s normal to use a subset of components in the Elastic Stack. For example, you might use Elasticsearch, Logstash, and Kibana; Elasticsearch and Kibana; Elasticsearch and Beats; or just Elasticsearch.

Intended Audience

edit

This guide is intended for existing users of the Elastic Stack, running specific version ranges of each component:

Component Version

Beats

1.0 or later

Elasticsearch

2.0 or later

Kibana

4.2 or later

Logstash

2.0 or later

Elasticsearch Hadoop

2.2 or later

Marvel, Shield, Watcher, Graph, Reporting [1]

2.0 or later

Kibana 4.2 and Elasticsearch Hadoop 2.2 were the first versions that were compatible with Elasticsearch 2.x!

The spread of version numbers in the above table is the reason that the Elastic Stack is moving to a unified release number: 5.0. Beginning with 5.0, all of the components above will be released at the same time with the same version number. As such, you can choose a version with confidence across the entire stack.

It is critical to note that you cannot upgrade data that was written using Elasticsearch 1.x to Elasticsearch 2.x, then upgrade directly to Elasticsearch 5.x. Elasticsearch uses Lucene to store its data and Lucene is only compatible with the current version of Lucene, and one major release behind it. The Elasticsearch upgrade instructions do cover that path.

This includes system indices, such as the .kibana index that is created by Kibana, and snapshots of indices from Elasticsearch 1.x!

Elasticsearch Version Lucene Version Path

5.x

6.x

Rolling Upgrade

2.x

5.x

Full Cluster Restart Upgrade <1>

1.x

4.x

Reindexing Required

  1. Some features require reindexing to take advantage of them, such as Lucene’s new Block KD tree support.

When upgrading from two major releases ago, it is important to read the breaking changes from 1.x to 2.x, as well as from 2.x to 5.x!

Upgrade Order

edit

As noted above, this only applies for users of Beats 1.x, Elasticsearch 2.x, Logstash 2.x, and Kibana 4.2+. For Elasticsearch in particular, it is critical to run the Elasticsearch Migration Plugin prior to any upgrade to verify upgrade compatibility!

To maintain the most compatibility, you must upgrade the stack in the recommended order. You may skip any components that you do not use in your own system. The upgrade requires a full cluster shutdown for both Elasicsearch and Kibana because neither Elasticsearch 5.0, nor Kibana 5.0, can communicate with earlier versions of Elasticsearch.

  1. Elasticsearch Hadoop (can talk to Elasticsearch 5.0 and 2.x)
  2. Elasticsearch

    • X-Pack for Elasticsearch (combines Marvel Agent, Shield, Watcher, and Graph)
  3. Kibana (now includes Timelion and Console, formerly known as Sense)

    • X-Pack for Kibana (combines Marvel, Shield, Graph, and Reporting)
  4. Logstash
  5. Beats

Elasticsearch Hadoop versions prior to 5.0 are not compatible with Elasticsearch 5.0, but Elasticsearch Hadoop 5.0 is compatible with Elasticsearch 2.0 and Elasticsearch 5.0.

Logstash 2.0+ and Beats 1.0+ are compatible with both Elasticsearch 2.0+ and Elasticsearch 5.0. This provides flexibility in when you schedule the upgrades for each Logstash instance and Beats agent.

By following the above order, you should upgrade Elasticsearch Hadoop first; then Elasticsearch by performing a full cluster restart and upgrade; then install X-Pack; then upgrade Kibana immediately afterward by restarting and upgrading all instances of Kibana; then installing X-Pack there as well. Afterward, you can choose when it makes the most sense to upgrade Logstash and Beats for your architecture. It is worth upgrading Logstash and Beats as soon as possible to take advantage of performance improvements and other enhancements.

The following table lists the upgrade instructions and breaking changes for each component. Before upgrading, make sure you read through the upgrade guide and breaking changes list for every component that you are upgrading.

Upgrading on Elastic Cloud

edit

A single click in the Elastic Cloud console can upgrade a cluster to a newer version, add more processing capacity, change plugins, and enable or disable high availability, all at the same time. During the upgrade process, Elasticsearch, Kibana, X-Pack and the officially included plugins are all upgraded to the correct version together.

Although the upgrade process on Elastic Cloud is simple, you are still subject to breaking changes in Elasticsearch, and major version upgrades require a full cluster restart. Minor version upgrades and all other cluster configuration changes are performed with no downtime.

To avoid downtime on production clusters due to major version upgrades:

  1. Provision an additional cluster with the new Elasticsearch version, reindex your data, and send index requests to both clusters temporarily.
  2. Verify that the new cluster performs as expected, fix any problems, and then swap in the new cluster permanently.
  3. Delete the old cluster to stop incurring additional costs. You are billed extra only for the time that the additional cluster was running. Billing for usage is by the hour.

To learn more about the upgrade process on Elastic Cloud, see Upgrade Versions and Configuring Elastic Cloud.

[1] Marvel, Shield, Watcher, Graph, and Reporting have all been combined into a new, unified plugin called X-Pack. Unlike before, the same X-Pack distribution works for both Elasticsearch and Kibana.