IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Upgrade between minor versions Troubleshooting Beats upgrade issues »
Elastic Docs ›Beats Platform Reference [7.3] ›Upgrading

Upgrade from 6.x to 7.x

edit

Upgrade from 6.x to 7.x

edit

Before upgrading your Beats, review the breaking changes and the Release notes.

If you’re upgrading other products in the stack, also read the Elastic Stack Installation and Upgrade Guide.

We recommend that you fully upgrade Elasticsearch and Kibana to version 7.0 before upgrading Beats. If you’re on Beats 6.0 through 6.6, upgrade the Elastic Stack and Beats to version 6.7 before proceeding with the 7.0 upgrade.

Upgrading between non-consecutive major versions (e.g. 5.x to 7.x) is not supported.

Please read through all upgrade steps before proceeding. These steps are required before running the software for the first time.

Upgrade to Beats 6.7 before upgrading to 7.0

edit

The upgrade procedure assumes that you have Beats 6.7 installed. If you’re on a previous 6.x version of Beats, upgrade to version 6.7 first. If you’re using other products in the Elastic Stack, upgrade Beats as part of the Elastic Stack upgrade process.

Upgrading to 6.7 is required because the Elasticsearch index template was modified to be compatible with Elasticsearch 7.0 (the _type setting changed from doc to _doc).

After upgrading to 6.7, use the Index Template API to verify that the 6.7 index template has been created in Elasticsearch.

To load the the index template, run the setup command with the --template option specified. For example, if the Beats output is Elasticsearch, run:

deb and rpm:

metricbeat setup --template

mac and linux:

./metricbeat setup --template

docker:

docker run docker.elastic.co/beats/metricbeat:7.0.0 setup --template

win:

PS > .\metricbeat.exe setup --template

If the Beats output is not elasticsearch, temporarily enable the elasticsearch output when you run the setup command. For example:

deb and rpm:

metricbeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

mac and linux:

./metricbeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

docker:

docker run docker.elastic.co/beats/metricbeat:7.0.0 setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

win:

PS > .\metricbeat.exe setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

In previous versions, we advised users to manually force loading of the index template. This is no longer recommended. Use the setup command instead.

Metricbeat and Filebeat users: If you use Beats central management, make sure you migrate the Beats central management index to 6.7 before you upgrade to 7.0. Although central management is not a GA-level feature in 7.0, we’ve provided a migration tool to help you migrate your configurations from version 6.6 to 6.7 or later. For more information, see the Beats 6.7.0 release blog.

Upgrade Beats binaries to 7.0

edit

Before upgrading:

  1. Stop the existing Beats process by using the appropriate command for your system.

  2. Back up the data and config directories by copying them to another location.

    The location of these directories depends on the installation method. To see the current paths, start the Beat from a terminal, and the data and config paths are printed at startup.

To upgrade using a Debian or RPM package:

  • Use rpm or dpkg to install the new package. All files are installed in the appropriate location for the operating system and Beats config files are not overwritten.

To upgrade using a zip or compressed tarball:

  1. Extract the zip or tarball to a new directory. This is critical if you are not using external config and data directories.

  2. Set the following options in the Beats configuration file:

    • Set path.config to point to your external config directory. If you are not using an external config directory, copy your old configuration over to the new installation.
    • Set path.data to point to your external data directory. If you are not using an external data directory, copy your old data directory over to the new installation.
    • Set path.logs to point to the location where you want to store your logs. If you do not specify this setting, logs are stored in the directory you extracted the archive to.

Complete the upgrade tasks described in the following sections before restarting the Beats process.

Migrate configuration files

edit

Beats 7.0 comes with several backwards incompatible configuration changes. Before upgrading, review the Breaking changes in 7.0 document. Also review the full list of breaking changes in the Release notes for 7.0.

Where possible, we kept the old configuration options working, but deprecated them. However, deprecation was not always possible, so if you use any of the settings described under breaking changes, make sure you understand the alternatives that we provide.

Enable the compatibility layer for Elastic Common Schema (ECS) fields

edit

Starting with 7.0, the fields exported by Beats conform to the Elastic Common Schema (ECS). Many of the exported fields have been renamed. See Breaking changes in 7.0 for the full list of changed names.

To help you transition to the new fields, we provide a compatibility layer in the form of ECS-compatible field aliases. To use the aliases, set the following option in the Beat’s configuration file before you upgrade the Elasticsearch index template to 7.0.

migration.6_to_7.enabled: true

The field aliases let you use 6.x dashboards and visualizations with indices created by Beats 7.0 or later. The aliases do not work with saved searches or with API calls that manipulate documents directly.

Some fields also have type changes in 7.0 that affect the behavior of older dashboards and visualizations. To clarify:

  • Some fields have type changes. Your Kibana visualizations and aggregations will not work on these fields until the conflicts are resolved.
  • Some fields have name changes, but no type changes. The field aliases created by the compatibility layer ensure that visualizations and aggregations on the old field names work on old and new data. 7.0 dashboards work only on new field names (and therefore only on new data).
  • Some fields have both name and type changes. Field aliases are created for these fields, but your Kibana visualizations and aggregations will not work on these fields until the conflicts are resolved. Some of your Elasticsearch API queries may continue to work, if the old and new types are compatible.

We strongly advise that you adjust your custom Kibana dashboards, machine learning jobs, and other content to use the new ECS field names. To learn more about migrating to ECS, see the Migrating to Elastic Common Schema (ECS) in Beats environments blog post.

After removing all references to old fields, you should set migration.6_to_7.enabled: false so that field aliases will not be created during your next minor upgrade.

The aliases will be removed in 8.0.

Did you run the Beat or load the index template before reading this section? That’s OK. See the clean-up steps described under Missing fields or too many fields in the index.

Upgrade the Elasticsearch index template

edit

Index templates and the default index names are versioned. For example, Metricbeat 7.3.2 typically creates indices like this:

metricbeat-7.3.2-2019.04.02

And the corresponding Elasticsearch index template is named metricbeat-7.3.2.

This means that each version of the Beat creates a new index, and it’s guaranteed that the correct index template for that version is applied. With these changes in place, you generally don’t have to do anything to upgrade the index template when you move to a new version. Just load the new version of the index template before ingesting any data into Elasticsearch.

If you plan to run Beats 6.7 and 7.0 in parallel, make sure you enable the compatibility layer before you load the index template.

To load the the index template, run the setup command with the --template option specified. For example, if the Beats output is Elasticsearch, run:

deb and rpm:

metricbeat setup --template

mac and linux:

./metricbeat setup --template

docker:

docker run docker.elastic.co/beats/metricbeat:7.0.0 setup --template

win:

PS > .\metricbeat.exe setup --template

If the Beats output is not elasticsearch, temporarily enable the elasticsearch output when you run the setup command. For example:

deb and rpm:

metricbeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

mac and linux:

./metricbeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

docker:

docker run docker.elastic.co/beats/metricbeat:7.0.0 setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

win:

PS > .\metricbeat.exe setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

When loading the index template, you can also specify -E setup.template.settings.index.number_of_shards=n where n is the number of shards to use for the index.

How to use versioned index templates when the output is not Elasticsearch

edit

If you’ve configured Beats to send events to a different output, such as Logstash, make sure you use versioned index templates and indices. Otherwise, after you upgrade, there will be field conflicts.

To use versioned index templates and indices with Logstash, configure your Logstash pipeline to use the metadata from Beats to set the index, and allow Beats to manage the index template:

    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

When you use this configuration, the index name is set to match the index pattern in the Beats index template. See the Beats input plugin documentation for more information.

For other non-Elasticsearch outputs, use the metadata from Beats to set the index, and allow Beats to load and manage the index template, as described under Upgrade the Elasticsearch index template.

The index name must match the index pattern in the Beats index template. For example, if Logstash sends events to an index called metricbeat-7-2019.04.02, but the index template expects indices to match metricbeat-7.3.2-*, you may encounter mapping errors and be unable to index Beats events.

Upgrade dashboards

edit

We recommend that you import the 7.0 Kibana dashboards after upgrading Kibana and Beats. This way, you can take advantage of the new dashboards created for the 7.0 release.

If you’ve enabled the compatibility layer before loading the index template and dashboards, 6.x dashboards can co-exist with 7.0 dashboards and will continue working after the upgrade.

To load the 7.0 dashboards, run the setup command with the --dashboards option specified. For example, if the Beats output is Elasticsearch, run:

deb and rpm:

metricbeat setup --dashboards

mac and linux:

./metricbeat setup --dashboards

docker:

docker run docker.elastic.co/beats/metricbeat:7.0.0 setup --dashboards

win:

PS > .\metricbeat.exe setup --dashboards

If the Beats output is not elasticsearch, temporarily enable the elasticsearch output when you run the setup command. For example:

deb and rpm:

metricbeat setup --dashboards -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

mac and linux:

./metricbeat setup --dashboards -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

docker:

docker run docker.elastic.co/beats/metricbeat:7.0.0 setup --dashboards -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

win:

PS > .\metricbeat.exe setup --dashboards -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

Migrate Filebeat registry to use new format

edit

Starting with version 7.0, Filebeat stores the registry in a directory structure. If Filebeat finds an old registry file at the path set by filebeat.registry.path, it will automatically migrate the registry file to the new format. If the registry file is at a different location, set filebeat.registry.migrate_file to point to the file.

The registry changes also require you to rename the following configuration settings:

Old config name New config name

filebeat.registry_file

filebeat.registry.path.

filebeat.registry_file_permissions

filebeat.registry.file_permissions

filebeat.registry_flush

filebeat.registry.flush

Before proceeding with the upgrade, make sure you back up the registry file.

Check privileges for index lifecycle management (on by default in 7.0)

edit

Starting with Beats 7.0, index lifecycle management is on by default when sending data to Beats clusters that support it. Make sure Beats users have the privileges needed to use index lifecycle management, or disable index lifecycle management.

For help troubleshooting authorization issues, see User is not authorized.

If you want to disable index lifecycle management, set setup.ilm.enabled: false in the Beats configuration file.

« Upgrade between minor versions Troubleshooting Beats upgrade issues »