Logstash 5.0.0 Breaking Changes

edit

Logstash 5.0.0 Breaking Changes

edit

This section discusses the changes that you need to be aware of when migrating your application to Logstash 5.0 from the previous major release of Logstash (2.x).

Changes in Logstash Core

edit

These changes can impact any instance of Logstash and are plugin agnostic, but only if you are using the features that are impacted.

Application Settings

edit

Logstash 5.0 introduces a new way to configure application settings for Logstash through a logstash.yml file.

This file is typically located in ${LS_HOME}/config, or /etc/logstash when installed via packages. Logstash will not be able to start without this file, so please make sure to pass in --path.settings if you are starting Logstash manually after installing it via a package (RPM, DEB).

bin/logstash --path.settings /path/to/logstash.yml

URL Changes for DEB/RPM Packages

edit

The previous packages.elastic.co URL has been altered to artifacts.elastic.co. Ensure you update your repository files before running the upgrade process, or your operating system may not see the new packages.

Release Packages

edit

When Logstash 5.0 is installed via DEB or RPM packages, it now uses /usr/share/logstash to install binaries. Previously it used to install in /opt/logstash directory. This change was done to make the user experience consistent with other products in the Elastic Stack.

DEB RPM

Logstash 2.x

/opt/logstash

/opt/logstash

Logstash 5.0

/usr/share/logstash

/usr/share/logstash

A complete directory layout is described in Logstash Directory Layout. This will likely impact any scripts that you may have written to support installing or manipulating Logstash, such as via Puppet.

Default Logging Level

edit

The default log severity level changed to INFO instead of WARN to match Elasticsearch. Existing logs (in core and plugins) were too noisy at the INFO level, so we auditted our log messages and switched some of them to DEBUG level.

You can use the new logstash.yml file to configure the log.level setting or continue to pass the new --log.level command line flag.

bin/logstash --log.level warn

Plugin Manager Renamed

edit

bin/plugin has been renamed to bin/logstash-plugin. This occurred in Logstash 2.3 and it was mainly prevent PATH being polluted when other components of the Elastic Stack are installed on the same machine. Also, this provides a foundation for future change which will allow Elastic Stack packs to be installed via this script.

Logstash 5.0 also adds a remove option, which is an alias for the now-deprecated uninstall option.

As with earlier releases, the updated script allows both online and offline plugin installation. For example, to install a plugin named “my-plugin”, it’s as simple as running:

bin/logstash-plugin install my-plugin

Similar to the package changes, this is likely to impact and scripts that have been written to follow Logstash installations.

Like earlier releases of Logstash, most plugins are bundled directly with Logstash, so no additional action is required while upgrading from earlier Logstash releases. However, if you are attempting to install a non-bundled plugin, then make sure that it supports Logstash 5.0 before upgrading!

Logstash with All Plugins Download

edit

The Logstash All Plugins download option has been removed. For users previously using this option as a convenience for offline plugin management purposes (air-gapped environments), please see the Offline Plugin Management documentation page.

There were 17 plugins removed from 5.0 default bundle. These plugins can still be installed manually for use:

  • logstash-codec-oldlogstashjson
  • logstash-input-eventlog
  • logstash-input-log4j
  • logstash-input-zeromq
  • logstash-filter-anonymize
  • logstash-filter-checksum
  • logstash-filter-multiline
  • logstash-output-email
  • logstash-output-exec
  • logstash-output-ganglia
  • logstash-output-gelf
  • logstash-output-hipchat
  • logstash-output-juggernaut
  • logstash-output-lumberjack
  • logstash-output-nagios_nsca
  • logstash-output-opentsdb
  • logstash-output-zeromq

Command Line Interface

edit

Some CLI Options changed in Logstash 5.0. If you were using the “long form” of the options, then this will impact the way that you launch Logstash. They were changed to match the logstash.yml format used to simplify future setup, as well as behave in the same way as other products in the Elastic Stack. For example, here’s two before-and-after examples. In Logstash 2.x, you may have run something:

bin/logstash --config my.conf --pipeline-workers 8 
bin/logstash -f my.conf -w 8 

Long form options config and pipeline-workers are used here.

Short form options f and w (aliases for the former` are used here.

But, in Logstash 5.0, this becomes:

bin/logstash --path.config my.conf --pipeline.workers 8 
bin/logstash -f my.conf -w 8 

Long form options are changed to reflect the new options.

Short form options are unchanged.

None of the short form options have changed!

RSpec testing script

edit

The rspec script is no longer bundled with Logstash release artifacts. This script has been used previously to run unit tests for validating Logstash configurations. While this was useful to some users, this mechanism assumed that Logstash users were familiar with the RSpec framework, which is a Ruby testing framework.

Breaking Changes in Plugins

edit

Elasticsearch Output workers Setting Removed

edit

Starting with Logstash 5.0, the workers setting in the Elasticsearch output plugin is no longer supported. Pipelines that specify this setting will no longer start up. You need to specify the pipeline.workers setting at the pipeline level instead. For more information about setting pipeline.workers, see Settings File.

Elasticsearch Output Index Template

edit

The index template for Elasticsearch 5.0 has been changed to reflect Elasticsearch’s mapping changes. Most importantly, the subfield for string multi-fields has changed from .raw to .keyword to match Elasticsearch’s default behavior. The impact of this change to various user groups is detailed below:

  • New Logstash 5.0 and Elasticsearch 5.0 users: Multi-fields (often called sub-fields) use .keyword from the outset. In Kibana, you can use my_field.keyword to perform aggregations against text-based fields, in the same way that it used to be my_field.raw.
  • Existing users with custom templates: Using a custom template means that you control the template completely, and our template changes do not impact you.
  • Existing users with default template: Logstash does not force you to upgrade templates if one already exists. If you intend to move to the new template and want to use .keyword, you will most likely want to reindex existing data so that it also uses the .keyword field, unless you are able to transition from .raw to .keyword. Elasticsearch’s reindexing API can help move your data from using .raw subfields to .keyword, thereby avoiding any transition time. You can use a custom template to get both .raw and .keyword so that you can wait until all .raw data has stopped existing before transitioning to only using .keyword; this will waste some storage space and memory, but it does help users to avoid having to relearn operations.

Plugin Versions

edit

Logstash is unique amongst the Elastic Stack with respect to its plugins. Unlike Elasticsearch and Kibana, which both require plugins to be targeted to a specific release, Logstash’s plugin ecosystem provides more flexibility so that it can support outside ecosystems within the same release. Unfortunately, that flexibility can cause issues when handling upgrades.

Non-standard plugins must always be checked for compatibility, but some bundled plugins are upgraded in order to remain compatible with the tools or frameworks that they use for communication. For example, the Kafka Input and Kafka Output plugins serve as a primary example of such compatibilty changes. The latest version of the Kafka plugins is only compatible with Kafka 0.10, but as the compatibility matrices show: earlier plugin versions are required for earlier versions of Kafka (e.g., Kafka 0.9).

Automatic upgrades generally lead to improved features and support, but network layer changes like those above may make part of your architecture incompatible. You should always test your Logstash configurations in a test environment before deploying to production, which would catch these kinds of issues. If you do face such an issue, then you should also check the specific plugin’s page to see how to get a compatible, older plugin version if necessary.

For example, if you upgrade to Logstash 5.0, but you want to run against Kafka 0.9, then you need to remove the bundled plugin(s) that only work with Kafka 0.10 and replace them:

bin/logstash-plugin remove logstash-input-kafka
bin/logstash-plugin remove logstash-output-kafka
bin/logstash-plugin install --version 4.0.0 logstash-input-kafka
bin/logstash-plugin install --version 4.0.1 logstash-output-kafka

The version numbers were found by checking the compatibility matrix for the individual plugins.

Kafka Input Configuration Changes

edit

As described in the section above, the Kafka plugin has been updated to bring in new consumer features. In addition, to the plugin being incompatible with 0.8.x version of the Kafka broker, most of the config options have been changed to match the new consumer configurations from the Kafka Java consumer. Here’s a list of important config options that have changed:

  • topic_id is renamed to topics and accepts an array of topics to consume from.
  • zk_connect has been dropped; you should use bootstrap_servers. There is no need for the consumer to go through ZooKeeper.
  • consumer_id is renamed to client_id.

We recommend users of the Kafka plugin to check the documentation for the latest config options.

File Input

edit

The File Input SinceDB file is now saved at <path.data>/plugins/inputs/file location, where path.data is the path defined in the new logstash.yml file.

Default sincedb_path

Logstash 2.x

$HOME/.sincedb*

Logstash 5.0

<path.data>/plugins/inputs/file

If you have manually specified sincedb_path as part of the configuration, this change will not affect you. If you are moving from Logstash 2.x to Logstash 5.0, and you would like to use the existing SinceDB file, then it must be copied over to path.data manually to use the save state (or the path needs to be changed to point to it).

GeoIP Filter

edit

The GeoIP filter has been updated to use MaxMind’s GeoIP2 database. Previous GeoIP version is now considered legacy by MaxMind. As a result of this, .dat version files are no longer supported, and only .mmdb format is supported. The new database will not include ASN data in the basic free database file.

Previously, when the filter encountered an IP address for which there were no results in the database, the event would just pass through the filter without modification. It will now add a _geoip_lookup_failure tag to the event which will allow for some subsequent stage of the pipeline to identify those events and perform some other operation. To simply get the same behavior as the earlier versions, just add a filter conditional on that tag which then drops the tag from the event.

Ruby Filter and Custom Plugin Developers

edit

With the migration to the new Event API, we have changed how you can access internal data compared to previous release. The event object no longer returns a reference to the data. Instead, it returns a copy. This might change how you perform manipulation of your data, especially when working with nested hashes. When working with nested hashes, it’s recommended that you use the field reference syntax instead of using multiple square brackets.

As part of this change, Logstash has introduced new Getter/Setter APIs for accessing information in the event object.

Examples:

Prior to Logstash 5.0, you may have used Ruby filters like so:

filter {
  ruby {
    code => "event['name'] = 'Logstash'"
  }
  ruby {
    code => "event['product']['version'] = event['major'] + '.' + event['minor']"
  }
}

The above syntax, which uses the event object as a reference, is no longer supported in Logstash 5.0. Fortunately, the change to make it work is very simple:

filter {
  ruby {
    code => "event.set('name', 'Logstash')"
  }
  ruby {
    code => "event.set('[product][version]', event.get('major') + '.' + event.get('minor'))"
  }
}

Moving from the old syntax to the new syntax, it can be easy to miss that ['product']['version'] became '[product][version]'. The quotes moved from inside of the square brackets to outside of the square brackets!

The Event API documentation describes the available syntax in great detail.