Product release

Logstash 5.0.0 Released

By
Suyog Rao

Logstash 5.0.0 has landed. 5 alphas, 1 beta and an RC later, we are super excited to announce the general availability of the biggest release of Logstash yet. And it’s not just Logstash, the entire Elastic Stack is having a release party and they’re all available on Elastic Cloud for you to try immediately.

If you can't wait to get your hands on it, head straight to our downloads page. But we do recommend reading through the important list of breaking changes. If you are looking for a complete set of release notes, you can find it here.

Breaking Changes

We recommend users to read this important information before upgrading from older versions.

Highlights

An important theme for Logstash in this release was to enable self-service troubleshooting. Improving performance is an aspect we try to focus on every release, and in 5.0.0, we've made great strides across the board. Below you will find the headlining features for this release:

Logstash is no longer a black box

A new monitoring feature provides runtime visibility into the Logstash pipeline and its plugins. This component collects various kinds of operational metrics while Logstash processes your data, and all of this information can be queried using simple APIs. Most of the APIs were modeled after Elasticsearch and we've tried to be consisted in regards to the end-points and even the JSON responses. The monitoring APIs can be grouped broadly into these areas:

  • Node Info API: Static information about the Logstash instance and its environment like JVM, OS-level info, etc
  • Plugins Info API: Get information about all Logstash plugins that are currently installed.
  • Node Stats API: Various runtime stats like events processed in stages, CPU and system stats.
  • Hot Threads API: Find which part of Logstash is keeping your CPU busy.

Please see the reference documentation for details.

Logging improvements

Continuing on our theme of providing more visibility into the inner workings of Logstash, we've enhanced Logstash's internal logging framework. First off, we've moved to Log4j2, the popular Java logging library. For users, this means you can now get logs granularly down to a single plugin level. To increase granularity and log levels, one could modify the log4j2.properties file and restart Logstash, but that is both tedious and leads to unnecessary downtime. Instead, we've exposed APIs that can be used to dynamically update logging levels. These settings get effective immediately and do not need a restart. Another benefit of moving to Log4j2 — file rotation policies can be configured directly using the properties file.

Performance: Java Event

In this release, we've completely rewritten a critical part of Logstash's pipeline infrastructure in Java. Data flow in Logstash is encapsulated using an internal Event object that is passed to plugins, used for conditionals, field reference lookups and in the future, will be persisted on disk. So what does this mean for users? In our performance testing, we've seen consistent throughput increases across multiple configurations. In some cases, we observed up to 75% increase in events processed through Logstash.

If you are a plugin developer or maintain a custom plugin, we encourage you to read this section to adapt your code to the new plugin API.

Ease of use

Settings File: Users can now set Logstash's options in a settings file, logstash.yml, instead of using command line arguments. For example, you can specify pipeline settings, the location of configuration files, logging options all in one file.

Release Packages: Logstash release packages (Debian, RPM) has been given an overhaul in this release. Previously, Logstash used /opt/logstash directory to install the binaries, whereas Elasticsearch used /usr/share, and /var. To make user experience consistent across our products — a theme you'll hear a lot in our 5.0.0 releases — we've changed Logstash to reflect Elasticsearch RPM and DEB layouts. Additionally, we added systemd and upstart support to run Logstash as a service on *nix platforms.

Enhanced integrations and new plugins

Elasticsearch Output: This plugin works out of the box with Elasticsearch 5.0.0. It has also been made threadsafe to take better advantage of the recent changes in the pipeline architecture. Other enhancements include a new connection pool to efficiently reuse connections to Elasticsearch, exponential backoff for connection retries, and better handling for sniffing.

Note: Logstash 5.0 is compatible with Elasticsearch 5.x, 2.x, and even 1.x.

Kafka 0.10 support: Apache Kafka had released version 0.9 and 0.10 couple of months ago which brings in new security features (SSL, client based auth, access control), improved consumer API, and much more. This Logstash release provides out of the box support for SSL encryption and client auth features in Kafka.

Introducing the dissect filter: Extracting fields from unstructured data is a popular Logstash feature. So far, Grok filter was the one and only way to extract fields. Grok uses regular expressions internally, and there was plenty of room for users to get tripped on costly expressions. In other words, for simple field extractions, using Grok was like using a sledgehammer to crack a nut. Dissect is an alternative for field extraction using just delimiter splitting. There's no regular expression in sight. This also means that dissect only works for log lines whose structure is well known. It is not a grok replacement, but our hope is that the dissect filter can complement it.

Google Protobuf codec: A new codec to parse protobuf messages and convert it to Logstash Events, contributed by Inga Feick.

Language Filter: Detects the language of a specific field(s). One of its use cases could be to set the right analyzer in Elasticsearch. Contributed by Justin Bovee.

Prereleases

Blog posts for each 5.0 prerelease can be found here:

Logstash 5.0.0-alpha1 release
Logstash 5.0.0-alpha2 release
Logstash 5.0.0-alpha3 release
Logstash 5.0.0-alpha4 release
Logstash 5.0.0-alpha5 release
Logstash 5.0.0-beta1 release

Feedback

Logstash 5.0.0 has been a huge endeavor which wouldn't have been possible without all the contributions, feedback, comments and enhancement requests from our user community. A special shout out goes to our awesome plugin maintainers who’ve helped shape 5.0.0. If you participated in the Pioneer Program, you will be hearing from our developer relationship team soon. So, from the entire Logstash team, thank you to our users for using and contributing back to Logstash.

Go on, unwrap the shiny new 5.0.0 and let us know what you think! Happy ‘stashing!