Logstash 2.2.0 and 2.1.2 released

The release bonanza continues…

Today, we announce new versions of the entire Elastic Stack, including a tighter integration of Shield with Kibana and an updated version of ES-Hadoop. Detailed blogs for product releases are available in the releases category of the blog. And yes, the blog has categories -- you know, for searchability.

Logstash 2.2.0 and 2.1.2 is available for download today as part of the release bonanza! 2.2.0 release includes changes to the internal architecture of the pipeline, but is still backward compatible with old configuration. Read below for highlights, and check the changelog for more details.

Logstash 2.1.2 is mainly a bug fix release, and its changelog can be found here.

Next Generation Pipeline

The single largest change in Logstash 2.2 release is the addition of the “Next Generation Pipeline” (also known as the “NG pipeline”). We’ve fully re-architected the pipeline to provide more performance and pave the way for future persistence efforts. Where the previous pipeline processed one event at a time, the new pipeline works in micro-batches, processing groups of events at a time. Additionally, the filter and output stages now run as separate execution phases in the same thread, vs. running in separate thread pools. Check out the diagram below for a richer view of these changes.

As illustrated above, the new pipeline first has each worker accumulate a batch of events before processing the filter + output stages. This batch has a maximum size of batch_size and will wait at most batch_delay milliseconds from the last event taken from the queue to proceed with filter/output processing. The new pipeline no longer has distinct threads for filtering and outputting as the old pipeline did, but rather runs the filter and output stage in sequence in each worker thread.

You will most likely find that you need more worker threads to achieve the same throughput as the old pipeline--perhaps even more than the number of real CPU cores--but that these threads run much more efficiently. The reason you’ll need more worker threads is that the output stage of execution is often idle waiting on IO while talking to, say, Elasticsearch or another remote service. We’ve seen performance increases of up to 28% vs. the original pipeline on real-world workloads. This comes from dramatically lowering overhead per-thread, and letting the OS scheduler smartly allocate resources.

For an in-depth guide to tuning the NG pipeline, check out the revamped pipeline documentation.

Smarter Defaults, and Better Output management.

We’ve made managing the performance of outputs much simpler in Logstash 2.2 with automatic output worker scaling. With the NG pipeline an output worker works in a fundamentally different way than previous Logstash releases. Before, each output worker would get its own thread. This led to increased cross-thread communication and context switching and was less efficient, in addition to complicating future persistence work. In the NG pipeline the number of output workers determines how many instances to fill an output worker ‘pool’ with. Output instances are pulled from this pool by pipeline worker threads as needed.

In previous versions of Logstash the default number of output workers was 1. In Logstash 2.2 this number is by default the number of pipeline workers, which now default to be equal to the number of cores on the system. So, if you have a four core system expect to see four pipeline workers with four output ‘workers’ available.This is actually a conservative number for the NG pipeline, as these threads are often idle when in I/O wait. Be sure to read the pipeline documentation for the full story on tuning values here.

The internals of the output stage of logstash have changed quite a bit as well, in ways that have significant implications for both users and plugin authors. Output plugins are now encouraged to implement multi_receive(events) as the primary interface for receiving events, and are discouraged from managing their own buffers. When run with Logstash 2.2 the Elasticsearch output will no longer use a special internal event buffer, but rather use multi_receive to receive events in the batch size the pipeline is using. This was a hard requirement to move us closer to persistence, and should also simplify user’s mental model of Logstash (no having to worry about buffers inside of individual plugins). Plugins that manage their own internal buffers will never be able to correctly persist events and guarantee no data loss in the event of a crash.

We also added the ability to declare plugins as fully threadsafe, meaning a single instance would be shared across all pipeline workers, allowing plugin authors to manage their own locking for even greater performance. Threadsafe plugins ignore any ‘workers’ settings a user might provide, because that number is effectively infinite. We also have improved the API for declaring plugins as not allowing for more than one worker. You can find the details in the revamped How to Write a Logstash Output Plugin page.

Elasticsearch Output Enhancements

Bundled with Logstash 2.2 is the newest version of the Elasticsearch output, designed to work optimally with the NG pipeline, while remaining backward compatible with older 2.x Logstash releases. This version has a vastly simplified and more correct algorithm around retry handling. In this new release each batch going through the NG pipeline blocks until all operations that do not produce unrecoverable errors succeed. This means that network connectivity errors, 429s, and 503s are retried indefinitely now. Previously, the plugin retried these a limited number of times, but we’ve decided to err on the side of data preservation here. Additionally, 409 conflict handling is now only retryable via the retry_on_conflict parameter.

Elasticsearch output now supports scripts to update documents (thanks to @Da-Wei for the contribution). New config options to set the script, language, and variable name have been added.

Deprecating Support For Node Protocol

We would like to announce that Elasticsearch node protocol will no longer be actively developed and no new features will be added. In 2.0 release, we changed the default communication protocol from node to http. This change makes it operationally easy to run Logstash and Elasticsearch, and provides out of the box support for newly released Elasticsearch versions. More details can be found in our post here.

Bug Fixes

Below we highlight some of the bugs fixes in this release:

• File Input: Added config option close_older which closes any files that remain unmodified for longer than the specified timespan in seconds (#44).
• Added config option ignore_older which monitors if a file that was last modified before the specified timespan in seconds, then its contents are ignored (#44).
• Kafka: Fixed an issue where the input was consuming more CPU when no new messages were present in Kafka (#59).
• HTTP: Support compressed and gzip POST requests (#33).

Feedback

Many thanks to our users and contributors for making 2.2.0 a successful release. Please download the GA binaries and give it a spin! Let us know what you think on our forum, IRC, and GitHub.