Previewing Native Support for Java Plugins in Logstash

By
Dan Hermann

The 6.6.0 release of Logstash includes the first iteration of native support for input, filter, and output Java plugins. Though Logstash is built on both JRuby and Java, its plugin API has always been defined solely in Ruby. Even plugins that incorporated Java classes or dependencies still had to conform to the Ruby plugin API. With native support for Java plugins, all that changes.

The goals for native Java plugins in Logstash are:

  • Provide first-class support for developers who wish to write pure Java plugins for Logstash through a new Java plugin API. “First-class support” means that Java developers should be able to write Logstash plugins using idiomatic Java code free of any dependencies on Ruby and its toolchain and ecosystem.
  • Enable seamless operation with Ruby plugins. Native Java plugins exist as peers of Ruby plugins. As such, any combination of Ruby and Java plugins should run smoothly and transparently within the same Logstash pipeline.  
  • Run with high performance in the Java execution engine. There can be a performance cost with interop between JRuby and Java code. Native Java plugins and the Java execution engine should be optimized to minimize that interop cost.

If you would like to experiment with building your own Java plugins for Logstash or just see what a Java plugin would look like, we have provided examples of a Java input, filter, and output. Let us know what you think by commenting on our Github issue or posting in the Logstash forum.

As a very brief and simple example, the following is a snippet of a hypothetical Java filter that tags events based on the datatype of a particular field: 

@Override
public Collection<Event> filter(Collection<Event> events) {
    for (Event e : events) {
        Object value = e.getField(targetField);
        if (value != null) {
            e.tag(targetField + “_” + value.getClass().getName());        
        } else {
            e.tag(targetField + “_missing”);
        }
    }
    return events;
}

Known Limitations

Native support for Java plugins in the 6.6.0 release of Logstash is in the experimental phase. It is intended to provide a preview of the Java plugin functionality and Java plugin API that will eventually be made GA. Though we intend to avoid unnecessary changes, the functionality and API is subject to change before GA. The current Java plugin support provides basic plugin functionality for inputs, filters, and outputs but does not yet provide full feature parity with Ruby plugins.

Java plugins run only in the Java execution engine which is being promoted to GA in the 6.6.0 release. You can enable the Java execution engine via the --java-execution command-line option or by adding pipeline.java_execution: true to your logstash.yml file.

Future Plans

Java plugins are planned as full peers of Ruby plugins with all the same features and capabilities. Full feature parity with Ruby plugins is planned for the beta phase of the native Java plugin support project which is already proceeding rapidly and will land in a future Logstash release!