<html><head></head><body>The <a href="/blog/logstash-6-6-0-released">6.6.0 release of Logstash</a> includes the first iteration of native support for <a href="https://github.com/logstash-plugins/logstash-input-java_input_example#how-to-write-a-java-input">input</a>, <a href="https://github.com/logstash-plugins/logstash-filter-java_filter_example#how-to-write-a-java-filter">filter</a>, and <a href="https://github.com/logstash-plugins/logstash-output-java_output_example#how-to-write-a-java-output">output</a> Java plugins. Though Logstash is built on both JRuby and Java, its plugin API has always been defined solely in Ruby. Even plugins that incorporated Java classes or dependencies still had to conform to the Ruby plugin API. With native support for Java plugins, all that changes.
The goals for native Java plugins in Logstash are:
<ul>
	<li>Provide first-class support for developers who wish to write pure Java plugins for Logstash through a new Java plugin API. &#x201C;First-class support&#x201D; means that Java developers should be able to write Logstash plugins using idiomatic Java code free of any dependencies on Ruby and its toolchain and ecosystem.</li>
	<li>Enable seamless operation with Ruby plugins. Native Java plugins exist as peers of Ruby plugins. As such, any combination of Ruby and Java plugins should run smoothly and transparently within the same Logstash pipeline. &#xA0;</li>
	<li>Run with high performance in the Java execution engine. There can be a performance cost with interop between JRuby and Java code. Native Java plugins and the Java execution engine should be optimized to minimize that interop cost.</li>
</ul>If you would like to experiment with building your own Java plugins for Logstash or just see what a Java plugin would look like, we have provided examples of a Java <a href="https://github.com/logstash-plugins/logstash-input-java_input_example">input</a>, <a href="https://github.com/logstash-plugins/logstash-filter-java_filter_example">filter</a>, and <a href="https://github.com/logstash-plugins/logstash-output-java_output_example">output</a>. Let us know what you think by commenting on our <a href="https://github.com/elastic/logstash/issues/9215">Github issue</a> or posting in the <a href="https://discuss.elastic.co/c/logstash">Logstash forum</a>.
As a very brief and simple example, the following is a snippet of a hypothetical Java filter that tags events based on the datatype of a particular field:
<pre class="prettyprint">@Override
public Collection&lt;Event&gt; filter(Collection&lt;Event&gt; events) {
 for (Event e : events) {
 Object value = e.getField(targetField);
 if (value != null) {
 e.tag(targetField + &#x201C;_&#x201D; + value.getClass().getName()); 
 } else {
 e.tag(targetField + &#x201C;_missing&#x201D;);
 }
 }
 return events;
}
</pre><h2>Known Limitations</h2>Native support for Java plugins in the 6.6.0 release of Logstash is in the experimental phase. It is intended to provide a preview of the Java plugin functionality and Java plugin API that will eventually be made GA. Though we intend to avoid unnecessary changes, the functionality and API is subject to change before GA. The current Java plugin support provides basic plugin functionality for inputs, filters, and outputs but does not yet provide full feature parity with Ruby plugins.
Java plugins run only in the Java execution engine which is being promoted to GA in the 6.6.0 release. You can enable the Java execution engine via the <code>--java-execution</code> command-line option or by adding <code>pipeline.java_execution: true</code> to your <code>logstash.yml</code> file.
<h2>Future Plans</h2>Java plugins are planned as full peers of Ruby plugins with all the same features and capabilities. Full feature parity with Ruby plugins is planned for the <a href="https://github.com/elastic/logstash/pull/10232">beta phase</a> of the native Java plugin support project which is already proceeding rapidly and will land in a future Logstash release!
</body></html>

logstash-persistent-queue-elasticsearch-events-fullbleed.jpg

logstash-persistent-queue-elasticsearch-events-thumbnail.jpeg

Previewing Native Support for Java Plugins in Logstash

<html><head></head><body>We&#x2019;re happy to announce that the new <a href="https://github.com/elastic/logstash/pull/8500">Java execution engine</a> in Logstash has reached the production candidate stage. It features faster performance, reduced memory usage, and lower config startup and reload times. And you can use it now in <a href="https://www.elastic.co/blog/logstash-6-3-0-released">Logstash 6.3.0</a> with the <code>--experimental-java-execution flag</code>! 
Logstash is built with JRuby and Java which combines the elegance and expressivity of Ruby with the maturity and performance of the Java Virtual Machine. However, the mixing of the Ruby and Java ecosystems must be done with care in performance-sensitive scenarios. The Java execution engine is the first major deliverable of a series of projects that are underway to optimize the interaction between Ruby and Java in Logstash in order to provide a first-class experience for Logstash plugin developers who wish to use either Ruby or Java.
Here are some of <a href="https://github.com/elastic/logstash/issues/9625">the performance improvements</a> we&#x2019;ve seen with the new Java execution engine (higher bars are better):
<img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/blt392dc8a176b08485/5c30597529d13af10bc2ab8c/logstash-throughput-apache-log-parsing.png" data-sys-asset-uid="blt392dc8a176b08485" alt="logstash-throughput-apache-log-parsing.png" width="390" height="240" style="float: left; width: 390px; height: 240px;"><img src="https://images.contentstack.io/v3/assets/bltefdd0b53724fa2ce/bltcf933fc166ae3cc9/5c305971ca38a16366a38054/logstash-throughput-generated-events.png" data-sys-asset-uid="bltcf933fc166ae3cc9" alt="logstash-throughput-generated-events.png" width="379" height="235" style="width: 379px; height: 235px;" rel="width: 379px; height: 235px;">
As always, no single benchmark can fully characterize the performance of Logstash because different workloads can have very different performance profiles. The two different scenarios above are intended to model workloads at opposite ends of the plugin resource demand spectrum. In the Apache log parsing scenario, reading the Apache logs and filtering them through grok and other filters requires lots of CPU and IO resources, so the time spent in plugins tends to be the dominating factor in the overall throughput of the pipeline. In the generated events scenario, very little time is spent in plugins, so the efficiency of the execution engine is the dominating factor in the overall throughput of the pipeline.
The two big takeaways from the performance charts above are:
<ul>
	<li>Across-the-board performance in both execution engines has improved significantly between 6.0 and 6.1 when the alpha version of the Java execution engine was completed. You should upgrade if you&#x2019;re on 6.0 or an earlier release!</li>
	<li>Both the Java and Ruby execution engines have shown progressive performance improvements from the first version in 6.1 through the current release in 6.3.</li>
</ul>There are two aspects of the Java execution engine that contribute to these performance improvements.
The first factor is the all-new config parsing and bytecode generation. The original Ruby execution engine reads the Logstash config files using <a href="https://github.com/cjheath/treetop">Treetop</a> to parse and convert them to executable Ruby source code which is then interpreted and executed at runtime by the JRuby runtime. The Java execution engine replaces the Ruby source code generation phase with compilation directly to JVM bytecode using <a href="https://janino-compiler.github.io/janino/">Janino</a>. Some execution optimizations can be applied at this stage and the startup and reload times for some large configurations have been reduced by as much as 20x. These improvements apply only to workloads running in the Java execution engine.
The second factor is optimizing the points of contact between Ruby and Java. Due to the dynamic nature of Ruby, there are <a href="https://github.com/jruby/jruby/wiki/ImprovingJavaIntegrationPerformance">performance costs</a> involved in calling from JRuby code into Java code and these can be significant when made in tight loops or other performance-sensitive areas. A concerted effort has been made to minimize those calls in the pipeline execution engine. Where such calls were unavoidable, <a href="https://github.com/jruby/jruby-examples/tree/master/extensions/basic">JRuby extension classes</a> were used which also bypass that performance penalty. Although the Ruby execution engine is slated for eventual removal, one nice thing about this work is that many of these performance improvements were able to be applied to both the Ruby and Java execution engines so you can benefit from them now.
The Java execution engine also positions us to be able to work more efficiently and deliver new features such as our upcoming pure <a href="https://github.com/elastic/logstash/issues/9215">Java plugin API</a> for Logstash. We expect that along with these new features, we&#x2019;ll continue to deliver incremental performance improvements as well. We&#x2019;d encourage you to <a href="http://elastic.co/downloads/logstash">give it a try in Logstash 6.3</a>, see how it works for you, and <a href="https://github.com/elastic/logstash/issues">let us know</a> if you encounter any problems!
Happy stashing!
</body></html>

Meet the New Logstash Java Execution Engine

Start free trial

Blog Footer CTA

Sign up for Elastic Cloud free trial

Dan Hermann

Надсилаючи цю форму ви погоджуєтеся з <a href="/legal/terms-of-use">Умовами надання послуг Elastic</a>. Ваші особисті дані будуть опрацьовані відповідно з <a href="/legal/privacy-statement">Положеннями про конфіденційність Elastic</a>.

Newsletter GDPR Text - Ukrainian

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_self">Terms of Service</a>, and you agree to receive the <a href="/legal/privacy-statement#how-we-use-the-information" target="_self">selected communications</a> at the contact details you provided above. See <a href="/legal/privacy-statement/" target="_self">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Communication Preferences - ONLY for /communication-preferences

By submitting you agree to <a href="/agreements/global/cloud-services-monthly" rel="nofollow">Elastic Cloud Terms of Service</a>. Your personal data will be processed in accordance with <a href="/legal/privacy-statement">Elastic's Privacy Statement</a>.

Cloud Trial GDPR Text

Intentionally Blank

By submitting you acknowledge that you've read and agree to our <a href="/legal/serverless-preview-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Serverless GDPR Text

提交即表示您確認已詳閱並同意我們的<a href="/legal/elastic-cloud-account-terms" target="_blank">服務條款</a>，且同意 Elastic 使用您在上方提供的個人資料<a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">與您聯絡</a>，為您提供相關產品和服務的資訊。如需詳細說明或隨時想停止收到產品和服務資訊，請參閱 <a href="/legal/privacy-statement/" target="_blank">Elastic 的隱私聲明</a>。

Newsletter GDPR Text - TW

By submitting you acknowledge that you've read and agree to our <a href="/legal/elastic-cloud-account-terms" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.By submitting this form, I agree to Elastic sharing my contact information with Google Cloud. I also agree to Elastic giving my information to the vendor for the purpose of shipping my food package.

GDPR Text + Google Cloud

Inviando la registrazione (1) esprimi il consenso ai <a href="/legal/terms-of-use">termini e condizioni di Elastic</a> e <a href="/legal/privacy-statement">all’informativa sulla privacy</a>, nonché (2) a ricevere saltuarie e-mail.

Newsletter GDPR Text - Italian

By submitting you acknowledge that you've read and agree to our <a href="/legal/terms-of-use" target="_blank">Terms of Service</a>, and that Elastic may <a href="/legal/privacy-statement#how-we-use-the-information" target="_blank">contact you</a> about our related products and services, using the details you provide above. See <a href="/legal/privacy-statement/" target="_blank">Elastic’s Privacy Statement</a> for more details or to opt-out at any time.

Newsletter GDPR Text

Load more

Overview

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content

Can't make it? Register and we'll send you the recording. You'll also receive an email with related content.

You'll also receive an email with related content.

You'll also receive an email with related content

Speakers

Close

See more insights

The content on this page is not available in the selected language. As Elastic grows globally, we continue to support content in multiple languages.

Author

Articles by Dan Hermann

Previewing Native Support for Java Plugins in Logstash

Meet the New Logstash Java Execution Engine