IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Logstash 8.0.1 Release Notes Logstash 8.0.0-rc2 Release Notes »
Elastic Docs ›Logstash Reference [8.1] ›Release Notes

Logstash 8.0.0 Release Notes

edit

Logstash 8.0.0 Release Notes

edit

The following list are changes in 8.0.0 as compared to 7.17.0, and combines release notes from the 8.0.0-alpha1, -alpha2, -beta1, -rc1 and -rc2 releases.

Breaking changes

edit
  • Many plugins can now be run in a mode that avoids implicit conflict with the Elastic Common Schema (ECS). This mode is controlled individually with each plugin’s ecs_compatibility option, which defaults to the value of the Logstash pipeline.ecs_compatibility setting. In Logstash 8, this compatibility mode will be on-by-default for all pipelines. If you wish to lock in a pipeline’s behavior from Logstash 7.x before upgrading to Logstash 8, you can set pipeline.ecs_compatibility: disabled to its definition in pipelines.yml (or globally in logstash.yml).
  • Starting from Logstash 8.0, the minimum required version of Java to run Logstash is Java 11. By default, Logstash will run with the bundled JDK, which has been verified to work with each specific version of Logstash, and generally provides the best performance and reliability.
  • Support for using JAVA_HOME to override the path to the JDK that Logstash runs with has been removed for this release. In the 8.x release, users should set the value of LS_JAVA_HOME to the path of their preferred JDK if they wish to use a version other than the bundled JDK. The value of JAVA_HOME will be ignored.
  • The Java Execution Engine has been the default engine since Logstash 7.0, and works with plugins written in either Ruby or Java. Removal of the Ruby Execution Engine will not affect the ability to run existing pipelines. #12517
  • We have added support for UTF-16 and other multi-byte-character when reading log files. #9702
  • Setting config.field_reference.parser has been removed. The Field Reference parser interprets references to fields in your pipelines and plugins. Its behavior was configurable in 6.x, and since 7.0 allowed only a single option: strict. 8.0 no longer recognizes the setting, but maintains the same behavior as the strict setting. Logstash rejects ambiguous and illegal inputs as standard behavior. #12466

For a more detailed view of these changes please check Breaking changes in 8.0.

New features and enhancements

edit
  • As processing times speed up, millisecond granularity is not always enough. Inbound data increasingly has sub-millisecond granularity timestamps. The pull request #12797 allows the internal mechanisms of Logstash that hold moment-in-time data - such as the Logstash Event, the Persistent Queue, the Dead Letter Queue and JSON encoding/decoding - to have nanosecond granularity.
  • We have added another flag to the Benchmark CLI to allow passing a data file with previously captured data to the custom test case. This feature allows users to run the Benchmark CLI in a custom test case with a custom config and a custom dataset. #12437

Plugins

edit

Logstash 8.0.0 includes the same versions of all bundled plugins as Logstash 7.17.0. If you upgrade to 7.17 before upgrading to 8.0 (as recommended), you won’t see any changes to plugin versions.

Clone Filter - 4.2.0

  • Added support for ECS v8 as alias for ECS v1 #27

Geoip Filter - 7.2.11

  • Improved compatibility with the Elastic Common Schema #206

    • Added support for ECS’s composite region_iso_code (US-WA), which replaces the non-ECS region_code (WA) as a default field with City databases. To get the stand-alone region_code in ECS mode, you must include it in the fields directive
    • [DOC] Improve ECS-related documentation
  • [DOC] Air-gapped environment requires both ASN and City databases #204

Http Filter - 1.2.1

  • Fix: do not set content-type if provided by user #36
  • Feat: improve ECS compatibility #35
  • Add support for PUT requests #34

Ruby Filter - 3.1.8

  • [DOC] Added doc to describe the option `tag_with_exception_message`https://github.com/logstash-plugins/logstash-filter-ruby/pull/62[#62]
  • Fix SyntaxError handling so other pipelines can shut down gracefully #64

Useragent Filter - 3.3.3

  • Docs: mention added fields in 3.3 with a note #78

Exec Input - 3.4.0

  • Feat: adjust fields for ECS compatibility #28
  • Plugin will no longer override fields if they exist in the decoded payload (It no longer sets the host field if decoded from the command’s output)

Gelf Input - 3.3.1

  • Fix: safely coerce the value of _@timestamp to avoid crashing the plugin #67

Generator Input - 3.1.0

  • Feat: adjusted fields for ECS compatibility #22
  • Fix: do not override the host field if it’s present in the generator line (after decoding)
  • Fix: codec flushing when closing input

Imap Input - 3.2.0

  • Feat: ECS compatibility #55
  • added (optional) headers_target configuration option
  • added (optional) attachments_target configuration option
  • Fix: plugin should not close $stdin, while being stopped

Jms Input - 3.2.1

  • Fix: improve compatibility with MessageConsumer implementations #51, such as IBM MQ.
  • Test: Fix test failures due to ECS compatibility default changes in 8.x of logstash #53
  • Feat: event_factory support + targets to aid ECS #49
  • Fix: when configured to add JMS headers to the event, headers whose value is not set no longer result in nil entries on the event
  • Fix: when adding the jms_reply_to header to an event, a string representation is set instead of an opaque object.

Pipe Input - 3.1.0

  • Feat: adjust fields for ECS compatibility #19

S3 Input - 3.8.3

  • Fix missing metadata and type of the last event #223
  • Refactor: read sincedb time once per bucket listing #233

Snmp Input - 1.3.1

  • Refactor: handle no response(s) wout error logging #105
  • Feat: ECS compliance + optional target #99
  • Internal: update to Gradle 7 #102

Snmptrap Input - 3.1.0

  • Feat: ecs_compatiblity support + (optional) target #37

Syslog Input - 3.6.0

  • Add support for ECS v8 as alias to v1 implementation #68

Twitter Input - 4.1.0

  • Feat: optional target + ecs_compatibility #72

Unix Input - 3.1.1

  • Fix: unable to stop plugin (on LS 6.x) #29
  • Refactor: plugin internals got reviewed for data_timeout => ... to work reliably
  • Feat: adjust fields for ECS compatibility #28

Jdbc Integration - 5.2.2

  • Feat: name scheduler threads + redirect error logging #102
  • Refactor: isolate paginated normal statement algorithm in a separate handler #101
  • Added jdbc_paging_mode option to choose if use explicit pagination in statements and avoid the initial count query or use auto to delegate to the underlying library #95

  • Several improvements to Java driver loading

    • Refactor: to explicit Java (driver) class name loading #96. The change is expected to provide a more robust fix for the driver loading issue #83.

      NOTE: A fatal driver error will no longer keep reloading the pipeline and now leads to a system exit.
    • Fix: regression due returning the Java driver class #98

Kafka Integration - 10.9.0

  • Refactor: leverage codec when using schema registry Previously using schema_registry_url parsed the payload as JSON even if codec => 'plain' was explicitly set, this is no longer the case. #106

Cloudwatch Output - 3.0.10

  • Fix: an old undefined method error which would surface with load (as queue fills up)
  • Deps: unpin rufus scheduler #20

Elasticsearch Output - 11.4.1

  • Feat: upgrade manticore (http-client) library #1063

    • the underlying changes include latest HttpClient (4.5.13)
    • resolves an old issue with ssl_certificate_verification => false still doing some verification logic

  • Updates ECS templates #1062

    • Updates v1 templates to 1.12.1 for use with Elasticsearch 7.x and 8.x
    • Updates BETA preview of ECS v8 templates for Elasticsearch 7.x and 8.x
  • Feat: add support for traces data stream type #1057
  • Refactor: review manticore error handling/logging, logging originating cause in case of connection related error when debug level is enabled. Java causes on connection related exceptions will now be extra logged when plugin is logging at debug level #1029

  • ECS-related fixes #1046

    • Data Streams requirement on ECS is properly enforced when running on Logstash 8, and warned about when running on Logstash 7.
    • ECS Compatibility v8 can now be selected

Core Patterns - 4.3.2

  • Fix: typo in BIN9_QUERYLOG pattern (in ECS mode) #307
« Logstash 8.0.1 Release Notes Logstash 8.0.0-rc2 Release Notes »