IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Example: Set up Filebeat modules to work with Kafka and Logstash Persistent Queues »
Elastic Docs ›Logstash Reference [7.3]

Data Resiliency

edit

Data Resiliency

edit

As data flows through the event processing pipeline, Logstash may encounter situations that prevent it from delivering events to the configured output. For example, the data might contain unexpected data types, or Logstash might terminate abnormally.

To guard against data loss and ensure that events flow through the pipeline without interruption, Logstash provides the following data resiliency features.

  • Persistent Queues protect against data loss by storing events in an internal queue on disk.
  • Dead Letter Queues provide on-disk storage for events that Logstash is unable to process. You can easily reprocess events in the dead letter queue by using the dead_letter_queue input plugin.

These resiliency features are disabled by default. To turn on these features, you must explicitly enable them in the Logstash settings file.

« Example: Set up Filebeat modules to work with Kafka and Logstash Persistent Queues »