- Logstash Reference: other versions:
- Logstash Introduction
- Getting Started with Logstash
- How Logstash Works
- Setting Up and Running Logstash
- Logstash Directory Layout
- Logstash Configuration Files
- logstash.yml
- Secrets keystore for secure settings
- Running Logstash from the Command Line
- Running Logstash as a Service on Debian or RPM
- Running Logstash on Docker
- Configuring Logstash for Docker
- Running Logstash on Windows
- Logging
- Shutting Down Logstash
- Setting Up X-Pack
- Upgrading Logstash
- Configuring Logstash
- Advanced Logstash Configurations
- Managing Logstash
- Working with Logstash Modules
- Working with Filebeat Modules
- Data Resiliency
- Transforming Data
- Deploying and Scaling Logstash
- Performance Tuning
- Monitoring Logstash
- Monitoring Logstash with APIs
- Working with plugins
- Integration plugins
- Input plugins
- azure_event_hubs
- beats
- cloudwatch
- couchdb_changes
- dead_letter_queue
- elasticsearch
- exec
- file
- ganglia
- gelf
- generator
- github
- google_cloud_storage
- google_pubsub
- graphite
- heartbeat
- http
- http_poller
- imap
- irc
- java_generator
- java_stdin
- jdbc
- jms
- jmx
- kafka
- kinesis
- log4j
- lumberjack
- meetup
- pipe
- puppet_facter
- rabbitmq
- redis
- relp
- rss
- s3
- s3-sns-sqs
- salesforce
- snmp
- snmptrap
- sqlite
- sqs
- stdin
- stomp
- syslog
- tcp
- udp
- unix
- varnishlog
- websocket
- wmi
- xmpp
- Output plugins
- boundary
- circonus
- cloudwatch
- csv
- datadog
- datadog_metrics
- elastic_app_search
- elasticsearch
- exec
- file
- ganglia
- gelf
- google_bigquery
- google_cloud_storage
- google_pubsub
- graphite
- graphtastic
- http
- influxdb
- irc
- java_stdout
- juggernaut
- kafka
- librato
- loggly
- lumberjack
- metriccatcher
- mongodb
- nagios
- nagios_nsca
- opentsdb
- pagerduty
- pipe
- rabbitmq
- redis
- redmine
- riak
- riemann
- s3
- sink
- sns
- solr_http
- sqs
- statsd
- stdout
- stomp
- syslog
- tcp
- timber
- udp
- webhdfs
- websocket
- xmpp
- zabbix
- Filter plugins
- aggregate
- alter
- bytes
- cidr
- cipher
- clone
- csv
- date
- de_dot
- dissect
- dns
- drop
- elapsed
- elasticsearch
- environment
- extractnumbers
- fingerprint
- geoip
- grok
- http
- i18n
- java_uuid
- jdbc_static
- jdbc_streaming
- json
- json_encode
- kv
- memcached
- metricize
- metrics
- mutate
- prune
- range
- ruby
- sleep
- split
- syslog_pri
- threats_classifier
- throttle
- tld
- translate
- truncate
- urldecode
- useragent
- uuid
- wurfl_device_detection
- xml
- Codec plugins
- Tips and Best Practices
- Troubleshooting
- Contributing to Logstash
- How to write a Logstash input plugin
- How to write a Logstash codec plugin
- How to write a Logstash filter plugin
- How to write a Logstash output plugin
- Logstash Plugins Community Maintainer Guide
- Document your plugin
- Publish your plugin to RubyGems.org
- List your plugin
- Contributing a patch to a Logstash plugin
- Extending Logstash core
- Contributing a Java Plugin
- Glossary of Terms
- Breaking Changes
- Release Notes
- Logstash 7.12.1 Release Notes
- Logstash 7.12.0 Release Notes
- Logstash 7.11.2 Release Notes
- Logstash 7.11.1 Release Notes
- Logstash 7.11.0 Release Notes
- Logstash 7.10.2 Release Notes
- Logstash 7.10.1 Release Notes
- Logstash 7.10.0 Release Notes
- Logstash 7.9.3 Release Notes
- Logstash 7.9.2 Release Notes
- Logstash 7.9.1 Release Notes
- Logstash 7.9.0 Release Notes
- Logstash 7.8.1 Release Notes
- Logstash 7.8.0 Release Notes
- Logstash 7.7.1 Release Notes
- Logstash 7.7.0 Release Notes
- Logstash 7.6.2 Release Notes
- Logstash 7.6.1 Release Notes
- Logstash 7.6.0 Release Notes
- Logstash 7.5.2 Release Notes
- Logstash 7.5.1 Release Notes
- Logstash 7.5.0 Release Notes
- Logstash 7.4.2 Release Notes
- Logstash 7.4.1 Release Notes
- Logstash 7.4.0 Release Notes
- Logstash 7.3.2 Release Notes
- Logstash 7.3.1 Release Notes
- Logstash 7.3.0 Release Notes
- Logstash 7.2.1 Release Notes
- Logstash 7.2.0 Release Notes
- Logstash 7.1.1 Release Notes
- Logstash 7.1.0 Release Notes
- Logstash 7.0.1 Release Notes
- Logstash 7.0.0 Release Notes
- Logstash 7.0.0-rc2 Release Notes
- Logstash 7.0.0-rc1 Release Notes
- Logstash 7.0.0-beta1 Release Notes
- Logstash 7.0.0-alpha2 Release Notes
- Logstash 7.0.0-alpha1 Release Notes
IMPORTANT: No additional bug fixes or documentation updates
will be released for this version. For the latest information, see the
current release documentation.
Data Resiliency
editData Resiliency
editAs data flows through the event processing pipeline, Logstash may encounter situations that prevent it from delivering events to the configured output. For example, the data might contain unexpected data types, or Logstash might terminate abnormally.
To guard against data loss and ensure that events flow through the pipeline without interruption, Logstash provides the following data resiliency features.
- Persistent Queues protect against data loss by storing events in an internal queue on disk.
-
Dead Letter Queues (DLQ) provide on-disk storage for events that Logstash is
unable to process. You can easily reprocess events in the dead letter queue by
using the
dead_letter_queue
input plugin.
These resiliency features are disabled by default. To turn on these features, you must explicitly enable them in the Logstash settings file.
Was this helpful?
Thank you for your feedback.