- Metricbeat Reference: other versions:
- Metricbeat overview
- Quick start: installation and configuration
- Set up and run
- Upgrade Metricbeat
- How Metricbeat works
- Configure
- Modules
- General settings
- Project paths
- Config file loading
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- append
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_duration
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- move_fields
- rate_limit
- registered_domain
- rename
- replace
- script
- syslog
- translate_sid
- truncate_fields
- urldecode
- Autodiscover
- Internal queue
- Logging
- HTTP endpoint
- Regular expression support
- Instrumentation
- Feature flags
- metricbeat.reference.yml
- How to guides
- Modules
- ActiveMQ module
- Aerospike module
- Airflow module
- Apache module
- AWS module
- AWS billing metricset
- AWS cloudwatch metricset
- AWS dynamodb metricset
- AWS ebs metricset
- AWS ec2 metricset
- AWS elb metricset
- AWS kinesis metricset
- AWS lambda metricset
- AWS natgateway metricset
- AWS rds metricset
- AWS s3_daily_storage metricset
- AWS s3_request metricset
- AWS sns metricset
- AWS sqs metricset
- AWS transitgateway metricset
- AWS usage metricset
- AWS vpn metricset
- AWS Fargate module
- Azure module
- Azure app_insights metricset
- Azure app_state metricset
- Azure billing metricset
- Azure compute_vm metricset
- Azure compute_vm_scaleset metricset
- Azure container_instance metricset
- Azure container_registry metricset
- Azure container_service metricset
- Azure database_account metricset
- Azure monitor metricset
- Azure storage metricset
- Beat module
- Ceph module
- Ceph cluster_disk metricset
- Ceph cluster_health metricset
- Ceph cluster_status metricset
- Ceph mgr_cluster_disk metricset
- Ceph mgr_cluster_health metricset
- Ceph mgr_osd_perf metricset
- Ceph mgr_osd_pool_stats metricset
- Ceph mgr_osd_tree metricset
- Ceph mgr_pool_disk metricset
- Ceph monitor_health metricset
- Ceph osd_df metricset
- Ceph osd_tree metricset
- Ceph pool_disk metricset
- Cloudfoundry module
- CockroachDB module
- Consul module
- Containerd module
- Coredns module
- Couchbase module
- CouchDB module
- Docker module
- Dropwizard module
- Elasticsearch module
- Elasticsearch ccr metricset
- Elasticsearch cluster_stats metricset
- Elasticsearch enrich metricset
- Elasticsearch index metricset
- Elasticsearch index_recovery metricset
- Elasticsearch index_summary metricset
- Elasticsearch ingest_pipeline metricset
- Elasticsearch ml_job metricset
- Elasticsearch node metricset
- Elasticsearch node_stats metricset
- Elasticsearch pending_tasks metricset
- Elasticsearch shard metricset
- Enterprise Search module
- Envoyproxy module
- Etcd module
- Google Cloud Platform module
- Google Cloud Platform billing metricset
- Google Cloud Platform compute metricset
- Google Cloud Platform dataproc metricset
- Google Cloud Platform firestore metricset
- Google Cloud Platform gke metricset
- Google Cloud Platform loadbalancing metricset
- Google Cloud Platform metrics metricset
- Google Cloud Platform pubsub metricset
- Google Cloud Platform storage metricset
- Golang module
- Graphite module
- HAProxy module
- HTTP module
- IBM MQ module
- IIS module
- Istio module
- Jolokia module
- Kafka module
- Kibana module
- Kubernetes module
- Kubernetes apiserver metricset
- Kubernetes container metricset
- Kubernetes controllermanager metricset
- Kubernetes event metricset
- Kubernetes node metricset
- Kubernetes pod metricset
- Kubernetes proxy metricset
- Kubernetes scheduler metricset
- Kubernetes state_container metricset
- Kubernetes state_cronjob metricset
- Kubernetes state_daemonset metricset
- Kubernetes state_deployment metricset
- Kubernetes state_job metricset
- Kubernetes state_node metricset
- Kubernetes state_persistentvolumeclaim metricset
- Kubernetes state_pod metricset
- Kubernetes state_replicaset metricset
- Kubernetes state_resourcequota metricset
- Kubernetes state_service metricset
- Kubernetes state_statefulset metricset
- Kubernetes state_storageclass metricset
- Kubernetes system metricset
- Kubernetes volume metricset
- KVM module
- Linux module
- Logstash module
- Memcached module
- MongoDB module
- MSSQL module
- Munin module
- MySQL module
- NATS module
- Nginx module
- Openmetrics module
- Oracle module
- PHP_FPM module
- PostgreSQL module
- Prometheus module
- RabbitMQ module
- Redis module
- Redis Enterprise module
- SQL module
- Stan module
- Statsd module
- SyncGateway module
- System module
- System core metricset
- System cpu metricset
- System diskio metricset
- System entropy metricset
- System filesystem metricset
- System fsstat metricset
- System load metricset
- System memory metricset
- System network metricset
- System network_summary metricset
- System process metricset
- System process_summary metricset
- System raid metricset
- System service metricset
- System socket metricset
- System socket_summary metricset
- System uptime metricset
- System users metricset
- Tomcat module
- Traefik module
- uWSGI module
- vSphere module
- Windows module
- ZooKeeper module
- Exported fields
- ActiveMQ fields
- Aerospike fields
- Airflow fields
- Apache fields
- AWS fields
- AWS Fargate fields
- Azure fields
- Beat fields
- Beat fields
- Ceph fields
- Cloud provider metadata fields
- Cloudfoundry fields
- CockroachDB fields
- Common fields
- Consul fields
- Containerd fields
- Coredns fields
- Couchbase fields
- CouchDB fields
- Docker fields
- Docker fields
- Dropwizard fields
- ECS fields
- Elasticsearch fields
- Enterprise Search fields
- Envoyproxy fields
- Etcd fields
- Google Cloud Platform fields
- Golang fields
- Graphite fields
- HAProxy fields
- Host fields
- HTTP fields
- IBM MQ fields
- IIS fields
- Istio fields
- Jolokia fields
- Jolokia Discovery autodiscover provider fields
- Kafka fields
- Kibana fields
- Kubernetes fields
- Kubernetes fields
- KVM fields
- Linux fields
- Logstash fields
- Memcached fields
- MongoDB fields
- MSSQL fields
- Munin fields
- MySQL fields
- NATS fields
- Nginx fields
- Openmetrics fields
- Oracle fields
- PHP_FPM fields
- PostgreSQL fields
- Process fields
- Prometheus fields
- Prometheus typed metrics fields
- RabbitMQ fields
- Redis fields
- Redis Enterprise fields
- SQL fields
- Stan fields
- Statsd fields
- SyncGateway fields
- System fields
- Tomcat fields
- Traefik fields
- uWSGI fields
- vSphere fields
- Windows fields
- ZooKeeper fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Common problems
- "open /compat/linux/proc: no such file or directory" error on FreeBSD
- Metricbeat collects system metrics for interfaces you didn’t configure
- Metricbeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- High RSS memory usage due to MADV settings
- Contribute to Beats
Configure the Kafka output
editConfigure the Kafka output
editThe Kafka output sends events to Apache Kafka.
To use this output, edit the Metricbeat configuration file to disable the Elasticsearch output by commenting it out, and enable the Kafka output by uncommenting the Kafka section.
For Kafka version 0.10.0.0+ the message creation timestamp is set by beats and equals to the initial timestamp of the event. This affects the retention policy in Kafka: for example, if a beat event was created 2 weeks ago, the retention policy is set to 7 days and the message from beats arrives to Kafka today, it’s going to be immediately discarded since the timestamp value is before the last 7 days. It’s possible to change this behavior by setting timestamps on message arrival instead, so the message is not discarded but kept for 7 more days. To do that, please set log.message.timestamp.type
to LogAppendTime
(default CreateTime
) in the Kafka configuration.
Example configuration:
output.kafka: # initial brokers for reading cluster metadata hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"] # message topic selection + partitioning topic: '%{[fields.log_topic]}' partition.round_robin: reachable_only: false required_acks: 1 compression: gzip max_message_bytes: 1000000
Events bigger than max_message_bytes
will be dropped. To avoid this problem, make sure Metricbeat does not generate events bigger than max_message_bytes
.
Compatibility
editThis output can connect to Kafka version 0.8.2.0 and later. Older versions might work as well, but are not supported.
Configuration options
editYou can specify the following options in the kafka
section of the metricbeat.yml
config file:
enabled
editThe enabled
config is a boolean setting to enable or disable the output. If set
to false, the output is disabled.
The default value is true
.
hosts
editThe list of Kafka broker addresses from where to fetch the cluster metadata. The cluster metadata contain the actual Kafka brokers events are published to.
version
editKafka protocol version that Metricbeat will request when connecting. Defaults to 1.0.0.
Valid values are all kafka releases in between 0.8.2.0
and 2.6.0
.
The protocol version controls the Kafka client features available to Metricbeat; it does not prevent Metricbeat from connecting to Kafka versions newer than the protocol version.
See Compatibility for information on supported versions.
username
editThe username for connecting to Kafka. If username is configured, the password must be configured as well.
password
editThe password for connecting to Kafka.
sasl.mechanism
editThe SASL mechanism to use when connecting to Kafka. It can be one of:
-
PLAIN
for SASL/PLAIN. -
SCRAM-SHA-256
for SCRAM-SHA-256. -
SCRAM-SHA-512
for SCRAM-SHA-512.
If sasl.mechanism
is not set, PLAIN
is used if username
and password
are provided. Otherwise, SASL authentication is disabled.
To use GSSAPI
mechanism to authenticate with Kerberos, you must leave this
field empty, and use the kerberos
options.
topic
editThe Kafka topic used for produced events.
You can set the topic dynamically by using a format string to access any
event field. For example, this configuration uses a custom field,
fields.log_topic
, to set the topic for each event:
topic: '%{[fields.log_topic]}'
To learn how to add custom fields to events, see the
fields
option.
See the topics
setting for other ways to set the
topic dynamically.
topics
editAn array of topic selector rules. Each rule specifies the topic
to use for
events that match the rule. During publishing, Metricbeat sets the topic
for each event based on the first matching rule in the array. Rules
can contain conditionals, format string-based fields, and name mappings. If the
topics
setting is missing or no rule matches, the
topic
field is used.
Rule settings:
-
topic
-
The topic format string to use. If this string contains field
references, such as
%{[fields.name]}
, the fields must exist, or the rule fails. -
mappings
-
A dictionary that takes the value returned by
topic
and maps it to a new name. -
default
-
The default string value to use if
mappings
does not find a match. -
when
- A condition that must succeed in order to execute the current rule. All the conditions supported by processors are also supported here.
The following example sets the topic based on whether the message field contains the specified string:
output.kafka: hosts: ["localhost:9092"] topic: "logs-%{[agent.version]}" topics: - topic: "critical-%{[agent.version]}" when.contains: message: "CRITICAL" - topic: "error-%{[agent.version]}" when.contains: message: "ERR"
This configuration results in topics named critical-8.7.1
,
error-8.7.1
, and logs-8.7.1
.
key
editOptional formatted string specifying the Kafka event key. If configured, the event key can be extracted from the event using a format string.
See the Kafka documentation for the implications of a particular choice of key; by default, the key is chosen by the Kafka cluster.
partition
editKafka output broker event partitioning strategy. Must be one of random
,
round_robin
, or hash
. By default the hash
partitioner is used.
random.group_events
: Sets the number of events to be published to the same
partition, before the partitioner selects a new partition by random. The
default value is 1 meaning after each event a new partition is picked randomly.
round_robin.group_events
: Sets the number of events to be published to the
same partition, before the partitioner selects the next partition. The default
value is 1 meaning after each event the next partition will be selected.
hash.hash
: List of fields used to compute the partitioning hash value from.
If no field is configured, the events key
value will be used.
hash.random
: Randomly distribute events if no hash or key value can be computed.
All partitioners will try to publish events to all partitions by default. If a
partition’s leader becomes unreachable for the beat, the output might block. All
partitioners support setting reachable_only
to overwrite this
behavior. If reachable_only
is set to true
, events will be published to
available partitions only.
Publishing to a subset of available partitions potentially increases resource usage because events may become unevenly distributed.
headers
editA header is a key-value pair, and multiple headers can be included with the same key
. Only string values are supported. These headers will be included in each produced Kafka message.
output.kafka: hosts: ["localhost:9092"] topic: "logs-%{[agent.version]}" headers: - key: "some-key" value: "some value" - key: "another-key" value: "another value"
client_id
editThe configurable ClientID used for logging, debugging, and auditing purposes. The default is "beats".
codec
editOutput codec configuration. If the codec
section is missing, events will be json encoded.
See Change the output codec for more information.
metadata
editKafka metadata update settings. The metadata do contain information about brokers, topics, partition, and active leaders to use for publishing.
-
refresh_frequency
- Metadata refresh interval. Defaults to 10 minutes.
-
full
-
Strategy to use when fetching metadata, when this option is
true
, the client will maintain a full set of metadata for all the available topics, if the this option is set tofalse
it will only refresh the metadata for the configured topics. The default is false. -
retry.max
- Total number of metadata update retries when cluster is in middle of leader election. The default is 3.
-
retry.backoff
- Waiting time between retries during leader elections. Default is 250ms.
max_retries
editThe number of times to retry publishing an event after a publishing failure. After the specified number of retries, the events are typically dropped.
Set max_retries
to a value less than 0 to retry until all events are published.
The default is 3.
backoff.init
editThe number of seconds to wait before trying to republish to Kafka
after a network error. After waiting backoff.init
seconds, Metricbeat
tries to republish. If the attempt fails, the backoff timer is increased
exponentially up to backoff.max
. After a successful publish, the backoff
timer is reset. The default is 1s.
backoff.max
editThe maximum number of seconds to wait before attempting to republish to Kafka after a network error. The default is 60s.
bulk_max_size
editThe maximum number of events to bulk in a single Kafka request. The default is 2048.
bulk_flush_frequency
editDuration to wait before sending bulk Kafka request. 0 is no delay. The default is 0.
timeout
editThe number of seconds to wait for responses from the Kafka brokers before timing out. The default is 30 (seconds).
broker_timeout
editThe maximum duration a broker will wait for number of required ACKs. The default is 10s.
channel_buffer_size
editPer Kafka broker number of messages buffered in output pipeline. The default is 256.
keep_alive
editThe keep-alive period for an active network connection. If 0s, keep-alives are disabled. The default is 0 seconds.
compression
editSets the output compression codec. Must be one of none
, snappy
, lz4
and gzip
. The default is gzip
.
Known issue with Azure Event Hub for Kafka
When targeting Azure Event Hub for Kafka, set compression
to none
as the provided codecs are not supported.
compression_level
editSets the compression level used by gzip. Setting this value to 0 disables compression. The compression level must be in the range of 1 (best speed) to 9 (best compression).
Increasing the compression level will reduce the network usage but will increase the cpu usage.
The default value is 4.
max_message_bytes
editThe maximum permitted size of JSON-encoded messages. Bigger messages will be dropped. The default value is 1000000 (bytes). This value should be equal to or less than the broker’s message.max.bytes
.
required_acks
editThe ACK reliability level required from broker. 0=no response, 1=wait for local commit, -1=wait for all replicas to commit. The default is 1.
Note: If set to 0, no ACKs are returned by Kafka. Messages might be lost silently on error.
ssl
editConfiguration options for SSL parameters like the root CA for Kafka connections.
The Kafka host keystore should be created with the
-keyalg RSA
argument to ensure it uses a cipher supported by
Filebeat’s Kafka library.
See SSL for more information.
kerberos
editThis functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
Configuration options for Kerberos authentication.
See Kerberos for more information.
On this page
- Compatibility
- Configuration options
enabled
hosts
version
username
password
sasl.mechanism
topic
topics
key
partition
headers
client_id
codec
metadata
max_retries
backoff.init
backoff.max
bulk_max_size
bulk_flush_frequency
timeout
broker_timeout
channel_buffer_size
keep_alive
compression
compression_level
max_message_bytes
required_acks
ssl
kerberos