- Filebeat Reference: other versions:
- Filebeat overview
- Quick start: installation and configuration
- Set up and run
- Upgrade
- How Filebeat works
- Configure
- Inputs
- Modules
- General settings
- Project paths
- Config file loading
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- append
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_duration
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- move_fields
- parse_aws_vpc_flow_log
- rate_limit
- registered_domain
- rename
- replace
- script
- syslog
- timestamp
- translate_sid
- truncate_fields
- urldecode
- Autodiscover
- Internal queue
- Logging
- HTTP endpoint
- Regular expression support
- Instrumentation
- Feature flags
- filebeat.reference.yml
- How to guides
- Override configuration settings
- Load the Elasticsearch index template
- Change the index name
- Load Kibana dashboards
- Load ingest pipelines
- Enrich events with geoIP information
- Deduplicate data
- Parse data using an ingest pipeline
- Use environment variables in the configuration
- Avoid YAML formatting problems
- Migrate
log
input configurations tofilestream
- Modules
- Modules overview
- ActiveMQ module
- Apache module
- Auditd module
- AWS module
- AWS Fargate module
- Azure module
- Barracuda module
- Bluecoat module
- CEF module
- Check Point module
- Cisco module
- CoreDNS module
- CrowdStrike module
- Cyberark PAS module
- Cylance module
- Elasticsearch module
- Envoyproxy Module
- F5 module
- Fortinet module
- Google Cloud module
- Google Workspace module
- HAproxy module
- IBM MQ module
- Icinga module
- IIS module
- Imperva module
- Infoblox module
- Iptables module
- Juniper module
- Kafka module
- Kibana module
- Logstash module
- Microsoft module
- MISP module
- MongoDB module
- MSSQL module
- MySQL module
- MySQL Enterprise module
- NATS module
- NetFlow module
- Netscout module
- Nginx module
- Office 365 module
- Okta module
- Oracle module
- Osquery module
- Palo Alto Networks module
- pensando module
- PostgreSQL module
- Proofpoint module
- RabbitMQ module
- Radware module
- Redis module
- Salesforce module
- Santa module
- Snort module
- Snyk module
- Sonicwall module
- Sophos module
- Squid module
- Suricata module
- System module
- Threat Intel module
- Tomcat module
- Traefik module
- Zeek (Bro) Module
- ZooKeeper module
- Zoom module
- Zscaler module
- Exported fields
- ActiveMQ fields
- Apache fields
- Auditd fields
- AWS fields
- AWS CloudWatch fields
- AWS Fargate fields
- Azure fields
- Barracuda Web Application Firewall fields
- Beat fields
- Blue Coat Director fields
- Decode CEF processor fields fields
- CEF fields
- Checkpoint fields
- Cisco fields
- Cloud provider metadata fields
- Coredns fields
- Crowdstrike fields
- CyberArk PAS fields
- CylanceProtect fields
- Docker fields
- ECS fields
- Elasticsearch fields
- Envoyproxy fields
- Big-IP Access Policy Manager fields
- Fortinet fields
- Google Cloud Platform (GCP) fields
- google_workspace fields
- HAProxy fields
- Host fields
- ibmmq fields
- Icinga fields
- IIS fields
- Imperva SecureSphere fields
- Infoblox NIOS fields
- iptables fields
- Jolokia Discovery autodiscover provider fields
- Juniper JUNOS fields
- Kafka fields
- kibana fields
- Kubernetes fields
- Log file content fields
- logstash fields
- Lumberjack fields
- Microsoft fields
- MISP fields
- mongodb fields
- mssql fields
- MySQL fields
- MySQL Enterprise fields
- NATS fields
- NetFlow fields
- Arbor Peakflow SP fields
- Nginx fields
- Office 365 fields
- Okta fields
- Oracle fields
- Osquery fields
- panw fields
- Pensando fields
- PostgreSQL fields
- Process fields
- Proofpoint Email Security fields
- RabbitMQ fields
- Radware DefensePro fields
- Redis fields
- s3 fields
- Salesforce fields
- Google Santa fields
- Snort/Sourcefire fields
- Snyk fields
- Sonicwall-FW fields
- sophos fields
- Squid fields
- Suricata fields
- System fields
- threatintel fields
- Apache Tomcat fields
- Traefik fields
- Zeek fields
- ZooKeeper fields
- Zoom fields
- Zscaler NSS fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Understand logged metrics
- Common problems
- Error extracting container id while using Kubernetes metadata
- Can’t read log files from network volumes
- Filebeat isn’t collecting lines from a file
- Too many open file handlers
- Registry file is too large
- Inode reuse causes Filebeat to skip lines
- Log rotation results in lost or duplicate events
- Open file handlers cause issues with Windows file rotation
- Filebeat is using too much CPU
- Dashboard in Kibana is breaking up data fields incorrectly
- Fields are not indexed or usable in Kibana visualizations
- Filebeat isn’t shipping the last line of a file
- Filebeat keeps open file handlers of deleted files for a long time
- Filebeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- High RSS memory usage due to MADV settings
- Contribute to Beats
Add cloud metadata
editAdd cloud metadata
editThe add_cloud_metadata
processor enriches each event with instance metadata
from the machine’s hosting provider. At startup it will query a list of hosting
providers and cache the instance metadata.
The following cloud providers are supported:
- Amazon Web Services (AWS)
- Digital Ocean
- Google Compute Engine (GCE)
- Tencent Cloud (QCloud)
- Alibaba Cloud (ECS)
- Huawei Cloud (ECS)
- Azure Virtual Machine
- Openstack Nova
- Hetzner Cloud
The Alibaba Cloud and Tencent cloud providers are disabled by default, because
they require to access a remote host. The providers
setting allows users to
select a list of default providers to query.
The simple configuration below enables the processor.
processors: - add_cloud_metadata: ~
The add_cloud_metadata
processor has three optional configuration settings.
The first one is timeout
which specifies the maximum amount of time to wait
for a successful response when detecting the hosting provider. The default
timeout value is 3s
.
If a timeout occurs then no instance metadata will be added to the events. This makes it possible to enable this processor for all your deployments (in the cloud or on-premise).
The second optional setting is providers
. The providers
settings accepts a
list of cloud provider names to be used. If providers
is not configured, then
all providers that do not access a remote endpoint are enabled by default.
List of names the providers
setting supports:
- "alibaba", or "ecs" for the Alibaba Cloud provider (disabled by default).
- "azure" for Azure Virtual Machine (enabled by default).
- "digitalocean" for Digital Ocean (enabled by default).
- "aws", or "ec2" for Amazon Web Services (enabled by default).
- "gcp" for Google Copmute Enging (enabled by default).
- "openstack", or "nova" for Openstack Nova (enabled by default).
- "openstack-ssl", or "nova-ssl" for Openstack Nova when SSL metadata APIs are enabled (enabled by default).
- "tencent", or "qcloud" for Tencent Cloud (disabled by default).
- "huawei" for Huawei Cloud (enabled by default).
- "hetzner" for Hetzner Cloud (enabled by default).
The third optional configuration setting is overwrite
. When overwrite
is
true
, add_cloud_metadata
overwrites existing cloud.*
fields (false
by
default).
The add_cloud_metadata
processor supports SSL options to configure the http
client used to query cloud metadata. See SSL for more information.
The metadata that is added to events varies by hosting provider. Below are examples for each of the supported providers.
AWS
{ "cloud": { "account.id": "123456789012", "availability_zone": "us-east-1c", "instance.id": "i-4e123456", "machine.type": "t2.medium", "image.id": "ami-abcd1234", "provider": "aws", "region": "us-east-1" } }
Digital Ocean
{ "cloud": { "instance.id": "1234567", "provider": "digitalocean", "region": "nyc2" } }
GCP
{ "cloud": { "availability_zone": "us-east1-b", "instance.id": "1234556778987654321", "machine.type": "f1-micro", "project.id": "my-dev", "provider": "gcp" } }
Tencent Cloud
{ "cloud": { "availability_zone": "gz-azone2", "instance.id": "ins-qcloudv5", "provider": "qcloud", "region": "china-south-gz" } }
Huawei Cloud
{ "cloud": { "availability_zone": "cn-east-2b", "instance.id": "37da9890-8289-4c58-ba34-a8271c4a8216", "provider": "huawei", "region": "cn-east-2" } }
Alibaba Cloud
This metadata is only available when VPC is selected as the network type of the ECS instance.
{ "cloud": { "availability_zone": "cn-shenzhen", "instance.id": "i-wz9g2hqiikg0aliyun2b", "provider": "ecs", "region": "cn-shenzhen-a" } }
Azure Virtual Machine
{ "cloud": { "provider": "azure", "instance.id": "04ab04c3-63de-4709-a9f9-9ab8c0411d5e", "instance.name": "test-az-vm", "machine.type": "Standard_D3_v2", "region": "eastus2" } }
Openstack Nova
{ "cloud": { "instance.name": "test-998d932195.mycloud.tld", "instance.id": "i-00011a84", "availability_zone": "xxxx-az-c", "provider": "openstack", "machine.type": "m2.large" } }
Hetzner Cloud
{ "cloud": { "availability_zone": "hel1-dc2", "instance.name": "my-hetzner-instance", "instance.id": "111111", "provider": "hetzner", "region": "eu-central" } }