- Filebeat Reference: other versions:
- Filebeat overview
- Quick start: installation and configuration
- Set up and run
- Upgrade
- How Filebeat works
- Configure
- Inputs
- Modules
- General settings
- Project paths
- Config file loading
- Output
- Kerberos
- SSL
- Index lifecycle management (ILM)
- Elasticsearch index template
- Kibana endpoint
- Kibana dashboards
- Processors
- Define processors
- add_cloud_metadata
- add_cloudfoundry_metadata
- add_docker_metadata
- add_fields
- add_host_metadata
- add_id
- add_kubernetes_metadata
- add_labels
- add_locale
- add_network_direction
- add_nomad_metadata
- add_observer_metadata
- add_process_metadata
- add_tags
- community_id
- convert
- copy_fields
- decode_base64_field
- decode_cef
- decode_csv_fields
- decode_json_fields
- decode_xml
- decode_xml_wineventlog
- decompress_gzip_field
- detect_mime_type
- dissect
- dns
- drop_event
- drop_fields
- extract_array
- fingerprint
- include_fields
- rate_limit
- registered_domain
- rename
- script
- timestamp
- translate_sid
- truncate_fields
- urldecode
- Autodiscover
- Internal queue
- Load balancing
- Logging
- HTTP endpoint
- Regular expression support
- Instrumentation
- filebeat.reference.yml
- How to guides
- Override configuration settings
- Load the Elasticsearch index template
- Change the index name
- Load Kibana dashboards
- Load ingest pipelines
- Enrich events with geoIP information
- Deduplicate data
- Parse data using an ingest pipeline
- Use environment variables in the configuration
- Avoid YAML formatting problems
- Migrate
log
input configurations tofilestream
- Modules
- Modules overview
- ActiveMQ module
- Apache module
- Auditd module
- AWS module
- AWS Fargate module
- Azure module
- Barracuda module
- Bluecoat module
- CEF module
- Check Point module
- Cisco module
- CoreDNS module
- CrowdStrike module
- Cyberark PAS module
- Cylance module
- Elasticsearch module
- Envoyproxy Module
- F5 module
- Fortinet module
- Google Cloud module
- Google Workspace module
- HAproxy module
- IBM MQ module
- Icinga module
- IIS module
- Imperva module
- Infoblox module
- Iptables module
- Juniper module
- Kafka module
- Kibana module
- Logstash module
- Microsoft module
- MISP module
- MongoDB module
- MSSQL module
- MySQL module
- MySQL Enterprise module
- NATS module
- NetFlow module
- Netscout module
- Nginx module
- Office 365 module
- Okta module
- Oracle module
- Osquery module
- Palo Alto Networks module
- pensando module
- PostgreSQL module
- Proofpoint module
- RabbitMQ module
- Radware module
- Redis module
- Santa module
- Snort module
- Snyk module
- Sonicwall module
- Sophos module
- Squid module
- Suricata module
- System module
- Threat Intel module
- Tomcat module
- Traefik module
- Zeek (Bro) Module
- ZooKeeper module
- Zoom module
- Zscaler module
- Exported fields
- ActiveMQ fields
- Apache fields
- Auditd fields
- AWS fields
- AWS CloudWatch fields
- AWS Fargate fields
- Azure fields
- Barracuda Web Application Firewall fields
- Beat fields
- Blue Coat Director fields
- Decode CEF processor fields fields
- CEF fields
- Checkpoint fields
- Cisco fields
- Cloud provider metadata fields
- Coredns fields
- Crowdstrike fields
- CyberArk PAS fields
- CylanceProtect fields
- Docker fields
- ECS fields
- Elasticsearch fields
- Envoyproxy fields
- Big-IP Access Policy Manager fields
- Fortinet fields
- Google Cloud Platform (GCP) fields
- google_workspace fields
- HAProxy fields
- Host fields
- ibmmq fields
- Icinga fields
- IIS fields
- Imperva SecureSphere fields
- Infoblox NIOS fields
- iptables fields
- Jolokia Discovery autodiscover provider fields
- Juniper JUNOS fields
- Kafka fields
- kibana fields
- Kubernetes fields
- Log file content fields
- logstash fields
- Microsoft fields
- MISP fields
- mongodb fields
- mssql fields
- MySQL fields
- MySQL Enterprise fields
- NATS fields
- NetFlow fields
- Arbor Peakflow SP fields
- Nginx fields
- Office 365 fields
- Okta fields
- Oracle fields
- Osquery fields
- panw fields
- Pensando fields
- PostgreSQL fields
- Process fields
- Proofpoint Email Security fields
- RabbitMQ fields
- Radware DefensePro fields
- Redis fields
- s3 fields
- Google Santa fields
- Snort/Sourcefire fields
- Snyk fields
- Sonicwall-FW fields
- sophos fields
- Squid fields
- Suricata fields
- System fields
- threatintel fields
- Apache Tomcat fields
- Traefik fields
- Zeek fields
- ZooKeeper fields
- Zoom fields
- Zscaler NSS fields
- Monitor
- Secure
- Troubleshoot
- Get help
- Debug
- Common problems
- Error extracting container id while using Kubernetes metadata
- Can’t read log files from network volumes
- Filebeat isn’t collecting lines from a file
- Too many open file handlers
- Registry file is too large
- Inode reuse causes Filebeat to skip lines
- Log rotation results in lost or duplicate events
- Open file handlers cause issues with Windows file rotation
- Filebeat is using too much CPU
- Dashboard in Kibana is breaking up data fields incorrectly
- Fields are not indexed or usable in Kibana visualizations
- Filebeat isn’t shipping the last line of a file
- Filebeat keeps open file handlers of deleted files for a long time
- Filebeat uses too much bandwidth
- Error loading config file
- Found unexpected or unknown characters
- Logstash connection doesn’t work
- Publishing to Logstash fails with "connection reset by peer" message
- @metadata is missing in Logstash
- Not sure whether to use Logstash or Beats
- SSL client fails to connect to Logstash
- Monitoring UI shows fewer Beats than expected
- Dashboard could not locate the index-pattern
- High RSS memory usage due to MADV settings
- Contribute to Beats
Define processors
editDefine processors
editYou can use processors to filter and enhance data before sending it to the configured output. To define a processor, you specify the processor name, an optional condition, and a set of parameters:
processors: - <processor_name>: when: <condition> <parameters> - <processor_name>: when: <condition> <parameters> ...
Where:
-
<processor_name>
specifies a processor that performs some kind of action, such as selecting the fields that are exported or adding metadata to the event. -
<condition>
specifies an optional condition. If the condition is present, then the action is executed only if the condition is fulfilled. If no condition is set, then the action is always executed. -
<parameters>
is the list of parameters to pass to the processor.
More complex conditional processing can be accomplished by using the if-then-else processor configuration. This allows multiple processors to be executed based on a single condition.
processors: - if: <condition> then: - <processor_name>: <parameters> - <processor_name>: <parameters> ... else: - <processor_name>: <parameters> - <processor_name>: <parameters> ...
|
|
|
Where are processors valid?
editProcessors are valid:
- At the top-level in the configuration. The processor is applied to all data collected by Filebeat.
-
Under a specific input. The processor is applied to the data collected for that input.
- type: <input_type> processors: - <processor_name>: when: <condition> <parameters> ...
Similarly, for Filebeat modules, you can define processors under the
input
section of the module definition.
Processors
editThe supported processors are:
-
add_cloud_metadata
-
add_cloudfoundry_metadata
-
add_docker_metadata
-
add_fields
-
add_host_metadata
-
add_id
-
add_kubernetes_metadata
-
add_labels
-
add_locale
-
add_nomad_metadata
-
add_observer_metadata
-
add_process_metadata
-
add_tags
-
community_id
-
convert
-
copy_fields
-
decode_base64_field
-
decode_cef
-
decode_csv_fields
-
decode_json_fields
-
decode_xml
-
decode_xml_wineventlog
-
decompress_gzip_field
-
detect_mime_type
-
dissect
-
dns
-
drop_event
-
drop_fields
-
extract_array
-
fingerprint
-
include_fields
-
rate_limit
-
registered_domain
-
rename
-
script
-
timestamp
-
translate_sid
-
truncate_fields
-
urldecode
Conditions
editEach condition receives a field to compare. You can specify multiple fields
under the same condition by using AND
between the fields (for example,
field1 AND field2
).
For each field, you can specify a simple field name or a nested map, for example
dns.question.name
.
See Exported fields for a list of all the fields that are exported by Filebeat.
The supported conditions are:
equals
editWith the equals
condition, you can compare if a field has a certain value.
The condition accepts only an integer or a string value.
For example, the following condition checks if the response code of the HTTP transaction is 200:
equals: http.response.code: 200
contains
editThe contains
condition checks if a value is part of a field. The field can be
a string or an array of strings. The condition accepts only a string value.
For example, the following condition checks if an error is part of the transaction status:
contains: status: "Specific error"
regexp
editThe regexp
condition checks the field against a regular expression. The
condition accepts only strings.
For example, the following condition checks if the process name starts with
foo
:
regexp: system.process.name: "^foo.*"
range
editThe range
condition checks if the field is in a certain range of values. The
condition supports lt
, lte
, gt
and gte
. The condition accepts only
integer or float values.
For example, the following condition checks for failed HTTP transactions by
comparing the http.response.code
field with 400.
range: http.response.code: gte: 400
This can also be written as:
range: http.response.code.gte: 400
The following condition checks if the CPU usage in percentage has a value between 0.5 and 0.8.
range: system.cpu.user.pct.gte: 0.5 system.cpu.user.pct.lt: 0.8
network
editThe network
condition checks if the field is in a certain IP network range.
Both IPv4 and IPv6 addresses are supported. The network range may be specified
using CIDR notation, like "192.0.2.0/24" or "2001:db8::/32", or by using one of
these named ranges:
-
loopback
- Matches loopback addresses in the range of127.0.0.0/8
or::1/128
. -
unicast
- Matches global unicast addresses defined in RFC 1122, RFC 4632, and RFC 4291 with the exception of the IPv4 broadcast address (255.255.255.255
). This includes private address ranges. -
multicast
- Matches multicast addresses. -
interface_local_multicast
- Matches IPv6 interface-local multicast addresses. -
link_local_unicast
- Matches link-local unicast addresses. -
link_local_multicast
- Matches link-local multicast addresses. -
private
- Matches private address ranges defined in RFC 1918 (IPv4) and RFC 4193 (IPv6). -
public
- Matches addresses that are not loopback, unspecified, IPv4 broadcast, link local unicast, link local multicast, interface local multicast, or private. -
unspecified
- Matches unspecified addresses (either the IPv4 address "0.0.0.0" or the IPv6 address "::").
The following condition returns true if the source.ip
value is within the
private address space.
network: source.ip: private
This condition returns true if the destination.ip
value is within the
IPv4 range of 192.168.1.0
- 192.168.1.255
.
network: destination.ip: '192.168.1.0/24'
And this condition returns true when destination.ip
is within any of the given
subnets.
network: destination.ip: ['192.168.1.0/24', '10.0.0.0/8', loopback]
has_fields
editThe has_fields
condition checks if all the given fields exist in the
event. The condition accepts a list of string values denoting the field names.
For example, the following condition checks if the http.response.code
field
is present in the event.
has_fields: ['http.response.code']
or
editThe or
operator receives a list of conditions.
or: - <condition1> - <condition2> - <condition3> ...
For example, to configure the condition
http.response.code = 304 OR http.response.code = 404
:
or: - equals: http.response.code: 304 - equals: http.response.code: 404
and
editThe and
operator receives a list of conditions.
and: - <condition1> - <condition2> - <condition3> ...
For example, to configure the condition
http.response.code = 200 AND status = OK
:
and: - equals: http.response.code: 200 - equals: status: OK
To configure a condition like <condition1> OR <condition2> AND <condition3>
:
or: - <condition1> - and: - <condition2> - <condition3>
not
editThe not
operator receives the condition to negate.
not: <condition>
For example, to configure the condition NOT status = OK
:
not: equals: status: OK
On this page