- Filebeat Reference: other versions:
- Overview
- Getting Started With Filebeat
- Step 1: Install Filebeat
- Step 2: Configure Filebeat
- Step 3: Configure Filebeat to use Logstash
- Step 4: Load the index template in Elasticsearch
- Step 5: Set up the Kibana dashboards
- Step 6: Start Filebeat
- Step 7: View the sample Kibana dashboards
- Quick start: modules for common log formats
- Repositories for APT and YUM
- Setting up and running Filebeat
- Upgrading Filebeat
- How Filebeat works
- Configuring Filebeat
- Specify which modules to run
- Configure inputs
- Manage multiline messages
- Specify general settings
- Load external configuration files
- Configure the internal queue
- Configure the output
- Load balance the output hosts
- Specify SSL settings
- Filter and enhance the exported data
- Parse data by using ingest node
- Set up project paths
- Set up the Kibana endpoint
- Load the Kibana dashboards
- Load the Elasticsearch index template
- Configure logging
- Use environment variables in the configuration
- Autodiscover
- YAML tips and gotchas
- Regular expression support
- HTTP Endpoint
- filebeat.reference.yml
- Beats central management
- Modules
- Exported fields
- Apache2 fields
- Auditd fields
- Beat fields
- Cloud provider metadata fields
- Docker fields
- elasticsearch fields
- haproxy fields
- Host fields
- Icinga fields
- IIS fields
- Kafka fields
- kibana fields
- Kubernetes fields
- Log file content fields
- logstash fields
- mongodb fields
- MySQL fields
- Nginx fields
- Osquery fields
- PostgreSQL fields
- Redis fields
- System fields
- Traefik fields
- Monitoring Filebeat
- Securing Filebeat
- Troubleshooting
- Migrating from Logstash Forwarder to Filebeat
- Contributing to Beats
Hints based autodiscover
editHints based autodiscover
editFilebeat supports autodiscover based on hints from the provider. The hints system looks for
hints in Kubernetes Pod annotations or Docker labels that have the prefix co.elastic.logs
. As soon as
the container starts, Filebeat will check if it contains any hints and launch the proper config for
it. Hints tell Filebeat how to get logs for the given container. By default logs will be retrieved
from the container using the docker
input. You can use hints to modify this behavior. This is the full
list of supported hints:
co.elastic.logs/disable
editFilebeat gets logs from all containers by default, you can set this hint to true
to ignore
the output of the container. Filebeat won’t read or send logs from it.
co.elastic.logs/multiline.*
editMultiline settings. See Manage multiline messages for a full list of all supported options.
co.elastic.logs/include_lines
editA list of regular expressions to match the lines that you want Filebeat to include. See Configure inputs for more info.
co.elastic.logs/exclude_lines
editA list of regular expressions to match the lines that you want Filebeat to exclude. See Configure inputs for more info.
co.elastic.logs/module
editInstead of using raw docker
input, specifies the module to use to parse logs from the container. See
Modules for the list of supported modules.
co.elastic.logs/fileset
editWhen module is configured, map container logs to module filesets. You can either configure a single fileset like this:
co.elastic.logs/fileset: access
Or configure a fileset per stream in the container (stdout and stderr):
co.elastic.logs/fileset.stdout: access co.elastic.logs/fileset.stderr: error
co.elastic.logs/raw
editWhen an entire input/module configuration needs to be completely set the raw
hint can be used. You can provide a
stringified JSON of the input configuration. raw
overrides every other hint and can be used to create bot a single or
a list of configurations.
co.elastic.logs/raw: "[{\"containers\":{\"ids\":[\"${data.container.id}\"]},\"multiline\":{\"negate\":\"true\",\"pattern\":\"^test\"},\"type\":\"docker\"}]"
co.elastic.logs/processors
editDefine a processor to be added to the Filebeat input/module configuration. See Filter and enhance the exported data for the list of supported processors.
In order to provide ordering of the processor definition, numbers can be provided. If not, the hints builder will do arbitrary ordering:
co.elastic.logs/processors.1.dissect.tokenizer: "%{key1} %{key2}" co.elastic.logs/processors.dissect.tokenizer: "%{key2} %{key1}"
In the above sample the processor definition tagged with 1
would be executed first.
Kubernetes
editKubernetes autodiscover provider supports hints in Pod annotations. To enable it just set hints.enabled
:
filebeat.autodiscover: providers: - type: kubernetes hints.enabled: true
You can annotate Kubernetes Pods with useful info to spin up Filebeat inputs or modules:
annotations: co.elastic.logs/multiline.pattern: '^\[' co.elastic.logs/multiline.negate: true co.elastic.logs/multiline.match: after
Multiple containers
editWhen a pod has multiple containers, the settings are shared unless you put the container name in the
hint. For example, these hints configure multiline settings for all containers in the pod, but set a
specific exclude_lines
hint for the container called sidecar
.
annotations: co.elastic.logs/multiline.pattern: '^\[' co.elastic.logs/multiline.negate: true co.elastic.logs/multiline.match: after co.elastic.logs.sidecar/exclude_lines: '^DBG'
Docker
editDocker autodiscover provider supports hints in labels. To enable it just set hints.enabled
:
filebeat.autodiscover: providers: - type: docker hints.enabled: true
You can label Docker containers with useful info to spin up Filebeat inputs, for example:
co.elastic.logs/module: nginx co.elastic.logs/fileset.stdout: access co.elastic.logs/fileset.stderr: error
The above labels configure Filebeat to use the Nginx module to harvest logs for this container. Access logs will be retrieved from stdout stream, and error logs from stderr.
On this page