Enriching logs with Docker metadata using Filebeat

By
Carlos Pérez-Aradros

Docker, and containers in general, have certainly changed the way we deploy applications. With many benefits on scalability and reliability they also bring new challenges, and both the methodologies and tools we use need to be updated to the new ecosystem.

Containers, unlike hosts, are ephemeral, a container can die in a host and trigger the creation of a new one in other. With this short lived instances of our applications we need the right data to track down these moving parts and keep up to speed with so many changes.

As part of our push on Beats support of containers we recently implemented a new processor add_docker_metadata, that will released with 6.0.0 beta1.

The idea is simple, yet really powerful. It enriches your logs and metrics with Docker metadata, this way you gain full visibility into your infrastructure and applications. Let’s see it in action:

Configuring Filebeat

Filebeat can easily ship Docker logs, by default they are written by Docker under /var/lib/docker/containers/<container_id>/<container_id>-json.log. As new containers are started, new files will be created to store their logs, following the same pattern, Filebeat can watch the entire directory and pick them as they appear.

These would be the settings to ship Docker container logs to Elasticsearch and enrich them with the correct metadata. If you have Filebeat installed, just edit filebeat.yml: 

filebeat.prospectors:
- type: log
  paths:
   - '/var/lib/docker/containers/*/*.log'
  json.message_key: log
  json.keys_under_root: true
  processors:
  - add_docker_metadata: ~
output.elasticsearch:
  hosts: ["elasticsearch:9200"]

Now to start shipping logs to Elasticsearch by running: 

$ ./filebeat -e -v

Exploring logs in Kibana

Once logs start flowing into Elasticsearch, you can start watching them from Kibana interface, let’s have a look to one of them. This is one of the event reported by Filebeat, corresponding to a new log line in a NGINX server running on our Docker scenario:

log event with docker metadata

Thanks to add_docker_metadata we not only get the log output but a series of fields enriching it, with useful context from Docker, like the container name, ID, Docker image, and labels!

As an example, you may want to debug what’s going on in a specific container, you just need to filter your search results by your container name.

docker logs filtering

That’s all! Simple things should be simple, and we strive to provide the best user experience, hiding the complexity where it belongs, in the code.

Try it yourself

We encourage everyone to join the testing effort and give this a try once beta1 is out! User feedback is important for us to shape features, so don’t hesitate to pass by and let us know what you think about this feature!