Running Logstash on Docker

edit

Docker images for Logstash are available from the Elastic Docker registry.

Obtaining Logstash for Docker is as simple as issuing a docker pull command against the Elastic Docker registry.

The Docker image for Logstash 5.3.3 can be retrieved with the following command:

docker pull docker.elastic.co/logstash/logstash:5.3.3

Configuring Logstash for Docker

edit

Logstash differentiates between two types of configuration: Settings and Pipeline Configuration.

Pipeline Configuration

edit

It is essential to place your pipeline configuration where it can be found by Logstash. By default, the container will look in /usr/share/logstash/pipeline/ for pipeline configuration files.

In this example we use a bind-mounted volume to provide the configuration via the docker run command:

docker run --rm -it -v ~/pipeline/:/usr/share/logstash/pipeline/ docker.elastic.co/logstash/logstash:5.3.3

Every file in the host directory ~/pipeline/ will then be parsed by Logstash as pipeline configuration.

If you don’t provide configuration to Logstash, it will run with a minimal config that listens for messages from the Beats input plugin and echoes any that are received to stdout. In this case, the startup logs will be similar to the following:

Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties.
[2016-10-26T05:11:34,992][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2016-10-26T05:11:35,068][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2016-10-26T05:11:35,078][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2016-10-26T05:11:35,078][INFO ][logstash.pipeline        ] Pipeline main started
[2016-10-26T05:11:35,105][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

This is the default configuration for the image, defined in /usr/share/logstash/pipeline/logstash.conf. If this is the behaviour that you are observing, ensure that your pipeline configuration is being picked up correctly, and that you are replacing either logstash.conf or the entire pipeline directory.

Settings Files

edit

Settings files can also be provided through bind-mounts. Logstash expects to find them at /usr/share/logstash/config/.

It’s possible to provide an entire directory containing all needed files:

docker run --rm -it -v ~/settings/:/usr/share/logstash/config/ docker.elastic.co/logstash/logstash:5.3.3

Alternatively, a single file can be mounted:

docker run --rm -it -v ~/settings/logstash.yml:/usr/share/logstash/config/logstash.yml docker.elastic.co/logstash/logstash:5.3.3

Bind-mounted configuration files will retain the same permissions and ownership within the container that they have on the host system. Be sure to set permissions such that the files will be readable and, ideally, not writeable by the container’s logstash user (UID 1000).

Custom Images

edit

Bind-mounted configuration is not the only option, naturally. If you prefer the Immutable Infrastructure approach, you can prepare a custom image containing your configuration by using a Dockerfile like this one:

FROM docker.elastic.co/logstash/logstash:5.3.3
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
ADD pipeline/ /usr/share/logstash/pipeline/
ADD config/ /usr/share/logstash/config/

Be sure to replace or delete logstash.conf in your custom image, so that you don’t retain the example config from the base image.

Logging Configuration

edit

Under Docker, Logstash logs go to standard output by default. To change this behaviour, use any of the techniques above to replace the file at /usr/share/logstash/config/log4j2.properties.