« Getting started with the Elastic Stack Running the Elastic Stack on Kubernetes »

›

Running the Elastic Stack ("ELK") on Docker

Running the Elastic Stack ("ELK") on Docker

The Elastic Docker registry contains Docker images for all the products in the Elastic Stack: https://www.docker.elastic.co/.

You can start the Elastic Stack in Docker with security enabled and configured by default. This option is great for quickly getting started with Elasticsearch and Kibana for testing and development.

You can also start the Elastic Stack with Docker Compose to create a secured, multi-node cluster with a connected Kibana instance. This option results in a more resilient cluster with greater capacity and reliability.

Start Elasticsearch and Kibana in Docker

Starting in Elasticsearch 8.0, security is enabled and configured by default.

If you’re starting a single-node Elasticsearch cluster in a Docker container, security will be automatically enabled and configured for you. When you start Elasticsearch for the first time, the following security configuration occurs automatically:

Certificates and keys are generated for the transport and HTTP layers.
The Transport Layer Security (TLS) configuration settings are written to elasticsearch.yml.
A password is generated for the elastic user.
An enrollment token is generated for Kibana.
An enrollment token is generated for other Elasticsearch nodes.

You can then start Kibana and enter the enrollment token, which is valid for 30 minutes. This token automatically applies the security settings from your Elasticsearch cluster, authenticates to Elasticsearch with the kibana service account, and writes the security configuration to kibana.yml.

Prerequisites

Install the appropriate Docker application for your operating system.

Pull the Elasticsearch Docker image.

docker pull docker.elastic.co/elasticsearch/elasticsearch:8.2.3

Pull the Kibana Docker image.

docker pull docker.elastic.co/kibana/kibana:8.2.3

Start a single-node Elasticsearch cluster and enroll Kibana

The following command starts a single-node Elasticsearch cluster for development or testing.

Create an elastic network for your containers.
```
docker network create elastic
```
Start Elasticsearch in Docker. A password is generated for the elastic user and output to the terminal, plus enrollment tokens for enrolling Kibana and adding additional nodes to your cluster.
```
docker run --name es01 --net elastic -p 9200:9200 -it docker.elastic.co/elasticsearch/elasticsearch:8.2.3
```
You might need to scroll back a bit in the terminal to view the password and enrollment token.
Copy the generated password and enrollment token and save them in a secure location. These values are shown only when you start Elasticsearch for the first time.
Copy the http_ca.crt security certificate from your Docker container to your local machine.
```
docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .
```

Open a new terminal and verify that you can connect to your Elasticsearch cluster by making an authenticated call, using the http_ca.crt file that you copied from your Docker container. Enter the password for the elastic user when prompted.

curl --cacert http_ca.crt -u elastic https://localhost:9200

If the connection is successful, the command returns a response like this:

{
  "name" : "Cp8oag6",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA",
  "version" : {
    "number" : "8.2.3",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "f27399d",
    "build_date" : "2021-11-04T12:35:26.989068569Z",
    "build_snapshot" : false,
    "lucene_version" : "9.0.0",
    "minimum_wire_compatibility_version" : "7.16.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

In a new terminal session, start Kibana and connect it to your Elasticsearch container:
```
docker run --name kibana --net elastic -p 5601:5601 docker.elastic.co/kibana/kibana:8.2.3
```
When you start Kibana, a unique link is output to your terminal.
To access Kibana, click the generated link in your terminal.
1. In your browser, paste the enrollment token that you copied when starting Elasticsearch and click the button to connect your Kibana instance with Elasticsearch.
2. Log in to Kibana as the elastic user with the password that was generated when you started Elasticsearch.

Enroll additional nodes

When you start Elasticsearch for the first time, the installation process configures a single-node cluster by default. This process also generates an enrollment token and prints it to your terminal. If you want a node to join an existing cluster, start the new node with the generated enrollment token.

Generating enrollment tokens

The enrollment token is valid for 30 minutes. If you need to generate a new enrollment token, run the elasticsearch-create-enrollment-token tool on your existing node. This tool is available in the Elasticsearch bin directory of the Docker container.

For example, run the following command on the existing es01 node to generate an enrollment token for new Elasticsearch nodes:

docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node

In the terminal where you started your first node, copy the generated enrollment token for adding new Elasticsearch nodes.
On your new node, start Elasticsearch and include the generated enrollment token.
```
docker run -e ENROLLMENT_TOKEN="<token>" --name es02 --net elastic -it docker.elastic.co/elasticsearch/elasticsearch:8.2.3
```
Elasticsearch is now configured to join the existing cluster.

If you experience issues where the container where your first node is running exits when your second node starts, explicitly set values for the JVM heap size. To manually configure the heap size, include the ES_JAVA_OPTS variable and set values for -Xms and -Xmx when starting each node. For example, the following command starts node es02 and sets the minimum and maximum JVM heap size to 1 GB:

docker run -e ES_JAVA_OPTS="-Xms1g -Xmx1g" -e ENROLLMENT_TOKEN="<token>" --name es02 -p 9201:9200 --net elastic -it docker.elastic.co/elasticsearch/elasticsearch:8.2.3

Generate passwords and enrollment tokens

If you need to reset the password for the elastic user or other built-in users, run the elasticsearch-reset-password tool. This tool is available in the Elasticsearch bin directory of the Docker container.

For example, to reset the password for the elastic user:

docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

If you need to generate new enrollment tokens for Kibana or Elasticsearch nodes, run the elasticsearch-create-enrollment-token tool. This tool is available in the Elasticsearch bin directory of the Docker container.

For example, to generate a new enrollment token for Kibana:

docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

Remove Docker containers

To remove the containers and their network, run:

docker network rm elastic
docker rm es01
docker rm kibana

Start the Elastic Stack with Docker Compose

To get a multi-node Elasticsearch cluster and Kibana up and running in Docker with security enabled, you can use Docker Compose.

This configuration provides a simple method of starting a secured cluster that you can use for development before building a distributed deployment with multiple hosts.

Prerequisites

Install the appropriate Docker application for your operating system.

If you’re running on Linux, install Docker Compose.

Make sure that Docker is allotted at least 4GB of memory. In Docker Desktop, you configure resource usage on the Advanced tab in Preferences (macOS) or Settings (Windows).

Prepare the environment

Create the following configuration files in a new, empty directory. These files are also available from the elastic/elasticsearch repository on GitHub.

`.env`

The .env file sets environment variables that are used when you run the docker-compose.yml configuration file. Ensure that you specify a strong password for the elastic and kibana_system users with the ELASTIC_PASSWORD and KIBANA_PASSWORD variables. These variable are referenced by the docker-compose.yml file.

# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=

# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=

# Version of Elastic products
STACK_VERSION=8.2.3

# Set the cluster name
CLUSTER_NAME=docker-cluster

# Set to 'basic' or 'trial' to automatically start the 30-day trial
LICENSE=basic
#LICENSE=trial

# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200
#ES_PORT=127.0.0.1:9200

# Port to expose Kibana to the host
KIBANA_PORT=5601
#KIBANA_PORT=80

# Increase or decrease based on the available host memory (in bytes)
MEM_LIMIT=1073741824

# Project namespace (defaults to the current folder name if not set)
#COMPOSE_PROJECT_NAME=myproject

`docker-compose.yml`

This docker-compose.yml file creates a three-node secure Elasticsearch cluster with authentication and network encryption enabled, and a Kibana instance securely connected to it.

version: "2.2"

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es02\n"\
          "    dns:\n"\
          "      - es02\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es03\n"\
          "    dns:\n"\
          "      - es03\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es02,es03
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es02:
    depends_on:
      - es01
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata02:/usr/share/elasticsearch/data
    environment:
      - node.name=es02
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es03
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es02/es02.key
      - xpack.security.http.ssl.certificate=certs/es02/es02.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es02/es02.key
      - xpack.security.transport.ssl.certificate=certs/es02/es02.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  es03:
    depends_on:
      - es02
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata03:/usr/share/elasticsearch/data
    environment:
      - node.name=es03
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01,es02,es03
      - discovery.seed_hosts=es01,es02
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es03/es03.key
      - xpack.security.http.ssl.certificate=certs/es03/es03.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es03/es03.key
      - xpack.security.transport.ssl.certificate=certs/es03/es03.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  kibana:
    depends_on:
      es01:
        condition: service_healthy
      es02:
        condition: service_healthy
      es03:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    mem_limit: ${MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  esdata02:
    driver: local
  esdata03:
    driver: local
  kibanadata:
    driver: local

Start your cluster with security enabled and configured

Modify the .env file and enter strong password values for both the ELASTIC_PASSWORD and KIBANA_PASSWORD variables.

You must use the ELASTIC_PASSWORD value for further interactions with the cluster. The KIBANA_PASSWORD value is only used internally when configuring Kibana.
Create and start the three-node Elasticsearch cluster and Kibana instance:
```
docker-compose up -d
```
When the deployment has started, open a browser and navigate to http://localhost:5601 to access Kibana, where you can load sample data and interact with your cluster. Log in as the elastic user with the ELASTIC_PASSWORD value to get access.

Stop and remove the deployment

When you’re done experimenting, you can remove the network, containers, and volumes:

docker-compose down -v

Load settings from a file

Specifying settings for Elasticsearch and Kibana directly in the Docker Compose file is a convenient way to get started, but loading settings from a file is preferable after you get past the experimental stage.

For example, to use a custom es01.yml as the configuration file for the es01 Elasticsearch node, you can create a bind mount in the volumes section for the es01 service.

    volumes:
      - ./es01.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - ...

Similarly, to load Kibana settings from a file, you can add the following mount in the volumes section for the kibana service.

    volumes:
      - ./kibana.yml:/usr/share/kibana/config/kibana.yml
      - ...

Product-specific instructions for Docker

See the product-specific documentation for information about running a specific Elastic product in Docker:

« Getting started with the Elastic Stack Running the Elastic Stack on Kubernetes »