Running the Elastic Stack ("ELK") on Docker
editRunning the Elastic Stack ("ELK") on Docker
editThe Elastic Docker registry contains Docker images for all the products in the Elastic Stack: https://www.docker.elastic.co/.
You can start the Elastic Stack in Docker with security enabled and configured by default. This option is great for quickly getting started with Elasticsearch and Kibana for testing and development.
You can also start the Elastic Stack with Docker Compose to create a secured, multi-node cluster with a connected Kibana instance. This option results in a more resilient cluster with greater capacity and reliability.
Start Elasticsearch and Kibana in Docker
editStarting in Elasticsearch 8.0, security is enabled and configured by default.
If you’re starting a single-node Elasticsearch cluster in a Docker container, security will be automatically enabled and configured for you. When you start Elasticsearch for the first time, the following security configuration occurs automatically:
- Certificates and keys are generated for the transport and HTTP layers.
-
The Transport Layer Security (TLS) configuration settings are written to
elasticsearch.yml
. -
A password is generated for the
elastic
user. - An enrollment token is generated for Kibana.
- An enrollment token is generated for other Elasticsearch nodes.
You can then start Kibana and enter the enrollment
token, which is valid for 30 minutes. This token automatically applies the
security settings from your Elasticsearch cluster, authenticates to Elasticsearch with the
kibana
service account, and writes the security configuration to kibana.yml
.
Prerequisites
edit- Install the appropriate Docker application for your operating system.
-
Pull the Elasticsearch Docker image.
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.0.1
-
Pull the Kibana Docker image.
docker pull docker.elastic.co/kibana/kibana:8.0.1
Start a single-node Elasticsearch cluster and enroll Kibana
editThe following command starts a single-node Elasticsearch cluster for development or testing.
-
Create an
elastic
network for your containers.docker network create elastic
-
Start Elasticsearch in Docker. A password is generated for the
elastic
user and output to the terminal, plus enrollment tokens for enrolling Kibana and adding additional nodes to your cluster.docker run --name es01 --net elastic -p 9200:9200 -it docker.elastic.co/elasticsearch/elasticsearch:8.0.1
You might need to scroll back a bit in the terminal to view the password and enrollment token.
- Copy the generated password and enrollment token and save them in a secure location. These values are shown only when you start Elasticsearch for the first time.
-
Copy the
http_ca.crt
security certificate from your Docker container to your local machine.docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt .
-
Open a new terminal and verify that you can connect to your Elasticsearch cluster by making an authenticated call, using the
http_ca.crt
file that you copied from your Docker container. Enter the password for theelastic
user when prompted.curl --cacert http_ca.crt -u elastic https://localhost:9200
If the connection is successful, the command returns a response like this:
{ "name" : "Cp8oag6", "cluster_name" : "docker-cluster", "cluster_uuid" : "AT69_T_DTp-1qgIJlatQqA", "version" : { "number" : "8.0.1", "build_flavor" : "default", "build_type" : "docker", "build_hash" : "f27399d", "build_date" : "2021-11-04T12:35:26.989068569Z", "build_snapshot" : false, "lucene_version" : "9.0.0", "minimum_wire_compatibility_version" : "7.16.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" }
-
In a new terminal session, start Kibana and connect it to your Elasticsearch container:
docker run --name kibana --net elastic -p 5601:5601 docker.elastic.co/kibana/kibana:8.0.1
When you start Kibana, a unique link is output to your terminal.
-
To access Kibana, click the generated link in your terminal.
- In your browser, paste the enrollment token that you copied when starting Elasticsearch and click the button to connect your Kibana instance with Elasticsearch.
-
Log in to Kibana as the
elastic
user with the password that was generated when you started Elasticsearch.
Enroll additional nodes
editWhen you start Elasticsearch for the first time, the installation process configures a single-node cluster by default. This process also generates an enrollment token and prints it to your terminal. If you want a node to join an existing cluster, start the new node with the generated enrollment token.
- In the terminal where you started your first node, copy the generated enrollment token for adding new Elasticsearch nodes.
-
On your new node, start Elasticsearch and include the generated enrollment token.
docker run -e ENROLLMENT_TOKEN="<token>" --name es02 --net elastic -it docker.elastic.co/elasticsearch/elasticsearch:8.0.1
Elasticsearch is now configured to join the existing cluster.
If you experience issues where the container where your first node is running
exits when your second node starts, explicitly set values for the JVM heap size.
To manually configure the heap size,
include the ES_JAVA_OPTS
variable and set values for -Xms
and -Xmx
when
starting each node. For example, the following command starts node es02
and
sets the minimum and maximum JVM heap size to 1 GB:
docker run -e ES_JAVA_OPTS="-Xms1g -Xmx1g" -e ENROLLMENT_TOKEN="<token>" --name es02 -p 9201:9200 --net elastic -it docker.elastic.co/elasticsearch/elasticsearch:8.0.1
Generate passwords and enrollment tokens
editIf you need to reset the password for the elastic
user or other
built-in users, run the elasticsearch-reset-password
tool. This tool is available in the Elasticsearch bin
directory of the Docker container.
For example, to reset the password for the elastic
user:
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
If you need to generate new enrollment tokens for Kibana or Elasticsearch nodes, run the
elasticsearch-create-enrollment-token
tool.
This tool is available in the Elasticsearch bin
directory of the Docker container.
For example, to generate a new enrollment token for Kibana:
docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
Remove Docker containers
editTo remove the containers and their network, run:
docker network rm elastic docker rm es01 docker rm kibana
Start the Elastic Stack with Docker Compose
editTo get a multi-node Elasticsearch cluster and Kibana up and running in Docker with security enabled, you can use Docker Compose.
This configuration provides a simple method of starting a secured cluster that you can use for development before building a distributed deployment with multiple hosts.
Prerequisites
editInstall the appropriate Docker application for your operating system.
If you’re running on Linux, install Docker Compose.
Make sure that Docker is allotted at least 4GB of memory. In Docker Desktop, you configure resource usage on the Advanced tab in Preferences (macOS) or Settings (Windows).
Prepare the environment
editCreate the following configuration files in a new, empty directory. These files are also available from the elastic/elasticsearch repository on GitHub.
.env
editThe .env
file sets environment variables that are used when you run the
docker-compose.yml
configuration file. Ensure that you specify a strong
password for the elastic
and kibana_system
users with the
ELASTIC_PASSWORD
and KIBANA_PASSWORD
variables. These variable are
referenced by the docker-compose.yml
file.
# Password for the 'elastic' user (at least 6 characters) ELASTIC_PASSWORD= # Password for the 'kibana_system' user (at least 6 characters) KIBANA_PASSWORD= # Version of Elastic products STACK_VERSION=8.0.1 # Set the cluster name CLUSTER_NAME=docker-cluster # Set to 'basic' or 'trial' to automatically start the 30-day trial LICENSE=basic #LICENSE=trial # Port to expose Elasticsearch HTTP API to the host ES_PORT=9200 #ES_PORT=127.0.0.1:9200 # Port to expose Kibana to the host KIBANA_PORT=5601 #KIBANA_PORT=80 # Increase or decrease based on the available host memory (in bytes) MEM_LIMIT=1073741824 # Project namespace (defaults to the current folder name if not set) #COMPOSE_PROJECT_NAME=myproject
docker-compose.yml
editThis docker-compose.yml
file creates a three-node secure Elasticsearch cluster with authentication and network encryption enabled, and a Kibana instance securely connected to it.
version: "2.2" services: setup: image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION} volumes: - certs:/usr/share/elasticsearch/config/certs user: "0" command: > bash -c ' if [ x${ELASTIC_PASSWORD} == x ]; then echo "Set the ELASTIC_PASSWORD environment variable in the .env file"; exit 1; elif [ x${KIBANA_PASSWORD} == x ]; then echo "Set the KIBANA_PASSWORD environment variable in the .env file"; exit 1; fi; if [ ! -f config/certs/ca.zip ]; then echo "Creating CA"; bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip; unzip config/certs/ca.zip -d config/certs; fi; if [ ! -f config/certs/certs.zip ]; then echo "Creating certs"; echo -ne \ "instances:\n"\ " - name: es01\n"\ " dns:\n"\ " - es01\n"\ " - localhost\n"\ " ip:\n"\ " - 127.0.0.1\n"\ " - name: es02\n"\ " dns:\n"\ " - es02\n"\ " - localhost\n"\ " ip:\n"\ " - 127.0.0.1\n"\ " - name: es03\n"\ " dns:\n"\ " - es03\n"\ " - localhost\n"\ " ip:\n"\ " - 127.0.0.1\n"\ > config/certs/instances.yml; bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key; unzip config/certs/certs.zip -d config/certs; fi; echo "Setting file permissions" chown -R root:root config/certs; find . -type d -exec chmod 750 \{\} \;; find . -type f -exec chmod 640 \{\} \;; echo "Waiting for Elasticsearch availability"; until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done; echo "Setting kibana_system password"; until curl -s -X POST --cacert config/certs/ca/ca.crt -u elastic:${ELASTIC_PASSWORD} -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done; echo "All done!"; ' healthcheck: test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"] interval: 1s timeout: 5s retries: 120 es01: depends_on: setup: condition: service_healthy image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION} volumes: - certs:/usr/share/elasticsearch/config/certs - esdata01:/usr/share/elasticsearch/data ports: - ${ES_PORT}:9200 environment: - node.name=es01 - cluster.name=${CLUSTER_NAME} - cluster.initial_master_nodes=es01,es02,es03 - discovery.seed_hosts=es02,es03 - ELASTIC_PASSWORD=${ELASTIC_PASSWORD} - bootstrap.memory_lock=true - xpack.security.enabled=true - xpack.security.http.ssl.enabled=true - xpack.security.http.ssl.key=certs/es01/es01.key - xpack.security.http.ssl.certificate=certs/es01/es01.crt - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.http.ssl.verification_mode=certificate - xpack.security.transport.ssl.enabled=true - xpack.security.transport.ssl.key=certs/es01/es01.key - xpack.security.transport.ssl.certificate=certs/es01/es01.crt - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.verification_mode=certificate - xpack.license.self_generated.type=${LICENSE} mem_limit: ${MEM_LIMIT} ulimits: memlock: soft: -1 hard: -1 healthcheck: test: [ "CMD-SHELL", "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'", ] interval: 10s timeout: 10s retries: 120 es02: depends_on: - es01 image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION} volumes: - certs:/usr/share/elasticsearch/config/certs - esdata02:/usr/share/elasticsearch/data environment: - node.name=es02 - cluster.name=${CLUSTER_NAME} - cluster.initial_master_nodes=es01,es02,es03 - discovery.seed_hosts=es01,es03 - bootstrap.memory_lock=true - xpack.security.enabled=true - xpack.security.http.ssl.enabled=true - xpack.security.http.ssl.key=certs/es02/es02.key - xpack.security.http.ssl.certificate=certs/es02/es02.crt - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.http.ssl.verification_mode=certificate - xpack.security.transport.ssl.enabled=true - xpack.security.transport.ssl.key=certs/es02/es02.key - xpack.security.transport.ssl.certificate=certs/es02/es02.crt - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.verification_mode=certificate - xpack.license.self_generated.type=${LICENSE} mem_limit: ${MEM_LIMIT} ulimits: memlock: soft: -1 hard: -1 healthcheck: test: [ "CMD-SHELL", "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'", ] interval: 10s timeout: 10s retries: 120 es03: depends_on: - es02 image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION} volumes: - certs:/usr/share/elasticsearch/config/certs - esdata03:/usr/share/elasticsearch/data environment: - node.name=es03 - cluster.name=${CLUSTER_NAME} - cluster.initial_master_nodes=es01,es02,es03 - discovery.seed_hosts=es01,es02 - bootstrap.memory_lock=true - xpack.security.enabled=true - xpack.security.http.ssl.enabled=true - xpack.security.http.ssl.key=certs/es03/es03.key - xpack.security.http.ssl.certificate=certs/es03/es03.crt - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.http.ssl.verification_mode=certificate - xpack.security.transport.ssl.enabled=true - xpack.security.transport.ssl.key=certs/es03/es03.key - xpack.security.transport.ssl.certificate=certs/es03/es03.crt - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.verification_mode=certificate - xpack.license.self_generated.type=${LICENSE} mem_limit: ${MEM_LIMIT} ulimits: memlock: soft: -1 hard: -1 healthcheck: test: [ "CMD-SHELL", "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'", ] interval: 10s timeout: 10s retries: 120 kibana: depends_on: es01: condition: service_healthy es02: condition: service_healthy es03: condition: service_healthy image: docker.elastic.co/kibana/kibana:${STACK_VERSION} volumes: - certs:/usr/share/kibana/config/certs - kibanadata:/usr/share/kibana/data ports: - ${KIBANA_PORT}:5601 environment: - SERVERNAME=kibana - ELASTICSEARCH_HOSTS=https://es01:9200 - ELASTICSEARCH_USERNAME=kibana_system - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD} - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt mem_limit: ${MEM_LIMIT} healthcheck: test: [ "CMD-SHELL", "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'", ] interval: 10s timeout: 10s retries: 120 volumes: certs: driver: local esdata01: driver: local esdata02: driver: local esdata03: driver: local kibanadata: driver: local
Start your cluster with security enabled and configured
edit-
Modify the
.env
file and enter strong password values for both theELASTIC_PASSWORD
andKIBANA_PASSWORD
variables.You must use the
ELASTIC_PASSWORD
value for further interactions with the cluster. TheKIBANA_PASSWORD
value is only used internally when configuring Kibana. -
Create and start the three-node Elasticsearch cluster and Kibana instance:
docker-compose up -d
-
When the deployment has started, open a browser and navigate to http://localhost:5601 to
access Kibana, where you can load sample data and interact with your cluster. Log in as the
elastic
user with theELASTIC_PASSWORD
value to get access.
Stop and remove the deployment
editWhen you’re done experimenting, you can remove the network, containers, and volumes:
docker-compose down -v
Load settings from a file
editSpecifying settings for Elasticsearch and Kibana directly in the Docker Compose file is a convenient way to get started, but loading settings from a file is preferable after you get past the experimental stage.
For example, to use a custom es01.yml
as the configuration file for the es01
Elasticsearch node, you can create a bind mount in the volumes
section for the es01
service.
volumes: - ./es01.yml:/usr/share/elasticsearch/config/elasticsearch.yml - ...
Similarly, to load Kibana settings from a file, you can add the following mount
in the volumes
section for the kibana
service.
volumes: - ./kibana.yml:/usr/share/kibana/config/kibana.yml - ...
Product-specific instructions for Docker
editSee the product-specific documentation for information about running a specific Elastic product in Docker: