IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« What’s new in 7.16 Set up Elasticsearch »

›

Quick start

edit

Quick start

edit

This guide helps beginners learn how to:

Install and run Elasticsearch in a test environment
Add data to Elasticsearch
Search and sort data
Extract fields from unstructured content during a search

Run Elasticsearch

edit

The simplest way to set up Elasticsearch is to create a managed deployment with Elasticsearch Service on Elastic Cloud. If you prefer to manage your own test environment, you can install and run Elasticsearch using Docker.

Get a free trial.
Log into Elastic Cloud.
Click Create deployment.
Give your deployment a name.
Click Create deployment and download the password for the elastic user.
Click Continue to open Kibana.
Click Explore on my own.

Install and run Elasticsearch

Install and start Docker Desktop.

Run:

docker network create elastic
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.16.3
docker run --name es01-test --net elastic -p 127.0.0.1:9200:9200 -p 127.0.0.1:9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.16.3

Install and run Kibana

To analyze, visualize, and manage Elasticsearch data using an intuitive UI, install Kibana.

In a new terminal session, run:

docker pull docker.elastic.co/kibana/kibana:7.16.3
docker run --name kib01-test --net elastic -p 127.0.0.1:5601:5601 -e "ELASTICSEARCH_HOSTS=http://es01-test:9200" docker.elastic.co/kibana/kibana:7.16.3

To access Kibana, go to http://localhost:5601

Send requests to Elasticsearch

edit

You send data and other requests to Elasticsearch using REST APIs. This lets you interact with Elasticsearch using any client that sends HTTP requests, such as curl. You can also use Kibana’s console to send requests to Elasticsearch.

Use Kibana

Open Kibana’s main menu and go to Dev Tools > Console.
Run the following example API request in the console:
```
GET /
```

Use curl

To communicate with Elasticsearch using curl or another client, you need your cluster’s endpoint.

Open Kibana’s main menu and click Manage this deployment.
From your deployment menu, go to the Elasticsearch page. Click Copy endpoint.
To submit an example API request, run the following curl command in a new terminal session. Replace <password> with the password for the elastic user. Replace <elasticsearch_endpoint> with your endpoint.
```
curl -u elastic:<password> <elasticsearch_endpoint>/
```

Use Kibana

Open Kibana’s main menu and go to Dev Tools > Console.
Run the following example API request in the console:
```
GET /
```

Use curl

To submit an example API request, run the following curl command in a new terminal session.

curl -X GET http://localhost:9200/

Add data

edit

You add data to Elasticsearch as JSON objects called documents. Elasticsearch stores these documents in searchable indices.

For time series data, such as logs and metrics, you typically add documents to a data stream made up of multiple auto-generated backing indices.

A data stream requires an index template that matches its name. Elasticsearch uses this template to configure the stream’s backing indices. Documents sent to a data stream must have a @timestamp field.

Add a single document

edit

Submit the following indexing request to add a single log entry to the logs-my_app-default data stream. Since logs-my_app-default doesn’t exist, the request automatically creates it using the built-in logs-*-* index template.

POST logs-my_app-default/_doc
{
  "@timestamp": "2099-05-06T16:21:15.000Z",
  "event": {
    "original": "192.0.2.42 - - [06/May/2099:16:21:15 +0000] \"GET /images/bg.jpg HTTP/1.0\" 200 24736"
  }
}

The response includes metadata that Elasticsearch generates for the document:

The backing _index that contains the document. Elasticsearch automatically generates the names of backing indices.
A unique _id for the document within the index.

{
  "_index": ".ds-logs-my_app-default-2099-05-06-000001",
  "_type": "_doc",
  "_id": "gl5MJXMBMk1dGnErnBW8",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 1,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}

Add multiple documents

edit

Use the _bulk endpoint to add multiple documents in one request. Bulk data must be newline-delimited JSON (NDJSON). Each line must end in a newline character (\n), including the last line.

PUT logs-my_app-default/_bulk
{ "create": { } }
{ "@timestamp": "2099-05-07T16:24:32.000Z", "event": { "original": "192.0.2.242 - - [07/May/2020:16:24:32 -0500] \"GET /images/hm_nbg.jpg HTTP/1.0\" 304 0" } }
{ "create": { } }
{ "@timestamp": "2099-05-08T16:25:42.000Z", "event": { "original": "192.0.2.255 - - [08/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638" } }

Search data

edit

Indexed documents are available for search in near real-time. The following search matches all log entries in logs-my_app-default and sorts them by @timestamp in descending order.

GET logs-my_app-default/_search
{
  "query": {
    "match_all": { }
  },
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

By default, the hits section of the response includes up to the first 10 documents that match the search. The _source of each hit contains the original JSON object submitted during indexing.

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": ".ds-logs-my_app-default-2099-05-06-000001",
        "_type": "_doc",
        "_id": "PdjWongB9KPnaVm2IyaL",
        "_score": null,
        "_source": {
          "@timestamp": "2099-05-08T16:25:42.000Z",
          "event": {
            "original": "192.0.2.255 - - [08/May/2099:16:25:42 +0000] \"GET /favicon.ico HTTP/1.0\" 200 3638"
          }
        },
        "sort": [
          4081940742000
        ]
      },
      ...
    ]
  }
}

Get specific fields

edit

Parsing the entire _source is unwieldy for large documents. To exclude it from the response, set the _source parameter to false. Instead, use the fields parameter to retrieve the fields you want.

GET logs-my_app-default/_search
{
  "query": {
    "match_all": { }
  },
  "fields": [
    "@timestamp"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

The response contains each hit’s fields values as a flat array.

{
  ...
  "hits": {
    ...
    "hits": [
      {
        "_index": ".ds-logs-my_app-default-2099-05-06-000001",
        "_type": "_doc",
        "_id": "PdjWongB9KPnaVm2IyaL",
        "_score": null,
        "fields": {
          "@timestamp": [
            "2099-05-08T16:25:42.000Z"
          ]
        },
        "sort": [
          4081940742000
        ]
      },
      ...
    ]
  }
}

Search a date range

edit

To search across a specific time or IP range, use a range query.

GET logs-my_app-default/_search
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2099-05-05",
        "lt": "2099-05-08"
      }
    }
  },
  "fields": [
    "@timestamp"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

You can use date math to define relative time ranges. The following query searches for data from the past day, which won’t match any log entries in logs-my_app-default.

GET logs-my_app-default/_search
{
  "query": {
    "range": {
      "@timestamp": {
        "gte": "now-1d/d",
        "lt": "now/d"
      }
    }
  },
  "fields": [
    "@timestamp"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

Extract fields from unstructured content

edit

You can extract runtime fields from unstructured content, such as log messages, during a search.

Use the following search to extract the source.ip runtime field from event.original. To include it in the response, add source.ip to the fields parameter.

GET logs-my_app-default/_search
{
  "runtime_mappings": {
    "source.ip": {
      "type": "ip",
      "script": """
        String sourceip=grok('%{IPORHOST:sourceip} .*').extract(doc[ "event.original" ].value)?.sourceip;
        if (sourceip != null) emit(sourceip);
      """
    }
  },
  "query": {
    "range": {
      "@timestamp": {
        "gte": "2099-05-05",
        "lt": "2099-05-08"
      }
    }
  },
  "fields": [
    "@timestamp",
    "source.ip"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

Combine queries

edit

You can use the bool query to combine multiple queries. The following search combines two range queries: one on @timestamp and one on the source.ip runtime field.

GET logs-my_app-default/_search
{
  "runtime_mappings": {
    "source.ip": {
      "type": "ip",
      "script": """
        String sourceip=grok('%{IPORHOST:sourceip} .*').extract(doc[ "event.original" ].value)?.sourceip;
        if (sourceip != null) emit(sourceip);
      """
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "2099-05-05",
              "lt": "2099-05-08"
            }
          }
        },
        {
          "range": {
            "source.ip": {
              "gte": "192.0.2.0",
              "lte": "192.0.2.240"
            }
          }
        }
      ]
    }
  },
  "fields": [
    "@timestamp",
    "source.ip"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

Aggregate data

edit

Use aggregations to summarize data as metrics, statistics, or other analytics.

The following search uses an aggregation to calculate the average_response_size using the http.response.body.bytes runtime field. The aggregation only runs on documents that match the query.

GET logs-my_app-default/_search
{
  "runtime_mappings": {
    "http.response.body.bytes": {
      "type": "long",
      "script": """
        String bytes=grok('%{COMMONAPACHELOG}').extract(doc[ "event.original" ].value)?.bytes;
        if (bytes != null) emit(Integer.parseInt(bytes));
      """
    }
  },
  "aggs": {
    "average_response_size":{
      "avg": {
        "field": "http.response.body.bytes"
      }
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "2099-05-05",
              "lt": "2099-05-08"
            }
          }
        }
      ]
    }
  },
  "fields": [
    "@timestamp",
    "http.response.body.bytes"
  ],
  "_source": false,
  "sort": [
    {
      "@timestamp": "desc"
    }
  ]
}

The response’s aggregations object contains aggregation results.

{
  ...
  "aggregations" : {
    "average_response_size" : {
      "value" : 12368.0
    }
  }
}

Explore more search options

edit

To keep exploring, index more data to your data stream and check out Common search options.

Clean up

edit

When you’re done, delete your test data stream and its backing indices.

DELETE _data_stream/logs-my_app-default

You can also delete your test deployment.

Click Delete deployment from the deployment overview page and follow the prompts.

To stop your Elasticsearch and Kibana Docker containers, run:

docker stop es01-test
docker stop kib01-test

To remove the containers and their network, run:

docker network rm elastic
docker rm es01-test
docker rm kib01-test

What’s next?

edit

Get the most out of your time series data by setting up data tiers and ILM. See Use Elasticsearch for time series data.
Use Fleet and Elastic Agent to collect logs and metrics directly from your data sources and send them to Elasticsearch. See the Ingest logs, metrics, and uptime data with Elastic Agent.
Use Kibana to explore, visualize, and manage your Elasticsearch data. See the Kibana quick start guide.

« What’s new in 7.16 Set up Elasticsearch »