Time series data stream (TSDS)

edit

Time series data stream (TSDS)

edit

This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

A time series data stream (TSDS) models timestamped metrics data as one or more time series.

You can use a TSDS to store metrics data more efficiently. In our benchmarks, metrics data stored in a TSDS used 70% less disk space than a regular data stream.

When to use a TSDS

edit

Both a regular data stream and a TSDS can store timestamped metrics data. Only use a TSDS if you typically add metrics data to Elasticsearch in near real-time and @timestamp order.

A TSDS is only intended for metrics data. For other timestamped data, such as logs or traces, use a regular data stream.

Differences from a regular data stream

edit

A TSDS works like a regular data stream with some key differences:

  • The matching index template for a TSDS requires a data_stream object with the index.mode: time_series option. This option enables most TSDS-related functionality.
  • In addition to a @timestamp, each document in a TSDS must contain one or more dimension fields. The matching index template for a TSDS must contain mappings for at least one keyword dimension.

    TSDS documents also typically contain one or more metric fields.

  • Elasticsearch generates a hidden _tsid metadata field for each document in a TSDS.
  • A TSDS uses time-bound backing indices to store data from the same time period in the same backing index.
  • The matching index template for a TSDS must contain the index.routing_path index setting. A TSDS uses this setting to perform dimension-based routing.
  • A TSDS uses internal index sorting to order shard segments by _tsid and @timestamp.
  • TSDS documents only support auto-generated document _id values. For TSDS documents, the document _id is a hash of the document’s dimensions and @timestamp. A TSDS doesn’t support custom document _id values.

What is a time series?

edit

A time series is a sequence of observations for a specific entity. Together, these observations let you track changes to the entity over time. For example, a time series can track:

  • CPU and disk usage for a computer
  • The price of a stock
  • Temperature and humidity readings from a weather sensor.
time series chart
Figure 1. Time series of weather sensor readings plotted as a graph

In a TSDS, each Elasticsearch document represents an observation, or data point, in a specific time series. Although a TSDS can contain multiple time series, a document can only belong to one time series. A time series can’t span multiple data streams.

Dimensions

edit

Dimensions are field names and values that, in combination, identify a document’s time series. In most cases, a dimension describes some aspect of the entity you’re measuring. For example, documents related to the same weather sensor may always have the same sensor_id and location values.

A TSDS document is uniquely identified by its time series and timestamp, both of which are used to generate the document _id. So, two documents with the same dimensions and the same timestamp are considered to be duplicates. When you use the _bulk endpoint to add documents to a TSDS, a second document with the same timestamp and dimensions overwrites the first. When you use the PUT /<target>/_create/<_id> format to add an individual document and a document with the same _id already exists, an error is generated.

You mark a field as a dimension using the boolean time_series_dimension mapping parameter. The following field types support the time_series_dimension parameter:

Metrics

edit

Metrics are fields that contain numeric measurements, as well as aggregations and/or downsampling values based off of those measurements. While not required, documents in a TSDS typically contain one or more metric fields.

Metrics differ from dimensions in that while dimensions generally remain constant, metrics are expected to change over time, even if rarely or slowly.

To mark a field as a metric, you must specify a metric type using the time_series_metric mapping parameter. The following field types support the time_series_metric parameter:

Accepted metric types vary based on the field type:

Valid values for time_series_metric
counter

A number that only increases or resets to 0 (zero). For example, a count of errors or completed tasks.

Only numeric and aggregate_metric_double fields support the counter metric type.

gauge

A number that can increase or decrease. For example, a temperature or available disk space.

Only numeric and aggregate_metric_double fields support the gauge metric type.

null (Default)
Not a time series metric.

Time series mode

edit

The matching index template for a TSDS must contain a data_stream object with the index_mode: time_series option. This option ensures the TSDS creates backing indices with an index.mode setting of time_series. This setting enables most TSDS-related functionality in the backing indices.

If you convert an existing data stream to a TSDS, only backing indices created after the conversion have an index.mode of time_series. You can’t change the index.mode of an existing backing index.

_tsid metadata field

edit

When you add a document to a TSDS, Elasticsearch automatically generates a _tsid metadata field for the document. The _tsid is an object containing the document’s dimensions. Documents in the same TSDS with the same _tsid are part of the same time series.

The _tsid field is not queryable or updatable. You also can’t retrieve a document’s _tsid using a get document request. However, you can use the _tsid field in aggregations and retrieve the _tsid value in searches using the fields parameter.

Time-bound indices

edit

In a TSDS, each backing index, including the most recent backing index, has a range of accepted @timestamp values. This range is defined by the index.time_series.start_time and index.time_series.end_time index settings.

When you add a document to a TSDS, Elasticsearch adds the document to the appropriate backing index based on its @timestamp value. As a result, a TSDS can add documents to any TSDS backing index that can receive writes. This applies even if the index isn’t the most recent backing index.

time bound indices

Some ILM actions, such as forcemerge, shrink, and searchable_snapshot, make a backing index read-only. You cannot add documents to read-only indices. Keep this in mind when defining the index lifecycle policy for your TSDS.

If no backing index can accept a document’s @timestamp value, Elasticsearch rejects the document.

Elasticsearch automatically configures index.time_series.start_time and index.time_series.end_time settings as part of the index creation and rollover process.

Look-ahead time

edit

Use the index.look_ahead_time index setting to configure how far into the future you can add documents to an index. When you create a new write index for a TSDS, Elasticsearch calculates the index’s index.time_series.end_time value as:

now + index.look_ahead_time

At the time series poll interval (controlled via time_series.poll_interval setting), Elasticsearch checks if the write index has met the rollover criteria in its index lifecycle policy. If not, Elasticsearch refreshes the now value and updates the write index’s index.time_series.end_time to:

now + index.look_ahead_time + time_series.poll_interval

This process continues until the write index rolls over. When the index rolls over, Elasticsearch sets a final index.time_series.end_time value for the index. This value borders the index.time_series.start_time for the new write index. This ensures the @timestamp ranges for neighboring backing indices always border but never overlap.

Accepted time range for adding data

edit

A TSDS is designed to ingest current metrics data. When the TSDS is first created the initial backing index has:

  • an index.time_series.start_time value set to now - index.look_ahead_time
  • an index.time_series.end_time value set to now + index.look_ahead_time

Only data that falls inside that range can be indexed.

In our TSDS example, index.look_ahead_time is set to three hours, so only documents with a @timestamp value that is within three hours previous or subsequent to the present time are accepted for indexing.

You can use the get data stream API to check the accepted time range for writing to any TSDS.

Dimension-based routing

edit

Within each TSDS backing index, Elasticsearch uses the index.routing_path index setting to route documents with the same dimensions to the same shards.

When you create the matching index template for a TSDS, you must specify one or more dimensions in the index.routing_path setting. Each document in a TSDS must contain one or more dimensions that match the index.routing_path setting.

Dimensions in the index.routing_path setting must be plain keyword fields. The index.routing_path setting accepts wildcard patterns (for example dim.*) and can dynamically match new fields. However, Elasticsearch will reject any mapping updates that add scripted, runtime, or non-dimension, non-keyword fields that match the index.routing_path value.

TSDS documents don’t support a custom _routing value. Similarly, you can’t require a _routing value in mappings for a TSDS.

Index sorting

edit

Elasticsearch uses compression algorithms to compress repeated values. This compression works best when repeated values are stored near each other — in the same index, on the same shard, and side-by-side in the same shard segment.

Most time series data contains repeated values. Dimensions are repeated across documents in the same time series. The metric values of a time series may also change slowly over time.

Internally, each TSDS backing index uses index sorting to order its shard segments by _tsid and @timestamp. This makes it more likely that these repeated values are stored near each other for better compression. A TSDS doesn’t support any index.sort.* index settings.

What’s next?

edit

Now that you know the basics, you’re ready to create a TSDS or convert an existing data stream to a TSDS.