Create transform API

edit

Instantiates a transform.

Request

edit

PUT _transform/<transform_id>

Prerequisites

edit

If the Elasticsearch security features are enabled, you must have the following built-in roles and privileges:

  • transform_admin
  • source index: read, view_index_metadata
  • destination index: read, create_index, manage and index

For more information, see Built-in roles, Security privileges, and Machine learning security privileges.

Description

edit

This API defines a transform, which copies data from source indices, transforms it, and persists it into an entity-centric destination index. If you choose to use the pivot method for your transform, the entities are defined by the set of group_by fields in the pivot object. If you choose to use the latest method, the entities are defined by the unique_key field values in the latest object.

You can also think of the destination index as a two-dimensional tabular data structure (known as a data frame). The ID for each document in the data frame is generated from a hash of the entity, so there is a unique row per entity. For more information, see Transforming data.

When the transform is created, a series of validations occur to ensure its success. For example, there is a check for the existence of the source indices and a check that the destination index is not part of the source index pattern. You can use the defer_validation parameter to skip these checks.

Deferred validations are always run when the transform is started, with the exception of privilege checks. When Elasticsearch security features are enabled, the transform remembers which roles the user that created it had at the time of creation and uses those same roles. If those roles do not have the required privileges on the source and destination indices, the transform fails when it attempts unauthorized operations.

You must use Kibana or this API to create a transform. Do not add a transform directly to any .transform-internal* indices using the Elasticsearch index API. If Elasticsearch security features are enabled, do not give users any privileges on .transform-internal* indices. If you used transforms prior to 7.5, also do not give users any privileges on .data-frame-internal* indices.

You must choose either the latest or pivot method for your transform; you cannot use both in a single transform.

Path parameters

edit
<transform_id>
(Required, string) Identifier for the transform. This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters.

Query parameters

edit
defer_validation
(Optional, Boolean) When true, deferrable validations are not run. This behavior may be desired if the source index does not exist until after the transform is created.

Request body

edit
description
(Optional, string) Free text description of the transform.
dest

(Required, object) The destination for the transform.

Properties of dest
index
(Required, string) The destination index for the transform. The mappings of the destination index are deduced based on the source fields when possible. If alternate mappings are required, use the Create index API prior to starting the transform.
pipeline
(Optional, string) The unique identifier for an ingest pipeline.
frequency
(Optional, time units) The interval between checks for changes in the source indices when the transform is running continuously. Also determines the retry interval in the event of transient failures while the transform is searching or indexing. The minimum value is 1s and the maximum is 1h. The default value is 1m.
latest

(Required*, object) [beta] This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features. The latest method transforms the data by finding the latest document for each unique key.

Properties of latest
sort
(Required, string) Specifies the date field that is used to identify the latest documents.
unique_key
(Required, array of strings) Specifies an array of one or more fields that are used to group the data.
pivot

(Required*, object) The pivot method transforms the data by aggregating and grouping it. These objects define the group by fields and the aggregation to reduce the data.

Properties of pivot
aggregations or aggs

(Required, object) Defines how to aggregate the grouped data. The following aggregations are currently supported:

group_by

(Required, object) Defines how to group the data. More than one grouping can be defined per pivot. The following groupings are currently supported:

max_page_search_size
(Optional, integer) [7.8.0] Deprecated in 7.8.0. Moved to settings.
settings

(Optional, object) Defines optional transform settings.

Properties of settings
dates_as_epoch_millis
(Optional, boolean) Defines if dates in the ouput should be written as ISO formatted string (default) or as millis since epoch. epoch_millis has been the default for transforms created before version 7.11. For compatible output set this to true. The default value is false.
docs_per_second
(Optional, float) Specifies a limit on the number of input documents per second. This setting throttles the transform by adding a wait time between search requests. The default value is null, which disables throttling.
max_page_search_size
(Optional, integer) Defines the initial page size to use for the composite aggregation for each checkpoint. If circuit breaker exceptions occur, the page size is dynamically adjusted to a lower value. The minimum value is 10 and the maximum is 10,000. The default value is 500.
source

(Required, object) The source of the data for the transform.

Properties of source
index
(Required, string or array) The source indices for the transform. It can be a single index, an index pattern (for example, "my-index-*"), an array of indices (for example, ["my-index-000001", "my-index-000002"]), or an array of index patterns (for example, ["my-index-*", "my-other-index-*"].
query
(Optional, object) A query clause that retrieves a subset of data from the source index. See Query DSL.
sync

(Optional, object) Defines the properties transforms require to run continuously.

Properties of sync
time

(Required, object) Specifies that the transform uses a time field to synchronize the source and destination indices.

Properties of time
delay
(Optional, time units) The time delay between the current time and the latest input data time. The default value is 60s.
field

(Required, string) The date field that is used to identify new documents in the source.

In general, it’s a good idea to use a field that contains the ingest timestamp. If you use a different field, you might need to set the delay such that it accounts for data transmission delays.

Examples

edit

The following transform uses the pivot method:

PUT _transform/ecommerce_transform1
{
  "source": {
    "index": "kibana_sample_data_ecommerce",
    "query": {
      "term": {
        "geoip.continent_name": {
          "value": "Asia"
        }
      }
    }
  },
  "pivot": {
    "group_by": {
      "customer_id": {
        "terms": {
          "field": "customer_id"
        }
      }
    },
    "aggregations": {
      "max_price": {
        "max": {
          "field": "taxful_total_price"
        }
      }
    }
  },
  "description": "Maximum priced ecommerce data by customer_id in Asia",
  "dest": {
    "index": "kibana_sample_data_ecommerce_transform1",
    "pipeline": "add_timestamp_pipeline"
  },
  "frequency": "5m",
  "sync": {
    "time": {
      "field": "order_date",
      "delay": "60s"
    }
  }
}

When the transform is created, you receive the following results:

{
  "acknowledged" : true
}

The following transform uses the latest method:

PUT _transform/ecommerce_transform2
{
  "source": {
    "index": "kibana_sample_data_ecommerce"
  },
  "latest": {
    "unique_key": ["customer_id"],
    "sort": "order_date"
  },
  "description": "Latest order for each customer",
  "dest": {
    "index": "kibana_sample_data_ecommerce_transform2"
  },
  "frequency": "5m",
  "sync": {
    "time": {
      "field": "order_date",
      "delay": "60s"
    }
  }
}