IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Get pipeline API Simulate ingest API »

› › ›

Simulate pipeline API

edit

Simulate pipeline API

edit

Executes an ingest pipeline against a set of provided documents.

POST /_ingest/pipeline/my-pipeline-id/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "rab"
      }
    }
  ]
}

Request

edit

POST /_ingest/pipeline/<pipeline>/_simulate

GET /_ingest/pipeline/<pipeline>/_simulate

POST /_ingest/pipeline/_simulate

GET /_ingest/pipeline/_simulate

Prerequisites

edit

If the Elasticsearch security features are enabled, you must have the read_pipeline, manage_pipeline, manage_ingest_pipelines, or manage cluster privilege to use this API.

Description

edit

The simulate pipeline API executes a specific pipeline against a set of documents provided in the body of the request.

You can either specify an existing pipeline to execute against the provided documents or supply a pipeline definition in the body of the request.

Path parameters

edit

<pipeline>: (Required*, string) Pipeline to test. If you don’t specify a pipeline in the request body, this parameter is required.

Query parameters

edit

verbose: (Optional, Boolean) If true, the response includes output data for each processor in the executed pipeline.

Request body

edit

pipeline

(Required*, object) Pipeline to test. If you don’t specify the <pipeline> request path parameter, this parameter is required. If you specify both this and the request path parameter, the API only uses the request path parameter.

Properties of pipeline

description

(Optional, string) Description of the ingest pipeline.

on_failure

(Optional, array of processor objects) Processors to run immediately after a processor failure.

Each processor supports a processor-level on_failure value. If a processor without an on_failure value fails, Elasticsearch uses this pipeline-level parameter as a fallback. The processors in this parameter run sequentially in the order specified. Elasticsearch will not attempt to run the pipeline’s remaining processors.

processors

(Required, array of processor objects) Processors used to perform transformations on documents before indexing. Processors run sequentially in the order specified.

version

(Optional, integer) Version number used by external systems to track ingest pipelines.

See the if_version parameter above for how the version attribute is used.

_meta

(Optional, object) Optional metadata about the ingest pipeline. May have any contents. This map is not automatically generated by Elasticsearch.

deprecated

(Optional, boolean) Marks this ingest pipeline as deprecated. When a deprecated ingest pipeline is referenced as the default or final pipeline when creating or updating a non-deprecated index template, Elasticsearch will emit a deprecation warning.

docs

(Required, array of objects) Sample documents to test in the pipeline.

Properties of docs objects

_id: (Optional, string) Unique identifier for the document. This ID must be unique within the _index.
_index: (Optional, string) Name of the index containing the document.
_routing: (Optional, string) Value used to send the document to a specific primary shard. See the _routing field.
_source: (Required, object) JSON body for the document.

Examples

edit

Specify a pipeline as a path parameter

edit

POST /_ingest/pipeline/my-pipeline-id/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "rab"
      }
    }
  ]
}

The API returns the following response:

{
   "docs": [
      {
         "doc": {
            "_id": "id",
            "_index": "index",
            "_version": "-3",
            "_source": {
               "field2": "_value",
               "foo": "bar"
            },
            "_ingest": {
               "timestamp": "2017-05-04T22:30:03.187Z"
            }
         }
      },
      {
         "doc": {
            "_id": "id",
            "_index": "index",
            "_version": "-3",
            "_source": {
               "field2": "_value",
               "foo": "rab"
            },
            "_ingest": {
               "timestamp": "2017-05-04T22:30:03.188Z"
            }
         }
      }
   ]
}

Specify a pipeline in the request body

edit

response = client.ingest.simulate(
  body: {
    pipeline: {
      description: '_description',
      processors: [
        {
          set: {
            field: 'field2',
            value: '_value'
          }
        }
      ]
    },
    docs: [
      {
        _index: 'index',
        _id: 'id',
        _source: {
          foo: 'bar'
        }
      },
      {
        _index: 'index',
        _id: 'id',
        _source: {
          foo: 'rab'
        }
      }
    ]
  }
)
puts response

POST /_ingest/pipeline/_simulate
{
  "pipeline" :
  {
    "description": "_description",
    "processors": [
      {
        "set" : {
          "field" : "field2",
          "value" : "_value"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "rab"
      }
    }
  ]
}

The API returns the following response:

{
   "docs": [
      {
         "doc": {
            "_id": "id",
            "_index": "index",
            "_version": "-3",
            "_source": {
               "field2": "_value",
               "foo": "bar"
            },
            "_ingest": {
               "timestamp": "2017-05-04T22:30:03.187Z"
            }
         }
      },
      {
         "doc": {
            "_id": "id",
            "_index": "index",
            "_version": "-3",
            "_source": {
               "field2": "_value",
               "foo": "rab"
            },
            "_ingest": {
               "timestamp": "2017-05-04T22:30:03.188Z"
            }
         }
      }
   ]
}

View verbose results

edit

You can use the simulate pipeline API to see how each processor affects the ingest document as it passes through the pipeline. To see the intermediate results of each processor in the simulate request, you can add the verbose parameter to the request.

response = client.ingest.simulate(
  verbose: true,
  body: {
    pipeline: {
      description: '_description',
      processors: [
        {
          set: {
            field: 'field2',
            value: '_value2'
          }
        },
        {
          set: {
            field: 'field3',
            value: '_value3'
          }
        }
      ]
    },
    docs: [
      {
        _index: 'index',
        _id: 'id',
        _source: {
          foo: 'bar'
        }
      },
      {
        _index: 'index',
        _id: 'id',
        _source: {
          foo: 'rab'
        }
      }
    ]
  }
)
puts response

POST /_ingest/pipeline/_simulate?verbose=true
{
  "pipeline" :
  {
    "description": "_description",
    "processors": [
      {
        "set" : {
          "field" : "field2",
          "value" : "_value2"
        }
      },
      {
        "set" : {
          "field" : "field3",
          "value" : "_value3"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "rab"
      }
    }
  ]
}

The API returns the following response:

{
  "docs" : [
    {
      "processor_results" : [
        {
          "processor_type" : "set",
          "status" : "success",
          "doc" : {
            "_index" : "index",
            "_id" : "id",
            "_version": "-3",
            "_source" : {
              "field2" : "_value2",
              "foo" : "bar"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-07-30T01:21:24.251836Z"
            }
          }
        },
        {
          "processor_type" : "set",
          "status" : "success",
          "doc" : {
            "_index" : "index",
            "_id" : "id",
            "_version": "-3",
            "_source" : {
              "field3" : "_value3",
              "field2" : "_value2",
              "foo" : "bar"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-07-30T01:21:24.251836Z"
            }
          }
        }
      ]
    },
    {
      "processor_results" : [
        {
          "processor_type" : "set",
          "status" : "success",
          "doc" : {
            "_index" : "index",
            "_id" : "id",
            "_version": "-3",
            "_source" : {
              "field2" : "_value2",
              "foo" : "rab"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-07-30T01:21:24.251863Z"
            }
          }
        },
        {
          "processor_type" : "set",
          "status" : "success",
          "doc" : {
            "_index" : "index",
            "_id" : "id",
            "_version": "-3",
            "_source" : {
              "field3" : "_value3",
              "field2" : "_value2",
              "foo" : "rab"
            },
            "_ingest" : {
              "pipeline" : "_simulate_pipeline",
              "timestamp" : "2020-07-30T01:21:24.251863Z"
            }
          }
        }
      ]
    }
  ]
}

« Get pipeline API Simulate ingest API »