IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Field extraction Scripting and security »
Elastic Docs ›Elasticsearch Guide [8.15] ›Scripting

Accessing document fields and special variables

edit
IMPORTANT: This documentation is no longer updated. Refer to Elastic's version policy and the latest documentation.

Accessing document fields and special variables

edit

Depending on where a script is used, it will have access to certain special variables and document fields.

Update scripts

edit

A script used in the update, update-by-query, or reindex API will have access to the ctx variable which exposes:

ctx._source

Access to the document _source field.

ctx.op

The operation that should be applied to the document: index or delete.

ctx._index etc

Access to document metadata fields, some of which may be read-only.

These scripts do not have access to the doc variable and have to use ctx to access the documents they operate on.

Search and aggregation scripts

edit

With the exception of script fields which are executed once per search hit, scripts used in search and aggregations will be executed once for every document which might match a query or an aggregation. Depending on how many documents you have, this could mean millions or billions of executions: these scripts need to be fast!

Field values can be accessed from a script using doc-values, the _source field, or stored fields, each of which is explained below.

Accessing the score of a document within a script

edit

Scripts used in the function_score query, in script-based sorting, or in aggregations have access to the _score variable which represents the current relevance score of a document.

Here’s an example of using a script in a function_score query to alter the relevance _score of each document:

resp = client.index(
    index="my-index-000001",
    id="1",
    refresh=True,
    document={
        "text": "quick brown fox",
        "popularity": 1
    },
)
print(resp)

resp1 = client.index(
    index="my-index-000001",
    id="2",
    refresh=True,
    document={
        "text": "quick fox",
        "popularity": 5
    },
)
print(resp1)

resp2 = client.search(
    index="my-index-000001",
    query={
        "function_score": {
            "query": {
                "match": {
                    "text": "quick brown fox"
                }
            },
            "script_score": {
                "script": {
                    "lang": "expression",
                    "source": "_score * doc['popularity']"
                }
            }
        }
    },
)
print(resp2)
response = client.index(
  index: 'my-index-000001',
  id: 1,
  refresh: true,
  body: {
    text: 'quick brown fox',
    popularity: 1
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  id: 2,
  refresh: true,
  body: {
    text: 'quick fox',
    popularity: 5
  }
)
puts response

response = client.search(
  index: 'my-index-000001',
  body: {
    query: {
      function_score: {
        query: {
          match: {
            text: 'quick brown fox'
          }
        },
        script_score: {
          script: {
            lang: 'expression',
            source: "_score * doc['popularity']"
          }
        }
      }
    }
  }
)
puts response
const response = await client.index({
  index: "my-index-000001",
  id: 1,
  refresh: "true",
  document: {
    text: "quick brown fox",
    popularity: 1,
  },
});
console.log(response);

const response1 = await client.index({
  index: "my-index-000001",
  id: 2,
  refresh: "true",
  document: {
    text: "quick fox",
    popularity: 5,
  },
});
console.log(response1);

const response2 = await client.search({
  index: "my-index-000001",
  query: {
    function_score: {
      query: {
        match: {
          text: "quick brown fox",
        },
      },
      script_score: {
        script: {
          lang: "expression",
          source: "_score * doc['popularity']",
        },
      },
    },
  },
});
console.log(response2);
PUT my-index-000001/_doc/1?refresh
{
  "text": "quick brown fox",
  "popularity": 1
}

PUT my-index-000001/_doc/2?refresh
{
  "text": "quick fox",
  "popularity": 5
}

GET my-index-000001/_search
{
  "query": {
    "function_score": {
      "query": {
        "match": {
          "text": "quick brown fox"
        }
      },
      "script_score": {
        "script": {
          "lang": "expression",
          "source": "_score * doc['popularity']"
        }
      }
    }
  }
}

Doc values

edit

By far the fastest most efficient way to access a field value from a script is to use the doc['field_name'] syntax, which retrieves the field value from doc values. Doc values are a columnar field value store, enabled by default on all fields except for analyzed text fields.

resp = client.index(
    index="my-index-000001",
    id="1",
    refresh=True,
    document={
        "cost_price": 100
    },
)
print(resp)

resp1 = client.search(
    index="my-index-000001",
    script_fields={
        "sales_price": {
            "script": {
                "lang": "expression",
                "source": "doc['cost_price'] * markup",
                "params": {
                    "markup": 0.2
                }
            }
        }
    },
)
print(resp1)
response = client.index(
  index: 'my-index-000001',
  id: 1,
  refresh: true,
  body: {
    cost_price: 100
  }
)
puts response

response = client.search(
  index: 'my-index-000001',
  body: {
    script_fields: {
      sales_price: {
        script: {
          lang: 'expression',
          source: "doc['cost_price'] * markup",
          params: {
            markup: 0.2
          }
        }
      }
    }
  }
)
puts response
const response = await client.index({
  index: "my-index-000001",
  id: 1,
  refresh: "true",
  document: {
    cost_price: 100,
  },
});
console.log(response);

const response1 = await client.search({
  index: "my-index-000001",
  script_fields: {
    sales_price: {
      script: {
        lang: "expression",
        source: "doc['cost_price'] * markup",
        params: {
          markup: 0.2,
        },
      },
    },
  },
});
console.log(response1);
PUT my-index-000001/_doc/1?refresh
{
  "cost_price": 100
}

GET my-index-000001/_search
{
  "script_fields": {
    "sales_price": {
      "script": {
        "lang":   "expression",
        "source": "doc['cost_price'] * markup",
        "params": {
          "markup": 0.2
        }
      }
    }
  }
}

Doc-values can only return "simple" field values like numbers, dates, geo- points, terms, etc, or arrays of these values if the field is multi-valued. It cannot return JSON objects.

Missing fields

The doc['field'] will throw an error if field is missing from the mappings. In painless, a check can first be done with doc.containsKey('field') to guard accessing the doc map. Unfortunately, there is no way to check for the existence of the field in mappings in an expression script.

Doc values and text fields

The doc['field'] syntax can also be used for analyzed text fields if fielddata is enabled, but BEWARE: enabling fielddata on a text field requires loading all of the terms into the JVM heap, which can be very expensive both in terms of memory and CPU. It seldom makes sense to access text fields from scripts.

The document _source

edit

The document _source can be accessed using the _source.field_name syntax. The _source is loaded as a map-of-maps, so properties within object fields can be accessed as, for example, _source.name.first.

Prefer doc-values to _source

Accessing the _source field is much slower than using doc-values. The _source field is optimised for returning several fields per result, while doc values are optimised for accessing the value of a specific field in many documents.

It makes sense to use _source when generating a script field for the top ten hits from a search result but, for other search and aggregation use cases, always prefer using doc values.

For instance:

resp = client.indices.create(
    index="my-index-000001",
    mappings={
        "properties": {
            "first_name": {
                "type": "text"
            },
            "last_name": {
                "type": "text"
            }
        }
    },
)
print(resp)

resp1 = client.index(
    index="my-index-000001",
    id="1",
    refresh=True,
    document={
        "first_name": "Barry",
        "last_name": "White"
    },
)
print(resp1)

resp2 = client.search(
    index="my-index-000001",
    script_fields={
        "full_name": {
            "script": {
                "lang": "painless",
                "source": "params._source.first_name + ' ' + params._source.last_name"
            }
        }
    },
)
print(resp2)
response = client.indices.create(
  index: 'my-index-000001',
  body: {
    mappings: {
      properties: {
        first_name: {
          type: 'text'
        },
        last_name: {
          type: 'text'
        }
      }
    }
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  id: 1,
  refresh: true,
  body: {
    first_name: 'Barry',
    last_name: 'White'
  }
)
puts response

response = client.search(
  index: 'my-index-000001',
  body: {
    script_fields: {
      full_name: {
        script: {
          lang: 'painless',
          source: "params._source.first_name + ' ' + params._source.last_name"
        }
      }
    }
  }
)
puts response
const response = await client.indices.create({
  index: "my-index-000001",
  mappings: {
    properties: {
      first_name: {
        type: "text",
      },
      last_name: {
        type: "text",
      },
    },
  },
});
console.log(response);

const response1 = await client.index({
  index: "my-index-000001",
  id: 1,
  refresh: "true",
  document: {
    first_name: "Barry",
    last_name: "White",
  },
});
console.log(response1);

const response2 = await client.search({
  index: "my-index-000001",
  script_fields: {
    full_name: {
      script: {
        lang: "painless",
        source: "params._source.first_name + ' ' + params._source.last_name",
      },
    },
  },
});
console.log(response2);
PUT my-index-000001
{
  "mappings": {
    "properties": {
      "first_name": {
        "type": "text"
      },
      "last_name": {
        "type": "text"
      }
    }
  }
}

PUT my-index-000001/_doc/1?refresh
{
  "first_name": "Barry",
  "last_name": "White"
}

GET my-index-000001/_search
{
  "script_fields": {
    "full_name": {
      "script": {
        "lang": "painless",
        "source": "params._source.first_name + ' ' + params._source.last_name"
      }
    }
  }
}

Stored fields

edit

Stored fields — fields explicitly marked as "store": true in the mapping — can be accessed using the _fields['field_name'].value or _fields['field_name'] syntax:

resp = client.indices.create(
    index="my-index-000001",
    mappings={
        "properties": {
            "full_name": {
                "type": "text",
                "store": True
            },
            "title": {
                "type": "text",
                "store": True
            }
        }
    },
)
print(resp)

resp1 = client.index(
    index="my-index-000001",
    id="1",
    refresh=True,
    document={
        "full_name": "Alice Ball",
        "title": "Professor"
    },
)
print(resp1)

resp2 = client.search(
    index="my-index-000001",
    script_fields={
        "name_with_title": {
            "script": {
                "lang": "painless",
                "source": "params._fields['title'].value + ' ' + params._fields['full_name'].value"
            }
        }
    },
)
print(resp2)
response = client.indices.create(
  index: 'my-index-000001',
  body: {
    mappings: {
      properties: {
        full_name: {
          type: 'text',
          store: true
        },
        title: {
          type: 'text',
          store: true
        }
      }
    }
  }
)
puts response

response = client.index(
  index: 'my-index-000001',
  id: 1,
  refresh: true,
  body: {
    full_name: 'Alice Ball',
    title: 'Professor'
  }
)
puts response

response = client.search(
  index: 'my-index-000001',
  body: {
    script_fields: {
      name_with_title: {
        script: {
          lang: 'painless',
          source: "params._fields['title'].value + ' ' + params._fields['full_name'].value"
        }
      }
    }
  }
)
puts response
const response = await client.indices.create({
  index: "my-index-000001",
  mappings: {
    properties: {
      full_name: {
        type: "text",
        store: true,
      },
      title: {
        type: "text",
        store: true,
      },
    },
  },
});
console.log(response);

const response1 = await client.index({
  index: "my-index-000001",
  id: 1,
  refresh: "true",
  document: {
    full_name: "Alice Ball",
    title: "Professor",
  },
});
console.log(response1);

const response2 = await client.search({
  index: "my-index-000001",
  script_fields: {
    name_with_title: {
      script: {
        lang: "painless",
        source:
          "params._fields['title'].value + ' ' + params._fields['full_name'].value",
      },
    },
  },
});
console.log(response2);
PUT my-index-000001
{
  "mappings": {
    "properties": {
      "full_name": {
        "type": "text",
        "store": true
      },
      "title": {
        "type": "text",
        "store": true
      }
    }
  }
}

PUT my-index-000001/_doc/1?refresh
{
  "full_name": "Alice Ball",
  "title": "Professor"
}

GET my-index-000001/_search
{
  "script_fields": {
    "name_with_title": {
      "script": {
        "lang": "painless",
        "source": "params._fields['title'].value + ' ' + params._fields['full_name'].value"
      }
    }
  }
}

Stored vs _source

The _source field is just a special stored field, so the performance is similar to that of other stored fields. The _source provides access to the original document body that was indexed (including the ability to distinguish null values from empty fields, single-value arrays from plain scalars, etc).

The only time it really makes sense to use stored fields instead of the _source field is when the _source is very large and it is less costly to access a few small stored fields instead of the entire _source.

« Field extraction Scripting and security »