_source field
edit_source
field
editThe _source
field contains the original JSON document body that was passed
at index time. The _source
field itself is not indexed (and thus is not
searchable), but it is stored so that it can be returned when executing
fetch requests, like get or search.
If disk usage is important to you, then consider the following options:
-
Using synthetic
_source
, which reconstructs source content at the time of retrieval instead of storing it on disk. This shrinks disk usage, at the cost of slower access to_source
in Get and Search queries. -
Disabling the
_source
field completely. This shrinks disk usage but disables features that rely on_source
.
Synthetic _source
editSynthetic _source
is Generally Available only for TSDB indices
(indices that have index.mode
set to time_series
). For other indices
synthetic _source
is in technical preview. Features in technical preview may
be changed or removed in a future release. Elastic will work to fix
any issues, but features in technical preview are not subject to the support SLA
of official GA features.
Though very handy to have around, the source field takes up a significant amount
of space on disk. Instead of storing source documents on disk exactly as you
send them, Elasticsearch can reconstruct source content on the fly upon retrieval.
Enable this by setting mode: synthetic
in _source
:
resp = client.indices.create( index="idx", mappings={ "_source": { "mode": "synthetic" } }, ) print(resp)
response = client.indices.create( index: 'idx', body: { mappings: { _source: { mode: 'synthetic' } } } ) puts response
PUT idx { "mappings": { "_source": { "mode": "synthetic" } } }
While this on the fly reconstruction is generally slower than saving the source
documents verbatim and loading them at query time, it saves a lot of storage
space. Additional latency can be avoided by not loading _source
field in queries when it is not needed.
Supported fields
editSynthetic _source
is supported by all field types. Depending on implementation details, field types have different properties when used with synthetic _source
.
Most field types construct synthetic _source
using existing data, most commonly doc_values
and stored fields. For these field types, no additional space is needed to store the contents of _source
field. Due to the storage layout of doc_values
, the generated _source
field undergoes modifications compared to original document.
For all other field types, the original value of the field is stored as is, in the same way as the _source
field in non-synthetic mode. In this case there are no modifications and field data in _source
is the same as in the original document. Similarly, malformed values of fields that use ignore_malformed
or ignore_above
need to be stored as is. This approach is less storage efficient since data needed for _source
reconstruction is stored in addition to other data required to index the field (like doc_values
).
Synthetic _source
restrictions
editSynthetic _source
cannot be used together with field mappings that use copy_to
.
Some field types have additional restrictions. These restrictions are documented in the synthetic _source
section of the field type’s documentation.
Synthetic _source
modifications
editWhen synthetic _source
is enabled, retrieved documents undergo some
modifications compared to the original JSON.
Arrays moved to leaf fields
editSynthetic _source
arrays are moved to leaves. For example:
resp = client.index( index="idx", id="1", document={ "foo": [ { "bar": 1 }, { "bar": 2 } ] }, ) print(resp)
response = client.index( index: 'idx', id: 1, body: { foo: [ { bar: 1 }, { bar: 2 } ] } ) puts response
const response = await client.index({ index: "idx", id: 1, document: { foo: [ { bar: 1, }, { bar: 2, }, ], }, }); console.log(response);
PUT idx/_doc/1 { "foo": [ { "bar": 1 }, { "bar": 2 } ] }
Will become:
{ "foo": { "bar": [1, 2] } }
This can cause some arrays to vanish:
resp = client.index( index="idx", id="1", document={ "foo": [ { "bar": 1 }, { "baz": 2 } ] }, ) print(resp)
response = client.index( index: 'idx', id: 1, body: { foo: [ { bar: 1 }, { baz: 2 } ] } ) puts response
const response = await client.index({ index: "idx", id: 1, document: { foo: [ { bar: 1, }, { baz: 2, }, ], }, }); console.log(response);
PUT idx/_doc/1 { "foo": [ { "bar": 1 }, { "baz": 2 } ] }
Will become:
{ "foo": { "bar": 1, "baz": 2 } }
Fields named as they are mapped
editSynthetic source names fields as they are named in the mapping. When used
with dynamic mapping, fields with dots (.
) in their names are, by
default, interpreted as multiple objects, while dots in field names are
preserved within objects that have subobjects
disabled. For example:
resp = client.index( index="idx", id="1", document={ "foo.bar.baz": 1 }, ) print(resp)
const response = await client.index({ index: "idx", id: 1, document: { "foo.bar.baz": 1, }, }); console.log(response);
PUT idx/_doc/1 { "foo.bar.baz": 1 }
Will become:
{ "foo": { "bar": { "baz": 1 } } }
Alphabetical sorting
editSynthetic _source
fields are sorted alphabetically. The
JSON RFC defines objects as
"an unordered collection of zero or more name/value pairs" so applications
shouldn’t care but without synthetic _source
the original ordering is
preserved and some applications may, counter to the spec, do something with
that ordering.
Field types that support synthetic source with no storage overhead
editThe following field types support synthetic source using data from doc_values
or stored fields, and require no additional storage space to construct the _source
field.
If you enable the ignore_malformed
or ignore_above
settings, then additional storage is required to store ignored field values for these types.
Disabling the _source
field
editThough very handy to have around, the source field does incur storage overhead within the index. For this reason, it can be disabled as follows:
resp = client.indices.create( index="my-index-000001", mappings={ "_source": { "enabled": False } }, ) print(resp)
response = client.indices.create( index: 'my-index-000001', body: { mappings: { _source: { enabled: false } } } ) puts response
const response = await client.indices.create({ index: "my-index-000001", mappings: { _source: { enabled: false, }, }, }); console.log(response);
PUT my-index-000001 { "mappings": { "_source": { "enabled": false } } }
Think before disabling the _source
field
Users often disable the _source
field without thinking about the
consequences, and then live to regret it. If the _source
field isn’t
available then a number of features are not supported:
-
The
update
,update_by_query
, andreindex
APIs. - In the Kibana Discover application, field data will not be displayed.
- On the fly highlighting.
- The ability to reindex from one Elasticsearch index to another, either to change mappings or analysis, or to upgrade an index to a new major version.
- The ability to debug queries or aggregations by viewing the original document used at index time.
- Potentially in the future, the ability to repair index corruption automatically.
If disk space is a concern, rather increase the
compression level instead of disabling the _source
.
Including / Excluding fields from _source
editAn expert-only feature is the ability to prune the contents of the _source
field after the document has been indexed, but before the _source
field is
stored.
Removing fields from the _source
has similar downsides to disabling
_source
, especially the fact that you cannot reindex documents from one
Elasticsearch index to another. Consider using
source filtering instead.
The includes
/excludes
parameters (which also accept wildcards) can be used
as follows:
resp = client.indices.create( index="logs", mappings={ "_source": { "includes": [ "*.count", "meta.*" ], "excludes": [ "meta.description", "meta.other.*" ] } }, ) print(resp) resp1 = client.index( index="logs", id="1", document={ "requests": { "count": 10, "foo": "bar" }, "meta": { "name": "Some metric", "description": "Some metric description", "other": { "foo": "one", "baz": "two" } } }, ) print(resp1) resp2 = client.search( index="logs", query={ "match": { "meta.other.foo": "one" } }, ) print(resp2)
response = client.indices.create( index: 'logs', body: { mappings: { _source: { includes: [ '*.count', 'meta.*' ], excludes: [ 'meta.description', 'meta.other.*' ] } } } ) puts response response = client.index( index: 'logs', id: 1, body: { requests: { count: 10, foo: 'bar' }, meta: { name: 'Some metric', description: 'Some metric description', other: { foo: 'one', baz: 'two' } } } ) puts response response = client.search( index: 'logs', body: { query: { match: { 'meta.other.foo' => 'one' } } } ) puts response
const response = await client.indices.create({ index: "logs", mappings: { _source: { includes: ["*.count", "meta.*"], excludes: ["meta.description", "meta.other.*"], }, }, }); console.log(response); const response1 = await client.index({ index: "logs", id: 1, document: { requests: { count: 10, foo: "bar", }, meta: { name: "Some metric", description: "Some metric description", other: { foo: "one", baz: "two", }, }, }, }); console.log(response1); const response2 = await client.search({ index: "logs", query: { match: { "meta.other.foo": "one", }, }, }); console.log(response2);
PUT logs { "mappings": { "_source": { "includes": [ "*.count", "meta.*" ], "excludes": [ "meta.description", "meta.other.*" ] } } } PUT logs/_doc/1 { "requests": { "count": 10, "foo": "bar" }, "meta": { "name": "Some metric", "description": "Some metric description", "other": { "foo": "one", "baz": "two" } } } GET logs/_search { "query": { "match": { "meta.other.foo": "one" } } }