IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Edge n-gram tokenizer Letter tokenizer »
Elastic Docs ›Elasticsearch Guide [8.7] ›Text analysis ›Tokenizer reference

Keyword tokenizer

edit

Keyword tokenizer

edit

The keyword tokenizer is a “noop” tokenizer that accepts whatever text it is given and outputs the exact same text as a single term. It can be combined with token filters to normalise output, e.g. lower-casing email addresses.

Example output

edit
response = client.indices.analyze(
  body: {
    tokenizer: 'keyword',
    text: 'New York'
  }
)
puts response
POST _analyze
{
  "tokenizer": "keyword",
  "text": "New York"
}

The above sentence would produce the following term:

[ New York ]

Combine with token filters

edit

You can combine the keyword tokenizer with token filters to normalise structured data, such as product IDs or email addresses.

For example, the following analyze API request uses the keyword tokenizer and lowercase filter to convert an email address to lowercase.

response = client.indices.analyze(
  body: {
    tokenizer: 'keyword',
    filter: [
      'lowercase'
    ],
    text: '[email protected]'
  }
)
puts response
POST _analyze
{
  "tokenizer": "keyword",
  "filter": [ "lowercase" ],
  "text": "[email protected]"
}

The request produces the following token:

[ [email protected] ]

Configuration

edit

The keyword tokenizer accepts the following parameters:

buffer_size

The number of characters read into the term buffer in a single pass. Defaults to 256. The term buffer will grow by this size until all the text has been consumed. It is advisable not to change this setting.

« Edge n-gram tokenizer Letter tokenizer »