New

The executive guide to generative AI

Read more
About usPartnersSupport|Login
IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.
« Porter stem token filter Remove duplicates token filter »
Elastic Docs ›Elasticsearch Guide [7.7] ›Text analysis ›Token filter reference

Predicate script token filter

edit
IMPORTANT: This documentation is no longer updated. Refer to Elastic's version policy and the latest documentation.

Predicate script token filter

edit

The predicate_token_filter token filter takes a predicate script, and removes tokens that do not match the predicate.

Options

edit

script

a predicate script that determines whether or not the current token will be emitted. Note that only inline scripts are supported.

Settings example

edit

You can set it up like:

PUT /condition_example
{
    "settings" : {
        "analysis" : {
            "analyzer" : {
                "my_analyzer" : {
                    "tokenizer" : "standard",
                    "filter" : [ "my_script_filter" ]
                }
            },
            "filter" : {
                "my_script_filter" : {
                    "type" : "predicate_token_filter",
                    "script" : {
                        "source" : "token.getTerm().length() > 5"  
                    }
                }
            }
        }
    }
}
Copy as curlTry in Elastic 

This will emit tokens that are more than 5 characters long

And test it like:

POST /condition_example/_analyze
{
  "analyzer" : "my_analyzer",
  "text" : "What Flapdoodle"
}
Copy as curlTry in Elastic 

And it’d respond:

{
  "tokens": [
    {
      "token": "Flapdoodle",        
      "start_offset": 5,
      "end_offset": 15,
      "type": "<ALPHANUM>",
      "position": 1                 
    }
  ]
}

The token What has been removed from the tokenstream because it does not match the predicate.

The position and offset values are unaffected by the removal of earlier tokens
« Porter stem token filter Remove duplicates token filter »
Was this helpful?
Feedback