New

The executive guide to generative AI

Read more
About usPartnersSupport|Login
« ICU normalization token filter ICU collation token filter »
Elastic Docs ›Elasticsearch Plugins and Integrations [8.17] ›Analysis plugins ›ICU analysis plugin

ICU folding token filter

edit

ICU folding token filter

edit

Case folding of Unicode characters based on UTR#30, like the ASCII-folding token filter on steroids. It registers itself as the icu_folding token filter and is available to all indices:

PUT icu_sample
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "folded": {
            "tokenizer": "icu_tokenizer",
            "filter": [
              "icu_folding"
            ]
          }
        }
      }
    }
  }
}
Copy as curlTry in Elastic 

The ICU folding token filter already does Unicode normalization, so there is no need to use Normalize character or token filter as well.

Which letters are folded can be controlled by specifying the unicode_set_filter parameter, which accepts a UnicodeSet.

The following example exempts Swedish characters from folding. It is important to note that both upper and lowercase forms should be specified, and that these filtered character are not lowercased which is why we add the lowercase filter as well:

PUT icu_sample
{
  "settings": {
    "index": {
      "analysis": {
        "analyzer": {
          "swedish_analyzer": {
            "tokenizer": "icu_tokenizer",
            "filter": [
              "swedish_folding",
              "lowercase"
            ]
          }
        },
        "filter": {
          "swedish_folding": {
            "type": "icu_folding",
            "unicode_set_filter": "[^åäöÅÄÖ]"
          }
        }
      }
    }
  }
}
Copy as curlTry in Elastic 
« ICU normalization token filter ICU collation token filter »
Was this helpful?
Feedback