Hunspell Token Filter

edit

Hunspell Token Filter

edit

Basic support for hunspell stemming. Hunspell dictionaries will be picked up from a dedicated hunspell directory on the filesystem (defaults to <path.conf>/hunspell). Each dictionary is expected to have its own directory named after its associated locale (language). This dictionary directory is expected to hold both the *.aff and \*.dic files (all of which will automatically be picked up). For example, assuming the default hunspell location is used, the following directory layout will define the en_US dictionary:

- conf
    |-- hunspell
    |    |-- en_US
    |    |    |-- en_US.dic
    |    |    |-- en_US.aff

The location of the hunspell directory can be configured using the indices.analysis.hunspell.dictionary.location settings in elasticsearch.yml.

Each dictionary can be configured with two settings:

ignore_case
If true, dictionary matching will be case insensitive (defaults to false)
strict_affix_parsing
Determines whether errors while reading a affix rules file will cause exception or simple be ignored (defaults to true)

These settings can be configured globally in elasticsearch.yml using

  • indices.analysis.hunspell.dictionary.ignore_case and
  • indices.analysis.hunspell.dictionary.strict_affix_parsing

or for specific dictionaries:

  • indices.analysis.hunspell.dictionary.en_US.ignore_case and
  • indices.analysis.hunspell.dictionary.en_US.strict_affix_parsing.

It is also possible to add settings.yml file under the dictionary directory which holds these settings (this will override any other settings defined in the elasticsearch.yml).

One can use the hunspell stem filter by configuring it the analysis settings:

{
    "analysis" : {
        "analyzer" : {
            "en" : {
                "tokenizer" : "standard",
                "filter" : [ "lowercase", "en_US" ]
            }
        },
        "filter" : {
            "en_US" : {
                "type" : "hunspell",
                "locale" : "en_US",
                "dedup" : true
            }
        }
    }
}

The hunspell token filter accepts four options:

locale
A locale for this filter. If this is unset, the lang or language are used instead - so one of these has to be set.
dictionary
The name of a dictionary. The path to your hunspell dictionaries should be configured via indices.analysis.hunspell.dictionary.location before.
dedup
If only unique terms should be returned, this needs to be set to true. Defaults to true.
recursion_level
Configures the recursion level a stemmer can go into. Defaults to 2. Some languages (for example czech) give better results when set to 1 or 0, so you should test it out. (since 0.90.3)

As opposed to the snowball stemmers (which are algorithm based) this is a dictionary lookup based stemmer and therefore the quality of the stemming is determined by the quality of the dictionary.

References

edit

Hunspell is a spell checker and morphological analyzer designed for languages with rich morphology and complex word compounding and character encoding.