WARNING: The 2.x versions of Elasticsearch have passed their EOL dates. If you are running a 2.x version, we strongly advise you to upgrade.
This documentation is no longer maintained and may be removed. For the latest information, see the current Elasticsearch documentation.
Lucene’s Practical Scoring Function
editLucene’s Practical Scoring Function
editFor multiterm queries, Lucene takes the Boolean model, TF/IDF, and the vector space model and combines them in a single efficient package that collects matching documents and scores them as it goes.
A multiterm query like
GET /my_index/doc/_search { "query": { "match": { "text": "quick fox" } } }
is rewritten internally to look like this:
GET /my_index/doc/_search { "query": { "bool": { "should": [ {"term": { "text": "quick" }}, {"term": { "text": "fox" }} ] } } }
The bool
query implements the Boolean model and, in this example, will
include only documents that contain either the term quick
or the term fox
or
both.
As soon as a document matches a query, Lucene calculates its score for that query, combining the scores of each matching term. The formula used for scoring is called the practical scoring function. It looks intimidating, but don’t be put off—most of the components you already know. It introduces a few new elements that we discuss next.
score(q,d) = queryNorm(q) · coord(q,d) · ∑ ( tf(t in d) · idf(t)² · t.getBoost() · norm(t,d) ) (t in q)
|
|
|
|
|
|
The sum of the weights for each term |
|
|
|
|
|
|
|
|
You should recognize score
, tf
, and idf
. The queryNorm
, coord
,
t.getBoost
, and norm
are new.
We will talk more about query-time boosting later in this chapter, but first let’s get query normalization, coordination, and index-time field-level boosting out of the way.
Query Normalization Factor
editThe query normalization factor (queryNorm
) is an attempt to normalize a
query so that the results from one query may be compared with the results of
another.
Even though the intent of the query norm is to make results from different
queries comparable, it doesn’t work very well. The only purpose of
the relevance _score
is to sort the results of the current query in the
correct order. You should not try to compare the relevance scores from
different queries.
This factor is calculated at the beginning of the query. The actual calculation depends on the queries involved, but a typical implementation is as follows:
queryNorm = 1 / √sumOfSquaredWeights
The |
The same query normalization factor is applied to every document, and you have no way of changing it. For all intents and purposes, it can be ignored.
Query Coordination
editThe coordination factor (coord
) is used to reward documents that contain a
higher percentage of the query terms. The more query terms that appear in
the document, the greater the chances that the document is a good match for
the query.
Imagine that we have a query for quick brown fox
, and that the
weight for each term is 1.5. Without the coordination factor, the score would
just be the sum of the weights of the terms in a document. For instance:
-
Document with
fox
→ score: 1.5 -
Document with
quick fox
→ score: 3.0 -
Document with
quick brown fox
→ score: 4.5
The coordination factor multiplies the score by the number of matching terms in the document, and divides it by the total number of terms in the query. With the coordination factor, the scores would be as follows:
-
Document with
fox
→ score:1.5 * 1 / 3
= 0.5 -
Document with
quick fox
→ score:3.0 * 2 / 3
= 2.0 -
Document with
quick brown fox
→ score:4.5 * 3 / 3
= 4.5
The coordination factor results in the document that contains all three terms being much more relevant than the document that contains just two of them.
Remember that the query for quick brown fox
is rewritten into a bool
query
like this:
GET /_search { "query": { "bool": { "should": [ { "term": { "text": "quick" }}, { "term": { "text": "brown" }}, { "term": { "text": "fox" }} ] } } }
The bool
query uses query coordination by default for all should
clauses,
but it does allow you to disable coordination. Why might you want to do this?
Well, usually the answer is, you don’t. Query coordination is usually a good
thing. When you use a bool
query to wrap several high-level queries like
the match
query, it also makes sense to leave coordination enabled. The more
clauses that match, the higher the degree of overlap between your search
request and the documents that are returned.
However, in some advanced use cases, it might make sense to disable
coordination. Imagine that you are looking for the synonyms jump
, leap
, and
hop
. You don’t care how many of these synonyms are present, as they all
represent the same concept. In fact, only one of the synonyms is likely to be
present. This would be a good case for disabling the coordination factor:
GET /_search { "query": { "bool": { "disable_coord": true, "should": [ { "term": { "text": "jump" }}, { "term": { "text": "hop" }}, { "term": { "text": "leap" }} ] } } }
When you use synonyms (see Synonyms), this is exactly what happens internally: the rewritten query disables coordination for the synonyms. Most use cases for disabling coordination are handled automatically; you don’t need to worry about it.
Index-Time Field-Level Boosting
editWe will talk about boosting a field—making it more important than other fields—at query time in Query-Time Boosting. It is also possible to apply a boost to a field at index time. Actually, this boost is applied to every term in the field, rather than to the field itself.
To store this boost value in the index without using more space
than necessary, this field-level index-time boost is combined with the field-length norm (see Field-length norm) and stored in the index as a single byte.
This is the value returned by norm(t,d)
in the preceding formula.
We strongly recommend against using field-level index-time boosts for a few reasons:
- Combining the boost with the field-length norm and storing it in a single byte means that the field-length norm loses precision. The result is that Elasticsearch is unable to distinguish between a field containing three words and a field containing five words.
- To change an index-time boost, you have to reindex all your documents. A query-time boost, on the other hand, can be changed with every query.
- If a field with an index-time boost has multiple values, the boost is multiplied by itself for every value, dramatically increasing the weight for that field.
Query-time boosting is a much simpler, cleaner, more flexible option.
With query normalization, coordination, and index-time boosting out of the way, we can now move on to the most useful tool for influencing the relevance calculation: query-time boosting.