Cardinality Aggregation

Computing exact counts requires loading values into a hash set and returning its size. This doesn’t scale when working on high-cardinality sets and/or large values as the required memory usage and the need to communicate those per-shard sets between nodes would utilize too many resources of the cluster.

This cardinality aggregation is based on the HyperLogLog++ algorithm, which counts based on the hashes of the values with some interesting properties:

For a precision threshold of c, the implementation that we are using requires about c * 8 bytes.

The following chart shows how the error varies before and after the threshold:

For all 3 thresholds, counts have been accurate up to the configured threshold (although not guaranteed, this is likely to be the case). Please also note that even with a threshold as low as 100, the error remains under 5%, even when counting millions of items.

If you don’t want Elasticsearch to re-compute hashes on every run of this aggregation, it is possible to use pre-computed hashes, either by computing a hash on client-side, indexing it and specifying rehash: false, or by using the special murmur3 field mapper, typically in the context of a multi-field in the mapping:

{
    "author": {
        "type": "string",
        "fields": {
            "hash": {
                "type": "murmur3"
            }
        }
    }
}

With such a mapping, Elasticsearch is going to compute hashes of the author field at indexing time and store them in the author.hash field. This way, unique counts can be computed using the cardinality aggregation by only loading the hashes into memory, not the values of the author field, and without computing hashes on the fly:

{
    "aggs" : {
        "author_count" : {
            "cardinality" : {
                "field" : "author.hash"
            }
        }
    }
}

The cardinality metric supports scripting, with a noticeable performance hit however since hashes need to be computed on the fly.

{
    "aggs" : {
        "author_count" : {
            "cardinality" : {
                "script": "doc['author.first_name'].value + ' ' + doc['author.last_name'].value"
            }
        }
    }
}