Elasticsearch 1.5.0 Released

This release adds one of the most frequently requested features in Elasticsearch: inner hits, the ability to return the child documents which matched a has_child or nested query alongside each parent document.

For instance, imagine that you have a blog parent document and comment child documents. You would like to search for blogs posts which have comments mentioning “full text search":

GET /my_index/blog/_search
{
  "query": {
    "has_child": {
      "type":       "comment",
      "score_mode": "sum",
      "query": {
        "match": {
          "body":   "full text search"
        }
      }
    }
  }
}
	

The above request returns the parent blog documents, but gives us no indication of which comments were the most relevant. We would have to do a second, much trickier query to retrieve the most interesting comments and to group them by parent.

Inner hits changes all this! Just add the inner_hits parameter to the above query as follows:

GET /my_index/blog/_search
{
  "query": {
    "has_child": {
      "type":       "comment",
      "score_mode": "sum",
      "query": {
        "match": {
          "body":   "full text search"
        }
      },
      "inner_hits": {}
    }
  }
}
	

Each matching blog post will be returned with an inner_hits section containing (by default) the top three best matching comments:

...
"hits": [
  {
    "_index":   "my_index",
    "_type":    "blog",
    "_id":      1,
    "_score":   3.68,
    "_source":  { ... },
    "inner_hits": {
      "comment": {
        "total": 16,
        "hits": [
          {
            "_type":    "comment",
            "_id":      5,
            "_score":   2.79,
            "_source": {
              "body":   "Full text search is the bomb"
            }
          },
          { ... },
          { ... }
        ]
      }
    }
  }
]
...
	

The inner_hits section is like a secondary search request. Its behaviour can be customised by including parameters like size and from. It supports the functionality that you would expect from search: pagination, sorting, highlighting, _source filtering, etc.

Inner hits is supported for parent-child relationships and for nested documents. The feature is currently labelled experimental, which means that it may be completely rewritten or even removed in the future. See the Inner Hits documentation for more.

Elasticseach has always taken care of its own redundancy. It has replica shards — redundant copies of each primary shard — to allow Elasticsearch to survive the loss of the primary shard without losing any data. Replica shards also allow you to scale search throughput: the more replicas (combined with more nodes), the more throughput.

However, some users are hosting Elasticsearch on distributed file systems which already take care of replication and redundancy. It makes little sense to make multiple copies of each shard when the file system is doing the same thing.

Shadow replicas allow you to scale search throughput by adding more nodes, without paying the price of extra storage and indexing for each node. Instead, each shadow replica has read-only access to the shared file system holding the primary shard. The shadow replica refreshes its view of the file system on a regular basis, and will see any changes that the primary shard has flushed to disk.

If the primary shard fails, a shadow replica will be promoted to primary and will be able to read and replay the transaction log written by the failed primary.

This feature is marked experimental. See the Shadow Replicas documentation for more.

Elasticsearch 1.1 to 1.3 focused on adding checksums to all files in an index and using them to validate whether a file has been corrupted or not. Version 1.4 made huge improvements to Zen discovery and our distributed model.

The more detailed statistics and more granular logging that accompanied these changes has brought to the light previously unknown problems that existed in earlier versions of both Elasticsearch and Lucene. Elasticsearch 1.5.0 addresses many of these issues, including: