The Great Mapping Refactoring

In the past we have said that document types are “like tables in a database”, which is a nice simple way to explain their purpose. Unfortunately, it is just not true: fields with the same name in different document types in the same index map to the same Lucene field name internally.

If you have an error field which is mapped as an integer in the apache document type and as a string in the nginx document type, Elasticsearch will end up mixing numeric and string data into the same Lucene field. Trying to search or aggregate on this field will either return wrong results, throw an exception, or even corrupt your index.

To resolve this problem, we first considered prefixing field names with the document type name, to make each field completely independent. The advantage of this approach is that document types would behave as real tables.

Unfortunately, it comes with a number of disadvantages:

The solution

In the end, we opted to enforce the rule that all fields with the same name in the same index must have the same mapping, although certain parameters such as copy_to and enabled are still allowed to be specified per-type. This resolves the issues with data corruption, query time exceptions, and incorrect results. Queries and aggregations remain as fast as they are today, and we can maximise compression and minimise heap and disk usage.

The disadvantage of this solution is that users who have treated types as completely separate tables need to revise their approach. This is not as problematic as it sounds. In reality, most field names represent a certain type of data: a created_date is always going to be a date, a number_of_hits field is always going to be numeric. Users who have conflicting field mappings today are either getting wrong results or losing their data to corruption. The real difference is that we’re now enforcing correct behaviour at index time, instead of relying on the user to follow best practice.

While the majority of users have non-conflicting field mappings, there are occasions when conflicts exist, so what techniques are available to deal with these situations? There are a few solutions:

PUT my_index/_mapping/mapping_one
{
  "properties": {
    "error": {
      "type": "string",
      "index": "no",
      "copy_to": "error_code"
    },
    "error_code": {
      "type": "integer"
    }
  }
}
			

PUT my_index/_mapping/mapping_two
{
  "properties": {
    "error": {
      "type": "string",
      "index": "no",
      "copy_to": "error_message"
    },
    "error_message": {
      "type": "string"
    }
  }
}
			

{
  "message": "some string",
  "count":   1,
  "date":    "2015-06-01"
}
			

{
  "data": [
    {"key": "message", "str_val":  "some_string" },
    {"key": "count",   "int_val":  1             },
    {"key": "date",    "date_val": "2015-06-01"  }
  ]
}
			

Today, it is possible to refer to fields using their “short name”, the full path, or the full path prefixed by the document type. These options lead to ambiguity. Take this mapping for example:

{
  "mappings": {
    "user": {
      "properties": {
        "title": {
          "type": "string"
        }
      }
    },
    "blog": {
      "properties": {
        "title": {
          "type": "string"
        },
        "user": {
          "type": "object",
          "fields": {
            "title": {
              "type": "string"
            }
          }
        }
      }
    }
  }
}
		

• Does title refer to user.title ,blog.title, or blog.user.title? 
• Does user.title refer to user.title or blog.user.title? 

The answer is: it depends which one Elasticsearch finds first. The field that is selected could even change from request to request, depending on how the mappings are serialised on each node.

In 2.0, you will have to use the full path name without the document type prefix to refer to fields:

• user.title maps only to the user.title field in the blog type, 
• title maps to the title field in user and in blog. 
• *title will match user.title and both title fields.

How would we differentiate between the title field in user and the title field in blog? 

We don’t have to. Because of the change explained in Conflicting field mappings, the mapping for the title field is the same in both types. In essence, there is only one field called title.

The type prefix (user. or blog.) used to have the side effect of filtering by the specified type. Querying the field blog.title would find only documents of type blog, not documents of type user. This syntax is misleading because it doesn’t work everywhere: aggregations or suggestions could contain results from any type. For this reason, plus the ambiguity demonstrated above, the type prefix is no longer supported.

IMPORTANT: You will need to update any percolators which make use of short names or the type prefix.

Every type has meta-fields like _id, _index, _routing, _parent, _timestamp, most of which supported various configuration options like index, store, or path. We have simplified these settings considerably.

It used to be possible to extract the _id, _routing, and _timestamp values from fields in the document body. This functionality has been removed because they required two rounds of document parsing and could result in conflicts. Instead, these values must be set explicitly in the URL or query string instead.

With the exception of the _boost and _analyzer fields, the existing meta-field configuration will be respected on old indices.

It used to be possible to specify index and search analyzers at index level, at type level, at field level, and even at document level (with the _analyzer field). Combining tokens from different analysis chains into the same field results in bad relevance scores. With the move to disallowing conflicting field mappings, we have simplified the analysis settings considerably:

The index_name and path settings have been removed in favour of copy_to, which has been available since Elasticsearch v1.0.0. They will continue to work on existing indices, but will no longer be accepted when creating new indices.

Today, when indexing a document which contains a previously unseen field, the field is added to the mapping locally, then forwarded to the master, which issues a cluster update to inform all shards of the new mapping. It is possible that the same field could be added to two shards at the same time. It is also possible that these two mappings could be different: one could be a double while the other is a long, or one could be a string while the other is a date.

In these cases, the first mapping to arrive to the master node wins. However, the shard with the “losing” mapping is already using a different data type, and will continue to use it. Later on, perhaps after restarting that node, the shard is moved to a different node and the official mapping from the master is applied. This results in index corruption and data loss.

To prevent this, a shard will now wait for the master to accept the new mapping before allowing indexing to continue. This makes all mapping updates deterministic and safe. Indexing a document containing new fields may be somewhat slower than before, because of the need to wait for acceptance, but the speed of cluster state updates has been greatly improved by two new features:

• Cluster state diffs:  Whenever possible, only changes to the cluster state are published instead of the entire cluster state.
• Async shard info requests: During the shard allocation process, the master node sends a request to the data nodes to find out which ones have the most up to date copies of unassigned shards. This used to be a blocking call which would delay changes to the cluster state. As of v1.6.0, this request happens asynchronously in the background, allowing pending tasks like mapping updates to be processed more quickly.

If you have conflicting mappings, you will either need to reindex your data into a new index with correct mappings, or to delete old indices that you no longer need. You will not be able to upgrade to 2.0 without resolving these conflicts.