allocation

edit

allocation

edit

Summary

edit

From the Elasticsearch Shard Allocation Filtering Documentation:

Allows to control the allocation of indices on nodes based on include/exclude filters. The filters can be set both on the index level and on the cluster level. Lets start with an example of setting it on the cluster level:

Lets say we have 4 nodes, each has specific attribute called tag associated with it (the name of the attribute can be any name). Each node has a specific value associated with tag. Node 1 has a setting node.tag: value1, Node 2 a setting of node.tag: value2, and so on.

We can create an index that will only deploy on nodes that have tag set to value1 and value2 by setting index.routing.allocation.include.tag to value1,value2. For example:
curl -XPUT localhost:9200/test/_settings -d '{
    "index.routing.allocation.include.tag" : "value1,value2"
}'
On the other hand, we can create an index that will be deployed on all nodes except for nodes with a tag of value value3 by setting index.routing.allocation.exclude.tag to value3. For example:
curl -XPUT localhost:9200/test/_settings -d '\{ +
 "index.routing.allocation.exclude.tag" : "value3" +
}'
index.routing.allocation.require.* can be used to specify a number of rules, all of which MUST match in order for a shard to be allocated to a node.

In versions 3.2.3 and older, Curator used this last method, index.routing.allocation.require.*, to force routing. Beginning in version 3.3.0, Curator also allows the use of include and exclude tagging.

Use-case

edit

You may be wondering why this is useful. Imagine that you have a tiered Elasticsearch cluster, with indexing happening on nodes with SSDs, and longer-term storage on nodes with traditional spinning hard disks. Using routing allocation, you can do indexing on the SSD nodes, then after a few days use curator to shift them to change the routing allocation tags to re-allocate them onto the slower storage nodes.

Extra configuration needed

edit

You can assign tags to your indices with Curator, but no allocation will occur unless you have the tags set in each node’s elasticsearch.yml:

SSD nodes:

node.tag: ssd

Spinning disk nodes:

node.tag: hdd

These tags can be anything. You could use fast and slow if you wanted. tag is only the example provided here. You can use other descriptive markers:

node.disk_type: ssd

node.disk_type: hdd

In like fashion, you would need to guarantee that new indexes are tagged with the appropriate markers, e.g. tag and ssd. This should be handled with an Elasticsearch mapping template. Logstash comes with one by default, but it has no routing tags. You can use that template as a guide and add the appropriate tags.

Flags

edit

$ curator allocation --help
Usage: curator allocation [OPTIONS] COMMAND [ARGS]...

  Index Allocation

Options:
  --rule TEXT  Routing allocation rule to apply, e.g. tag=ssd
  --type TEXT  Specify an allocation type, include, exclude or require
               [default: require]
  --help       Show this message and exit.

Commands:
  indices  Index selection.

This command requires the indices subcommand for index selection.

Example

edit

Apply the tag value of hdd:

curator allocation --rule tag=hdd indices <<index selection parameters>>

This will append the setting index.routing.allocation.require.tag=hdd to the indices specified by index selection.

« alias bloom »