allocation
editallocation
editSummary
editFrom the Elasticsearch Shard Allocation Filtering Documentation:
Allows to control the allocation of indices on nodes based on include/exclude filters. The filters can be set both on the index level and on the cluster level. Lets start with an example of setting it on the cluster level:
Lets say we have 4 nodes, each has specific attribute called
tag
associated with it (the name of the attribute can be any name). Each node has a specific value associated withtag
. Node 1 has a settingnode.tag: value1
, Node 2 a setting ofnode.tag: value2
, and so on.We can create an index that will only deploy on nodes that have tag set to value1 and value2 by setting index.routing.allocation.include.tag to value1,value2. For example:
curl -XPUT localhost:9200/test/_settings -d '{ "index.routing.allocation.include.tag" : "value1,value2" }'On the other hand, we can create an index that will be deployed on all nodes except for nodes with a
tag
of valuevalue3
by settingindex.routing.allocation.exclude.tag
tovalue3
. For example:curl -XPUT localhost:9200/test/_settings -d '\{ + "index.routing.allocation.exclude.tag" : "value3" + }'
index.routing.allocation.require.*
can be used to specify a number of rules, all of which MUST match in order for a shard to be allocated to a node.
In versions 3.2.3 and older, Curator used this last method,
index.routing.allocation.require.*
, to force routing. Beginning in version
3.3.0, Curator also allows the use of include
and exclude
tagging.
Use-case
editYou may be wondering why this is useful. Imagine that you have a tiered Elasticsearch cluster, with indexing happening on nodes with SSDs, and longer-term storage on nodes with traditional spinning hard disks. Using routing allocation, you can do indexing on the SSD nodes, then after a few days use curator to shift them to change the routing allocation tags to re-allocate them onto the slower storage nodes.
Extra configuration needed
editYou can assign tags to your indices with Curator, but no allocation will occur
unless you have the tags set in each node’s elasticsearch.yml
:
SSD nodes:
node.tag: ssd
Spinning disk nodes:
node.tag: hdd
These tags can be anything. You could use fast
and slow
if you wanted.
tag
is only the example provided here. You can use other descriptive markers:
node.disk_type: ssd
or
node.disk_type: hdd
In like fashion, you would need to guarantee that new indexes are tagged with
the appropriate markers, e.g. tag
and ssd
. This should be handled with an
Elasticsearch mapping template. Logstash comes with one by default, but it has
no routing tags. You can use that template as a guide and add the appropriate
tags.
Flags
edit$ curator allocation --help Usage: curator allocation [OPTIONS] COMMAND [ARGS]... Index Allocation Options: --rule TEXT Routing allocation rule to apply, e.g. tag=ssd --type TEXT Specify an allocation type, include, exclude or require [default: require] --help Show this message and exit. Commands: indices Index selection.
This command requires the indices subcommand for index selection.
Example
editApply the tag
value of hdd
:
curator allocation --rule tag=hdd indices <<index selection parameters>>
This will append the setting index.routing.allocation.require.tag=hdd
to the
indices specified by index selection.