Total number of shards for an index on a single node exceeded

Elasticsearch takes advantage of all available resources by distributing data (index shards) among the cluster nodes.

You can influence this data distribution by configuring the index.routing.allocation.total_shards_per_node dynamic index setting to restrict the maximum number of shards from a single index that can be allocated to a node. For example, in case of a highly trafficked index, the value can be set to 1.

Various configurations limiting how many shards an index can have located on one node can lead to shards being unassigned, because the cluster does not have enough nodes to satisfy the index configuration. To fix this issue, complete the following steps:

1. Check and adjust the index allocation settings to determine the current value and increase it if needed.
2. Determine which data tier needs more capacity to identify the tier where shards need to be allocated.
3. Resize your deployment to add capacity and accommodate additional shards.

The index.routing.allocation.total_shards_per_node setting controls the maximum number of shards that can be collocated on a node in your cluster. When this limit is reached, Elasticsearch cannot assign new shards to that node, leading to unassigned shards in your cluster.

By checking the current value and increasing it, you allow more shards to be collocated on each node, which might resolve the allocation issue without adding more capacity to your cluster.

You can run the following steps using either API console or direct Elasticsearch API calls.

Use the get index settings API to inspect the index.routing.allocation.total_shards_per_node value for the index with unassigned shards:

				GET /my-index-000001/_settings/index.routing.allocation.total_shards_per_node?flat_settings
		

The response looks like this:

{
  "my-index-000001": {
    "settings": {
      "index.routing.allocation.total_shards_per_node": "1"
    }
  }
}
		

1. Represents the current configured value for the total number of shards that can reside on one node for the my-index-000001 index.

Use the update index settings API to increase the value for the total number of shards that can be assigned on a node to a higher value that accommodates your workload:

				PUT /my-index-000001/_settings
					{
  "index" : {
    "routing.allocation.total_shards_per_node" : "2"
  }
}
		

1. The new value for the total_shards_per_node configuration for the my-index-000001 index is increased from the previous value of 1 to 2. The total_shards_per_node configuration can also be set to -1, which represents no upper bound with regards to how many shards of the same index can reside on one node.

If increasing the index shard limit alone doesn't resolve the issue, or if you want to distribute shards more evenly, you need to identify which data tier requires additional capacity.

Use the get index settings API to retrieve the configured value for the index.routing.allocation.include._tier_preference setting:

				GET /my-index-000001/_settings/index.routing.allocation.include._tier_preference?flat_settings
		

The response looks like this:

{
  "my-index-000001": {
    "settings": {
      "index.routing.allocation.include._tier_preference": "data_warm,data_hot"
    }
  }
}
		

1. Represents a comma-separated list of data tier node roles this index is allowed to be allocated on. The first tier in the list has the highest priority and is the tier the index is targeting. In this example, the tier preference is data_warm,data_hot, so the index is targeting the warm tier. If the warm tier lacks capacity, the index will fall back to the data_hot tier.

After you've identified the tier that needs more capacity, you can resize your deployment to distribute the shard load and allow previously unassigned shards to be allocated.