Total number of shards per node has been reached

Elasticsearch takes advantage of all available resources by distributing data (index shards) among the cluster nodes.

You can influence the data distribution by configuring the cluster.routing.allocation.total_shards_per_node dynamic cluster setting to restrict the number of shards that can be hosted on a single node in the cluster.

In earlier Elasticsearch versions, cluster.routing.allocation.total_shards_per_node is set to 1000. Reaching that limit causes the following error: Total number of shards per node has been reached and requires adjusting this setting or reducing the number of shards. In Elasticsearch 9.x, this setting is not configured by default, which means there is no upper bound on the number of shards per node unless the setting is explicitly defined.

Various configurations limiting how many shards can be hosted on a single node can lead to shards being unassigned, because the cluster does not have enough nodes to satisfy the configuration. To ensure that each node carries a reasonable shard load, you might need to resize your deployment.

Follow these steps to resolve this issue:

1. Check and adjust the cluster shard limit to determine the current value and increase it if needed.
2. Determine which data tier needs more capacity to identify the tier where shards need to be allocated.
3. Resize your deployment to add capacity and accommodate additional shards.

The cluster.routing.allocation.total_shards_per_node setting controls the maximum number of shards that can be allocated to each node in a cluster. When this limit is reached, Elasticsearch cannot assign new shards to that node, leading to unassigned shards in your cluster.

By checking the current value and increasing it, you allow more shards to be collocated on each node, which might resolve the allocation issue without adding more capacity to your cluster.

You can run the following steps using either API console or direct Elasticsearch API calls.

Use the get cluster-wide settings API to inspect the current value of cluster.routing.allocation.total_shards_per_node:

				GET /_cluster/settings?flat_settings
		

The response looks like this:

{
  "persistent": {
    "cluster.routing.allocation.total_shards_per_node": "300"
  },
  "transient": {}
}
		

1. Represents the current configured value for the total number of shards that can reside on one node in the cluster. If the value is null or absent, no explicit limit is configured.

Use the update the cluster settings API to increase the value to a higher number that accommodates your workload:

				PUT _cluster/settings
					{
  "persistent" : {
    "cluster.routing.allocation.total_shards_per_node" : 400
  }
}
		

1. The new value for the system-wide total_shards_per_node configuration is increased from the previous value of 300 to 400. The total_shards_per_node configuration can also be set to null, which represents no upper bound with regards to how many shards can be collocated on one node in the system.

If increasing the cluster shard limit alone doesn't resolve the issue, or if you want to distribute shards more evenly, you need to identify which data tier requires additional capacity.

Use the get index settings API to retrieve the configured value for the index.routing.allocation.include._tier_preference setting:

				GET /my-index-000001/_settings/index.routing.allocation.include._tier_preference?flat_settings
		

The response looks like this:

{
  "my-index-000001": {
    "settings": {
      "index.routing.allocation.include._tier_preference": "data_warm,data_hot"
    }
  }
}
		

1. Represents a comma-separated list of data tier node roles this index is allowed to be allocated on. The first tier in the list has the highest priority and is the tier the index is targeting. In this example, the tier preference is data_warm,data_hot, so the index is targeting the warm tier. If the warm tier lacks capacity, the index will fall back to the data_hot tier.

After you've identified the tier that needs more capacity, you can resize your deployment to distribute the shard load and allow previously unassigned shards to be allocated.