Nodes orchestration

NodeSets are used to specify the topology of the Elasticsearch cluster. Each NodeSet represents a group of Elasticsearch nodes that share the same Elasticsearch configuration and Kubernetes Pod configuration.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.16.1
  nodeSets:
  - name: master-nodes
    count: 3
    config:
      node.roles: ["master"]
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: standard
  - name: data-nodes
    count: 10
    config:
      node.roles: ["data"]
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1000Gi
        storageClassName: standard

In this example, the Elasticsearch resource defines two NodeSets:

The Elasticsearch cluster is composed of 13 nodes: 3 master nodes and 10 data nodes.

ECK handles smooth upgrades from one cluster specification to another. You can apply a new Elasticsearch specification at any time. For example, based on the Elasticsearch specification described in the NodeSets overview, you can:

ECK orchestrates NodeSet changes with no downtime and makes sure that:

Behind the scenes, ECK translates each NodeSet specified in the Elasticsearch resource into a StatefulSet in Kubernetes. The NodeSet specification is based on the StatefulSet specification:

The actual Pod creation is handled by the StatefulSet controller in Kubernetes. ECK relies on the OnDelete StatefulSet update strategy since it needs full control over when and how Pods get upgraded to a new revision.

When a Pod is removed and recreated (maybe with a newer revision), the StatefulSet controller makes sure that the PersistentVolumes attached to the original Pod are then attached to the new Pod.

Depending on how the NodeSets are updated, ECK handles the Kubernetes resource reconciliation in various ways.

In all these cases, ECK handles StatefulSet operations according to the Elasticsearch orchestration best practices by adjusting the following orchestration settings:

Due to relying on Kubernetes primitives such as StatefulSets, the ECK orchestration process has some inherent limitations.

If an Elasticsearch node holds the only copy of a shard, this shard becomes unavailable while the node is upgraded. To ensure high availability it is recommended to configure clusters with three master nodes, more than one node per data tier and at least one replica per index.

If these conditions are met, then ECK can delete a Pod for upgrade even if the cluster health is yellow, as long as the Pod is not holding the last available replica of a shard.

The health of the cluster is deliberately ignored in the following cases:

Advanced users may force an upgrade by manually deleting Pods themselves. The deleted Pods are automatically recreated at the latest revision.

Operations that reduce the number of nodes in the cluster cannot make progress without user intervention, if the Elasticsearch index replica settings are incompatible with the intended downscale. Specifically, if the Elasticsearch index settings demand a higher number of shard copies than data nodes in the cluster after the downscale operation, ECK cannot migrate the data away from the node about to be removed. You can address this in the following ways:

The rules (otherwise known as predicates) that the ECK operator follows during an Elasticsearch upgrade can be selectively disabled for certain scenarios where the ECK operator will not proceed with an Elasticsearch cluster upgrade because it deems it to be "unsafe".

Any of these predicates can be disabled by adding an annotation with the key of eck.k8s.elastic.co/disable-upgrade-predicates to the Elasticsearch metadata, specifically naming the predicate that is needing to be disabled. Also, all predicates can be disabled by replacing the name of any predicatae with "*".

Assume a given Elasticsearch cluster is a "red" state because of an un-allocatable shard setting that was applied to the cluster:

{
	"settings": {
		"index.routing.allocation.include._id": "does not exist"
	}
}

This cluster would never be allowed to be upgraded with the standard set of upgrade predicates in place, as the cluster is in a "red" state, and the named predicate only_restart_healthy_node_if_green_or_yellow prevents the upgrade.

If the following annotation was added to the cluster specification, and the version was increased from 7.15.2 → 7.15.3

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: testing
  annotations:
    eck.k8s.elastic.co/disable-upgrade-predicates: "only_restart_healthy_node_if_green_or_yellow"
    # Also note that eck.k8s.elastic.co/disable-upgrade-predicates: "*" would work as well, but is much less selective.
spec:
  version: 7.15.3 # previously set to 7.15.2, for example

The ECK operator would allow this upgrade to proceed, even though the cluster was in a "red" state during this upgrade process.