Add remote clusters using TLS certificate authentication

edit

Add remote clusters using TLS certificate authentication

edit

To add a remote cluster using TLS certificate authentication:

If you run into any issues, refer to Troubleshooting.

Prerequisites

edit
  1. The Elasticsearch security features need to be enabled on both clusters, on every node. Security is enabled by default. If it’s disabled, set xpack.security.enabled to true in elasticsearch.yml. Refer to General security settings.
  2. The local and remote clusters versions must be compatible.

    • Any node can communicate with another node on the same major version. For example, 7.0 can talk to any 7.x node.
    • Only nodes on the last minor version of a certain major version can communicate with nodes on the following major version. In the 6.x series, 6.8 can communicate with any 7.x node, while 6.7 can only communicate with 7.0.
    • Version compatibility is symmetric, meaning that if 6.7 can communicate with 7.0, 7.0 can also communicate with 6.7. The following table depicts version compatibility between local and remote nodes.

      Version compatibility table

      Local cluster

      Remote cluster

      5.0–5.5

      5.6

      6.0–6.6

      6.7

      6.8

      7.0

      7.1–7.16

      7.17

      8.0–8.16

      5.0–5.5

      Yes

      Yes

      No

      No

      No

      No

      No

      No

      No

      5.6

      Yes

      Yes

      Yes

      Yes

      Yes

      No

      No

      No

      No

      6.0–6.6

      No

      Yes

      Yes

      Yes

      Yes

      No

      No

      No

      No

      6.7

      No

      Yes

      Yes

      Yes

      Yes

      Yes

      No

      No

      No

      6.8

      No

      Yes

      Yes

      Yes

      Yes

      Yes

      Yes

      Yes

      No

      7.0

      No

      No

      No

      Yes

      Yes

      Yes

      Yes

      Yes

      No

      7.1–7.16

      No

      No

      No

      No

      Yes

      Yes

      Yes

      Yes

      No

      7.17

      No

      No

      No

      No

      Yes

      Yes

      Yes

      Yes

      Yes

      8.0–8.16

      No

      No

      No

      No

      No

      No

      No

      Yes

      Yes

      Elastic only supports cross-cluster search on a subset of these configurations. See Supported cross-cluster search configurations.

Establish trust with a remote cluster

edit

To use cross-cluster replication or cross-cluster search safely with remote clusters, enable security on all connected clusters and configure Transport Layer Security (TLS) on every node. Configuring TLS security on the transport interface is minimally required for remote clusters. For additional security, configure TLS on the HTTP interface as well.

All connected clusters must trust one another and be mutually authenticated with TLS on the transport interface. This means that the local cluster trusts the certificate authority (CA) of the remote cluster, and the remote cluster trusts the CA of the local cluster. When establishing a connection, all nodes will verify certificates from nodes on the other side. This mutual trust is required to securely connect a remote cluster, because all connected nodes effectively form a single security domain.

User authentication is performed on the local cluster and the user and user’s roles names are passed to the remote clusters. A remote cluster checks the user’s role names against its local role definitions to determine which indices the user is allowed to access.

Before using cross-cluster replication or cross-cluster search with secured Elasticsearch clusters, complete the following configuration task:

  1. Configure Transport Layer Security (TLS) on every node to encrypt internode traffic and authenticate nodes in the local cluster with nodes in all remote clusters. Refer to set up basic security for the Elastic Stack for the required steps to configure security.

    This procedure uses the same CA to generate certificates for all nodes. Alternatively, you can add the certificates from the local cluster as a trusted CA in each remote cluster. You must also add the certificates from remote clusters as a trusted CA on the local cluster. Using the same CA to generate certificates for all nodes simplifies this task.

Connect to a remote cluster

edit

You must have the manage cluster privilege to connect remote clusters.

The local cluster uses the transport interface to establish communication with remote clusters. The coordinating nodes in the local cluster establish long-lived TCP connections with specific nodes in the remote cluster. Elasticsearch requires these connections to remain open, even if the connections are idle for an extended period.

To add a remote cluster from Stack Management in Kibana:

  1. Select Remote Clusters from the side navigation.
  2. Enter a name (cluster alias) for the remote cluster.
  3. Specify the Elasticsearch endpoint URL, or the IP address or host name of the remote cluster followed by the transport port (defaults to 9300). For example, cluster.es.eastus2.staging.azure.foundit.no:9300 or 192.168.1.1:9300.

Alternatively, use the cluster update settings API to add a remote cluster. You can also use this API to dynamically configure remote clusters for every node in the local cluster. To configure remote clusters on individual nodes in the local cluster, define static settings in elasticsearch.yml for each node.

The following request adds a remote cluster with an alias of cluster_one. This cluster alias is a unique identifier that represents the connection to the remote cluster and is used to distinguish between local and remote indices.

resp = client.cluster.put_settings(
    persistent={
        "cluster": {
            "remote": {
                "cluster_one": {
                    "seeds": [
                        "127.0.0.1:{remote-interface-default-port}"
                    ]
                }
            }
        }
    },
)
print(resp)
const response = await client.cluster.putSettings({
  persistent: {
    cluster: {
      remote: {
        cluster_one: {
          seeds: ["127.0.0.1:{remote-interface-default-port}"],
        },
      },
    },
  },
});
console.log(response);
PUT /_cluster/settings
{
  "persistent" : {
    "cluster" : {
      "remote" : {
        "cluster_one" : {    
          "seeds" : [
            "127.0.0.1:9300" 
          ]
        }
      }
    }
  }
}

The cluster alias of this remote cluster is cluster_one.

Specifies the hostname and transport port of a seed node in the remote cluster.

You can use the remote cluster info API to verify that the local cluster is successfully connected to the remote cluster:

resp = client.cluster.remote_info()
print(resp)
response = client.cluster.remote_info
puts response
const response = await client.cluster.remoteInfo();
console.log(response);
GET /_remote/info

The API response indicates that the local cluster is connected to the remote cluster with the cluster alias cluster_one:

{
  "cluster_one" : {
    "seeds" : [
      "127.0.0.1:9300"
    ],
    "connected" : true,
    "num_nodes_connected" : 1,  
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : true, 
    "mode" : "sniff"
  }
}

The number of nodes in the remote cluster the local cluster is connected to.

Indicates whether to skip the remote cluster if searched through cross-cluster search but no nodes are available.

Dynamically configure remote clusters

edit

Use the cluster update settings API to dynamically configure remote settings on every node in the cluster. The following request adds three remote clusters: cluster_one, cluster_two, and cluster_three.

The seeds parameter specifies the hostname and transport port (default 9300) of a seed node in the remote cluster.

The mode parameter determines the configured connection mode, which defaults to sniff. Because cluster_one doesn’t specify a mode, it uses the default. Both cluster_two and cluster_three explicitly use different modes.

resp = client.cluster.put_settings(
    persistent={
        "cluster": {
            "remote": {
                "cluster_one": {
                    "seeds": [
                        "127.0.0.1:{remote-interface-default-port}"
                    ]
                },
                "cluster_two": {
                    "mode": "sniff",
                    "seeds": [
                        "127.0.0.1:{remote-interface-default-port-plus1}"
                    ],
                    "transport.compress": True,
                    "skip_unavailable": True
                },
                "cluster_three": {
                    "mode": "proxy",
                    "proxy_address": "127.0.0.1:{remote-interface-default-port-plus2}"
                }
            }
        }
    },
)
print(resp)
const response = await client.cluster.putSettings({
  persistent: {
    cluster: {
      remote: {
        cluster_one: {
          seeds: ["127.0.0.1:{remote-interface-default-port}"],
        },
        cluster_two: {
          mode: "sniff",
          seeds: ["127.0.0.1:{remote-interface-default-port-plus1}"],
          "transport.compress": true,
          skip_unavailable: true,
        },
        cluster_three: {
          mode: "proxy",
          proxy_address: "127.0.0.1:{remote-interface-default-port-plus2}",
        },
      },
    },
  },
});
console.log(response);
PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "cluster_one": {
          "seeds": [
            "127.0.0.1:9300"
          ]
        },
        "cluster_two": {
          "mode": "sniff",
          "seeds": [
            "127.0.0.1:9301"
          ],
          "transport.compress": true,
          "skip_unavailable": true
        },
        "cluster_three": {
          "mode": "proxy",
          "proxy_address": "127.0.0.1:9302"
        }
      }
    }
  }
}

You can dynamically update settings for a remote cluster after the initial configuration. The following request updates the compression settings for cluster_two, and the compression and ping schedule settings for cluster_three.

When the compression or ping schedule settings change, all existing node connections must close and re-open, which can cause in-flight requests to fail.

resp = client.cluster.put_settings(
    persistent={
        "cluster": {
            "remote": {
                "cluster_two": {
                    "transport.compress": False
                },
                "cluster_three": {
                    "transport.compress": True,
                    "transport.ping_schedule": "60s"
                }
            }
        }
    },
)
print(resp)
response = client.cluster.put_settings(
  body: {
    persistent: {
      cluster: {
        remote: {
          cluster_two: {
            'transport.compress' => false
          },
          cluster_three: {
            'transport.compress' => true,
            'transport.ping_schedule' => '60s'
          }
        }
      }
    }
  }
)
puts response
const response = await client.cluster.putSettings({
  persistent: {
    cluster: {
      remote: {
        cluster_two: {
          "transport.compress": false,
        },
        cluster_three: {
          "transport.compress": true,
          "transport.ping_schedule": "60s",
        },
      },
    },
  },
});
console.log(response);
PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "cluster_two": {
          "transport.compress": false
        },
        "cluster_three": {
          "transport.compress": true,
          "transport.ping_schedule": "60s"
        }
      }
    }
  }
}

You can delete a remote cluster from the cluster settings by passing null values for each remote cluster setting. The following request removes cluster_two from the cluster settings, leaving cluster_one and cluster_three intact:

resp = client.cluster.put_settings(
    persistent={
        "cluster": {
            "remote": {
                "cluster_two": {
                    "mode": None,
                    "seeds": None,
                    "skip_unavailable": None,
                    "transport.compress": None
                }
            }
        }
    },
)
print(resp)
response = client.cluster.put_settings(
  body: {
    persistent: {
      cluster: {
        remote: {
          cluster_two: {
            mode: nil,
            seeds: nil,
            skip_unavailable: nil,
            'transport.compress' => nil
          }
        }
      }
    }
  }
)
puts response
const response = await client.cluster.putSettings({
  persistent: {
    cluster: {
      remote: {
        cluster_two: {
          mode: null,
          seeds: null,
          skip_unavailable: null,
          "transport.compress": null,
        },
      },
    },
  },
});
console.log(response);
PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "cluster_two": {
          "mode": null,
          "seeds": null,
          "skip_unavailable": null,
          "transport.compress": null
        }
      }
    }
  }
}

Statically configure remote clusters

edit

If you specify settings in elasticsearch.yml, only the nodes with those settings can connect to the remote cluster and serve remote cluster requests.

Remote cluster settings that are specified using the cluster update settings API take precedence over settings that you specify in elasticsearch.yml for individual nodes.

In the following example, cluster_one, cluster_two, and cluster_three are arbitrary cluster aliases representing the connection to each cluster. These names are subsequently used to distinguish between local and remote indices.

cluster:
    remote:
        cluster_one:
            seeds: 127.0.0.1:9300
        cluster_two:
            mode: sniff
            seeds: 127.0.0.1:9301
            transport.compress: true      
            skip_unavailable: true        
        cluster_three:
            mode: proxy
            proxy_address: 127.0.0.1:9302 

Compression is explicitly enabled for requests to cluster_two.

Disconnected remote clusters are optional for cluster_two.

The address for the proxy endpoint used to connect to cluster_three.

Configure roles and users for remote clusters

edit

After connecting remote clusters, you create a user role on both the local and remote clusters and assign necessary privileges. These roles are required to use cross-cluster replication and cross-cluster search.

You must use the same role names on both the local and remote clusters. For example, the following configuration for cross-cluster replication uses the remote-replication role name on both the local and remote clusters. However, you can specify different role definitions on each cluster.

You can manage users and roles from Stack Management in Kibana by selecting Security > Roles from the side navigation. You can also use the role management APIs to add, update, remove, and retrieve roles dynamically. When you use the APIs to manage roles in the native realm, the roles are stored in an internal Elasticsearch index.

The following requests use the create or update roles API. You must have at least the manage_security cluster privilege to use this API.

Configure privileges for cross-cluster replication

edit

The cross-cluster replication user requires different cluster and index privileges on the remote cluster and local cluster. Use the following requests to create separate roles on the local and remote clusters, and then create a user with the required roles.

Remote cluster
edit

On the remote cluster that contains the leader index, the cross-cluster replication role requires the read_ccr cluster privilege, and monitor and read privileges on the leader index.

If requests are authenticated with an API key, the API key requires the above privileges on the local cluster, instead of the remote.

If requests are issued on behalf of other users, then the authenticating user must have the run_as privilege on the remote cluster.

The following request creates a remote-replication role on the remote cluster:

resp = client.security.put_role(
    name="remote-replication",
    cluster=[
        "read_ccr"
    ],
    indices=[
        {
            "names": [
                "leader-index-name"
            ],
            "privileges": [
                "monitor",
                "read"
            ]
        }
    ],
)
print(resp)
const response = await client.security.putRole({
  name: "remote-replication",
  cluster: ["read_ccr"],
  indices: [
    {
      names: ["leader-index-name"],
      privileges: ["monitor", "read"],
    },
  ],
});
console.log(response);
POST /_security/role/remote-replication
{
  "cluster": [
    "read_ccr"
  ],
  "indices": [
    {
      "names": [
        "leader-index-name"
      ],
      "privileges": [
        "monitor",
        "read"
      ]
    }
  ]
}
Local cluster
edit

On the local cluster that contains the follower index, the remote-replication role requires the manage_ccr cluster privilege, and monitor, read, write, and manage_follow_index privileges on the follower index.

The following request creates a remote-replication role on the local cluster:

resp = client.security.put_role(
    name="remote-replication",
    cluster=[
        "manage_ccr"
    ],
    indices=[
        {
            "names": [
                "follower-index-name"
            ],
            "privileges": [
                "monitor",
                "read",
                "write",
                "manage_follow_index"
            ]
        }
    ],
)
print(resp)
const response = await client.security.putRole({
  name: "remote-replication",
  cluster: ["manage_ccr"],
  indices: [
    {
      names: ["follower-index-name"],
      privileges: ["monitor", "read", "write", "manage_follow_index"],
    },
  ],
});
console.log(response);
POST /_security/role/remote-replication
{
  "cluster": [
    "manage_ccr"
  ],
  "indices": [
    {
      "names": [
        "follower-index-name"
      ],
      "privileges": [
        "monitor",
        "read",
        "write",
        "manage_follow_index"
      ]
    }
  ]
}

After creating the remote-replication role on each cluster, use the create or update users API to create a user on the local cluster cluster and assign the remote-replication role. For example, the following request assigns the remote-replication role to a user named cross-cluster-user:

resp = client.security.put_user(
    username="cross-cluster-user",
    password="l0ng-r4nd0m-p@ssw0rd",
    roles=[
        "remote-replication"
    ],
)
print(resp)
const response = await client.security.putUser({
  username: "cross-cluster-user",
  password: "l0ng-r4nd0m-p@ssw0rd",
  roles: ["remote-replication"],
});
console.log(response);
POST /_security/user/cross-cluster-user
{
  "password" : "l0ng-r4nd0m-p@ssw0rd",
  "roles" : [ "remote-replication" ]
}

You only need to create this user on the local cluster.

You can then configure cross-cluster replication to replicate your data across datacenters.

Configure privileges for cross-cluster search

edit

The cross-cluster search user requires different cluster and index privileges on the remote cluster and local cluster. The following requests create separate roles on the local and remote clusters, and then create a user with the required roles.

Remote cluster
edit

On the remote cluster, the cross-cluster search role requires the read and read_cross_cluster privileges for the target indices.

If requests are authenticated with an API key, the API key requires the above privileges on the local cluster, instead of the remote.

If requests are issued on behalf of other users, then the authenticating user must have the run_as privilege on the remote cluster.

The following request creates a remote-search role on the remote cluster:

resp = client.security.put_role(
    name="remote-search",
    indices=[
        {
            "names": [
                "target-indices"
            ],
            "privileges": [
                "read",
                "read_cross_cluster"
            ]
        }
    ],
)
print(resp)
const response = await client.security.putRole({
  name: "remote-search",
  indices: [
    {
      names: ["target-indices"],
      privileges: ["read", "read_cross_cluster"],
    },
  ],
});
console.log(response);
POST /_security/role/remote-search
{
  "indices": [
    {
      "names": [
        "target-indices"
      ],
      "privileges": [
        "read",
        "read_cross_cluster"
      ]
    }
  ]
}
Local cluster
edit

On the local cluster, which is the cluster used to initiate cross cluster search, a user only needs the remote-search role. The role privileges can be empty.

The following request creates a remote-search role on the local cluster:

resp = client.security.put_role(
    name="remote-search",
)
print(resp)
const response = await client.security.putRole({
  name: "remote-search",
});
console.log(response);
POST /_security/role/remote-search
{}

After creating the remote-search role on each cluster, use the create or update users API to create a user on the local cluster and assign the remote-search role. For example, the following request assigns the remote-search role to a user named cross-search-user:

resp = client.security.put_user(
    username="cross-search-user",
    password="l0ng-r4nd0m-p@ssw0rd",
    roles=[
        "remote-search"
    ],
)
print(resp)
const response = await client.security.putUser({
  username: "cross-search-user",
  password: "l0ng-r4nd0m-p@ssw0rd",
  roles: ["remote-search"],
});
console.log(response);
POST /_security/user/cross-search-user
{
  "password" : "l0ng-r4nd0m-p@ssw0rd",
  "roles" : [ "remote-search" ]
}

You only need to create this user on the local cluster.

Users with the remote-search role can then search across clusters.

Configure privileges for cross-cluster search and Kibana

edit

When using Kibana to search across multiple clusters, a two-step authorization process determines whether or not the user can access data streams and indices on a remote cluster:

  • First, the local cluster determines if the user is authorized to access remote clusters. The local cluster is the cluster that Kibana is connected to.
  • If the user is authorized, the remote cluster then determines if the user has access to the specified data streams and indices.

To grant Kibana users access to remote clusters, assign them a local role with read privileges to indices on the remote clusters. You specify data streams and indices in a remote cluster as <remote_cluster_name>:<target>.

To grant users read access on the remote data streams and indices, you must create a matching role on the remote clusters that grants the read_cross_cluster privilege with access to the appropriate data streams and indices.

For example, you might be actively indexing Logstash data on a local cluster and and periodically offload older time-based indices to an archive on your remote cluster. You want to search across both clusters, so you must enable Kibana users on both clusters.

Local cluster
edit

On the local cluster, create a logstash-reader role that grants read and view_index_metadata privileges on the local logstash-* indices.

If you configure the local cluster as another remote in Elasticsearch, the logstash-reader role on your local cluster also needs to grant the read_cross_cluster privilege.

resp = client.security.put_role(
    name="logstash-reader",
    indices=[
        {
            "names": [
                "logstash-*"
            ],
            "privileges": [
                "read",
                "view_index_metadata"
            ]
        }
    ],
)
print(resp)
const response = await client.security.putRole({
  name: "logstash-reader",
  indices: [
    {
      names: ["logstash-*"],
      privileges: ["read", "view_index_metadata"],
    },
  ],
});
console.log(response);
POST /_security/role/logstash-reader
{
  "indices": [
    {
      "names": [
        "logstash-*"
        ],
        "privileges": [
          "read",
          "view_index_metadata"
          ]
    }
  ]
}

Assign your Kibana users a role that grants access to Kibana, as well as your logstash_reader role. For example, the following request creates the cross-cluster-kibana user and assigns the kibana-access and logstash-reader roles.

resp = client.security.put_user(
    username="cross-cluster-kibana",
    password="l0ng-r4nd0m-p@ssw0rd",
    roles=[
        "logstash-reader",
        "kibana-access"
    ],
)
print(resp)
const response = await client.security.putUser({
  username: "cross-cluster-kibana",
  password: "l0ng-r4nd0m-p@ssw0rd",
  roles: ["logstash-reader", "kibana-access"],
});
console.log(response);
PUT /_security/user/cross-cluster-kibana
{
  "password" : "l0ng-r4nd0m-p@ssw0rd",
  "roles" : [
    "logstash-reader",
    "kibana-access"
    ]
}
Remote cluster
edit

On the remote cluster, create a logstash-reader role that grants the read_cross_cluster privilege and read and view_index_metadata privileges for the logstash-* indices.

resp = client.security.put_role(
    name="logstash-reader",
    indices=[
        {
            "names": [
                "logstash-*"
            ],
            "privileges": [
                "read_cross_cluster",
                "read",
                "view_index_metadata"
            ]
        }
    ],
)
print(resp)
const response = await client.security.putRole({
  name: "logstash-reader",
  indices: [
    {
      names: ["logstash-*"],
      privileges: ["read_cross_cluster", "read", "view_index_metadata"],
    },
  ],
});
console.log(response);
POST /_security/role/logstash-reader
{
  "indices": [
    {
      "names": [
        "logstash-*"
        ],
        "privileges": [
          "read_cross_cluster",
          "read",
          "view_index_metadata"
          ]
    }
  ]
}