Access clusters of a self-managed environment

edit

This section explains how to configure a deployment to connect remotely to self-managed clusters.

Allow the remote connection

edit

Before you start, consider the security model that you would prefer to use for authenticating remote connections between clusters, and follow the corresponding steps.

API key
For deployments based on Elastic Stack version 8.10 or later, you can use an API key to authenticate and authorize cross-cluster operations to a remote cluster. This model offers administrators of both the local and the remote deployment fine-grained access controls.
TLS certificate
This model uses mutual TLS authentication for cross-cluster operations. User authentication is performed on the local cluster and a user’s role names are passed to the remote cluster. A superuser on the local deployment gains total read access to the remote deployment, so it is only suitable for deployments that are in the same security domain.
Specify the deployments trusted to be used as remote clusters
edit

A deployment can be configured to trust all or specific deployments in any environment:

  1. From the Security menu, select Remote Connections > Add trusted environment and choose Self-managed, then click Next.
  2. Select Certificates as authentication mechanism and click Next.
  3. Upload the public certificate for the Certificate Authority of the self-managed environment (the one used to sign all the cluster certificates). The certificate needs to be in PEM format and should not contain the private key. If you only have the key in p12 format, then you can create the necessary file like this: openssl pkcs12 -in elastic-stack-ca.p12 -out newfile.crt.pem -clcerts -nokeys
  4. Select the clusters to trust. There are two options here depending on the subject name of the certificates presented by the nodes in your self managed cluster:

    • Following the Elastic Cloud pattern. In Elastic Cloud, the certificates of all Elasticsearch nodes follow this convention: CN = {node_id}.node.{cluster_id}.cluster.{scope_id}. If you follow the same convention in your self-managed environment, then choose this option and you will be able to select all or specific clusters to trust.
    • If your clusters don’t follow the previous convention for the certificates subject name of your nodes, you can still specify the node name of each of the nodes that should be trusted by this deployment. (Keep in mind that following this convention will simplify the management of this cluster since otherwise this configuration will need to be updated every time the topology of your self-managed cluster changes along with the trust restriction file. For this reason, it is recommended migrating your cluster certificates to follow the previous convention).

      Trust management will not work properly in clusters without an otherName value specified, as is the case by default in an out-of-the-box Elasticsearch installation. To have the Elasticsearch certutil generate new certificates with the otherName attribute, use the file input with the cn attribute as in the example below.

  5. . Provide a name for the trusted environment. That name will appear in the trust summary of your deployment’s Security page.
  6. Select Create trust to complete the configuration.
  7. Configure the self-managed cluster to trust this deployment, so that both deployments are configured to trust each other:

    • Download the Certificate Authority used to sign the certificates of your deployment nodes (it can be found in the Security page of your deployment)
    • Trust this CA either using the setting xpack.security.transport.ssl.certificate_authorities in elasticsearch.yml or by adding it to the trust store.
  8. Generate certificates with an otherName attribute using the Elasticsearch certutil. Create a file called instances.yaml with all the details of the nodes in your on-premise cluster like below. The dns and ip settings are optional, but cn is mandatory for use with the trust_restrictions path setting in the next step. Next, run ./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 -in instances.yaml to create new certificates for all the nodes at once. You can then copy the resulting files into each node.

    instances:
      - name: "node1"
        dns: ["node1.mydomain.com"]
        ip: ["192.168.1.1"]
        cn: ["node1.node.1234567abcd.cluster.myscope.account"]
      - name: "node2"
        dns: ["node2.mydomain.com"]
        ip: ["192.168.1.2"]
        cn: ["node2.node.1234567abcd.cluster.myscope.account"]
  9. Restrict the trusted clusters to allow only the ones which your self-managed cluster should trust.

    • All the clusters in an Elastic Cloud region are signed by the same certificate authority. Therefore, adding this CA would make the self-managed cluster trust all the clusters in that region, including clusters from other organizations. This can be limited using the setting xpack.security.transport.ssl.trust_restrictions.path which points to a file that limits the certificates to trust based on their otherName-attribute.
    • For example, the following file would trust:

      • two specific clusters with the cluster IDs aaaabbbbaaaabbbb<1> and xxxxyyyyxxxxyyyy<2> from an organization with organization ID 1053523734
      • <3> any cluster from an organization with organization ID 83988631
      • <4> The nodes from its own cluster (whose certificates follow a different convention: CN = node1.example.com, CN = node2.example.com and CN = node3.example.com)
  trust.subject_name:
  - *.node.aaaabbbbaaaabbbb.cluster.1053523734.account 
  - *.node.xxxxyyyyxxxxyyyy.cluster.1053523734.account 
  - *.node.*.cluster.83988631.account 
  - node*.example.com 

Generate new node certificates for an entire cluster using the file input mode of the certutil.

Using the API

You can update a deployment using the appropriate trust settings for the elasticsearch payload.

In order to trust a cluster whose nodes present certificates with the subject names: "CN = node1.example.com", "CN = node2.example.com" and "CN = node3.example.com" in a self-managed environment, you could update the trust settings with an additional direct trust relationship like this:

{
  "trust":{
    "accounts":[
      {
         "account_id":"ec38dd0aa45f4a69909ca5c81c27138a",
         "trust_all":true
      }
    ],
    "direct": [
      {
        "type" : "generic",
        "name" : "My Self-managed environment",
        "additional_node_names" : ["node1.example.com", "node2.example.com", "node3.example.com",],
        "certificates" : [
            {
                "pem" : "-----BEGIN CERTIFICATE-----\nMIIDTzCCA...H0=\n-----END CERTIFICATE-----"
            }
         ],
         "trust_all":false
       }
    ]
  }
}

You can now connect remotely to the trusted clusters.

Connect to the remote cluster

edit

On the local cluster, add the remote cluster using Kibana or the Elasticsearch API.

Using Kibana

edit
  1. Open the Kibana main menu, and select Stack Management > Data > Remote Clusters > Add a remote cluster.
  2. Enable Manually enter proxy address and server name.
  3. Fill in the following fields:

    • Name: This cluster alias is a unique identifier that represents the connection to the remote cluster and is used to distinguish between local and remote indices.
    • Proxy address: This value can be found on the Security page of the Elasticsearch Service deployment you want to use as a remote.

      If you’re using API keys as security model, change the port into 9443.

    • Server name: This value can be found on the Security page of the Elasticsearch Service deployment you want to use as a remote.

      Remote Cluster Parameters in Deployment

      If you’re having issues establishing the connection and the remote cluster is part of an Elastic Cloud Enterprise environment with a private certificate, make sure that the proxy address and server name match with the the certificate information. For more information, refer to Administering endpoints in Elastic Cloud Enterprise.

  4. Click Next.
  5. Click Add remote cluster (you have already established trust in a previous step).

Using the Elasticsearch API

edit

To configure a deployment as a remote cluster, use the cluster update settings API. Configure the following fields:

  • mode: proxy
  • proxy_address: This value can be found on the Security page of the Elasticsearch Service deployment you want to use as a remote. Also, using the API, this value can be obtained from the Elasticsearch resource info, concatenating the field metadata.endpoint and port 9400 using a semicolon.

If you’re using API keys as security model, change the port into 9443.

  • server_name: This value can be found on the Security page of the Elasticsearch Service deployment you want to use as a remote. Also, using the API, this can be obtained from the Elasticsearch resource info field metadata.endpoint.

This is an example of the API call to _cluster/settings:

PUT /_cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "alias-for-my-remote-cluster": {
          "mode":"proxy",
          "proxy_address": "a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io:9400",
          "server_name": "a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io"
        }
      }
    }
  }
}
Stack Version above 6.7.0 and below 7.6.0

This section only applies if you’re using TLS certificates as cross-cluster security model.

When the cluster to be configured as a remote is above 6.7.0 and below 7.6.0, the remote cluster must be configured using the sniff mode with the proxy field. For each remote cluster you need to pass the following fields:

  • Proxy: This value can be found on the Security page of the deployment you want to use as a remote under the name Proxy Address. Also, using the API, this can be obtained from the elasticsearch resource info, concatenating the fields metadata.endpoint and metadata.ports.transport_passthrough using a semicolon.
  • Seeds: This field is an array that must contain only one value, which is the server name that can be found on the Security page of the {s} deployment you want to use as a remote concatenated with :1. Also, using the API, this can be obtained from the Elasticsearch resource info, concatenating the fields metadata.endpoint and 1 with a semicolon.
  • Mode: sniff (or empty, since sniff is the default value)

This is an example of the API call to _cluster/settings:

{
  "persistent": {
    "cluster": {
      "remote": {
        "my-remote-cluster-1": {
          "seeds": [
            "a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io:1"
          ],
          "proxy": "a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io:9400"
        }
      }
    }
  }
}

Using the Elasticsearch Service RESTful API

edit

This section only applies if you’re using TLS certificates as cross-cluster security model and when both clusters belong to the same organization (for other scenarios,the Elasticsearch API should be used instead):

curl -H 'Content-Type: application/json' -X PUT -H "Authorization: ApiKey $EC_API_KEY" https://api.elastic-cloud.com/api/v1/deployments/$DEPLOYMENT_ID/elasticsearch/$REF_ID/remote-clusters -d '
{
  "resources" : [
    {
      "deployment_id": "$DEPLOYMENT_ID_REMOTE",
      "elasticsearch_ref_id": "$REF_ID_REMOTE",
      "alias": "alias-your-remote",
      "skip_unavailable" : true
    }
  ]
}'
DEPLOYMENT_ID_REMOTE
The ID of your remote deployment, as shown in the Cloud UI or obtained through the API.
REF_ID_REMOTE
The unique ID of the Elasticsearch resources inside your remote deployment (you can obtain these values through the API).

Note the following when using the Elasticsearch Service RESTful API:

  1. A cluster alias must contain only letters, numbers, dashes (-), or underscores (_).
  2. To learn about skipping disconnected clusters, refer to the Elasticsearch documentation.
  3. When remote clusters are already configured for a deployment, the PUT request replaces the existing configuration with the new configuration passed. Passing an empty array of resources will remove all remote clusters.

The following API request retrieves the remote clusters configuration:

curl -X GET -H "Authorization: ApiKey $EC_API_KEY" https://api.elastic-cloud.com/api/v1/deployments/$DEPLOYMENT_ID/elasticsearch/$REF_ID/remote-clusters

The response will include just the remote clusters from the same organization in Elasticsearch Service. In order to obtain the whole list of remote clusters, use Kibana or the Elasticsearch API Elasticsearch API directly.

Configure roles and users

edit

To use a remote cluster for cross-cluster replication or cross-cluster search, you need to create user roles with remote indices privileges on the local cluster. Refer to Configure roles and users.