Access other deployments of the same Elasticsearch Service organization

edit

This section explains how to configure a deployment to connect remotely to clusters belonging to the same Elasticsearch Service organization.

Allow the remote connection

edit

Before you start, consider the security model that you would prefer to use for authenticating remote connections between clusters, and follow the corresponding steps.

API key
For deployments based on Elastic Stack version 8.10 or later, you can use an API key to authenticate and authorize cross-cluster operations to a remote cluster. This model offers administrators of both the local and the remote deployment fine-grained access controls.
TLS certificate
This model uses mutual TLS authentication for cross-cluster operations. User authentication is performed on the local cluster and a user’s role names are passed to the remote cluster. A superuser on the local deployment gains total read access to the remote deployment, so it is only suitable for deployments that are in the same security domain.
Set the default trust with other clusters in the same Elasticsearch Service organization
edit

By default, any deployment that you create trusts all other deployments in the same organization. You can change this behavior in the Elasticsearch Service Console under Features > Trust, so that when a new deployment is created it does not automatically trust any other deployment. You can choose one of the following options:

  • Trust all my deployments - All of your organization’s deployments created while this option is selected already trust each other. If you keep this option, that includes any deployments you’ll create in the future. You can directly jump to Connect to the remote cluster to finalize the CCS or CCR configuration.
  • Trust no deployment - New deployments won’t trust any other deployment when they are created. You can instead configure trust individually for each of them in their security settings, as described in the next section.
Trust management at the account Level
  • The level of trust of existing deployments is not modified when you change this setting. You must instead update the trust settings individually for each deployment you wish to change.
  • Deployments created before the Elasticsearch Service February 2021 release trust only themselves. You have to update the trust setting for each deployment that you want to either use as a remote cluster or configure to work with a remote cluster.
Specify the deployments trusted to be used as remote clusters
edit

If your organization’s deployments already trust each other by default, you can skip this section. If that’s not the case, follow these steps to configure which are the specific deployments that should be trusted.

  1. Go to the Security page of your deployment.
  2. From the list of existing trust configurations, edit the one labeled as your organization.
  3. Choose one of following options to configure the level of trust on each of your deployments:

    • Trust all deployments - This deployment trusts all other deployments in this environment, including new deployments when they are created.
    • Trust specific deployments - Choose which of the existing deployments from your environment you want to trust.
    • Trust no deployment - No deployment in this Elasticsearch Service environment is trusted.

When trusting specific deployments, the more restrictive CCS version policy is used (even if you only want to use CCR). To work around this restriction for CCR-only trust, it is necessary to use the API as described below.

  1. Repeat these steps from each of the deployments you want to use for CCS or CCR. You will only be able to connect 2 deployments successfully when both of them trust each other.
Using the API

You can update a deployment using the appropriate trust settings for the elasticsearch payload.

The current trust settings can be found in the path .resources.elasticsearch[0].info.settings.trust when calling:

curl -k -X GET -H "Authorization: ApiKey $ECE_API_KEY" https://COORDINATOR_HOST:12443/api/v1/deployments/$DEPLOYMENT_ID?show_settings=true

For example:

{
  "accounts": [
    {
      "account_id": "ec38dd0aa45f4a69909ca5c81c27138a",
      "trust_all": true
    }
  ]
}

The account_id above represents the only account in an {s} environment, and therefore is the one used to update the trust level with deployments in the current {s} environment. For example, to update the trust level to trust only the deployment with cluster ID cf659f7fe6164d9691b284ae36811be1 (NOTE: use the Elasticsearch cluster ID, not the deployment ID), the trust settings in the body would look like this:

{
  "trust":{
    "accounts":[
      {
         "account_id":"ec38dd0aa45f4a69909ca5c81c27138a",
         "trust_all":false,
         "trust_allowlist":[
            "cf659f7fe6164d9691b284ae36811be1"
         ]
      }
    ]
  }
}

You can now connect remotely to the trusted clusters.

Connect to the remote cluster

edit

On the local cluster, add the remote cluster using Kibana or the Elasticsearch API.

Using Kibana

edit
  1. Open the Kibana main menu, and select Stack Management > Data > Remote Clusters > Add a remote cluster.
  2. Enable Manually enter proxy address and server name.
  3. Fill in the following fields:

    • Name: This cluster alias is a unique identifier that represents the connection to the remote cluster and is used to distinguish between local and remote indices.
    • Proxy address: This value can be found on the Security page of the Elasticsearch Service deployment you want to use as a remote.

      If you’re using API keys as security model, change the port into 9443.

    • Server name: This value can be found on the Security page of the Elasticsearch Service deployment you want to use as a remote.

      Remote Cluster Parameters in Deployment

      If you’re having issues establishing the connection and the remote cluster is part of an Elastic Cloud Enterprise environment with a private certificate, make sure that the proxy address and server name match with the the certificate information. For more information, refer to Administering endpoints in Elastic Cloud Enterprise.

  4. Click Next.
  5. Click Add remote cluster (you have already established trust in a previous step).

Using the Elasticsearch API

edit

To configure a deployment as a remote cluster, use the cluster update settings API. Configure the following fields:

  • mode: proxy
  • proxy_address: This value can be found on the Security page of the Elasticsearch Service deployment you want to use as a remote. Also, using the API, this value can be obtained from the Elasticsearch resource info, concatenating the field metadata.endpoint and port 9400 using a semicolon.

If you’re using API keys as security model, change the port into 9443.

  • server_name: This value can be found on the Security page of the Elasticsearch Service deployment you want to use as a remote. Also, using the API, this can be obtained from the Elasticsearch resource info field metadata.endpoint.

This is an example of the API call to _cluster/settings:

PUT /_cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "alias-for-my-remote-cluster": {
          "mode":"proxy",
          "proxy_address": "a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io:9400",
          "server_name": "a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io"
        }
      }
    }
  }
}
Stack Version above 6.7.0 and below 7.6.0

This section only applies if you’re using TLS certificates as cross-cluster security model.

When the cluster to be configured as a remote is above 6.7.0 and below 7.6.0, the remote cluster must be configured using the sniff mode with the proxy field. For each remote cluster you need to pass the following fields:

  • Proxy: This value can be found on the Security page of the deployment you want to use as a remote under the name Proxy Address. Also, using the API, this can be obtained from the elasticsearch resource info, concatenating the fields metadata.endpoint and metadata.ports.transport_passthrough using a semicolon.
  • Seeds: This field is an array that must contain only one value, which is the server name that can be found on the Security page of the {s} deployment you want to use as a remote concatenated with :1. Also, using the API, this can be obtained from the Elasticsearch resource info, concatenating the fields metadata.endpoint and 1 with a semicolon.
  • Mode: sniff (or empty, since sniff is the default value)

This is an example of the API call to _cluster/settings:

{
  "persistent": {
    "cluster": {
      "remote": {
        "my-remote-cluster-1": {
          "seeds": [
            "a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io:1"
          ],
          "proxy": "a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io:9400"
        }
      }
    }
  }
}

Using the Elasticsearch Service RESTful API

edit

This section only applies if you’re using TLS certificates as cross-cluster security model and when both clusters belong to the same organization (for other scenarios,the Elasticsearch API should be used instead):

curl -H 'Content-Type: application/json' -X PUT -H "Authorization: ApiKey $EC_API_KEY" https://api.elastic-cloud.com/api/v1/deployments/$DEPLOYMENT_ID/elasticsearch/$REF_ID/remote-clusters -d '
{
  "resources" : [
    {
      "deployment_id": "$DEPLOYMENT_ID_REMOTE",
      "elasticsearch_ref_id": "$REF_ID_REMOTE",
      "alias": "alias-your-remote",
      "skip_unavailable" : true
    }
  ]
}'
DEPLOYMENT_ID_REMOTE
The ID of your remote deployment, as shown in the Cloud UI or obtained through the API.
REF_ID_REMOTE
The unique ID of the Elasticsearch resources inside your remote deployment (you can obtain these values through the API).

Note the following when using the Elasticsearch Service RESTful API:

  1. A cluster alias must contain only letters, numbers, dashes (-), or underscores (_).
  2. To learn about skipping disconnected clusters, refer to the Elasticsearch documentation.
  3. When remote clusters are already configured for a deployment, the PUT request replaces the existing configuration with the new configuration passed. Passing an empty array of resources will remove all remote clusters.

The following API request retrieves the remote clusters configuration:

curl -X GET -H "Authorization: ApiKey $EC_API_KEY" https://api.elastic-cloud.com/api/v1/deployments/$DEPLOYMENT_ID/elasticsearch/$REF_ID/remote-clusters

The response will include just the remote clusters from the same organization in Elasticsearch Service. In order to obtain the whole list of remote clusters, use Kibana or the Elasticsearch API Elasticsearch API directly.

Configure roles and users

edit

To use a remote cluster for cross-cluster replication or cross-cluster search, you need to create user roles with remote indices privileges on the local cluster. Refer to Configure roles and users.