Enable cross-cluster search and cross-cluster replication
editEnable cross-cluster search and cross-cluster replication
editCross-cluster search (CCS) allows you to configure multiple remote clusters across different ECE environments, and enable federated search queries across all of the configured clusters.
Cross-cluster replication (CCR) allows you to replicate indices across multiple clusters regardless of where they’re located. This provides tremendous benefit in scenarios of disaster recovery or data locality.
These steps describe how to configure remote clusters. Once that’s done, you’ll be able to run CCS queries from Elasticsearch or set up CCR.
You no longer need to use the dedicated cross-cluster search template to search across clusters. You can now use any template to configure remote clusters and search across them when using stack version 6.7 or above. We’ve deprecated the cross-cluster search template. Existing deployments created using this template are not affected by this change, but they need to migrate to another template before upgrading to the next 8.0 major version.
Before you begin
editTo use CCS or CCR, your environment must meet the following criteria:
-
Deployments must be version 6.7.x (to use CCS) or later and be compatible:
- Review the Elasticsearch version compatibility table.
- System deployments can not be used as remote clusters or have remote clusters.
- To use CCR, deployments must be at version 6.8.9 or higher (in the 6.x branch), or 7.7.1 or higher (in the 7.x branch).
- Deployments must trust each other.
- Proxies must answer TCP requests on the port 9400. Check the prerequisites for the ports that must permit outbound or inbound traffic.
- Load balancers must pass-through TCP requests on port 9400. Check the configuration details.
The deprecated CCS Deployment Template supports CCS for deployments in version 6.1.x and above without the need to have proxies and load balancers answer on port 9400. However, these remote clusters must be in the same ECE environment and they do not support Cross-cluster Replication. Read more about migrating your deployments using the CCS Deployment Template.
Configure remote clusters using Kibana
editYou can configure remote clusters in Kibana starting with version 7.7.0 (for any deployment except deployments using the deprecated CCS deployment template):
-
Open the Stack Management section and select Remote Clusters from the side navigation.
-
To add a new remote cluster, fill in the following fields:
- Name: this is the alias you will use to run CCS queries.
- Proxy Address: This value can be found on the Security page of the deployment you want to use as a remote.
- Server Name: This value can be found on the Security page of the deployment you want to use as a remote.
This mechanism will work for remote clusters in other ECE environments as long as they trust each other.
This configuration of remote clusters uses the Proxy mode and it requires that the allocators can communicate via http with the proxies.
Configure remote clusters in the same ECE environment using the Cloud RESTful API
editThe following PUT request configures a remote cluster from the same ECE environment:
curl -k -H 'Content-Type: application/json' -X PUT -H "Authorization: ApiKey $ECE_API_KEY" https://COORDINATOR_HOST:12443/api/v1/deployments/$DEPLOYMENT_ID/elasticsearch/$REF_ID/remote-clusters -d '
{
"resources" : [
{
"deployment_id": "$DEPLOYMENT_ID_REMOTE",
"elasticsearch_ref_id": "$REF_ID_REMOTE",
"alias": "alias-your-remote",
"skip_unavailable" : true
}
]
}'
-
DEPLOYMENT_ID_REMOTE - The ID of your remote deployment, as shown in the Cloud UI or obtained through the API.
-
REF_ID_REMOTE - The unique ID of the Elasticsearch resources inside your remote deployment (you can obtain these values through the API).
Note the following when using the API:
- A cluster alias must contain only letters, numbers, dashes (-), or underscores (_).
- To learn about skipping disconnected clusters, refer to the Elasticsearch documentation.
-
When remote clusters are already configured for a deployment, the
PUTrequest replaces the existing configuration with the new configuration passed. Passing an empty array of resources will remove all remote clusters.
The following API request retrieves the remote clusters configuration:
curl -k -X GET -H "Authorization: ApiKey $ECE_API_KEY" https://COORDINATOR_HOST:12443/api/v1/deployments/$DEPLOYMENT_ID/elasticsearch/$REF_ID/remote-clusters
Configure remote clusters in other ECE environment using the Elasticsearch API
editUsage of this API is incompatible with the CCS deployment template. Use the remote Cloud API instead if you are using this template.
Elasticsearch provides an API to manage remote clusters. This API supports two different modes to configure the remote clusters, and depending on the version of the remote cluster you need to use the following parameters:
Stack Version 7.6.0 and above
editRemote clusters must be configured using proxy mode. For each remote cluster you need to pass the following fields:
-
Proxy address: This value can be found on the Security page of the deployment you want to use as a remote. Also, using the API, this value can be obtained from the elasticsearch resource info, concatenating the fields
metadata.endpointandmetadata.ports.transport_passthroughusing a semicolon. -
Server name: This value can be found on the Security page of the deployment you want to use as a remote. Also, using the API, this can be obtained from the Elasticsearch resource info field
metadata.endpoint.
This is an example of the API call to _cluster/settings:
{
"persistent": {
"cluster": {
"remote": {
"my-remote-cluster-1": {
"mode":"proxy",
"proxy_address": "a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io:9400",
"server_name": "a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io"
}
}
}
}
}
Stack Version above 6.7.0 and below 7.6.0
editRemote clusters must be configured using sniff mode with the proxy field. For each remote cluster you need to pass the following fields:
-
Proxy: This value can be found on the Security page of the deployment you want to use as a remote under the name
Proxy Address. Also, using the API, this can be obtained from the elasticsearch resource info, concatenating the fieldsmetadata.endpointandmetadata.ports.transport_passthroughusing a semicolon. -
Seeds: This field is an array that must contain only one value, which is the
server namethat can be found on the Security page of the deployment you want to use as a remote concatenated with:1. Also, using the API, this can be obtained from the Elasticsearch resource info, concatenating the fieldsmetadata.endpointand1with a semicolon.
This is an example of the API call to _cluster/settings:
{
"persistent": {
"cluster": {
"remote": {
"my-remote-cluster-1": {
"seeds": [
"a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io:1"
],
"proxy": "a542184a7a7d45b88b83f95392f450ab.192.168.44.10.ip.es.io:9400"
}
}
}
}
}
Configure remote clusters using the CCS deployment template (deprecated)
editElastic Cloud Enterprise currently includes a Cross Cluster Search deployment template, which is now deprecated and will be removed in ECE version 3.0.
This deployment template allows you to configure remote clusters for deployments with Elastic stack version 6.1.x or later. However, it does not support remote clusters in other remote ECE environments and it does not support Cross Cluster Replication.
The Cross-cluster search deployment template is not supported for AWS Marketplace accounts.
This deployment template configures remote clusters using sniff mode, which stores in the cluster settings the IP address of the nodes of the remote clusters. This deployment template automatically takes care of that configuration, and the cluster settings are automatically maintained and updated when cluster topologies change. For example, all connected clusters are updated when:
- A cluster is deleted.
- A node is added or removed from a cluster.
The Cross-cluster search deployment acts as a hub that manages all of the remote deployment connections.
- Log into the Cloud UI.
- Select Create deployment.
- Select the Elastic Stack solution
- Select the Cross Cluster Search hardware profile.
-
Decide when to add the remote connections:
- If you know the deployments that you want to include, select Customize deployment to define the remotes while creating the deployment.
- After the deployment is spun up, you can add remotes from the deployment overview page.
- Select Add deployment.
-
Select a target deployment from the Deployment name drop-down list.
Only the compatible deployments appear on this list.
- Optional: Update the generated alias to something meaningful to you.
- Save your changes.
- Verify the configuration by running CCS queries from Elasticsearch.
- As needed, from the Elasticsearch API or Kibana, update the Elasticsearch cluster security and user access roles to allow them to access the remote deployments.
The target deployment overview page shows which CCS deployments can search the target.
Deployments using this template are also compatible with the remotes Cloud API, so you can use the API too to manage the remote clusters.
Enable CCR and the Remote Clusters UI in Kibana
editIf your deployment was created before ECE version 2.9.0, CCR won’t be enabled and you won’t find the Remote Clusters UI in Kibana even though your deployment meets all the criteria.
To enable these features, go to the Security page of your deployment and under Trust management select Enable CCR.
CCR is not supported for indices used by Enterprise Search.
Remote Clusters and Traffic Filtering
editTraffic filtering can be enabled to restrict access to deployments that are used as a local or remote cluster without any impact to cross-cluster search or cross-cluster replication.