IMPORTANT: No additional bug fixes or documentation updates will be released for this version. For the latest information, see the current release documentation.

« Retrieve selected fields from a search Search multiple data streams and indices »

› ›

Search across clusters

edit

Search across clusters

edit

Cross-cluster search lets you run a single search request against one or more remote clusters. For example, you can use a cross-cluster search to filter and analyze log data stored on clusters in different data centers.

Supported APIs

edit

The following APIs support cross-cluster search:

Search
Async search
Multi search
Search template
Multi search template
Field capabilities
[preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. EQL search
[preview] This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. SQL search
Vector tile search

Prerequisites

edit

Cross-cluster search requires remote clusters. To set up remote clusters on Elasticsearch Service, see configure remote clusters on Elasticsearch Service. If you run Elasticsearch on your own hardware, see Remote clusters.

To ensure your remote cluster configuration supports cross-cluster search, see Supported configurations.
The local coordinating node must have the remote_cluster_client node role.

If you use sniff mode, the local coordinating node must be able to connect to seed and gateway nodes on the remote cluster.

We recommend using gateway nodes capable of serving as coordinating nodes. The seed nodes can be a subset of these gateway nodes.

If you use proxy mode, the local coordinating node must be able to connect to the configured proxy_address. The proxy at this address must be able to route connections to gateway and coordinating nodes on the remote cluster.
Cross-cluster search requires different security privileges on the local cluster and remote cluster. See Configure privileges for cross-cluster search and Configure privileges for cross-cluster search and Kibana.

Cross-cluster search examples

edit

Remote cluster setup

edit

The following cluster update settings API request adds three remote clusters: cluster_one, cluster_two, and cluster_three.

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "cluster_one": {
          "seeds": [
            "127.0.0.1:9300"
          ]
        },
        "cluster_two": {
          "seeds": [
            "127.0.0.1:9301"
          ]
        },
        "cluster_three": {
          "seeds": [
            "127.0.0.1:9302"
          ]
        }
      }
    }
  }
}

Search a single remote cluster

edit

In the search request, you specify data streams and indices on a remote cluster as <remote_cluster_name>:<target>`.

The following search API request searches the my-index-000001 index on a single remote cluster, cluster_one.

GET /cluster_one:my-index-000001/_search
{
  "query": {
    "match": {
      "user.id": "kimchy"
    }
  },
  "_source": ["user.id", "message", "http.response.status_code"]
}

The API returns the following response:

{
  "took": 150,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0,
    "skipped": 0
  },
  "_clusters": {
    "total": 1,
    "successful": 1,
    "skipped": 0
  },
  "hits": {
    "total" : {
        "value": 1,
        "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "cluster_one:my-index-000001", 
        "_type": "_doc",
        "_id": "0",
        "_score": 1,
        "_source": {
          "user": {
            "id": "kimchy"
          },
          "message": "GET /search HTTP/1.1 200 1070000",
          "http": {
            "response":
              {
                "status_code": 200
              }
          }
        }
      }
    ]
  }
}

The search response body includes the name of the remote cluster in the _index parameter.

Search multiple remote clusters

edit

The following search API request searches the my-index-000001 index on three clusters:

Your local cluster
Two remote clusters, cluster_one and cluster_two

GET /my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001/_search
{
  "query": {
    "match": {
      "user.id": "kimchy"
    }
  },
  "_source": ["user.id", "message", "http.response.status_code"]
}

The API returns the following response:

{
  "took": 150,
  "timed_out": false,
  "num_reduce_phases": 4,
  "_shards": {
    "total": 3,
    "successful": 3,
    "failed": 0,
    "skipped": 0
  },
  "_clusters": {
    "total": 3,
    "successful": 3,
    "skipped": 0
  },
  "hits": {
    "total" : {
        "value": 3,
        "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "my-index-000001", 
        "_type": "_doc",
        "_id": "0",
        "_score": 2,
        "_source": {
          "user": {
            "id": "kimchy"
          },
          "message": "GET /search HTTP/1.1 200 1070000",
          "http": {
            "response":
              {
                "status_code": 200
              }
          }
        }
      },
      {
        "_index": "cluster_one:my-index-000001", 
        "_type": "_doc",
        "_id": "0",
        "_score": 1,
        "_source": {
          "user": {
            "id": "kimchy"
          },
          "message": "GET /search HTTP/1.1 200 1070000",
          "http": {
            "response":
              {
                "status_code": 200
              }
          }
        }
      },
      {
        "_index": "cluster_two:my-index-000001", 
        "_type": "_doc",
        "_id": "0",
        "_score": 1,
        "_source": {
          "user": {
            "id": "kimchy"
          },
          "message": "GET /search HTTP/1.1 200 1070000",
          "http": {
            "response":
              {
                "status_code": 200
              }
          }
        }
      }
    ]
  }
}

	This document’s `_index` parameter doesn’t include a cluster name. This means the document came from the local cluster.
	This document came from `cluster_one`.
	This document came from `cluster_two`.

Skip unavailable clusters

edit

By default, a cross-cluster search returns an error if any cluster in the request is unavailable.

To skip an unavailable cluster during a cross-cluster search, set the skip_unavailable cluster setting to true.

The following cluster update settings API request changes cluster_two's skip_unavailable setting to true.

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.cluster_two.skip_unavailable": true
  }
}

If cluster_two is disconnected or unavailable during a cross-cluster search, Elasticsearch won’t include matching documents from that cluster in the final results.

How cross-cluster search handles network delays

edit

Because cross-cluster search involves sending requests to remote clusters, any network delays can impact search speed. To avoid slow searches, cross-cluster search offers two options for handling network delays:

Minimize network roundtrips

By default, Elasticsearch reduces the number of network roundtrips between remote clusters. This reduces the impact of network delays on search speed. However, Elasticsearch can’t reduce network roundtrips for large search requests, such as those including a scroll or inner hits.

See Minimize network roundtrips to learn how this option works.

Don’t minimize network roundtrips

For search requests that include a scroll or inner hits, Elasticsearch sends multiple outgoing and ingoing requests to each remote cluster. You can also choose this option by setting the ccs_minimize_roundtrips parameter to false. While typically slower, this approach may work well for networks with low latency.

See Don’t minimize network roundtrips to learn how this option works.

The vector tile search API always minimizes network roundtrips and doesn’t include the ccs_minimize_roundtrips parameter.

Minimize network roundtrips

edit

Here’s how cross-cluster search works when you minimize network roundtrips.

You send a cross-cluster search request to your local cluster. A coordinating node in that cluster receives and parses the request.
The coordinating node sends a single search request to each cluster, including the local cluster. Each cluster performs the search request independently, applying its own cluster-level settings to the request.
Each remote cluster sends its search results back to the coordinating node.
After collecting results from each cluster, the coordinating node returns the final results in the cross-cluster search response.

Don’t minimize network roundtrips

edit

Here’s how cross-cluster search works when you don’t minimize network roundtrips.

You send a cross-cluster search request to your local cluster. A coordinating node in that cluster receives and parses the request.
The coordinating node sends a search shards API request to each remote cluster.
Each remote cluster sends its response back to the coordinating node. This response contains information about the indices and shards the cross-cluster search request will be executed on.
The coordinating node sends a search request to each shard, including those in its own cluster. Each shard performs the search request independently.

When network roundtrips aren’t minimized, the search is executed as if all data were in the coordinating node’s cluster. We recommend updating cluster-level settings that limit searches, such as action.search.shard_count.limit, pre_filter_shard_size, and max_concurrent_shard_requests, to account for this. If these limits are too low, the search may be rejected.
Each shard sends its search results back to the coordinating node.
After collecting results from each cluster, the coordinating node returns the final results in the cross-cluster search response.

Supported configurations

edit

To run a cross-cluster search, the local and remote clusters must be compatible as outlined in the following matrix.

For the EQL search API, the local and remote clusters must use the same Elasticsearch version.

Version compatibility matrix

	Local cluster
Remote cluster	5.0–5.5	5.6	6.0–6.6	6.7	6.8	7.0	7.1–7.16
5.0–5.5
5.6
6.0–6.6
6.7
6.8
7.0
7.1–7.16

Cross-cluster search can also search remote clusters that are being upgraded so long as both the "upgrade from" and "upgrade to" version are compatible with the gateway node.

For example, a coordinating node running Elasticsearch 5.6 can search a remote cluster running Elasticsearch 6.8, but that cluster can not be upgraded to 7.1. In this case you should first upgrade the coordinating node to 7.1 and then upgrade remote cluster.

Running multiple versions of Elasticsearch in the same cluster beyond the duration of an upgrade is not supported.

Only features that exist across all searched clusters are supported. Using a recent feature with a remote cluster where the feature is not supported will result in undefined behavior.

« Retrieve selected fields from a search Search multiple data streams and indices »