Bootstrapping CCR with remote cluster connections

This walkthrough establishes the persistent cluster.remote.<alias>.seeds connection between a follower and a leader cluster and proves it is genuinely open — resolving connectivity, transport-port, and role prerequisites — before any follower index replication is attempted.

The remote connection is the wire everything else in Cross-Cluster Replication rides on. Skip its verification and you push the failure downstream: _start accepts your call, the follower sits in BOOTSTRAPPING, and only minutes later does _status reveal the real problem was an unreachable seed or a node missing remote_cluster_client. This page is the connectivity foundation for CCR Setup and Bootstrap; once the connection here reports connected: true, you move on to Starting follower index replication from a leader.

Prerequisites #

Confirm each item on the follower cluster before registering the remote — a connection that appears to save but never opens is almost always one of these.

• The follower nodes that will proxy replication carry the remote_cluster_client role in opensearch.yml (node.roles: [ data, remote_cluster_client ]), per Node Role Allocation.
• The leader’s transport port 9300 (not the REST port 9200) is reachable from every follower node, verified with a raw nc -vz <leader-host> 9300.
• The leader and follower share a trusted transport TLS chain — CCR transport connections require mutual TLS, and a CA mismatch surfaces as a silent connection failure.
• The leader_read_role and follower_write_role exist with identical names on both clusters, scoped as in Security & Access Boundaries.
• You have decided between sniff mode (leader nodes directly routable) and proxy mode (traffic funneled through one reachable address) using the decision flow below.
• The replication user can write _cluster/settings on the follower (cluster:admin/settings/update).

Choosing sniff versus proxy mode #

The connection mode determines how the follower discovers and reaches leader nodes. sniff mode connects to the seeds, then learns and connects directly to the rest of the leader’s eligible nodes — ideal when the two clusters share a routable network. proxy mode sends every connection through a single proxy_address, which is what you need across NAT, a load balancer, or a region boundary where individual leader node IPs are not reachable.

Step-by-step procedure #

1. Register the remote connection #

Register the leader under a stable alias — this alias becomes the leader_alias in every replication call, so pick a name that survives the life of the topology (leader-use1, not temp). Point seeds at transport port 9300.

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.leader-use1.seeds": [
      "10.20.0.11:9300",
      "10.20.0.12:9300"
    ],
    "cluster.remote.leader-use1.transport.ping_schedule": "30s",
    "cluster.remote.leader-use1.transport.compress": true
  }
}

Use persistent, never transient — a transient remote setting is dropped on the next cluster-state recovery and silently kills replication.

{ "acknowledged": true, "persistent": { "cluster": { "remote": { "leader-use1": { ... } } } } }

Gotcha: an acknowledged: true here means the setting was stored, not that the connection opened. The two are independent — verification in step 4 is the only proof the wire is live.

2. Switch to proxy mode when nodes are not routable #

If the decision flow sent you down the proxy path, add the mode and a single reachable address instead of relying on node sniffing. In proxy mode the seeds are ignored in favor of proxy_address.

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.leader-use1.mode": "proxy",
    "cluster.remote.leader-use1.proxy_address": "ccr-proxy.leader.internal:9300",
    "cluster.remote.leader-use1.server_name": "ccr-proxy.leader.internal"
  }
}

Gotcha: server_name must match the SNI on the leader’s transport certificate when traffic crosses a TLS-terminating proxy, or the mutual-TLS handshake fails before any replication logic runs.

3. Confirm transport reachability and role placement #

Before trusting the OpenSearch-level status, prove the raw socket and the node role. A blocked 9300 is the most common root cause and is invisible in the REST layer.

# Raw transport reachability from a follower node
nc -vz 10.20.0.11 9300

# Which follower nodes actually carry remote_cluster_client
curl -s "https://<follower>:9200/_cat/nodes?v&h=name,node.role"

Connection to 10.20.0.11 9300 port [tcp/*] succeeded!
name        node.role
follower-a  dimr          # 'r' = remote_cluster_client present
follower-b  dim           # missing 'r' -> cannot proxy replication

Gotcha: a node without the r role can still hold follower shards, but it cannot open the remote channel — pin the replication client calls to a node that carries the role, or add the role fleet-wide.

4. Verify the connection is open #

GET _remote/info is the authoritative check. It reports connected, the negotiated number of nodes, and the mode. Only a connected: true here means you may proceed to start followers.

curl -s "https://<follower>:9200/_remote/info?pretty"

{
  "leader-use1": {
    "connected": true,
    "mode": "sniff",
    "seeds": ["10.20.0.11:9300", "10.20.0.12:9300"],
    "num_nodes_connected": 3,
    "max_connections_per_cluster": 3,
    "initial_connect_timeout": "30s"
  }
}

Read this response critically rather than glancing at the boolean. num_nodes_connected should equal the number of eligible leader nodes (bounded by max_connections_per_cluster); a connected: true with num_nodes_connected well below your leader’s node count means the follower reached the seeds but could not open channels to the rest of the leader cluster — typically a firewall that allows the seed hosts but not the sniffed nodes behind them. In proxy mode, expect num_nodes_connected: 1, because every connection funnels through the single proxy_address; that is correct, not a shortfall. The connection is negotiated lazily and re-established automatically after a transient network drop, which is exactly why it must be persistent: the recovery logic depends on the setting surviving in cluster state.

Gotcha: _remote/info reflects the view of the node you queried. On a large follower cluster, poll it against the specific node that will issue your replication calls — a coordinating node without remote_cluster_client can report a stale or absent connection even when data nodes are connected fine.

Verification #

Run these two follow-up checks to confirm the connection is not just open but usable for replication — connectivity and authorization are separate gates.

# 1. Connection health and node count (must be connected: true)
curl -s "https://<follower>:9200/_remote/info?pretty" | grep -E 'connected|num_nodes'
# "connected" : true
# "num_nodes_connected" : 3

# 2. Prove the replication roles resolve on the leader side by validating
#    a setup against a real leader index (dry authorization check)
curl -s -X POST "https://<follower>:9200/_plugins/_replication/_autofollow?pretty" \
  -H "Content-Type: application/json" \
  -d '{"leader_alias":"leader-use1","name":"probe","pattern":"__none-*",
       "use_roles":{"leader_cluster_role":"leader_read_role",
                    "follower_cluster_role":"follower_write_role"}}'
# acknowledged: true  -> roles resolved on both clusters (delete the probe rule after)

A connected: true with num_nodes_connected matching your leader’s eligible node count, plus an accepted role probe, means the wire is fully ready. Remove the throwaway probe auto-follow rule once it acknowledges.

Common failures #

Frequently asked questions #

Related #

• CCR Setup and Bootstrap — the full leader–follower bootstrap sequence this connection is the first step of.
• Starting follower index replication from a leader — the next step once _remote/info reports connected: true.
• Security & Access Boundaries — defining the leader-read and follower-write roles the connection carries.

Up: CCR Setup and Bootstrap