CCR Setup and Bootstrap

Cross-Cluster Replication (CCR) setup and bootstrap is the sequence that turns two independent OpenSearch clusters into a leader–follower pair: you register a persistent remote connection, scope the security roles that carry replication authority across the wire, start replication on a follower index, and watch it walk from an empty shell through a leader snapshot restore into a live translog tail. Get any step out of order — a missing seed, an under-scoped role, a follower that is not actually read-only — and replication either refuses to start or bootstraps once and then silently stops shipping operations. This page is the operational core of Cross-Cluster Replication Operations; it treats bootstrap as a state machine you can observe and recover, not a fire-and-forget API call.

CCR exists so a second cluster in another region or failure domain holds a byte-for-byte copy of your leader indices, ready to take reads immediately and writes after a promotion. Because that copy is produced by replaying the leader’s operations rather than re-indexing, the follower is deliberately write-blocked, and the bootstrap phase — a one-time remote snapshot restore before the translog tail begins — is where most setup failures surface. The two deep procedures branch off from here: establishing the wire itself in Bootstrapping CCR with remote cluster connections, and starting per-index replication in Starting follower index replication from a leader.

Leader and follower cluster alignment #

CCR is not symmetric. The leader keeps serving production writes at full ingest rate while the follower spends its I/O budget replaying operations and holding a searchable replica. The follower does not need to match the leader hardware-for-hardware, but it does need enough network throughput to keep replication lag inside your recovery objective and enough soft-delete retention on the leader side to avoid a full remote resync after a blip. The table aligns each role with the profile that keeps bootstrap and steady-state replication healthy; the node-attribute mechanics behind the “routing” column are the same ones described in Node Role Allocation.

The single non-negotiable attribute is remote_cluster_client: every node that either issues the _start call or holds a shard of a follower index must carry that role, or the replication engine cannot open a transport channel to the leader. The follower’s write-block is not a tuning choice — it is enforced by the plugin, and it is why ISM lifecycle actions on a follower behave differently from a leader, a distinction developed in the follower-policy pages. Scope the roles that cross the wire before you touch any index, following Security & Access Boundaries; an over-broad replication role is the most common audit finding in a CCR rollout.

What CCR replicates — and what it does not #

Before wiring anything up, be precise about the replication boundary, because misjudging it is the root of most “why didn’t my change propagate” incidents. CCR replicates the data-plane operations of a leader index: every indexed, updated, and deleted document arrives on the follower in leader order, reconstructed by replaying the leader’s translog against the restored shard set. It also carries index mapping and most index-level setting changes, synced on the metadata_sync_interval cadence, so a mapping field added on the leader shows up on the follower without intervention.

What CCR does not replicate is just as important. It does not copy cluster-level state — templates, ISM policies, ingest pipelines, roles, and _cluster/settings all live per-cluster and must be provisioned on the follower independently. It does not replicate the lifecycle actions themselves: a rollover on the leader creates a new leader backing index, and unless an auto-follow rule matches that new index, the follower will not begin replicating it. And it never replicates writes from the follower, because the follower cannot take any. This split is why a CCR topology is really two coordinated deployments: the data flows one way over the wire, while the control plane — policies, templates, security — is deployed to both sides by your own automation. Keeping the two in step is the discipline that separates a resilient standby from a follower that silently diverges the first time someone edits a template on only one cluster.

The bootstrap state machine #

Replication for a single follower index moves through a small, observable set of states. GET _plugins/_replication/<follower>/_status reports the current one, and every setup failure lands the index in FAILED with a reason you can act on. Understanding the transitions is what separates “wait longer” from “stop and fix a role” when a follower stalls.

INIT is transient — you rarely observe it — and exists only while the follower registers the leader shard set. BOOTSTRAPPING is the expensive phase: the follower pulls a remote snapshot of each leader shard, so on a large index it can run for minutes and consume peering bandwidth. SYNCING means the restore finished and the follower is replaying the leader translog to close the gap; a healthy follower spends almost all of its life oscillating between SYNCING and SYNCED as new operations arrive. FAILED is terminal until you intervene: the plugin does not auto-retry a role or block error, because retrying a misconfigured _start just fails again. The corrective action is always to fix the underlying cause and re-issue _start, which walks the index back through BOOTSTRAPPING.

1. Register the remote cluster connection #

Replication needs a persistent remote connection — a transient one evaporates on the next elected-master change and takes replication down with it. Register the leader on the follower cluster under a stable alias; that alias is the leader_alias every _start call references. Point seeds at the leader’s transport port (9300), not the REST port (9200).

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.leader-use1.seeds": [
      "10.20.0.11:9300",
      "10.20.0.12:9300"
    ],
    "cluster.remote.leader-use1.transport.ping_schedule": "30s"
  }
}

When the leader nodes are not directly routable from the follower — a common cross-region or NAT situation — use proxy mode instead of sniff, so all traffic flows through a single reachable address:

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.leader-use1.mode": "proxy",
    "cluster.remote.leader-use1.proxy_address": "ccr-proxy.leader.internal:9300"
  }
}

The full connectivity-and-verification walkthrough — including how to confirm the connection actually opened before you rely on it — is in Bootstrapping CCR with remote cluster connections.

2. Scope roles and replication security #

CCR carries the caller’s authority across the wire through use_roles: the replication engine executes leader reads and follower writes under two named roles you specify at start time, not under the human who ran the command. Define a tight leader-read role and a follower-write role, and grant a dedicated replication user only those two roles plus cluster:admin/plugins/replication/*.

{
  "leader_read_role": {
    "cluster_permissions": [
      "cluster:monitor/state",
      "indices:admin/plugins/replication/index/setup/validate"
    ],
    "index_permissions": [
      {
        "index_patterns": ["logs-*"],
        "allowed_actions": [
          "indices:data/read/plugins/replication/changes",
          "indices:admin/plugins/replication/index/setup/validate"
        ]
      }
    ]
  }
}

{
  "follower_write_role": {
    "index_permissions": [
      {
        "index_patterns": ["logs-*"],
        "allowed_actions": [
          "indices:data/write/plugins/replication/changes",
          "indices:admin/plugins/replication/index/setup/validate"
        ]
      }
    ]
  }
}

Both roles must exist on both clusters with identical names — the follower resolves use_roles against the leader’s role registry, and a missing role on either side fails the start with security_exception. Keep the replication user out of your interactive admin group; scope it exactly as Security & Access Boundaries prescribes so the managed-index and replication audit trail stays clean.

3. Start replication on the follower #

With the wire up and roles in place, start replication for a single follower index. The follower index must not exist beforehand — CCR creates it during BOOTSTRAPPING. Passing use_roles binds the leader-read and follower-write identities to this replication task.

PUT _plugins/_replication/logs-000001/_start
{
  "leader_alias": "leader-use1",
  "leader_index": "logs-000001",
  "use_roles": {
    "leader_cluster_role": "leader_read_role",
    "follower_cluster_role": "follower_write_role"
  }
}

For many indices sharing a pattern, register an auto-follow rule instead of calling _start per index, so new leader indices matching the pattern are picked up automatically:

POST _plugins/_replication/_autofollow
{
  "leader_alias": "leader-use1",
  "name": "logs-rule",
  "pattern": "logs-*",
  "use_roles": {
    "leader_cluster_role": "leader_read_role",
    "follower_cluster_role": "follower_write_role"
  }
}

The per-index start path, its exact status progression, and how it interacts with a follower-side ISM policy are covered in Starting follower index replication from a leader.

4. Verify the bootstrap #

Never assume replication started because _start returned 200 — that only means the task was accepted. Confirm the follower left BOOTSTRAPPING, that lag is closing, and that the index is genuinely write-blocked.

# 1. Replication status and current state for the follower
curl -s "https://<follower>:9200/_plugins/_replication/logs-000001/_status?pretty"

# 2. Confirm the follower index is write-blocked (it must be)
curl -s "https://<follower>:9200/logs-000001/_settings?pretty" \
  | grep -i "index.blocks.write"

# 3. Compare leader vs follower doc counts as a coarse lag check
curl -s "https://<leader>:9200/_cat/count/logs-000001?v"
curl -s "https://<follower>:9200/_cat/count/logs-000001?v"

A healthy follower reports "status": "SYNCING" (or SYNCED), a non-null leader_checkpoint and follower_checkpoint that are converging, and index.blocks.write: true. If status is FAILED, the reason field names the cause directly — an under-scoped role, an unresolvable seed, or a follower index that already existed.

Production automation with opensearch-py #

Starting one follower by hand is fine; onboarding a fleet is not. The manager below registers a follower, polls _status until it leaves BOOTSTRAPPING, and fails fast with structured logs if the index lands in FAILED. It uses narrow exception handling so it is safe to run from a pipeline, and it never issues a blind retry against a role error — it surfaces the reason and stops, matching the state machine above.

import logging
import time
from opensearchpy import OpenSearch, exceptions

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("ccr-bootstrap")

USE_ROLES = {
    "leader_cluster_role": "leader_read_role",
    "follower_cluster_role": "follower_write_role",
}
TERMINAL_OK = {"SYNCING", "SYNCED"}


class CCRBootstrapper:
    def __init__(self, follower: OpenSearch):
        self.follower = follower

    def start_replication(self, index: str, leader_alias: str, leader_index: str) -> bool:
        body = {
            "leader_alias": leader_alias,
            "leader_index": leader_index,
            "use_roles": USE_ROLES,
        }
        try:
            self.follower.transport.perform_request(
                "PUT", f"/_plugins/_replication/{index}/_start", body=body
            )
            logger.info("start accepted for %s <- %s/%s", index, leader_alias, leader_index)
            return True
        except exceptions.RequestError as exc:
            # 400 with resource_already_exists means the follower index pre-existed.
            logger.error("start rejected for %s: %s", index, exc.error)
            return False
        except exceptions.AuthorizationException:
            logger.error("replication user lacks cluster:admin/plugins/replication/* for %s", index)
            return False

    def wait_until_synced(self, index: str, timeout_s: int = 600, poll_s: int = 10) -> str:
        deadline = time.monotonic() + timeout_s
        while time.monotonic() < deadline:
            status = self.follower.transport.perform_request(
                "GET", f"/_plugins/_replication/{index}/_status"
            )
            state = status.get("status", "UNKNOWN")
            if state == "FAILED":
                logger.error("%s FAILED: %s", index, status.get("reason", "no reason"))
                return state
            if state in TERMINAL_OK:
                logger.info("%s reached %s", index, state)
                return state
            logger.info("%s still %s; waiting", index, state)
            time.sleep(poll_s)
        logger.warning("%s did not leave BOOTSTRAPPING within %ds", index, timeout_s)
        return "TIMEOUT"


if __name__ == "__main__":
    follower = OpenSearch(
        hosts=[{"host": "follower.internal", "port": 9200}],
        http_auth=("ccr-replicator", "..."),
        use_ssl=True,
        verify_certs=True,
    )
    mgr = CCRBootstrapper(follower)
    if mgr.start_replication("logs-000001", "leader-use1", "logs-000001"):
        mgr.wait_until_synced("logs-000001")

Run it once per index during onboarding, or wrap start_replication in a loop over a leader index list. Export a counter for indices that reach SYNCING and a separate one for FAILED so a partial fleet rollout is visible on a dashboard rather than buried in logs.

Bootstrap is a one-time event, but replication health is continuous, so the same client should also drive a steady-state monitor. The two numbers worth alerting on are the checkpoint gap (leader_checkpoint - follower_checkpoint, your live lag in operations) and the count of followers whose status is anything other than SYNCING/SYNCED. A checkpoint gap that trends upward over several poll intervals means the follower cannot keep pace — usually peer bandwidth saturation or an under-provisioned follower tier — and a follower that drops to FAILED in steady state, having previously been healthy, is almost always a retention-lease expiry rather than a fresh misconfiguration. Emitting both as gauges lets you distinguish “onboarding not finished” from “an established replica is falling behind,” which demand very different responses.

Operational guardrails #

Bootstrap is safe only when the leader retains enough history for the follower to tail without falling off the end of the translog and triggering a full remote resync. The recovery point objective you can actually hit is bounded by the ingest rate and how long the follower can be behind before the leader garbage-collects the operations it still needs:

where $\lambda$ is the leader write rate in ops/sec and $t_{detect} + t_{failover}$ is the time to notice a leader loss and promote the follower. Keep soft-delete and translog retention comfortably above $\lambda \times$ your worst tolerated lag so a network blip resumes from the remote translog instead of re-restoring every shard.

Undersizing retention_lease.period is the single most expensive mistake: a follower that lags past it cannot resume incrementally and is forced back through a full BOOTSTRAPPING restore, re-consuming peering bandwidth you were trying to protect.

Troubleshooting #

A follower that repeatedly returns to FAILED after a successful start is almost always a soft-delete retention shortfall on the leader, not a network fault — check index.soft_deletes.enabled on the leader index before touching the connection.

Frequently asked questions #

Related #

• Bootstrapping CCR with remote cluster connections — establishing and verifying the persistent cluster.remote.<alias>.seeds wire before any follower starts.
• Starting follower index replication from a leader — the per-index _start call and watching _status through the bootstrap states.
• Security & Access Boundaries — scoping the leader-read and follower-write roles that CCR carries across the wire.
• Node Role Allocation — assigning the remote_cluster_client and data roles the replication engine depends on.

Up: Cross-Cluster Replication Operations