OpenSearch ISM & Cross-Cluster Replication

Starting follower index replication from a leader

This procedure issues PUT _plugins/_replication/<follower>/_start with an explicit leader_alias, leader_index, and use_roles binding, then watches _status walk the follower from BOOTSTRAPPING through SYNCING to SYNCED — turning an accepted API call into a verified, lag-free replica.

Starting replication is deceptively simple: one PUT and the API returns 200. The work is everything after that acknowledgement, because a 200 only means the replication task was scheduled. Whether the follower actually restored the leader shards, began tailing the translog, and closed its checkpoint gap is a question only _status answers. This page assumes the remote wire is already verified in Bootstrapping CCR with remote cluster connections; it is the second half of the flow described in CCR Setup and Bootstrap.

Prerequisites

Each of these must hold before the _start call, or the follower lands in FAILED within the first status poll.

The start-to-synced progression

The _start call schedules a task; the follower then advances through the bootstrap states on its own clock. The diagram traces the request and the status polls that confirm each transition, ending at the steady-state checkpoint loop where a SYNCED follower briefly re-enters SYNCING each time new leader operations arrive.

Follower replication start and status progression Sequence timeline. The client issues PUT _start and receives 202 accepted. A poll loop of GET _status observes three stages: BOOTSTRAPPING as the leader snapshot restores, SYNCING as the follower tails the leader translog, and SYNCED once the follower checkpoint equals the leader checkpoint. From SYNCED an arrow returns to SYNCING labeled new leader ops, showing the steady-state loop. PUT _start leader_alias + use_roles 202 task scheduled poll _status → BOOTSTRAPPING restore leader snapshot SYNCING tail leader translog SYNCED follower_cp = leader_cp restore done lag = 0 new leader ops re-enter SYNCING GET _status every ~10s until BOOTSTRAPPING clears

Step-by-step procedure

1. Issue the start call

Bind the leader index to a new follower index of the same (or a different) name. The follower index must not exist yet. use_roles names the two roles the replication engine assumes for leader reads and follower writes.

HTTP
PUT _plugins/_replication/logs-000007/_start
{
  "leader_alias": "leader-use1",
  "leader_index": "logs-000007",
  "use_roles": {
    "leader_cluster_role": "leader_read_role",
    "follower_cluster_role": "follower_write_role"
  }
}
Text
{ "acknowledged": true }

Gotcha: acknowledged: true confirms the task was scheduled, nothing more. Do not chain a dependent step off this response — gate on _status reaching SYNCING instead.

2. Poll status through the bootstrap

Immediately begin polling _status. The follower first reports BOOTSTRAPPING while it restores the remote snapshot; on a large index this is the longest stage.

Shell
curl -s "https://<follower>:9200/_plugins/_replication/logs-000007/_status?pretty"
JSON
{
  "status": "BOOTSTRAPPING",
  "reason": "Bootstrapping the follower index: logs-000007",
  "leader_alias": "leader-use1",
  "leader_index": "logs-000007",
  "follower_index": "logs-000007"
}

Gotcha: a long BOOTSTRAPPING is normal for a big index but a stuck one usually means the follower cannot allocate the restored shards — check follower disk watermarks and remote_cluster_client capacity before assuming a CCR fault.

3. Confirm SYNCING and a converging checkpoint

Once the restore completes, status flips to SYNCING and the payload exposes syncing_details with leader and follower checkpoints. The gap between them is your live replication lag in operations.

Shell
curl -s "https://<follower>:9200/_plugins/_replication/logs-000007/_status?pretty"
JSON
{
  "status": "SYNCING",
  "syncing_details": {
    "leader_checkpoint": 480123,
    "follower_checkpoint": 479980,
    "seq_no": 480123
  }
}

The remaining operations to replay, and therefore the recovery-point gap at any instant, is simply

lagops=leader_checkpoint−follower_checkpoint\text{lag}_{ops} = \text{leader\_checkpoint} - \text{follower\_checkpoint}

Gotcha: a small non-zero lag under steady write load is healthy, not a failure — a follower on a busy leader will hover at SYNCING and never sit permanently at SYNCED. Only a lag that trends upward without recovering signals a real throughput or retention problem.

The distinction between a healthy oscillating lag and a genuine backlog is a matter of trend, not a single reading. Sample syncing_details on a short interval and watch whether the gap returns toward zero between leader write bursts. A follower whose follower_checkpoint advances but never closes the gap is bandwidth- or reader-limited, and the fix is more concurrent_readers_per_shard or more peer throughput — not restarting replication. A follower whose follower_checkpoint stops advancing entirely, while the leader’s climbs, is a stalled task and warrants inspecting reason and the follower’s shard allocation. Restarting a follower that is merely behind throws away the incremental progress it has made and forces another full BOOTSTRAPPING, so reach for _start again only after _status actually reports FAILED.

4. Attach a follower-side lifecycle policy (optional)

A follower is write-blocked, so lifecycle management on it must use only read-safe actions. If you want the follower to age out replicated indices, attach an ISM policy scoped to allocation and a stop-gated delete, as designed in Follower Policy Design.

HTTP
POST _plugins/_ism/add/logs-000007
{
  "policy_id": "ccr_follower_readsafe_policy"
}

Gotcha: never attach a policy with rollover, force_merge, or shrink to a follower — those are write actions and will fail the ISM step against the follower’s write block. Manage those on the leader and let CCR replicate the results.

Verification

Confirm the follower is genuinely replicating and read-only, not merely present.

Shell
# 1. Status is SYNCING or SYNCED, never FAILED
curl -s "https://<follower>:9200/_plugins/_replication/logs-000007/_status" \
  | python -c "import sys,json; print(json.load(sys.stdin)['status'])"
# SYNCING
Shell
# 2. Doc counts track between leader and follower (coarse lag proxy)
curl -s "https://<leader>:9200/_cat/count/logs-000007?h=count"     # 480123
curl -s "https://<follower>:9200/_cat/count/logs-000007?h=count"   # 479980  (catching up)
Shell
# 3. The follower is write-blocked as expected
curl -s -X POST "https://<follower>:9200/logs-000007/_doc" \
  -H "Content-Type: application/json" -d '{"probe": true}'
# 403 cluster_block_exception: index [logs-000007] blocked by: FORBIDDEN/1000/... (read-only)

A healthy result is SYNCING/SYNCED, converging doc counts, and a 403 on the write probe — the write rejection is the proof the follower is a true replica, not a defect.

Common failures

Symptom Root cause Fix command
_start returns 400 resource_already_exists A follower index of that name already exists DELETE logs-000007 on the follower, then re-issue _start
status: FAILED, reason mentions security_exception use_roles names a role missing on one cluster Recreate both roles identically, then _start again
Stuck in BOOTSTRAPPING indefinitely Restored shards cannot allocate on the follower GET _cluster/allocation/explain; free disk or add remote_cluster_client nodes
status: FAILED, reason soft deletes Leader index has soft deletes disabled Recreate the leader index with index.soft_deletes.enabled: true; CCR cannot tail without it
Follower falls behind and returns to BOOTSTRAPPING Follower lagged past the leader retention lease Raise leader index.soft_deletes.retention_lease.period; re-_start
ISM step fails on the follower A write action (rollover/force_merge) applied to a read-only follower Remove it via POST _plugins/_ism/remove/logs-000007; attach a read-safe policy per Follower Policy Design

Frequently asked questions

Can the follower index have a different name than the leader index?

Yes. leader_index and the index in the request path are independent, so PUT _plugins/_replication/logs-mirror/_start with "leader_index": "logs-000007" replicates the leader logs-000007 into a follower named logs-mirror. Keep names aligned when you use auto-follow, but for a one-off _start a rename is perfectly valid and common when the follower cluster has its own naming scheme.

My follower sits at SYNCING and never shows SYNCED — is that broken?

Almost certainly not. On a leader taking continuous writes, the follower is always a few operations behind and reports SYNCING with a small, stable checkpoint gap. SYNCED is the instantaneous state where the follower checkpoint has momentarily caught the leader — you see it on idle or bursty leaders. Watch the trend of leader_checkpoint - follower_checkpoint: flat or oscillating is healthy, monotonically rising is not.

How do I stop replication and make the follower writable?

Call POST _plugins/_replication/logs-000007/_stop. That detaches the index from the leader and lifts the write block, promoting it to an ordinary writable index — the supported failover path. Once stopped, replication cannot be resumed in place; you would start a fresh follower to re-establish it. Use _pause/_resume instead when you only need a temporary halt without breaking the leader binding.

Up: CCR Setup and Bootstrap