Cross-Cluster Replication Operations

OpenSearch Cross-Cluster Replication (CCR) continuously ships index operations from a leader cluster to one or more read-only follower clusters, giving you an active-passive disaster-recovery posture and geo-local read replicas that stay within seconds of the source. This guide is written for search and log-platform engineers, data-platform operators, and Python automation builders who must run CCR alongside Index State Management (ISM) in production — where the follower’s write-block, replication lag, checkpoint retention, and failover discipline all have to behave deterministically under partition and node loss.

CCR treats one cluster as the authoritative writer and every other as a passive mirror. The leader accepts all indexing traffic and journals each operation in its translog with soft-deletes retained; the follower opens a persistent connection to the leader, pulls batches of operations from a shard-level changes API, and applies them in order to a write-blocked local copy. Because the follower never accepts client writes, it can never diverge from the leader as long as replication is running — the entire operational discipline of CCR is about keeping that invariant true through network partitions, node loss, and lifecycle transitions on both clusters. The sections below map the topology and remote-connection model, the auto-follow mechanism that keeps replication comprehensive, the way CCR collides with ISM’s write actions, the checkpoint and lag signals you monitor, and the failover procedure that promotes a follower without corrupting data.

Replication Topology & the Remote Connection Model #

A CCR deployment is defined by two clusters and a directional relationship between them. The follower initiates and owns the connection: you register the leader on the follower cluster as a named remote using a persistent cluster.remote.<alias>.seeds setting that lists one or more of the leader’s transport-port (9300) addresses. The follower’s cross-cluster nodes then perform a sniff or proxy handshake, discover the leader’s shards, and maintain long-lived connections over which the changes API streams. This is a pull model: the follower drives the pace, which is why the follower cluster must be sized for both its local read traffic and the replay throughput of the leader’s write rate.

Topology choices follow directly from that model. A single leader with one follower is the canonical active-passive disaster-recovery pair. A single leader with several followers gives you geo-distributed read replicas — each region opens its own remote connection to the same leader and serves local search traffic at low latency while writes route back to the one leader. What CCR deliberately does not offer is native multi-primary or bidirectional replication of the same index; two clusters both accepting writes to the “same” index is precisely the split-brain failure mode that CCR’s single-writer discipline exists to prevent, and it is treated in depth in Split-Brain Prevention. The mechanics of registering the remote, establishing trust, and starting the first follower index are the subject of CCR Setup and Bootstrap.

The remote connection carries more than raw documents. Each replicated batch advances a per-shard checkpoint — a monotonically increasing sequence number the follower persists so it can resume from exactly where it left off after a restart, rather than re-pulling the whole shard. Retaining enough history on the leader for that resume to succeed is a capacity decision, not a default you can ignore; the retention knobs and their sizing appear later in this guide and in full under Split-Brain Prevention.

The core replication commands run against the _plugins/_replication/* API on the follower cluster:

# On the FOLLOWER: register the leader as a persistent remote
PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.dr-leader.seeds": ["10.20.0.11:9300", "10.20.0.12:9300"]
  }
}

# On the FOLLOWER: start replicating one leader index into a new follower index
PUT _plugins/_replication/logs-app-prod-000042/_start
{
  "leader_alias": "dr-leader",
  "leader_index": "logs-app-prod-000042",
  "use_roles": {
    "leader_cluster_role": "ccr_leader_role",
    "follower_cluster_role": "ccr_follower_role"
  }
}

# On the FOLLOWER: inspect replication health for that index
GET _plugins/_replication/logs-app-prod-000042/_status

The use_roles block is not optional hygiene — it pins the exact security roles CCR assumes on each side, so a follower cannot quietly read leader indices it was never granted. Those role boundaries build on the same fine-grained access-control model described in Security & Access Boundaries, extended across the cross-cluster trust boundary.

Auto-Follow: Keeping Replication Comprehensive #

Starting replication one index at a time is fine for a fixed set of indices, but time-series and log workloads roll over constantly — every rollover on the leader mints a new backing index that would otherwise sit unreplicated until an operator noticed. Auto-follow closes that gap. You register a pattern on the follower cluster; whenever the leader creates an index whose name matches, the follower automatically starts a follower index for it. This is what makes CCR safe to pair with ISM rollover: the leader’s Index Lifecycle Basics rollover cadence generates logs-app-prod-000043, -000044, and so on, and auto-follow picks each one up without human intervention.

# On the FOLLOWER: auto-follow every new leader index matching the pattern
POST _plugins/_replication/_autofollow
{
  "leader_alias": "dr-leader",
  "name": "logs-app-prod-rule",
  "pattern": "logs-app-prod-*",
  "use_roles": {
    "leader_cluster_role": "ccr_leader_role",
    "follower_cluster_role": "ccr_follower_role"
  }
}

Two operational caveats matter here. First, the auto-follow pattern must exclude indices you never want replicated — a too-broad pattern like * will pull system and history indices and waste follower capacity. Second, auto-follow reacts to index creation, so an index that already existed on the leader before you registered the rule is not retroactively followed; back-fill those with an explicit _start call. When auto-follow is combined with ISM rollover on the leader, the follower ends up mirroring an ever-growing set of read-only backing indices, each of which needs its own follower-side lifecycle handling — the design problem addressed next.

How CCR Collides with ISM: the Follower Write-Block #

The single most consequential fact about running ISM with CCR is that a follower index is write-blocked for its entire replicating life. Replication holds the index read-only so that the only writer is the replication engine applying leader operations. Any ISM action that mutates the index — rollover, force_merge, shrink, open/close, replica_count changes that require a write, or delete while replication is active — will fail on a follower, and the managed index parks in a failed action state exactly as a stuck transition does on a normal index.

This is not a bug to work around with force; it is the invariant that keeps the leader and follower consistent. The correct model is a division of labor: the leader cluster owns the write lifecycle (rollover to bound shard size, force_merge to reclaim segments, tier allocation), and those structural changes replicate to the follower as data. The follower cluster runs a deliberately narrower policy that only performs read-safe actions — allocation to place replicated shards on the follower’s own tier hardware, snapshot for independent archival, and a delete that is gated on replication having been stopped first. Designing that narrower policy correctly, including how to resolve an index that has already jammed on a write action, is the whole subject of Follower Policy Design.

A minimal follower-side policy skeleton illustrates the discipline — note the absence of rollover and the terminal delete reached only after an operator or automation has issued _stop:

{
  "policy": {
    "description": "Follower-side lifecycle: place replicated shards, archive, then delete after stop",
    "default_state": "replicating",
    "ism_template": [
      { "index_patterns": ["logs-app-prod-*"], "priority": 90 }
    ],
    "states": [
      {
        "name": "replicating",
        "actions": [
          { "allocation": { "require": { "data": "warm" }, "wait_for": true } }
        ],
        "transitions": [
          { "state_name": "archive", "conditions": { "min_index_age": "30d" } }
        ]
      },
      {
        "name": "archive",
        "actions": [
          { "snapshot": { "repository": "s3-dr-archive", "snapshot": "follower-lifecycle" } }
        ],
        "transitions": [
          { "state_name": "delete", "conditions": { "min_index_age": "97d" } }
        ]
      },
      {
        "name": "delete",
        "actions": [
          { "retry": { "count": 5, "backoff": "exponential", "delay": "1h" },
            "delete": {} }
        ],
        "transitions": []
      }
    ]
  }
}

The allocation action is safe because it changes routing metadata, not index contents, and CCR permits the follower to place its own replicated shards on whatever tier hardware the follower cluster exposes — which may differ entirely from the leader’s, so the follower’s routing attributes are chosen against the follower’s own Node Role Allocation layout rather than inherited blindly. The delete state only succeeds once replication is stopped, because deleting a replicating follower index fails the same way a write does. In practice you either sequence a _stop call before the retention deadline through automation, or design the follower’s retention to outlive the leader’s so the leader-side delete propagates as a natural close.

Crucially, ISM policy attachments do not replicate. A policy attached on the leader has no effect on the follower, because CCR carries index data and settings, not ISM metadata from the .opendistro-ism-config index. You attach the follower policy independently on the follower cluster. Getting that separation wrong — attaching a leader-style rollover policy on the follower — is the fastest way to a follower cluster full of managed indices stuck on write-block failures.

Security & Access Control Across the Trust Boundary #

CCR widens the blast radius of a credential leak: a compromised follower can read every leader index its role permits. Scope the two use_roles identities to the minimum. The leader-side role needs only the cross-cluster read and changes-tracking permissions on the replicated index patterns; the follower-side role needs the replication management permissions plus write on its own follower indices. Neither role should carry broad cluster-admin rights.

{
  "ccr_leader_role": {
    "index_permissions": [
      {
        "index_patterns": ["logs-app-prod-*"],
        "allowed_actions": [
          "indices:data/read/plugins/replication/changes",
          "indices:admin/plugins/replication/index/setup/validate",
          "indices:monitor/stats"
        ]
      }
    ]
  },
  "ccr_follower_role": {
    "cluster_permissions": ["cluster:admin/plugins/replication/autofollow/update"],
    "index_permissions": [
      {
        "index_patterns": ["logs-app-prod-*"],
        "allowed_actions": [
          "indices:admin/plugins/replication/index/start",
          "indices:admin/plugins/replication/index/pause",
          "indices:admin/plugins/replication/index/stop",
          "indices:data/write/plugins/replication/changes"
        ]
      }
    ]
  }
}

Treat the remote-connection seeds and the replication roles as production secrets, rotate them out of band, and audit the _plugins/_replication/_autofollow rules the same way you audit ISM policy writes — an over-broad auto-follow pattern is both a capacity risk and a data-exposure risk. The broader FGAC scoping for the ISM endpoints these roles sit beside is detailed in Security & Access Boundaries.

Python Automation for CCR Operations #

Bootstrapping replication for a fleet of indices, verifying lag, and orchestrating a controlled failover are all tasks you want as idempotent, logged code rather than ad-hoc curl. The opensearch-py client drives the _plugins/_replication/* API the same way it drives ISM. The example below starts replication idempotently — treating an already-replicating index as a no-op — and reads back the status with typed signatures and structured error handling suitable for a CI/CD job or a failover runbook:

import logging
from typing import Any, Dict
from opensearchpy import OpenSearch, ConnectionError, RequestError, TransportError

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)


def start_replication(
    follower: OpenSearch,
    follower_index: str,
    leader_alias: str,
    leader_index: str,
    roles: Dict[str, str],
) -> Dict[str, Any]:
    """Start CCR for one index. Idempotent: an already-replicating index
    returns a resource-already-exists error that we treat as success."""
    try:
        resp = follower.transport.perform_request(
            method="PUT",
            url=f"/_plugins/_replication/{follower_index}/_start",
            body={
                "leader_alias": leader_alias,
                "leader_index": leader_index,
                "use_roles": roles,
            },
        )
        logger.info("Replication started for '%s' <- %s:%s",
                    follower_index, leader_alias, leader_index)
        return resp
    except RequestError as e:
        if "resource_already_exists" in str(e).lower() or e.status_code == 400:
            logger.info("'%s' already replicating; no-op.", follower_index)
            return {"acknowledged": True, "already_running": True}
        logger.error("Replication start rejected for '%s': %s", follower_index, e.info)
        raise
    except (ConnectionError, TransportError) as e:
        logger.error("Transport failure starting replication for '%s': %s", follower_index, e)
        raise


def replication_lag(follower: OpenSearch, follower_index: str) -> Dict[str, Any]:
    """Return leader vs follower checkpoints so callers can alert on lag."""
    status = follower.transport.perform_request(
        method="GET", url=f"/_plugins/_replication/{follower_index}/_status"
    )
    idx = status.get("syncing_indices", [{}])[0] if status.get("syncing_indices") else status
    leader_cp = idx.get("leader_checkpoint")
    follower_cp = idx.get("follower_checkpoint")
    lag = None
    if isinstance(leader_cp, int) and isinstance(follower_cp, int):
        lag = leader_cp - follower_cp
    logger.info("'%s' lag=%s (leader=%s follower=%s)",
                follower_index, lag, leader_cp, follower_cp)
    return {"index": follower_index, "operations_behind": lag}


if __name__ == "__main__":
    follower_client = OpenSearch(
        hosts=[{"host": "opensearch-follower.dr.internal", "port": 9200}],
        http_auth=("ccr_automation_svc", "REDACTED"),
        use_ssl=True,
        verify_certs=True,
        maxsize=10,
        retry_on_timeout=True,
        max_retries=3,
    )
    ccr_roles = {
        "leader_cluster_role": "ccr_leader_role",
        "follower_cluster_role": "ccr_follower_role",
    }
    start_replication(follower_client, "logs-app-prod-000042",
                      "dr-leader", "logs-app-prod-000042", ccr_roles)
    replication_lag(follower_client, "logs-app-prod-000042")

The idempotency guard matters because failover runbooks and drift-reconciliation jobs re-run the same start call repeatedly; a transient rejection must never be mistaken for a real failure, and a genuine transport error must never be swallowed. Wrapping replication start, pause, and resume in the same bounded-retry discipline you use for stuck ISM transitions keeps a partition from cascading — the retry and backoff patterns are developed under Error Handling & Retries, and this CCR automation slots into the broader ISM Policy Implementation & Python Automation toolchain.

Monitoring Replication Lag & Checkpoints #

Replication health reduces to one question asked continuously: how far behind the leader is each follower shard? CCR answers it with checkpoints. The follower persists a follower_checkpoint (the last sequence number it applied) and tracks the leader_checkpoint (the leader’s latest); the difference is the operations backlog, and the wall-clock age of that backlog is your effective recovery-point objective. In formal terms:

A steadily growing lag_ops means the follower cannot keep up with the leader’s write rate — usually follower CPU, disk, or network saturation — and your RPO is silently degrading. A lag that spikes and recovers is a transient the retention window should absorb; a lag that grows unbounded will eventually exhaust the leader’s retained history and force a full remote resync, which is the expensive failure that checkpoint retention tuning exists to prevent.

# On the FOLLOWER: per-index replication status, checkpoints, and any failures
curl -s "https://<follower>:9200/_plugins/_replication/logs-app-prod-000042/_status?pretty"

# On the FOLLOWER: autofollow rule stats — how many indices each rule has picked up
curl -s "https://<follower>:9200/_plugins/_replication/autofollow_stats?pretty"

# On the FOLLOWER: fast index inventory to cross-check replicated sizes and health
curl -s "https://<follower>:9200/_cat/indices/logs-app-prod-*?v&h=index,health,docs.count,store.size&s=index"

Alert on the derivative of lag, not a single reading — a follower that is 500 operations behind at steady state is healthy, while one climbing by thousands per minute is heading for resync. Cross-reference lag with the follower’s disk watermarks and unassigned-shard count exactly as you would for a stalled ISM transition, so you can distinguish a follower capacity problem from a leader-side stall. Retention sizing that keeps a recovered follower resuming from the remote translog instead of a full resync is covered in detail in Split-Brain Prevention; the retention knobs are index.plugins.replication.translog.retention_size on the follower side and the leader’s soft-delete retention lease.

Disaster Recovery & Controlled Failover #

The payoff for all of this discipline is a clean failover. When the leader is lost — region outage, cluster failure, or a planned migration — you promote the follower to a writable primary. The sequence is deterministic and must not be shortcut, because the write-block is the only thing that has kept the follower consistent, and lifting it prematurely is how you create split-brain.

A controlled failover proceeds as: (1) confirm the follower has drained the leader’s remaining operations by checking that lag_ops has reached zero, or accept the current RPO if the leader is truly unreachable; (2) _stop replication on the follower index, which removes the write-block and detaches the CCR relationship, leaving a normal writable index; (3) point write clients at the newly promoted cluster; and (4) attach the promoted index to a full leader-style ISM policy so rollover and tiering resume under the new writer. Only after the original leader recovers do you rebuild replication in the reverse direction, and only from a checkpoint the retention window still covers — otherwise the recovered cluster must remote-resync the entire dataset.

# On the FOLLOWER being promoted: drain then stop to remove the write-block
GET _plugins/_replication/logs-app-prod-000042/_status
POST _plugins/_replication/logs-app-prod-000042/_stop
{}

# The index is now a normal writable index; attach a full leader-style ISM policy
POST _plugins/_ism/add/logs-app-prod-000042
{ "policy_id": "leader_retention_policy" }

Use _pause rather than _stop for a transient leader blip you expect to recover from quickly — pausing preserves the replication relationship and resumes from the retained checkpoint, whereas stopping is a one-way promotion. The distinction is the difference between a five-minute network hiccup and a genuine disaster, and confusing the two either wastes a full resync or accidentally promotes a follower you meant to keep passive. Whichever path you take, never allow both clusters to accept writes to the same index simultaneously during the transition — that overlap is the split-brain condition, and preventing it under partition is treated fully in Split-Brain Prevention.

Frequently asked questions #

Related #

• CCR Setup and Bootstrap — registering the leader as a persistent remote, establishing cross-cluster trust and roles, and starting the first follower index from a clean bootstrap.
• Follower Policy Design — building the deliberately narrow ISM policy a write-blocked follower can run, and resolving indices already jammed on write-block conflicts.
• Split-Brain Prevention — single-writer discipline, fencing under network partitions, and tuning checkpoint retention so a recovered follower resumes from the remote translog instead of a full resync.

Up: OpenSearch ISM & Cross-Cluster Replication