Split-Brain Prevention

Split-brain in a Cross-Cluster Replication (CCR) topology is the moment two clusters both accept writes to what operators still call the “same” index after a network partition severs the leader from its follower. The leader keeps ingesting; an impatient failover promotes the follower and it starts ingesting too; when the link heals there is no single truth to converge on and the two write histories have diverged irreconcilably. This guide, part of Cross-Cluster Replication Operations, shows how to make that outcome structurally impossible: a single-writer discipline that only ever grants the write role to one cluster, a fencing step that provably stops the old leader before the follower is promoted, a failover runbook that sequences those steps, and checkpoint retention sized so a recovered follower resumes from the remote translog instead of paying for a full remote resync.

Cluster Roles and Write-Authority Alignment #

Split-brain prevention is a question of who is allowed to write, so the foundational table is not tier-versus-hardware but cluster-role-versus-write-authority. Every cluster in the replication topology must carry an unambiguous role, a write mode enforced at the index level, and a fencing mechanism that can be actuated during a partition. The alignment below is what a failover runbook reads from; if any row is ambiguous, a partition will resolve it the wrong way.

The invariant across every row is that exactly one cluster holds read-write authority at any instant. A follower is write-blocked by CCR itself — the follower policy design rules make that block explicit rather than incidental — so the dangerous transition is never “two leaders exist” by accident; it is an operator promoting a follower while the old leader is still reachable by application traffic. The witness row exists so that “which side kept authority” is answered by a quorum vote, not by whichever operator reacted first. Fencing is the mechanism that turns the single-writer policy into a single-writer guarantee.

The Partition-to-Reconcile State Machine #

A partition does not cause split-brain; a bad response to a partition does. The safe response is a fixed four-phase sequence — detect the partition, fence the old leader, fail over to the follower, then reconcile when the link heals — and the state machine below is the contract every runbook and automation step must honour. The critical property is that the fence state is a hard gate: promotion is unreachable until fencing is confirmed, so there is no path in the graph where both clusters are writable at once.

The state machine also fixes what recovery looks like. Reconciliation does not mean “merge two histories” — that is unsolvable once both sides accepted conflicting writes. It means the fence held, so only one side has authoritative writes, and the other side re-bootstraps against it. Whether that re-bootstrap is cheap or catastrophic is decided entirely by checkpoint retention, covered in section 4 and in Tuning CCR checkpoint retention to avoid remote resync. The full operator sequence lives in Preventing split-brain during CCR network partitions.

1. Enforce Single-Writer Discipline at the Index Level #

Single-writer discipline means the application can only ever address the write role, never a specific cluster. Route all writes through a write alias and keep the follower index write-blocked by CCR so a misdirected write to the follower fails loudly rather than silently forking history.

PUT logs-000001
{
  "aliases": {
    "logs-write": { "is_write_index": true }
  }
}

On the leader, confirm the write alias resolves to exactly one backing index; on the follower, confirm CCR’s write block is active so nothing can write there while it is a follower.

GET _plugins/_replication/logs-000001/_status

{
  "status": "SYNCING",
  "leader_alias": "dc-primary",
  "leader_index": "logs-000001",
  "follower_index": "logs-000001",
  "syncing_details": { "leader_checkpoint": 84213, "follower_checkpoint": 84210 }
}

A SYNCING status with a small, bounded gap between leader_checkpoint and follower_checkpoint is the signal that the follower is genuinely tracking the leader and can be promoted with minimal loss. A gap that grows without bound means retention is undersized — fix that before you ever need to fail over.

Single-writer discipline is a design constraint on the application, not only the OpenSearch cluster. The moment any producer is configured to write to a named cluster endpoint rather than the write alias, a partition can leave half the fleet writing to the old leader and half to a freshly promoted follower — split-brain introduced by client configuration, invisible to the clusters themselves. Enforce the discipline at deploy time: producers resolve logs-write through service discovery, the alias lives in exactly one place, and promotion is a single atomic repoint of that alias. That way there is never a window in which two endpoints each believe they are “the writer”, and the clusters never have to arbitrate a conflict they cannot solve.

2. Prepare the Fence #

Fencing is the act of guaranteeing the old leader cannot accept another write before the follower is promoted. Two mechanisms combine: an index-level write block, and removal of the remote connection so the fenced cluster cannot re-establish a replication relationship that would resurrect it as an authority.

PUT logs-000001/_block/write

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.dc-primary.seeds": null
  }
}

Dropping the seeds is applied on the follower side during failover so the promoted cluster stops trusting the old leader as a source. Keep the fence actuable during a partition — the block must be settable from the surviving side without needing the unreachable cluster to acknowledge it. That is why the write block is set locally on each side rather than coordinated across the link.

There is a subtlety worth stating plainly: fencing does not require reaching the old leader. In an asymmetric partition where the operator can reach the follower’s data centre but not the leader’s, you cannot set a write block on the old leader — so fencing degrades to the seed-drop plus an arbiter decision that names the surviving side. What you must never do is treat “I cannot reach the old leader” as proof that the old leader is not taking writes. It may be perfectly healthy and serving application traffic from its own region while merely unreachable from your vantage point. The fence is a guarantee about the old leader’s write capability, not about your ability to observe it, and the automation in the next section encodes exactly that distinction.

3. Execute the Failover #

Only after the fence is confirmed does the follower stop replication and take the write role. Stopping replication is what converts a read-only follower into a writable index; do it in the wrong order and you have created the split-brain the fence was meant to prevent.

POST _plugins/_replication/logs-000001/_stop
{
  "leader_alias": "dc-primary",
  "leader_index": "logs-000001"
}

After _stop returns, the former follower is a normal writable index. Repoint the write alias to it and let application traffic follow. Handle the inevitable transient errors during the cutover with bounded retries rather than tight loops, following the same discipline as error handling and retries — a failover that hammers a recovering cluster only extends the outage.

POST _aliases
{
  "actions": [
    { "add": { "index": "logs-000001", "alias": "logs-write", "is_write_index": true } }
  ]
}

4. Size Checkpoint Retention so Recovery Resumes, Not Resyncs #

When the partition heals, the old leader must rejoin as a follower. If the new leader still retains the operations from the checkpoint the old leader last acknowledged, replication resumes incrementally from the remote translog. If those operations have aged out, CCR falls back to a full remote bootstrap — re-copying every segment across the link. Retention size is the single lever that decides which happens.

PUT logs-000001/_settings
{
  "index.plugins.replication.translog.retention_size": "512mb",
  "index.soft_deletes.retention_lease.period": "12h"
}

Size retention to cover the longest plausible partition times the peak write rate — the arithmetic is worked through in Tuning CCR checkpoint retention to avoid remote resync. The retention lease period is the grace window during which the leader promises not to reclaim operations for a disconnected follower; if it expires the lease is dropped and the incremental resume path closes. When routing during recovery, keep the fallback routing strategies in mind so a re-bootstrapping follower does not strand shards on a saturated tier.

The reason retention belongs in a split-brain guide at all is behavioural, not just mechanical. Teams that have watched a failover trigger a multi-hour full resync across a congested inter-region link learn to dread failover — and dread breeds shortcuts. The most common shortcut is skipping or rushing the fence to “get writes flowing again”, which is precisely the action that manufactures split-brain. Right-sized retention removes the incentive: recovery becomes a cheap incremental catch-up measured in seconds, the failover drill stops being frightening, and the fence step is never the thing under time pressure. Retention sizing is therefore a safety control, not merely a performance tuning, and it should be reviewed on the same cadence as write-rate growth.

Production Automation: A Fenced-Failover Check #

Manual failover under pressure is where split-brain is born. The opensearch-py helper below is the guard you run before promoting a follower: it verifies the old leader is genuinely fenced (write-blocked and unreachable as a remote) and only then reports the follower as safe to promote. It never promotes on its own — a human or an orchestration step makes that call — but it refuses to green-light a promotion while the old leader could still take writes. Scope its service account to hold only cluster:monitor/* and indices:admin/settings/update on the target index pattern.

import logging
from opensearchpy import OpenSearch, exceptions

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("fenced-failover")


class FencedFailoverCheck:
    """Verify the old leader is fenced before a follower may be promoted."""

    def __init__(self, leader: OpenSearch, follower: OpenSearch, index: str):
        self.leader = leader
        self.follower = follower
        self.index = index

    def leader_reachable(self) -> bool:
        """True if the old leader still answers — i.e. it is NOT fenced by partition."""
        try:
            self.leader.cluster.health(request_timeout=3)
            return True
        except (exceptions.ConnectionError, exceptions.ConnectionTimeout):
            return False

    def leader_write_blocked(self) -> bool:
        """True if the old leader carries an index write block (explicit fence)."""
        if not self.leader_reachable():
            # Unreachable counts as fenced ONLY if we also confirm seeds are dropped.
            return True
        try:
            settings = self.leader.indices.get_settings(index=self.index, flat_settings=True)
            blocks = settings.get(self.index, {}).get("settings", {})
            return blocks.get("index.blocks.write") == "true"
        except exceptions.TransportError as exc:
            logger.error("Cannot read leader settings for %s: %s", self.index, exc)
            return False

    def follower_caught_up(self, max_lag_ops: int = 100) -> bool:
        """True if follower checkpoint is within max_lag_ops of the last leader checkpoint."""
        try:
            status = self.follower.transport.perform_request(
                "GET", f"/_plugins/_replication/{self.index}/_status"
            )
        except exceptions.TransportError as exc:
            logger.error("Cannot read replication status: %s", exc)
            return False
        details = status.get("syncing_details", {})
        lag = details.get("leader_checkpoint", 0) - details.get("follower_checkpoint", 0)
        logger.info("Replication lag for %s: %d ops", self.index, lag)
        return 0 <= lag <= max_lag_ops

    def safe_to_promote(self) -> bool:
        """A follower is safe to promote only when the old leader is fenced."""
        fenced = self.leader_write_blocked()
        caught_up = self.follower_caught_up()
        if fenced and caught_up:
            logger.info("%s: old leader fenced and follower caught up -> SAFE", self.index)
            return True
        logger.warning(
            "%s: NOT safe (fenced=%s, caught_up=%s) -> refusing promotion",
            self.index, fenced, caught_up,
        )
        return False


if __name__ == "__main__":
    leader = OpenSearch(hosts=[{"host": "dc-primary", "port": 9200}],
                        http_auth=("admin", "admin"), use_ssl=True, verify_certs=True)
    follower = OpenSearch(hosts=[{"host": "dc-standby", "port": 9200}],
                          http_auth=("admin", "admin"), use_ssl=True, verify_certs=True)
    check = FencedFailoverCheck(leader, follower, "logs-000001")
    if check.safe_to_promote():
        logger.info("Promotion gate PASSED. Proceed with _stop then repoint write alias.")
    else:
        logger.error("Promotion gate FAILED. Do not promote; investigate fence state.")

The single most important line is the conjunction in safe_to_promote: promotion requires both a fenced old leader and a caught-up follower. Dropping either check is how teams reintroduce split-brain — a caught-up follower promoted against a still-writable leader forks history, and a fenced leader promoted against a lagging follower loses acknowledged writes. Run this as the gate ahead of any automated or manual failover.

Operational Guardrails #

The settings below keep the single-writer invariant enforceable and the fence actuable during a real partition.

An odd-numbered arbiter set is what prevents the cluster-internal split-brain that would otherwise let each partition half elect its own master; it is the same quorum logic applied one level up from the index. Retention sizing follows the capacity envelope you set when designing follower policies — under-provision it and every partition becomes a full resync.

Troubleshooting #

The recurring root cause across these rows is ordering: a fence applied after promotion, or a _stop issued before the fence is confirmed. The state machine exists precisely to make that ordering non-negotiable. When a follower refuses to stop, check that the _stop body carries the exact leader_alias and leader_index the replication was started with — a mismatch there leaves the follower stuck write-blocked.

Frequently asked questions #

Related #

• Preventing split-brain during CCR network partitions — the operator runbook that sequences detect, fence, promote, and reconcile.
• Tuning CCR checkpoint retention to avoid remote resync — sizing translog and soft-delete retention so recovery resumes incrementally.
• Follower Policy Design — the follower-side ISM rules that keep the write block explicit.

Up: Cross-Cluster Replication Operations