Preventing split-brain during CCR network partitions

This runbook keeps exactly one writer alive while a Cross-Cluster Replication (CCR) link is partitioned, then promotes the follower only after the old leader is provably fenced — the sequence that stops a partition from forking one index into two divergent histories. It is the step-by-step operator procedure behind the model in Split-Brain Prevention.

Prerequisites #

Confirm each item before a partition happens — this runbook is executed under pressure, and a missing prerequisite is discovered at the worst possible moment.

• A leader-follower CCR pair is running with GET _plugins/_replication/logs-000001/_status returning SYNCING on the follower.
• The application writes only through a logs-write alias, never to a named cluster endpoint, so redirecting writes is a single alias repoint — the single-writer basis from Split-Brain Prevention.
• Checkpoint retention is sized to cover your longest expected partition, per Tuning CCR checkpoint retention to avoid remote resync, so recovery resumes instead of resyncing.
• The follower’s ISM configuration keeps it write-blocked and terminal-action-safe, following Follower Policy Design.
• A scoped service account exists with cluster:monitor/*, indices:admin/settings/update, and the _plugins/_replication stop permission on the target index pattern.
• An arbiter or documented decision rule names which side keeps write authority in a symmetric partition, so promotion is never a judgement call.

Decision flow #

The whole runbook is one decision: promote the follower only if the old leader is fenced, otherwise wait. The flow below is the shape of that decision — every step in the procedure maps to one node in it.

Step-by-step procedure #

1. Detect the partition and freeze the promotion instinct #

Before touching anything, confirm the partition is real and not a slow link. Check replication status from the follower side and the remote connection health.

GET _plugins/_replication/logs-000001/_status
GET _remote/info

# follower _status during a partition
"status" : "SYNCING",
"syncing_details" : { "leader_checkpoint" : 90418, "follower_checkpoint" : 90418 }
# _remote/info
"dc-primary" : { "connected" : false, "num_nodes_connected" : 0 }

Gotcha: a follower still reports SYNCING immediately after a partition because it does not yet know the leader is gone — "connected": false from _remote/info is the authoritative partition signal, not the replication status.

2. Fence the old leader before promoting anything #

If the old leader is still reachable by application traffic, write-block it locally so it cannot accept another write. This is settable from the leader’s own side without the follower’s participation.

PUT logs-000001/_block/write

{ "acknowledged" : true, "shards_acknowledged" : true, "indices" : [ { "name" : "logs-000001", "blocked" : true } ] }

Gotcha: if the old leader is unreachable from your operator station too, fencing by write block is impossible — rely on the dropped-seeds step on the follower side and your arbiter’s decision, and never promote on the assumption that “unreachable to me” means “writing to no one”.

3. Confirm the follower is caught up within the lag budget #

Promoting a lagging follower loses acknowledged writes. Check the checkpoint gap and only proceed if it is within your budget.

GET _plugins/_replication/logs-000001/_status

"syncing_details" : { "leader_checkpoint" : 90418, "follower_checkpoint" : 90418 }
# gap = 0 ops -> zero-loss promotion is possible

Gotcha: the last checkpoint the follower saw is the last one before the partition — a zero gap here means no acknowledged writes are lost, but writes the old leader accepted after the partition are gone unless the fence let you recover them. That is expected and correct: those post-partition writes are exactly what fencing discards to keep one history.

4. Stop replication and promote #

With the leader fenced and the follower caught up, stop replication to make the former follower writable, then repoint the write alias.

POST _plugins/_replication/logs-000001/_stop
{
  "leader_alias": "dc-primary",
  "leader_index": "logs-000001"
}

POST _aliases
{
  "actions": [
    { "add": { "index": "logs-000001", "alias": "logs-write", "is_write_index": true } }
  ]
}

Gotcha: the _stop body must repeat the exact leader_alias and leader_index the replication was started with; a mismatch leaves the follower write-blocked and un-promotable, which under partition pressure looks identical to a hung cluster.

5. Reconcile the old leader as a follower after heal #

When the link returns, the old leader rejoins as a follower against the new leader. Drop its stale role and start replication in the reverse direction.

POST _plugins/_replication/logs-000001/_start
{
  "leader_alias": "dc-standby",
  "leader_index": "logs-000001",
  "use_roles": { "leader_cluster_role": "ccr_leader", "follower_cluster_role": "ccr_follower" }
}

Gotcha: if retention on the new leader still covers the old leader’s last checkpoint, this _start resumes incrementally; if not, it silently falls back to a full remote bootstrap. Watch the status for BOOTSTRAPPING versus SYNCING to tell which path you got.

Verification #

Confirm exactly one writer exists and replication now flows the new direction.

# Only the promoted cluster should own the write alias
GET logs-write/_alias
# promoted cluster: { "logs-000001": { "aliases": { "logs-write": { "is_write_index": true } } } }
# fenced cluster:   write block present, alias NOT is_write_index

# Reverse replication is healthy
GET _plugins/_replication/logs-000001/_status
# "status": "SYNCING" with a shrinking checkpoint gap

A correct end state has the write alias resolving to one cluster only, the other cluster either write-blocked or replicating as a follower, and a bounded, shrinking checkpoint gap.

Common failures #

Frequently asked questions #

Related #

• Split-Brain Prevention — the split-brain model and fenced-failover automation this runbook executes.
• Tuning CCR checkpoint retention to avoid remote resync — sizing retention so step 5 resumes instead of bootstrapping.
• Follower Policy Design — keeping the follower write-blocked and terminal-action-safe.

Up: Split-Brain Prevention