Tuning CCR checkpoint retention to avoid remote resync

When a Cross-Cluster Replication (CCR) follower briefly disconnects, whether it resumes from where it left off or re-copies the entire index across the link is decided by one thing: whether the leader still retains the operations the follower last acknowledged. This procedure sizes leader soft-deletes and translog retention so a short partition costs an incremental catch-up, not a full remote bootstrap.

Prerequisites #

Confirm these before tuning — retention that is too small silently degrades to a resync, and retention that is too large wastes leader disk on operations no follower will ever request.

• A leader-follower CCR pair exists and GET _plugins/_replication/logs-000001/_status returns SYNCING on the follower.
• You have a measured or estimated peak write rate for the leader index in operations per second and in bytes per second.
• You have a target maximum partition window — the longest disconnection you want to survive without a resync — as required by the Split-Brain Prevention recovery model.
• The follower index was created through a healthy bootstrap, per CCR Setup and Bootstrap, so its checkpoints align with the leader’s history.
• A scoped service account with indices:admin/settings/update on the leader index pattern is available to apply the settings.

How retention decides resume versus resync #

CCR is a translog-following mechanism: the follower pulls operations from the leader starting at the checkpoint sequence number it last acknowledged. The leader can only serve an operation that still exists — soft-deletes and translog retention are what keep historical operations alive past the point they would normally be merged away. If the follower reconnects while its last checkpoint is still retained, replication resumes from that sequence number. If the checkpoint has aged out, CCR has no incremental starting point and falls back to a full remote bootstrap, re-streaming every segment.

The retention you must keep is the number of operations that can accumulate during the longest partition you want to survive. With a peak write rate $R$ in operations per second and a target partition window $T$ in seconds, the minimum operation count to retain is:

where $\beta$ is a safety margin (typically $0.2$–$0.5$) covering burst above the measured peak. Retention is configured as a byte budget, so convert with the mean operation size $\bar{s}$ in bytes:

The incremental resume path holds only while the follower’s disconnection stays inside the retained window. Expressed as lag, if the follower is behind by $L$ operations when it reconnects, resume succeeds precisely when $L \le N_{\text{ops}}$; beyond that boundary the operations it needs are gone and a resync is unavoidable.

The crossing point $T_{\text{max}} = N_{\text{ops}} / R$ is the maximum partition you can survive on the current budget — the single number this whole exercise produces. Sizing retention is the recovery-cost half of the fencing discipline in Preventing split-brain during CCR network partitions; a clean fenced failover is wasted if reconnection then forces a full resync across an already-stressed link.

Step-by-step procedure #

1. Measure the peak write rate and mean operation size #

Pull indexing throughput and document counts so $R$ and $\bar{s}$ are measured, not guessed.

GET logs-000001/_stats/indexing,store

"indexing" : { "index_total" : 41880000, "index_time_in_millis" : 6980000 },
"store" : { "size_in_bytes" : 268435456000 }
# R  ≈ index_total / uptime_seconds  (ops/s at peak, not average)
# s̄ ≈ size_in_bytes / index_total    (mean bytes per operation)

Gotcha: use the peak rate, not the lifetime average from _stats — a nightly batch load can be ten times the daytime mean, and retention sized to the average resyncs every time the batch window overlaps a partition.

2. Compute the retention byte budget #

Apply the formula with your target window $T$ and safety margin $\beta$. For $R = 6{,}000$ ops/s, $T = 1800$ s (30 min), $\beta = 0.3$, and $\bar{s} = 512$ bytes: $N_{\text{ops}} = 6000 \times 1800 \times 1.3 \approx 1.40 \times 10^{7}$ ops, so $S_{\text{retain}} \approx 1.40 \times 10^{7} \times 512 \approx 7.2$ GB.

N_ops     = 6000 * 1800 * 1.3      = 14,040,000 ops
S_retain  = 14,040,000 * 512 bytes ≈ 7.2 GB
T_max     = 14,040,000 / 6000      = 2340 s (39 min headroom incl. margin)

Gotcha: retention is per-shard-aware but you configure a byte budget per index setting; if the index has many primaries, the effective per-shard retention is the budget divided across shards, so size against the busiest shard’s share of $R$, not the index total blindly.

3. Apply retention on the leader index #

Set the translog retention byte budget and the soft-deletes retention lease period. Both must be generous enough to cover $T_{\text{max}}$.

PUT logs-000001/_settings
{
  "index.plugins.replication.translog.retention_size": "8gb",
  "index.soft_deletes.retention_lease.period": "45m"
}

{ "acknowledged" : true }

Gotcha: round the byte budget up from the computed $S_{\text{retain}}$ (7.2 GB → 8 GB here) and set the lease period at or above $T_{\text{max}}$ (39 min → 45 min) so the two limits agree; the tighter of the two is what actually bounds recovery, and a mismatched pair silently caps you at the smaller window.

4. Verify the retention lease is held for the follower #

A retention lease is the leader’s promise to keep operations for a specific follower. Confirm the lease exists and its retained sequence number trails the follower’s checkpoint.

GET logs-000001/_stats?level=shards&filter_path=**.retention_leases

"retention_leases" : { "leases" : [
  { "id" : "peer_recovery/ccr_follower", "retaining_seq_no" : 90190, "source" : "ccr" }
] }

Gotcha: if retaining_seq_no is far behind the leader’s current checkpoint, that is healthy — it means the leader is holding history back for the follower. A retaining_seq_no that jumps forward to near-current means the lease expired and was renewed from scratch, which is the resync signal.

Verification #

Prove that a short disconnect resumes instead of bootstrapping.

# Pause then resume replication to simulate a brief disconnect
POST _plugins/_replication/logs-000001/_pause
POST _plugins/_replication/logs-000001/_resume
GET  _plugins/_replication/logs-000001/_status

"status" : "SYNCING"        <- resumed incrementally; a full resync would show BOOTSTRAPPING
"syncing_details" : { "leader_checkpoint" : 90512, "follower_checkpoint" : 90480 }

A resume shows SYNCING with a shrinking checkpoint gap and no bootstrap phase. If you see BOOTSTRAPPING, retention aged out during the pause — raise the budget and lease period and retest.

Common failures #

Frequently asked questions #

Related #

• Preventing split-brain during CCR network partitions — the failover runbook whose recovery step depends on this retention window.
• Split-Brain Prevention — the split-brain model tying retention to safe recovery.
• CCR Setup and Bootstrap — establishing the leader-follower pair whose checkpoints this tuning preserves.

Up: Split-Brain Prevention