Resolving ISM write-block conflicts on follower indices

This is a diagnostic runbook for the specific incident where an Index State Management (ISM) action has already failed against a Cross-Cluster Replication (CCR) follower with a cluster_block_exception — the managed index is parked in Failed, and you need to identify which action hit the write block, unblock any legitimate terminal action, and retry the policy cleanly. It is the recovery counterpart to the preventative design in Follower Policy Design, for on-call engineers staring at a stuck follower rather than authoring a policy from scratch.

The workflow is: read the failure with _plugins/_ism/explain, decide whether the failed action belongs on the leader (most cases) or is a terminal action that just needs replication stopped, act accordingly, then _plugins/_ism/retry to clear the Failed state. Getting the decision right matters — retrying a force_merge on a live follower just fails again, while force-clearing a write block on a still-replicating follower corrupts its checkpoint.

Prerequisites #

• You can reach the follower cluster and call GET _plugins/_ism/explain/<follower>.
• You know whether the failed follower is expected to be at end-of-life (a legitimate delete) or mid-lifecycle (a write action that never belonged on the follower), per Follower Policy Design.
• A service account with cluster:admin/plugins/replication/index/stop is available for the case where a terminal action needs replication stopped first.
• You have confirmed the leader is healthy, so any action you push back to the leader can actually run.
• For retry backoff and idempotency patterns you understand the approach in Error Handling & Retries.

How a write-block conflict arises #

A follower carries index.blocks.write: true, set by the replication engine. When ISM’s worker executes an action that mutates segments or shard topology, the block converts the action into a cluster_block_exception and ISM records the step as failed. The follower keeps replicating; only the policy is stuck. That decoupling is important to internalise before you touch anything: the data on the follower is fine and still current, so there is no urgency to force the action through. The failed action is a policy-authoring bug surfacing at runtime, and the correct response is almost always to fix the policy or stop replication cleanly — never to fight the block.

Two error codes look nearly identical in the ISM explain output but demand opposite fixes. FORBIDDEN/8/index write (api) is the replication-owned write block, cleared only by stopping replication. FORBIDDEN/12/index read-only / allow delete (api) is the disk flood-stage block that OpenSearch sets when a node crosses the flood_stage watermark, cleared by freeing disk and resetting index.blocks.read_only_allow_delete. Misreading a /12 as a /8 sends you to stop replication when the real problem is a full cold tier. The diagram below traces the decision for the CCR write block specifically; the code-check in step 1 tells you which block you are actually looking at.

Step-by-step procedure #

1. Read the failure with ISM explain #

Pull the managed-index state and the exact failed action. The info field carries the cluster_block_exception message that names the write block.

curl -s "https://<follower>:9200/_plugins/_ism/explain/logs-follower-000007?pretty"

{
  "logs-follower-000007": {
    "state": { "name": "warm" },
    "action": { "name": "force_merge", "failed": true },
    "step": { "name": "attempt_call_force_merge", "step_status": "failed" },
    "info": {
      "message": "Failed to start force_merge",
      "cause": "cluster_block_exception ... index [logs-follower-000007] blocked by: [FORBIDDEN/8/index write (api)]"
    }
  }
}

Gotcha: FORBIDDEN/8/index write (api) is the replication-owned write block, not a disk flood-stage read-only block (FORBIDDEN/12). Confirm the block number — a /12 block means a full disk, a different problem entirely.

2. Classify the failed action #

Decide which branch you are on. If the failed action is rollover, force_merge, shrink, open, or close, it never belonged on a follower and must move to the leader. If it is delete (or snapshot before delete), it is legitimate but needs replication stopped first.

# Confirm which actions the follower policy defines, to see if a leader-only action leaked in
curl -s "https://<follower>:9200/_plugins/_ism/explain/logs-follower-000007?show_policy=true&pretty" \
  | grep -oE '"(rollover|force_merge|shrink|delete)"'

"force_merge"
"delete"

Gotcha: seeing force_merge in a follower policy at all confirms a leader-shaped policy was mis-attached. The retry in step 5 will not stick until you remove that action.

3a. Write action — push it to the leader and detach from the follower #

For a leaked write action, remove the follower’s current policy and attach the follower-safe one. The write action’s result (merged segments) will replicate down from the leader once you run it there.

POST _plugins/_ism/remove/logs-follower-000007

POST _plugins/_ism/add/logs-follower-000007
{ "policy_id": "follower_safe_policy" }

Then run the actual merge on the leader, where it is a legal write:

curl -s -X POST "https://<leader>:9200/logs-000007/_forcemerge?max_num_segments=1"

The merged segments replicate down to the follower through CCR’s normal segment shipping, so the follower ends up with the same one-segment-per-shard layout you wanted — without ISM ever issuing a write against it. This leader-owns-writes discipline is the whole reason the follower policy in Follower Policy Design contains no force_merge: the follower inherits the result rather than performing the action.

3b. Terminal delete — stop replication to lift the block #

If the follower is genuinely at end-of-life and the failed action is delete, stop replication. Stopping is what removes the engine’s write block and turns the follower into a plain, deletable index.

curl -s -X POST "https://<follower>:9200/_plugins/_replication/logs-follower-000007/_stop" \
  -H 'Content-Type: application/json' -d '{}'

{ "acknowledged": true }

Gotcha: stopping is irreversible — a stopped follower cannot resume, only re-bootstrap from the leader. Only do this when the index is about to be deleted, never to “temporarily” push an action through mid-lifecycle.

4. Confirm the write block is gone #

Before retrying, verify the block actually lifted. A retry against a still-blocked index just re-fails and burns a retry attempt.

curl -s "https://<follower>:9200/logs-follower-000007/_settings?filter_path=**.blocks.write"

{}

An empty object means no write block remains. If {"...":{"settings":{"index":{"blocks":{"write":"true"}}}}} still comes back, replication did not fully stop — re-check _plugins/_replication/<follower>/_status.

5. Retry the policy to clear the Failed state #

_plugins/_ism/retry re-runs the failed step from where it stopped. It is safe to call repeatedly; it only advances if the underlying block is resolved, mirroring the idempotent retry discipline in Error Handling & Retries.

curl -s -X POST "https://<follower>:9200/_plugins/_ism/retry/logs-follower-000007?pretty"

{ "updated_indices": 1, "failures": false }

Verification #

Confirm the managed index left the Failed state and settled on a follower-safe step.

curl -s "https://<follower>:9200/_plugins/_ism/explain/logs-follower-000007?pretty"

{
  "logs-follower-000007": {
    "state": { "name": "cold_hold" },
    "action": { "name": "allocation", "failed": false },
    "step": { "name": "attempting_to_apply_allocation", "step_status": "completed" }
  }
}

For the terminal-delete case, confirm the index is actually gone rather than still parked:

curl -s "https://<follower>:9200/_cat/indices/logs-follower-000007?v"

# (empty) — index removed after replication stop + retry

A healthy result is failed: false on an allocation action for the mid-lifecycle case, or a removed index for the end-of-life case. A recurring failed: true on the same write action means step 3a was skipped — the leaked action is still in the attached policy.

Common failures #

Frequently asked questions #

Related #

• Follower Policy Design — the preventative design that stops these conflicts happening, plus the replication-stop controller.
• Designing ISM policies for CCR follower indices — authoring a follower-only policy so no write action can leak onto a follower.
• Error Handling & Retries — the idempotent retry and backoff patterns this runbook’s _ism/retry step relies on.

Up: Follower Policy Design