OpenSearch ISM & Cross-Cluster Replication

Designing ISM policies for CCR follower indices

This walkthrough authors a single follower-only Index State Management (ISM) policy from scratch: one that manages a read-only Cross-Cluster Replication (CCR) follower’s lifecycle using only allocation and metadata actions, and gates its final delete on replication having been stopped first. It is the hands-on procedure behind the design principles in Follower Policy Design, aimed at operators who need a follower cluster to tier and expire its replicas without ever attempting a write the follower will reject.

The goal is a policy you can attach to a logs-follower-* index and trust to progress to Completed rather than parking in Failed on the first rollover or force_merge. Every step below is written so the policy contains zero write actions and the only shard-topology change — deletion — happens after the follower is no longer a follower.

Prerequisites

Confirm each item before authoring the policy — a missing write alias is irrelevant here, but an unstopped-replication assumption or a leader-shaped template is the usual reason a first attempt fails.

How the policy stays follower-safe

The policy’s safety comes from what it omits. A follower’s write block rejects any action that mutates segments or shard layout, so the entire policy is built from allocation (routing metadata), read_only (a metadata block), and a delete reached only after an intermediate stop state. The diagram traces the one authoring decision that matters at each action: is this write-safe on a follower?

Choosing follower-safe ISM actions Decision flowchart. Start: candidate ISM action. Decision: does it write to shard segments or topology? If yes, follow the reject branch to "move to leader policy" because the follower write block rejects it. If no, follow to "safe: include in follower policy" covering allocation, read_only, and snapshot. A separate note shows delete is allowed only after a stop_follower state stops replication. candidate ISM action writes to segments? yes rejected by write block move to leader policy no safe: allocation, read_only, snapshot delete? only after _stop via stop_follower state

The one subtlety the diagram hides is precedence. When both an ism_template on the follower cluster and an inherited plugins.index_state_management.policy_id setting (replicated from the leader’s index settings) point at a policy, the explicitly-attached policy wins, but a leaked leader policy_id in the replicated settings is a common way a leader-shaped policy silently re-attaches after a follower is re-bootstrapped. The procedure below removes that ambiguity by attaching the follower policy explicitly and scoping it with a high-priority template, so neither an inherited setting nor a broad catch-all template can reintroduce a write action.

Step-by-step procedure

The steps build the policy in the order you reason about it: first inventory what the leader does, then translate only the follower-safe actions into tiering states, add the delete gate, assemble and scope the document, and finally attach it to indices that already exist.

1. Inventory the actions your leader policy uses

List the actions in the policy attached to the leader so you know which ones must stay on the leader and which have follower-safe equivalents. Anything that writes segments or shard topology is disqualified from the follower policy.

Shell
curl -s "https://<leader>:9200/_plugins/_ism/policies/logs_leader_policy?pretty" \
  | grep -oE '"(rollover|force_merge|shrink|allocation|snapshot|delete|read_only)"'
Text
"rollover"
"force_merge"
"allocation"
"delete"

Gotcha: rollover and force_merge in this output stay on the leader — CCR replicates their results down. Only allocation and a gated delete carry over to the follower policy.

2. Draft the follower states with allocation only

Write the two tiering states. Each uses allocation with wait_for: false so a temporarily full tier does not block the ISM worker; the next sweep re-attempts placement. This is the same non-blocking approach used for leaders, but here it is the only work these states do.

JSON
{
  "states": [
    {
      "name": "replicating",
      "actions": [
        { "allocation": { "require": { "data": "warm" }, "wait_for": false } }
      ],
      "transitions": [
        { "state_name": "cold_hold", "conditions": { "min_index_age": "14d" } }
      ]
    },
    {
      "name": "cold_hold",
      "actions": [
        { "allocation": { "require": { "data": "cold" }, "wait_for": false } }
      ],
      "transitions": [
        { "state_name": "stop_follower", "conditions": { "min_index_age": "90d" } }
      ]
    }
  ]
}

Gotcha: do not add a force_merge to cold_hold “to save space” — it is the single most common way a follower policy fails. Merge on the leader; the merged segments replicate down to the follower for free.

3. Add the stop-follower gate and the delete state

The stop_follower state parks the index for a settling day and sets a read_only metadata block (safe on a follower). The delete gate is enforced by the controller from Follower Policy Design, which stops replication for any follower in this state before ISM’s delete transition fires.

JSON
{
  "name": "stop_follower",
  "actions": [ { "read_only": {} } ],
  "transitions": [
    { "state_name": "delete", "conditions": { "min_index_age": "91d" } }
  ]
},
{
  "name": "delete",
  "actions": [ { "delete": {} } ]
}

Gotcha: the one-day gap between day 90 and day 91 is not cosmetic — it is the window the replication-stop controller needs to run and confirm the write block is gone. Collapse it and delete can fire while the follower is still replicating.

4. Assemble, register the policy, and scope it with an ism_template

Combine the states into a full policy document and PUT it, then bind it to the follower pattern with an ism_template so a leader-shaped policy can never auto-attach to a follower.

HTTP
PUT _plugins/_ism/policies/follower_safe_policy
{
  "policy": {
    "description": "Follower-only lifecycle: allocation tiering + replication-gated delete",
    "default_state": "replicating",
    "ism_template": { "index_patterns": ["logs-follower-*"], "priority": 50 },
    "states": [ ... states from steps 2 and 3 ... ]
  }
}

Gotcha: give the follower template a higher priority than any broad catch-all template, or a generic logs-* policy will win the match and attach leader actions to your follower.

5. Attach to existing followers and confirm the initial state

ism_template only auto-attaches on index creation, so any follower that was already replicating when you PUT the policy is unmanaged until you add it explicitly. Batch-add across the whole pattern in one call:

HTTP
POST _plugins/_ism/add/logs-follower-*
{ "policy_id": "follower_safe_policy" }
Text
{ "updated_indices": 7, "failures": false }

Gotcha: updated_indices should equal the number of live followers. A lower count means some followers already carry a different policy_id (often a leaked leader policy) and were skipped — remove that policy first with POST _plugins/_ism/remove/<follower>, then re-add.

Verification

Confirm the policy attached and that its first action is a clean allocation, not a failed write.

Shell
curl -s "https://<follower>:9200/_plugins/_ism/explain/logs-follower-000007?pretty"
Text
{
  "logs-follower-000007": {
    "index.plugins.index_state_management.policy_id": "follower_safe_policy",
    "state": { "name": "replicating" },
    "action": { "name": "allocation", "failed": false },
    "step": { "name": "attempting_to_apply_allocation", "step_status": "completed" }
  }
}

Cross-check that replication is still healthy — the policy must not have disturbed it:

Shell
curl -s "https://<follower>:9200/_plugins/_replication/logs-follower-000007/_status?pretty"
Text
{ "status": "SYNCING", "leader_alias": "dc-primary", "leader_index": "logs-000007" }

A healthy result is failed: false on the allocation action and SYNCING (or SYNCED) replication status. If the action shows failed: true naming rollover or force_merge, a leader-shaped policy is still attached — remove it and re-add the follower policy.

Common failures

Symptom Root cause Fix command
Action failed: true on rollover/force_merge Leader-shaped policy attached to the follower POST _plugins/_ism/remove/<follower> then re-add follower_safe_policy
Policy never auto-attaches to new followers Missing or low-priority ism_template Re-PUT policy with ism_template.priority above the catch-all template
Follower UNASSIGNED after cold_hold Cold tier full or missing data: cold nodes PUT <follower>/_settings {"index.routing.allocation.include.data":"warm,cold"}
delete transition never completes Replication still active; write block present Stop replication via the controller, then POST _plugins/_ism/retry/<follower>
security_exception attaching the policy Service account lacks cluster:admin/opendistro/ism/* Grant the ISM action per Security & Access Boundaries

Frequently asked questions

Can I reuse my leader policy on the follower if I just delete the write actions?

You can, and that is effectively what this procedure produces — but do it deliberately, not by trimming a copy. Start from the follower-safe vocabulary (allocation, read_only, snapshot, gated delete) so you never accidentally leave a force_merge behind. A single stray write action parks the whole managed index in Failed.

Why `read_only` in the stop_follower state if the follower is already write-blocked?

read_only sets an ISM-owned metadata block that persists after replication is stopped and the engine’s own write block is lifted. It keeps the index read-only during the settling day between stopping replication and deleting, so nothing writes to the briefly-writable index before it is removed.

Does the follower policy need to match the leader's retention exactly?

No. The follower can retain longer (for local DR) or shorter (to save follower-cluster disk) than the leader. Only the delete gate is non-negotiable: it must sit behind a stopped-replication check regardless of the retention numbers you choose.

Up: Follower Policy Design