Follower Policy Design

A Cross-Cluster Replication (CCR) follower index is a read-only, write-blocked mirror of a leader index, and that single property breaks most Index State Management (ISM) policies: the moment a policy reaches a rollover, force_merge, shrink, open, or close action, OpenSearch rejects the write against the follower and the managed index parks in a Failed state. This guide, part of Cross-Cluster Replication Operations, shows how to design a follower-safe lifecycle — one that manages tiering and expiry on a replica whose contents you are not allowed to mutate directly. The core discipline is to let the leader own every write action, let CCR replicate the resulting settings, and attach a follower-only policy that restricts itself to read-safe actions plus a delete that is gated on replication having been stopped first.

Getting this wrong is quiet and expensive: a follower cluster keeps ingesting replicated segments while its ISM policy silently fails on a force_merge, the managed-index history fills with retries, and an operator eventually deletes a follower that is still actively replicating — corrupting the follower’s checkpoint and forcing a full remote resync. Follower policy design exists to make the lifecycle deterministic on an index you can read but not write.

Follower Index Constraints and Safe Actions #

Before writing a line of policy JSON, map which ISM actions are physically possible on a follower. A follower carries the write block index.blocks.write: true (set by the replication engine, not by you), so every action that mutates shard contents or shard topology fails. Actions that only change metadata or allocation succeed, because they do not write into the follower’s Lucene segments. The table below is the contract the rest of this page builds on; it mirrors the tier model from Index Lifecycle Basics but re-scores each action for the follower case.

The decisive rows are allocation, snapshot, and the gated delete: these are the entire vocabulary a follower-only policy is allowed to speak. Any state that reaches for rollover or force_merge on a follower is a design error, not a tuning problem. Scoping who can even attach such a policy is a security concern covered in Security & Access Boundaries — a service account that can call _plugins/_ism/add on a follower but not _plugins/_replication/_stop will strand every delete.

A Follower-Safe State Machine #

The state machine below is the shape a follower-only policy must take. It never mutates shard contents. The follower spends its managed life in a replicating state whose only action is tier allocation; a terminal delete is unreachable until replication is explicitly stopped, which is modelled as an intermediate stop-follower state. The transition into stop-follower is what converts a read-only follower into a plain index that ISM is finally allowed to delete.

The reason stop-follower is a distinct state rather than a step inside the delete action is that stopping replication is an operation on the replication engine, not on the index, and it must succeed and settle before the index is writable. Modelling it as its own state gives ISM a clean transition boundary: the delete state is only entered once the _stop call has returned and the write block is genuinely gone. Trying to collapse the two — deleting straight from replicating — is exactly the failure the crossed-out guard depicts, and it is where the write-block conflicts covered later in this guide originate.

1. Bootstrap the Follower and Confirm the Write Block #

Start replication so the follower exists, then confirm it carries the write block ISM must respect. The follower is created by the replication start call, not by a template, and inherits the leader’s settings. Remote connection and start mechanics are covered in CCR Setup and Bootstrap; the snippet below is the minimum needed before attaching a policy.

PUT _plugins/_replication/logs-follower-000007/_start
{
  "leader_alias": "dc-primary",
  "leader_index": "logs-000007",
  "use_roles": {
    "leader_cluster_role": "ccr_leader_role",
    "follower_cluster_role": "ccr_follower_role"
  }
}

Confirm the write block is present — this is the property your policy design has to survive:

# The follower carries index.blocks.write=true, set by the replication engine
curl -s "https://<follower>:9200/logs-follower-000007/_settings?filter_path=**.blocks" | python -m json.tool

{
  "logs-follower-000007": {
    "settings": {
      "index": { "blocks": { "write": "true" } }
    }
  }
}

2. Author the Follower-Only ISM Policy #

The policy below encodes the state machine. Every action is drawn from the safe-action vocabulary: allocation to tier the follower’s shards, then a stop-follower state whose real work is done out-of-band (ISM has no native “stop replication” action, so the transition into delete is gated by a Python controller described in the automation section — the policy models the state so the controller has something to key on). The delete state runs only after replication is confirmed stopped.

{
  "policy": {
    "description": "Follower-safe lifecycle: tier via allocation, gate delete on stopped replication",
    "default_state": "replicating",
    "ism_template": {
      "index_patterns": ["logs-follower-*"],
      "priority": 50
    },
    "states": [
      {
        "name": "replicating",
        "actions": [
          {
            "allocation": { "require": { "data": "warm" }, "wait_for": false }
          }
        ],
        "transitions": [
          { "state_name": "cold_hold", "conditions": { "min_index_age": "14d" } }
        ]
      },
      {
        "name": "cold_hold",
        "actions": [
          {
            "allocation": { "require": { "data": "cold" }, "wait_for": false }
          }
        ],
        "transitions": [
          { "state_name": "stop_follower", "conditions": { "min_index_age": "90d" } }
        ]
      },
      {
        "name": "stop_follower",
        "actions": [
          { "read_only": {} }
        ],
        "transitions": [
          { "state_name": "delete", "conditions": { "min_index_age": "91d" } }
        ]
      },
      {
        "name": "delete",
        "actions": [
          { "delete": {} }
        ]
      }
    ]
  }
}

Attach it and register it against the follower explicitly rather than relying only on the template, so an already-created follower is picked up immediately:

PUT _plugins/_ism/policies/follower_safe_policy

POST _plugins/_ism/add/logs-follower-*
{
  "policy_id": "follower_safe_policy"
}

Note what is absent: no rollover, no force_merge, no shrink. The read_only action in stop_follower is deliberately the most aggressive write-adjacent call the policy makes, and even that only sets a metadata block — it does not touch segments. The one-day gap between stop_follower and delete (day 90 to 91) is the settling window the automation uses to actually stop replication.

3. Tier Follower Shards with Allocation #

Because allocation is metadata-only, it is the workhorse of a follower policy. A follower can live on a different tier layout than its leader; the follower cluster’s hardware envelope, not the leader’s, decides where replicated shards land. Confirm each allocation transition with the ISM explain endpoint before trusting it:

curl -s "https://<follower>:9200/_plugins/_ism/explain/logs-follower-000007?pretty"

{
  "logs-follower-000007": {
    "index.plugins.index_state_management.policy_id": "follower_safe_policy",
    "state": { "name": "cold_hold", "start_time": 1752... },
    "action": { "name": "allocation", "failed": false },
    "step": { "name": "attempting_to_apply_allocation", "step_status": "completed" }
  }
}

A failed: false on an allocation action is the signal that the follower’s lifecycle is progressing without touching the write block. If you instead see failed: true on a rollover or force_merge, the wrong policy is attached — remove it with POST _plugins/_ism/remove/logs-follower-000007 and re-add the follower-only policy.

4. Gate Delete on Stopped Replication #

The delete state must never be reached while replication is live. ISM cannot itself call _plugins/_replication/.../_stop, so the gate is enforced two ways working together: the policy holds the index in stop_follower for a settling day, and the Python controller in the next section stops replication for any follower that has entered stop_follower. Only once GET .../_status reports PAUSED or the replication metadata is gone is the follower a plain, writable index that delete can remove cleanly.

# Manually, the gate is: stop, confirm, then let ISM delete
curl -s -X POST "https://<follower>:9200/_plugins/_replication/logs-follower-000007/_stop" -H 'Content-Type: application/json' -d '{}'
curl -s "https://<follower>:9200/_plugins/_replication/logs-follower-000007/_status?pretty"

Stopping replication is irreversible for that index — a stopped follower cannot be resumed, only re-bootstrapped from the leader. That is acceptable at end-of-life (the index is about to be deleted) but is why the gate sits at day 90, not at ingestion time.

Production Automation for Follower Lifecycle #

The controller below closes the loop the policy cannot: it finds every follower sitting in the stop_follower state, stops its replication, verifies the write block is gone, and lets ISM’s own delete transition complete on the next sweep. It uses opensearch-py, structured logging, and narrow exception handling so it is safe to schedule. Scope its service account to only cluster:admin/plugins/replication/index/stop and indices:admin/settings on the follower pattern, per Security & Access Boundaries.

import logging
from opensearchpy import OpenSearch, exceptions

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("follower_lifecycle")

STOP_STATE = "stop_follower"
FOLLOWER_PATTERN = "logs-follower-*"


class FollowerLifecycleController:
    def __init__(self, client: OpenSearch):
        self.client = client

    def followers_in_stop_state(self) -> list[str]:
        """Return followers whose ISM policy has reached the stop_follower state."""
        try:
            explain = self.client.transport.perform_request(
                "GET", f"/_plugins/_ism/explain/{FOLLOWER_PATTERN}"
            )
        except exceptions.ConnectionError:
            logger.error("Cannot reach follower cluster for ISM explain.")
            return []
        pending = []
        for index, meta in explain.items():
            if index.startswith("_"):  # skip total_managed_indices key
                continue
            state = (meta.get("state") or {}).get("name")
            if state == STOP_STATE:
                pending.append(index)
        return pending

    def stop_replication(self, index: str) -> bool:
        """Stop CCR replication so the write block lifts before ISM deletes the index."""
        try:
            self.client.transport.perform_request(
                "POST", f"/_plugins/_replication/{index}/_stop", body={}
            )
        except exceptions.NotFoundError:
            logger.info("%s already detached from replication; nothing to stop.", index)
            return True
        except exceptions.TransportError as exc:
            logger.error("Failed to stop replication for %s: %s", index, exc)
            return False
        return self._write_block_cleared(index)

    def _write_block_cleared(self, index: str) -> bool:
        """Verify index.blocks.write is no longer set after stopping replication."""
        try:
            settings = self.client.indices.get_settings(
                index=index, params={"filter_path": "**.blocks.write"}
            )
        except exceptions.TransportError as exc:
            logger.error("Could not read settings for %s: %s", index, exc)
            return False
        blocks = (
            settings.get(index, {}).get("settings", {})
            .get("index", {}).get("blocks", {})
        )
        cleared = blocks.get("write") in (None, "false", False)
        level = logging.INFO if cleared else logging.WARNING
        logger.log(level, "%s write block cleared=%s", index, cleared)
        return cleared

    def reconcile(self) -> None:
        pending = self.followers_in_stop_state()
        if not pending:
            logger.info("No followers awaiting replication stop.")
            return
        for index in pending:
            if self.stop_replication(index):
                logger.info("%s ready for ISM delete transition.", index)
            else:
                logger.warning("%s NOT ready; ISM delete will be blocked.", index)


if __name__ == "__main__":
    client = OpenSearch(
        hosts=[{"host": "follower.data.svc", "port": 9200}],
        http_auth=("ism_follower_svc", "..."),
        use_ssl=True,
        verify_certs=True,
    )
    FollowerLifecycleController(client).reconcile()

Run it as a CronJob on a cadence tighter than the one-day stop_follower → delete gap — hourly is ample — and export a counter for followers that reach stop_follower but fail _write_block_cleared, because that gap is the leading indicator of the stuck-delete failure mode below. Because stop_replication is idempotent (a NotFoundError is treated as success), re-running the controller never double-stops or errors on an already-detached follower.

Operational Guardrails #

Follower policy safety is mostly about not doing things, but a few settings keep the design honest. The lag headroom formula below decides whether a follower is safe to tier down: only demote to cold once replication lag is small, or a cold-tier follower that is still catching up will throttle both recovery I/O and query latency. Define follower lag as

where $C_{leader}$ and $C_{follower}$ are the leader and follower global checkpoints. Only allow the cold_hold transition when $L < L_{max}$ for the follower’s tier.

Keep follower number_of_replicas decoupled from the leader — a follower used only for local DR can run with fewer replicas to save disk, since it re-bootstraps from the leader on loss anyway. Sizing the follower tiers follows the same envelope logic as Index Lifecycle Basics, applied to the follower cluster’s hardware rather than the leader’s.

Troubleshooting #

The recurring root cause behind the first three rows is a policy authored for a leader being attached to a follower. The durable fix is the ism_template scoping in Step 2, which binds the follower-only policy to the logs-follower-* pattern so a leader-shaped policy never lands on a follower in the first place. Diagnosing an already-failed write block in depth is the subject of Resolving ISM write-block conflicts on follower indices.

Frequently asked questions #

Related #

• Designing ISM policies for CCR follower indices — the step-by-step walkthrough of authoring the follower-only policy above.
• Resolving ISM write-block conflicts on follower indices — diagnosing and clearing an action that already failed against the follower write block.
• Index Lifecycle Basics — the state/transition model this follower-safe policy is a constrained form of.
• Security & Access Boundaries — scoping the service account that attaches follower policies and stops replication.

Up: Cross-Cluster Replication Operations