OpenSearch ISM & Cross-Cluster Replication

Verifying bulk ISM policy attachment at scale

After a bulk attach touches thousands of indices, an HTTP 200 proves the request was accepted — not that every index is actually managed on the policy you intended. This walkthrough builds the verification pass that closes that gap: sweep _plugins/_ism/explain across the whole pattern, classify each index as managed-on-target, on-the-wrong-policy, unmanaged, or failed-to-initialise, then reconcile the stragglers. It is the confirm-and-reconcile half of the pipeline in Bulk Policy Attachment, and the natural next step after Attaching ISM policies to thousands of indices in bulk.

Prerequisites

Verification reads a lot of state and then writes reconcile actions, so confirm both sides of that before running — a read-only account cannot reconcile, and an unfiltered pattern makes the counts meaningless.

Step-by-step procedure

The pass runs in four stages: sweep explain across the pattern, classify every index into one of four buckets, report a coverage ratio against the expected denominator, and reconcile each straggler bucket with the right corrective call.

1. Sweep the explain endpoint across the pattern

GET _plugins/_ism/explain/<pattern> returns one entry per matching index plus a total_managed_indices tally. At scale, ask for the policy id and failure detail so the classify step has everything it needs in one response.

Python
from opensearchpy import OpenSearch

def sweep_explain(client: OpenSearch, pattern: str) -> dict:
    # show_policy surfaces the attached policy id; the response also carries
    # per-index state and any action failure.
    return client.transport.perform_request(
        "GET", f"/_plugins/_ism/explain/{pattern}?show_policy=true"
    )
JSON
{
  "logs-2026.06.01-000001": {
    "index.plugins.index_state_management.policy_id": "bulk_hot_delete",
    "state": { "name": "hot" }, "action": { "failed": false }, "enabled": true
  },
  "logs-2026.06.14-000003": {
    "index.plugins.index_state_management.policy_id": "bulk_hot_delete",
    "state": {}, "action": { "failed": true, "name": "rollover" }, "enabled": true
  },
  "total_managed_indices": 12586
}

Gotcha: total_managed_indices counts indices ISM manages, not indices on your policy. An index managed by some other policy still increments it, so never treat that number alone as success — the classify step below keys on the policy_id string.

2. Classify every index into four buckets

Walk the sweep response and sort each index into one of four states. This is the core of verification: a single index can be managed on the target and healthy, managed on the wrong policy, unmanaged entirely, or managed on the target but stuck with a failed action.

Python
def classify(explain: dict, target_policy: str) -> dict:
    buckets = {"ok": [], "wrong_policy": [], "unmanaged": [], "failed": []}
    for name, meta in explain.items():
        if name == "total_managed_indices" or not isinstance(meta, dict):
            continue
        policy = meta.get("index.plugins.index_state_management.policy_id")
        action = meta.get("action") or {}
        if policy is None:
            buckets["unmanaged"].append(name)
        elif policy != target_policy:
            buckets["wrong_policy"].append(name)
        elif action.get("failed"):
            buckets["failed"].append(name)
        else:
            buckets["ok"].append(name)
    return buckets
Text
ok=12401  wrong_policy=3  unmanaged=171  failed=11

Gotcha: an index with state: {} but policy_id set is attached-but-not-yet-initialised — the sweep has not reached it. Treat a small, shrinking unmanaged/empty-state count as “wait one sweep and re-check,” not a failure. Only a count that stays flat across two sweeps is a real straggler.

3. Report the coverage ratio

Reduce the buckets to a single coverage figure against the expected denominator N from the attach ledger. Coverage is the fraction of expected indices that are managed on the target policy and healthy:

C=∣ok∣NC = \frac{\lvert \text{ok} \rvert}{N}

Anything below 1.0 names exactly how many indices still need reconciliation, and the buckets say which kind.

Python
import logging
logger = logging.getLogger("ism.verify")

def report(buckets: dict, expected_total: int) -> float:
    ok = len(buckets["ok"])
    coverage = ok / expected_total if expected_total else 1.0
    logger.info(
        "coverage=%.4f ok=%d wrong_policy=%d unmanaged=%d failed=%d",
        coverage, ok, len(buckets["wrong_policy"]),
        len(buckets["unmanaged"]), len(buckets["failed"]),
    )
    return coverage
Text
[INFO] coverage=0.9853 ok=12401 wrong_policy=3 unmanaged=171 failed=11

Gotcha: use the expected total from the attach ledger as the denominator, not len(ok) + len(stragglers) from this sweep. If the attach pattern silently missed a whole day of indices, those never appear in the explain response at all, and a self-referential denominator would hide the miss behind a perfect-looking 1.0.

4. Reconcile each straggler bucket

Each bucket has a different corrective call. Unmanaged indices get a fresh add. Wrong-policy indices get change_policy — never add, which refuses a managed index. Failed-action indices get retry, which re-runs the stuck action in place.

Python
def reconcile(client, buckets, target_policy):
    if buckets["unmanaged"]:
        client.transport.perform_request(
            "POST", "/_plugins/_ism/add/" + ",".join(buckets["unmanaged"]),
            body={"policy_id": target_policy})
    if buckets["wrong_policy"]:
        client.transport.perform_request(
            "POST", "/_plugins/_ism/change_policy/" + ",".join(buckets["wrong_policy"]),
            body={"policy_id": target_policy})
    for name in buckets["failed"]:
        client.transport.perform_request("POST", f"/_plugins/_ism/retry/{name}")
Text
[INFO] reconcile: re-added 171, changed_policy 3, retried 11

Gotcha: reconcile is itself idempotent — re-running the whole verify-then-reconcile pass converges. But retry only helps if the underlying cause of the failed action is gone; retrying a rollover that failed for a missing write alias just fails again. Diagnose persistent failures through Error Handling & Retries before looping retry blindly.

The verify pass is a classify-and-route loop that either declares full coverage or feeds each straggler bucket to its matching corrective call:

Explain-based verification classifying indices into four buckets and routing stragglers to matched corrective calls Flow diagram. An explain sweep feeds a classifier that sorts each index into ok, wrong-policy, unmanaged, or failed. A coverage check follows: coverage of one means done. Otherwise unmanaged indices route to an add call, wrong-policy indices to a change-policy call, and failed indices to a retry call; all three feed back into a re-sweep that repeats until coverage reaches one. C = 1 C < 1 Explain sweep across pattern Classify 4 buckets Coverage C = |ok|/N against ledger N Done unmanaged POST add wrong_policy change_policy failed POST retry

Verification

Two independent cross-checks confirm the verify pass itself is honest — that the denominator is right and no stragglers are hiding behind an initialising sweep.

Shell
# 1. Does ISM's managed count reconcile with your expected total plus prior managed?
GET _plugins/_ism/explain/logs-2026.06-*?pretty
# "total_managed_indices": 12587   <- must equal ledger N + previously-managed
Shell
# 2. Are any target-policy indices still sitting on a failed action?
GET _plugins/_ism/explain/logs-2026.06-*?pretty | grep -A2 '"failed": true'
# expect no matches once retries have drained

A clean pass shows total_managed_indices equal to the expected denominator plus indices managed before the attach, and zero "failed": true entries under the target policy. If the managed count is short, the attach pattern missed indices that never entered the explain response — go back to the attach step rather than reconciling, because there is nothing in these buckets to reconcile.

Common failures

Symptom Root cause Fix command
Coverage looks perfect but indices are missing Denominator taken from the sweep, not the attach ledger Recompute C against the ledger’s expected N; re-list the pattern with _cat/indices
Large unmanaged bucket that never shrinks Coordinator sweep not initialising attached indices Confirm plugins.index_state_management.job_interval; re-add the bucket and wait a sweep
failed bucket refills after every retry Underlying action still cannot succeed (missing write alias, blocked tier) Diagnose via Error Handling & Retries; fix the root cause before POST _plugins/_ism/retry/<idx>
wrong_policy indices rejected by reconcile Used add (refuses managed) instead of change_policy POST _plugins/_ism/change_policy/<idx> with the target policy_id
explain request times out on huge patterns Single sweep over tens of thousands of indices Split the pattern by date sub-range and sweep each; aggregate the buckets client-side

Frequently asked questions

Why isn't `total_managed_indices` enough to confirm success?

Because it counts every index ISM manages under the pattern, regardless of which policy. An index left on an old policy, or managed by an unrelated one, still increments it. You can hit your expected total_managed_indices while dozens of indices sit on the wrong policy. Verification has to key on the per-index policy_id string, which is exactly what the classify step does.

How do I tell "still initialising" apart from "genuinely stuck"?

Compare two sweeps a sweep-interval apart. An index that is attached but shows an empty state in the first sweep and a concrete state in the second was simply waiting for the coordinator — normal. One that stays state-less or "failed": true across both sweeps is a real straggler and belongs in the reconcile pass. Never reconcile off a single sweep taken seconds after the attach.

Is it safe to run verification repeatedly?

Yes, and you should — the verify-then-reconcile pass is idempotent and convergent. Each run re-classifies from live state, so already-healthy indices fall into ok and are untouched, add skips managed indices, and change_policy/retry are no-ops once their target is correct. Schedule it to run a few times after a large attach so transient stragglers clear themselves before you page anyone.

Up: Bulk Policy Attachment