OpenSearch ISM & Cross-Cluster Replication

Diagnosing stuck ISM policy transitions

When an OpenSearch Index State Management (ISM) index sits in a state whose transition conditions look satisfied — the age is past min_index_age, the size is over min_size — yet the index never advances, the fault is almost always in the execution layer between the policy and the scheduler, not in the conditions themselves. This walkthrough reads _plugins/_ism/explain for the three fields that reveal the real cause (retry_count, failed, and info), checks that the background sweep is actually evaluating the index, and clears the stall with _plugins/_ism/retry.

A transition that “should have fired an hour ago” is a diagnosis problem before it is a fix problem: _cat/indices shows a healthy green index and gives no hint that ISM has quietly parked it. This is the detection half of the parent Error Handling & Retries discipline — you first prove where the lifecycle stopped, then decide whether the sibling automated recovery in Implementing retry logic for stuck ISM transitions applies, or whether the condition itself is misconfigured back in Phase Transition Logic.

Decision tree for locating why an ISM transition never fires A top-to-bottom decision tree rooted at GET _plugins/_ism/explain. Node one checks whether the index is managed: if enabled is false or policy_id is absent it was never attached. Node two checks action.failed: if true, read info.cause, clear the block, then POST _plugins/_ism/retry. Node three checks whether retry_count reached the policy retry.count ceiling: if exhausted, only an explicit retry re-arms execution. Node four re-evaluates the transition condition against live age, size and doc count: a disagreement means the condition is misconfigured, not stuck. Node five compares explain start_time to job_interval to see whether the background sweep has run since the condition became true: a lagging sweep resolves on the next tick with no action. GET _ism/explain read failed · retry_count · info managed? enabled & policy_id action failed? action.failed == true retries exhausted? retry_count ≥ retry.count condition really met? re-check live age · size sweep lag → wait one job_interval never attached → add policy clear block → retry re-arm with _ism/retry fix condition, not runtime

Prerequisites

Confirm each item before you conclude an index is “stuck” — several of these rule out the common false alarm where the transition was simply going to fire on the next sweep.

Step-by-step procedure

The order matters: prove the index is managed, then read the failure fields, then rule out an exhausted retry budget, and only then question the condition itself.

1. Confirm the index is actually managed by ISM

An index that was created before its policy was attached, or that lost its policy in a template change, is not “stuck” — ISM is simply not looking at it. The explain endpoint is the authoritative check; _cat/indices cannot show management status.

HTTP
GET _plugins/_ism/explain/logs-prod-000042?pretty
Text
{
  "logs-prod-000042": {
    "index.plugins.index_state_management.policy_id": "logs-hot-warm-cold",
    "index.opendistro.index_state_management.policy_id": "logs-hot-warm-cold",
    "enabled": true,
    "index_creation_date": 1752624000000
  },
  "total_managed_indices": 1
}

Gotcha: if policy_id is null or enabled is false, the index is unmanaged and no condition will ever fire — attach or re-enable the policy rather than retrying. A policy_id that is present but points at a policy the index no longer matches is the classic template-drift case covered separately for the phase-transition cluster.

2. Read the failure triad: failed, retry_count, and info

When the index is managed, expand the explain output for the current action. Three fields decide everything: action.failed, retry_count, and info.cause.

HTTP
GET _plugins/_ism/explain/logs-prod-000042?pretty
Text
{
  "logs-prod-000042": {
    "state": { "name": "hot", "start_time": 1752624300000 },
    "action": { "name": "transition", "start_time": 1752624300000, "failed": true },
    "step": { "name": "attempt_transition", "status": "failed" },
    "retry_count": 4,
    "info": {
      "message": "Failed to evaluate transition conditions",
      "cause": "index [logs-prod-000042] missing rollover_alias setting"
    }
  }
}

Gotcha: action.failed: true with a populated info.cause means ISM tried and hit a real error — the transition never even got to condition evaluation. Read info.cause literally; here the transition action cannot resolve the write index because plugins.index_state_management.rollover_alias is unset, so no age/size math is ever reached.

3. Determine whether the retry budget is exhausted

Every action carries an implicit or policy-defined retry block. Once retry_count reaches the policy’s retry.count ceiling, ISM stops re-attempting on its own — the index then stays failed forever with no further sweep activity, which is exactly what makes it look permanently stuck.

HTTP
GET _plugins/_ism/policies/logs-hot-warm-cold?pretty
Text
"retry": { "count": 3, "backoff": "exponential", "delay": "1m" }

Gotcha: when retry_count (4 above) already exceeds retry.count (3), the background sweep will not touch the index again no matter how long you wait. Only an explicit _plugins/_ism/retry call resets the counter and re-arms execution — waiting is futile at this point.

4. Re-evaluate the transition condition against live values

If action.failed is false and the index still has not moved, the condition may simply not be met despite appearances. Compare the policy’s transitions conditions against the live index age and size — operators routinely misread min_index_age (measured from index creation, not from the last document) or a store.size that counts replicas.

Shell
GET _cat/indices/logs-prod-000042?v&h=index,docs.count,pri.store.size,creation.date.string
Text
index             docs.count pri.store.size creation.date.string
logs-prod-000042  81034112   47.9gb         2026-07-16T00:00:00.000Z

Gotcha: min_index_age counts from creation.date, so an index re-created two hours ago will not satisfy a min_index_age: "7d" transition even though it is “full”. And min_size compares against primary store size — a store.size that looks over threshold in _cat/indices includes replicas and can mislead you by a factor of two.

5. Clear the block and re-arm with _plugins/_ism/retry

Once the root cause from Step 2 is fixed (here, set the missing rollover alias), reset the failure flag and counter. An empty body re-runs the failed step in the current state.

HTTP
POST _plugins/_ism/retry/logs-prod-000042
Text
{ "updated_indices": 1, "failures": false, "failed_indices": [] }

Gotcha: _retry only re-queues the step — actual execution still waits for the next sweep tick, so do not expect the state to advance instantly. If you must skip a permanently blocked action, pass {"state": "warm"} to jump straight to a known-good state, but only when skipping is safe for the lifecycle defined in Phase Transition Logic.

Verification

Confirm the index resumed its lifecycle and the failure fields reset.

Shell
# 1. Failure cleared and counter reset to zero?
curl -s "https://os.internal:9200/_plugins/_ism/explain/logs-prod-000042?pretty"
# "action": { "failed": false }, "retry_count": 0, "step": { "status": "starting" }
Shell
# 2. Did the state actually advance on the following sweep?
curl -s "https://os.internal:9200/_plugins/_ism/explain/logs-prod-000042?filter_path=**.state"
# "state": { "name": "warm" }   <- transition fired

A healthy result shows action.failed: false, retry_count: 0, and — after one job_interval — a state.name that has moved on from hot. If retry_count climbs again immediately, the Step 2 root cause is not truly cleared and you are back to a persistent block, not a transient stall.

Common failures

Symptom Root cause Fix command
explain shows policy_id: null on a full index Policy never attached (index predates policy or template drift) POST _plugins/_ism/add/logs-prod-000042 {"policy_id":"logs-hot-warm-cold"}
action.failed: true, info.cause mentions rollover_alias Transition action cannot resolve the write index without the alias PUT logs-prod-000042/_settings {"plugins.index_state_management.rollover_alias":"logs-write"} then POST _plugins/_ism/retry/logs-prod-000042
retry_count frozen above retry.count, no sweep activity Retry budget exhausted; ISM stopped attempting on its own POST _plugins/_ism/retry/logs-prod-000042 to reset the counter
Condition looks met but index never moves, failed: false min_index_age counts from creation, or min_size compares primaries only Recompute against creation.date and pri.store.size; adjust the transition value
Follower index “stuck” on a write action CCR follower cannot execute write actions locally Drive the transition on the leader; do not retry the follower

Frequently asked questions

Why does the transition never fire even though `min_index_age` has clearly passed?

Two usual causes: the sweep has not run yet (wait up to one job_interval), or action.failed is true and ISM stopped before evaluating the condition at all. Always read _plugins/_ism/explain first — if info.cause is populated, the condition math was never reached, so no amount of waiting helps.

What is the difference between `retry_count` and the policy `retry.count`?

retry_count is the live per-index counter of how many times ISM has re-attempted the failed step; retry.count is the policy-defined ceiling. When the live counter reaches or exceeds the ceiling, ISM stops retrying automatically and the index parks indefinitely until you call _plugins/_ism/retry, which resets the live counter to zero.

Does calling `_plugins/_ism/retry` advance the state immediately?

No. It clears the failure flag and re-queues the step, but execution still happens on the next background sweep tick. Expect up to one job_interval of delay before _plugins/_ism/explain shows the state moving on. If you need it faster for one index, an out-of-band trigger is more appropriate than shortening the OpenSearch cluster-wide sweep.

Up: Error Handling & Retries