OpenSearch ISM & Cross-Cluster Replication

Recovering unassigned shards after a tier migration

When an OpenSearch Index State Management (ISM) allocation action moves an index from the hot tier to the warm tier and its shards drop to UNASSIGNED, the index goes yellow or red the instant the write path expected it to keep serving reads. The cause is nearly always that the routing attribute ISM stamped onto the index matches no eligible node — a missing node.attr.data: warm, an awareness rule that forbids the placement, or a filter that excludes every warm node at once. This procedure reads the decisive _cluster/allocation/explain output, repairs the routing attribute or awareness constraint, and forces the shards back onto the warm tier.

An ISM allocation action only writes index.routing.allocation.require.<attr> settings; it does not verify that any node can satisfy them. So a policy that is correct on paper strands shards the moment the warm fleet’s attributes drift from what the policy names. This is the failure mode behind the parent Data Tier Routing Patterns model, and it is distinct from the disk-pressure case in the sibling Resolving watermark-blocked cold-tier allocation — here the nodes have space but the wrong labels, which ties directly back to how tiers map to labels in Node Role Allocation.

From an ISM allocation action to a recovered warm-tier shard Left to right. An ISM allocation action writes index.routing.allocation.require.data equals warm. The OpenSearch cluster-manager runs the allocation decider, which checks three gates in sequence: attribute match against node.attr.data equals warm, allocation awareness across zones, and include/exclude filters. All gates passing yields a STARTED shard on a warm node. Any gate failing leaves the shard UNASSIGNED; _cluster/allocation/explain returns decision NO naming the failing decider, which maps to one of three fixes — relabel or restart the node, correct the awareness attribute list, or clear the exclude filter — followed by POST _cluster/reroute with retry_failed true. ISM allocation require.data: warm hot → warm 1 · attribute match node.attr.data == warm? 2 · awareness zone rule permits? 3 · include/exclude a candidate remains? all pass shard STARTED on a warm node any NO UNASSIGNED explain names the decider fix → _cluster/reroute

Prerequisites

Confirm each item before forcing shards back onto a tier — most unassigned-shard incidents are a label mismatch you can prove in one API call, not a hardware fault.

Step-by-step procedure

Diagnose before you reroute — a blind _cluster/reroute against a real constraint just fails again and burns the per-shard allocation retry budget.

1. Identify which shards went unassigned and why the state changed

Start with cluster health scoped to the migrated index, then list the unassigned shards with their unassigned.reason so you can tell an ISM-driven move from a node departure.

Shell
GET _cat/shards/logs-prod-000042?v&h=index,shard,prirep,state,node,unassigned.reason
Text
index             shard prirep state      node unassigned.reason
logs-prod-000042  0     p      STARTED    warm-1
logs-prod-000042  0     r      UNASSIGNED      NODE_LEFT
logs-prod-000042  1     p      UNASSIGNED      ALLOCATION_FAILED
logs-prod-000042  1     r      UNASSIGNED      ALLOCATION_FAILED

Gotcha: ALLOCATION_FAILED after an ISM allocation action points at a routing/attribute mismatch, while NODE_LEFT points at a departed node. If you see ALLOCATION_FAILED, the shard has already burned retries and will not re-attempt on its own until you reroute with retry_failed.

2. Read the decisive _cluster/allocation/explain output

This is the one call that names the failing decider. Target a specific unassigned primary rather than letting the API pick an arbitrary shard.

HTTP
POST _cluster/allocation/explain
{
  "index": "logs-prod-000042",
  "shard": 1,
  "primary": true
}
Text
{
  "index": "logs-prod-000042", "shard": 1, "primary": true,
  "can_allocate": "no",
  "node_allocation_decisions": [
    {
      "node_name": "warm-1",
      "node_decision": "no",
      "deciders": [
        { "decider": "filter", "decision": "NO",
          "explanation": "node does not match index setting [index.routing.allocation.require.data] filters [data:\"warm\"]" }
      ]
    }
  ]
}

Gotcha: the filter decider firing on require.data:"warm" means the node simply lacks node.attr.data: warm — the shard is unassignable because no node carries the label the policy demanded, not because the node is unhealthy. Read the decider field literally; a awareness or disk_threshold decider points at a completely different fix.

3. Repair the routing attribute or awareness constraint

Fix the mismatch named by the decider. If the warm nodes are missing the label, add it to opensearch.yml and restart, or — for an immediate unblock — relax the index requirement to a label the fleet actually carries.

Shell
# On each warm node's opensearch.yml, then rolling-restart:
# node.attr.data: warm

# Or, to unblock immediately without a restart, repoint the index requirement:
PUT logs-prod-000042/_settings
{
  "index.routing.allocation.require.data": "warm"
}

If an awareness rule is the blocker, confirm the warm nodes actually span the required zones:

Shell
GET _cat/nodeattrs?v&h=node,attr,value | grep -E 'data|zone'
Text
node   attr  value
warm-1 data  warm
warm-1 zone  a
warm-2 data  warm
warm-2 zone  a

Gotcha: with awareness.attributes: zone, having every warm node in zone: a forces a replica to stay unassigned because awareness forbids two copies in one zone. Either add a warm node in zone: b or, for a single-zone warm tier, drop zone from the awareness attributes for this tier — the trade-off is discussed under Fallback Routing Strategies.

4. Force a fresh allocation attempt with _cluster/reroute

Shards that hit ALLOCATION_FAILED will not retry until you clear the retry counter. Reroute with retry_failed after the constraint is fixed.

HTTP
POST _cluster/reroute?retry_failed=true
Text
{ "acknowledged": true, "state": { "routing_table": { "indices": { "logs-prod-000042": {
  "shards": { "1": [ { "state": "INITIALIZING", "node": "warm-2" } ] } } } } } }

Gotcha: retry_failed=true is what actually re-arms shards that exhausted index.allocation.max_retries (default 5) — a plain _cluster/reroute without it leaves ALLOCATION_FAILED shards untouched and you will wrongly conclude the fix did not work.

Verification

Confirm every shard is STARTED on a warm node and the index is green again.

Shell
# 1. All shards assigned and on the warm tier?
curl -s "https://os.internal:9200/_cat/shards/logs-prod-000042?v&h=shard,prirep,state,node"
# 0 p STARTED warm-1 / 0 r STARTED warm-2 / 1 p STARTED warm-2 / 1 r STARTED warm-1
Shell
# 2. Index health back to green with zero unassigned?
curl -s "https://os.internal:9200/_cluster/health/logs-prod-000042?pretty"
# "status": "green", "unassigned_shards": 0, "active_shards_percent_as_number": 100.0

A healthy result shows all primaries and replicas STARTED on nodes carrying node.attr.data: warm, status: green, and unassigned_shards: 0. If allocation/explain still returns can_allocate: no after the reroute, the decider named in Step 2 is not the one you fixed — re-read it rather than rerouting again.

Common failures

Symptom Root cause Fix command
filter decider: node does not match require.data:"warm" No warm node carries node.attr.data: warm Add the attr in opensearch.yml + restart, or repoint require.data to an existing label
awareness decider blocks the replica All warm nodes share one zone; awareness forbids two copies per zone Add a warm node in another zone, or drop zone from awareness for this tier
Reroute acknowledged but shards stay UNASSIGNED ALLOCATION_FAILED retry budget exhausted POST _cluster/reroute?retry_failed=true
disk_threshold decider, not filter Warm nodes past the high watermark — disk, not labels Free space or follow the watermark recovery guide
Shards allocate then move straight back to hot Index still carries a stale require.data: hot from a prior state PUT /<index>/_settings {"index.routing.allocation.require.data":"warm"}

Frequently asked questions

Why did shards go unassigned when the ISM `allocation` action reported success?

The allocation action only writes the routing settings onto the index; it returns success as soon as the settings are applied, without checking that any node can satisfy them. If no node carries the required attribute, the OpenSearch cluster-manager then fails to place the shards and they go UNASSIGNED. Always confirm the target tier’s node attributes match what the policy writes.

What is the difference between `require`, `include`, and `exclude` routing?

require demands a node match all listed attributes, include allows a node matching any listed attribute, and exclude forbids nodes matching the listed attributes. An ISM tier migration usually writes require.data, which is the strictest — a single mismatched label leaves shards with no eligible node, whereas include degrades more gracefully.

Do I need `retry_failed=true`, or will shards eventually re-allocate on their own?

Once a shard hits ALLOCATION_FAILED it has exhausted index.allocation.max_retries (default 5) and will not re-attempt automatically, even after you fix the constraint. _cluster/reroute?retry_failed=true resets that counter and forces a fresh attempt — without it the shards stay unassigned indefinitely.

Up: Data Tier Routing Patterns