OpenSearch ISM & Cross-Cluster Replication

Resolving watermark-blocked cold-tier allocation

When an OpenSearch Index State Management (ISM) policy moves an index to the cold tier and the shards refuse to allocate — or an already-cold index suddenly turns read-only — the culprit is usually the disk-based shard allocator refusing the cold nodes because they have crossed the high or flood_stage watermark. Cold nodes run deliberately dense, so they sit close to the watermark by design, and one large force_merge output or a fat incoming shard is enough to tip them over. This procedure measures each cold node against the watermark thresholds, frees or re-thresholds the tier, and clears the read-only-allow-delete block that flood_stage stamps onto the index.

Cold-tier density is the whole point of the tier — but density means the disk-threshold decider is the first thing to reject a placement, not the last. Unlike the label-mismatch case in the sibling Recovering unassigned shards after a tier migration, here the routing attributes are correct and the nodes simply have no room. Understanding why cold nodes run near the line is the capacity-planning story in Hot-Warm-Cold Tier Design, and the whole thing sits under the parent Data Tier Routing Patterns model.

Disk watermark bands and their effect on cold-tier ISM allocation A vertical usage gauge from zero at the bottom to one hundred percent at the top, split into four bands. Below eighty-five percent, normal: shards allocate freely. Eighty-five to ninety, low watermark: no new shards allocate to the node. Ninety to ninety-five, high watermark: OpenSearch relocates shards off the node, so an ISM cold migration cannot land. Above ninety-five, flood stage: indices with a shard on the node are forced read-only-allow-delete, blocking writes. Callouts map three fixes to the bands: free disk or add cold nodes, temporarily raise the watermark thresholds, and clear the read-only block after usage drops below flood stage. flood_stage ≥ 95% · read-only high 90–95% · relocate off low 85–90% · no new shards normal < 85% · allocates freely 100% 95% 90% 85% 0% ISM cold migration blocked ≥ high Fix 3 · clear read-only-allow-delete block once usage falls back under flood_stage Fix 2 · raise watermark thresholds (temporary) disk.watermark.high — buys time, not space Fix 1 · free disk or add cold nodes the only durable fix — drop usage below high

The disk decider works in percentage of used space, so translate your raw node stats before comparing against a threshold. For a node with total capacity CC and free bytes FF, the used percentage the allocator checks is:

used%=(1−FC)×100\text{used\%} = \left(1 - \frac{F}{C}\right) \times 100

A cold node blocks new cold-tier allocation once used%≥90\text{used\%} \ge 90 (the high watermark) and forces every index with a shard on it to read-only once used%≥95\text{used\%} \ge 95 (the flood_stage watermark). The headroom a single incoming shard of size ss needs is therefore F≥s+(1−0.90) CF \ge s + (1 - 0.90)\,C to land without tripping high.

Prerequisites

Confirm each item before touching a watermark setting — raising a threshold to unblock allocation without also freeing space just defers a flood_stage read-only event by hours.

Step-by-step procedure

Measure first: confirm the block is disk pressure and identify exactly which cold nodes are over the line before you change any threshold.

1. Confirm the block is a disk watermark, not a routing failure

Ask the allocator directly. The disk_threshold decider in _cluster/allocation/explain is unambiguous — it names the watermark and the node’s free space.

HTTP
POST _cluster/allocation/explain
{ "index": "logs-prod-000030", "shard": 0, "primary": true }
Text
{
  "can_allocate": "no",
  "node_allocation_decisions": [
    { "node_name": "cold-2", "node_decision": "no",
      "deciders": [
        { "decider": "disk_threshold", "decision": "NO",
          "explanation": "the node is above the high watermark cluster setting [cluster.routing.allocation.disk.watermark.high=90%], having less than the minimum required [12gb] free space, actual free: [6.4gb]" }
      ] } ]
}

Gotcha: if the decider named is disk_threshold you are in the right guide; if it is filter or awareness, the cold nodes have the wrong labels and you want the sibling tier-migration recovery page instead. Do not raise watermarks to “fix” a routing problem — it does nothing and masks the real cause.

2. Measure every cold node against the watermark

Read per-node disk usage so you know how far over the line the tier is and how much you must free.

Shell
GET _cat/allocation?v&h=node,disk.percent,disk.used,disk.avail,disk.total
Text
node   disk.percent disk.used disk.avail disk.total
cold-1 88           1.4tb     190gb      1.6tb
cold-2 96           1.53tb    64gb       1.6tb
cold-3 91           1.45tb    150gb      1.6tb

Gotcha: cold-2 at 96% is already past flood_stage, so every index with a shard there is read-only right now — not just the one ISM tried to migrate. cold-3 at 91% is past high, which is why the incoming shard cannot land. Only cold-1 (88%) is below high and eligible.

3. Free space or add cold capacity — the durable fix

The only fix that actually resolves the pressure is reducing usage below high. Delete or snapshot-and-delete the oldest cold indices (this is what the ISM delete action exists for), or add a cold node so shards rebalance.

HTTP
DELETE logs-prod-000004,logs-prod-000005
Text
{ "acknowledged": true }

Gotcha: prefer letting the ISM policy’s own delete state expire the oldest indices over ad-hoc DELETE calls — a manual delete that removes an index still inside its retention window can breach a compliance hold. Reach for the manual delete only to relieve an active flood_stage incident.

4. Temporarily raise the watermark to unblock relocation (buys time, not space)

If you cannot free space immediately, nudge the thresholds up transiently so relocation and allocation resume while you provision capacity. This is a stopgap, not a fix.

HTTP
PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "90%",
    "cluster.routing.allocation.disk.watermark.high": "93%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "97%"
  }
}
Text
{ "acknowledged": true, "transient": { "cluster": { "routing": { "allocation": {
  "disk": { "watermark": { "high": "93%", "flood_stage": "97%" } } } } } } }

Gotcha: use transient so the override evaporates on cluster restart and cannot silently become permanent — a persistently raised flood_stage means the node fills to 97% before protection kicks in, risking a wedged node with no recovery headroom. Revert these the moment capacity is added.

5. Clear the read-only-allow-delete block once usage drops below flood_stage

flood_stage sets index.blocks.read_only_allow_delete: true on affected indices, and OpenSearch does not auto-clear it on every version once disk recovers. Clear it explicitly after usage falls back under the threshold.

HTTP
PUT logs-prod-*/_settings
{ "index.blocks.read_only_allow_delete": null }
Text
{ "acknowledged": true }

Gotcha: setting the value to null removes the block; setting it to false can leave a stale explicit setting that confuses later diagnosis. Only clear it after _cat/allocation shows the node back under flood_stage, or it re-arms within one cluster-info refresh (default 30s).

Verification

Confirm the cold shards allocated and no index remains read-only.

Shell
# 1. Is the migrated index green with all shards on cold nodes?
curl -s "https://os.internal:9200/_cat/shards/logs-prod-000030?v&h=shard,prirep,state,node"
# 0 p STARTED cold-1 / 0 r STARTED cold-3
Shell
# 2. Are all cold nodes back under the high watermark?
curl -s "https://os.internal:9200/_cat/allocation?v&h=node,disk.percent" | grep cold
# cold-1 86 / cold-2 89 / cold-3 87
Shell
# 3. Any lingering read-only block?
curl -s "https://os.internal:9200/logs-prod-000030/_settings?filter_path=**.blocks"
# {}   <- empty means no block remains

A healthy result shows the shards STARTED on cold nodes, every cold node’s disk.percent below 90, and an empty blocks response. If disk.percent still reads above 90 after the fix, the space you freed was reclaimed by an incoming shard — add capacity rather than raising watermarks again.

Common failures

Symptom Root cause Fix command
disk_threshold decider: less than minimum free space Cold node past the high watermark (≥90%) Free disk / add a cold node, then POST _cluster/reroute?retry_failed=true
Index turned read-only; writes fail Node crossed flood_stage (≥95%), stamping read_only_allow_delete Free space, then PUT /<index>/_settings {"index.blocks.read_only_allow_delete": null}
Watermark raised but shards still won’t allocate Change applied to the wrong scope, or free space is below the absolute min_bytes form Set the percentage form transiently and confirm with GET _cluster/settings?include_defaults
Block returns seconds after being cleared Cleared while node was still above flood_stage Wait until _cat/allocation shows usage under the threshold, then clear again
Cold node fills again within hours Watermark raised instead of adding capacity Revert transient watermarks; provision cold nodes per the tier design guide

Frequently asked questions

Why is the cold tier hitting watermarks when the hot tier never does?

Cold nodes are provisioned for density — large, cheap disks holding many read-only shards — so they intentionally run much closer to the watermark than hot nodes. That leaves little headroom, so a single large force_merge output or an incoming migrated shard can push a cold node from high to flood_stage. Size the cold tier with explicit watermark headroom rather than filling it to the line.

Should I raise the watermark thresholds or free space?

Freeing space (or adding a cold node) is the only durable fix; raising the watermark just moves the line and gives OpenSearch less recovery headroom. Use a transient watermark bump only to unblock relocation while you provision capacity, and revert it immediately afterward — a permanently raised flood_stage risks a node filling to the point where it cannot even relocate shards off itself.

Why does the index stay read-only after disk recovers?

flood_stage sets index.blocks.read_only_allow_delete, and depending on your OpenSearch version that block is not always auto-cleared when disk usage drops. Clear it explicitly with PUT /<index>/_settings {"index.blocks.read_only_allow_delete": null} once _cat/allocation confirms the node is back under the threshold, or writes stay blocked indefinitely.

Up: Data Tier Routing Patterns