Choosing between force_merge and shrink for warm indices

When a rolled-over log index lands on the warm tier it is usually both over-segmented and over-sharded, and the two ISM actions that fix those problems — force_merge and shrink — are frequently confused, run in the wrong order, or run against an index that fails their preconditions. This procedure separates the two decisions: force_merge (max_num_segments) reduces the segment count to cut per-segment memory and query overhead, while shrink (num_new_shards) reduces the primary shard count and requires every primary on one node plus a read-only source. It shows both actions in a single warm state, the mandatory shrink-then-force_merge ordering, the preconditions each enforces, and how to verify the result. It sits under ISM Policy Actions Comparison.

Prerequisites #

Confirm each of these before adding the actions to your warm state — the usual first-run failure is a shrink that aborts because the source shards were never colocated or the index was never made read-only.

• The index has already rolled over and is no longer the write index, so no active ingest fights the merge — the roll is set up in Rollover Trigger Configuration.
• Warm nodes carry the data: warm routing attribute the allocation step targets, per Node Role Allocation, so shards can colocate for the shrink.
• The target new-shard count divides the source shard count evenly (e.g. 6 → 3 → 2 → 1); a non-divisor value fails validation.
• There is a warm node with enough free disk to hold a full second copy of the index during the shrink, below the high watermark defined in Hot-Warm-Cold Tier Design.
• A scoped service account can call _plugins/_ism/explain and _cat/segments for verification.

Which action solves which problem #

The two actions look similar — both rewrite the index and both are one-way — but they attack different overhead. The decision flow below routes a warm index to shrink, force_merge, both, or neither based on its shard count and segment count.

The reason the order is fixed: shrink builds a new index with fewer shards and deletes the source, so any force_merge done first is thrown away with the source. Merge the shrunk target and you rewrite the data exactly once.

Sizing the target shard count #

Pick the shrink target from the physical shard-size sweet spot, not from a round number. If $S$ is the total primary store size and $T$ is your target per-shard size (20–50 GB is the usual warm-tier band), the target primary shard count is

so a 90 GB index aiming at ~45 GB shards wants 2 primaries — and if it started at 6, then 6 mod 2 = 0 makes 2 a legal target. The memory you reclaim with the subsequent force_merge is roughly proportional to the segment count you remove, since each segment carries fixed per-segment structures:

where $m_{seg}$ is the average per-segment heap overhead. Merging 40 segments down to 1 reclaims 39 segments’ worth of that fixed overhead per shard.

Step-by-step procedure #

1. Make the source read-only and colocate its primaries #

shrink requires every primary shard on a single node and the source blocked for writes. Do both in one settings call before the shrink runs.

PUT logs-000042/_settings
{
  "index.blocks.write": true,
  "index.routing.allocation.require._name": "warm-node-3"
}

{ "acknowledged": true }

Gotcha: index.blocks.write blocks writes but still allows metadata changes; do not use index.blocks.read_only, which blocks the shrink’s own settings update and makes the action fail.

2. Encode both actions in one warm state, in order #

Put shrink ahead of force_merge in the same state’s actions array. ISM runs a state’s actions sequentially, so the shrunk target is what gets merged.

{
  "name": "warm",
  "actions": [
    { "shrink": { "num_new_shards": 2, "switch_aliases": false } },
    { "force_merge": { "max_num_segments": 1 } },
    { "allocation": { "require": { "data": "warm" } } }
  ],
  "transitions": [
    { "state_name": "cold", "conditions": { "min_index_age": "30d" } }
  ]
}

Gotcha: if you reverse the two actions, the force_merge runs on the soon-to-be-discarded source and the new shrunk index arrives un-merged — you pay the merge I/O twice.

3. Attach the policy and let the state run #

Register the policy and confirm the index is picked up on the next sweep.

curl -s -X POST "https://<cluster>:9200/_plugins/_ism/change_policy/logs-000042" \
  -H "Content-Type: application/json" \
  -d '{"policy_id": "warm-optimize-policy"}'

{ "updated_indices": 1, "failures": false }

Gotcha: the shrink produces a new index named shrink_logs-000042 by default. If a downstream alias or query targets the original name, set switch_aliases or repoint the alias explicitly after the shrink completes.

Verification #

Confirm each action ran and produced the intended physical layout. Run all three checks.

# 1. Which action is the index currently on, and did it fail?
GET _plugins/_ism/explain/logs-000042
# "action": { "name": "force_merge", "failed": false }

# 2. Did the shard count actually drop?
GET _cat/indices/shrink_logs-000042?v&h=index,pri,rep,store.size
# index               pri rep store.size
# shrink_logs-000042  2   1   88.4gb        <- 2 primaries, down from 6

# 3. Did the merge collapse segments to one per shard?
GET _cat/segments/shrink_logs-000042?v&h=shard,segment,size
# shard 0 -> a single segment; shard 1 -> a single segment

A healthy result shows the shrunk index at the target primary count, exactly one segment per shard, and the ISM action reporting failed: false. The tier placement that finishes the state follows the data tier routing patterns — the shards should now sit on data: warm nodes.

Common failures #

Frequently asked questions #

Related #

• ISM Policy Actions Comparison — how force_merge and shrink compare against every other ISM action.
• Hot-Warm-Cold Tier Design — sizing the warm tier that runs these layout rewrites.
• Node Role Allocation — colocating primaries via routing attributes so shrink can run.

Up: ISM Policy Actions Comparison