Mapping data tiers to OpenSearch node roles

This guide gives the exact strings that bind each data tier — hot, warm, cold, frozen — to an OpenSearch node role and routing attribute, so Index State Management (ISM) transitions land shards on the tier you intend instead of parking them UNASSIGNED.

Deterministic tier routing depends on one thing being true end to end: the value in your opensearch.yml, the value in your index template, and the value in your ISM allocation action are byte-identical. When they drift, hot nodes absorb cold workloads, shard relocation storms saturate the network, and Cross-Cluster Replication (CCR) checkpoints stall. This procedure sits under Node Role Allocation and applies the broader OpenSearch ISM Architecture & Fundamentals execution model: it declares the tier attributes, stamps them onto new indices, maps each ISM phase to a node role, extends the mapping across a CCR follower, and audits for drift.

Prerequisites #

Confirm every item before you touch a live policy. A single mismatched attribute silently removes a node from a tier’s eligible pool without raising an error.

• Each data node group runs on distinct hardware (NVMe hot, SSD warm, HDD cold), sized against the ratios in Hot-Warm-Cold Tier Design.
• You have chosen one routing convention per cluster — native data_hot/data_warm/data_cold/data_frozen roles plus a matching node.attr.data value — and will hold it, per Data Tier Routing Patterns.
• An Index Template v2 owns the creation-time index.routing.allocation.require.data baseline.
• The automation service account holds fine-grained access to _plugins/_ism/*, _index_template/*, and _settings, scoped per Security & Access Boundaries.
• Disk watermarks (low, high, flood_stage) are set for multi-tier hardware, since they gate placement before the routing filter is ever evaluated.

Step-by-step procedure #

1. Declare tier roles and the matching attribute in opensearch.yml #

OpenSearch evaluates routing against explicit node role tags. Legacy nodes carrying only the generic data role bypass tier-aware allocation and fall back to capacity-based placement, which breaks ISM lifecycle guarantees. Set both the native tier role and the node.attr.data value ISM’s allocation action keys off — they are the same tier expressed two ways, and both must be present:

# opensearch.yml — a hot-tier data node
node.roles: ["data_hot", "ingest"]
node.attr.data: hot
node.attr.rack_id: az-1

Repeat per tier, swapping data_hot/hot for data_warm/warm, data_cold/cold, or data_frozen/frozen. Restart nodes sequentially to preserve quorum. Gotcha: node.attr.data values are case-sensitive. A node advertising Hot instead of hot is excluded from the hot pool but throws no error — it simply never matches a require.data: hot filter.

2. Verify role and attribute propagation #

Confirm OpenSearch resolves both the tier role and the attribute on every node before any policy references them:

curl -s "https://<cluster-endpoint>:9200/_cat/nodes?v&h=name,node.role,attr.data"

Expected output — each node lists its tier role letters and a populated attr.data column:

name        node.role attr.data
hot-node-1  himr      hot
warm-node-1 mr        warm
cold-node-1 mr        cold

Cross-check the raw roles with the nodes info API when the node.role letters are ambiguous:

curl -s "https://<cluster-endpoint>:9200/_nodes?filter_path=nodes.*.roles" | jq '.nodes[].roles'

Gotcha: if roles resolve to only data or data_content, the node will not be selected by tier-specific routing. Correct opensearch.yml and restart that node before deploying any lifecycle policy.

3. Stamp creation-time placement with an Index Template v2 #

New indices inherit routing from Index Template v2. Setting index.routing.allocation.require.data at creation pins every new index to the hot tier and establishes the baseline ISM later rewrites. The template value must equal the node.attr.data value from Step 1:

PUT _index_template/observability-hot-warm-cold
{
  "index_patterns": ["logs-app-*", "metrics-infra-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.routing.allocation.require.data": "hot",
      "index.refresh_interval": "15s",
      "index.plugins.index_state_management.policy_id": "observability-lifecycle"
    }
  },
  "priority": 500,
  "version": 2,
  "_meta": { "description": "Enforces hot-warm-cold routing for application telemetry" }
}

Gotcha: template priority must exceed any legacy v1 template or component template that also matches the pattern, or the older definition wins and your require.data baseline is dropped. Omitting the setting entirely routes new indices to any eligible data node, bypassing tier boundaries and exhausting NVMe capacity.

4. Map each ISM phase to its node role #

Index templates anchor creation routing; the ISM policy governs migration. Each state must declare its target tier through the allocation action, whose require block matches the node.attr.data value — it does not read _tier_preference, which is a separate index-level setting. Without an explicit require, shards stay pinned to their origin node regardless of age or size. The order in which routing changes relative to shrink and force_merge follows Index Lifecycle Basics.

PUT _plugins/_ism/policies/observability-lifecycle
{
  "policy": {
    "description": "Automated tier migration with strict allocation boundaries",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          { "rollover": { "min_primary_shard_size": "50gb", "min_index_age": "1d" } }
        ],
        "transitions": [{ "state_name": "warm", "conditions": { "min_index_age": "7d" } }]
      },
      {
        "name": "warm",
        "actions": [
          { "allocation": { "require": { "data": "warm" }, "wait_for": true } },
          { "shrink": { "num_new_shards": 1 } }
        ],
        "transitions": [{ "state_name": "cold", "conditions": { "min_index_age": "30d" } }]
      },
      {
        "name": "cold",
        "actions": [
          { "allocation": { "require": { "data": "cold" }, "wait_for": true } },
          { "force_merge": { "max_num_segments": 1 } }
        ]
      }
    ]
  }
}

Gotcha: wait_for: true blocks the state until relocation completes, preventing premature advancement. If no node carries the required attribute, the shards go UNASSIGNED and the lifecycle halts — the exact failure that Fallback Routing Strategies exist to absorb.

5. Align CCR follower routing to the same tier map #

A CCR follower index inherits routing from the leader only when you configure it to. Default behaviour replicates the leader’s shard placement, which mismatches the moment leader and follower run heterogeneous hardware. Override routing on the follower using the same node.attr.data value its own cluster declares:

PUT _plugins/_replication/logs-app-2026.01-follower/_start
{
  "leader_alias": "prod-cluster",
  "leader_index": "logs-app-2026.01",
  "settings": { "index.routing.allocation.require.data": "warm" }
}

Gotcha: the follower tier must offer equal or greater storage capacity than the leader tier it mirrors. A follower pointed at a smaller tier raises ALLOCATION_FAILED and stalls replication checkpoints. Validate cross-cluster tier parity before enabling auto-follow patterns.

6. Audit and correct tier drift #

Drift creeps in when manual relocations, node decommissions, or template overrides bypass ISM. This worker reads each index’s require.data setting, flags any value outside the expected set, and reapplies the correct attribute. _cat/indices does not expose routing settings, so it queries the settings API directly.

import requests
from urllib3.util.retry import Retry
from requests.adapters import HTTPAdapter

CLUSTER_URL = "https://<cluster-endpoint>:9200"
AUTH = ("automation-svc", "changeme")      # FGAC-scoped service account
EXPECTED_DATA_ATTRS = {"hot", "warm", "cold", "frozen"}

def get_session() -> requests.Session:
    session = requests.Session()
    session.auth = AUTH
    session.verify = "/path/to/ca-bundle.crt"
    retries = Retry(total=3, backoff_factor=0.5, status_forcelist=[502, 503, 504])
    session.mount("https://", HTTPAdapter(max_retries=retries))
    return session

def audit_tier_alignment(session: requests.Session) -> list:
    """Return indices whose routing.allocation.require.data is missing or unexpected."""
    resp = session.get(
        f"{CLUSTER_URL}/_all/_settings"
        "?filter_path=*.settings.index.routing.allocation.require.data"
    )
    resp.raise_for_status()
    misaligned = []
    for idx_name, idx_cfg in resp.json().items():
        data_attr = (
            idx_cfg.get("settings", {}).get("index", {}).get("routing", {})
            .get("allocation", {}).get("require", {}).get("data", "")
        )
        if data_attr not in EXPECTED_DATA_ATTRS:
            misaligned.append(idx_name)
    return misaligned

def enforce_routing(session: requests.Session, index_name: str, target_tier: str) -> dict:
    payload = {"index.routing.allocation.require.data": target_tier}
    resp = session.put(f"{CLUSTER_URL}/{index_name}/_settings", json=payload)
    resp.raise_for_status()
    return resp.json()

if __name__ == "__main__":
    session = get_session()
    drift = audit_tier_alignment(session)
    if not drift:
        print("All indices aligned with expected tier routing.")
    else:
        print(f"Detected {len(drift)} misaligned indices. Reapplying warm-tier routing...")
        for idx in drift:
            try:
                enforce_routing(session, idx, "warm")
                print(f"  OK: {idx} routed to warm")
            except requests.exceptions.HTTPError as e:
                print(f"  FAIL: {idx}: {e.response.text}")

Gotcha: the default target here forces every drifted index to warm; in production, derive the correct tier from the index’s age or ISM state rather than assuming warm, or you will pull hot indices off NVMe. Schedule the audit as a Kubernetes CronJob every 15 minutes and alert when drift exceeds a threshold.

Verification #

After a transition, confirm the shards actually landed on the intended tier and that the ISM state completed.

Confirm the allocator chose a matching node:

curl -s -X POST "https://<cluster-endpoint>:9200/_cluster/allocation/explain" \
  -H "Content-Type: application/json" \
  -d '{"index": "logs-app-2026.01", "shard": 0, "primary": true}'

Inspect node_allocation_decisions: the selected node must carry attr.data equal to the phase’s require.data value with "node_decision": "yes". A filter decider returning "NO" means no node matched the attribute.

Confirm the ISM state advanced:

curl -s "https://<cluster-endpoint>:9200/_plugins/_ism/explain/logs-app-2026.01?pretty"

A healthy result shows the expected state.name, action.name: allocation, and step.status: completed. A retry_failed_count above zero points at a blocked allocation.

Confirm per-tier disk headroom:

curl -s "https://<cluster-endpoint>:9200/_cat/allocation?v&h=shards,disk.percent,disk.avail,node&s=disk.percent:desc"

The routing filter is evaluated only after the disk-watermark decider passes, so a shard can match its attribute and still stay UNASSIGNED when a node sits above watermark.high. Keep enough headroom that used bytes plus the incoming shard stay under the high watermark:

Common failures #

If shards are already parked, unblock them temporarily by relaxing watermarks, then revert once recovery completes — a persistent breach is a capacity problem, not a routing one:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%"
  }
}

Frequently asked questions #

Related guides #

• Node Role Allocation — the decider pipeline and tier-attribute contract this mapping enforces.
• Data Tier Routing Patterns — choosing require vs include vs exclude and detecting routing drift at scale.
• Fallback Routing Strategies — graceful degradation when a require filter matches no eligible node.
• Hot-Warm-Cold Tier Design — capacity ratios and watermark tuning behind each tier.

Up one level: Node Role Allocation · Foundations: OpenSearch ISM Architecture & Fundamentals