Index Lifecycle Basics

Index lifecycle basics are the rules that decide when an OpenSearch index rolls over, migrates to cheaper hardware, sheds replicas, and is finally deleted — expressed as a declarative Index State Management (ISM) policy OpenSearch runs on its own. In production telemetry and log workloads, hand-rolled rotation via external cron introduces configuration drift, uneven shard distribution, and unbounded storage growth: one missed job and a hot node crosses its flood-stage watermark and locks every index on it read-only. A policy replaces that fragility with a finite state machine the engine evaluates on a fixed interval, executing deterministic transitions against measurable thresholds. This model is the timing layer that the rest of OpenSearch ISM Architecture & Fundamentals builds on — it is what decides when an index moves, while node roles and routing decide where it lands.

How a lifecycle phase maps to cluster hardware #

A lifecycle policy only makes economic sense when each phase lands on hardware matched to that phase’s access pattern. The phase names in the policy (hot, warm, cold, delete) are arbitrary labels; the routing attribute each phase writes is what actually binds it to a hardware pool. Declaring and verifying those attributes is the job of Node Role Allocation, and sizing the pools so a migration wave never overruns the next tier is covered in Hot-Warm-Cold Tier Design. The table below is the reference every template and allocation action in this guide points back to.

The delete phase has no hardware because it holds no data — it exists only to run the delete action once the retention threshold is met. Everything upstream of it is a trade of query latency for bytes-per-dollar, which is why the routing attribute, not the phase name, is the load-bearing part of the policy.

The policy state machine #

An ISM policy is a JSON document that defines a finite state machine for one index (or index alias). Each policy is a set of states; each state carries actions it runs on entry and transitions it evaluates on every poll. The coordinator re-evaluates every managed index at the interval set by plugins.index_state_management.job_interval (default 5 minutes). When a transition’s conditions evaluate true, OpenSearch runs the destination state’s actions and moves the index there.

Three architectural properties fall out of this model and shape every policy you write:

• Deterministic, at-least-once execution. Actions run in listed order. If an action fails — say force_merge cannot proceed because the node is above watermark.high — the index stays in its current state and the step retries on the next poll. Nothing is skipped; a phase is never half-applied.
• Stateless evaluation. Conditions are checked against live index metadata (_stats, _cat/indices, index.creation_date). ISM keeps no external bookkeeping, so a coordinator failover or cluster restart resumes cleanly from the index’s own state.
• Coordinator-scheduled fan-out. One elected coordinator node owns the execution queue and spreads expensive actions (shrink, force_merge, snapshot) across data nodes to avoid hot-spotting a single node during a migration wave.

Because evaluation is periodic rather than continuous, an index can overshoot a threshold before the transition fires. The worst-case overshoot is bounded by the poll interval plus the action’s own runtime:

where $I_{\text{job}}$ is job_interval and $t_{\text{action}}$ is how long the entering action takes. Size rollover thresholds with that slack in mind — a min_primary_shard_size of 50 GB on a 5-minute interval can realistically peak higher before the roll completes.

Step-by-step: building a lifecycle policy #

The four steps below stand up a complete tiered-retention policy for a telemetry-logs-* index set: node attributes so phases have somewhere to route, a template so new indices start hot under management, the policy itself, then verification of both the declared state and the physical placement.

1. Node configuration #

Each phase’s allocation action targets a routing attribute, so that attribute must exist on the right nodes first. Declare it in opensearch.yml; the value is the exact string the policy references — a case or whitespace mismatch yields a shard that can never route to the tier.

# opensearch.yml — value matches the node's physical tier
node.name: os-data-hot-01
node.roles: [ data, ingest ]
node.attr.data: hot          # hot | warm | cold on their respective pools

2. Index template #

Attach the policy at the template level so every new rollover index inherits it automatically — never rely on a human running the add API per index. The template also seeds the rollover alias the hot state needs.

PUT _index_template/telemetry-logs
{
  "index_patterns": ["telemetry-logs-*"],
  "template": {
    "settings": {
      "plugins.index_state_management.policy_id": "telemetry-lifecycle",
      "plugins.index_state_management.rollover_alias": "telemetry-logs",
      "index.routing.allocation.require.data": "hot",
      "number_of_shards": 3,
      "number_of_replicas": 1
    }
  }
}

3. Policy JSON #

This policy rolls over in the hot phase, force-merges and reduces replicas in warm, drops to zero replicas in cold, and hard-deletes at 30 days. The transition conditions time each move; the allocation actions bind each phase to the attributes from step 1. The exact ordering of force_merge before allocation in warm is deliberate — it lets one controlled relocation carry both the merge and the routing change, a pattern detailed in Data Tier Routing Patterns.

PUT _plugins/_ism/policies/telemetry-lifecycle
{
  "policy": {
    "description": "Automated tiered retention for high-volume telemetry indices",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_index_age": "1d",
              "min_primary_shard_size": "45gb",
              "min_doc_count": 75000000
            }
          }
        ],
        "transitions": [
          { "state_name": "warm", "conditions": { "min_index_age": "3d" } }
        ]
      },
      {
        "name": "warm",
        "actions": [
          { "replica_count": { "number_of_replicas": 1 } },
          { "force_merge": { "max_num_segments": 1 } },
          { "allocation": { "require": { "data": "warm" } } }
        ],
        "transitions": [
          { "state_name": "cold", "conditions": { "min_index_age": "14d" } }
        ]
      },
      {
        "name": "cold",
        "actions": [
          { "replica_count": { "number_of_replicas": 0 } },
          { "allocation": { "require": { "data": "cold" } } }
        ],
        "transitions": [
          { "state_name": "delete", "conditions": { "min_index_age": "30d" } }
        ]
      },
      {
        "name": "delete",
        "actions": [ { "delete": {} } ]
      }
    ]
  }
}

For indices created before the template existed, attach the policy directly with the add API scoped to a pattern:

POST _plugins/_ism/add/telemetry-logs-*
{
  "policy_id": "telemetry-lifecycle"
}

4. Verification #

Confirm both that ISM picked the index up and that its declared routing matches its physical placement. The explain API reports the managed state; _cat/shards confirms the shards actually moved.

# Managed state, current phase, and any action failure
curl -s "https://<cluster>:9200/_plugins/_ism/explain/telemetry-logs-2026.07.04?pretty"

# Physical placement — the node column must sit on the phase's tier
curl -s "https://<cluster>:9200/_cat/shards/telemetry-logs-2026.07.04?v&h=index,shard,prirep,state,node"

A healthy index reports a state matching its age, action.failed: false, and shards resident on nodes carrying the matching node.attr.data value.

Production automation with Python #

Attachment and drift correction belong in CI/CD, not in a runbook. The handler below applies a policy to a pattern with bounded retries, injects credentials from the environment, and verifies attachment by parsing the explain response — skipping the total_managed_indices integer summary that sits alongside the per-index keys. It exits non-zero on failure so a pipeline step fails loudly rather than silently leaving indices unmanaged.

import os
import time
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry


class ISMManager:
    def __init__(self, host: str, policy_id: str, index_pattern: str):
        self.base_url = host.rstrip("/")
        self.policy_id = policy_id
        self.index_pattern = index_pattern
        self.session = requests.Session()
        self.session.auth = (os.getenv("OPENSEARCH_USER"), os.getenv("OPENSEARCH_PASS"))

        retry_strategy = Retry(
            total=3, backoff_factor=1, status_forcelist=[429, 500, 502, 503, 504]
        )
        self.session.mount("https://", HTTPAdapter(max_retries=retry_strategy))

    def attach_policy(self) -> dict:
        endpoint = f"{self.base_url}/_plugins/_ism/add/{self.index_pattern}"
        response = self.session.post(endpoint, json={"policy_id": self.policy_id})
        response.raise_for_status()
        return response.json()

    def verify_attachment(self) -> bool:
        """True when at least one managed index carries the expected policy."""
        endpoint = f"{self.base_url}/_plugins/_ism/explain/{self.index_pattern}"
        response = self.session.get(endpoint)
        response.raise_for_status()

        # The explain response keys index names at the top level;
        # `total_managed_indices` is an integer summary — skip it.
        for _idx, details in response.json().items():
            if isinstance(details, dict) and details.get("policy_id") == self.policy_id:
                return True
        return False


if __name__ == "__main__":
    manager = ISMManager(
        host=os.getenv("OPENSEARCH_ENDPOINT", "https://localhost:9200"),
        policy_id="telemetry-lifecycle",
        index_pattern="telemetry-logs-*",
    )

    try:
        manager.attach_policy()
        time.sleep(2)  # allow cluster-state propagation before verifying
        if manager.verify_attachment():
            print("Policy attached and verified.")
        else:
            raise SystemExit("Attachment reported success but no managed index carries the policy.")
    except requests.exceptions.RequestException as exc:
        raise SystemExit(f"ISM operation failed: {exc}")

Operational guardrails #

Two policies with identical states behave very differently under load depending on cluster-level settings. Tune the poll cadence and the disk watermarks together — an aggressive job_interval fires transitions faster but adds coordinator overhead, while watermarks decide whether a migration wave completes or wedges an index read-only. Aligning these with the routing attributes above is what keeps Fallback Routing Strategies from having to catch a stalled transition in the first place.

Give every action a retry block instead of relying on the default single attempt, so a transient watermark breach or a busy node does not strand a phase:

{
  "force_merge": { "max_num_segments": 1 },
  "retry": { "count": 3, "backoff": "exponential", "delay": "10m" }
}

Cross-cluster replication and tier routing alignment #

When ISM runs alongside Cross-Cluster Replication (CCR), policy execution must respect the replication topology. Follower indices inherit the leader’s policy by default but operate under a write block, so any action that writes — rollover, force_merge, shrink — will fail on a follower. Attach a separate, read-optimized policy to followers via POST _plugins/_ism/add/<follower_index> that carries only tier-transition and replica actions; let rollover happen on the leader and propagate through replication.

Routing during warm and cold transitions depends on node attributes existing on the follower cluster too — a follower that lacks a data: cold node will strand the transition, which is exactly the shard-unassignment class that Node Role Allocation diagnoses. Finally, the _plugins/_ism/* endpoints are privileged: a service account that can rewrite a policy can delete production data on a schedule, so scope those endpoints to platform-engineering roles as described in Security & Access Boundaries.

Troubleshooting #

Most lifecycle incidents present the same way — an index that will not leave its current phase. Diagnose with the explain API first; it names the failing step and its retry count. The pairs below cover the failure modes that account for the bulk of stuck transitions.

Transition never fires. The index sits in hot well past its min_index_age.

curl -s ".../_plugins/_ism/explain/<index>?pretty"        # inspect: is a policy even attached?
POST _plugins/_ism/add/<index> {"policy_id":"telemetry-lifecycle"}   # attach if policy_id is null

Action stuck on force_merge. The step retries forever and never advances.

curl -s ".../_cat/allocation?v"                            # check the node is below watermark.high
curl -s -X POST ".../_plugins/_ism/retry/<index>"          # re-drive the step once space is freed

Shards unassigned after a tier move. The transition ran but shards went UNASSIGNED.

curl -s -X POST ".../_cluster/allocation/explain" -H 'Content-Type: application/json' \
  -d '{"index":"<index>","shard":0,"primary":true}'        # reveals the decider that rejected placement
# Fix: add a node carrying the target node.attr.data value, or correct the allocation attribute

Policy detached after a template update. New indices are created unmanaged.

curl -s ".../_plugins/_ism/explain/<pattern>*" | grep -c null   # count unmanaged matches
# Fix: ensure the template still sets plugins.index_state_management.policy_id, then re-add by pattern

Index locked read-only. Writes fail with FORBIDDEN/12/index read-only.

curl -s ".../_cat/nodes?v&h=name,disk.used_percent"        # find the node past flood_stage
curl -s -X PUT ".../<index>/_settings" -d '{"index.blocks.read_only_allow_delete":null}'  # clear after freeing disk

Operational validation and execution monitoring #

Beyond attachment, treat the explain API as the source of truth for lifecycle health:

GET _plugins/_ism/explain/telemetry-logs-2026.07.04

The response carries the current state, the last transition timestamp, and any action failure. For alerting, parse action.failed and fire a webhook when step.status stays failed across consecutive poll cycles rather than on a single miss — a one-off failure is usually a transient watermark breach that self-heals on retry. Pair that with index-template versioning so an OpenSearch upgrade cannot silently detach policies mid-flight. The end-to-end tuning, capacity, and recovery playbook lives in Best practices for OpenSearch index lifecycle management.

For the full setting reference and endpoint semantics, consult the official OpenSearch Index State Management documentation, and standardize policy-payload construction with Python’s native json module.

Frequently asked questions #

Related #

• Hot-Warm-Cold Tier Design — sizing the hardware pools each lifecycle phase migrates onto.
• Node Role Allocation — declaring and verifying the routing attributes the policy writes.
• Data Tier Routing Patterns — how ISM stamps the routing attribute at each phase transition.
• Fallback Routing Strategies — recovering a transition when a target tier has no eligible node.
• Security & Access Boundaries — scoping the _plugins/_ism/* endpoints that can delete data on a schedule.
• Best practices for OpenSearch index lifecycle management — the deep tuning, capacity, and recovery playbook.

Up: OpenSearch ISM Architecture & Fundamentals