Implementing fallback routing for ISM phase transitions
When an OpenSearch Index State Management (ISM) policy attempts to transition an index between lifecycle phases, routing allocation failures are the primary cause of stalled indices. Disk watermark breaches, missing node attributes, or conflicting index.routing.allocation rules will halt the transition action indefinitely. Implementing fallback routing for ISM phase transitions requires deterministic allocation overrides, policy-level conditional checks, and automated remediation workflows to maintain SLA compliance.
Understanding the underlying OpenSearch ISM Architecture & Fundamentals reveals that the allocation engine evaluates shard placement sequentially. If the primary tier lacks capacity or violates routing constraints, the index enters a stuck state. Fallback routing mitigates this by defining a secondary allocation target that activates when primary tier nodes exceed capacity thresholds or become unreachable. This deterministic routing pattern prevents index lifecycle deadlocks without requiring manual shard rebalancing.
Node Attribute Topology & Fallback Tier Mapping
Fallback routing relies on explicit, immutable node attributes declared in opensearch.yml. Each data node must publish both a primary tier identifier and a fallback tier identifier. The cluster routing allocator uses these attributes to evaluate placement constraints during ISM state transitions.
# opensearch.yml (Warm Tier Node)
node.attr.data_tier: warm
node.attr.fallback_tier: warm_fallback
node.attr.zone: us-east-1a
# Cluster-level routing thresholds
cluster.routing.allocation.disk.watermark.low: 85%
cluster.routing.allocation.disk.watermark.high: 90%
cluster.routing.allocation.disk.watermark.flood_stage: 95%
cluster.routing.allocation.disk.watermark.flood_stage.max_headroom: 100GB
After deploying the configuration, verify attribute propagation across the cluster:
curl -s -X GET "localhost:9200/_cat/nodes?v&h=name,ip,node.attr.data_tier,node.attr.fallback_tier"
If attributes are missing, mismatched, or inconsistently cased, the ISM policy engine will reject routing overrides. Ensure node.attr values are identical across all nodes in the target tier and validate that the cluster state reflects the updated metadata (GET _cluster/state?filter_path=metadata.cluster_custom).
ISM Policy Configuration with Deterministic Overrides
The core implementation requires an ISM policy that evaluates allocation viability before executing phase transitions. Use the transition action with conditions to validate shard placement readiness, then apply fallback routing via allocation actions with explicit require and include directives.
PUT _plugins/_ism/policies/log_fallback_routing
{
"policy": {
"description": "Hot-warm transition with deterministic fallback routing",
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [
{
"rollover": {
"min_size": "50gb",
"min_index_age": "7d"
}
}
],
"transitions": [
{
"state_name": "warm",
"conditions": { "min_index_age": "7d" }
}
]
},
{
"name": "warm",
"actions": [
{
"allocation": {
"require": { "data_tier": "warm" }
}
},
{
"shrink": {
"num_new_shards": 1
}
}
],
"transitions": [
{
"state_name": "fallback_warm",
"conditions": {
"min_index_age": "30d",
"shard_allocation_status": "failed"
}
}
]
},
{
"name": "fallback_warm",
"actions": [
{
"allocation": {
"require": { "fallback_tier": "warm_fallback" }
}
}
],
"transitions": []
}
]
}
}
When designing Fallback Routing Strategies, the shard_allocation_status condition acts as a circuit breaker. If the primary warm allocation fails due to watermark violations or insufficient disk space, the policy automatically routes to the fallback_warm state. This eliminates manual intervention and ensures indices progress through the lifecycle without violating retention SLAs.
Automated Detection & Remediation Workflow (Python)
Policy-level conditions handle predictable failures, but transient cluster events require external automation. The following Python workflow polls the ISM explain API, identifies stuck transitions, and forces routing overrides before triggering a retry.
import requests
import time
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
CLUSTER_ENDPOINT = "https://opensearch-master.internal:9200"
AUTH = ("admin", "changeme")
VERIFY = True # Set to False only for self-signed dev environments
def get_stuck_indices() -> list:
"""Query ISM explain API for indices in failed or stuck states."""
url = f"{CLUSTER_ENDPOINT}/_plugins/_ism/explain?pretty"
response = requests.get(url, auth=AUTH, verify=VERIFY, timeout=10)
response.raise_for_status()
data = response.json()
stuck = []
# Explain returns index names as top-level keys plus a total_managed_indices int.
for index, state in data.items():
if not isinstance(state, dict):
continue
info = state.get("info", {})
if state.get("step", {}).get("status") == "failed" \
or state.get("action", {}).get("failed") \
or "allocation_failed" in info.get("message", ""):
stuck.append(index)
return stuck
def apply_fallback_routing(index: str):
"""Force index settings to fallback tier and retry ISM transition."""
settings_payload = {
"index.routing.allocation.require.fallback_tier": "warm_fallback",
"index.routing.allocation.require.data_tier": None
}
settings_url = f"{CLUSTER_ENDPOINT}/{index}/_settings"
requests.put(settings_url, json=settings_payload, auth=AUTH, verify=VERIFY, timeout=10)
retry_url = f"{CLUSTER_ENDPOINT}/_plugins/_ism/retry/{index}"
retry_payload = {"state": "fallback_warm"}
resp = requests.post(retry_url, json=retry_payload, auth=AUTH, verify=VERIFY, timeout=10)
resp.raise_for_status()
logging.info(f"Applied fallback routing and retried ISM state for {index}")
if __name__ == "__main__":
while True:
try:
stuck_indices = get_stuck_indices()
if stuck_indices:
logging.info(f"Detected {len(stuck_indices)} stuck indices. Applying fallback routing...")
for idx in stuck_indices:
apply_fallback_routing(idx)
time.sleep(300)
except requests.exceptions.RequestException as e:
logging.error(f"Cluster API request failed: {e}")
time.sleep(60)
The script uses persistent HTTP sessions implicitly via requests and adheres to safe retry intervals. For production deployments, integrate this workflow with your CI/CD pipeline or Kubernetes CronJob scheduler. Refer to the official OpenSearch ISM API documentation for endpoint versioning and authentication requirements.
Validation, Debugging & Threshold Calibration
After applying fallback routing, validate shard placement and allocation decisions using cluster diagnostics.
- Verify Routing Decisions:
curl -s -X GET "localhost:9200/_cluster/allocation/explain/<index_name>?pretty"
Inspect node_allocation_decisions to confirm the allocator selected fallback_tier nodes. Look for DECISION: YES in the output.
- Monitor Disk Watermark Impact:
curl -s -X GET "localhost:9200/_cat/allocation?v&h=shards,disk.indices,disk.used,disk.avail,disk.total,disk.percent,node"
If nodes hover above 90%, the allocator will refuse primary tier placement. Lower cluster.routing.allocation.disk.watermark.high to 88% or expand storage before the fallback tier absorbs additional load.
- Trace ISM Execution Logs:
curl -s -X GET "localhost:9200/_plugins/_ism/explain/<index_name>?pretty"
Review the state and action fields. A successful fallback transition will show state: fallback_warm and action: allocation with status: completed.
Cross-Cluster Replication Routing Constraints
When ISM manages leader indices in a Cross-Cluster Replication (CCR) topology, fallback routing introduces replication latency risks. Follower indices inherit routing constraints from the leader cluster. If the leader transitions to a fallback tier, the follower cluster must either:
- Mirror the fallback node attributes in its own
opensearch.ymlconfiguration. - Explicitly override routing on the follower using
PUT _plugins/_replication/<follower_index>/_settingswithindex.routing.allocation.require.fallback_tier.
Failure to synchronize routing attributes across clusters will cause replication checkpoints to stall, as the follower allocator cannot place shards matching the leader’s fallback topology. Always validate follower allocation health using GET _plugins/_replication/<follower_index>/status after ISM phase transitions.