Policy Drift Detection

Policy drift is the divergence between the ISM policy running live on an OpenSearch cluster and the version-controlled JSON you consider the source of truth. It creeps in through console edits, a half-finished change_policy rollout, a hotfix applied straight to production, or two automation jobs racing to PUT the same policy id. Left undetected, drift means the lifecycle your git history describes is not the lifecycle actually deleting and tiering your data — retention windows silently lengthen, force-merges stop firing, and an audit cannot be answered from the repository alone. This guide, part of ISM Policy Implementation & Python Automation, shows how to pull the live policy from GET _plugins/_ism/policies/<id>, read what each index is actually running through _plugins/_ism/explain, canonicalize both against the desired JSON, and drive a deterministic detect → diff → reconcile loop that converges the OpenSearch cluster back onto its declared state.

Drift detection is a control problem, not a one-off script. The OpenSearch cluster is the plant, git is the setpoint, and the reconciler is the controller that closes the gap on every cycle. Everything below assumes you treat the repository as authoritative and the OpenSearch cluster as the thing to be corrected — never the reverse.

Drift Sources and Detection Signals #

Not all drift looks the same on the wire, and each source needs a different signal to catch it. A console edit bumps the policy document’s seq_no/primary_term; a stalled change_policy leaves indices pinned to a superseded policy_seq_no in their explain output; a deleted-and-recreated policy resets the version counters entirely. The table below maps each drift source to the endpoint that reveals it, the field you compare, and how urgently it needs reconciling.

The last two rows are why canonicalization matters. OpenSearch stores and returns the policy with server-added fields (policy_id, last_updated_time, schema_version) and does not preserve your key ordering, so a naive byte-for-byte comparison reports drift on every run. A useful detector strips server-managed fields, sorts keys, and only then diffs — turning “everything is different” into “these three transitions are different.”

The Detect → Diff → Reconcile Loop #

The core of drift management is a closed loop that runs on a schedule and is safe to repeat. It fetches the live policy and the per-index managed state, canonicalizes both sides, computes a semantic diff against the git desired document, and branches: no diff means log-and-exit; a diff means either alert-only (the default guardrail) or reconcile by PUT-ting the desired document with the live seq_no/primary_term for optimistic concurrency, then re-applying it to stuck indices with change_policy. The state diagram below shows the branches and the two write actions the loop can take.

The two write actions are deliberately separate. PUT _plugins/_ism/policies/<id> corrects the policy document; it does not retroactively move indices already managed under the old version. Those indices keep running their pinned policy_seq_no until you issue POST _plugins/_ism/change_policy/<index> to move them to the current version at a safe state boundary. A reconciler that updates the document but forgets the second step reports “in sync” on the next run while half the fleet still executes stale rules. This distinction is the whole reason the loop re-verifies before exiting, and it connects directly to the state-machine reasoning in Phase Transition Logic.

1. Establish the desired-state document #

Keep one canonical JSON per policy id in the repository, containing only the fields you author — never the server-added ones. This is the setpoint every diff runs against.

{
  "policy": {
    "description": "hot-warm-delete for application logs",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          { "rollover": { "min_primary_shard_size": "50gb", "min_index_age": "1d" } }
        ],
        "transitions": [
          { "state_name": "warm", "conditions": { "min_index_age": "7d" } }
        ]
      },
      {
        "name": "warm",
        "actions": [
          { "force_merge": { "max_num_segments": 1 } },
          { "allocation": { "require": { "data": "warm" } } }
        ],
        "transitions": [
          { "state_name": "delete", "conditions": { "min_index_age": "30d" } }
        ]
      },
      { "name": "delete", "actions": [ { "delete": {} } ] }
    ]
  }
}

Store it at a stable path (for example policies/logs-hot-warm-delete.json) so the reconciler can resolve the policy id from the filename. Do not embed policy_id, last_updated_time, last_updated_user, or schema_version — the server owns those, and including them guarantees a permanent false positive.

2. Fetch the live policy and its version markers #

Read the running document and capture _seq_no and _primary_term. These two integers implement optimistic concurrency: a later PUT that carries them will only succeed if nobody else has written the policy in the meantime.

# Live policy document plus the version markers used for concurrency
curl -s "https://<cluster>:9200/_plugins/_ism/policies/logs-hot-warm-delete?pretty"

{
  "_id": "logs-hot-warm-delete",
  "_seq_no": 42,
  "_primary_term": 6,
  "policy": {
    "policy_id": "logs-hot-warm-delete",
    "last_updated_time": 1752800000000,
    "schema_version": 21,
    "description": "hot-warm-delete for application logs",
    "default_state": "hot",
    "states": [ "..." ]
  }
}

A jump in _seq_no since the value you last recorded is the cheapest possible drift signal — it means someone wrote this policy, whether or not the body changed. Persist the last-known _seq_no per policy id (in a small state file or a metrics label) so a version move alone can page you even before a body diff runs.

3. Detect indices pinned to an old policy version #

The policy document can be perfectly in sync while individual indices still execute an older revision, because change_policy moves managed indices lazily. _plugins/_ism/explain reports each index’s policy_id and the policy_seq_no it is actually running.

curl -s "https://<cluster>:9200/_plugins/_ism/explain/logs-*?pretty"

{
  "logs-000019": {
    "index.plugins.index_state_management.policy_id": "logs-hot-warm-delete",
    "policy_seq_no": 37,
    "policy_primary_term": 6,
    "state": { "name": "warm" },
    "action": { "name": "force_merge", "failed": false }
  }
}

Here the policy document is at _seq_no: 42 but logs-000019 is pinned to policy_seq_no: 37 — it is five revisions behind and will keep applying the old warm rules until reconciled. Any index whose policy_seq_no trails the current policy _seq_no is drifted at the index level even when the document is correct.

4. Canonicalize before diffing #

Both sides must be reduced to the same normal form or the diff is pure noise. Strip the server-managed keys, sort every object’s keys recursively, and serialize deterministically.

import json

SERVER_FIELDS = {"policy_id", "last_updated_time", "last_updated_user", "schema_version"}

def canonicalize(policy_doc: dict) -> str:
    """Reduce a policy body to a stable, comparable string."""
    body = policy_doc.get("policy", policy_doc)
    cleaned = {k: v for k, v in body.items() if k not in SERVER_FIELDS}
    # sort_keys makes key order irrelevant; separators strip incidental whitespace
    return json.dumps(cleaned, sort_keys=True, separators=(",", ":"))

Run canonicalize on both the live policy body and the desired document’s policy body. If the two strings are equal, the document is in sync regardless of formatting; if not, the difference is semantic and worth reporting.

5. Reconcile with optimistic concurrency #

When the document has drifted and the reconciler is allowed to write, PUT the desired body back with the live _seq_no/_primary_term as query parameters. If another writer slipped in between your read and your write, the version guard fails with 409 Conflict instead of silently clobbering their change.

# Re-assert the desired policy only if nobody else has written since seq_no 42
curl -s -X PUT \
  "https://<cluster>:9200/_plugins/_ism/policies/logs-hot-warm-delete?if_seq_no=42&if_primary_term=6" \
  -H "Content-Type: application/json" \
  -d @policies/logs-hot-warm-delete.json

Then move any pinned indices to the freshly written version, one state boundary at a time:

curl -s -X POST \
  "https://<cluster>:9200/_plugins/_ism/change_policy/logs-000019" \
  -H "Content-Type: application/json" \
  -d '{ "policy_id": "logs-hot-warm-delete", "state": "warm" }'

Naming the state in the change_policy body pins where the index resumes, so an index mid-warm does not get bounced back to hot. Getting this handoff right is the same concurrency discipline the Python Orchestration Frameworks cluster applies to any dynamic policy update.

Production Automation with Structured Logging #

The manual calls above become a scheduled reconciler. The opensearch-py class below fetches the live document, canonicalizes both sides, logs a structured drift record, and — only when apply=True — re-asserts the policy under optimistic concurrency and re-applies it to pinned indices. Narrow exception handling keeps a transient 409 or connection blip from crashing an unattended run. Scope its service account to just cluster:admin/opendistro/ism/* on the target policy ids before deploying it.

import json
import logging
from opensearchpy import OpenSearch, exceptions

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("ism.drift")

SERVER_FIELDS = {"policy_id", "last_updated_time", "last_updated_user", "schema_version"}

def canonicalize(body: dict) -> str:
    cleaned = {k: v for k, v in body.items() if k not in SERVER_FIELDS}
    return json.dumps(cleaned, sort_keys=True, separators=(",", ":"))

class DriftReconciler:
    def __init__(self, client: OpenSearch, apply: bool = False):
        self.client = client
        self.apply = apply  # dry-run guard: default is detect-only

    def _get_live(self, policy_id: str) -> dict | None:
        try:
            return self.client.transport.perform_request(
                "GET", f"/_plugins/_ism/policies/{policy_id}")
        except exceptions.NotFoundError:
            logger.error("policy_missing id=%s", policy_id)
            return None
        except exceptions.ConnectionError:
            logger.error("connect_failed id=%s", policy_id)
            return None

    def check(self, policy_id: str, desired_path: str) -> bool:
        """Return True when live matches desired; reconcile if drift and apply=True."""
        with open(desired_path) as fh:
            desired = json.load(fh)
        live = self._get_live(policy_id)
        if live is None:
            return False

        seq_no = live["_seq_no"]
        primary_term = live["_primary_term"]
        live_canon = canonicalize(live["policy"])
        want_canon = canonicalize(desired["policy"])

        if live_canon == want_canon:
            logger.info("in_sync id=%s seq_no=%s", policy_id, seq_no)
            return True

        logger.warning(
            "drift_detected id=%s seq_no=%s live_bytes=%d want_bytes=%d apply=%s",
            policy_id, seq_no, len(live_canon), len(want_canon), self.apply,
        )
        if not self.apply:
            return False
        return self._reconcile(policy_id, desired, seq_no, primary_term)

    def _reconcile(self, policy_id, desired, seq_no, primary_term) -> bool:
        try:
            self.client.transport.perform_request(
                "PUT", f"/_plugins/_ism/policies/{policy_id}",
                params={"if_seq_no": seq_no, "if_primary_term": primary_term},
                body=desired,
            )
            logger.info("policy_reasserted id=%s from_seq_no=%s", policy_id, seq_no)
            return True
        except exceptions.ConflictError:
            # Someone wrote between our read and write; back off and retry next cycle.
            logger.error("concurrent_write id=%s seq_no=%s; skipping", policy_id, seq_no)
            return False

if __name__ == "__main__":
    client = OpenSearch(
        hosts=[{"host": "localhost", "port": 9200}],
        http_auth=("admin", "admin"), use_ssl=True, verify_certs=True,
    )
    DriftReconciler(client, apply=False).check(
        "logs-hot-warm-delete", "policies/logs-hot-warm-delete.json")

Emit each drift_detected line to your central log pipeline with the policy id as a label so a dashboard can count drift events per policy over time. The apply=False default is the single most important guardrail: a scheduler should detect on every run and only reconcile when a human or a pipeline flips the flag. The scheduled-remediation wrapper that adds alerting and a dry-run gate around this class is covered in Automating ISM drift remediation with scheduled jobs, and the focused single-policy version with full diff output is in Detecting and reconciling ISM policy drift.

Operational Guardrails #

Reconciliation writes to a live cluster, so the settings that bound when and how aggressively it acts matter as much as the diff logic. The drift-event rate that a scheduled reconciler can safely absorb is bounded by the sweep interval $I$ and the reconcile budget $B$ (writes allowed per cycle):

Keep $B$ small — one or two policies per cycle — so a mass false positive from a bad desired document cannot rewrite the whole fleet in a single run.

Align the check cadence with the ISM job_interval: running the detector far more often than the sweep just re-observes the same pinned indices before ISM has had a chance to advance them. A detector that runs every sweep interval and reconciles only on the second consecutive confirmed diff avoids paging on a transition that is simply in flight.

Troubleshooting #

The version-conflict row is the expected, healthy failure under concurrency — it means the optimistic guard did its job and refused to overwrite a change it had not seen. Treat a burst of conflicts as a signal that two writers (say a CI deploy and the reconciler) are fighting over the same policy id, and serialize them rather than raising the retry count.

Frequently asked questions #

Related #

• Detecting and reconciling ISM policy drift — the focused script that fetches, canonicalizes, diffs, and repairs a single policy.
• Automating ISM drift remediation with scheduled jobs — running the detector on a CronJob with alerting and a dry-run guard.
• Python Orchestration Frameworks — the client patterns and idempotency discipline this reconciler builds on.
• Phase Transition Logic — the state model that determines where a change_policy can safely resume an index.

Up: ISM Policy Implementation & Python Automation