OpenSearch ISM & Cross-Cluster Replication

Versioning ISM policies with a GitOps workflow

This walkthrough makes a git repository the single source of truth for your OpenSearch ISM policies: every change lands through a reviewed pull request, each merge is tagged with a semantic version, a reconciler continuously drives the live cluster back to the committed state, and a bad release rolls back by reverting one commit. It extends the deploy job in Integrating ISM policy deployment into CI/CD pipelines from a push-on-merge model to a pull-based reconciliation loop, for teams that want the OpenSearch cluster’s lifecycle configuration to be as auditable and reversible as their application manifests.

Prerequisites

Confirm each of these before adopting the GitOps loop — a missing version tag or a reconciler running with an over-broad token is the usual reason a rollback either cannot find its target or reconciles the wrong cluster.

The reconciliation loop

GitOps inverts the push pipeline: instead of a merge pushing to the live cluster, a reconciler pulls the desired state from a git ref and continuously closes the gap between what git says and what the live cluster runs. The loop below is the whole model — a merged, tagged commit is the desired state; the reconciler reads it, diffs it against the live policy, applies the difference with a conditional PUT, and records the observed version. A failed release is undone by reverting the commit, which the same loop then reconciles back to the previous good version.

GitOps reconciliation loop for ISM policies Cycle. A reviewed pull request merges to main and is tagged with a semantic version, producing the desired state. The reconciler reads the tagged commit, fetches the live policy from the OpenSearch cluster, and computes a diff. If they match it records observed=desired and waits. If they differ it applies a conditional PUT and loops back to re-diff. A separate revert-commit path feeds a reverted version back into the desired state so the loop reconciles the OpenSearch cluster to the previous good version. PR reviewed & merged tag: logs_lifecycle@2.3.0 desired state tagged commit in git reconciler read · diff · apply live == desired? cluster live ISM policy apply conditional PUT if_seq_no · if_primary_term record observed = 2.3.0 revert → new desired rolls back to 2.2.0 differ match

Step-by-step procedure

The five steps take a change from a proposed edit to a reconciled, versioned, and reversible release.

1. Propose the change as a pull request

Every edit is a branch and a pull request against the policy repository — never a console edit. The PR diff is the change review: a reviewer sees the exact state, action, or transition delta before it can reach any cluster.

Shell
git checkout -b shorten-warm-retention
# edit policies/logs_lifecycle.json — warm→delete min_index_age 30d -> 21d
git commit -am "logs_lifecycle: shorten warm retention to 21d"
git push -u origin shorten-warm-retention

Gotcha: the reviewer must scrutinise any change to a delete action or a min_index_age, because those shorten retention and destroy data soonest. Encode that as a CODEOWNERS rule so retention edits require a data-owner approval, not just any teammate.

2. Tag the merge with a semantic version

On merge to main, tag the commit with a per-policy semantic version so the running configuration is always traceable to an immutable ref. The version is the anchor rollback and drift reporting both hang off.

Shell
git checkout main && git pull
git tag logs_lifecycle@2.3.0 -m "warm retention 21d"
git push origin logs_lifecycle@2.3.0

Embed that version inside the policy so the OpenSearch cluster can report it back:

JSON
{ "policy": { "description": "logs_lifecycle | version=2.3.0 | managed-by=gitops", "default_state": "hot", "states": [ "..." ] } }

Gotcha: bump the version on every functional change, even a one-line threshold edit. A reconciler that reports observed=2.3.0 is only trustworthy if that string uniquely identifies one committed document.

3. Reconcile the OpenSearch cluster to the tagged commit

The reconciler reads the desired document from the tagged ref, fetches the live policy for its concurrency tokens, and applies a conditional PUT only when they differ. Running it on a schedule (or triggered by a git webhook) is what makes the model continuous rather than merge-triggered — a manual console edit is reverted on the next pass.

Python
import json, logging, subprocess
from opensearchpy import OpenSearch, exceptions

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
log = logging.getLogger("gitops.reconcile")

def desired_at(ref: str, path: str) -> dict:
    """Read the committed policy document at a git ref."""
    blob = subprocess.check_output(["git", "show", f"{ref}:{path}"])
    return json.loads(blob)

def reconcile(client: OpenSearch, policy_id: str, desired: dict) -> str:
    try:
        live = client.transport.perform_request("GET", f"/_plugins/_ism/policies/{policy_id}")
        seq_no, primary_term, current = live["_seq_no"], live["_primary_term"], live["policy"]
    except exceptions.NotFoundError:
        seq_no = primary_term = current = None
    if current == desired["policy"]:
        log.info("in-sync policy_id=%s", policy_id)
        return "in-sync"
    params = {} if current is None else {"if_seq_no": seq_no, "if_primary_term": primary_term}
    try:
        client.transport.perform_request(
            "PUT", f"/_plugins/_ism/policies/{policy_id}", params=params, body=desired)
        log.info("reconciled policy_id=%s -> desired", policy_id)
        return "reconciled"
    except exceptions.ConflictError:
        log.warning("drift race policy_id=%s — retry next cycle", policy_id)
        return "conflict"

if __name__ == "__main__":
    client = OpenSearch(hosts=[{"host": "os-prod", "port": 9200}],
                        http_auth=("gitops-reconciler", "***"), use_ssl=True, verify_certs=True)
    reconcile(client, "logs_lifecycle",
              desired_at("logs_lifecycle@2.3.0", "policies/logs_lifecycle.json"))

The reconciler is idempotent by construction: once the OpenSearch cluster equals the tagged document it logs in-sync and writes nothing. This is the same conditional-PUT contract the push pipeline uses; the difference is that here the desired ref, not a merge event, drives the write.

4. Roll existing indices onto the new version

Reconciling the policy document does not move indices already managed under an older version. Advance them deliberately with change_policy, in a batch you can watch, so a retention change takes effect on a schedule you control.

Shell
POST _plugins/_ism/change_policy/logs-*
{ "policy_id": "logs_lifecycle", "state": { "name": "hot" } }
Text
{ "updated_indices": 37, "failures": false }

Gotcha: change_policy only re-evaluates an index at its next transition, so a shortened retention does not retroactively delete already-aged indices in one sweep — it applies going forward. Verify with explain before assuming an index moved.

5. Roll back a bad version by reverting the commit

Rollback in GitOps is not a special path — it is an ordinary change that happens to restore an earlier state. Revert the bad commit, tag the revert with a new version, and let the reconciler drive the OpenSearch cluster back. Because git holds every prior document, the previous good policy is always exactly recoverable.

Shell
git revert <bad-sha> --no-edit           # restores the 2.2.0 document
git tag logs_lifecycle@2.3.1 -m "rollback: revert 21d retention change"
git push origin main logs_lifecycle@2.3.1
# reconciler's next pass applies the reverted document with a conditional PUT

Gotcha: reverting the commit fixes the desired state, but indices already advanced to the bad version in Step 4 stay there until you change_policy them back. Roll the policy back and the indices back — the two are separate actions.

Verification

Confirm three things after a reconcile or rollback: the live policy matches the tagged commit, the OpenSearch cluster reports the version you expect, and no managed index is in a failed action.

Shell
# 1. The live description carries the version git just reconciled
GET _plugins/_ism/policies/logs_lifecycle
# "description": "logs_lifecycle | version=2.3.1 | managed-by=gitops", "_seq_no": 51
Shell
# 2. No managed index is stuck after the change
GET _plugins/_ism/explain/logs-*
# "logs-000014": { "state": {"name":"warm"}, "action": {"failed": false} }
Shell
# 3. The reconciler reports convergence, not a standing conflict
# in-sync policy_id=logs_lifecycle    <- steady state

A healthy result shows the live version= string equal to the latest tag, every index at "failed": false, and the reconciler logging in-sync on its next pass. If the live version keeps disagreeing with the tag between reconciles, an out-of-band editor is fighting the loop — hand that off to the deeper remediation flow in Detecting and reconciling ISM policy drift.

Common failures

Symptom Root cause Fix command
Reconciler logs conflict every cycle someone edits the policy in the console between reconciles disable console write for the reconciler’s account; escalate to drift remediation
Rollback tag applied but cluster unchanged reconciler pinned to an old ref, not the newest tag point the reconciler at the latest logs_lifecycle@* tag or a moving branch
version= in cluster never matches the tag version string not bumped in the committed document embed and increment the version on every functional change
Indices still on the bad version after revert policy reverted but indices not moved POST _plugins/_ism/change_policy/<pattern> back to the good version
Reconciler applies to the wrong cluster shared token or host misconfig scope one reconciler + token per cluster per Security & Access Boundaries
Revert reintroduces an even older bug blind git revert of a squashed range revert the precise commit; verify the resulting document is the intended prior version

Frequently asked questions

How is this different from the push pipeline in the sibling guide?

The push model in Integrating ISM policy deployment into CI/CD pipelines applies on merge and then stops — the OpenSearch cluster can drift until the next deploy. The GitOps model runs a reconciler continuously against a tagged git ref, so the OpenSearch cluster is always being pulled back toward the committed state. Push is event-driven; GitOps is convergence-driven. Many teams use the push pipeline for the reviewed apply and add the reconciler to guarantee the state holds between merges.

Do I tag every policy separately or the whole repo?

Tag per policy (logs_lifecycle@2.3.0, metrics_lifecycle@1.4.2) when policies change on independent cadences, so one policy’s rollback never drags another’s version with it. A single repo-wide tag is simpler but couples unrelated policies — a revert of one forces a version bump on all. Per-policy tags keep rollbacks surgical.

What stops the reconciler from fighting a legitimate emergency console fix?

Nothing should — that is the point. If an operator makes an emergency console edit, the reconciler will revert it on the next pass, which is correct: the fix must be committed to git to survive. For a genuine break-glass change, pause the reconciler, apply the fix, then immediately backport it to git and resume. A change that only lives in the OpenSearch cluster is a change that will be lost.

Can I reconcile more than one cluster from one repository?

Yes — run one reconciler instance per cluster, each with a token scoped to that cluster and each pointed at the ref that cluster should track (e.g. staging follows main, prod follows the latest release tag). Keeping the ref-per-cluster mapping explicit is what lets staging validate a version before prod’s reconciler ever sees it.

Up: CI/CD Pipeline Integration