CI/CD Pipeline Integration

CI/CD pipeline integration treats every OpenSearch Index State Management (ISM) policy as source-controlled JSON that a pipeline lints, validates, diffs against the live cluster, and applies idempotently through PUT _plugins/_ism/policies/<id> before verifying the result — the same discipline you already apply to application code, now applied to lifecycle configuration. This guide sits within ISM Policy Implementation & Python Automation and covers the full promotion path from a developer’s laptop through dev, staging, and production clusters, where an unreviewed policy edit that silently detaches a warm-tier transition or shortens a retention window can quietly delete data weeks later. The failure ISM engineers hit again and again is the hand-edited policy: someone pastes JSON into a console, the change works, nobody records it, and three clusters drift out of agreement until an incident forces an archaeology dig. A pipeline removes the console from the loop entirely — git is the only writer, and every change is reviewable, reversible, and reproducible.

Environment and promotion alignment #

A policy does not graduate to production in one leap. It flows through environments that each answer a different question, and every stage maps to a real cluster, a distinct credential scope, and an explicit gate. The table below aligns each promotion stage with the OpenSearch cluster it targets, what the pipeline is allowed to do there, the approval it requires, and the signal that lets the change advance. Getting this alignment right is what keeps a plan/diff step in dev from ever silently mutating production.

The critical property is that the credential the pipeline uses in each stage is scoped to that stage’s cluster only, so a bug in the dev job physically cannot reach production. Scope those service accounts per the model in Security & Access Boundaries: the CI identity that plans against production holds read on _plugins/_ism/policies/* and nothing more, while the apply identity is a separate, short-lived token minted only inside the gated production job. The two walkthroughs below build the concrete jobs — Integrating ISM policy deployment into CI/CD pipelines wires the pipeline job itself, and Versioning ISM policies with a GitOps workflow makes git the reconciliation source of truth.

The pipeline as a state machine #

A policy deployment pipeline is a linear progression of gates, each of which either passes the artifact forward or fails the whole run. Nothing mutates an OpenSearch cluster until the artifact has cleared lint, schema validation, and a diff that a human (or an automated policy) has accepted. The diagram below shows the five stages and the fact that only apply writes to the live cluster — every earlier stage is read-only, and verify reads back to confirm the write landed.

The design principle is that the pipeline is fail-closed: a schema error, a diff the reviewer did not expect, or a failed verification all halt the run before or immediately after the single write, and the OpenSearch cluster is never left in a half-applied state. Because a PUT to _plugins/_ism/policies/<id> replaces the entire policy document atomically, there is no partial-write window to worry about within a single policy — the risk is applying the wrong whole document, which is exactly what the plan/diff stage exists to catch. When your fleet grows past a handful of policies you will want the reconciliation loop from Policy Drift Detection running continuously behind the pipeline, so a manual console edit that bypasses git is detected and reverted rather than surviving to the next deploy.

1. Structure the repository so the artifact is the source of truth #

Store each policy as a standalone JSON file named for its policy id, with the environment-specific values factored out. The pipeline reads the file, substitutes per-environment parameters, and produces the exact document it will PUT. Keep the JSON free of any live-cluster metadata (_seq_no, _primary_term, _version) — those belong to the OpenSearch cluster, not the repository.

{
  "policy": {
    "description": "Logs lifecycle — managed by CI, do not edit in console",
    "default_state": "hot",
    "ism_template": [
      { "index_patterns": ["logs-*"], "priority": 100 }
    ],
    "states": [
      {
        "name": "hot",
        "actions": [ { "rollover": { "min_primary_shard_size": "50gb", "min_index_age": "1d" } } ],
        "transitions": [ { "state_name": "warm", "conditions": { "min_index_age": "7d" } } ]
      },
      {
        "name": "warm",
        "actions": [ { "allocation": { "require": { "data": "warm" } } }, { "force_merge": { "max_num_segments": 1 } } ],
        "transitions": [ { "state_name": "delete", "conditions": { "min_index_age": "30d" } } ]
      },
      { "name": "delete", "actions": [ { "delete": {} } ] }
    ]
  }
}

The ism_template block is what makes the policy self-attaching: any new index matching logs-* picks up the policy on creation, so the pipeline never has to run a separate POST _plugins/_ism/add/<index> for greenfield indices. Existing indices still need an explicit attach or change_policy, covered in Step 4.

2. Lint and schema-validate before any cluster is touched #

The first two pipeline stages run entirely inside the CI runner with no cluster credentials. Lint proves the file is well-formed JSON; validation proves it is a well-formed ISM policy — the required keys are present, every transition names a state that exists, and no action references an unknown type. Catching a dangling transition here saves a FAILED managed index later.

import json, sys

REQUIRED = {"description", "default_state", "states"}
KNOWN_ACTIONS = {
    "rollover", "force_merge", "shrink", "allocation",
    "replica_count", "snapshot", "index_priority", "read_only",
    "read_write", "close", "open", "delete", "retry", "notification",
}

def validate_policy(path: str) -> list[str]:
    errors: list[str] = []
    with open(path) as fh:
        doc = json.load(fh)                      # raises on malformed JSON = lint failure
    policy = doc.get("policy", {})
    missing = REQUIRED - policy.keys()
    if missing:
        errors.append(f"missing top-level keys: {sorted(missing)}")
    state_names = {s["name"] for s in policy.get("states", [])}
    if policy.get("default_state") not in state_names:
        errors.append(f"default_state '{policy.get('default_state')}' is not a defined state")
    for state in policy.get("states", []):
        for action in state.get("actions", []):
            for key in action:
                if key not in KNOWN_ACTIONS and key not in {"retry", "timeout"}:
                    errors.append(f"state '{state['name']}': unknown action '{key}'")
        for t in state.get("transitions", []):
            if t["state_name"] not in state_names:
                errors.append(f"state '{state['name']}': transition to undefined state '{t['state_name']}'")
    return errors

if __name__ == "__main__":
    problems = validate_policy(sys.argv[1])
    for p in problems:
        print(f"::error::{p}")          # surfaces inline in the CI job log
    sys.exit(1 if problems else 0)

Run this against every changed policy file. Because it needs no network access, it belongs in the fast, always-on part of the pipeline and gives contributors sub-second feedback on a bad edit.

3. Plan: diff the desired policy against the live cluster #

The plan stage is the safety interlock. It fetches the currently attached policy with GET _plugins/_ism/policies/<id>, strips the OpenSearch cluster-owned concurrency metadata, and compares the remaining document to what the repository wants to apply. An empty diff means the apply stage will be a no-op; a non-empty diff is exactly the change set a reviewer approves before production.

# Fetch the live policy (read-only credential is sufficient here)
curl -s "https://os-prod:9200/_plugins/_ism/policies/logs_lifecycle" \
  -H "Authorization: Bearer $OS_READ_TOKEN" -o live.json

# Expected shape — note the concurrency metadata the diff must ignore

{
  "_id": "logs_lifecycle",
  "_seq_no": 42,
  "_primary_term": 6,
  "policy": { "description": "Logs lifecycle ...", "states": [ ... ] }
}

The _seq_no (42) and _primary_term (6) are the optimistic-concurrency-control tokens the OpenSearch cluster assigns; they change on every write and must never appear in a diff or in the repository. Compare only the policy object. If your diff tool reports a change, the plan stage prints it to the job summary so the reviewer sees precisely which states, actions, or transitions move.

4. Apply: an idempotent conditional PUT #

Applying a policy is a single PUT _plugins/_ism/policies/<id>. The idempotency guarantee comes from OpenSearch’s optimistic concurrency control: when the policy already exists you must pass the if_seq_no and if_primary_term you read during plan, and the write succeeds only if no one has modified the policy since. If a console edit slipped in between plan and apply, the tokens no longer match, the OpenSearch cluster returns 409 Conflict, and the pipeline fails instead of clobbering the intervening change.

# CREATE (policy does not yet exist): no concurrency tokens
curl -s -X PUT "https://os-prod:9200/_plugins/_ism/policies/logs_lifecycle" \
  -H "Authorization: Bearer $OS_WRITE_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @desired-policy.json

# UPDATE (policy exists): guard the write with the tokens read during plan
curl -s -X PUT "https://os-prod:9200/_plugins/_ism/policies/logs_lifecycle?if_seq_no=42&if_primary_term=6" \
  -H "Authorization: Bearer $OS_WRITE_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @desired-policy.json

Re-running the apply with the same desired document and the new tokens is a safe no-op at the policy level, which is what makes the whole pipeline replayable. Applying the policy does not retroactively change indices already running under an old version — OpenSearch keeps each managed index pinned to the policy version it started on until you explicitly move it forward with POST _plugins/_ism/change_policy/<index>. Roll existing indices onto the new version deliberately, in a batch you can watch, rather than assuming the PUT re-managed them.

5. A GitHub Actions pipeline example #

The pipeline below ties the stages together. Lint and validate run on every push; plan runs against production read-only on pull requests; apply runs only on merge to main inside an environment that carries a manual-approval gate and the short-lived write token.

name: ism-policy-deploy
on:
  pull_request: { paths: ["policies/**"] }
  push: { branches: [main], paths: ["policies/**"] }

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python -m pip install opensearch-py
      - run: |
          for f in policies/*.json; do
            python ci/validate_policy.py "$f"      # lint + schema, stages 1-2
          done

  plan:
    needs: validate
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - env:
          OS_HOST: $
          OS_READ_TOKEN: $   # read-only
        run: python ci/deploy.py --plan policies/          # stage 3, prints diff

  apply:
    needs: validate
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    environment: production            # manual approval gate lives here
    steps:
      - uses: actions/checkout@v4
      - env:
          OS_HOST: $
          OS_WRITE_TOKEN: $ # short-lived, gated
        run: python ci/deploy.py --apply --verify policies/  # stages 4-5

The environment: production key is what pauses the apply job for a human approval and injects the write secret only after approval, keeping the powerful credential out of every other run. Package and version the ci/deploy.py image the same way you would any release artifact — the mechanics of that build live in Python Orchestration Frameworks.

Production deploy step in opensearch-py #

The deploy.py invoked above wraps the plan/apply/verify sequence in one typed, structured-logging module. It reads the desired document from disk, fetches the live policy for its concurrency tokens, issues the conditional PUT, and reads back the explain output — logging a machine-parseable line at every step so a failed run is diagnosable from logs alone. Narrow exception handling keeps a 409 (concurrency conflict) and a 404 (first-time create) on separate, intentional paths.

import json, logging, sys
from opensearchpy import OpenSearch, exceptions

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("ism.deploy")

class PolicyDeployer:
    def __init__(self, client: OpenSearch):
        self.client = client

    def _fetch_tokens(self, policy_id: str) -> tuple[dict | None, int | None, int | None]:
        """Return (live_policy, seq_no, primary_term) or (None, None, None) if absent."""
        try:
            resp = self.client.transport.perform_request(
                "GET", f"/_plugins/_ism/policies/{policy_id}")
            return resp["policy"], resp["_seq_no"], resp["_primary_term"]
        except exceptions.NotFoundError:
            return None, None, None

    def apply(self, policy_id: str, desired: dict) -> bool:
        live, seq_no, primary_term = self._fetch_tokens(policy_id)
        if live == desired["policy"]:
            logger.info("noop policy_id=%s reason=already-in-sync", policy_id)
            return True
        params = {} if live is None else {"if_seq_no": seq_no, "if_primary_term": primary_term}
        try:
            self.client.transport.perform_request(
                "PUT", f"/_plugins/_ism/policies/{policy_id}",
                params=params, body=desired)
            logger.info("applied policy_id=%s mode=%s",
                        policy_id, "create" if live is None else "update")
            return True
        except exceptions.ConflictError:
            logger.error("conflict policy_id=%s reason=seq-no-stale-rerun-plan", policy_id)
            return False

    def verify(self, index_pattern: str) -> bool:
        explain = self.client.transport.perform_request(
            "GET", f"/_plugins/_ism/explain/{index_pattern}")
        failed = [i for i, s in explain.items()
                  if isinstance(s, dict) and s.get("action", {}).get("failed")]
        if failed:
            logger.error("verify-failed indices=%s", failed)
            return False
        logger.info("verify-ok pattern=%s managed=%d", index_pattern, len(explain))
        return True

if __name__ == "__main__":
    client = OpenSearch(hosts=[{"host": "os-prod", "port": 9200}],
                        http_auth=("ci-deploy", "***"), use_ssl=True, verify_certs=True)
    with open("policies/logs_lifecycle.json") as fh:
        desired = json.load(fh)
    deployer = PolicyDeployer(client)
    ok = deployer.apply("logs_lifecycle", desired) and deployer.verify("logs-*")
    sys.exit(0 if ok else 1)

The apply method short-circuits to a logged no-op when the live policy already equals the desired document, so re-running the pipeline after a network blip never issues a redundant write. The verify method fails the process when any managed index reports action.failed, which turns a bad policy into a red build instead of a slow-burn production problem.

Operational guardrails #

A deployment pipeline needs limits that keep a well-meaning change from becoming an outage. The settings and conventions below are the guardrails that separate a safe rollout from a risky one; treat the deviation budget as a hard gate, not a suggestion.

Keep the deviation budget explicit: if the plan diff touches a delete action or shortens any min_index_age, require a second reviewer, because those are the two edits that destroy data soonest. Pair the pipeline with continuous drift detection so a change that never went through git still gets caught and reconciled.

Troubleshooting #

The two most common real-world snags are the stale-token 409, which the conditional PUT is designed to produce (it is protecting you), and the “applied but nothing changed” surprise, which is OpenSearch correctly refusing to yank live indices onto a new policy version without an explicit change_policy. Both are features, not bugs, once the pipeline models them.

Frequently asked questions #

Related #

• Integrating ISM policy deployment into CI/CD pipelines — the concrete pipeline job that validates, diffs, applies, and gates on verification.
• Versioning ISM policies with a GitOps workflow — making git the source of truth with PR review, tagging, reconciliation, and rollback.
• Policy Drift Detection — the continuous audit loop that catches changes that bypass the pipeline.
• Python Orchestration Frameworks — packaging and shipping the deploy tooling itself.

Up: ISM Policy Implementation & Python Automation