Integrating ISM policy deployment into CI/CD pipelines

This walkthrough builds one concrete GitLab CI pipeline job that validates an OpenSearch ISM policy, computes a plan/diff against the target cluster, applies it with an idempotent conditional PUT, and gates the merge on a post-deploy verification that reads the policy back. It is the hands-on job behind the promotion model in CI/CD Pipeline Integration, aimed at teams who already keep policies in git but still apply them by hand and want the deploy to become a reviewable, replayable pipeline stage instead.

Prerequisites #

Confirm each of these before wiring the job — a missing scoped token or an un-stripped _seq_no in the committed file is the usual reason the first pipeline run either fails to authenticate or reports an endless spurious diff.

• The policy JSON lives in the repository at a stable path (e.g. policies/logs_lifecycle.json) and contains no cluster metadata (_id, _seq_no, _primary_term, _version) — only the {"policy": {...}} object.
• Two scoped service accounts exist: a read-only identity for the diff stage and a write identity for apply, each restricted to its target cluster per Security & Access Boundaries.
• Their credentials are stored as masked, protected CI/CD variables (OS_READ_TOKEN, OS_WRITE_TOKEN) — never in the repo.
• A protected environment or manual-approval gate is configured on the branch that applies to production.
• python >= 3.11 and opensearch-py are available in the job image.
• The target cluster is reachable from the CI runner’s network (a private runner or an allow-listed egress IP).

How the job flows #

The pipeline is three sequential stages with one hard gate. Validation runs with no cluster access; the diff runs read-only and prints the change set into the merge request; apply runs only after the diff is approved, guards its write with the concurrency tokens read during the diff, and immediately verifies. A failure anywhere before apply means the OpenSearch cluster is never touched.

Step-by-step procedure #

The four stages map one-to-one to jobs in a single .gitlab-ci.yml. Build them in order; each stage assumes the previous one passed.

1. Validate the policy with no cluster access #

The first job proves the committed file is a structurally valid ISM policy before any credential is loaded. It parses the JSON (a lint failure raises here) and checks that default_state names a real state and every transition targets a defined state.

stages: [validate, diff, apply]

validate:
  stage: validate
  image: python:3.11-slim
  script:
    - python ci/validate_policy.py policies/logs_lifecycle.json
  rules:
    - changes: [policies/**/*]

$ python ci/validate_policy.py policies/logs_lifecycle.json
policy 'logs_lifecycle' OK — 3 states, default_state=hot, 2 transitions resolved

Gotcha: run validation on a slim image with no network egress at all. If this stage can reach the OpenSearch cluster, a contributor’s typo can eventually reach it too — keep the read/write tokens out of the job entirely.

2. Diff the desired policy against the live cluster #

The diff job fetches the currently attached policy read-only, strips the OpenSearch cluster-owned concurrency metadata, and compares the remaining policy object to the committed file. It writes the diff into the merge request and stashes the _seq_no/_primary_term as a job artifact so the apply job can reuse them.

diff:
  stage: diff
  image: python:3.11-slim
  before_script: [pip install --quiet opensearch-py]
  script:
    - python ci/deploy.py --plan --emit-tokens tokens.env policies/logs_lifecycle.json
  artifacts:
    reports: { dotenv: tokens.env }     # carries IF_SEQ_NO / IF_PRIMARY_TERM forward
  variables:
    OS_HOST: $OS_PROD_HOST
    OS_TOKEN: $OS_READ_TOKEN            # read-only
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

$ python ci/deploy.py --plan ...
plan logs_lifecycle: UPDATE (live _seq_no=42 _primary_term=6)
  states.warm.transitions[0].conditions.min_index_age: "30d" -> "21d"
1 change — review required before apply

Gotcha: strip _id, _seq_no, _primary_term, and _version from the fetched document before diffing. Leave any of them in and the diff is never empty, because those tokens change on every write — you would review the same phantom change forever. This is the single most common reason a “working” pipeline reports noise; the plan/diff contract is spelled out in the parent CI/CD Pipeline Integration guide.

3. Apply with a conditional PUT behind a manual gate #

The apply job is when: manual so it will not run until a reviewer clicks it, and it only exists on the default branch. It consumes the tokens from the diff artifact and passes them as if_seq_no/if_primary_term, so if the live policy changed since the diff, the write is rejected rather than clobbering the newer state.

apply:
  stage: apply
  image: python:3.11-slim
  before_script: [pip install --quiet opensearch-py]
  script:
    - python ci/deploy.py --apply --verify
        --if-seq-no "$IF_SEQ_NO" --if-primary-term "$IF_PRIMARY_TERM"
        policies/logs_lifecycle.json
  environment: { name: production }     # protected env injects the write token
  variables:
    OS_HOST: $OS_PROD_HOST
    OS_TOKEN: $OS_WRITE_TOKEN
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: manual                       # human approval gate

The deploy.py core is a single conditional request. On update it sends the tokens; a mismatch returns 409 and the job fails loudly:

def apply(client, policy_id, desired, seq_no, primary_term):
    params = {"if_seq_no": seq_no, "if_primary_term": primary_term} if seq_no else {}
    try:
        client.transport.perform_request(
            "PUT", f"/_plugins/_ism/policies/{policy_id}", params=params, body=desired)
        logging.info("applied policy_id=%s guarded=%s", policy_id, bool(seq_no))
    except exceptions.ConflictError:
        logging.error("stale tokens for %s — re-run diff and retry", policy_id)
        raise SystemExit(1)

Gotcha: the 409 is the guardrail doing its job. Do not “fix” it by dropping the tokens — that reintroduces the clobber it prevents. Re-run the diff to read fresh tokens, review the new change set, then re-apply.

4. Verify inside the same job and fail closed #

Verification runs in the apply job immediately after the PUT, so an apply that attaches a broken policy turns red before the pipeline reports success. It reads GET _plugins/_ism/explain/<pattern> and fails the job if any managed index reports action.failed.

explain = client.transport.perform_request("GET", "/_plugins/_ism/explain/logs-*")
failed = [i for i, s in explain.items()
          if isinstance(s, dict) and s.get("action", {}).get("failed")]
if failed:
    raise SystemExit(f"verify failed for {failed}")
logging.info("verify-ok managed=%d", len(explain))

Verification #

After the first gated apply, confirm three things: the policy version advanced, the intended indices are managed with no failed action, and — if you moved existing indices — they landed on the new version.

# 1. The stored policy reflects the committed change and a bumped _seq_no
GET _plugins/_ism/policies/logs_lifecycle
# "_seq_no": 43, "_primary_term": 6, "policy": { ... "min_index_age": "21d" ... }

# 2. Managed indices show no failed action
GET _plugins/_ism/explain/logs-*
# "logs-000012": { "state": {"name":"hot"}, "action": {"failed": false} }

# 3. If you rolled existing indices forward, they run the new policy version
POST _plugins/_ism/change_policy/logs-000009
# { "updated_indices": 1, "failures": false }

A healthy result shows the _seq_no incremented by exactly one per apply, every managed index at "failed": false, and no index still pinned to the pre-change policy version. A pipeline that keeps drifting back should hand off to the continuous audit loop in Policy Drift Detection, which catches out-of-band edits between deploys.

Common failures #

Frequently asked questions #

Related #

• CI/CD Pipeline Integration — the promotion model and guardrails this job implements.
• Versioning ISM policies with a GitOps workflow — making git the reconciliation source of truth and rolling back a bad version.
• Policy Drift Detection — catching out-of-band edits that never went through this pipeline.

Up: CI/CD Pipeline Integration