Attaching ISM policies to thousands of indices in bulk

This walkthrough builds a Python script that attaches one Index State Management (ISM) policy across thousands of pre-existing indices safely — discovering the matching set, chunking it into fixed-size batches, attaching each batch with exponential backoff, and recording every outcome so the run is resumable. It is the deep implementation behind the pipeline introduced in Bulk Policy Attachment, for the day you turn on a lifecycle policy over an OpenSearch cluster that already holds months of daily indices.

Prerequisites #

Confirm each of these before the first run — an over-broad pattern or an unscoped service account is the usual reason a large attach either misses indices or trips permission errors halfway through.

• The target policy already exists: GET _plugins/_ism/policies/bulk_hot_delete returns 200. add fails for every index if the policy_id is unknown.
• You have the exact index pattern the attach should cover, and you have listed it once with GET _cat/indices/<pattern>?h=index&s=index to confirm it matches only what you intend — no system or .opendistro indices.
• A scoped service account exists whose role grants cluster:admin/opendistro/ism/managedindex/add and read (indices:monitor/*, indices:admin/get) on the target pattern, and nothing broader — scoping is detailed in Security & Access Boundaries.
• python >= 3.10 and opensearch-py are installed.
• You have a steady-state value for plugins.index_state_management.job_interval recorded, so you can widen it during the attach and restore it afterward.

Step-by-step procedure #

The script is built in five stages: compute the batch plan, discover the indices that still need the policy, attach one batch with backoff, drive the batched loop with a throttle, and persist a resumable ledger of results.

1. Compute the batch plan #

Given N target indices and a batch size s, the number of attach calls is a ceiling division:

For 12,400 indices at s = 250, that is $B = \lceil 49.6 \rceil = 50$ calls. Keep s in the 100–500 range: small enough that each call’s coordinator-initialisation cost stays bounded, large enough that per-request overhead does not dominate. Express the plan in code so logs report progress as “batch 7 of 50”.

import math

def batch_count(total: int, batch_size: int) -> int:
    """Number of attach calls needed: ceil(N / s)."""
    if batch_size < 1:
        raise ValueError("batch_size must be >= 1")
    return math.ceil(total / batch_size)

batch_count(12400, 250) -> 50
batch_count(999, 250)    -> 4
batch_count(0, 250)      -> 0

Gotcha: an empty target list must yield 0, not 1. math.ceil(0/250) is 0, so the loop below simply never runs — the resumable, idempotent design means “nothing to do” is a valid, successful outcome.

2. Discover only the indices that still need the policy #

Attaching an index that is already managed fails in-band, so discover current state first and keep only the indices whose policy_id is not already the target. GET _plugins/_ism/explain/<pattern> returns a policy_id (or null) per index.

from opensearchpy import OpenSearch

def discover_targets(client: OpenSearch, pattern: str, policy_id: str) -> list[str]:
    explain = client.transport.perform_request("GET", f"/_plugins/_ism/explain/{pattern}")
    targets = []
    for name, meta in explain.items():
        if name == "total_managed_indices" or not isinstance(meta, dict):
            continue
        current = meta.get("index.plugins.index_state_management.policy_id")
        if current != policy_id:            # unmanaged OR on a different policy
            targets.append(name)
    return sorted(targets)

[INFO] explain matched 12587 indices; 12400 need attach, 187 already on bulk_hot_delete

Gotcha: an index already on a different policy also shows up here, because its policy_id is not the target. add will reject it. If you intend to move those onto the new policy, route them to change_policy instead — this discovery step is where you split the two populations.

3. Attach one batch with exponential backoff #

A batch is a comma-separated list of concrete index names in the add path, with the policy_id in the body. Retry only on 429 too_many_requests — that is backpressure and safe to replay — and re-raise everything else so a real error stops the run instead of looping.

import time, logging
from opensearchpy import exceptions

logger = logging.getLogger("ism.attach")

def attach_batch(client, policy_id, indices, max_retries=4, base_delay=2.0) -> dict:
    target = ",".join(indices)
    for attempt in range(max_retries):
        try:
            return client.transport.perform_request(
                "POST", f"/_plugins/_ism/add/{target}", body={"policy_id": policy_id}
            )
        except exceptions.TransportError as exc:
            if getattr(exc, "status_code", None) == 429 and attempt < max_retries - 1:
                delay = base_delay * (2 ** attempt)
                logger.warning("429; backoff %.1fs (attempt %d)", delay, attempt + 1)
                time.sleep(delay)
                continue
            raise
    return {}

{ "updated_indices": 250, "failures": false, "failed_indices": [] }

Gotcha: inspect failed_indices, not just the HTTP status. A 200 with entries in failed_indices (already-managed, closed, or blocked indices) is a partial success — the loop must record those names so verification and reconcile can pick them up later.

4. Drive the batched loop with a throttle #

Iterate the plan, attaching one batch per call and pausing between calls so config-document creation spreads across coordinator sweeps rather than landing in one initialisation storm. This is the client half of the throttle; widening plugins.index_state_management.job_interval on the OpenSearch cluster is the server half.

from itertools import islice

def chunked(seq, size):
    it = iter(seq)
    while batch := list(islice(it, size)):
        yield batch

def run_attach(client, policy_id, pattern, batch_size=250, pause_s=3.0):
    targets = discover_targets(client, pattern, policy_id)
    total_batches = batch_count(len(targets), batch_size)
    ledger = []
    for i, batch in enumerate(chunked(targets, batch_size), start=1):
        logger.info("batch %d/%d: %d indices", i, total_batches, len(batch))
        result = attach_batch(client, policy_id, batch)
        ledger.append({"batch": i, "indices": batch, "result": result})
        time.sleep(pause_s)           # throttle: protect the coordinator sweep
    return ledger

Gotcha: concurrencyPolicy: Forbid on any scheduler that runs this, plus a single worker per cluster, is what stops two attach runs from racing. Parallel workers multiply the config-write rate and defeat the pause — the coordinator sees one combined storm regardless.

5. Persist a resumable ledger #

Write the ledger to disk after the run so a crashed or interrupted attach resumes from where it stopped. Because discovery in step 2 already skips managed indices, resuming is simply re-running: the next pass discovers a shorter target list.

import json
from pathlib import Path

def save_ledger(ledger, path="attach-ledger.json"):
    failed = [ix for b in ledger for ix in b["result"].get("failed_indices", [])]
    summary = {
        "batches": len(ledger),
        "attached": sum(b["result"].get("updated_indices", 0) for b in ledger),
        "failed_indices": failed,
    }
    Path(path).write_text(json.dumps({"summary": summary, "ledger": ledger}, indent=2))
    logger.info("ledger saved: attached=%d failed=%d", summary["attached"], len(failed))
    return summary

[INFO] ledger saved: attached=12400 failed=0

Gotcha: the ledger is your input to verification, not proof of success. attached=12400 means the API accepted 12,400 indices; whether each one actually initialised is a separate question answered by Verifying bulk ISM policy attachment at scale.

The full run is a straight-line pipeline from discovery to a saved ledger, with the throttle pacing the loop:

Verification #

Two quick checks confirm the run did what the ledger claims before you trust it. Full-fleet verification is its own procedure, but these catch the obvious misses immediately.

# 1. How many indices does ISM now consider managed under this pattern?
GET _plugins/_ism/explain/logs-2026.06-*
# tail of the response:
# "total_managed_indices": 12587   <- expect discovered count + previously managed

# 2. Spot-check a single index landed on the expected policy and initialised
GET _plugins/_ism/explain/logs-2026.06.01-000001
# "index.plugins.index_state_management.policy_id": "bulk_hot_delete",
# "state": { "name": "hot" }, "enabled": true

A healthy result shows total_managed_indices equal to your ledger’s attached count plus any indices already managed before the run, and a spot-checked index reporting both the expected policy_id and a concrete state — not a null state, which would mean it was attached but never initialised by the sweep.

Common failures #

Frequently asked questions #

Related #

• Bulk Policy Attachment — the section overview of the whole discover-batch-attach-verify pipeline.
• Verifying bulk ISM policy attachment at scale — proving every attached index is managed on the expected policy.
• Python Orchestration Frameworks — packaging and scheduling this attach script as production automation.

Up: Bulk Policy Attachment