Bulk Policy Attachment

Bulk policy attachment is the problem of binding one Index State Management (ISM) policy to thousands of existing indices at once — the day-two task that appears the moment you introduce a lifecycle policy on an OpenSearch cluster that already carries months of daily log indices. A single POST _plugins/_ism/add/logs-* call looks like the answer, but at fleet scale it hides three failure surfaces: the ISM coordinator has to initialise every newly managed index on its next sweep, so attaching ten thousand indices in one request spikes coordinator load; a naive re-run collides with indices that are already managed and reports partial failures you have to interpret; and “the call returned 200” is not the same as “every index is managed on the right policy.” This page, part of ISM Policy Implementation & Python Automation, shows how to attach a policy across large index sets deterministically — batching the work, throttling to protect the sweep, making re-runs idempotent, and proving attachment through _plugins/_ism/explain rather than trusting a status code.

Attach approaches and their scale profile #

There are two primitives for attaching a policy, and the right one depends on how many indices you touch, whether the set is stable, and how much control you need over failure reporting. The table aligns each approach with the index-count band where it stays safe, the pressure it puts on the ISM coordinator, and how it behaves on a second run.

The wildcard add is the correct tool for a few hundred indices created in the last day or two. Past that, the chunked approach — discover the matching indices, split them into fixed-size batches, and attach each batch with a pause between calls — is what keeps the coordinator’s sweep from stalling and gives you a per-index audit trail. Use change_policy rather than add when the indices are already managed and you only want to move them onto a newer policy version; add refuses an index that already has a policy, whereas change_policy re-points it cleanly. The template policy_id route only ever helps future indices, so it complements a bulk attach but never replaces it.

The batched attach pipeline #

Bulk attachment at scale is a pipeline, not a single API call. You discover the target indices, filter out the ones already managed so the run is idempotent, split the remainder into fixed-size batches, attach each batch with a throttle pause and bounded retries, record what each batch returned, and finally sweep _plugins/_ism/explain to confirm every index landed on the expected policy — feeding any stragglers back into a reconcile pass. The diagram below shows that flow and where the two verification gates sit.

The two gates on the right of that diagram are what separate a bulk attach from a bulk hope. The idempotency filter early in the pipeline lets you re-run the whole job after a partial failure without tripping over already-managed indices, and the explain sweep at the end is the only authority on whether the job actually succeeded. The deep procedural build of the attach side lives in Attaching ISM policies to thousands of indices in bulk; the verification and reconcile side is covered in Verifying bulk ISM policy attachment at scale.

1. Create the policy before attaching anything #

An add call fails for every index if the policy id does not exist, so the policy is always the first object to deploy. Keep the policy body itself minimal here — this page is about attachment, not lifecycle design — but note the policy_id you PUT, because both the add call and the explain verification key on it.

PUT _plugins/_ism/policies/bulk_hot_delete
{
  "policy": {
    "description": "Baseline lifecycle attached in bulk to existing logs indices",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [ { "rollover": { "min_index_age": "1d", "min_primary_shard_size": "50gb" } } ],
        "transitions": [ { "state_name": "delete", "conditions": { "min_index_age": "30d" } } ]
      },
      { "name": "delete", "actions": [ { "delete": {} } ] }
    ]
  }
}

2. Pattern add versus per-index add #

POST _plugins/_ism/add/<index> attaches the named policy to whatever the target resolves to. The target can be a single concrete index, a comma-separated list, or a wildcard pattern. The body carries the policy_id.

# Wildcard add: every current match initialises on the next coordinator sweep
POST _plugins/_ism/add/logs-2026.06-*
{ "policy_id": "bulk_hot_delete" }

The response reports how many indices were updated and which, if any, failed:

{
  "updated_indices": 187,
  "failures": false,
  "failed_indices": []
}

A wildcard is convenient but blunt: every match is scheduled to initialise on the same sweep, and the whole call is one transaction from your side — you get a single aggregate result with no control over ordering or pacing. For a few hundred indices that is fine. For thousands, prefer building explicit batches of concrete index names so you can pace and audit them, which is what step 3 sets up.

3. Batching over thousands of indices #

Split the discovered index list into fixed-size batches and attach one batch per call. Given N target indices and a batch size s, the number of attach calls is

so 12,400 indices at s = 250 is 50 calls rather than one 12,400-index request. A batch size in the 100–500 range is the sweet spot: small enough that each call’s initialisation cost stays bounded, large enough that request overhead does not dominate. The add API accepts a comma-separated list, so a batch maps directly to one call.

# One batch of concrete index names (comma-separated in the path)
POST _plugins/_ism/add/logs-2026.06.01-000001,logs-2026.06.01-000002,logs-2026.06.02-000001
{ "policy_id": "bulk_hot_delete" }

Batching also localises failure. If one batch throws a 429 under load, you retry that batch alone rather than replaying the entire fleet, and your result record tells you exactly which 250 indices to re-check.

4. Throttling and sweep pressure #

Attaching a policy writes a managed-index config document; the ISM coordinator picks those up on its sweep and initialises each one, moving it into its default_state. The sweep interval is governed by plugins.index_state_management.job_interval (default 5 minutes). Push thousands of new config documents in seconds and the next sweep tries to initialise all of them at once, competing with real ISM work — rollovers and transitions — already scheduled. The fix is a deliberate pause between batches so initialisation spreads across several sweeps instead of landing on one.

# Widen the sweep window while a large bulk attach is in flight, then restore it
PUT _cluster/settings
{
  "transient": {
    "plugins.index_state_management.job_interval": 10,
    "plugins.index_state_management.jitter": 0.6
  }
}

Raising job_interval during the attach lengthens the window each sweep has to absorb newly managed indices; the jitter fraction spreads job execution so managed indices do not all wake on the same tick. Keep these transient so an OpenSearch cluster restart restores your steady-state values. Pair the server-side widening with a client-side pause between batches — the two together are what keep a bulk attach from degrading live lifecycle work.

5. Idempotent re-runs #

add refuses an index that already has a policy: that index appears in failed_indices with a message noting it is already managed. This is exactly what you want for safety, but it means a blind re-run of a partially-completed job reports a wall of “failures” that are really no-ops. Make the run idempotent by discovering current state first and attaching only the unmanaged indices. GET _plugins/_ism/explain/<pattern> returns the policy_id (or none) for every match.

# Which of these indices are already managed, and on what policy?
GET _plugins/_ism/explain/logs-2026.06-*

{
  "logs-2026.06.01-000001": { "index.plugins.index_state_management.policy_id": "bulk_hot_delete", "enabled": true },
  "logs-2026.06.02-000001": { "index.plugins.index_state_management.policy_id": null, "enabled": null },
  "total_managed_indices": 1
}

An index whose policy_id is null is unmanaged and belongs in the next attach batch; one already on bulk_hot_delete is done; one on a different policy is a reconcile case for change_policy, not add. Filtering on this before every batch turns a re-run into a safe convergence step — it only ever acts on what still needs acting on.

Production automation #

The following opensearch-py manager wraps the whole pipeline: discover, filter for idempotency, chunk, attach with bounded retries and structured logging, and record every batch’s outcome. Scope its service account narrowly first — a bulk-attach identity needs cluster:admin/opendistro/ism/managedindex/add and read on the target patterns, and nothing else. Endpoint-level scoping is covered in Security & Access Boundaries; the packaging and scheduling of a manager like this is the subject of Python Orchestration Frameworks.

import logging
import time
from itertools import islice
from opensearchpy import OpenSearch, exceptions

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("ism.bulk_attach")


def chunked(seq, size):
    """Yield successive fixed-size batches from a list."""
    it = iter(seq)
    while batch := list(islice(it, size)):
        yield batch


class BulkPolicyAttacher:
    def __init__(self, client: OpenSearch, policy_id: str):
        self.client = client
        self.policy_id = policy_id

    def discover_unmanaged(self, pattern: str) -> list[str]:
        """List indices matching pattern that are not yet on this policy (idempotency)."""
        explain = self.client.transport.perform_request(
            "GET", f"/_plugins/_ism/explain/{pattern}"
        )
        unmanaged = []
        for name, meta in explain.items():
            if not isinstance(meta, dict) or name == "total_managed_indices":
                continue
            current = meta.get("index.plugins.index_state_management.policy_id")
            if current != self.policy_id:
                unmanaged.append(name)
        logger.info("discover: %d index(es) need attach on %r", len(unmanaged), pattern)
        return unmanaged

    def attach_batch(self, indices: list[str], max_retries: int = 4) -> dict:
        """Attach one batch with exponential backoff; return the API result."""
        target = ",".join(indices)
        for attempt in range(max_retries):
            try:
                result = self.client.transport.perform_request(
                    "POST", f"/_plugins/_ism/add/{target}",
                    body={"policy_id": self.policy_id},
                )
                logger.info("batch ok: updated=%s failed=%s",
                            result.get("updated_indices"), result.get("failed_indices"))
                return result
            except exceptions.TransportError as exc:
                status = getattr(exc, "status_code", None)
                if status == 429 and attempt < max_retries - 1:
                    delay = 2.0 * (2 ** attempt)
                    logger.warning("429 on batch; backing off %.1fs (attempt %d)", delay, attempt + 1)
                    time.sleep(delay)
                    continue
                logger.error("batch failed permanently: %s", exc)
                raise
        return {}

    def run(self, pattern: str, batch_size: int = 250, pause_s: float = 3.0) -> list[dict]:
        """Full pipeline: discover -> chunk -> attach with throttle -> record."""
        targets = self.discover_unmanaged(pattern)
        results = []
        for i, batch in enumerate(chunked(targets, batch_size), start=1):
            logger.info("batch %d: attaching %d index(es)", i, len(batch))
            results.append(self.attach_batch(batch))
            time.sleep(pause_s)  # client-side throttle protects the coordinator sweep
        logger.info("attach complete: %d batch(es), %d index(es)", len(results), len(targets))
        return results


if __name__ == "__main__":
    client = OpenSearch(
        hosts=[{"host": "localhost", "port": 9200}],
        http_auth=("admin", "admin"),
        use_ssl=True,
        verify_certs=True,
    )
    BulkPolicyAttacher(client, "bulk_hot_delete").run("logs-2026.06-*", batch_size=250, pause_s=3.0)

The manager never attaches an index it can already see is on the target policy, retries only on 429 (backpressure, safe to replay) and re-raises everything else, and records each batch result so a downstream verification pass has something to reconcile against. It is safe to run twice: the second run discovers a much shorter unmanaged list and converges on zero.

Operational guardrails #

Bulk attachment is safe only when its pacing leaves the coordinator enough headroom to initialise managed indices without starving live lifecycle work. The settings below are the knobs that matter, with values tuned for a fleet in the low tens of thousands of indices.

Keep concurrency at a single bulk-attach worker per cluster. Running several in parallel multiplies the config-doc write rate and defeats the throttle — the coordinator sees one combined storm regardless of how many workers produced it. Restore job_interval and jitter to their steady-state values once the explain sweep confirms full attachment.

Troubleshooting #

A batch that fails wholesale on 429 is backpressure, not a broken payload — the same batch usually succeeds after a longer pause, which is why the automation retries only that status. An index stuck uninitialised after attachment almost always means the sweep is not keeping up; confirm job_interval is a value the coordinator can actually service before assuming the attach itself failed.

Frequently asked questions #

Related #

• Attaching ISM policies to thousands of indices in bulk — the step-by-step Python build that discovers, chunks, and attaches with backoff.
• Verifying bulk ISM policy attachment at scale — proving every index is managed on the expected policy and reconciling the stragglers.
• Python Orchestration Frameworks — packaging and scheduling an attach manager as production automation.
• Security & Access Boundaries — scoping the service account that a bulk-attach identity runs as.

Up: ISM Policy Implementation & Python Automation