Skip to main content

Set local OpenSearch disk watermarks

Use this guide to select fixed free-space reserves for local OpenSearch 2.18 in FullLocal or PartialRemote mode. A watermark is a disk threshold that restricts shard allocation or writes before storage becomes full.

The optional configuration affects only the local OpenSearch service. It does not change remote dependencies, grants, or operating-system limits. It keeps disk protection enabled.

Before you start​

Choose the physical runtime root that owns the running instance's .env, state, and volumes. An application source snapshot can differ from this root. See source selection.

With Docker access on the runtime host, locate the selected running instance:

  1. List the running OpenSearch containers:

    docker ps --filter label=com.docker.compose.service=opensearch --format '{{.ID}} {{.Names}}'

  2. Choose the exact container for the intended instance from its name and instance identity. Do not select another instance.

  3. Read that container's physical runtime root:

    docker inspect OPENSEARCH_CONTAINER_ID --format '{{index .Config.Labels "ecs.clone-root"}}'

Replace OPENSEARCH_CONTAINER_ID with the chosen container ID in every command below. The ecs.clone-root label gives the physical root. If the label is absent or the identity is unclear, stop before editing configuration.

Back up the selected instance's data, keys, and private environment configuration before changes. Preserve existing volumes. Use the environment configuration procedure for private files.

Choose reserves that suit the runtime's actual storage. The override is portable configuration, not hardware detection. Copying an enabled .env to another runtime enables it there too. Other runtimes remain off by default.

Set and apply the reserves​

  1. Open the selected physical runtime root's .env.

  2. Set these non-sensitive values:

    OPENSEARCH_DISK_WATERMARK_OVERRIDE_ENABLED=true
    OPENSEARCH_DISK_WATERMARK_LOW=15gb
    OPENSEARCH_DISK_WATERMARK_HIGH=10gb
    OPENSEARCH_DISK_WATERMARK_FLOOD_STAGE=5gb

  3. From that same root, check the configuration:

    bash devstack.sh validate

  4. Apply the change with the ordinary restart:

    bash restartDevServers.sh

An ordinary restart interrupts users and preserves data. Follow the supported lifecycle procedure and its capacity checks. A quick start resumes existing containers without applying new configuration. Use ordinary restart after this .env change, rather than a scoped service restart.

The enabled overlay sets the static cluster.routing.allocation.disk.watermark.enable_for_single_data_node setting to true. It also supplies the three startup watermarks. Byte values mean free space. Percentage defaults mean used space, so their ordering differs.

Check the effective settings​

From the runtime host, query the selected container with this read-only command:

docker exec OPENSEARCH_CONTAINER_ID curl --fail --silent --show-error 'http://127.0.0.1:9200/_cluster/settings?include_defaults=true'

After restart, repeat the container list above to get the current ID for the same instance. Replace OPENSEARCH_CONTAINER_ID with that ID.

This command uses loopback inside the container. The local service in compose.yaml sets DISABLE_SECURITY_PLUGIN=true and publishes no host port. Only this local service needs no API authentication. Remote authentication is outside this guide.

The command sends GET /_cluster/settings?include_defaults=true.

Inspect transient, persistent, and defaults for these exact keys:

cluster.routing.allocation.disk.watermark.low
cluster.routing.allocation.disk.watermark.high
cluster.routing.allocation.disk.watermark.flood_stage

Transient settings take precedence over persistent settings. Both take precedence over startup values. With no API overrides, the enabled configuration supplies 15gb, 10gb, and 5gb, unless you selected other valid sizes.

If an operator previously set these watermarks through the API, review those values before changing them. To return control to startup configuration, reset only the previously set keys to null in each affected scope. The following request applies only when all three keys need a reset in both scopes:

PUT /_cluster/settings
Content-Type: application/json

{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": null,
    "cluster.routing.allocation.disk.watermark.high": null,
    "cluster.routing.allocation.disk.watermark.flood_stage": null
  },
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": null,
    "cluster.routing.allocation.disk.watermark.high": null,
    "cluster.routing.allocation.disk.watermark.flood_stage": null
  }
}

Omit keys and scopes that the operator did not previously set. Never clear unrelated cluster or index settings. Do not disable disk protection. Repeat the GET request after an approved reset.

The disk monitor normally checks space about every 30 seconds. Flood-stage read-only index blocks release after disk usage drops below the high watermark. With free-space values, available space must exceed the high reserve.

After a monitor cycle, repeat the settings GET command. Also inspect active blocks:

docker exec OPENSEARCH_CONTAINER_ID curl --fail --silent --show-error 'http://127.0.0.1:9200/_cluster/state/blocks'

No global block 10 and no index block 12 means no active index-creation or flood-stage block. A persistent cluster.blocks.create_index=true setting can remain after restart even when no active global block exists. Do not assume that the monitor always clears this stored flag. In OpenSearch 2.18, its automatic release path requires an active global block 10.

If the historical persistent flag remains, first confirm all three conditions:

  • Available disk bytes exceed the effective high reserve.
  • The effective cluster.blocks.create_index.auto_release setting is true.
  • The flag has known automatic disk-monitor provenance, rather than a manual operator restriction.

Inspect available bytes through the same selected container:

docker exec OPENSEARCH_CONTAINER_ID curl --fail --silent --show-error 'http://127.0.0.1:9200/_nodes/stats/fs'

Use fs.total.available_in_bytes, not only free_in_bytes. If any condition is uncertain, stop and diagnose the cause. After these checks, an authorized operator can reset only the historical persistent flag:

PUT /_cluster/settings
Content-Type: application/json

{
  "persistent": {
    "cluster.blocks.create_index": null
  }
}

Repeat both GET requests after the approved reset. Never clear manual or unrelated blocks, index settings, or disk protection. Mailbox indexing acceptance is outside this guide. Service readiness alone does not prove that mailbox indexing completed.

See the OpenSearch 2.18 cluster settings reference and Cluster Settings API for the upstream contracts.

Return to the defaults​

  1. Set OPENSEARCH_DISK_WATERMARK_OVERRIDE_ENABLED=false in the selected runtime root's .env.
  2. Run bash devstack.sh validate from that root.
  3. Run bash restartDevServers.sh from that root.
  4. Check the effective settings with the GET request above.

The disabled configuration omits the overlay and preserves the original OpenSearch defaults: low 85%, high 90%, flood stage 95%. These defaults apply only when no stronger API overrides remain. Disabled size settings are ignored.

Configuration reference​

VariableConsumerPurposeType or formatRequired whenDefaultExampleConstraints or security
OPENSEARCH_DISK_WATERMARK_OVERRIDE_ENABLEDRoot lifecycle Compose configurationSelect the local OpenSearch overlayLiteral true or falseOptionalfalsetrueAn empty or other value is invalid. No secret.
OPENSEARCH_DISK_WATERMARK_LOWLocal OpenSearch overlaySet the low free-space reservePositive integer with lowercase mb or gbOptional when enabled15gb15gbMust exceed HIGH. Ignored when disabled. No secret.
OPENSEARCH_DISK_WATERMARK_HIGHLocal OpenSearch overlaySet the high free-space reservePositive integer with lowercase mb or gbOptional when enabled10gb10gbMust exceed FLOOD_STAGE. Ignored when disabled. No secret.
OPENSEARCH_DISK_WATERMARK_FLOOD_STAGELocal OpenSearch overlaySet the flood-stage free-space reservePositive integer with lowercase mb or gbOptional when enabled5gb5gbMust exceed zero. Ignored when disabled. No secret.

Missing or blank sizes use the stated defaults when enabled. Sizes must satisfy LOW > HIGH > FLOOD_STAGE > 0 after conversion to bytes. Each byte count must fit a positive signed 64-bit integer, at most 9223372036854775807 bytes. Here, 1mb means 1024² bytes and 1gb means 1024³ bytes.

Percentages, ratios, decimals, uppercase units, other units, and variable expressions are invalid size inputs. The lifecycle rejects invalid enabled settings before Compose changes resources. The enable flag comes from the runtime root's .env. A shell-only flag does not enable the overlay.