Elasticsearch / OpenSearch 'flood stage disk watermark [95%] exceeded' - indices become read-only ('index read-only / allow delete', HTTP 429 cluster_block_exception): how do I recover?

Cause: Elasticsearch watches disk usage per node. At the low watermark (default 85%) it stops allocating new shards there, at the high watermark (default 90%) it tries to move shards away, and at the flood-stage watermark (default 95%) it applies the block index.blocks.read_only_allow_delete to every index that has a shard on that node, so writes fail with a cluster_block_exception (HTTP 429) while reads and deletes still work. OpenSearch has the same mechanism with equivalent cluster.routing.allocation.disk.watermark settings; check its docs for exact defaults. In Elasticsearch the block is released automatically once disk usage falls below the high watermark. Fix (best first): 1. Confirm the cause in the logs (the message names the node and threshold) and check usage per node with GET _cat/allocation?v. 2. Free space for real: delete indices you no longer need (for example old time-based indices; use ILM to automate), or add disk or nodes to the affected tier. Elasticsearch relocates shards on its own once nodes are below the high watermark; check GET _cluster/health and the cat recovery API. 3. If shards will not move, run the cluster allocation explain API for one shard to see why (rules such as awareness, filtering, or all nodes being full). 4. As temporary relief you can raise the three watermarks and clear the block on affected indices by setting index.blocks.read_only_allow_delete to null in the index settings. The Elastic guide presents this strictly as a stopgap. 5. Afterwards reset the watermark settings to null to return to defaults. Elastic recommends the defaults, since running disks nearly full leaves no room for merges and recovery. 6. Prevent recurrence: alert well before 85%, use lifecycle policies to move or delete data, and avoid mixing very large and very small indices. Verify: GET _cat/allocation?v shows every node under the low watermark, and a test index write returns 201 instead of a cluster_block_exception. Sources: https://www.elastic.co/guide/en/elasticsearch/reference/current/fix-watermark-errors.html https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html