Support for automatic release of index block once disk frees up

[Bukhtawar]

The Problem

The DiskThresholdMonitor in Elasticsearch monitors for disk utilization across nodes periodically based on cluster.info.update.interval and ends up putting a INDEX_READ_ONLY_ALLOW_DELETE_BLOCK on every index that has atleast one shard on a node that breaches cluster.routing.allocation.disk.watermark.flood_stage but leaves those indices in the same state even after disk frees up, putting the onus on the end user to manually unblock those indices. This issue has been initiated as a part of #24299(comment)

Proposed Solution

The master is already checking for disk utilization across nodes to see if nodes are going over the flood threshold. So if we find any index block in the cluster state that should no longer be blocked based on the latest run of the disk monitor we should go ahead unblock those indices. The solution should mark index block which breach the threshold as INDEX_READ_ONLY_ALLOW_DELETE_BLOCK and the unblock the ones which already have blocks based on state.getBlocks().indexBlocked(ClusterBlockLevel.WRITE, index) but no longer match the indices. The only challenge would be to distinguish a customer requested block for SETTING_READ_ONLY_ALLOW_DELETE vs the one added by the disk monitor. To solve this we can either introduce a parameter in the request and a separate block to mark the indices from monitor differently.

[elasticmachine]

Pinging @elastic/es-distributed

[DaveCTurner]

We discussed this today and it seems like a reasonable change to make. The INDEX_READ_ONLY_ALLOW_DELETE_BLOCK was introduced purely to support the flood_stage watermark so we don't think there's a need for another kind of block.

We are concerned about flapping, i.e., being in a state where the block is added and removed and then added again in a loop, and the solution to this is not to remove the block as soon as the disk usage drops below the watermark itself. We think the block should be removed only when the disk usage is at least 5% below the flood_stage watermark and below the high watermark.

Marking this as help wanted because we do not anticipate working on this as a priority, but we welcome contributions from the community.

[alex101101]

Hi @DaveCTurner,
Could I look into this?

[DaveCTurner]

Sure @alex101101, go ahead.

[alex101101]

Thanks @DaveCTurner, I have just raised a pull request. Could you please review it when you have time?

[Bukhtawar]

@DaveCTurner would be great if you could spare sometime for the review

[Bukhtawar]

@DaveCTurner can you please review this. Also by

We think the block should be removed only when the disk usage is at least 5% below the flood_stage watermark and below the high watermark.

you mean with low/high/flood at 85/90/95 respectively release should happen at 5% less from 95 i.e below 90(which is also where high disk watermark is)