[Feature Request] Make elastic search crash after `Caused by: java.io.IOException: No space left on device`, rather than spamminng logs

[nullpixel]

Describe the feature: As per #20354, logs are spammed with Caused by: java.io.IOException: No space left on device when the disk space runs out. Why not make it leave a single message in the logs, then just crash? As the disk is full, the database cannot run anyway, so rather than spamming logs, just adapt a simpler approach.

This is a production database, and I wouldn't expect my logs to look like that after waking up.

[martijnvg]

I don't think that crashing the node is the best approach when there is no space left on device. The node can still serve read requests and if indices are removed/relocated (or in this case old log files are removed) then there should be sufficient space to handle write requests.

In case of log files filling up, maybe ES should try to prevent this by capping the size of the log file, for example like is done for deprecation logging (with log4j's SizeBasedTriggeringPolicy)?

[clintongormley]

Also, a disk might fill up because of a big merge. As soon as that fails, disk space could be freed up again allowing the node to continue working.

[khionu]

In that case, it might be best to put a hold on the operations that are unable to complete.

A queue, that will retry at a much lower rate, until operations are successful, would appease the concerns of being available for read and waiting for disk to be freed, without causing undue spam. For an exception that requires manual intervention, a retry rate of once a minute would be acceptable, in my opinion.

[s1monw]

I agree killing the node might not be the right thing here from a 10k feet view while I can see it as a viable solution for some use - cases. Out of the box I'd like us to rather detect that we are close to full disk and then simply stop write operations like merging and indexing. Yet, this still has issues since if we have a replica on the node we'd reject a write which would bring the replica out of sync in that moment. Disk-threshold allocation deciders should help here moving stuff away from the node but essentially killing it and leave the cluster would be the right thing to do here IMO. If we wanna do that we need to somehow add corresponding handlers in many places or we start with adding it into the engine and allow folks to opt out of it? I generally think we should kill nodes more-often in disaster situations instead of just sitting there and wait for more disaster to happen.

[nullpixel]

Yeah, so you cannot write to the disk if it's full, but spamming the logs isn't the way to go.

Could do a "read only" style mode where it just says it went into read only because of no disk space. there's no need to spam the logs; even with a cap

[jasontedor]

Yeah, so you cannot write to the disk if it's full, but spamming the logs isn't the way to go.

This is not necessarily true, the logs could (and should!) be on a separate mount, they could have log rotation applied to them, etc.

[jasontedor]

I generally think we should kill nodes more-often in disaster situations instead of just sitting there and wait for more disaster to happen.

I agree but I'm unsure if disk-full qualifies as such a disaster situation since it's possible to recover.

[jasontedor]

Why not make it leave a single message in the logs, then just crash? As the disk is full, the database cannot run anyway, so rather than spamming logs, just adapt a simpler approach.

In a concurrent server application, there are likely many disk operations in flight, expecting a single message is not realistic. As others have mentioned, this situation is not completely fatal, and can be recovered from so crashing on disk-full should not be a first option.

[khionu]

One option is to provide a few behaviors for the host to pick from. Fatal crash, start refusing writes, or something else.

Something else to be considered is maybe preventative? It should be reasonable to check the space remaining daily, and if it goes below, say, 5%, log a severe warning.

[s1monw]

@khionu just out of curiosity did you look into this

[s1monw]

I agree but I'm unsure if disk-full qualifies as such a disaster situation since it's possible to recover.

the question is what the recovery path is? Most likely we need to relocate shards but shouldn't the disk-threshold decider have taken care of this already? Once we are like at 99% there is not much we can do but failing? it's likely the most healthy option here, it tells the cluster to heal itself by allocating shards on other nodes, it notifies the users since a node died. The log message might be clear and we can refuse to start up until we have at lest 5% disk back? I kind of like this option the more I think about it.