Improve node health checks to detect slow disk IO

[Bukhtawar]

This is a follow-up from #52680 (comment) where we'd want to detect slow disk IO, mark the node UNHEALTHY and eventually remove such node from the cluster as a single slow disk can cause operations like indexing to slow down the entire _bulk.

We need to finalise reasonable thresholds for marking the node UNHEALTHY.

[DaveCTurner]

I think this is covered by our earlier discussions:

... once again we feel that your problems with IO that hangs indefinitely are most likely a bug in your infrastructure and not something that Elasticsearch should be working around. This doesn't seem to be a problem we experience in practice on a wide variety of production systems, and we still cannot find evidence that any other software has needed to implement protections against indefinitely unresponsive disks either.

Therefore I am closing this.

[Bukhtawar]

I think this is around slow disk and not IO hangs @DaveCTurner do you think disks don't degrade, espl Network backed file systems(I think a lot of folks use this.) I remember we enabled slow logging threshold for detecting such problems.

[DaveCTurner]

I think the best way to address that would be to document a more explicit recommendation to avoid using network file systems for the data path. Today we say to avoid them for performance reasons, but as you note they are also bad for resilience. I've added this to my TODO list.

[Bukhtawar]

Thanks @DaveCTurner while as you agree network backed disks aren't resilient enough the way it stands. But do you think this proposal can improve resiliency for these systems if you think this might be straight forward and user configurable, maybe off by default or a reasonably safe(high) default