Scheduler plugin to dump cluster state on close

[gjoseph92]

We've run into issues where the scheduler unexpectedly cleanly shuts itself down after running for a very long time. Having a dump of cluster state would help to debug this.

After #5659 is implemented, write a SchedulerPlugin with a close hook that dumps cluster state. The filename where the state is written can be either passed into the plugin instance.

If the cluster state dump fails, or writing to the destination fails, this should not affect the shutdown process—just log the problem and move on.

[gjoseph92]

Actually, a SchedulerPlugin may not work for debugging #5675. The close callback on plugins doesn't run until after the workers have already been removed:

distributed/distributed/diagnostics/plugin.py

Lines 55 to 59 in 2a3ee56

	async def close(self):
	"""Run when the scheduler closes down
	This runs at the beginning of the Scheduler shutdown process, but after
	workers have been asked to shut down gracefully

We could, however, use a preload, since teardown() gets called before any workers are removed:

distributed/distributed/scheduler.py

Lines 4155 to 4159 in 2a3ee56

	for preload in self.preloads:
	await preload.teardown()
	if close_workers:
	await self.broadcast(msg={"op": "close_gracefully"}, nanny=True)

A preload will be a bit less ergonomic to install, though.

[ian-r-rose]

Actually, a SchedulerPlugin may not work for debugging #5675. The close callback on plugins doesn't run until after the workers have already been removed:

We could also add a before_close() plugin hook. I've also run into this issue before in a different context, and adding some more granularity to plugin lifecycles is pretty cheap.

[jrbourbeau]

Yeah, if adding a hook for before workers have been removed would be useful for other problems, then adding a new method seems sensible

[gjoseph92]

Sounds good. Looking back through blames, it seems that scheduler plugin behavior has always been this way, but without much explanation of why. Having this option would probably be useful in many cases (especially if you're doing something dangerous and have worker plugins doing some stateful thing, and want to communicate to your worker plugins before shutdown).

[gjoseph92]

Hm, I don't know what I was talking about here.

This runs at the beginning of the Scheduler shutdown process, but after workers have been asked to shut down gracefully

is only the case if close_workers=True is passed to close. This doesn't happen when invoked from the idle timeout (I don't know why—I feel like close_workers should be true if the cluster is shutting down due to idleness). It seems to only ever be set to True if Client.shutdown() is called and the client doesn't have a cluster object attached.

So though I'm not sure if close_workers actually should be True for the idle timeout, for the purposes of debugging #5675, we don't a before_close hook or anything like that.