Self-restarting cache pruner

[jc-harrison]

The flowmachine server runs a repeating task to shrink the query cache at regular intervals. However, if an error occurs during a cache shrink, this task is killed and does not restart.

It would be preferable for the server to restart the cache-prune task on failure. This could be achieved by forcing an error during cache-pruning to bring down the docker container, but this would have the adverse impact of interrupting any flowmachine queries running at the same time. I think it would probably be preferable to log the error message and restart the cache-pruning task without interrupting the container.

[greenape]

And/or to move the cache pruner to a separate container.

[jc-harrison]

A separate container is the way to go here, I think.

Currently the cache pruner runs as a background task within the main loop of the flowmachine server:

FlowKit/flowmachine/flowmachine/core/server/server.py

Lines 239 to 246 in 2d6f9d5

	main_loop.create_task(
	watch_and_shrink_cache(
	flowdb_connection=get_db(),
	pool=get_executor(),
	sleep_time=config.cache_pruning_frequency,
	timeout=config.cache_pruning_timeout,
	)
	)

I think we should instead have this as an entirely independent, alternative entrypoint for the flowmachine server. So when running in docker, one would have two flowmachine containers, one run via docker run flowminder/flowmachine which handles the usual flowmachine server stuff (running queries etc.), and a second container run with a different command, e.g. docker run flowminder/flowmachine cache-pruner, to handle the cache pruning. That way, we can have the cache pruner raise an error on failure, which will bring down the cache-pruner container (without interrupting the main flowmachine server), and docker can handle the restarting.