[meta] Pending image downloads of deleted containers are never interrupted, leading to possible DoS

[rgaiacs]

Related to #3056

The behaviour observed with GESIS server of many pending pods like in

that is also observed with OVH server like in

matches the description of issue reported upstream kubernetes/kubernetes#122905:

When an user deploys a pod, the deployed container(s) may rely on image(s) that are not present on the target node, triggering a download by the container runtime. However, if that pod is deleted (either with or without the --force flag) before the container runtime completes the download of the image, this download will be carried on anyway. Kubernetes tracks no information at all about the download, and there is no way of interrupting it unless the container runtime is forcibly terminated and recreated.

This (useless, as the images will never be used) sequence of downloads will waste the node’s CPU and network bandwidth and block the download queue, preventing the deployment of other containers with images yet to be downloaded. We experimented on a wide variety of settings, including but not limited to private clouds, public clouds and edge devices, with different container runtimes (containerd and CRI-O) and different Kubernetes providers (kubeadm, microk8s, and GKE).

@sgibson91 @manics @minrk @yuvipanda @arnim what do you think of us to drop the timeout that JupyterHub or BinderHub has and the automatic relaunch? This should reduce the number of pending pods.