Integrate jupyterhub-idle-culler with KubeIngressProxy

[droctothorpe]

Proposed change

last_activity data for routes will be available when using configurable-http-proxy as JupyterHub does by default, but if for example traefik-proxy is used as it is in the TLJH distribution, no such data will be available.

We use JupyterHub + KubeIngressProxy + the nginx ingress controller. Based on the docs, I suspect that the last_activity is never updated, so notebooks are culled even when they're active.

Are there any plans to integrate these two capabilities? If not, would you be open to a PR to that effect (assuming it's possible)?

Thanks!

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[manics]

The idle culler is agnostic to the underlying spawner. Assuming you're right about the cause it sounds like the fix needs to be in KubeIngressProxy not this repo. Is that your understanding? If so I'll transfer the issue to KubeSpawner.

[droctothorpe]

Thanks for getting back to me, @manics!

KIP just creates the ingress resources. The nginx ingress controller is what's actually responsible for managing the proxy, and it's completely oblivious to the concept of last_activity, since that's a very JH / configurable-http-proxy convention.

I guess what I'm wondering if it makes sense to either modify jupyterhub-idle-culler or create a discrete culler that's more generic / isn't contingent on the use of configurable-http-proxy.

Some approaches that come to mind:

1. Evaluating idleness using a daemon on the notebook container that terminates the container locally instead of relying on a centralized service.
2. Determining idleness by checking HTTP requests frequency via prometheus.

Totally understand if this is out of scope. We're pretty unique in our adoption of KIP I suspect.

[droctothorpe]

Hm, alternatively, why not hit the /api/status endpoints of the notebooks? I think that's what KF does: https://github.com/kubeflow/kubeflow/blob/master/components/notebook-controller/pkg/culler/culler.go.

[manics]

There are some past discussions on supporting plugins or configurable hooks:

• Support custom culling logic through hooks (like Spawner's pre_spawn_hook etc) #25
• How configurable should the culler be? #9
• Adding the option to delete persisted user data as well. #8

No progress yet, but maybe your use-case could be a driver?

If you haven't already seen https://discourse.jupyter.org/t/which-is-the-correct-way-to-cull-idle-kernels-and-notebook/8123/7 it explains some of the complexity of idle detection?

Did you see https://discourse.jupyter.org/t/jupyterhub-amalthea/12208 (posted today)? Might be worth seeing how they implemented idle culling.