Use idle culler from jupyterhub-idle-culler package #559
Conversation
|
If all tests pass, I will make a release of the idle culler and reference that instead of the git url |
|
Similar PR in z2jh: jupyterhub/zero-to-jupyterhub-k8s#1648 |
|
I'll probably move the repo to the JupyterHub org once a few people agree. |
There was a problem hiding this comment.
Yay ❤️
This is great @yuvipanda, thanks for working on it.
P.S: It would be great to have it in the JupyterHub org.
|
Upgrade tests are failing with JupyterLab builds - I suspect this is another memory issue. I have bumped up the memory limit to 2G to test that, but I hope this doesn't mean we have to bump our minimum requirement to 2G, just for JupyterLab builds :'( |
|
@yuvipanda, unfortunately the upgrade test is failing often when building jupyterlab :( ( Also, part of the solution in the past was this PR. |
|
yeah, 2G fixes it :( I'm trying 1.5G. See jupyterlab/jupyterlab#4824 for prior work |
The idle culler lives as a script in at least 3 different places: - In the JupyterHub repo, as an 'example' https://github.com/jupyterhub/jupyterhub/tree/d126baa443ad7d893be2ff4a70afe9ef5b8a4a1a/examples/cull-idle - In the TLJH repo, as a core part of the service https://github.com/jupyterhub/the-littlest-jupyterhub/blob/01ba34857dd4e316d839034ae2b3cc400b929964/tljh/cull_idle_servers.py. This is an import from a specific version of the JupyterHub repo, and has had a couple of changes made to it since. - In the z2jh repo, as a core part of the service https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/c3f3be25f8ae6c72d02f385f41983b70ee1d416e/jupyterhub/files/hub/cull_idle_servers.py This is also an import from a specific version of the JupyterHub repo, but has had a lot more work done on it. Most had been sync'd back the JupyterHub repo, but some had not been. See https://github.com/jupyterhub/zero-to-jupyterhub-k8s/commits/9c15a42b1227f3b54826f273f1689e4dc8c8e12e/images/hub/cull_idle_servers.py and https://github.com/jupyterhub/zero-to-jupyterhub-k8s/commits/master/jupyterhub/files/hub/cull_idle_servers.py The idle culler is a core integral part of every JupyterHub deployment these days. It would be great if it was maintained separately on its own, without being split across multiple repos. The latest changes had been to the version in the JupyterHub repo, so I copied it (while preserving commit history, because credit is important) to a new repository: https://github.com/yuvipanda/jupyterhub-idle-culler I looked through z2jh and tljh copies, and cherry-picked the following changes manually jupyterhub/zero-to-jupyterhub-k8s@ae80fb5 jupyterhub/zero-to-jupyterhub-k8s@836f19a jupyterhub/zero-to-jupyterhub-k8s@a0787c6 jupyterhub/zero-to-jupyterhub-k8s@b230ef8 jupyterhub@20374db#diff-f00cd100e9f673285208aaa6fc0c3212 There were a few from https://github.com/jupyterhub/zero-to-jupyterhub-k8s/commits/9c15a42b1227f3b54826f273f1689e4dc8c8e12e/images/hub/cull_idle_servers.py I could not apply, but mostly because those features had been re-implemented already. Right now, the package is a direct port of the code we had. Once this settles in, I am hopefull we can iterate faster and make cool new changes.
A v1.0 release has been made!
yuvipanda
force-pushed
the
feat/upstream-culler
branch
from
a47ad7c to
1c7e89a
Compare
|
Looks like I just didn't restart the tests enough times at 1G! I'm going to remove all the commits I made moving the memory limit around. This is ready to merge! \o/ |
|
hmmm, so it fails now but succeeds if I make a few commits that eventually bring the memory requirement to 1G? |
I couldn't figure out why JupyterLab fails to build with 1G on initial commit, but succeeds if you do a bunch of other commits after. Previously, I binary searched down from 2G. Here, I just add a new, no-op commit to see if that helps
|
@yuvipanda, I've restarted the tests and now they pass again 😕 |
|
I think we should merge this PR as it's a great addition ❤️ and open an issue about the tests failing intermittently again (😞) |
|
@GeorgianaElena ah, damn - re: intermittent. At least that makes more sense and I'm not going bananas... Would love to get this merged and open another issue for the flaky tests! |
|
Things seem to work as intented in the Z2JH project from what I can tell after trying it, and the changes look good to me! Merging given Georgianas approval! |
|
Thanks, @GeorgianaElena and @consideRatio! |
Has been moved to its own repo. See jupyterhub/the-littlest-jupyterhub#559 for more info
The idle culler lives as a script in at least 3 different
places:
https://github.com/jupyterhub/jupyterhub/tree/d126baa443ad7d893be2ff4a70afe9ef5b8a4a1a/examples/cull-idle
https://github.com/jupyterhub/the-littlest-jupyterhub/blob/01ba34857dd4e316d839034ae2b3cc400b929964/tljh/cull_idle_servers.py.
This is an import from a specific version of the JupyterHub repo,
and has had a couple of changes made to it since.
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/c3f3be25f8ae6c72d02f385f41983b70ee1d416e/jupyterhub/files/hub/cull_idle_servers.py
This is also an import from a specific version of the JupyterHub
repo, but has had a lot more work done on it. Most had been sync'd
back the JupyterHub repo, but some had not been. See
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/commits/9c15a42b1227f3b54826f273f1689e4dc8c8e12e/images/hub/cull_idle_servers.py
and https://github.com/jupyterhub/zero-to-jupyterhub-k8s/commits/master/jupyterhub/files/hub/cull_idle_servers.py
The idle culler is a core integral part of every JupyterHub deployment
these days. It would be great if it was maintained separately on
its own, without being split across multiple repos.
The latest changes had been to the version in the JupyterHub repo, so I
copied it (while preserving commit history, because credit is important)
to a new repository: https://github.com/yuvipanda/jupyterhub-idle-culler
I looked through z2jh and tljh copies, and cherry-picked the following
changes manually
jupyterhub/zero-to-jupyterhub-k8s@ae80fb5
jupyterhub/zero-to-jupyterhub-k8s@836f19a
jupyterhub/zero-to-jupyterhub-k8s@a0787c6
jupyterhub/zero-to-jupyterhub-k8s@b230ef8
20374db#diff-f00cd100e9f673285208aaa6fc0c3212
There were a few from https://github.com/jupyterhub/zero-to-jupyterhub-k8s/commits/9c15a42b1227f3b54826f273f1689e4dc8c8e12e/images/hub/cull_idle_servers.py
I could not apply, but mostly because those features had been
re-implemented already.
Right now, the package is a direct port of the code we had. Once
this settles in, I am hopefull we can iterate faster and make cool
new changes.