Issue with Tornado and Cull_idle_servers

[rtadewald]

Hello.
I'm having an issue with TLJH on my company. When all my colleagues start to use the server, sometimes it freezes. Looking at the logs I saw the following error:

Mar 02 20:33:16 asimov_base python3[1270]: [E 200302 20:33:08 ioloop:763] Exception in callback functools.partial(<bound method IOLoop._discard_future_result of <tornado.platform.asyncio.AsyncIOMainLoop object at 0x7fc6152b6da0>>, <Future finished exception=HTTP 599: Operation timed out after 20000 milliseconds with 0 bytes received>)
Mar 02 20:33:16 asimov_base python3[1270]:     Traceback (most recent call last):
Mar 02 20:33:16 asimov_base python3[1270]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 743, in _run_callback
Mar 02 20:33:16 asimov_base python3[1270]:         ret = callback()
Mar 02 20:33:16 asimov_base python3[1270]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/ioloop.py", line 767, in _discard_future_result
Mar 02 20:33:16 asimov_base python3[1270]:         future.result()
Mar 02 20:33:16 asimov_base python3[1270]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 742, in run
Mar 02 20:33:16 asimov_base python3[1270]:         yielded = self.gen.throw(*exc_info)  # type: ignore
Mar 02 20:33:16 asimov_base python3[1270]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/cull_idle_servers.py", line 112, in cull_idle
Mar 02 20:33:16 asimov_base python3[1270]:         resp = yield fetch(req)
Mar 02 20:33:16 asimov_base python3[1270]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Mar 02 20:33:16 asimov_base python3[1270]:         value = future.result()
Mar 02 20:33:16 asimov_base python3[1270]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 748, in run
Mar 02 20:33:16 asimov_base python3[1270]:         yielded = self.gen.send(value)
Mar 02 20:33:16 asimov_base python3[1270]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tljh/cull_idle_servers.py", line 106, in fetch
Mar 02 20:33:16 asimov_base python3[1270]:         return (yield client.fetch(req))
Mar 02 20:33:16 asimov_base python3[1270]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 735, in run
Mar 02 20:33:16 asimov_base python3[1270]:         value = future.result()
Mar 02 20:33:16 asimov_base python3[1270]:     tornado.curl_httpclient.CurlError: HTTP 599: Operation timed out after 20000 milliseconds with 0 bytes received

Does anyone knows what could it be?

[betatim]

This could be related to jupyterhub/jupyterhub#2954

How many people are using your hub?

[rtadewald]

There are 4 users, but they use much RAM memory.

[consideRatio]

This may be a relevant issue still, but perhaps not. It is not critical though. If the idle-culler fails it means that it will not shutdown a server that is inactive at worst.

Since this issue was opened, #559 was merged where a new version of the jupyterhub-idle-culler was installed that may have better error handling. I think this failure is acceptable.

I'll go for a close, but keep my eyes open for any future issues like this while I triage all the issues in this repos.