Description
Description
Connecting JupyterLab 4.2.2 or Vscode (Jupyter extension) to Google Colab Enterprise fails.
Scenario:
- JupyterLab 4.2.2 with Gateway enabled. (
c.GatewayClient.url
) - Colab Enterprise (Using Proxy URL)
In Colab Enterprise we have a internal proxy which expects session_id
Currently JupyterLab passes:
GET /api/kernels/342eba8f-aa8c-47a9-8fee-8541cee8ddfe/channels
vs
GET /api/kernels/342eba8f-aa8c-47a9-8fee-8541cee8ddfe/channels?session_id=4feb3f46-4e83-4fba-bebd-3aba95e96fc0
When I create a new Kernel from JupyterLab I can see the session_id
HTTP query param, but when JupyterLab Gateway is building this same request it does not pass it to remote side here hence in Colab Enterprise I see this error.
2024/06/17 04:45:11 Websocket failure: failed to read a websocket message from the server: websocket: close 1006 (abnormal closure): unexpected EOF
2024-06-16 21:45:11.616 PDT
E0617 04:45:11.404645 154 ipython_mitm.go:1398] handleKernels5: websocket request missing required parameter session_id "/api/kernels/73f2bc65-bb6a-444c-a728-b2252ec9ed37/channels"
2024-06-16 21:45:11.632 PDT
JupyterLab to Jupyter Server session_id
is correctly generated but not passed to c.GatewayClient.url
here
GET /api/kernels/342eba8f-aa8c-47a9-8fee-8541cee8ddfe/channels?session_id=4feb3f46-4e83-4fba-bebd-3aba95e96fc0 HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:127.0) Gecko/20100101 Firefox/127.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br, zstd
Sec-WebSocket-Version: 13
Origin: http://localhost:8080
Sec-WebSocket-Protocol: v1.kernel.websocket.jupyter.org
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: 5E7hLeW9SV/Dck4S4jgEAQ==
DNT: 1
Connection: keep-alive, Upgrade
Cookie: username-localhost-8080=2|1:0|10:1719307104|23:username-localhost-8080|188:eyJ1c2VybmFtZSI6ICJmNjdlMDQ1YmZkMmE0MDllOTU4MzMzZThlY2NlNTcwNyIsICJuYW1lIjogIkFub255bW91cyBLb3JlIiwgImRpc3BsYXlfbmFtZSI6ICJBbm9ueW1vdXMgS29yZSIsICJpbml0aWFscyI6ICJBSyIsICJjb2xvciI6IG51bGx9|806e6a65fe8c8092fc40bef8b65ed0edeaec2f440850c61df7d63742f2de96f7; _xsrf=2|8cd09498|e00723a11572d27ce0a1f55b25dc0735|1719307104
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-origin
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket
Reproduce
import datetime
import subprocess
import hashlib
import json
PROXY_URL = "https://2itk4z2isb2va-dot-datalab-vm-staging.googleusercontent.com"
c.NotebookApp.open_browser = False
c.ServerApp.token = ""
c.ServerApp.password = ""
c.ServerApp.port = 8080
c.ServerApp.allow_remote_access = True
c.ServerApp.disable_check_xsrf = True
def get_gcloud_token():
"""Helper method to get an OAuth token from gcloud."""
p = subprocess.run(
["gcloud", "auth", "print-access-token"],
capture_output=True,
check=True,
encoding="UTF-8",
)
bearer_token = p.stdout.strip()
return bearer_token
c.GatewayClient.auth_scheme = "Bearer"
c.GatewayClient.auth_token = get_gcloud_token()
c.GatewayClient.headers = json.dumps({"Origin": PROXY_URL})
c.GatewayClient.validate_cert = False
c.GatewayClient.url = PROXY_URL
c.Application.log_level = 0
Started JupyterLab locally:
jupyter lab --port-retries=0 --ip 0.0.0.0 --allow-root --config=jupyter_notebook_config.py --debug
Open JupyterLab locally and try to connect to remote server (Colab Enterprise), error is seen on JupyterLab and Colab Enterprise logs.
[I 2024-06-16 21:56:34.143 ServerApp] Request start kernel: kernel_id=None, path=''
[D 2024-06-16 21:56:34.145 ServerApp] Request new kernel at: https://tovggcxuxwpac-dot-asia-east1.aiplatform-notebook.googleusercontent.com/api/kernels
[I 2024-06-16 21:56:34.243 ServerApp] GatewayKernelManager started kernel: 48cf0082-b7c8-4f43-b13c-bffa39db467b, args: {'kernel_id': None, 'kernel_name': 'python3', 'env': {'TERM_PROGRAM': 'Apple_Terminal', 'SHELL': '/bin/zsh', 'TERM': 'xterm-256color', 'TMPDIR': '/var/folders/n_/vtwpxlys2cngxlzl4v35747000_wjb/T/', 'TERM_PROGRAM_VERSION': '453', 'TERM_SESSION_ID': 'CB2A71D6-9B3E-4201-8BB8-3A53D8757F89', 'USER': 'gogasca', 'SSH_AUTH_SOCK': '/private/tmp/com.apple.launchd.3jQ3nSOvmk/Listeners', 'PATH': '/Users/gogasca/opt/anaconda3/bin:/Users/gogasca/opt/anaconda3/condabin:/Users/gogasca/Downloads/google-cloud-sdk/bin:/usr/local/git/git-google/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Applications/Wireshark.app/Contents/MacOS:/usr/local/go/bin', 'LaunchInstanceID': 'CB01220E-F880-4642-BEA1-3C64FC7484E3', '__CFBundleIdentifier': 'com.apple.Terminal', 'PWD': '/Users/gogasca/Downloads/vscode', 'XPC_FLAGS': '0x0', 'XPC_SERVICE_NAME': '0', 'SHLVL': '1', 'HOME': '/Users/gogasca', 'LOGNAME': 'gogasca', 'SECURITYSESSIONID': '186a3', 'OLDPWD': '/Users/gogasca/Downloads/vscode', 'SK_SIGNING_PLUGIN': 'gnubbyagent', 'PROMPT': '${COLOR_DIR}${PWD} ${COLOR_GIT}$(parse_git_branch)${COLOR_DEF}% > ', 'CLICOLOR': '1', 'LSCOLORS': 'ExGxBxDxCxEgEdxbxgxcxd', 'CONDA_EXE': '/Users/gogasca/opt/anaconda3/bin/conda', '_CE_M': '', '_CE_CONDA': '', 'CONDA_PYTHON_EXE': '/Users/gogasca/opt/anaconda3/bin/python', 'CONDA_SHLVL': '1', 'CONDA_PREFIX': '/Users/gogasca/opt/anaconda3', 'CONDA_DEFAULT_ENV': 'base', 'CONDA_PROMPT_MODIFIER': '(base) ', 'LANG': 'en_US.UTF-8', '_': '/Users/gogasca/opt/anaconda3/bin/jupyter', '__CF_USER_TEXT_ENCODING': '0x5722B:0x0:0x0', 'KERNEL_LAUNCH_TIMEOUT': '40', 'JPY_SESSION_NAME': '/Users/gogasca/Downloads/vscode/BigQuery basics-88b61f81-d47b-437f-b598-8becabba33f2.ipynb'}, 'cwd': '/Users/gogasca/Downloads/vscode'}
[D 2024-06-16 21:56:34.253 ServerApp] 201 POST /api/sessions?1718600194138 (be9e028811694a53aecd99ba777a6721@127.0.0.1) 111.75ms
[D 2024-06-16 21:56:34.255 ServerApp] Generating new user for token-authenticated request: 3e9e4575787a42c1b817daf8bb14ac42
[D 2024-06-16 21:56:34.256 ServerApp] 200 GET /api/sessions?1718600194253 (3e9e4575787a42c1b817daf8bb14ac42@127.0.0.1) 1.23ms
[D 2024-06-16 21:56:34.263 ServerApp] Generating new user for token-authenticated request: a12888ed50b84bf480ada03042447a13
[D 2024-06-16 21:56:34.263 ServerApp] Request list kernels: https://tovggcxuxwpac-dot-asia-east1.aiplatform-notebook.googleusercontent.com/api/kernels
[D 2024-06-16 21:56:34.272 ServerApp] Generating new user for token-authenticated request: 79c2aed07aef4bc4b98aca2405415a8f
[D 2024-06-16 21:56:34.275 ServerApp] 101 GET /api/kernels/48cf0082-b7c8-4f43-b13c-bffa39db467b/channels?session_id=598b4dab-b7c1-403b-aea9-c96f382b268b (79c2aed07aef4bc4b98aca2405415a8f@127.0.0.1) 3.70ms
[D 2024-06-16 21:56:34.275 ServerApp] Opening websocket /api/kernels/48cf0082-b7c8-4f43-b13c-bffa39db467b/channels
[I 2024-06-16 21:56:34.275 ServerApp] Connecting to kernel 48cf0082-b7c8-4f43-b13c-bffa39db467b.
[I 2024-06-16 21:56:34.275 ServerApp] Connecting to wss://tovggcxuxwpac-dot-asia-east1.aiplatform-notebook.googleusercontent.com/api/kernels/48cf0082-b7c8-4f43-b13c-bffa39db467b/channels
[D 2024-06-16 21:56:34.377 ServerApp] Connection is ready: ws: <tornado.websocket.WebSocketClientConnection object at 0x7fe27f85b8b0>
[D 2024-06-16 21:56:34.384 ServerApp] 200 GET /api/kernels?1718600194261 (a12888ed50b84bf480ada03042447a13@127.0.0.1) 121.80ms
[W 2024-06-16 21:56:34.430 ServerApp] Lost connection to Gateway: 48cf0082-b7c8-4f43-b13c-bffa39db467b
[I 2024-06-16 21:56:34.430 ServerApp] Attempting to re-establish the connection to Gateway in 1.71 secs (1/5): 48cf0082-b7c8-4f43-b13c-bffa39db467b
[I 2024-06-16 21:56:36.140 ServerApp] Connecting to wss://tovggcxuxwpac-dot-asia-east1.aiplatform-notebook.googleusercontent.com/api/kernels/48cf0082-b7c8-4f43-b13c-bffa39db467b/channels
[D 2024-06-16 21:56:36.262 ServerApp] Connection is ready: ws: <tornado.websocket.WebSocketClientConnection object at 0x7fe27f84ba30>
[W 2024-06-16 21:56:36.323 ServerApp] Lost connection to Gateway: 48cf0082-b7c8-4f43-b13c-bffa39db467b
[I 2024-06-16 21:56:36.323 ServerApp] Attempting to re-establish the connection to Gateway in 1.9 secs (1/5): 48cf0082-b7c8-4f43-b13c-bffa39db467b
[I 2024-06-16 21:56:38.224 ServerApp] Connecting to wss://tovggcxuxwpac-dot-asia-east1.aiplatform-notebook.googleusercontent.com/api/kernels/48cf0082-b7c8-4f43-b13c-bffa39db467b/channels
Expected behavior
Pass session_id during channels request.
I'm able to solve this issue by passing session_id and modifying CORS config in Colab Enterprise side. This issue is opened to track Jupyter Server side.
Context
This is used to allow VScode connectivity via JupyterLab Gateway feature. https://medium.com/@gogasca_/how-to-connect-from-vscode-to-vertex-notebooks-directly-8c078d538b0b
- Operating System and version:
- Browser and version:
- Jupyter Server version: 4.2.2
Troubleshoot Output
Paste the output from running `jupyter troubleshoot` from the command line here. You may want to sanitize the paths in the output.
Command Line Output
Paste the output from your command line running `jupyter lab` here, use `--debug` if possible.
Browser Output
Paste the output from your browser Javascript console here, if applicable.