Cannot create a Notebook when starting KFP from cloud shell

[ajayalfred]

I use Cloud Shell to start KFP. I am able to access KFP UI and complete workflows.
I can spin up Jupyter Notebook but I'm unable to create a notebook. And I am unable to open an uploaded Notebook – getting the error shown below

[qimingj]

Adding some context --- this is done through Cloud shell. Maybe Cloud shell does not handle web socket well.

[yebrahim]

This issue should be on kubeflow/kubeflow.
I wonder if websockets should be shimmed for CloudShell somehow like Datalab does.

[fdasilva59]

I experienced the same issue. While investigating on my side, I found this usefull post about "Jupyter does not work when accessed from Google Cloud Shell preview" GoogleCloudDataproc/initialization-actions#230 (comment)

As a bypass solution, I was able to start a Jupyter Notebook after a little bit of hacking. Here is what I've done:

Regular GKE Cluster setup and ML Pipeline installation (using CloudShell)

# See https://cloud.google.com/compute/docs/regions-zones/
gcloud config set compute/zone europe-west1-b

export CLUSTER_NAME=ml-pipelines

gcloud container clusters create $CLUSTER_NAME \
  --scopes cloud-platform \
  --enable-cloud-logging \
  --enable-cloud-monitoring \
  --machine-type n1-standard-2 \
  --num-nodes 4

# Grant your user account permission to create new cluster roles. 
kubectl create clusterrolebinding ml-pipeline-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value account)

# Deploy the ML Pipeline Services and Kubeflow to your cluster.
export PIPELINE_VERSION=0.1.2
kubectl create -f https://storage.googleapis.com/ml-pipeline/release/$PIPELINE_VERSION/bootstrapper.yaml

• Wait for the cluster to be ready. You can check the progress with kubectl get jobs and then kubectl get pods -n kubeflow

Access to Jupyter and try to create a Notebook (that will fail)

• At this stage if you look at the running pods in the kubeflow namespace kubectl get pods -n kubeflow, there is no running Jupyter notebooks yet
• In CloudShell, forward a local port to visit the Kubeflow UI dashboard:

export NAMESPACE=kubeflow
kubectl port-forward -n ${NAMESPACE} $(kubectl get pods -n ${NAMESPACE} --selector=service=ambassador -o jsonpath='{.items[0].metadata.name}') 8080:80

• You can then open Kubeflow home from the CloudShell WebPreview, and click on JupyterHub link in the menu
• You may have to wait until Jupyter Hub is started.
• You should land to the "Jupyter Sign In" page. Proceed with the default user : user admin and password admin
• Then the "Spawner options" form is displayed: Choose the image "gcr.io/kubeflow-images-public/tensorflow-1.10.1-notebook-cpu:v0.3.1" anc click the "Spawn" button. "Your server is starting up." message is showed up. This may takes a while (download the docker image...)
• You can check the progression after opening a second CloudShell session:

# Check for the Jupyter pod name `jupyter-<USER>` (Here user is 'admin')
kubectl get pods -n kubeflow

• Once ready, in the WebPreview, it looks like we are not automatically redirected to the Jupyter Notebooks. Click on the "Home" Link next to the Jupyter Logo. Then click on "My Server" Button

• At this stage, if you try to create a new Notebook, a popup will display a "Not Found Error"

• You can check the container logs for more details with kubectl logs -n kubeflow jupyter-admin. You should see the error "Blocking Cross Origin API request for /user/admin/api/contents. Origin: [...]"

Fix Jupyter configuration

• Attach to the corresponding docker container in the K8S cluster

kubectl exec -it -n kubeflow jupyter-admin  bash

• Edit Jupyter configuration file .jupyter/jupyter_notebook_config.pyin order to add c.NotebookApp.allow_origin='*'

• Unfortunately, Jupyter does not automatically reload its configuration. We have to restart the server, which mean killing the entry point process, which also means killing the docker container. But that's OK as K8S will restart the container. (Hopefully, it also looks like the Jupyter configuration is persisted somewhere but I have not been able yet to locate the Dockerfile for the "kubeflow-images-public/tensorflow-1.10.1-notebook-cpu:v0.3.1" )

jovyan@jupyter-admin:~$ vim .jupyter/jupyter_notebook_config.py`

jovyan@jupyter-admin:~$ ps -auxw
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
jovyan       1  0.0  0.0   4528   820 ?        Ss   12:44   0:00 tini -- start-singleuser.sh --ip="0.0.0.0" --port=8888 --allow-root
jovyan       5  0.1  0.8 292128 62168 ?        Sl   12:44   0:01 /opt/conda/bin/python /opt/conda/bin/jupyterhub-singleuser --ip="0.0.0.0" --port=8888 --allow-root
jovyan      33  0.0  0.0  20316  4108 ?        Ss   12:52   0:00 bash
jovyan      41  0.0  0.0  36080  3300 ?        R+   12:53   0:00 ps -auxw

jovyan@jupyter-admin:~$ kill 1
jovyan@jupyter-admin:~$ command terminated with exit code 137

<USER>@CloudShell:~$ kubectl get pods  -n kubeflow |grep jupyter
jupyter-admin                                            1/1       Running   2          16m

• If you reload the Jupyter Notebook dashboard in the WebPreview, you can now create a Notebook without the error.

• By the way, if you wan to display the ML pipeline UI, just add "pipeline" at the root URL of the WebView URL (Ex: https://8080-dot-[ID]-dot-devshell.appspot.com/pipeline/)

PS : With this conf, I was able to successfully run the Notebook example https://github.com/kubeflow/pipelines/blob/master/samples/notebooks/Lightweight%20Python%20components%20-%20basics.ipynb

[yebrahim]

Thanks @fdasilva59 for finding this out.
You don't need to open your Jupyter CORS config that wide to get it to work though, since this is only needed for Cloud Shell. Instead of allowing all domains, you can just add the Cloud Shell origin pattern:
c.NotebookApp.allow_origin_pat = ".*-dot-devshell.appspot.com$"
(Take out the c.NotebookApp.allow_origin='*' part, since it'll override the pattern argument).

[yebrahim]

@jlewi is this something you'll willing to change in the notebook config? @fdasilva59 or I can send a PR your way if this is the case.

[fdasilva59]

@yebrahim Thanks for correcting my configuration change. That's indeed a better choice 👍 Let's wait for the feedback on creating a PR or not !

[yebrahim]

@fdasilva59 can you send a PR to fix this?

[fdasilva59]

@yebrahim Yes I can look to create a PR in Kubeflow/Kubeflow - That will be my first PR for an Open Source project ! :) I'll check for the Contributor License Agreement today.

[fdasilva59]

CLA submited today. PR created kubeflow/kubeflow#1956