Documentation: How to access Dashboard when using dask-kubernetes in a Jupyterhub environment

[guillaumeeb]

Hi!

We are currently setting up Dask enabled Jupyterhub over Kubernetes, and want to use dask-kubernetes as it is better maintained than dask-gateway and fits our use case.

The problem we are facing is how to give the user access to the Dashboard from the Jupyter lab, as Scheduler/Dashboard are running in a separated pod from the Jupyter server one.

After some searching, I essentially found https://kubernetes.dask.org/en/stable/installing.html#dashboard-access, which is linked to Kubeflow, but I believe can be re-used in some way for Jupyterhub environments. Digging in issues here didn't help me better, found some ingress of kube forward discussion, but I'm not sure about those.

Are there some best practices in order to implement this need? We would be happy to help contributing some documentation about the good solution here!

[jacobtomlinson]

Thanks for raising this @guillaumeeb.

You're right that the section on Kubeflow can be mostly reused for this. The main problem is that the jupyter-server-proxy will only forward services running on localhost by default, so when the Dask scheduler Service gets created we need to tell the proxy that it's ok to forward that too. The example you linked shows you how to do it with Kubeflow.

For JupyterHub there is probably a setting that needs tweaking to allow jupyter-server-proxy in each notebook server to forward all of the Dask services. Given that it uses an allowlist it's up to you to decide whether to just set it to * or whether you will limit things down to some specific patterns that you know the scheduler services will use.

It would be great to improve things and get some good documentation on how to do this.

[Montes-L]

Hey, with @guillaumeeb we found the correct way to access the Dask scheduler dashboard using dask-kubernetes and JupyterHub on Kubernetes.
In fact, the section on Kubeflow shows the correct config to add to your singleuser configuration. However, in the JupyterHub Helm chart, it needs to be added at a different path, as explained in this doc, in order to be recognized as a valid config file.

In JupyterHub values-yaml, it looks like this: :

singleuser:
  extraFiles:
    jupyter-server-config:
      mountPath: /etc/jupyter/jupyter_server_config.py
      stringData: |
        c.ServerProxy.host_allowlist = lambda app, host: True

This approach worked for us when enabling Dask dashboard links from notebooks.
Considering that kubeconfig is already loaded for the user of the pod, and that network policies are modified as shown below: :

singleuser:
  serviceAccountName: my-account
  networkPolicy:
    enabled: True
    ingress: []
    egress:
      - ports:
          - port: 443
            protocol: TCP
        to:
          - namespaceSelector:
              matchLabels:
                name: kube-system
    egressAllowRules:
      cloudMetadataServer: false
      dnsPortsCloudMetadataServer: true
      dnsPortsKubeSystemNamespace: true
      dnsPortsPrivateIPs: true
      nonPrivateIPs: true
      privateIPs: true

And, the ~/.config/dask/distributed.yaml has the following configuration :

distributed:
  dashboard:
    link: "/user/{JUPYTERHUB_USER}/proxy/{host}:{port}/status"

I'd be happy to help improve the docs on this if needed!

[guillaumeeb]

We'll also look at how to properly set the Dashboard link directly in Jupyterhub values.yaml file so that it's pre configured for the user. Shouldn't be difficult.

[Montes-L]

We'll also look at how to properly set the Dashboard link directly in Jupyterhub values.yaml file so that it's pre configured for the user. Shouldn't be difficult.

I'm using a different method now :

singleuser:
  extraFiles:
    dask-dashboard-config:
      mountPath: /home/jovyan/.config/dask/distributed.yaml
      stringData: |
        distributed:
          dashboard:
            link: "/user/{JUPYTERHUB_USER}/proxy/{host}:{port}/status"

I've tried using an environment variable, but it was unsuccessful.