Local Development Cycle With Kubernetes Backend

[droctothorpe]

Would it be possible to provide some further explanation on what the local development cycle for the gateway server looks like?

This page describes the build process.

This page describes the local execution process.

When attempting the latter, I've been unable to establish communication with the Kubernetes cluster described in my kube config (a local deployment of Docker Desktop Kubernetes).

I've implemented a local development cycle that leverages Skaffold in conjunction with Helm. Skaffold watches the source files. When a change is detected, it automatically rebuilds the image (I wrote a Dockerfile to COPY and pip install the source code), stores it locally (no pulling to and from a remote registry), and deploys an updated helm chart. The process takes 12 seconds from saved file to functional deployment. It works, and I'd be happy to contribute it upstream if anyone else finds it useful, but I'm wondering what the existing development cycle when interacting with the Kubernetes backend looks like, and if it's possible to implement a cycle that runs the gateway server directly on the host for improved iteration time.

Thanks!

[jcrist]

The gateway requires bidirectional communication with dask clusters, which usually means it needs to be running in a pod inside the cluster.

When developing locally I've been using the script here: https://github.com/dask/dask-gateway/blob/master/continuous_integration/kubernetes/start.sh, which starts a local cluster using k3d, and starts a pod mounting the local dev directory. This can then be used to run tests and develop without rebuilding an image, provided you're only developing the dask-gateway-server library.

Previously very little of the implementation was kubernetes-specific, so more complicated features could be tested using the local-processes backend, but as we move towards a separate kubernetes implementation it would be nice to improve the dev tooling here. I don't have any experience with dev tooling like skaffold, so any recommendations you might have would be nice.

[mmccarty]

+1 for Skaffold - Some Googling resulted in an interesting article with k3s and skaffold.

@dsludwig - I remember your sweet dev tooling based on skaffold. Is it something you would recommend?

[droctothorpe]

Should we close this issue given that this PR was merged?

[jcrist]

Yeah I think so, thanks for the reminder!