Support using pod names as worker names

[bnaul]

In our helm chart we pass dask-worker --name $(POD_NAME) in the deployment definition; I tried to set something similar up when migrating to KubeCluster, but it seems that these lines https://github.com/dask/dask-kubernetes/blob/main/dask_kubernetes/core.py#L149-L151 make it impossible to set your worker name at runtime bc you'll get a duplicate version of that argument (obviously the pod names are not known yet when you call KubeCluster()).

I would think using pod names instead of 0, 1, ... would be preferable for almost all use cases, it makes the logs/dashboard a lot more interpretable. Maybe making this the default would be too drastic but I think at least supporting it would be a nice usability improvement.

[jacobtomlinson]

Agree this would be nice!

[bhperry]

So the name gets set to the index in SpecCluster, but only if there isn't another name parameter set in the worker_spec.

There could be a parameter for kube-cluster like worker_name_from_env where you pass in the env var name that you want, and it sets "name": f"$({worker_name_from_env}) under opts in the worker_spec.

Or another way that might be a bit more flexible would be to have an arg like worker_name_func that accepts a function with index as a parameter. And then you can return anything you want from that, including $(POD_NAME)

[jacobtomlinson]

The classic KubeCluster was removed in #890. All users will need to migrate to the Dask Operator. Closing.