Add full Istio support

[jacobtomlinson]

Closes #497.

The PR aims to ensure the operator will play nicely with Istio. As discussed in #482 the way to achieve this is for the workers to have a service each and ensure all network connections within the Dask cluster are going via services rather than directly pod to pod.

To kick things off in this PR I've added a fixture that installs Istio into the kind cluster that gets created, but it is opt-in. To enable it you need to set the TEST_ISTIO=true env var. This means we can enable Istio in the operator CI but not the Helm or classic KubeCluster CI.

I also had to shorten the uuid that was being appended to the worker name because Istio was reusing that name and appending more and was hitting a character limit. We should possibly consider reducing the length of some of our naming schemes.

I've then added a service in front of the workers and tried to get the Dask cluster to only speak via those. Things are not 100% working with this though.

I'm also doing some environment variable injection as discussed in #474. The use case here is so that the worker can correctly set its name.

[rrmistry]

Bringing the comment here from 674d8bb#commitcomment-133495956

We cannot use Dask in production because the Dask operator is injecting pod labels "sidecar.istio.io/inject": "false", which disables Istio side-car injection. With this label, the mesh communication breaks down and the Dask cluster does not behave as expected with other components within the Kubernetes cluster.

@jacobtomlinson can we collaborate offline to discuss options? We can connect on Gitter or via email

[jacobtomlinson]

@rrmistry I'd prefer to keep discussion on GitHub rather than moving to private channels. That way more people can engage and help out. We have #497 open to track enabling Istio support, however it hasn't been a high-priority activity for us lately. Do you think this is something you have the time/interest to help out with?

[rrmistry]

@rrmistry Do you think this is something you have the time/interest to help out with?

Sure. I can invest some time for this as Dask is planned to be a major investment for us going forward.

I will take a look at any CONTRIBUTING docs or guidelines for best practices.

Thanks!