Autoscaling of different WorkerGroups

[ghalter]

Hi,

I was wondering if there is any plans for allowing to define autoscalers per worker group, I think this could be quite a useful feature.

E.g. in our case we have two groups, "low-mem" which is the default group and "high-mem", the service picks the workers based on guessed workload. Small tasks go to low-mem, big tasks go to high-mem.
We then have an autoscaler, but as the docs states, only the default group is auto scaled.

Our current solution is to have to clusters and pick between the two, but maybe I'm missing something?

Cheers

[jacobtomlinson]

I would love to see this feautre make its way in, but we would need to make a bunch of scheduler changes in dask/distributed to make that happen. If folks are motivated to work on this then I would be happy to review PRs and walk them through the process.

the service picks the workers based on guessed workload

I think this is the biggest challenge. How do we know which group to scale. We have previously explored using resource annotations to do this, but nothing came of it.

[ghalter]

Thanks for your answer @jacobtomlinson
Yes for us, even the possiblity to annotate tasks with different worker groups to pick, and then setting an autoscaler per worker group independently would be a big improvement.