Improve performance polling / task sharing mechanism in Ballista #700
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
In #698 some improvements where added to avoid sleeping when there are enough tasks around.
In order to support quicker query responses (e.g. <100ms) and avoid delays in between stages we need to remove calls like sleep(Duration::from_millis(100)) from executor/scheduler and move away from interval-based polling.
This also will reduce CPU/network waste, as there are less calls being done (currently I am seeing around 1-2% of CPU being used when idling, not much, but when going to higher polling frequencies this increases. Setting it to 1ms gives it a very high CPU usage.
Describe the solution you'd like
Wait on tasks to be available in the scheduler and only send a response when having tasks for the worker.
Remove calls to sleep.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.