Using coroutines to execute multiple steps and subworkflows in a single worker #1218
Description
Right now our worker is executed like this:
- worker executes a step or nested workflow,
- worker sends pending requests to controller, waiting for a response
- controller sends the requests to an idle worker or start a new worker to work on the request
- controller sends the result back to the worker
- worker continues to run
The problem here is that there is potentially a large number of workers waiting for their requests to be satisfied.
This ticket proposes the use of Python coroutines/generator to resolve this problem. The technique is demonstrated in the following example:
def step():
requested = yield 'request 1'
return 'step completed'
def run_step():
try:
runner = step()
pending = next(runner)
while True:
# sends result back to the generator AND fetch the next one
runner.send('resolved')
except StopIteration as e:
# returned value is the value of StopIterator
return e.value
That is to say,
- step executor becomes a generator function
- Instead of sending requests directly to controller, the step executor
yieldthe request to the runner. - the runner receives the request and send to the controller.
- the runner can at the same time request more work for the worker and starts running
- the runner receives result from the controller and stores it (a stack)
- Once the additional work has been completed (stack popped), the returned result will be send back to the generator function so that the original step can continue