feature: long running steps

[maximilien]

In some workflows a step might take a "huge" amount of time to complete. This could be minutes, hours, or days. We need a way to deal with this. One proposal is to have async steps and allow for callbacks. For instance:

    steps:
      - name: long-running
        agent: slow-agent
        type: async
        callback: long-running-callback
            webhook: http://localhost:8000/long-running-callback

WDYT @akihikokuroda? We could probably dynamically create the callback for such steps.

We might also need to have a way to specify that a step is waiting on a callback. Like a trigger definition.

Anyhow, LMK what you think of above and let's try it. I can then create a "trigger" issue to address.

CC: @AngeloDanducci @psschwei for long running jobs need.

[akihikokuroda]

After issuing call to the agent, does this async mean continuing to the next step? Can the next step continue working without the response from this step?

[psschwei]

async component rides again! 😄

[maximilien]

In my mind I'd see both cases useful. One where next step executes (no wait) and one where next steps waits.

Either way the asynchronous step's results will be communicated in future in the callback. So if the step waits then basically waiting for call back. WDYT?

[akihikokuroda]

I can not think what workflow can do other than wait while a step is taking long time. Is there any example use case idea for this function?

[psschwei]

What about splitting the workflow in two? First part does up through launching the quantum job, and when the job completes it triggers the second part.

[akihikokuroda]

It's another option. In that case, the first workflow waits until the job completion and sends a slack message.

[akihikokuroda]

Behavior for this:

    steps:
      - name: long-running
        agent: slow-agent
        type: async
        callback: long-running-callback
            webhook: http://localhost:8000/long-running-callback

1. start a new server or add the new entry to the existing server that waits for the call back
2. start the slow-agent
3. if type is async, move to the next step, otherwise wait for the slow-agent response

Is this correct?

There are some other questions.

• s async valid without callback?
• Is there any option for callback other than webhook?

@maximilien

Thanks!

[akihikokuroda]

@maximilien I would like understand how the call back used in the scenario.

We could probably dynamically create the callback for such steps.

We can start a FastAPI server as the call back. What does call the callback? Do we pass the call back address to the agent so that it can call back or can give the address to what makes the call?

We might also need to have a way to specify that a step is waiting on a callback. Like a trigger definition.

We can put an agent or workflow http as the trigger.

        callback: long-running-callback
            webhook: http://localhost:8000/long-running-callback
            trigger: remote_workflow_1

WDYT?

[maximilien]

OK, here’s a more detailed design that @akihikokuroda and I came up with:

  triggers:
   - name: test_callback
      kind: callback
        callback: /test_callback
   - name: test_slack
       kind: slack
          channel: test_channel
          message: done
   - name: test_email
        kind: email
          address: maxim@us.ibm.com
step:
  …
  name: long_running
  async: true
    on_start: test_email
    on_finish: test_callback
step:
  …
step:
  …
step:
  …
  name: test_wait
  trigger: test_callback

@akihikokuroda feel free to edit so it matches the schema we have but the gist of this is:

1. triggers is a new top level concept to specify different kind of triggers: callback, email, and slack
2. async: true on a step means that the step will not wait for execution to complete and setup the trigger that it refers to. Also async can specify on_start and on_finish triggers to notify when start and when finish
3. any step can specify they are waiting on a trigger. This can be dangerous since if the trigger does not happen the step and workflow will block. So we will need to improve workflow validate to at minimum prevent these deadlocks. Also we could add a timeout for waits.