Task state change callback mechanism #57
Description
Is your feature request related to a problem? Please describe.
Currently, tracking of task states is (or will be) implemented by parsing cwl-tes logs in conjunction with polling of the relevant TES endpoint by cwl-tes. The latter imposes a high burden on the TES instance when handling many tasks simultaneously and may further require unnecessary token refreshing.
Describe the solution you'd like
Current WES/TES specs do not provide a callback solution. An outline for such a solution could be:
- The
POST /tasksrequest to TES should contain a fieldcallback_urlto let the TES instance know to which WES instance it should report back a state change - A
POST /task-state(or similar) endpoint should be added to the WES specs, requiring a TEStask_idand the new state of the task as parameters - A directive should be added to the TES specs, instructing TES to report task state changes back to WES (if
callback_urlis non-empty); if callback remains an optional TES feature, a Boolean field such ascallback_supportedcould further be added to the TESservice-infoendpoint to allow WES to verify whether a given TES instance supports callback
In this way, WES should be able to handle incoming task state changes gracefully, without having to resort to polling. In order to prevent TES having to have access to a refresh token (in case a task lasts longer than the auth token remains to be valid), access to the new WES endpoint should be granted either by a valid token or by the token that was attached when the corresponding task was originally posted by WES.
Describe alternatives you've considered
Given that distributed task execution requires that WES and TES not be coupled, even for a given workflow run, and that, moreover, statelessness of WES and TES should be maintained, the number of possible alternatives is limited. As WES cannot know what happens inside TES, it either has to ask or rely on a signal sent by TES to WES.
Additional context
Polling can be retained as a fallback mechanism. Make sure to consider integrated WES-ELIXIR/TESK solution and other solutions without proTES and with other TES implementations, e.g., Funnel.