Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improve granularity of flow state reporting. (Velocidex#3628)
Currently flows can be in the running, completed or errored states. This is not enough granularity leaving users wondering what is happening to their flows in case the client crashes or reboots for some reason, or may become unresponsive. In such cases sometimes flows remain in the running state so it is not easy to retry them. This PR adds a server side mechanism to actively check up on the progress of the flows. This ensures that we can keep track of running flows better and terminate them when the client reboots. Flows now move through more fine grained states: 1. RUNNING or Scheduled - in this state the flow is scheduled for the client which may not be online. 2. WAITING - The flow is waiting on the client to run - usually clients have a concurrency limit which blocks too many queries from running at the same time. Additional queries will be blocked in the WAITING state until they can begin running. 3. IN_PROGRESS - this state indicates that the query is running on the client. Periodic status updates will be sent to the server to make sure the query is still alive. 4. UNRESPONSIVE - if no periodic updates arrived for the query for a while, the flow goes into the UNRESPONSIVE state. This could happen if the client crashed or disappeared 5. FINISHED - This indicates the flow is completes successfully. 6. ERROR - The flow is in error. If a flow has not been seen for a while, and the client connects again, the server will now send a special request to the client asking it to update the status of those in flight flows. The flows will go from the UNRESPONSIVE state to the IN_PROGRESS state if the flow is still running. But in the more common case where the client restarted and lot progress on the flow, the client will send an ERROR message to the flow causing the in flight flow to be properly closed off with an error. This is a more robust mechanism which should ensure that flows are not left in the running state without being finalized one way or the other.
- Loading branch information