Fix potential race condition if worker disconnects

[allada]

In the event the worker disconnects from the scheduler, we now wait until all actions are killed before we attempt to reconnect to the scheduler. This prevents a race condition where the scheduler does not know about an action the worker is doing resulting in a potential of the scheduler sending the same action to the worker that it has not cleaned up yet.

fixes #246

---

This change is 

[allada]

+@chrisstaite-menlo

Reviewable status: 0 of 3 files reviewed, all discussions resolved (waiting on @chrisstaite-menlo)

[chrisstaite-menlo]

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on @allada)

---

cas/worker/local_worker.rs line 487 at r1 (raw file):

                        (sleep_fn_pin)(Duration::from_secs_f32(sleep_duration)).await;
                    }
                    if inner.actions_in_transit.load(Ordering::Acquire) != 0 {

Might be a good use for a label to perform a for-else rather than check the value again.

---

cas/worker/running_actions_manager.rs line 1164 at r1 (raw file):

    fn cleanup_action(&self, action_id: &ActionId) -> Result<(), Error> {
        println!("cleanup_action triggered");

Debug value

---

cas/worker/tests/running_actions_manager_test.rs line 140 at r1 (raw file):

#[cfg(test)]
mod running_actions_manager_tests {
    use std::sync::atomic::AtomicBool;

Weird to place this in the mod?

[allada]

Reviewable status: 0 of 3 files reviewed, 3 unresolved discussions (waiting on @chrisstaite-menlo)

---

cas/worker/local_worker.rs line 487 at r1 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

Might be a good use for a label to perform a for-else rather than check the value again.

Done.

---

cas/worker/running_actions_manager.rs line 1164 at r1 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

Debug value

Done.

---

cas/worker/tests/running_actions_manager_test.rs line 140 at r1 (raw file):

Previously, chrisstaite-menlo (Chris Staite) wrote…

Weird to place this in the mod?

Done. My editor did this.

[chrisstaite-menlo]

Reviewed 2 of 3 files at r2.
Reviewable status: 2 of 3 files reviewed, all discussions resolved (waiting on @allada)

---

cas/worker/tests/running_actions_manager_test.rs line 140 at r1 (raw file):

Previously, allada (Nathan (Blaise) Bruer) wrote…

Done. My editor did this.

Mine did the same for somewhere else. They're so helpful... sometimes.

[chrisstaite-menlo]

Reviewed 1 of 3 files at r2, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved (waiting on @allada)