Test cases are racy!

[hyperthunk]

I've seen this fail intermittently several times now. Have also seen testSendAfter and testFlushTimer bomb out a couple of times. The typical symptom of this is that the tests never complete - an apparent deadlock.

In the timer tests, this is often due to the ProcessMonitorNotification in testCancelTimer not arriving. For testTimerFlush I'm less certain of what's happening as we're using timeouts for all of the cases here AFAICT.

I'm not really convinced that there isn't a race in Cloud Haskell's monitoring implementation at the moment, but without further evidence, I'm not going to point the finger either. Part of resolving this bug is getting the right profiling tools enabled so we can inspect what is happening when the tests do start hanging, so I'm opening another bug to look into that.

[hyperthunk]

Well there's definitely a race in the code somewhere. This happens on 1 out of 4 test runs for me. Is anyone else seeing the tests hang?

I wonder if there's a subtle race in the timer module that I'm missing....

[hyperthunk]

There's also some raciness in the asyncChan tests for waitAnyTimeout. We end up blocked on an mvar operation (presumably the result mvar) and it's not obvious why.

[hyperthunk]

Argh, no no no no no. For the benefit of future readers....

It turns out there was no races in the tests or the production code at all! It turns out that I was mis-reading the various timeout APIs in distributed-process and passing what I thought were milliseconds, where microseconds was the expected unit of measure. Seemingly random/intermittent timing failures due to scheduling ensued. Fixing the code in the Time module has resolved this and I've not seen any test failures since, so => resolved.