Monotonic timestamps

[coolreader18]

Description of Changes

Fixes for timestamps, so that a reducer that happens after another is observed to have a later timestamp. Specifically:

• when running a scheduled reducer, clamp it so that ctx.timestamp will be at least the timestamp the reducer was scheduled for.
• when scheduling a reducer for now + delay, clamp now so it is at least the timestamp of the start of the current reducer run.

The first commit is pulled out from the second, to make it clearer what the actual logic change is.

Expected complexity level and risk

2 - timing stuff is intricate and this could mess with assumptions people have made about the way it works, but I'm pretty confident this is strictly a strong guarantee.

Testing

• Manually ran a repeating reducer "loop" which asserted monotonicity of its ctx.timestamp and set back the system clock during the run.
  • Triggered the assertion prior to this PR.
  • Did not trigger the assertion with this PR.

[gefjon]

Code changes look good, and I like your approach. Thanks for splitting into separate commits; that made review super easy!

I would like to see a test, though it may be manual or a smoketest at your discretion. I would write a module with a reducer that re-schedules itself on a very small delay (a zero delay, perhaps?) while decrementing a counter so that it runs a large number of times, and at each run asserts that its new timestamp is >= the previous one, panicking and breaking the loop if not. You should make sure to observe this failing at least once on master, and then run a loop, say, twice that length on this PR branch and see it succeed.

[coolreader18]

Alright, added a test. I couldn't get the issue to reproduce naturally, but calling timedatectl set-time <the_past> in the middle of the test run caused it to fail on master, but still succeed on this branch.

[gefjon]

I also can't get the issue to repro naturally on master. I tried pretty hard, too: I changed the test to run 100000 iterations, set the delay to 0, and still didn't see an out-of-order run. Makes me wonder what was going on with the reporter's setup, that they apparently saw multiple anomalies of several seconds.

I'm not sure there's much value in having the test checked in, in that case; I think a test with a 100% rate of false negatives is either useless or worse than useless. The manual test where you set the system clock is enough to convince me to merge this. Remove the new smoketest, then I'll approve. Thanks!

[coolreader18]

Yeah, I'm wondering if it's a windows-specific issue or something like that. At the very least it's a their-machine-specific issue.

[gefjon]

Yeah, I'm wondering if it's a windows-specific issue or something like that. At the very least it's a their-machine-specific issue.

They used the Linux names for CLOCK_MONOTONIC and CLOCK_REALTIME, so I assumed they were on Linux, but who knows?