apps/schedule test is racy

[zarvox]

As discussed in #3293, apps/schedule can fail due to a race condition. In the success case, the following sequence occurs:

1. the test's .click(selector) causes the JS in the test app to make some HTTP or websocket request to the grain backend.
2. the grain's backend makes some request over capnproto RPC to the sandstorm meteor backend to schedule a job to be run in 5 minutes
3. the sandstorm meteor backend's capnproto interface implementation inserts some record into the mongo database
4. the test's .execute('Meteor.call("runDueJobsAt", ' + firstCheck.toString() + ');') triggers a request to the sandstorm meteor backend
5. the meteor backend runs a db query for scheduled jobs due at a particular time, finds a record, and triggers the scheduled job.

But there's actually no synchronization forcing 4 to come after 3, so there's another sequence of events that is also possible:

1. the test's .click(selector) causes the JS in the test app to make some HTTP or websocket request to the grain backend.
2. the grain's backend makes some request over capnproto RPC to the sandstorm meteor backend to schedule a job to be run in 5 minutes
3. the test's .execute('Meteor.call("runDueJobsAt", ' + firstCheck.toString() + ');') triggers a request to the sandstorm meteor backend
4. the meteor backend runs a db query for scheduled jobs due at a particular time, finds no records, and triggers nothing.
5. the sandstorm meteor backend's capnproto interface implementation, responding to the RPC in step 2, inserts some record into the mongo database. In five minutes, it might fire, but our test framework has given up by then.

The solution is to enforce the intended ordering with a barrier of some sort. One way to go about that would be:

• Make the test app grain backend return something in the request to schedule the job, but only once the sandstorm backend has acknowledged the request.
• Make the test app frontend change something in the page when that request completes successfully (adding an element with an ID like #success-oneshot would do)
• Make the test itself wait on that frontend change to appear before calling the runDueJobsAt Meteor method.

(We've worked around this in the meantime with a five-second pause, but that increases the total runtime of the testsuite, and it'd be good to fix this the "right" way)

[ocdtrekkie]

We can close this now, yes?

[zarvox]

Yes, the race condition I identified was fixed by #3349, so I'll close this.

There's a much-less-pressing code health thing where we still call .pause() in a couple places when ideally we'd be checking for a condition until a deadline instead, but that can be a separate ticket if we want to write that down.

[zarvox]

I have unfortunate news -- the latest master, as of writing this, flaked on apps/schedule: https://github.com/sandstorm-io/sandstorm/runs/718849515

I'm gonna reopen this until we have a better idea what's going on here.