Wait for up to a second for existing database lock to clear

[Mark-Simulacrum]

This lets users queue new builds and otherwise schedule work while an ongoing
run is proceeding (with lots of record-progress hits). Arguably we want to
prioritize user-initiated commands to take effect and cancel ongoing
record-progress operations (and other automated work), but that will need more
care.

Should fix cases like rust-lang/rust#94775 (comment),
or at least make them much more rare (individual database operations should very rarely take >1s,
though our logs do suggest it is not entirely infrequent today. (That's something that's worth looking into,
just haven't had time yet).

[2022-03-10T16:08:38Z DEBUG crater::db] sql query "INSERT INTO results (experiment, crate, toolchain, result, log, encoding) VALUES (?1, ?2, ?3, ?4, ?5, ?6);" executed in 5.009915144s
[2022-03-10T16:08:43Z DEBUG crater::db] sql query "INSERT INTO results (experiment, crate, toolchain, result, log, encoding) VALUES (?1, ?2, ?3, ?4, ?5, ?6);" executed in 5.009725792s
[2022-03-10T16:08:46Z DEBUG crater::db] sql query "INSERT INTO results (experiment, crate, toolchain, result, log, encoding) VALUES (?1, ?2, ?3, ?4, ?5, ?6);" executed in 3.235525973s

[Mark-Simulacrum]

@bors r+

[bors]

📌 Commit 3802622 has been approved by Mark-Simulacrum

[bors]

⌛ Testing commit 3802622 with merge 1691543...

[bors]

💔 Test failed - checks-actions

[Mark-Simulacrum]

It looks like the default busy timeout is claimed to be 5 seconds, so this would not actually help. I'm trying to figure out why we're seeing the database is locked error in the first place then.