test(sync): take the lock by retrying, not by assuming it is free

[fujibee]

Closes #934.

The suite's most frequent failure. It fails in setup, on mkdir "$lock", which reads as a broken fixture — it is a race.

What races what

sync start holds this team's lock across the launch:

remote.sh:2801	the starter acquires it
remote.sh:2005	the shell writes the pidfile, right after nohup … &
remote.sh:2838	the starter releases it, once the engine is running
remote.sh:2904	the cleanup tries to retake it

The test waited for the pidfile and then took the lock. But the pidfile appears at the second line and the lock is still the starter's until the third — so a single mkdir there raced the starter's release, and lost often enough to be the most common red in the suite.

The pidfile means "the engine was launched", not "nothing holds the lock".

The fix

Retry the mkdir until it succeeds. It takes the lock the instant the starter drops it, so there is no gap to lose.

Waiting for the lock to disappear and then taking it would be the same race one step later. Retrying has no such gap.

The window it then owns is cmd_sync_start's readiness wait, which begins the line after the release: 1600 turns at 0.01s — a floor of 16 seconds, and longer in practice because each turn also spawns a status probe, a tail and an awk (remote.sh:2873-2886). That is what the test's own comment already asks for: a helper holding the lock "for the whole window". The retry ceiling, 400 × 0.05s, is sized to reach into it.

An earlier revision of this description cited 50 × 0.1s at remote.sh:1660. That loop is real, but it belongs to a different function and is not this window — the number was wrong, and review caught it.

Failing to take it now says so, instead of surfacing as EEXIST in setup.

Measured

Same harness both arms, unloaded:

before	4 / 10 failed
after	0 / 10 failed

Load is not the condition — it reproduces on an idle machine, which is worth knowing because "run it under load" was the obvious next step and would have been a detour.

Control, because 0/10 does not distinguish fixed from no longer measuring: with the retry removed and everything else identical, 1 / 6 failed. The harness still produces red for the shape this fixes.

Whole file: 9/9.

Two approaches that looked right and are not

Recorded because each one costs an hour to rediscover.

Take the lock before starting the engine. The pidfile then never appears — the starter needs that same lock to launch, so it blocks. [ -f "$pidfile" ] fails instead, and the test still cannot run.

Pass _REMOTE_ENGINE_CALLER_HOLDS_LOCK=1. remote.sh:49 resets it at startup on purpose: an exported variable of that name would let anything upstream make the engine skip its own locking and run the check-then-act unserialised (#762). That door is closed by design, and a test should not pry it open.

What this does not do

#828 is not addressed, and I could not establish that it is the same shape. It looks like the same family — sleep 4 standing in for "the watcher stopped", where the watcher writes a line to stderr before it exits (watch.sh:640, :661) — but on this machine it is 0 / 20, so the condition is environmental. Fixing it here would be a change I cannot tell had any effect. Better to catch it red in CI and read the log first.