Wait for in-progress ephemeral server download by progress, not a fixed deadline#1332
Merged
Conversation
… deadline When a concurrent process is already downloading the dev/test server binary, lazy_download_exe waited a fixed 20s for the temp file to disappear and then errored. A large binary on a slow connection (or several test workers starting servers at once) legitimately takes longer, so waiters would spuriously fail with 'Temp download file not complete after 20 seconds'. Wait based on progress instead: keep waiting as long as the temp file's mtime keeps advancing, and only reclaim it (once) if it stops making progress for 60 seconds, indicating the downloading process died.
Sushisource
approved these changes
Jun 15, 2026
brianstrauch
enabled auto-merge (squash)
June 15, 2026 22:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When a concurrent process is already downloading the dev/test server binary,
lazy_download_exewaited a fixed 20s for the.downloadingtemp file to disappear and then errored:A large binary on a slow connection — or several test workers starting servers concurrently (e.g.
pytest -n auto) — legitimately takes longer than 20s, so waiters spuriously fail even though the download is alive and progressing. This was observed flaking the Python SDK's CI on3.14, macos-armwhile downloading a ~578 MB custom build.Fix
Wait based on progress rather than a fixed deadline. The
.downloadingtemp file is written incrementally bystd::io::copy, so its mtime advances while the download is alive. We keep waiting as long as the mtime keeps advancing, and only reclaim the download (once) if it makes no progress for 60s — which indicates the downloading process died. This distinguishes "slow but alive" from "abandoned" instead of capping total wait time.The 60s no-progress threshold replaces the previous one-shot 90s abandoned-file check; reclaim remains a single attempt (
already_tried_cleaning_old) before failing loudly, so two live processes can't fight over the temp file indefinitely.Notes
The reclaim still operates on the shared
<dest>.downloadingpath, so 60s is deliberately well above realistic network-stall jitter: reclaiming a still-live download would unlink the temp file out from under the writer and risk publishing a partial binary. A future hardening could use a per-process unique temp name to make reclaim non-destructive and allow a shorter threshold.