Skip to content

fix: serialize artifact cache upload and commit for the same entry - #6156

Closed
hamodywe wants to merge 1 commit into
nektos:masterfrom
hamodywe:fix/artifactcache-upload-commit-race
Closed

fix: serialize artifact cache upload and commit for the same entry#6156
hamodywe wants to merge 1 commit into
nektos:masterfrom
hamodywe:fix/artifactcache-upload-commit-race

Conversation

@hamodywe

@hamodywe hamodywe commented Aug 8, 2026

Copy link
Copy Markdown

upload and commit both close their database handle before touching storage, so nothing prevented a commit from finalizing a cache -- and, on cleanup, deleting its temporary chunk directory -- while a concurrent upload for the same cache ID was still writing into that directory. Depending on timing this drops or corrupts the uploaded bytes, or causes the in-flight upload request to fail outright.

Add a per-cache-ID mutex (cacheLock) and hold it for the full check-then-write sequence in both handlers, so upload and commit for the same cache ID are now mutually exclusive while unrelated cache IDs remain fully concurrent.

TestHandler_UploadCommitRace uses a small test-only hook to force the exact interleaving deterministically rather than relying on winning a nanosecond-wide race by chance: it pauses upload right after its completeness check, starts a concurrent commit, confirms commit cannot finish while upload holds the lock, then verifies the committed artifact matches exactly what was uploaded. Reverting just the lock (keeping the hook) makes this test fail every time with "commit finished while upload still held the cache lock", often alongside a connection reset on the upload request itself -- confirming both that the race is real and that the test catches it.

fixes #6012

upload and commit both close their database handle before touching
storage, so nothing prevented a commit from finalizing a cache -- and,
on cleanup, deleting its temporary chunk directory -- while a
concurrent upload for the same cache ID was still writing into that
directory. Depending on timing this drops or corrupts the uploaded
bytes, or causes the in-flight upload request to fail outright.

Add a per-cache-ID mutex (cacheLock) and hold it for the full
check-then-write sequence in both handlers, so upload and commit for
the same cache ID are now mutually exclusive while unrelated cache IDs
remain fully concurrent.

TestHandler_UploadCommitRace uses a small test-only hook to force the
exact interleaving deterministically rather than relying on winning a
nanosecond-wide race by chance: it pauses upload right after its
completeness check, starts a concurrent commit, confirms commit cannot
finish while upload holds the lock, then verifies the committed
artifact matches exactly what was uploaded. Reverting just the lock
(keeping the hook) makes this test fail every time with "commit
finished while upload still held the cache lock", often alongside a
connection reset on the upload request itself -- confirming both that
the race is real and that the test catches it.

fixes nektos#6012
@hamodywe

Copy link
Copy Markdown
Author

Closing this to keep my open-PR list honest - no hard feelings, I know maintainer time is the scarce thing. The branch stays on my fork, so if this is ever useful just say the word and I'll reopen it.

@hamodywe hamodywe closed this Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Race Condition in Artifact Cache Upload/Commit Operations

1 participant