fix: serialize artifact cache upload and commit for the same entry - #6156
Closed
hamodywe wants to merge 1 commit into
Closed
fix: serialize artifact cache upload and commit for the same entry#6156hamodywe wants to merge 1 commit into
hamodywe wants to merge 1 commit into
Conversation
upload and commit both close their database handle before touching storage, so nothing prevented a commit from finalizing a cache -- and, on cleanup, deleting its temporary chunk directory -- while a concurrent upload for the same cache ID was still writing into that directory. Depending on timing this drops or corrupts the uploaded bytes, or causes the in-flight upload request to fail outright. Add a per-cache-ID mutex (cacheLock) and hold it for the full check-then-write sequence in both handlers, so upload and commit for the same cache ID are now mutually exclusive while unrelated cache IDs remain fully concurrent. TestHandler_UploadCommitRace uses a small test-only hook to force the exact interleaving deterministically rather than relying on winning a nanosecond-wide race by chance: it pauses upload right after its completeness check, starts a concurrent commit, confirms commit cannot finish while upload holds the lock, then verifies the committed artifact matches exactly what was uploaded. Reverting just the lock (keeping the hook) makes this test fail every time with "commit finished while upload still held the cache lock", often alongside a connection reset on the upload request itself -- confirming both that the race is real and that the test catches it. fixes nektos#6012
Author
|
Closing this to keep my open-PR list honest - no hard feelings, I know maintainer time is the scarce thing. The branch stays on my fork, so if this is ever useful just say the word and I'll reopen it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
upload and commit both close their database handle before touching storage, so nothing prevented a commit from finalizing a cache -- and, on cleanup, deleting its temporary chunk directory -- while a concurrent upload for the same cache ID was still writing into that directory. Depending on timing this drops or corrupts the uploaded bytes, or causes the in-flight upload request to fail outright.
Add a per-cache-ID mutex (cacheLock) and hold it for the full check-then-write sequence in both handlers, so upload and commit for the same cache ID are now mutually exclusive while unrelated cache IDs remain fully concurrent.
TestHandler_UploadCommitRace uses a small test-only hook to force the exact interleaving deterministically rather than relying on winning a nanosecond-wide race by chance: it pauses upload right after its completeness check, starts a concurrent commit, confirms commit cannot finish while upload holds the lock, then verifies the committed artifact matches exactly what was uploaded. Reverting just the lock (keeping the hook) makes this test fail every time with "commit finished while upload still held the cache lock", often alongside a connection reset on the upload request itself -- confirming both that the race is real and that the test catches it.
fixes #6012