Skip to content

Prevent concurrent lockfile conflicts using flock and add backup#851

Merged
nothingmuch merged 1 commit intopayjoin:masterfrom
0xZaddyy:fix/test-lockfile-safety
Jul 10, 2025
Merged

Prevent concurrent lockfile conflicts using flock and add backup#851
nothingmuch merged 1 commit intopayjoin:masterfrom
0xZaddyy:fix/test-lockfile-safety

Conversation

@0xZaddyy
Copy link
Contributor

@0xZaddyy 0xZaddyy commented Jul 4, 2025

Summary

This PR introduces a safe and stable mechanism to manage Cargo.lock during testing across multiple scripts.

  • File Locking with flock: Ensures only one script can manipulate Cargo.lock at a time using a .cargo.lock.flock file.
  • Backup and Restore: test.sh creates a backup of the existing Cargo.lock before overwriting it and restores it afterward using an EXIT trap.

This prevents race conditions and corruption when running tests concurrently

closes #773

@0xZaddyy 0xZaddyy marked this pull request as ready for review July 4, 2025 12:18
@coveralls
Copy link
Collaborator

coveralls commented Jul 4, 2025

Pull Request Test Coverage Report for Build 16203386079

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 85.558%

Totals Coverage Status
Change from base Build 16201907810: 0.0%
Covered Lines: 7571
Relevant Lines: 8849

💛 - Coveralls

contrib/test.sh Outdated
flock 9

# Backup original lockfile
if [ -f "LOCKFILE" ]; then
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you really intend to check for the literal string "LOCKFILE" ? . if you meant the variable , then you forgot the "$" character

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing it out!

@0xZaddyy 0xZaddyy force-pushed the fix/test-lockfile-safety branch from 3dd8fa1 to 78d42ee Compare July 4, 2025 21:12
contrib/test.sh Outdated
LOCKFILE_LOCK=".cargo.lock.flock"

# Acquire file lock to prevent concurrent modification
exec 9>"$LOCKFILE_LOCK"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure what this is trying to do, but exec is definitely the wrong thing here, it replaces the current shell script with the program 9 which should not normally exist

Suggested change
exec 9>"$LOCKFILE_LOCK"
touch "$LOCKFILE_LOCK"

contrib/test.sh Outdated

# Acquire file lock to prevent concurrent modification
exec 9>"$LOCKFILE_LOCK"
flock 9
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this would need to be something like:

(
    flock 9
    ...
) 9>"$LOCKFILE_LOCK"

() 9>"$LOCKFILE_LOCK" creates a subshell, opening $LOCKFILE_LOCK for writing as file descriptor 9 and then forking the current program

within the subshell, flock can then inherit file descriptor 9 from the subshell, creating the locking behavior

the ... represents the entirety of the test script, i.e. all the commands that need to be protected by the lock

contrib/test.sh Outdated
fi

# Restore the original lockfile on exit
trap 'if [ -f "$LOCKFILE_BAK" ]; then mv "$LOCKFILE_BAK" "$LOCKFILE"; fi; flock -u 9' EXIT
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
trap 'if [ -f "$LOCKFILE_BAK" ]; then mv "$LOCKFILE_BAK" "$LOCKFILE"; fi; flock -u 9' EXIT
trap '[ -f "$LOCKFILE_BAK" ] && mv "$LOCKFILE_BAK" "$LOCKFILE"' EXIT

since the file descriptor will be closed after the subshell exits, there's no need to unlock

make sure the trap command is within the ( ... ) subshell, or there will be a race condition and the lock will be dropped before restoring

contrib/test.sh Outdated

# Backup original lockfile
if [ -f "$LOCKFILE" ]; then
cp "$LOCKFILE" "$LOCKFILE_BAK"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems a bit more intuitive to me but doesn't really matter:

Suggested change
cp "$LOCKFILE" "$LOCKFILE_BAK"
mv "$LOCKFILE" "$LOCKFILE_BAK"

@0xZaddyy 0xZaddyy force-pushed the fix/test-lockfile-safety branch from 78d42ee to ddb20cb Compare July 10, 2025 02:40
@0xZaddyy 0xZaddyy requested a review from nothingmuch July 10, 2025 02:42
Copy link
Collaborator

@nothingmuch nothingmuch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i as a little confused when no Cargo.lock.bak was present because it's cargo.lock.bak

other than that, tACK, tested concurrently, including interrupting

contrib/test.sh Outdated
Comment on lines 5 to 6
LOCKFILE_BAK="cargo.lock.bak"
LOCKFILE_LOCK=".cargo.lock.flock"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
LOCKFILE_BAK="cargo.lock.bak"
LOCKFILE_LOCK=".cargo.lock.flock"
LOCKFILE_BAK="Cargo.lock.bak"
LOCKFILE_LOCK=".Cargo.lock.flock"

@0xZaddyy 0xZaddyy force-pushed the fix/test-lockfile-safety branch from f72f3e1 to 79316f1 Compare July 10, 2025 18:43
- Added a file lock using `flock` to ensure only one test script modifies Cargo.lock at a time
- Created a backup of Cargo.lock before overwriting it
- Added an exit trap to restore the original lockfile and release the lock

Add comment explaining file lock workaround due to MSRV constraints
@0xZaddyy 0xZaddyy force-pushed the fix/test-lockfile-safety branch from 79316f1 to 8c9bc73 Compare July 10, 2025 18:45
@nothingmuch nothingmuch merged commit bb63890 into payjoin:master Jul 10, 2025
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

contrib/test.sh should not overwrite Cargo.lock

4 participants