Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix diff-file being created when not configured #1716

Merged
merged 4 commits into from
May 3, 2024

Conversation

flyinghyrax
Copy link
Contributor

@flyinghyrax flyinghyrax commented Apr 28, 2024

This fixes path handling in the mirror subcommand that caused a diff file to be written to the current directory if diff-file wasn't set in the configuration file. This is the issue where Path("") becomes Path(".") and boolean-checks as truthy.

I encountered what might be an unexpected behavior when diff-file is set to an existing directory and diff-append-epoch is enabled. I've added a unit test to capture the current behavior, but it might be worth changing depending on how you feel about the compatibility break.

Split into 3 commits:

  1. Removes a diff_file argument and attribute from BandersnatchMirror. The initializer accepted both diff_file and diff_full_path, and diff_file doesn't appear to be used anywhere. I went ahead and removed it while I was around then extracted this change into its own commit so it can be accepted/removed on its own.

  2. Adds 2 tests for the mirror subcommand:

    • check whether a diff file is being created when not configured
    • capture the interaction between the diff-file and diff-append-epoch options when diff-file is set to an existing directory

    If you check out this commit, the first test is expected to fail.

  3. Changing the path handling in the mirror subcommand for diff-file.

Copy link

codecov bot commented Apr 28, 2024

Codecov Report

Attention: Patch coverage is 92.18750% with 5 lines in your changes are missing coverage. Please review.

Project coverage is 83.92%. Comparing base (4d020e8) to head (6cbab83).
Report is 74 commits behind head on main.

Files Patch % Lines
src/bandersnatch/mirror.py 82.60% 2 Missing and 2 partials ⚠️
src/bandersnatch/tests/test_mirror.py 97.56% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1716      +/-   ##
==========================================
+ Coverage   79.69%   83.92%   +4.23%     
==========================================
  Files          31       32       +1     
  Lines        4324     4405      +81     
  Branches      780      791      +11     
==========================================
+ Hits         3446     3697     +251     
+ Misses        721      526     -195     
- Partials      157      182      +25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@flyinghyrax flyinghyrax force-pushed the diff-file-optionality branch from 48553e6 to f406612 Compare April 28, 2024 18:34
Copy link
Contributor

@cooperlees cooperlees left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - Just a couple NITs and questions ... If I misunderstood just explain so and we can look at accept and merge.

src/bandersnatch/mirror.py Show resolved Hide resolved
src/bandersnatch/mirror.py Outdated Show resolved Hide resolved
)
assert expected_diff_file.is_file()
lines = expected_diff_file.read_text().splitlines()
assert len(lines) > 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
assert len(lines) > 0
assert len(lines) == 3

This should always be the same number of lines right? Or did I count wrong / miss why > 0 is better here?

Copy link
Contributor Author

@flyinghyrax flyinghyrax Apr 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You didn’t miss anything, I think this was just laziness 😆 And after already mocking synchronize it’s kind of silly.

I’ll double-check this and update it.

Copy link
Contributor Author

@flyinghyrax flyinghyrax Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay so something interesting happened here after all, on my Windows host anyway, here's my longwinded over explanation. 😬

When I changed the assertion to len(lines) == 3, it failed on 5 != 3. Stepping through the test in the debugger, sure enough lines had 2 extra blank lines for some reason:

[
'C:\\Users\\mrsei\\AppData\\Local\\Temp\\pytest-of-mrsei\\pytest-160\\test_mirror_subcommand_diff_fi0\\mirror\\AB\\C1\\ABC123\\black-123-py3-non-any.whl', 
'', 
'C:\\Users\\mrsei\\AppData\\Local\\Temp\\pytest-of-mrsei\\pytest-160\\test_mirror_subcommand_diff_fi0\\mirror\\98\\7D\\987DEF\\ruff-123.tar.gz', 
'', 
'C:\\Users\\mrsei\\AppData\\Local\\Temp\\pytest-of-mrsei\\pytest-160\\test_mirror_subcommand_diff_fi0\\mirror\\98\\7D\\987DEF\\ruff-123-py3-none-any.whl'
]

And opening the file in a text editor it appears that way as well.

But if break before the text gets written to the file and look at diff_text:

'C:\\Users\\mrsei\\AppData\\Local\\Temp\\pytest-of-mrsei\\pytest-161\\test_mirror_subcommand_diff_fi0\\mirror\\AB\\C1\\ABC123\\black-123-py3-non-any.whl\r\nC:\\Users\\mrsei\\AppData\\Local\\Temp\\pytest-of-mrsei\\pytest-161\\test_mirror_subcommand_diff_fi0\\mirror\\98\\7D\\987DEF\\ruff-123-py3-none-any.whl\r\nC:\\Users\\mrsei\\AppData\\Local\\Temp\\pytest-of-mrsei\\pytest-161\\test_mirror_subcommand_diff_fi0\\mirror\\98\\7D\\987DEF\\ruff-123.tar.gz'

Since that's escaped by the debugger it looks like hot garbage, but you can see the \r\ns in there and running it through print shows exactly three lines:

C:\Users\mrsei\AppData\Local\Temp\pytest-of-mrsei\pytest-161\test_mirror_subcommand_diff_fi0\mirror\AB\C1\ABC123\black-123-py3-non-any.whl
C:\Users\mrsei\AppData\Local\Temp\pytest-of-mrsei\pytest-161\test_mirror_subcommand_diff_fi0\mirror\98\7D\987DEF\ruff-123-py3-none-any.whl
C:\Users\mrsei\AppData\Local\Temp\pytest-of-mrsei\pytest-161\test_mirror_subcommand_diff_fi0\mirror\98\7D\987DEF\ruff-123.tar.gz

The real weirdness (and hint) was when I looked at the diff file in a hex editor:

Screenshot 2024-04-29 201553

The paths are separated by \r\r\n. And after staring a while I remembered universal newlines.


This is easy enough to fix, but do you think this may impact anyway scripting against the diff file output? I'm guessing if I were to bash such a script, I'd be stripping the blank lines out of the file, and that would fail gracefully because there just wouldn't be any blank lines to remove. 🤷

Copy link
Contributor

@cooperlees cooperlees Apr 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good old Windows. If nothing goes wrong here on Linux and if your fix doesn't impact Linux/MacOS (POSIX filesystems) then I am cool for the fix to come in this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can do a before/after check in a docker container this weekend to make certain, since if it did affect Linux I'm not sure there's an existing unit test that would catch the regression in CI.

Removed the 'diff_file' init argument and object attribute.
The attribute doesn't appear to be used, and the class
also takes a 'diff_full_path' argument.
Diff file lines were joined with os.linesep, but also written
in text mode where universal newlines are enabled by default,
so \r\n was being expanded to \r\t\n.
@flyinghyrax flyinghyrax force-pushed the diff-file-optionality branch from 16f1b0c to 6cbab83 Compare May 3, 2024 00:41
Copy link
Contributor

@cooperlees cooperlees left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me - Lets fix other nits in follow up diffs!

@cooperlees cooperlees merged commit 250a59c into pypa:main May 3, 2024
11 checks passed
@flyinghyrax flyinghyrax deleted the diff-file-optionality branch May 26, 2024 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants