Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] test_registration_mappings.py::test_pandas_indexing reformat #2853

Merged
merged 2 commits into from
Aug 9, 2024

Conversation

ryan-williams
Copy link
Member

Issue and/or context:
test_registration_mappings.py::test_pandas_indexing is one of the tests closest to the issues described in #2829, but I struggled to parse what the 14 cases were verifying, so I reformatted and documented them in a way that I feel makes them easier to understand.

Changes:
The first commit surfaces that the 14 cases are 2 similar blocks of 7 cases each (one where the default_index_name is obs_id, one where it's alt_id). Each set of 7 can be further subdivided into:

  • 3 cases where default_index_name matches the name of the column that was promoted to df.index (and df.index.name is either preserved, set to "index", or set to None)
  • 3 cases where it's the other column (and similarly df.index.name is either preserved, set to "index", or set to None)
  • 1 where no index is set (df.index is a default RangeIndex, and both columns remain present)

The 2nd commit inverts the grouping above (2x(3+3+1) → 2x3 + 2x3 + 2x1), and adds a comment above each group. Let me know if one presentation seems clearer than the other.

No functionality should have been changed, these are cosmetic changes.

Copy link

codecov bot commented Aug 7, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.19%. Comparing base (3fae283) to head (fa0b02d).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2853      +/-   ##
==========================================
+ Coverage   90.04%   90.19%   +0.15%     
==========================================
  Files          37       37              
  Lines        3957     3957              
==========================================
+ Hits         3563     3569       +6     
+ Misses        394      388       -6     
Flag Coverage Δ
python 90.19% <ø> (+0.15%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
python_api 90.19% <ø> (+0.15%) ⬆️
libtiledbsoma ∅ <ø> (∅)

"expected_signature": {"alt_id": "string", "obs_id": "string"},
},
],
[ "index_col_and_name" , "default_index_name" , "signature_col_names" ],
Copy link
Member

@johnkerl johnkerl Aug 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I love this

I've also really struggled with auto-format of all the parametrize blocks over the last couple years ... I don't know why I never used # fmt: off / #fmt: on ... regardless I'm delighted you're doing so

Moving forward I'll try to use this in some more places, some of the more complex parametrize blocks ...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I won't comment further since this is a draft PR -- just yay :) )

@johnkerl johnkerl self-requested a review August 7, 2024 16:58
@johnkerl johnkerl changed the title test_registration_mappings.py::test_pandas_indexing reformat [python] test_registration_mappings.py::test_pandas_indexing reformat Aug 7, 2024
@johnkerl
Copy link
Member

johnkerl commented Aug 7, 2024

Let me know if one presentation seems clearer than the other.

They're both fine BTW (since you asked) :)

@ryan-williams
Copy link
Member Author

ryan-williams commented Aug 7, 2024

The "(macos-12, 3.11) / lint" failure is the same as we were seeing yesterday, in #2839.

One thing I see, that doesn't make sense, is this job's setup-python restored a cache with key setup-python-macOS-python-3.11.9-pip-0bc0ed4dc8ce9f9b48ebc42fa10052206992eb7a7705c7a75ec8fcbe15e3e65f.

That is the same as was restored here, in a build which failed yesterday, which had different python-ci-single.yml contents (diff).

setup-python should just be sha256'ing python-ci-single.yml's contents, then sha256'ing that once more (per @actions/glob internal-hash-files.ts). I am looking into verifying that, but it seems that actions/setup-python is restoring an old cached Python environment that it should not be.

@johnkerl
Copy link
Member

johnkerl commented Aug 7, 2024

The "(macos-12, 3.11) / lint" failure is the same as we were seeing yesterday, in #2839.

One thing I see, that doesn't make sense, is this job's setup-python restored a cache with key setup-python-macOS-python-3.11.9-pip-0bc0ed4dc8ce9f9b48ebc42fa10052206992eb7a7705c7a75ec8fcbe15e3e65f.

That is was restored here, in a build which failed yesterday, which had different python-ci-single.yml contents (diff).

setup-python should just be sha256'ing python-ci-single.yml's contents, then sha256'ing that once more (per @actions/glob internal-hash-files.ts). I am looking into verifying that, but it seems that actions/setup-python is restoring an old cached Python environment that it should not be.

Great catch @ryan-williams ! Please create a GitHub issue to track this. (I'll do the usual Shortcut replications.)

@johnkerl
Copy link
Member

johnkerl commented Aug 7, 2024

Also, do we have a measurement of how much time the caching is saving? Perhaps we can live without it? I didn't see any quantification on #756.) Please let me know ...

@johnkerl
Copy link
Member

johnkerl commented Aug 7, 2024

... oop, found this: #693 (comment) -- suggesting #756 saves us 40 seconds per run which IMO isn't worth the trouble -- the hours we've spent debugging caching issue this week far exceed even a high multiple of 40 seconds ...

@ryan-williams
Copy link
Member Author

The "40 seconds" there is referring to caching ~/.cache/pre-commit (using @actions/cache, here), but it seems the issue here is caching some pip dependencies (using @actions/setup-python, here), which likely has a more substantial impact.

@johnkerl
Copy link
Member

johnkerl commented Aug 7, 2024

OK cool thanks @ryan-williams .... if it's possible to get some measurement on what the time savings really is, that'll make it easier to have an informed conversation on whether that particular caching is worth our effort ....

Copy link
Member

@johnkerl johnkerl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late approve! :|

@ryan-williams ryan-williams merged commit b9ec3c5 into main Aug 9, 2024
11 checks passed
@ryan-williams ryan-williams deleted the rw/tpi branch August 9, 2024 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants