Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add test for duplicate data when resaving model to fits #7358

Merged
merged 2 commits into from
Nov 29, 2022

Conversation

braingram
Copy link
Collaborator

@braingram braingram commented Nov 17, 2022

When resaving a model to fits, asdf loses track of array-hdu links and saves arrays in both hdu and as internal blocks in the asdf tree. This can make an asdf tree that is larger than what can be stored in a fits hdu.

See asdf issue: asdf-format/asdf#1232

This test is skipped as it fails with the current asdf version.

Checklist for maintainers

  • added entry in CHANGES.rst within the relevant release section
  • updated or added relevant tests
  • updated relevant documentation
  • added relevant milestone
  • added relevant label(s)
  • ran regression tests, post a link to the Jenkins job below.
    How to run regression tests on a PR
  • Make sure the JIRA ticket is resolved properly

@codecov
Copy link

codecov bot commented Nov 17, 2022

Codecov Report

Base: 79.63% // Head: 79.76% // Increases project coverage by +0.12% 🎉

Coverage data is based on head (818ea50) compared to base (fa17e4c).
Patch has no changes to coverable lines.

❗ Current head 818ea50 differs from pull request most recent head a551775. Consider uploading reports for the commit a551775 to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7358      +/-   ##
==========================================
+ Coverage   79.63%   79.76%   +0.12%     
==========================================
  Files         412      412              
  Lines       37572    37572              
==========================================
+ Hits        29922    29970      +48     
+ Misses       7650     7602      -48     
Flag Coverage Δ *Carryforward flag
nightly 79.74% <ø> (+0.12%) ⬆️ Carriedforward from 5a0634e
unit 52.23% <ø> (+<0.01%) ⬆️

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files Coverage Δ
jwst/regtest/regtestdata.py 86.23% <0.00%> (+1.37%) ⬆️
jwst/transforms/integration.py 92.85% <0.00%> (+7.14%) ⬆️
jwst/datamodels/integration.py 90.90% <0.00%> (+9.09%) ⬆️
jwst/regtest/conftest.py 88.04% <0.00%> (+23.36%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

jwst/datamodels/tests/test_fits.py Outdated Show resolved Hide resolved
jwst/datamodels/tests/test_fits.py Outdated Show resolved Hide resolved
jwst/datamodels/tests/test_fits.py Outdated Show resolved Hide resolved
@braingram
Copy link
Collaborator Author

Thanks! I made the improvements your described. Let me know if there's anything else and if you think this should either be merged now (with the test skipped) or wait until there's a fix for the bug in asdf. Either way I'll mark it ready for review.

@braingram braingram marked this pull request as ready for review November 18, 2022 14:23
Copy link
Collaborator

@jdavies-st jdavies-st left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Love the proactive test. 👍

@nden nden added this to the Build 9.1 milestone Nov 21, 2022
@hbushouse
Copy link
Collaborator

Shall we go ahead and merge this now, because it's being skipped and hence can't cause any damage, or wait for fix in asdf to be released (which could be a while)?

When resaving a model to fits, asdf loses track of array-hdu
links and saves arrays in both hdu and as internal blocks in the
asdf tree. This can make an asdf tree that is larger than what
can be stored in a fits hdu.

See asdf issue: asdf-format/asdf#1232

This test is skipped as it fails with the current asdf version.

modify imports and use context manager for test_fits

Co-authored-by: James Davies <jdavies@mpia.de>

use context manager in test_fits

Co-authored-by: James Davies <jdavies@mpia.de>

improve test docstring in test_fits
@braingram
Copy link
Collaborator Author

I removed the skip and bumped the asdf version to 2.14.1 to include the fix added in asdf 2.14. For reference here's the relevant PR to asdf with the fix:
asdf-format/asdf#1234

@braingram
Copy link
Collaborator Author

The CI test-xdist-cov errors appear to be unrelated and possibly addressed in this PR: #7370
The CI test-oldestdeps-xdist-cov appears to be a failure to upload results to codecov.

@nden
Copy link
Collaborator

nden commented Nov 29, 2022

Agreed, the failure is unrelated. Merging this so we can rerun the failing OPS case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants