Skip to content

FIREFLY-1923: Fix inability to load large FITS tables#1981

Merged
loitly merged 1 commit into
devfrom
FIREFLY-1923-2047-nan-values
Jul 16, 2026
Merged

FIREFLY-1923: Fix inability to load large FITS tables#1981
loitly merged 1 commit into
devfrom
FIREFLY-1923-2047-nan-values

Conversation

@loitly

@loitly loitly commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

https://jira.ipac.caltech.edu/browse/FIREFLY-1923
https://jira.ipac.caltech.edu/browse/FIREFLY-2047

FIREFLY-1923:

  • FITSTableReader: replaced getRow()'s per-cell reflection-based type
    dispatch (getValAsObject(), nRow x nCol times) with getColumn() based
    reads that resolve dispatch once per column instead.
  • deg2pix: remove NaN, clamp jp/jm to [0, nside) on both bounds (fixes left-shift overflow)
  • decimate_key/decimation stats: guard NaN and +/-Infinity, not just NaN
  • test file: It's over 3GB. Trey has one. Let me know if you need me to provide it.

FIREFLY-2047: Improve decimation handling of non-finite values

Test: https://firefly-1923-2047-nan-values.irsakubedev.ipac.caltech.edu/firefly/

- FITSTableReader: read columns via getColumn() instead of row-by-row
  getRow(), resolving type dispatch once per column instead of per cell
- deg2pix: scrub NaN, clamp jp/jm to [0, nside) on both bounds (fixes
  left-shift overflow)
- decimate_key/decimation stats: guard NaN and +/-Infinity, not just NaN

FIREFLY-2047: Improve decimation handling of non-finite values
- throw a clear error when a column has no finite values
- exclude merged invalid-rows bucket from decimated output
@loitly loitly added this to the 2026.2 milestone Jul 15, 2026
@loitly
loitly requested a review from robyww July 15, 2026 19:19
@loitly loitly self-assigned this Jul 15, 2026
@loitly loitly added the bug label Jul 15, 2026

@robyww robyww left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked a FITSTableReader pretty closely. Your changes look good and help improve speed. However, I think you need to put hduTable.getKernel() back in the code. I did several test with the SPLICES file. getKernal seemed to improve performance by about 30 seconds. Surprisingly the memory consumption is about the same. getKernal seems to grab more memory upfront but by the end both ways use the same amount. That is consistent with my experience with the FITS library. It attempts to cache HDU memory.

@loitly

loitly commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

put hduTable.getKernel() back in the code. I did several test with the SPLICES file. getKernal seemed to improve

Both hduTable.getKernel() and hduTable.getColumn() load the entire table into memory. I’m surprised that their performance differs. I’ll test it again.

@loitly
loitly merged commit cd13e48 into dev Jul 16, 2026
@loitly
loitly deleted the FIREFLY-1923-2047-nan-values branch July 16, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants