FIREFLY-1923: Fix inability to load large FITS tables#1981
Conversation
- FITSTableReader: read columns via getColumn() instead of row-by-row getRow(), resolving type dispatch once per column instead of per cell - deg2pix: scrub NaN, clamp jp/jm to [0, nside) on both bounds (fixes left-shift overflow) - decimate_key/decimation stats: guard NaN and +/-Infinity, not just NaN FIREFLY-2047: Improve decimation handling of non-finite values - throw a clear error when a column has no finite values - exclude merged invalid-rows bucket from decimated output
robyww
left a comment
There was a problem hiding this comment.
I looked a FITSTableReader pretty closely. Your changes look good and help improve speed. However, I think you need to put hduTable.getKernel() back in the code. I did several test with the SPLICES file. getKernal seemed to improve performance by about 30 seconds. Surprisingly the memory consumption is about the same. getKernal seems to grab more memory upfront but by the end both ways use the same amount. That is consistent with my experience with the FITS library. It attempts to cache HDU memory.
Both hduTable.getKernel() and hduTable.getColumn() load the entire table into memory. I’m surprised that their performance differs. I’ll test it again. |
https://jira.ipac.caltech.edu/browse/FIREFLY-1923
https://jira.ipac.caltech.edu/browse/FIREFLY-2047
FIREFLY-1923:
dispatch (getValAsObject(), nRow x nCol times) with getColumn() based
reads that resolve dispatch once per column instead.
FIREFLY-2047: Improve decimation handling of non-finite values
Test: https://firefly-1923-2047-nan-values.irsakubedev.ipac.caltech.edu/firefly/