Remove write_index_tables option of DataWriter, seems to corrupt files #2853
Conversation
maxnoe
force-pushed
the
remove-index-tables
branch
from
da5ff99 to
e6b629d
Compare
|
|
That is strange, since that is part of pytables, so shouldn't cause any corruption - I wonder if there is an upstream bug? Nominally adding the index tables should increase the speed of doing merges, since it basically pre-computes the event locations. However as we now do the merging with My question is more: should we just turn it off by default until we see if there is an upstream bug, before we just delete this feature? There was no test for this, since I wasn't sure really how to test it. However, we also don'thave a test for the compression settings we use, etc, and those were determined from a set of IO studies I did a long time ago, when the HDF5TableWriter was first developed. Adding the index tables did show some increase in read speed in these early tests, but as I said, maybe with the way we read data now, it's moot. |
|
It's already turned off by default, and it wasn't tested at all. You can see here that I completely removed it and didn't modify any tests. We also didn't activate it e.g. for grid productions. |
|
I also don't see how it would speed up merges using tables, as that doesn't perform any joins or similar on columns, it just appends data |
|
And yes, I suspect an upstream bug and will try to produce a minimal example for a report |
|
Ok, fine to remove it then -I did check and there is no real need since we do the merging in-memory with astropy.table, so these indices are never used. I think pytables doesn't even have a join operation, which is a bit odd, since that is where these would be useful... |
|
I prefer to remove now and eventually add it back later once we understood why the corruption is happening and if these indices are actually useful |
3 participants
Fixes #2839