API: HDFStore closing is buggy

[michaelaye]

Testing and playing with HDFStores a lot, I have found several times that a store.close() leaves things hanging, like this:

store.close()
# no output
store
ClosedFileError: the file object is closed
df.to_hdf('test.h5','df',data_columns=['A','B'],mode='w',table=True)
ValueError: The file 'test.h5' is already opened.  Please close it before reopening in write mode.
# The store was opened with 'test.h5' as filename.

pandas version 0.12.0

[jreback]

the only time I find this happening, is if something is sharing the file (e.g. its opened for reading in several places)..

do you have a reproducible example?

[michaelaye]

Yes, it is like you say, only happening when one opens the HDF file more than once. (Which I guess happens easily jumping between cells in a notebook). My reproducible example is here: http://nbviewer.ipython.org/6117992

2 interesting things:

• Even if the 2nd opening was under a different Python variable name, when closing the first store, it automatically finds itself the still open file-handle from the second store.
• I can't catch the ClosedFileError exception. Must I import its definition from somewhere?

[jreback]

just FYI, I always open in mode='a' (e.g. never specify the mode). If I need to overwrite, I just safe remove the file first. This avoids issues like this. I also never use mode='r'; I think I had similar issues that you are showing.

see also #4367 going to make errors a bit more explicity.

To catch ClosedFileError try:

import tables
from tables import ClosedFileError

I guess it makes sense for open/close to deal with the ClosedFileError so this doesn't happen

[michaelaye]

I usually do like you describe. But that creates another pitfall: As the store.append does not check if the data is exactly the same, I could accidentally blow up the store by appending the same data twice or more. I will publish a set of timing tests with stores as an answer to my own stackoverflow question in a minute. Thanks for your input!

[jreback]

though looking at your example again

• the variable name doesn't matter, however the fact that you called open (implicity by constructing the HDFStore twice does).

and I think that if you close it, the 2nd time you close it throws an error is correct.

[jreback]

@michaelaye

to solve that problem; I pare a delete with an append. if there is nothing there, then the delete does nothing; otherwise it actually deletes the data (which can be time-consuminng FYI!)

you could simply check if your data is there before appending, e.g. just do a simple select (of say the endpoints)?

[jreback]

How's this
(and I added is_open property)

In [1]: path = 'test.h5'

In [2]: df = DataFrame(randn(10,2),columns=list('AB'))

In [3]: df.to_hdf(path,'df',mode='w',table=True)

In [4]: store1 = pd.HDFStore(path)

In [5]: store2 = pd.HDFStore(path)

In [6]: store1
Out[6]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/df            frame_table  (typ->appendable,nrows->10,ncols->2,indexers->[index])

In [7]: store2
Out[7]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/df            frame_table  (typ->appendable,nrows->10,ncols->2,indexers->[index])

In [8]: store1.close()

In [9]: store1
Out[9]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/df            frame_table  (typ->appendable,nrows->10,ncols->2,indexers->[index])

In [10]: store2
Out[10]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/df            frame_table  (typ->appendable,nrows->10,ncols->2,indexers->[index])

In [11]: store1.close()

In [12]: store1
Out[12]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
File is CLOSED

In [13]: store2
Out[13]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
File is CLOSED

In [14]: store1.is_open
Out[14]: 0

In [15]: store2.is_open
Out[15]: 0

[jreback]

@michaelaye changed the api slightly; got rid of _quiet & warn. let me know if you thin it makes sense

[michaelaye]

I'm still puzzled by the fact that store1 stays 'legal' after a store1.close() because it picks up on the file being open via store2. Wouldn't there a more strict relationship be reasonable?

[jreback]

ok....PyTables essentially reference counts the opens (e.g. increments) and closes (decrements) and don't actually close the file until store._handle._open_count gets to 0. I think you are suggesting that:

• a reopen (in the same mode, e.g. 'a' should essentially do nothing)
• a close should completely close the file

?

only issue here is that say you do this:

store = HDFStore('test.h5')
store.append('df',df)

store2 = HDFStore('test.h5')
store2.append('df2',df2)
store2.close()

store.close()

essentially a nested opening

why should store be closed?

[michaelaye]

But with the current way, a double close on store pulls the carpet underneath the user of store2? Isn't that 'unfriendly'?

store = HDFStore('test.h5')
store.append('df', df)

store2 = HDFStore('test.h5')
df2 = store2.select('df') # 2nd user/process should only read, no?

store.close()
store.close()
# At this moment the second process can not access its store anymore, even so it never tried to close it.

[jreback]

I think your example is a bug. latest commit fixes that and gives nice semantics

see here (I have an example in the docs too...but not built yet)

lmk what you think

In [1]: df = DataFrame(randn(10,5))

In [2]: store1 = pd.HDFStore('test.h5',mode='w')

In [3]: store2 = pd.HDFStore('test.h5')

In [4]: store1.append('df',df)

In [5]: store2.append('df2',df)

In [6]: store2.close()

In [7]: store2
Out[7]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
File is CLOSED

In [8]: store1
Out[8]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/df             frame_table  (typ->appendable,nrows->10,ncols->5,indexers->[index])
/df2            frame_table  (typ->appendable,nrows->10,ncols->5,indexers->[index])

In [9]: store1.close()

In [10]: store1
Out[10]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
File is CLOSED

In [11]: store1.close()

In [12]: store1
Out[12]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
File is CLOSED

In [13]: store1.is_open
Out[13]: False

In [14]: store1 = pd.HDFStore('test.h5')

In [15]: store2 = pd.HDFStore('test.h5')

In [16]: store1
Out[16]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/df             frame_table  (typ->appendable,nrows->10,ncols->5,indexers->[index])
/df2            frame_table  (typ->appendable,nrows->10,ncols->5,indexers->[index])

In [17]: store1.close()

In [18]: store1.close()

In [19]: store2
Out[19]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/df             frame_table  (typ->appendable,nrows->10,ncols->5,indexers->[index])
/df2            frame_table  (typ->appendable,nrows->10,ncols->5,indexers->[index])

In [20]: store2.close()

In [21]: store2
Out[21]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
File is CLOSED

[michaelaye]

Very nice and clean! But it's still true that I have to be careful with concurrent writing to a store, because pytables doesn't support that, correct?

[jreback]

yes concurrent writing from thread/process or even multi processing can cause havoc

you basically have to do this yourself

one easy way is to set an attribute in the file when u write (or use a separate lock file)

really just avoid it

I basically use multiple processes to do my work each outputting to separate files

then when done have a single process read and append to a single store (which is pretty fast)