DataFrame with PeriodIndex written in Python2 gets an Int64Index when read back in Python3

[forbdonut]

Code Sample, a copy-pastable example if possible

df = pd.DataFrame([1.,2,3], index=pd.PeriodIndex(['2015-01-01', '2015-01-02', '2015-01-05'], freq='B'))
df.to_hdf('pind2.7', 'p')

Problem description

Writing this DataFrame out via HDF5 in Python2.7, then reading back in Python3.6 causes the DataFrame to have an Int64Index rather than a PeriodIndex:

in python3.6:

In [6]: pd.read_hdf('pind2.7').index
Out[6]: Int64Index([11740, 11741, 11742], dtype='int64')

Expected Output

The deserialized index should look like this:

in python2.7:

In [5]: pd.read_hdf('pind2.7').index
Out[5]: PeriodIndex(['2015-01-01', '2015-01-02', '2015-01-05'], dtype='period[B]', freq='B')

The index is correctly deserialized when reading it back in from Python2.7. The index is also correctly deserialized when reading a file that was written in Python3.6. I don't know if we guarantee 2.7 -> 3.6 compatibility for an on-disk HDF file?

I've attached an HDF file written in Python2.7 and a similar HDF file written in Python3.6.

pind.zip

Output of pd.show_versions()

pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Darwin
OS-release: 16.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.1
pytest: 3.0.7
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
xarray: None
IPython: 5.3.0
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.3.0
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.7
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.7.3
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: 1.1.9
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

[forbdonut]

It's the index_class coming out as the bytes b'period' rather than the string
period' in Python3 that seems to be the issue... this hack seems to fix the problem but I don't know what the right way is:

--- pytables.py 2017-05-08 14:40:21.000000000 -0700
+++ pytables.py 2017-06-27 12:37:50.000000000 -0700
@@ -2563,8 +2563,12 @@
         if 'name' in node._v_attrs:
             name = node._v_attrs.name

-        index_class = self._alias_to_class(getattr(node._v_attrs,
-                                                   'index_class', ''))
+        index_class = getattr(node._v_attrs, 'index_class', '')
+
+        if index_class == b'period':
+            index_class = 'period'
+
+        index_class = self._alias_to_class(index_class)
         factory = self._get_index_factory(index_class)

         kwargs = {}

[jorisvandenbossche]

I can confirm this on master

I don't know if we guarantee 2.7 -> 3.6 compatibility for an on-disk HDF file?

That's indeed the good question. Our docs at least don't mention it is not supported.
@jreback ?

[jreback]

this is for a fixed store
we have very little testing or support for PeriodIndex. That said would accept a PR to fix this.

The way this is done in the code is probably not ideal. The hard part here is make this backward compat.

[jreback]

I think you can use

_ensure_decoded(index_class) and this will just work.

[forbdonut]

Confirmed.. this fixes the issue for me:

--- pytables.py 2017-05-08 14:40:21.000000000 -0700
+++ pytables.py 2017-06-28 06:15:57.000000000 -0700
@@ -2563,8 +2563,8 @@
         if 'name' in node._v_attrs:
             name = node._v_attrs.name

-        index_class = self._alias_to_class(getattr(node._v_attrs,
-                                                   'index_class', ''))
+        index_class = self._alias_to_class(_ensure_decoded(
+            getattr(node._v_attrs, 'index_class', '')))
         factory = self._get_index_factory(index_class)

         kwargs = {}

[jreback]

great if u can put a PR with the above test and fix would be great

[forbdonut]

ok i'll work on it today.