BUG: 1.2.5 -> 1.3.x breaks yaml.dump(DataFrameObject)

[gunthergl]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd

tmp = pd.DataFrame(
        {'col_name': [1,2,3,4]}
)

import yaml
dumped = yaml.dump(tmp)  # Fails starting with pandas==1.3.0
print(dumped)

Problem description

Another package, pytorch-lightning, has the option to save all given hyperparameters. They yaml all given parameters, which resulted in the kind of ugly output shown in the expected output for dataframes. Starting with pandas==1.3.0 this breaks because of the MWE shown above.

If this is expected behaviour from pandas I am sorry, then the issue should go to pytorch-lightning probably.

Expected Output

    !!python/object:pandas.core.frame.DataFrame
    _flags:
      allows_duplicate_labels: true
    _metadata: []
    _mgr: !!python/object/new:pandas.core.internals.managers.BlockManager
      state: !!python/tuple
      - &id004
        - !!python/object/apply:pandas.core.indexes.base._new_Index
          - &id002 !!python/name:pandas.core.indexes.base.Index ''
          - data: !!python/object/apply:numpy.core.multiarray._reconstruct
              args:
              - &id001 !!python/name:numpy.ndarray ''
              - !!python/tuple
                - 0
              - !!binary |
                Yg==
              state: !!python/tuple
              - 1
              - !!python/tuple
                - 1
              - &id003 !!python/object/apply:numpy.dtype
                args:
                - O8
                - false
                - true
                state: !!python/tuple
                - 3
                - '|'
                - null
                - null
                - null
                - -1
                - -1
                - 63
              - false
              - - col_name
            name: null
        - !!python/object/apply:pandas.core.indexes.base._new_Index
          - !!python/name:pandas.core.indexes.range.RangeIndex ''
          - name: null
            start: 0
            step: 1
            stop: 4
      - - &id005 !!python/object/apply:numpy.core.multiarray._reconstruct
          args:
          - *id001
          - !!python/tuple
            - 0
          - !!binary |
            Yg==
          state: !!python/tuple
          - 1
          - !!python/tuple
            - 1
            - 4
          - !!python/object/apply:numpy.dtype
            args:
            - i8
            - false
            - true
            state: !!python/tuple
            - 3
            - <
            - null
            - null
            - null
            - -1
            - -1
            - 0
          - false
          - !!binary |
            AQAAAAAAAAACAAAAAAAAAAMAAAAAAAAABAAAAAAAAAA=
      - - !!python/object/apply:pandas.core.indexes.base._new_Index
          - *id002
          - data: !!python/object/apply:numpy.core.multiarray._reconstruct
              args:
              - *id001
              - !!python/tuple
                - 0
              - !!binary |
                Yg==
              state: !!python/tuple
              - 1
              - !!python/tuple
                - 1
              - *id003
              - false
              - - col_name
            name: null
      - 0.14.1:
          axes: *id004
          blocks:
          - mgr_locs: !!python/object/apply:builtins.slice
            - 0
            - 1
            - 1
            values: *id005
    _typ: dataframe
    attrs: {}

Output of pd.show_versions()

pandas 1.2.5:

INSTALLED VERSIONS

commit : 7c48ff4
python : 3.8.10.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19042
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : de_DE.cp1252

pandas : 1.2.5
numpy : 1.21.1
pytz : 2021.1
dateutil : 2.8.2
pip : 21.1.3
setuptools : 52.0.0.post20210125
Cython : None
pytest : 6.2.4
hypothesis : None
sphinx : 3.5.4
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 2021.07.0
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.51.2

pandas 1.3.1:

INSTALLED VERSIONS

commit : c7f7443
python : 3.8.10.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19042
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : de_DE.cp1252

pandas : 1.3.1
numpy : 1.21.1
pytz : 2021.1
dateutil : 2.8.2
pip : 21.1.3
setuptools : 52.0.0.post20210125
Cython : None
pytest : 6.2.4
hypothesis : None
sphinx : 3.5.4
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 2021.07.0
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.51.2

[MarcoGorelli]

Thanks @gunthergl for the report - I just left git bisect running and #40842 was found to be the first bad commit (cc @jbrockmendel ). Haven't looked into it further though

[jreback]

if yaml is looking at the internal structures then they need to update the package - these are by definition private and can /
do break

[jbrockmendel]

xref #40226

[simonjayhawkins]

close as won't fix? or add 1.3.2 milestone to track discussion?

[jreback]

close as won't fix as no details on what is broken

[simonjayhawkins]

Thanks @gunthergl for the report. have closed as upstream issue.

[simonjayhawkins]

Thanks @gunthergl for the report. have closed as upstream issue.

sorry. downstream issue.

[nitzmahone]

The underlying problem here that's making pyyaml's default representer blow up is that the __reduce__ impl of one of these objects changed recently to return a partial for the factory function; I assume that's related to the Cythonizing of one of those types I saw referenced above. Basically if reduce doesn't return a function whose name we can import and call, pyyaml won't be able to serialize those objects anymore without a custom dumper (and I'd guess this is a problem for other things as well).

yaml/pyyaml#541 (comment)

[jbrockmendel]

fair enough. PR to avoid the partial would be welcome

[biiiipy]

Having the same issue

[simonjayhawkins]

changing milestone to 1.3.5

[jbrockmendel]

@gunthergl working on a fix for this and i could use help writing a test. starting with the dumped from the OP, id like to reconstruct from dumped and then assert that the result matches the original. how do i do that re-loading? yaml.load(dumped) doesnt do it

[gunthergl]

Hi @jbrockmendel, pytorch-lightning uses yaml.UnsafeLoader - I read about the reason sometime somewhere but don't know anymore exactly. However, the following should be appropriate:

import pandas as pd

tmp = pd.DataFrame(
                {'col_name': [1,2,3,4]}
                )

import yaml
dumped = yaml.dump(tmp)  # Fails starting with pandas==1.3.0
print(dumped)

loaded_UnsafeLoader = yaml.load(dumped, Loader=yaml.UnsafeLoader)
print(loaded_UnsafeLoader)
loaded_Loader = yaml.load(dumped, Loader=yaml.Loader)
print(loaded_Loader)

assert tmp.equals(loaded_UnsafeLoader)
assert tmp.equals(loaded_Loader)