Parquet metadata persistence of DataFrame.attrs

[SanjithChockan]

• closes #ENH: Allow to_parquet to save the metadata from DataFrame.attrs and load it back #54321
• Tests added and passed
• All code checks passed.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

DataFrame.attrs seems to not be attached to pyarrow schema metadata as it is experimental. Added it to the pyarrow table (schema.metadata) so it persists when the parquet file is read back. One issue I am facing is DataFrame.attrs dictionary can't have int as keys as encoding it for pyarrow converts it to string, but not sure if this is a problem.

[xiki-tempula]

On one hand, this is not ideal as the round trip is not complete. On the other hand, in the current implementation, if the column name of a pandas dataframe is not a string, it will be converted to a string, when converting to a parquet file so is still consistent with other behaviours.

Similar to Duplicate column names and non-string columns names are not supported.

Several caveats.

Duplicate column names and non-string columns names are not supported.

, we just need to add a note saying that non-string attrs keys are not supported and will will be converted to a string.

[SanjithChockan]

On one hand, this is not ideal as the round trip is not complete.
I'm not sure I understand. Could you explain this?

[xiki-tempula]

One issue I am facing is DataFrame.attrs dictionary can't have int as keys as encoding it for pyarrow converts it to string, but not sure if this is a problem.
So I would imagine that
if I do

df.attrs = {1:1}
df.to_parquet('test.p')
new_df = pd.read_parquet('test.p')

The new_df.attrs ({'1':1}) would not be the same as the df.attrs ({1:1}). This is what I mean by round trip, where you go to parquet file then go back.

[SanjithChockan]

oh yeah. Can't really think of another approach other than typecasting keys to an int if possible but doesn't seem feasible.

[xiki-tempula]

@SanjithChockan I think it would be fine. The key to the attrs is not the only thing that would be converted to string when saving to parquet file.

[SanjithChockan]

I'll fix the failing checks and wait for someone else to review to see if this approach is okay

[mroeschke]

Suggested change

	- Added ``PANDAS_ATTRS`` to :attr:`Schema.metadata` in :class:`PyArrowImpl` for parquet metadata persistence using pyarrow engine (:issue:`54346`)
	- :meth:`DataFrame.to_parquet` and :func:`read_parquet` will now write and read ``attrs`` respectively (:issue:`54346`)

[SanjithChockan]

done!

[mroeschke]

Thanks @SanjithChockan

[martindurant]

fastparquet here - would have appreciated at least some notification of this.

dask/fastparquet#900

[mroeschke]

Ah sorry @martindurant. The original request mentioned pyarrow so fastparquet slipped the review. Happy to have PRs to add this to the fastparquet engine too!

[aufdenkampe]

@mroeschke, does this PR include the ability to read/write column-level metadata from pandas.Series.attrs? This feature is particularly important to environmental data scientists who need to keep track of column/variable metadata fields such as:

• long name
• description
• units

It's also a data sharing requirement of FAIR Data Principles.

[mroeschke]

I believe not, no. This PR only supports attrs from DataFrame