Use numpy representation for hashing

[hagenw]

This change is in response to

The hash key's value seems kind of generic but I wonder whether it only had been introduced recently, or whether it is changed sometimes?

from #419 (review) and has as target branch pyarrow from #419.

Response to the question

The result of the hashing has indeed changed recently in pandas, see pandas-dev/pandas#58999. That is the reason why I require pandas>2.2.0 in #419 as with lower versions of pandas you get a different hash.
I also find this worrisome as this might indicate that they might change it again in the future.

I also checked what changed between pandas 2.1.x and 2.2.x inside pandas/core/util/hashing.py, but it's only related to the documentation:

3a4
> 
108c109,110
<     Series of uint64, same length as the object
---
>     Series of uint64
>         Same length as the object.
244a247
>         The input array to hash.
256a260,264
> 
>     See Also
>     --------
>     util.hash_pandas_object : Return a data hash of the Index/Series/DataFrame.
>     util.hash_tuples : Hash an MultiIndex / listlike-of-tuples efficiently.

Which means the change in the resulting hash with pandas 2.2.x must be related to some restructuring/fixes of the dataframes themselves.

Description of proposed changes

Instead of relying on the solution introduced in #419 with pandas.util.hash_pandas_object() and pickle.dumps(), which might also change, I suggest here instead to hash each column separately, by first converting it to a numpy array with pandas.Series.to_numpy() and then hashing its string representation.

The behavior of pandas.Series.to_numpy() might change over time, and did already for pandas 2.2.x). We can handle with that by adding custom code, as I do here for Int64 dtypes. The same goes with the string representation of a numpy array, which might also change over time.

I could also lower the dependency for pandas to pandas >=2.1.0.

The new hash calculation is slightly slower than before. Storing the big table to CSV now takes 5.0 s instead of 4.2 s in #419.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.0%. Comparing base (2912f76) to head (575d2fe).

Additional details and impacted files

Files	Coverage Δ
audformat/core/table.py	100.0% <100.0%> (ø)

[ChristianGeng]

Review

As we need a way to identify objects, and the introduction of parquet favoring non terministic compression we no longer can use md5sums "on disk" for that purpose.

In order not to depend on pandas specific implementation details in pd.util.hash_pandas_object, this MR introduces an md5 sum based hashing using numpy representations and the python builtin hashlib.md5 representation directly.

I checked the oldest numpy version compatible with pandas 2.2.2 (1.22.4) , and the

line

md5.update(bytes(str(y.to_numpy()), "utf-8"))

does not introduce any changes. So my understanding is that this is safe, and therefore I will approve.