timezone aware timestamps in micro-seconds (us) read as '1970-01-01'

[jlequeux]

Describe the issue:

When reading timezone aware timestamps data stored with micro-second unit, parquet corrupt them to unix time.

Minimal Complete Verifiable Example:

import pandas as pd
timestamps = pd.date_range(
        "2024-01-01 12:00:00", 
        "2024-01-01 16:00:00", 
         freq="1h", 
         tz="UTC", 
         unit="us")
df = pd.DataFrame({"ts": timestamps})
df.to_parquet("test.pq", engine="fastparquet")
pd.read_parquet("test.pq", engine="fastparquet").head(2)

output:

	ts
0	1970-01-01 00:00:00+00:00
1	1970-01-01 00:00:00+00:00

Anything else we need to know?:

• changing unit="us" to unit="ns" works fine.
• using timestamps in unit="us" but without timezone works fine as well.

Only the combination of the 2 creates the issue.

• reading the file (test.pq) with another engine (pyarrow) works fine.

The issue seems to comes from the reading part.

Note: I discovered this issue while storing BigQuery data. They are using micro-second (us) timestamps.

Environment:

• Dask version:
• fastparquet: 2024.5.0
• Python version: 3.12
• Operating System: macOS 14.6
• Install method (conda, pip, source): conda

[martindurant]

This is fixed in #933 , which has been waiting for pandas maintainers' approval for merge. Since a month has passed, I will merge and release it now anyway.