Skip to content

Conversation

@jrbourbeau
Copy link
Member

Custom serialization for pyarrow.fs.FileInfo was recently added to help with Parquet in Dask DataFrame (xref #9025) but this import isn't quite right

In [1]: import pyarrow

In [2]: pyarrow.fs.FileInfo
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 1
----> 1 pyarrow.fs.FileInfo

AttributeError: module 'pyarrow' has no attribute 'fs'

I think this went unnoticed because in the Dask DataFrame case we trigger this correct import first

https://github.com/dask/dask/blob/c8ef947bd38e2b739ad3eccdbcce42e08d8c3f4a/dask/dataframe/dask_expr/io/parquet.py#L21

first which allows the import here to work.

However in the non-Dask DataFrame parquet case, that correct import doesn't happen and folks hit the AttributeError

@jrbourbeau jrbourbeau requested a review from fjetter as a code owner May 15, 2025 17:34
@github-actions
Copy link
Contributor

github-actions bot commented May 15, 2025

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    27 files  ±0      27 suites  ±0   11h 13m 32s ⏱️ + 1m 14s
 4 115 tests ±0   3 999 ✅  - 5    111 💤 ±0  4 ❌ +4  1 🔥 +1 
51 596 runs  +1  49 307 ✅  - 3  2 284 💤  - 1  4 ❌ +4  1 🔥 +1 

For more details on these failures and errors, see this check.

Results for commit 729e13a. ± Comparison against base commit bf499a2.

♻️ This comment has been updated with latest results.

@jrbourbeau jrbourbeau merged commit ca6efac into dask:main Jun 4, 2025
55 of 61 checks passed
@jrbourbeau jrbourbeau deleted the pa-fileinfo-import branch June 4, 2025 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant