-
-
Notifications
You must be signed in to change notification settings - Fork 18.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
BUG/REF: ArrowExtensionArray non-nanosecond units #53171
Conversation
pandas/core/arrays/arrow/array.py
Outdated
else: | ||
# GH 53171: pyarrow does not yet handle pandas non-nano correctly | ||
# see https://github.com/apache/arrow/issues/33321 | ||
from pandas._libs.tslibs import ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a circular import if this is put at the top?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated, this is now at the top
Is there some amount of this we can hope to rip out once pyarrow is updated? |
I think it can all be removed eventually. It may take some time as it needs to be fixed upstream first and then the pyarrow min version needs to catch up. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks okay. Just a merge conflict
Thanks @lukemanley |
* BUG/REF: ArrowExtensionArray non-nanosecond units * mypy * gh refs * fixes * xfail min versions * docstrings * fix test * fix test * update imports * move imports
* BUG/REF: ArrowExtensionArray non-nanosecond units * mypy * gh refs * fixes * xfail min versions * docstrings * fix test * fix test * update imports * move imports
* BUG/REF: ArrowExtensionArray non-nanosecond units * mypy * gh refs * fixes * xfail min versions * docstrings * fix test * fix test * update imports * move imports
doc/source/whatsnew/v2.1.0.rst
file if fixing a bug or adding a new feature.Fixes a few bugs related to non-nanosecond units for pyarrow duration and timestamp types.
The underlying issue is upstream (apache/arrow#33321) where pyarrow does not handle pandas
Timedelta
andTimestamp
non-nano units correctly.Example (1 seconds -> 0 seconds):