Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

check ExtensionType in is_datetime64_any_dtype for array-likes #57060

Merged
merged 10 commits into from
Feb 2, 2024

Conversation

jmoralez
Copy link
Contributor

@jmoralez jmoralez commented Jan 24, 2024

@jmoralez jmoralez changed the title check for ExtensionType in is_datetime64_any_dtype check for ExtensionType in is_datetime64_any_dtype for array-likes Jan 24, 2024
@jmoralez
Copy link
Contributor Author

Failures in CI look unrelated to this PR (logs)

FAILED pandas/tests/arrays/categorical/test_sorting.py::TestCategoricalSort::test_numpy_argsort - Failed: DID NOT RAISE <class 'ValueError'>
FAILED pandas/tests/extension/test_categorical.py::TestCategorical::test_argsort - AssertionError: Series are different

@jmoralez jmoralez marked this pull request as ready for review January 25, 2024 02:11
Copy link
Member

@mroeschke mroeschke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a test in test_arrow.py?

@mroeschke mroeschke added the ExtensionArray Extending pandas with custom dtypes or arrays. label Jan 25, 2024
@jmoralez
Copy link
Contributor Author

jmoralez commented Jan 25, 2024

Thanks @mroeschke. The test is failing (log) for arrow date types

FAILED pandas/tests/extension/test_arrow.py::test_is_datetime64_any_dtype[date32[day]] - AssertionError
FAILED pandas/tests/extension/test_arrow.py::test_is_datetime64_any_dtype[date64[ms]] - AssertionError

since for those types pa.types.is_timestamp returns False. However the current implementation of is_datetime64_any_dtype returns True when provided as a dtype, e.g. pd.api.types.is_datetime64_any_dtype(serie_with_pyarrow_date32.dtype). What should I do here? Is it expected for that to return True or should I adjust the above condition to also check if pa.types.is_date returns False?

Edit
I checked that both of those types translate to datetime64 in numpy:

import pyarrow as pa

type(pa.date32().to_pandas_dtype())
# numpy.dtypes.DateTime64DType 
type(pa.date64().to_pandas_dtype())
# numpy.dtypes.DateTime64DType

so I've included those in the test for True

@jmoralez jmoralez changed the title check for ExtensionType in is_datetime64_any_dtype for array-likes check ExtensionType in is_datetime64_any_dtype for array-likes Jan 25, 2024
@mroeschke
Copy link
Member

I think the date types should return False. In pandas they don't always behave like date times

@jmoralez
Copy link
Contributor Author

Agree, for example trying to extract the second raises an error. I'll make the changes.

@mroeschke mroeschke added this to the 3.0 milestone Feb 2, 2024
@mroeschke mroeschke merged commit 9c76d54 into pandas-dev:main Feb 2, 2024
50 of 51 checks passed
@mroeschke
Copy link
Member

Thanks @jmoralez

@jmoralez jmoralez deleted the check-extension-isdt64any branch February 2, 2024 18:01
pmhatre1 pushed a commit to pmhatre1/pandas-pmhatre1 that referenced this pull request May 7, 2024
…s-dev#57060)

* check for ExtensionType in is_datetime64_any_dtype

* use pre-commit

* add test and move doc entry

* check not date in test

* fix condition

* check type is not datetime.date

* fix comparison

* move description to ExtensionArray

* return True for date types
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ExtensionArray Extending pandas with custom dtypes or arrays.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: is_datetime64_any_dtype returns False for Series with pyarrow dtype
2 participants