Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: prevent overflow in npy_datetimestruct_to_datetime #55111

Open
3 tasks done
WillAyd opened this issue Sep 12, 2023 · 2 comments
Open
3 tasks done

BUG: prevent overflow in npy_datetimestruct_to_datetime #55111

WillAyd opened this issue Sep 12, 2023 · 2 comments
Labels
Bug Datetime Datetime data dtype Internals Related to non-user accessible pandas implementation

Comments

@WillAyd
Copy link
Member

WillAyd commented Sep 12, 2023

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

We have a lot of multiplication in npy_datetimestruct_to_datetime that can overflow:

npy_datetime npy_datetimestruct_to_datetime(NPY_DATETIMEUNIT base,

UBSAN detects this from running test_sql.py (first error only shown):

/pandas/_libs/src/vendored/numpy/datetime/np_datetime.c:359:33: runtime error: signed integer overflow: 253370764800000000 * 1000 cannot be represented in type 'long int'

What will be tricky about this function is it has an integral return value, where all values are technically valid. Ideally we have some way to avoid the overflow and signal an error via CPython

Issue Description

see above

Expected Behavior

see above

Installed Versions

main

@WillAyd WillAyd added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 12, 2023
@lithomas1 lithomas1 added Datetime Datetime data dtype Internals Related to non-user accessible pandas implementation and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 12, 2023
@tpcgold
Copy link

tpcgold commented Mar 8, 2024

running also into this issue since today. getting data from bigquery to dataframe and then pandas fails to convert it to json

  File "<ipython-input-2-fa9883de5fe4>", line 82, in my_function
    rows = df.to_json(orient="records")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/PycharmProjects/data-etl/venv/lib/python3.11/site-packages/pandas/util/_decorators.py", line 333, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/PycharmProjects/data-etl/venv/lib/python3.11/site-packages/pandas/core/generic.py", line 2699, in to_json
    return json.to_json(
           ^^^^^^^^^^^^^
  File "/PycharmProjects/data-etl/venv/lib/python3.11/site-packages/pandas/io/json/_json.py", line 210, in to_json
    ).write()
      ^^^^^^^
  File "/PycharmProjects/data-etl/venv/lib/python3.11/site-packages/pandas/io/json/_json.py", line 263, in write
    return ujson_dumps(
           ^^^^^^^^^^^^
OverflowError: Overflow occurred in npy_datetimestruct_to_datetime

@moble
Copy link

moble commented Aug 20, 2024

Maybe, like #57349, this is also closed by #55901?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Datetime Datetime data dtype Internals Related to non-user accessible pandas implementation
Projects
None yet
Development

No branches or pull requests

4 participants