Skip to content

BUG: Merges erroneously match integer join keys between int64 and uint64 when using values >= 2**53 #56367

Open
@wence-

Description

@wence-

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

left = pd.DataFrame({"key": pd.Series([2**53], dtype="int64"), "value": [1]})
right = pd.DataFrame({"key": pd.Series([2**53 + 1], dtype="uint64"), "value": [2]})

left.key.equals(right.key) # => False

left.merge(right, on="key", how="inner")
#                key  value_x  value_y
# 0  9007199254740992        1        2

Issue Description

The merge between the left and right frames produces a non-empty result, claiming that $2^{53} = 2^{53} + 1$.

My guess is that somewhere in the guts of the merge, to perform the matching between the left int64 and right uint64 columns, the keys are converted to float64 which is lossy above $2^{53}$. float(2**53) + 1 == float(2**53) is True.

Expected Behavior

I would expect an empty result, since the left and right key columns do not have matching keys.

Installed Versions

INSTALLED VERSIONS
------------------
commit                : b864b8eb54605124827e6af84aa7fbc816cfc1e0
python                : 3.10.12.final.0
python-bits           : 64
OS                    : Linux
OS-release            : 6.2.0-37-generic
Version               : #38~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov  2 18:01:13 UTC 2
machine               : x86_64
processor             : x86_64
byteorder             : little
LC_ALL                : None
LANG                  : en_GB.UTF-8
LOCALE                : en_GB.UTF-8

pandas                : 2.1.0.dev0+1414.gb864b8eb54
numpy                 : 1.24.4
pytz                  : 2023.3
dateutil              : 2.8.2
setuptools            : 67.7.2
pip                   : 23.2.1
Cython                : 3.0.0
pytest                : 7.4.0
hypothesis            : 6.82.2
sphinx                : 6.2.1
blosc                 : None
feather               : None
xlsxwriter            : 3.1.2
lxml.etree            : 4.9.3
html5lib              : 1.1
pymysql               : 1.4.6
psycopg2              : 2.9.6
jinja2                : 3.1.2
IPython               : 8.14.0
pandas_datareader     : None
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.12.2
bottleneck            : 1.3.7
dataframe-api-compat  : None
fastparquet           : 2023.7.0
fsspec                : 2023.6.0
gcsfs                 : 2023.6.0
matplotlib            : 3.7.2
numba                 : 0.57.1
numexpr               : 2.8.4
odfpy                 : None
openpyxl              : 3.1.2
pandas_gbq            : None
pyarrow               : 12.0.1
pyreadstat            : 1.2.2
python-calamine       : None
pyxlsb                : 1.0.10
s3fs                  : 2023.6.0
scipy                 : 1.11.1
sqlalchemy            : 2.0.19
tables                : 3.8.0
tabulate              : 0.9.0
xarray                : 2023.7.0
xlrd                  : 2.0.1
zstandard             : 0.19.0
tzdata                : 2023.3
qtpy                  : 2.4.0
pyqt5                 : None

Metadata

Metadata

Assignees

Labels

BugDtype ConversionsUnexpected or buggy dtype conversionsReshapingConcat, Merge/Join, Stack/Unstack, Explode

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions