BUG: sort_values fails because of TypeError: Cannot cast array data from dtype('<M8[ns]') to dtype('float64') according to the rule 'safe' #5648
Labels
bug 🦗
Something isn't working
External
Pull requests and issues from people who do not regularly contribute to modin
Needs more information ❔
Issues that require more information from the reporter
Comments
samyoung-dsci
added
bug 🦗
|
@samyoung-dsci thanks for opening the issue! I wasn't able to reproduce this bug on the latest master of modin on my machine. I tried reproducing with the same dependencies as well. I'm not sure if this is because I'm on a Mac. @vnlitvinov could you try reproducing this bug on your machine? |
pyrito
added
Needs more information ❔
|
I've tried it again with a fresh conda environment and the main branch of Modin, and can confirm that I still get the issue. |
|
@modin-project/modin-contributors @modin-project/modin-core is anyone else able to reproduce this on their machines? |
|
@samyoung-dsci @pyrito unfortunately I can't reproduce the error. My steps were:
I've also run the script 100 times in a loop to see whether I could catch a nondeterministic error. I'm on:
|
|
I've tried both Linux and Windows and have only been able to reproduce the failure on Windows. Steps I used:
Here is my issue-5648 envAnd spoiler |
|
@dchigarev can you take a look at this issue since you're able to reproduce it on Windows? |
anmyachev
added
the
External
|
@dchigarev @samyoung-dsci would it be possible for any of you to post Here's sample output from my laptop: python -m modin.config |
|
@vnlitvinov the error is seem to be only reproducible if the (I've also been able to reproduce the problem at the latest master be98fe6 with pandas 2.0.2) scriptimport modin.pandas as pd
import ray
import modin.config as cfg
cfg.CpuCount.put(8)
ray.init(num_cpus=cfg.CpuCount.get(), runtime_env={'env_vars': {'__MODIN_AUTOIMPORT_PANDAS__': '1'}})
import requests
# Get the example data
r = requests.get('https://raw.githubusercontent.com/ES-Catapult/clock_plot/main/data/eden_2_houseid_324_combined_data.csv')
with open('data.csv', 'w') as f:
f.write(r.text)
data = pd.read_csv('data.csv')
data = data.loc[8029:8033, ['datetime', 'reading_elec']].reset_index(drop=True)
data['reading_elec'] = 0
data['datetime'] = pd.to_datetime(data['datetime'])
# Try to sort the datetimes
data = data.sort_values('datetime')
print(data)python -m modin.configpd.show_versions()UserWarning: Setuptools is replacing distutils. INSTALLED VERSIONScommit : be98fe6 Modin dependenciesmodin : 0.17.0+434.gbe98fe6c pandas dependenciespandas : 2.0.2 conda list |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Modin version checks
I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest released version of Modin.
I have confirmed this bug exists on the main branch of Modin. (In order to do this you can follow this guide.)
Reproducible Example
Issue Description
When trying to use sort_values on a datetime column, it fails with
TypeError: Cannot cast array data from dtype('<M8[ns]') to dtype('float64') according to the rule 'safe'.This presumably has something to do with partitioning (potentially linked to #5552), because in attempting to create a MCVE I found that the issue does not occur when creating an identical minimal dataframe from scratch (using
data = pd.DataFrame({'datetime': ["2018-11-07 23:00:00", "2018-11-08 00:00:00", "2018-11-08 01:00:00", "2018-11-08 02:00:00", "2018-11-08 03:00:00"], 'reading_elec': [0]*5})) rather than as a subset of the full dataframe.This doesn't affect all timeseries - for example, I get the issue with the 2531:2535 of the above dataset, but not with 0:2534. However, I have observed it with multiple files, not just the one above.
Changing from Ray to Dask doesn't solve the issue.
Expected Behavior
The dateframe should end up sorted by datetime (without the exception being raised). Interestingly, this is what happens if you create the smaller dataframe from scratch, as shown below:
Installed Versions
INSTALLED VERSIONS
commit : 9068fbc
python : 3.10.8.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19044
machine : AMD64
processor : Intel64 Family 6 Model 140 Stepping 1, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United Kingdom.1252
Modin dependencies
modin : 0.18.1
ray : 2.2.0
dask : 2023.2.0
distributed : 2023.2.0
hdk : None
pandas dependencies
pandas : 1.5.3
numpy : 1.24.1
pytz : 2022.7
dateutil : 2.8.2
setuptools : 65.6.3
pip : 22.3.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.8.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli :
fastparquet : None
fsspec : 2023.1.0
gcsfs : None
matplotlib : 3.6.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.0
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None
tzdata : None
The text was updated successfully, but these errors were encountered: