Different behaviour on two different environments. TypeError: data type "datetime" not understood

[hussaintamboli]

I have an aggregation logic

In [19]: datewise_prices
Out[19]: 
[{'arrivalDate': 1490227200000,
  'maxPrice': 3300,
  'minPrice': 3300,
  'modalPrice': 3300},
 {'arrivalDate': 1490400000000,
  'maxPrice': 3300,
  'minPrice': 3300,
  'modalPrice': 3300},
 {'arrivalDate': 1490832000000,
  'maxPrice': 3500,
  'minPrice': 3500,
  'modalPrice': 3500},
 {'arrivalDate': 1490918400000,
  'maxPrice': 3300,
  'minPrice': 3300,
  'modalPrice': 3300},
 {'arrivalDate': 1491091200000,
  'maxPrice': 3300,
  'minPrice': 3300,
  'modalPrice': 3300}]

In [20]: weekly_dataframe = pandas.DataFrame(datewise_prices)
    ...: weekly_dataframe.drop('minPrice', axis=1, inplace=True)
    ...: weekly_dataframe.drop('maxPrice', axis=1, inplace=True)
    ...: weekly_dataframe['arrivalDate'] = pandas.to_datetime(weekly_dataframe['arrivalDate'], unit='ms')
    ...: weekly_dataframe = weekly_dataframe.resample('W', on='arrivalDate')['modalPrice'].mean().dropna().reset_index()
    ...: weekly_dataframe['label'] = (weekly_dataframe['arrivalDate'] - pandas.offsets.DateOffset(days=6)).dt.strftime('%d %b') + ' to ' + weekly_dataframe['arriv
    ...: alDate'].dt.strftime('%d %b')

It results in an error on my Staging environment but works fine on my local.

The staging environment gives an error in (weekly_dataframe['arrivalDate'] - pandas.offsets.DateOffset(days=6)).dt.strftime('%d %b')

Traceback (most recent call last):
  File "/root/myapp/myapp/handlers/aggregation_handler.py", line 290, in get_daily_weekly_and_monthly_aggregates
    weekly_dataframe['label'] = (weekly_dataframe['arrivalDate'] - pandas.offsets.DateOffset(days=6)).dt.strftime('%d %b') + ' to ' + weekly_dataframe['arrivalDate'].dt.strftime('%d %b')
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py", line 696, in wrapper
    converted = _Op.get_op(left, right, name, na_op)
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py", line 330, in get_op
    return _TimeOp(left, right, name, na_op)
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py", line 343, in __init__
    lvalues = self._convert_to_array(left, name=name)
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py", line 454, in _convert_to_array
    if (inferred_type in ('datetime64', 'datetime', 'date', 'time') or
TypeError: data type "datetime" not understood

but it works fine on my local.

Output of pd.show_versions() on local

INSTALLED VERSIONS ------------------ commit: None python: 2.7.6.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-78-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_IN LOCALE: None.None

pandas: 0.19.2
nose: None
pip: 9.0.1
setuptools: 36.0.1
Cython: None
numpy: 1.12.1
scipy: None
statsmodels: None
xarray: None
IPython: 5.4.1
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.7
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 0.9.6
lxml: None
bs4: None
html5lib: 0.999
httplib2: 0.8
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: 2.40.0
pandas_datareader: None

Output of pd.show_versions() on Staging

INSTALLED VERSIONS ------------------ commit: None python: 2.7.6.final.0 python-bits: 64 OS: Linux OS-release: 3.13.0-57-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: None.None

pandas: 0.19.2
nose: None
pip: 1.5.4
setuptools: 3.3
Cython: None
numpy: 1.13.0
scipy: None
statsmodels: None
xarray: None
IPython: 4.0.0
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 0.9.6
lxml: None
bs4: None
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.0.8
pymysql: None
psycopg2: None
jinja2: 2.9.6
boto: 2.40.0
pandas_datareader: None

Please help me out.

[TomAugspurger]

Can you make a copy-pastable example, as requested?

[BranYang]

import pandas as pd

datewise_prices = [
    {'arrivalDate': 1490227200000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1490400000000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1490832000000,
    'maxPrice': 3500,
    'minPrice': 3500,
    'modalPrice': 3500},
    {'arrivalDate': 1490918400000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1491091200000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300}
]


weekly_dataframe = pd.DataFrame(datewise_prices)
weekly_dataframe.drop('minPrice', axis=1, inplace=True)
weekly_dataframe.drop('maxPrice', axis=1, inplace=True)
weekly_dataframe['arrivalDate'] = pd.to_datetime(weekly_dataframe['arrivalDate'], unit='ms')
weekly_dataframe = weekly_dataframe.resample('W', on='arrivalDate')['modalPrice'].mean().dropna().reset_index()
weekly_dataframe['label'] = (weekly_dataframe['arrivalDate'] - pd.offsets.DateOffset(days=6)).dt.strftime('%d %b') + ' to ' + weekly_dataframe['arrivalDate'].dt.strftime('%d %b')

Transferred from @hussaintamboli 's input. It will help if you provide more information, for example, the full trace when it error.

[hussaintamboli]

@BranYang and @TomAugspurger, I have added the stacktrace.

[hussaintamboli]

Just figured it out myself.

It's because of numpy==1.13.0. For numpy==1.12.1, it's working fine.

[jreback]

this looks fine. what exactly is the issue?

[hussaintamboli]

On staging environment with the dependencies mentioned in Output of pd.show_versions() on Staging, I get TypeError: data type "datetime" not understood.

But on local environment with dependencies mentioned in Output of pd.show_versions() on local, I get desired output.

[jreback]

your example does not repro. I would guess that you have strings (and not datetimes) in one of your fields

[hussaintamboli]

Reproduction steps.

This works

$ pip freeze | grep numpy 
numpy==1.12.1

import pandas as pd

datewise_prices = [
    {'arrivalDate': 1490227200000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1490400000000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1490832000000,
    'maxPrice': 3500,
    'minPrice': 3500,
    'modalPrice': 3500},
    {'arrivalDate': 1490918400000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1491091200000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300}
]


weekly_dataframe = pd.DataFrame(datewise_prices)
weekly_dataframe.drop('minPrice', axis=1, inplace=True)
weekly_dataframe.drop('maxPrice', axis=1, inplace=True)
weekly_dataframe['arrivalDate'] = pd.to_datetime(weekly_dataframe['arrivalDate'], unit='ms')
weekly_dataframe = weekly_dataframe.resample('W', on='arrivalDate')['modalPrice'].mean().dropna().reset_index()
weekly_dataframe['label'] = (weekly_dataframe['arrivalDate'] - pd.offsets.DateOffset(days=6)).dt.strftime('%d %b') + ' to ' + weekly_dataframe['arrivalDate'].dt.strftime('%d %b')

weekly_dataframe.drop('arrivalDate', axis=1, inplace=True)

print weekly_dataframe.to_dict(orient='records')
[{'label': u'20 Mar to 26 Mar', 'modalPrice': 3300.0},
 {'label': u'27 Mar to 02 Apr', 'modalPrice': 3366.6666666666665}]

On the staging environment where it gives error

pip install numpy==1.13.0

weekly_dataframe = pd.DataFrame(datewise_prices)
weekly_dataframe.drop('minPrice', axis=1, inplace=True)
weekly_dataframe.drop('maxPrice', axis=1, inplace=True)
weekly_dataframe['arrivalDate'] = pd.to_datetime(weekly_dataframe['arrivalDate'], unit='ms')
weekly_dataframe = weekly_dataframe.resample('W', on='arrivalDate')['modalPrice'].mean().dropna().reset_index()
weekly_dataframe['label'] = (weekly_dataframe['arrivalDate'] - pd.offsets.DateOffset(days=6)).dt.strftime('%d %b') + ' to ' + weekly_dataframe['arrivalDate'].dt.strftime('%d %b')

TypeError: data type "datetime" not understood

[jreback]

this works on master for both numpies so not sure what your issue is

[jorisvandenbossche]

I can reproduce it on 0.19.2 / 1.13.0. Small reproducible example:

pd.Series(pd.date_range("2012-01-01", periods=3)) - pd.offsets.DateOffset(days=6)

and the error actually comes from

np.dtype('M8[ns]') in ('datetime', 'datetime64')

But on master / 0.20.2, this does not raise anymore, so I suppose we have already fixed this (this line of code is also removed).
Nonetheless, always welcome to do a PR to add a test for this specific case (the small reproducible example I showed above), to make sure it keeps on working.

[jreback]

yeah we had some refactorings of this in 0.20.0. ok test PR it is.