SQL: datetime writing: add tests + issue with pymysql

[jorisvandenbossche]

There were apparantly not yet tests for writing datetime data. So added a test case for both query/table.

For MySQL using the pymysql driver, writing a datetime column is not working ... It gives following error:

In [65]: import pymysql
In [66]: engine_mysql = sqlalchemy.create_engine('mysql+pymysql://root@localhost/pandas_nosetest')
In [67]: df = pd.DataFrame({'A': pd.date_range('2013-01-01 09:00:00', periods=3),
    ...:                    'B': np.arange(3.0)})
In [68]: df.to_sql('test_datetime', engine_mysql, if_exists='replace')
Traceback (most recent call last):
  File "<ipython-input-68-95961c7ce232>", line 1, in <module>
    df.to_sql('test_datetime', engine_mysql, if_exists='replace')
...
  File "C:\Anaconda\envs\devel\lib\site-packages\pymysql\converters.py", line 24, in escape_item
    encoder = encoders[type(val)]

KeyError: <class 'pandas.tslib.Timestamp'>

The data are feeded to the database using df.itertuples() and then np.asscalar(), and apparantly this gives data of the Timestamp class. And for some reason the other drivers can handle this (because it is a subclass of datetime.datetime?), and pymysql not. Which is not good...

[jreback]

does this handle NaT / None conversions?

[rockg]

I don't know if we need a new item, but pandas is good at handling timezone aware dates, but mysql has no knowledge of them. I don't know if we can lay down some suggestions/code for dealing with that data. For example, an IS_DST field along with a MySQL DATETIME field. It's unfortunate that the capability of MySQL is so limiting in this respect.

[jorisvandenbossche]

No, not yet tested NaT/None conversions (and there are certainly things not working with that, see #2754).
@rockg Indeed we should see what we do with timezones, but here, even just a simple datetime isn't even working with mysql (but not due to mysql, but pymysql I think). For timezones, I think we should first ensure it works (does not error, maybe by storing it as utc), and then later maybe see to add options to store it in some other way. If you have concrete ideas of proposals for that, certainly let that know. Although I think eg storing it as a seperate column is up to the user to do this themselves.

[jorisvandenbossche]

Merging this, and I will open a new issue for the issues with MySQL/pymysql