BUG: in read_csv, keep_date_cols doesn't result in correct dtype

[Thomasillo]

>>> import pandas
>>> import io
>>> data = """A
20150908
20150909
"""
>>> t=pandas.read_csv(io.StringIO(data))
>>> t.dtypes
A int64
dtype: object
>>> t=pandas.read_csv(io.StringIO(data),parse_dates={'date':['A']},keep_date_col=True)
>>> t.dtypes
date datetime64[ns]
A object
dtype: object

The second time, the datatype of the column 'A' should also be int64.

[jreback]

@Thomasillo not sure the doc-string actually says that it should be parsed first. as it only applies to a list-lf-list-of dates.

Not very many tests for this either.

So not clear if this is an issue or not.

cc @gfyoung

[gfyoung]

@jreback : I consider this behaviour a bug. Casting should still be done if the column is being kept because it still is a data column after. I think we'll need to remove this function here and just do the date casting first before doing other data conversions, but not sure yet.

@Thomasillo : For the time being, try using Python engine instead. It will correctly convert to int64 for the retained parsed date column.

[gfyoung]

Again, this issue highlights a major issue between the two engines in that the order in which things are applied do not match. As we can see here, the C engine applies the data conversions first before doing date conversions whereas the Python engine does it the other way around. Another reason why the ordering needs to be aligned so that the behaviour can be fixed.

[gfyoung]

@jreback : IMO I think that the difficulty is at least "intermediate," as this issue is another manifestation of the larger issue I described above.

[jreback]

whomever fixes then will get 'intermediate' points (to be spent just like virtual cash:>) 😉