(WIP) BUG: DatetimeTZBlock.fillna raises TypeError

[sinhrks]

import pandas as pd
idx = pd.DatetimeIndex(['2011-01-01', pd.NaT, '2011-01-03'], tz='Asia/Tokyo')
s = pd.Series(idx)
s
#0   2011-01-01 00:00:00+09:00
#1                         NaT
#2   2011-01-03 00:00:00+09:00
# dtype: datetime64[ns, Asia/Tokyo]

# NG, the result must be DatetimeTZBlock
s.fillna(pd.Timestamp('2011-01-02', tz='Asia/Tokyo'))
# TypeError: putmask() argument 1 must be numpy.ndarray, not DatetimeIndex

# NG, it should be object dtype, as the same as when Series creation with mixed tz
s.fillna(pd.Timestamp('2011-01-02'))
# TypeError: putmask() argument 1 must be numpy.ndarray, not DatetimeIndex

Also, found existing DatetimeBlock doesn't handle tz properly. Closes #7095.

idx = pd.DatetimeIndex(['2011-01-01', pd.NaT, '2011-01-03'])
s = pd.Series(idx)

# OK, result must be DatetimeBlock
s.fillna(pd.Timestamp('2011-01-02'))
#0   2011-01-01
#1   2011-01-02
#2   2011-01-03
# dtype: datetime64[ns]

# NG, it should be object dtype, as the same as when Series creation with mixed tz
s.fillna(pd.Timestamp('2011-01-02', tz='Asia/Tokyo'))
#0   2011-01-01 00:00:00
#1   2011-01-01 15:00:00
#2   2011-01-03 00:00:00
# dtype: datetime64[ns]

# NG, unable to fill different dtypes
s.fillna('AAA')
# ValueError: Error parsing datetime string "AAA" at position 0

[jreback]

this is #7095, yes?

[jreback]

hah, already exists in core/index.py

[sinhrks]

Yes, shoud I include the logic to putmask? I think it is better, but I concerned it may break something.

[sinhrks]

Ah my explanation is incorrect. This is remained because of my early attempt, it can be removed.

[sinhrks]

Correct, this should fix #7095. I'll add test case for this.

Also, I found current impl can't handle inplace option properly. inplace should be effective if target contains no NaN values in case of DatetimeTZBlock.

[jreback]

you don't need to test this because you know inplace, e.g. just use an if?

[sinhrks]

I added mask.any() condition to handle the case when the block contains no Nan. In this case:

• We can return the block as it is.
• Don't want to coerce data to object even if the fill value is object.
• DatetimeTZBlock can perform inplace op in this case. Otherwise can't because it's internal value is DatetimeIndex.

Let me do a follow-up if I do something wrong.

[jreback]

hah...i already merged. the code looked reasonable.

have a look and see if anything needs cleaning (or not ok too).

thanks!

[jreback]

thanks @sinhrks nice catch