Categoricals with NaNs

[jseabold]

Not sure how to handle this yet. It looks like NaNs do not become a level. Should they? Maybe so. Also describe fails if NaNs are present.

pandas.Categorical([np.nan, np.nan, 1, 1, 1, 2, 3, 4, 5, 5, 4, 3, 3]).describe()

[cpcloud]

their labels become -1...maybe it's being used as a sentinel here. i vote for nan levels, but i don't really use this much

[jreback]

yes
the -1 means it was not found so u could then do something with it/ignore whatever

[jankatins]

right now, you can use np.nan as a level: pd.Categorical(values, levels=[1,2,3,np.nan]). Per default, NaN is not used as a level but as NaN (-1). AFAIK there is no way to "rename" NaN other than first addin np.nan to the levels and then doing a reassign (cat[isnull(cat)] = np.nan).

@jseabold is that enough? Or do you need an explicit "use NaN as a level" keyword?

[jreback]

@JanSchulz I think this is a bug actually. When you construct this categorical, the levels are float64; like np.nan is being intermixed somehow. When it should be specially treated. What should this looks like? (I think the describe part is a sympton not the problem).

[jankatins]

I have a fix in #7768, but this doesn't work yet:

cat = pd.Categorical([1,2,3, np.nan], levels=[1,2,3])
cat.levels = [1,2,3,np.nan]
cat[pd.isnull(cat)] = np.nan
Traceback (most recent call last):
  File "C:\portabel\miniconda\envs\pandas_dev\lib\site-packages\IPython\core\interactiveshell.py", line 2883, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-33-e0614b20be67>", line 1, in <module>
    cat[pd.isnull(cat)] = np.nan
  File "c:\data\external\pandas\pandas\core\categorical.py", line 771, in __setitem__
    # require identical level set
ValueError: cannot setitem on a Categorical with a new level, set the levels first

[jankatins]

The describe call also still fails... I investigate...

[jankatins]

describe is fixed as well: jankatins@1334684

[jankatins]

all three issues should be done in #7768 . If one wants to have np.nan in levels, one would have to specify the levels: Categorical(values, levels=[1,2,3,np.nan]).