Series values cannot be accessed by numeric categorical index

[alex-filatov]

Code Sample

>>> s1 = pd.Series(['a', 'b', 'c'], index=pd.Series([1, 2, 3]))
>>> s1.get(3)
'c'

>>> s2 = pd.Series(['a', 'b', 'c'], index=pd.Series([1, 2, 3], dtype='category'))
>>> s2.get(3) is None
True
>>> s2.get(0)
'a'

Problem description

In my opinion behavior in the second case can be error-prone (when there is an overlap between positional index and categorical one) and inconvenient (forces to use positional index).

Expected Output

>>> s2.get(3)
'c'

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Darwin
OS-release: 16.1.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.19.1
nose: None
pip: 9.0.1
setuptools: 27.2.0
Cython: None
numpy: 1.11.2
scipy: 0.18.1
statsmodels: None
xarray: None
IPython: 5.1.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2016.7
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

[jreback]

this is raising on a validation check and hitting positional indexing, rather than label indexing:

In [12]: s.index.holds_integer()
Out[12]: True

In [13]: s.index.is_categorical()
Out[13]: False

In [14]: s.index.is_integer()
Out[14]: True

In [15]: s.index.holds_integer()
Out[15]: True

In [16]: s2.index.is_categorical()
Out[16]: True

In [17]: s2.index.holds_integer()
Out[17]: False

https://github.com/pandas-dev/pandas/blob/master/pandas/indexes/base.py#L1128

    def holds_integer(self):
        return self.inferred_type in ['integer', 'mixed-integer']

something like might work (though might cause other things to break)
In pandas/indexes/category.py override to

    def holds_integer(self):
        return self.categories.holds_integer()

[alex-filatov]

I was mistaken regarding positional indexing. It seems that category codes are used as the index:

>>> s3 = pd.Series(['a', 'b', 'c'], index=pd.Series([3, 1, 2], dtype='category'))
>>> s3.get(3) is None
True
>>> s3.get(0)
'b'
>>> s3.get(1)
'c'
>>> s3.get(2)
'a'
>>> s3.index.codes
array([2, 0, 1], dtype=int8)

Tried to override holds_integer as advised - didn't resolve the issue. Not sure the method is invoked in this case.

[elsander]

I was looking for a first issue-- I can take a look at this.

[TomAugspurger]

Cool, thanks. If making that change to indexes/category.py does end up breaking other things, then this could turn out to be difficult.

[elsander]

I've been playing around with this to understand the behavior, and the category codes can be used as the index for non-numeric categorical columns as well:

In [35]: s4 = pd.Series(['a', 'b', 'c'], index=pd.Series(['hello', 'there', 'world'], dtype='category
    ...: '))

In [36]: s4.get(2)
Out[36]: 'c'

This makes me wonder whether this is a bug at all-- should the category codes just be the default here? The behavior is different than you might expect at first, but it is consistent. I'm going to look for a fix to see if we can default to using the category over the category code, but if it breaks other things, it might make sense to document the behavior rather than changing it.

[elsander]

Also, I get different behavior from @alex-filatov here.

In [59]: s3 = pd.Series(['a', 'b', 'c'], index=pd.Series([3, 1, 2], dtype='category'))

In [60]: s3.get(3)
Out[60]: 'a' # not None!

In [61]: s3.get(2)
Out[61]: 'c'

In [62]: s3.get(1)
Out[62]: 'b'

In [63]: s3.get(0)
Out[63]: 'b'

[elsander]

Maybe there's been a version release where this got fixed? I can't replicate the behavior.