Series.reindex() does nothing

[dniku]

Code Sample, a copy-pastable example if possible

import numpy as np
import pandas as pd

np.random.seed(42)

nums = np.random.choice(range(0, 999), 1000)
cut = pd.cut(nums, 10)

discretization = cut.value_counts()
discretization.sort_index(inplace=True)

intervals = list(discretization.index)
mids = [i.mid for i in intervals]

print(discretization.reindex(index=mids))

Output:

(-0.998, 99.8]    100
(99.8, 199.6]     103
(199.6, 299.4]     97
(299.4, 399.2]     94
(399.2, 499.0]     97
(499.0, 598.8]     85
(598.8, 698.6]    118
(698.6, 798.4]    103
(798.4, 898.2]    108
(898.2, 998.0]     95
dtype: int64

Problem description

Series.reindex() returns the original Series even though the index is changed.

Expected Output

print(pd.Series(discretization.values, index=mids)) produces:

49.401     100
149.700    103
249.500     97
349.300     94
449.100     97
548.900     85
648.700    118
748.500    103
848.300    108
948.100     95
dtype: int64

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.39-1-MANJARO
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.20.3
pytest: 3.1.3
pip: 9.0.1
setuptools: 36.2.2
Cython: None
numpy: 1.13.1
scipy: 0.19.1
xarray: None
IPython: 5.3.0
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: None
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999999999
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: 0.4.0

[gfyoung]

@Pastafarianist : Thanks for the report! Unfortunately, we can't run this code because discretization_classes is not defined. Could you provide a reproducible code example?

In addition, if you could copy + paste the output (both actual and expected), that would be useful for anyone who's reading these issues.

[dniku]

Apologies; please set it to any integer like 10. I've updated the code in the issue description.

[gfyoung]

I've updated the code in the issue description.

Could you move whichever is your expected output under your sentence regarding the "second print" ?

[dniku]

Done.

[gfyoung]

Did you reverse the two? The expected output should be at the bottom, but I think you put it second-to-last if I understand the issue properly.

[dniku]

No, I did not. The issue description reads correctly right now.

[gfyoung]

Ah, actually, yes I see that now. One other thing: you're missing a definition for intervals in your code. Could you add a definition for that?

[dniku]

So sorry for that. I was copy-pasting from a Jupyter Notebook. I've updated the description and made sure that the code runs as-is.

[gfyoung]

No worries! Can confirm the code is runnable standalone.

That's odd...not sure why you can't reindex when the index are Interval. This re-indexing works if the index is a sequence of integers. Marking as a bug unless otherwise explained. PR is welcome!

[jreback]

@Pastafarianist not sure what you think this should do. .reindex matches up the existing with new. Reindexing with floats is meaningless with an IntervalIndex (which is the index backing a CategoricalIndex).

do you simply want

In [3]: discretization.index = mids

In [4]: discretization
Out[4]: 
49.401     100
149.700    103
249.500     97
349.300     94
449.100     97
548.900     85
648.700    118
748.500    103
848.300    108
948.100     95
dtype: int64

This is reindexing.

In [7]: discretization.reindex(discretization.index[[0, 2, 5]])
Out[7]: 
(-0.998, 99.8]    100
(199.6, 299.4]     97
(499.0, 598.8]     85
dtype: int64

[gfyoung]

@jreback : Ah, I see. I also had the implementation mixed up in my mind, but given the behavior, why don't we just get a Series of NaN ?

[jorisvandenbossche]

IMO this is certainly a bug. It should never return the original series, as it should either do an actual reindex (if we decide that eg 1 matches Interval(0, 2)) or if no matches our found indeed return a Series of all NaN.
The reindex method should always return an object with the new index that you passed.

[jorisvandenbossche]

Actually, on second look, it is reindexing, it just looks identical because you get the same values back (it just doing the same as discretization.loc[mids].
This is another example of the discussion we had yesterday of differences between loc and reindex. IMO the expected output of @Pastafarianist is more logical (more in line of what reindex is supposed to do).

[jreback]

IMO the expected output of @Pastafarianist is more logical (more in line of what reindex is supposed to do).

not at all. This is a correct result. The points happen to be reindexer of the intervals. E.g.

Not very useful, but the first 2 intervals are picked out because the contain the points 20 and 200. The 2000 gets nan because its not found.

In [13]: discretization.reindex(index=[20, 200, 2000])
Out[13]: 
(-0.998, 99.8]    100.0
(199.6, 299.4]     97.0
2000                NaN
dtype: float64

[jreback]

xref #16386

[jorisvandenbossche]

@jreback can you then explain the docstring? (emphasis mine)

Conform Series to new index with optional filling logic, placing
NA/NaN in locations having no value in the previous index. A new object
is produced unless the new index is equivalent to the current one and
copy=False

(the docstring could also have been incorrect for a long time of course)

I would argue that the IntervalIndex and the integers (the mids) are not equivalent in this case

[jreback]

what part is not correct? seems ok to me.

I would argue that the IntervalIndex and the integers (the mids) are not equivalent in this case

this is correct based on what Interval selection does. selection in an interval select an interval. Hence you get back the original index.

[gfyoung]

But they are not identical though, which is @jorisvandenbossche point, and if your explanation is expected behavior, I don't see that in the docs.

[ceciliassis]

Sorry for the late reply. but i seam to have the same problem with INT indexes.
The code:

import math
from IPython import display
from matplotlib import cm
from matplotlib import gridspec
from matplotlib import pyplot as plt
import numpy as np
import pandas as pd
from sklearn import metrics
import tensorflow as tf
from tensorflow.python.data import Dataset

california_housing_dataframe = pd.read_csv("https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv", sep=",")
print(california_housing_dataframe.head())

california_housing_dataframe = california_housing_dataframe.reindex(np.random.permutation(california_housing_dataframe.index).tolist())
print(california_housing_dataframe.head())

Whenever I use the numpy permutation the result goes wrong. I've tried shell, jupyter notebook and script.

Here's the output

Configs

Python version: 2.7 and 3.6
Pandas version: 0.22.0 and 0.23.6
Conda: Anaconda3

[jorisvandenbossche]

@ceciliassis It's not directly clear what is wrong in your output, or how it is related to this issue. If you think there is a bug, please open a new issue.