Description
-
I have checked that this issue has not already been reported.
-
I have confirmed this bug exists on the latest version of pandas.
-
(optional) I have confirmed this bug exists on the master branch of pandas.
Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.
Code Sample, a copy-pastable example
import pandas as pd
def f(s):
s.pop("A")
df = pd.DataFrame({"A": [1, 2, 3]})
df.apply(f, axis="columns")Traceback (most recent call last):
File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3080, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 101, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 4554, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 4562, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'A'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/frame.py", line 7765, in apply
return op.get_result()
File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/apply.py", line 185, in get_result
return self.apply_standard()
File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/apply.py", line 276, in apply_standard
results, res_index = self.apply_series_generator()
File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/apply.py", line 290, in apply_series_generator
results[i] = self.f(v)
File "<stdin>", line 1, in f
File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/series.py", line 4467, in pop
return super().pop(item=item)
File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/generic.py", line 768, in pop
result = self[item]
File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/series.py", line 824, in __getitem__
return self._get_value(key)
File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/series.py", line 932, in _get_value
loc = self.index.get_loc(label)
File "/usr/local/google/home/rchristy/.local/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 3082, in get_loc
raise KeyError(key) from err
KeyError: 'A'Problem description
When the function being applied by apply pops a column from the row, the popped column is missing on subsequent calls of the function. This seems to stem from apply reusing the same Series instance across multiple invocations but failing to fully reset its state (i.e., the popped column stays popped).
This is an apparent regression from the behavior in v1.0.5 which returned a DataFrame with three rows. The timing of the regression suggests that it is possibly an unintended consequence of #34909.
The problem is relatively straightforward to work around. Performing a shallow copy on the Series fixes it:
>>> def f(s):
... s.copy(deep=False).pop("A")
...
>>> df = pd.DataFrame({"A": [1, 2, 3]})
>>> df.apply(f, axis="columns")
0 None
1 None
2 None
dtype: object
Expected Output
The expected output is what is shown above with the workaround which is what v1.0.5 output.
Output of pd.show_versions()
INSTALLED VERSIONS
commit : 9d598a5
python : 3.8.7.final.0
python-bits : 64
OS : Linux
OS-release : 5.7.17-1rodete4-amd64
Version : #1 SMP Debian 5.7.17-1rodete4 (2020-10-01)
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.2.1
numpy : 1.19.4
pytz : 2020.5
dateutil : 2.8.1
pip : 20.1.1
setuptools : 51.1.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None