Skip to content

ENH: .shift optionally takes multiple periods #54115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 32 commits into from
Jul 28, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
4e1a84e
init
jona-sassenhagen Jul 13, 2023
db9cd03
precommit
jona-sassenhagen Jul 13, 2023
9def045
slightly update test
jona-sassenhagen Jul 13, 2023
b7ea297
Fix groupby API tests
jona-sassenhagen Jul 13, 2023
299927a
mostly types, also exclude groupby
jona-sassenhagen Jul 14, 2023
6cf632e
fix test
jona-sassenhagen Jul 14, 2023
23d9142
mypy
jona-sassenhagen Jul 14, 2023
b78d7d6
fix docstring
jona-sassenhagen Jul 14, 2023
cabd28c
change how futurewarning is handled in the test
jona-sassenhagen Jul 15, 2023
5017721
fix docstring
jona-sassenhagen Jul 15, 2023
6f3ec9b
remove debug statement
jona-sassenhagen Jul 15, 2023
8d085f3
address comments
jona-sassenhagen Jul 16, 2023
6e66a19
refactor
jona-sassenhagen Jul 16, 2023
f7ea7a3
handle default
jona-sassenhagen Jul 17, 2023
f8e29d9
pylint
jona-sassenhagen Jul 17, 2023
e54ba33
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 17, 2023
715c397
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 18, 2023
21e8e70
merge conflicts and mypy
jona-sassenhagen Jul 18, 2023
0e78963
split tests, remove checking for None default
jona-sassenhagen Jul 18, 2023
2c602e2
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 18, 2023
d9bf54f
address comments
jona-sassenhagen Jul 19, 2023
778b7e9
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 19, 2023
ad49861
Merge branch 'jona/44424_shift' of github.com:jona-sassenhagen/pandas…
jona-sassenhagen Jul 19, 2023
28469db
mypy
jona-sassenhagen Jul 19, 2023
c8e5bad
mypy again
jona-sassenhagen Jul 19, 2023
cb4013d
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 19, 2023
226fa6f
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 21, 2023
cb49cac
black
jona-sassenhagen Jul 21, 2023
44b0866
address comments
jona-sassenhagen Jul 25, 2023
5257292
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 25, 2023
eff4ed2
mypy
jona-sassenhagen Jul 25, 2023
04d4513
Merge branch 'jona/44424_shift' of github.com:jona-sassenhagen/pandas…
jona-sassenhagen Jul 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
precommit
  • Loading branch information
jona-sassenhagen committed Jul 13, 2023
commit db9cd03c1d0a4c804e3e0b38e799f4e31d499fcd
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v2.1.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -123,12 +123,12 @@ Other enhancements
- Added :meth:`ExtensionArray.interpolate` used by :meth:`Series.interpolate` and :meth:`DataFrame.interpolate` (:issue:`53659`)
- Added ``engine_kwargs`` parameter to :meth:`DataFrame.to_excel` (:issue:`53220`)
- Added a new parameter ``by_row`` to :meth:`Series.apply` and :meth:`DataFrame.apply`. When set to ``False`` the supplied callables will always operate on the whole Series or DataFrame (:issue:`53400`, :issue:`53601`).
- :meth:`DataFrame.shift` and :meth:`Series.shift` now allow shifting by multiple periods by supplying a list of periods (:issue:`44424`)
- Groupby aggregations (such as :meth:`DataFrameGroupby.sum`) now can preserve the dtype of the input instead of casting to ``float64`` (:issue:`44952`)
- Improved error message when :meth:`DataFrameGroupBy.agg` failed (:issue:`52930`)
- Many read/to_* functions, such as :meth:`DataFrame.to_pickle` and :func:`read_csv`, support forwarding compression arguments to lzma.LZMAFile (:issue:`52979`)
- Performance improvement in :func:`concat` with homogeneous ``np.float64`` or ``np.float32`` dtypes (:issue:`52685`)
- Performance improvement in :meth:`DataFrame.filter` when ``items`` is given (:issue:`52941`)
- :meth:`DataFrame.shift` and :meth:`Series.shift` now allow shifting by multiple periods by supplying a list of periods (:issue:`44424`)
-

.. ---------------------------------------------------------------------------
Expand Down
9 changes: 6 additions & 3 deletions pandas/core/frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -5511,15 +5511,18 @@ def shift(

if is_list_like(periods):
if axis == 1:
raise ValueError('If `periods` contains multiple shifts, `axis` cannot be 1.')
raise ValueError(
"If `periods` contains multiple shifts, `axis` cannot be 1."
)
if len(periods) == 0:
raise ValueError('If `periods` is an iterable, it cannot be empty.')
raise ValueError("If `periods` is an iterable, it cannot be empty.")
from pandas.core.reshape.concat import concat

result = []
for period in periods:
if not isinstance(period, int):
raise TypeError(
f"Value {period} in periods must be integer, but is type {type(period)}."
f"Periods must be integer, but {period} is {type(period)}."
)
result.append(
super()
Expand Down
4 changes: 2 additions & 2 deletions pandas/core/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
Any,
Callable,
ClassVar,
Iterable,
Literal,
NoReturn,
cast,
Expand Down Expand Up @@ -198,6 +197,7 @@
if TYPE_CHECKING:
from collections.abc import (
Hashable,
Iterable,
Iterator,
Mapping,
Sequence,
Expand Down Expand Up @@ -10655,7 +10655,7 @@ def shift(
if is_list_like(periods) and len(self.shape) == 1:
return self.to_frame().shift(
periods=periods, freq=freq, axis=axis, fill_value=fill_value
)
)

if freq is None:
# when freq is None, data is shifted, index is not
Expand Down
16 changes: 8 additions & 8 deletions pandas/tests/frame/methods/test_shift.py
Original file line number Diff line number Diff line change
Expand Up @@ -678,22 +678,22 @@ def test_shift_with_iterable(self):
tm.assert_frame_equal(expected, shifted)

# test pd.Series
s: pd.Series = df['a']
tm.assert_frame_equal(s.shift(shifts), df[['a']].shift(shifts))
s: Series = df["a"]
tm.assert_frame_equal(s.shift(shifts), df[["a"]].shift(shifts))

# test suffix
columns = df[['a']].shift(shifts, suffix='_suffix').columns
assert columns.tolist() == ['a_suffix_0', 'a_suffix_1', 'a_suffix_2']
columns = df[["a"]].shift(shifts, suffix="_suffix").columns
assert columns.tolist() == ["a_suffix_0", "a_suffix_1", "a_suffix_2"]

# check bad inputs when doing multiple shifts
msg = "If `periods` contains multiple shifts, `axis` cannot be 1."
with pytest.raises(ValueError, match=msg):
df.shift([1, 2], axis=1)
msg = f"Value s in periods must be integer, but is type <class 'str'>."

msg = "Periods must be integer, but s is <class 'str'>."
with pytest.raises(TypeError, match=msg):
df.shift(['s'])
df.shift(["s"])

msg = f"If `periods` is an iterable, it cannot be empty."
msg = "If `periods` is an iterable, it cannot be empty."
with pytest.raises(ValueError, match=msg):
df.shift([])