Skip to content

PERF: always slice when indexing on columns #33597

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 63 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
1697252
PERF: block-wise arithmetic for frame-with-frame
jbrockmendel Mar 17, 2020
a7764d6
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Mar 17, 2020
30a836d
lint fixup
jbrockmendel Mar 17, 2020
3559698
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Mar 18, 2020
4334353
troubleshoot npdev build
jbrockmendel Mar 18, 2020
cb40b0c
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Mar 24, 2020
713a776
comment
jbrockmendel Mar 25, 2020
95ef3ad
checkpoint passing
jbrockmendel Mar 25, 2020
61e5cd6
checkpoint passing
jbrockmendel Mar 25, 2020
89c3d7b
refactor
jbrockmendel Mar 25, 2020
e348e46
blackify
jbrockmendel Mar 25, 2020
519c757
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Mar 31, 2020
2b1ba18
disable assertions for perf
jbrockmendel Mar 31, 2020
53e93fc
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 1, 2020
91c86a3
asv
jbrockmendel Apr 1, 2020
2034084
whatsnew
jbrockmendel Apr 1, 2020
8aedf35
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 3, 2020
0c12d35
revert warning suppression
jbrockmendel Apr 3, 2020
9727562
Fixupm indentation
jbrockmendel Apr 3, 2020
6661dd3
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 4, 2020
42bbbf3
suppress warning
jbrockmendel Apr 4, 2020
65ab023
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 6, 2020
0d958a3
update asv
jbrockmendel Apr 6, 2020
7f91e74
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 6, 2020
56eef51
_data->_mgr
jbrockmendel Apr 6, 2020
4baea6f
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 7, 2020
41a4e7a
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 7, 2020
b23144e
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 9, 2020
7f24d57
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 9, 2020
ae744b7
update to use faspath constructor
jbrockmendel Apr 9, 2020
b14a98c
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 10, 2020
f42c403
update import
jbrockmendel Apr 10, 2020
8a2807e
remove unused import
jbrockmendel Apr 10, 2020
fa046f0
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 10, 2020
fd10fb6
rebase compat
jbrockmendel Apr 10, 2020
7ea5d3a
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 11, 2020
7150e87
slice instead of take
jbrockmendel Apr 12, 2020
bddfbb0
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 13, 2020
25f83d6
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 13, 2020
2142d29
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 14, 2020
0ca2125
Dummy commit to force CI
jbrockmendel Apr 14, 2020
1ea0cc0
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 15, 2020
2bfc308
update call bound
jbrockmendel Apr 15, 2020
d5ad2a0
update max_len
jbrockmendel Apr 15, 2020
92007e8
revert
jbrockmendel Apr 16, 2020
10f6349
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 16, 2020
3ce6f51
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 16, 2020
df471d8
PERF: always slice when indexing on columns
jbrockmendel Apr 16, 2020
b323ea6
typo fixup
jbrockmendel Apr 16, 2020
e557ace
Merge branch 'master' of https://github.com/pandas-dev/pandas into pe…
jbrockmendel Apr 21, 2020
96b7d6a
avoid take in one more case
jbrockmendel Apr 21, 2020
2ab3917
Merge branch 'master' into perf-never-take
jbrockmendel Jul 16, 2021
c63464f
Merge branch 'master' into perf-never-take
jbrockmendel Jul 18, 2021
6e34e3e
Merge branch 'master' into perf-never-take
jbrockmendel Jul 21, 2021
314a0f8
Merge branch 'master' into perf-never-take
jbrockmendel Jul 21, 2021
587ffef
Merge branch 'master' into perf-never-take
jbrockmendel Jul 24, 2021
7fad752
update
jbrockmendel Jul 24, 2021
7e8b0bd
Merge branch 'master' into perf-never-take
jbrockmendel Jul 31, 2021
6d82359
Merge branch 'master' into perf-never-take
jbrockmendel Aug 8, 2021
201e57d
fix last 2 failing tests
jbrockmendel Aug 14, 2021
69f08cb
mypy fixup
jbrockmendel Aug 15, 2021
f65cbf1
Merge branch 'master' into perf-never-take
jbrockmendel Aug 24, 2021
f337dfe
Merge branch 'master' into perf-never-take
jbrockmendel Aug 26, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' into perf-never-take
  • Loading branch information
jbrockmendel committed Aug 24, 2021
commit f65cbf19530d7eaa6dd960cfd3f7e426d0aff86b
1 change: 1 addition & 0 deletions pandas/_libs/internals.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ class BlockPlacement:
def __len__(self) -> int: ...
def delete(self, loc) -> BlockPlacement: ...
def append(self, others: list[BlockPlacement]) -> BlockPlacement: ...
def tile_for_unstack(self, factor: int) -> np.ndarray: ...
def to_slices(self) -> list[BlockPlacement]: ...

class SharedBlock:
Expand Down
54 changes: 54 additions & 0 deletions pandas/_libs/internals.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -210,6 +210,60 @@ cdef class BlockPlacement:

return self._as_slice

cpdef BlockPlacement increment_above(self, Py_ssize_t loc):
"""
Increment any entries of 'loc' or above by one.
"""
cdef:
slice nv, s = self._ensure_has_slice()
Py_ssize_t other_int, start, stop, step, l
ndarray newarr

if s is not None:
# see if we are either all-above or all-below, each of which
# have fastpaths available.

start, stop, step, l = slice_get_indices_ex(s)

if start < loc and stop <= loc:
# We are entirely below, nothing to increment
return self

if start >= loc and stop >= loc:
# We are entirely above, we can efficiently increment out slice
nv = slice(start + 1, stop + 1, step)
return BlockPlacement(nv)

if loc == 0:
# fastpath where we know everything is >= 0
newarr = self.as_array + 1
return BlockPlacement(newarr)

newarr = self.as_array.copy()
newarr[newarr >= loc] += 1
return BlockPlacement(newarr)

def tile_for_unstack(self, factor: int) -> np.ndarray:
"""
Find the new mgr_locs for the un-stacked version of a Block.
"""
cdef:
slice slc = self._ensure_has_slice()
slice new_slice
ndarray new_placement

if slc is not None and slc.step == 1:
new_slc = slice(slc.start * factor, slc.stop * factor, 1)
new_placement = np.arange(new_slc.start, new_slc.stop, dtype=np.intp)
else:
# Note: test_pivot_table_empty_aggfunc gets here with `slc is not None`
mapped = [
np.arange(x * factor, (x + 1) * factor, dtype=np.intp)
for x in self
]
new_placement = np.concatenate(mapped)
return new_placement

def to_slices(self):
"""
Decompose a BlockPlacement into a list of BlockPlacements, each of
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.