Skip to content

BUG/PERF: Sparse get_dummies uses concat #24372

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Dec 21, 2018
Merged

BUG/PERF: Sparse get_dummies uses concat #24372

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
aa08a6d
Fixed warnings in asv files
TomAugspurger Dec 19, 2018
ae026b2
avoid series constructor
TomAugspurger Dec 20, 2018
b253674
BUG: Fix concat(Series[sparse], axis=1)
TomAugspurger Dec 20, 2018
6a65cbc
SparseSeries unstack
TomAugspurger Dec 20, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
avoid series constructor 
(cherry picked from commit eb219ac)
  • Loading branch information
@TomAugspurger
TomAugspurger committed Dec 20, 2018
commit ae026b227b56f1ad73fcccc6a20b62a7cc069c13
8 changes: 4 additions & 4 deletions pandas/core/reshape/reshape.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -853,6 +853,7 @@ def check_len(item, name):

def _get_dummies_1d(data, prefix, prefix_sep='_', dummy_na=False,
sparse=False, drop_first=False, dtype=None):
from pandas.core.reshape.concat import concat
# Series avoids inconsistent NaN handling
codes, levels = _factorize_from_iterable(Series(data))

Expand Down Expand Up @@ -909,7 +910,7 @@ def _make_col_name(prefix, prefix_sep, level):
index = None

if sparse:
sparse_series = {}
sparse_series = []
N = len(data)
sp_indices = [[] for _ in range(len(dummy_cols))]
mask = codes != -1
Expand All @@ -928,10 +929,9 @@ def _make_col_name(prefix, prefix_sep, level):
sarr = SparseArray(np.ones(len(ixs), dtype=dtype),
sparse_index=IntIndex(N, ixs), fill_value=0,
dtype=dtype)
sparse_series[col] = Series(data=sarr, index=index)
sparse_series.append(Series(data=sarr, index=index, name=col))

out = DataFrame(sparse_series, index=index, columns=dummy_cols,
dtype=dtype)
out = concat(sparse_series, axis=1, copy=False)
return out

else:
Expand Down