pandas MultiIndex series "unstack" to scipy sparse array functionality

[cottrell]

related #4343

I have found myself on occasion writing code to convert from a pandas Series (with n-level MultiIndex) to a scipy sparse n-dim array (plus dimension labels) and back. I am not terribly familiar with the newer sparse data structure features but my impression is that it provides a totally different kind of functionality. Does this kind of functionality exist somewhere? If not, would something like this be of interest?

[TomAugspurger]

Do you have any short, concrete examples of what you're doing? Also let us know why using a SparseSeries doesn't work for it.

[cottrell]

Here is a quick example of going from a scipy.sparse array to a pandas Series and back. Mostly I am thinking of the case where you are data munging and are reading in labeled data and need to then switch to a sparse matrix in order to use something in scipy.sparse (multiplication, sparse svds or whatever). I am running this on 0.14.1. Apologies if this functionality is available on the dev branch. I have seen related question on stackoverflow but they all seem to indicate this is not yet implemented and that no one has given a strong opinion that it should be included either (maybe there is a good reason to avoid this kind of thing).

import scipy.sparse.linalg
import scipy.sparse
import pandas

# * limit outselves to 2d situation for now
# * also, for simplicity, all column and row labels are just the integer indices

# create a sparse array
m = int(1e6)
n = int(1e5)
A = scipy.sparse.rand(m, n, density=1e-8)

# scipy.sparse -> sparse series? Is there a better way?
s = pandas.Series(A.data)
s.index = pandas.MultiIndex.from_tuples(list(zip(A.row, A.col)))
s = s.to_sparse(fill_value=0)

# sparse series -> scipy.sparse array? s.unstack() is producing errors for me
i, j = list(zip(*s.index))
data = s.values
sA = scipy.sparse.coo_matrix((data, (i, j)), shape=(m, n))

check = A - sA
print('This should be zero', abs(check).max())

[TomAugspurger]

Thanks for the example. I agree with your intuition that s.unstack() should do something like what you want. I haven't used the sparse structures at all, but I'll take a look later. May just be not implemented yet.

[jreback]

@cottrell This would definitly be an improvements. See the related issue I linked to. I think several of the sparse routines could easily have an option to return a scipy coo type (or other sparse type matrix). It should be straightforward from a pandas sparse structure (which are very similar to coo types).

[jreback]

Here's an idea of the internal structure.

In [40]: ts = Series(randn(10))

In [41]: ts[2:-2] = np.nan

In [42]: sts = ts.to_sparse()

In [43]: sts
Out[43]: 
0   -0.785433
1    1.791538
2         NaN
3         NaN
4         NaN
5         NaN
6         NaN
7         NaN
8   -1.192896
9    2.672896
dtype: float64
BlockIndex
Block locations: array([0, 8], dtype=int32)
Block lengths: array([2, 2], dtype=int32)

In [44]: sts._data.values
Out[44]: 
[-0.785433163534, 1.7915375118, nan, nan, nan, nan, nan, nan, -1.19289573606, 2.67289566074]
Fill: nan
BlockIndex
Block locations: array([0, 8], dtype=int32)
Block lengths: array([2, 2], dtype=int32)

In [45]: sts._data
Out[45]: 
SingleBlockManager
Items: Int64Index([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype='int64')
SparseBlock: 10 dtype: float64

In [46]: sts._data.values.sp_index.blengths
Out[46]: array([2, 2], dtype=int32)

In [47]: sts._data.values.sp_index.blocs
Out[47]: array([0, 8], dtype=int32)

[cottrell]

Cool. I am not quite clear on whether the pandas.sparse structure should be the attachment point of some sort of unstack_to_scipy_sparse (this is a bad name) routine or not. Which do you think would make more sense (consider Series only for simplicity):

1. unstack_to_scipy_sparse for generic MultiIndexed Series
2. unstack_to_scipy_sparse for pandas.sparse MultiIndex Series

Basically, I think unstacking to scipy.sparse can happen with or without pandas.sparse storage.

[jreback]

I think it should go from a sparse structure. if you have a series it is dense by definition. (wether it is unstack from something else or not). The key is you can efficiency translate a pandas sparse structure to scipy w/o densing. any else doesn't make much sense.

[shoyer]

@cottrell Think of s.to_sparse().unstack().to_coo() as a better way of spelling s.unstack_to_scipy_sparse().

[byronyi]

I would like to know that, as v0.15.2 has been released, is this feature implemented?

[shoyer]

@byronyi I don't think so -- that's why this issue is still open! :)

[byronyi]

As a suggestion, you can simply do it by this:

from scipy.sparse import coo_matrix
coo = coo_matrix((series.values, zip(*series.index.values)))

Given the series has two-level integer multi-index.

But we need to deal with the fact when the index is not integer, and some kind of mapping is necessary.

[byronyi]

I think that implementing this will be kind of tricky when index is not numerical (or even not starting from 0).

But nevertheless, we can still put it on documentation of the Sparse page so people don't bother asking for answers again.