Should SparseArray.astype be dense or Sparse

[TomAugspurger]

Right now SparseArray.astype(numpy_dtype) is sparse:

In [6]: a = pd.SparseArray([0, 1, 0, 1])

In [7]: a.astype(np.dtype('float'))
Out[7]:
[0, 1.0, 0, 1.0]
Fill: 0
IntIndex
Indices: array([1, 3], dtype=int32)

This is potentially confusing. I did it to match the behavior of SparseSeries, but we may not want that.

[TomAugspurger]

This also leads to confusing behavior like

In [10]: a.astype(bool)
Out[10]:
[0, True, 0, True]
Fill: 0
IntIndex
Indices: array([1, 3], dtype=int32)

since .astype(bool) is interpreted as .astype(SparseDtype(bool, fill_value=0))

[jorisvandenbossche]

I think this is a regression compared to released version. Compare the last example:

In [18]: a = pd.SparseArray([0, 1, 0, 1])

In [20]: a.astype(bool)
Out[20]: 
[False, True, False, True]
Fill: False
IntIndex
Indices: array([1, 3], dtype=int32)

When astype-ing with a numpy dtype, I think we should simply "astype" the scalar fill_value as well.

[TomAugspurger]

So we have a few choices

1. Change SparseArray.astype to be dense for non-SparseDtype dtype
2. Keep SparseArray.astype always sparse, but change Series[sparse].astype to be dense for non-SparseDtype dtypes.

2 can be done and is backwards compatible. It means a special case in Block.astype, and means Series[sparse].astype isn't equivalent to SparseArray.astype.

1 is backwards incompatible. It'd be hard to do with a deprecation cycle I think.

[jorisvandenbossche]

Isn't there also option 3: keep both SparsArray.astype and Series[sparse].astype always sparse?

[TomAugspurger]

Yes, I was assuming we didn't want Series[sparse].astype(numpy_type) to be sparse though. Do we?

[jorisvandenbossche]

That's in any case what it is doing now (and in 0.23), so not doing that would also be a backwards incompatible change.

On the one hand it is convenient to keep it sparse if you quickly want to change the sparse type (without needing to use SparseDtype), but on the other hand it also might be strange that the result of Series.astype(dtype).dtype != dtype

[jorisvandenbossche]

But didn't we discuss that on the SparseArray PR? I seem to remember that you and @kernc argued for keeping it sparse?