.values on ExtensionArray-backed containers

[TomAugspurger]

Discussed briefly on the call today, but we should go through things formally.

What should the return type of Series[extension_array].values and Index[extension_array].values be? I believe the two options are

1. Return the ExtensionArray backing it (e.g. like what Categorical does)
2. Return an ndarray with some information loss / performance cost
   • e.g. like Series[datetimeTZ].values -> datetime64ns at UTC
   • e.g. Series[period].values -> ndarray[Period objects]

Current State

Not sure how much weight we should put on the current behavior, but for reference:

type	Series.values	Index.values
datetime	datetime64ns	datetime64ns
datetime-tz	datetine64ns(UTC&naive)	datetime64ns(UTC&naive)
categorical	Categorical	Categorical
period	NA	ndarray[Period objects]
interval	NA	ndarray[Interval objects]

In [5]: pd.Series(pd.date_range('2017', periods=1)).values
Out[5]: array(['2017-01-01T00:00:00.000000000'], dtype='datetime64[ns]')

In [6]: pd.Series(pd.date_range('2017', periods=1, tz='US/Eastern')).values
Out[6]: array(['2017-01-01T05:00:00.000000000'], dtype='datetime64[ns]')

In [7]: pd.Series(pd.Categorical([1])).values
Out[7]:
[1]
Categories (1, int64): [1]

In [8]: pd.Series(pd.SparseArray([1])).values
Out[8]:
[1]
Fill: 0
IntIndex
Indices: array([0], dtype=int32)

In [9]: pd.date_range('2017', periods=1).values
Out[9]: array(['2017-01-01T00:00:00.000000000'], dtype='datetime64[ns]')

In [10]: pd.date_range('2017', periods=1, tz='US/Central').values
Out[10]: array(['2017-01-01T06:00:00.000000000'], dtype='datetime64[ns]')

In [11]: pd.period_range('2017', periods=1, freq='D').values
Out[11]: array([Period('2017-01-01', 'D')], dtype=object)

In [12]: pd.interval_range(start=0, periods=1).values
Out[12]: array([Interval(0, 1, closed='right')], dtype=object)

In [13]: pd.CategoricalIndex([1]).values
Out[13]:
[1]
Categories (1, int64): [1]

If we decide to have the return values be ExtensionArrays, we'll need to discuss
to what extent they're part of the public API.

Regardless of the choice for .values, we'll probably want to support the other
use case (maybe just by documenting "call np.asarray on it). Internally, we
have ._values ("best" array, ndarray or EA) and ._ndarray_values (always an
ndarray).

cc @jreback @jorisvandenbossche @jschendel @jbrockmendel @shoyer @chris-b1

[TomAugspurger]

Not to derail the discussion from the get-go, but there's a closely related topic of how many "nuisance" ndarray methods we throw on the ExtensionArray base class to make writing code that's ndarray vs. extensionarray agnostic easier. For example tolist, which just calls list(self). .reshape, which just returns self. The list goes on (and on).

[shoyer]

I was pretty happy with always returning an ndarray from .values, but we broke that a long time ago with Categorical. For consistency we should probably clean this up (even for existing extension arrays), but unfortunately this will be a painful compatibility break.

I think my preferred option would be to make obj.values always return the underlying ExtensionArray, guaranteed to be the same data without any copies (so it can be modified in place). For converting to NumPy, we should probably add a obj.to_numpy() method on Series/DataFrame that is basically sugar for np.asarray(obj)

datetine64ns(UTC&naive)

NumPy's datetime64 is now always timezone naive, not UTC. I guess you meant the times are converted into UTC before being passed into NumPy?

there's a closely related topic of how many "nuisance" ndarray methods we throw on the ExtensionArray base class to make writing code that's ndarray vs. extensionarray agnostic easier

My preference would be to make the base class as abstract as possible, only implementing methods that are actually used by pandas. This will minimize the effort required to author new ExtensionArray classes, rather than requiring that authors override pre-existing methods. Extension arrays should be a low-level interface -- we already have the high-level interface for extension arrays in Series/DataFrame.

[chris-b1]

I think my preferred option would be to make obj.values always return the underlying ExtensionArray, guaranteed to be the same data without any copies (so it can be modified in place). For converting to NumPy, we should probably add a obj.to_numpy() method on Series/DataFrame that is basically sugar for np.asarray(obj)

+1 on this. I think the break in compatibility is less painful than the inconsistency and current situation of, e.g. PeriodIndex.values having very surprising performance costs.

xref to #15750 - was about changing the return type of tz-aware datetimes to boxed scalars.

[TomAugspurger]

I guess you meant the times are converted into UTC before being passed into NumPy?

Yep

I think I share the preference for .values being the actual array backing the data (even though that's not always possible with DataFrame.values).

Also +1 for a to_numpy method on Series / Index / Frame, with the same copy vs. view behavior as asarray.

[TomAugspurger]

Are people OK with Series[datetime64ns].values being an ndarray, but Series[datetime64ns, tz].values an ExtensionArray? That seems potentially confusing.

[jreback]

it has been this way for a long time
we should change the former to be honest - but last time there were some objections

[TomAugspurger]

it has been this way for a long time

What do you mean by that?

we should change the former to be honest - but last time there were some objections

Why would we change tz-naive to return anything other than a datetime64[ns] ndarray?

[chris-b1]

As stated above I'm OK with the inconsistency, but see where it could be surprising.

Another option would be to deprecate .values in favor of an explicit to_numpy() and another accessor that always returns the backing array, numpy or not, something like Series.array?

[TomAugspurger]

As stated above I'm OK with the inconsistency, but see where it could be surprising.

👍 . I wanted to get some explicit thoughts on that one, since I didn't initially comprehend that two seemingly similar types could have different outcomes.

another accessor that always returns the backing array

Yes. We'll want something like that. Internally, we've used ._values and ._ndarray_values. I like the name array.