Decide whether (new) ExtensionArrays and Dtypes are public

[TomAugspurger]

This mostly affects

1. IntervalArray
2. PeriodArray
3. DatetimeArray (and maybe TimedeltaArray if we do that)
4. SparseArray
5. IntegerArray

Categorical is already public, so let's leave that out.

---

A few questions

1. do we allow users to construct these directly (via a set of to_*_array methods, or a top-ish-level pd.array([...], dtype) method)?
2. Do users see these when they do .values or any operation returning an array (.unique, probably others)?

[TomAugspurger]

@pandas-dev/pandas-core here's a proposal: Everything goes at the top level.

1. That's one extra *_array per array.
2. One extra Dtype per array (more for IntegerArray)
3. (Maybe) add the actual Array itself to the top-level.

---

This is consistent with Categorical and SparseArray, meaning we wouldn't have to deprecate those if we want all the arrays in one place.

I'm concerned about nesting them in pandas.api.types.extensions.* In the past, users have had reservations about reaching that far into pandas' namespace. I think that's fine for library developers subclassing ExtensionArray, but not for interactive use.

So we add

1. categorical_array
2. CategoricalDtype
3. period_array
4. PeriodDtype
5. datetime_array
6. DatetimeDtype
7. sparse_array
8. interval_array
9. IntervalDtype
10. integer_array
11. IntegerDtype(...) (several of theses)

I don't think we need a timedelta_array / TimedeltaDtype in the public API. These shouldn't be user-facing at all.

[jbrockmendel]

Do users see these when they do .values or any operation returning an array

I would be ecstatic if we got to a place where a) .values always points to an EA, or at least a lossless non-costly array and b) ._values were rendered unnecessary. Even better if .get_values() also become a synonym.

or a top-ish-level pd.array([...], dtype) method)?

A lot of the pandas-internal EA code could be simplified if there were something analogous to Index.__new__ that returned the appropriately-typed array

[jreback]

I think we should leave the dtypes themselves in pandas.api.types

no objection to the *_array in the pandas namespace

[jschendel]

+1 on making these public in general.

8. interval_array

This does not currently exist and I'm a little confused as to what it would look like if implemented. I understand how this makes sense for the non-interval EA's, as they are usually constructed in a fairly direct manner. With intervals, my impression is that users have generally been using the IntervalIndex.from_* methods as opposed to the constructor itself, and I imagine the same will hold for IntervalArray.

Seems like trying to support all these construction methods in a single interval_array function would be a bit convoluted. Or am I missing something here? If not, my preference would be to expose the actual IntervalArray class itself.

I think we should leave the dtypes themselves in pandas.api.types

I agree, these don't seem to provide much additional utility in general. We could maybe make an exception for CategoricalDtype though, as it's useful for creating non-default categoricals via astype; this is what the docs recommend as well. The other dtypes don't have similar non-default behavior iirc, so should be fine just directing users to the string equivalent when specifying dtypes.

[TomAugspurger]

I agree with @jschendel that CategoricalDtype is especially useful on its own and should be in the top-level namespace.

[TomAugspurger]

Also agreed that IntervalArray's alternative constructors are especially useful, and conflating them in a single interval_array method isn't a great idea.

[jorisvandenbossche]

That's one extra *_array per array.

I have the feeling that adding a function for each array type is some API bloat. I think it would be nice to restrict ourselves to a general array(..) function, if that would be sufficient. It is eg also what pyarrow has to convert python array-likes (pure python/numpy/pandas containers) to an arrow array.

I suppose the main drawback of having a single function vs specialized functions for each array is the ability of having additional kwargs? (eg freq for period_array, or tz for datetime_array)

Further, I think we also need to discuss and somewhat define the "scope" of those .._array() functions. Although it might differ for each data type, I think we should have some general idea on this.
For example, is this function expected to do parsing of strings? (period_array does a limited form (only default format), integer_array does not; of course for periods this is a more typical use case) Is this function expected to accept "low-level values" (eg ordinals for period_array, ints for datetime_array) ...

I think for me the core use case that should certainly be covered is the round trip to/from "python objects" (so something like pd.array(np.asarray(EA, dtype=object), dtype=EA.dtype) should always work). Kind of the public version of _from_sequence.
But strictly this use case is of course more limited than eg what period_array already can do.

[jorisvandenbossche]

Another remark about where to put them in our API: I don't think those belong in pandas.api.extension.
Yes, we call them ExtensionArrays, and they are extending the numpy arrays, but they are a fundamental part of pandas, not a kind of secondary extension of pandas. For me, pandas.api.extension exposes the API for external people to extend pandas.

From the existing submodules, I would say that pandas.api.types is more fitting, but we might also consider adding something like pandas.api.arrays or even pandas.arrays (although that might feel strange to put the dtype objects)

[jbrockmendel]

Yes, we call them ExtensionArrays, and they are extending the numpy arrays, but they are a fundamental part of pandas, not a kind of secondary extension of pandas.

+1 on being clear about this.

[TomAugspurger]

On the idea of a single pd.array() function, that's fine, but would require adding all of the types to the top-level so that users can easily do

arr = pd.array([1, 2, 3], type=pd.Int64())

• Categorical: fine. Requires a bit more typing to get CategoricalDtype(..., ordered=...)
• Integer: fine, unless we want to accept a mask for the missing / valid values
• period: fine
• Sparse: not great, since the main SparseArray constructor accepts sparse_index and kind (and index which was want to remove).
• Interval: not being considered? Since it requires the alternative constructors?
• Datetime: Probably not sufficient, if we want to handle errors, dayfirst, year first, utc, etc.

I'd really like to avoid pd.array(data, type, **kwargs) if we can avoid it...

I think that a simple array(data, type) would suffice for categorical, period, integer (unless we want to support a mask= argument like pyarrow), datetime / timedelta, sparse.