Default string dtype (PDEP-14): naming convention to distinguish the dtype variants

[jorisvandenbossche]

Context: the future string dtype for 3.0 (currently enabled with pd.options.futu.infer_string = True) is being formalized in a PDEP in #58551, and one of the discussion points is how to name the different variants of the StringDtype that will exist with the PDEP (whether using pyarrow or numpy object dtype for the data under the hood, and whether using NA or NaN as missing value sentinel).

As explained in #54792 (comment), we introduced the NaN-variant of the dtype for 3.0 as pd.StringDtype(storage="pyarrow_numpy") because we wanted to reuse the storage keyword but "pyarrow" is already taken (by the dtype using pd.NA introduced in pandas 1.3), and because we couldn't think of a better name at the time. But as also mentioned back then, that is far from a great name.

But as mentioned by @jbrockmendel in #58551 (comment), we don't necessarily need to reuse just the storage keyword, but we could also add new keywords to distinguish the dtype variants.

That got me thinking and listing some possible options here:

• Add an extra keyword that distinguishes the NA sentinel (and with that implicitly the type of missing value semantics):
  • Possible names for pd.StringDtype(storage="python"|"pyarrow", <something>):
    • semantics="numpy" (and the other would then be "nullable" or 'arrow" or ..?)
    • na_value=np.nan
    • na_marker=np.nan
    • missing=np.nan
    • nullable=False (although we have used "nullable dtypes" in the past to denote the dtypes using NA, it's also confusing here because the False variant does support missing values as well)
  • One drawback here that I don't think users should actually ever explicitly do pd.StringDtype(storage="pyarrow", na_value=np.nan) as that is not future proof. But defaulting to na_value=np.nan (to avoid requiring to specify it) is then not backwards compatible with current pd.StringDtype(storage="pyarrow")
• Add a new keyword separate from storage to determine the storage/backend that only controls the new variants with NaN.
  • Given we are using storage right now, but speak about "backend" in other places, we could add for example a backend keyword, where StringDtype(storage="python"|"pyarrow") keeps resulting in the dtypes using NA (backwards compatible), while doing StringDtype(backend="python"|"pyarrow") gives you the new dtypes using NaN (and specifying both then obviously errors)
  • This is not great API design to have two keywords that are mutually exclusive but are essentially controlling the same thing, but, it does avoid having to specify two keywords (or having the confusing names)
  • One question is which keyword name to use. backend has prior use in the "dtypes_backend" terminology. Irv suggested nature below.
• For completeness, we can also still come up with a better storage name than "pyarrrow_numpy" and stick to that single existing keyword. Suggestions from the PDEP PR:
  • "pyarrow_nan"
  • "pyarrow_legacy" (I wouldn't go with this one, because for users it is not "legacy" right now, rather it would be the default. It will only become legacy later if we decide on switching to NA later)

After writing this down, I think my current preference would go to StringDtype(backend="python"|"pyarrow"), as that seems the simplest for most users (it's a bit confusing for those who already explicitly used storage, but most users have never done that)

[jorisvandenbossche]

FWIW, there is also a fourth option: not have any keyword for this, and don't give users a way to control this through StringDtype().
Calling the default pd.StringDtype() will under the hood still create a dtype instance either backed by numpy object or pyarrow depending on whether pyarrow is installed, but that choice would then always be done automatically. And we can still have a private API to create the dtype with one of the specific backends for testing.

[simonjayhawkins]

I guess I was hinting at something similar for the semantics keyword #58551 (comment). i.e. We do don't have that public on the dtype (but as a dtype property) and perhaps control that at the DataFrame (and maybe Series) level, so that a nullable array assigned to a DataFrame would coerce to NumPy semantics.

[Dr-Irv]

It seems to me that there are 2 choices that a user could make, along with options for those choices

• How strings are stored and manipulated:
  • pyarrow
  • numpy objects (current 2.x behavior)
  • numpy strings (requires numpy 2.0)
• How missing values are handled:
  • Use np.nan
  • Use pd.NA

Aside from the numpy 2.0 option, I think we have implementation of all combinations of the storage and missing values available, and it seems to me that when we implement support for numpy strings that depends on numpy 2.0, we'd want to support np.nan and pd.NA for missing values.

There are then a few questions to address:

1. How should a user choose the storage/manipulation/backend and what the keyword should be
2. How should a user choose the missing value behavior
3. What should the defaults of each be
4. What do we do to handle compatibility from version 2.2 to 3.0
5. What is the class name for specifying these dtypes
6. What is the string representation of these dtypes

I'd like to suggest the word nature as the keyword to represent the storage/backend and a keyword missing for the missing value. We deprecate the word storage in pd.StringDtype(), and document how the current storage argument for pd.StringDtype() map to nature and missing . That will address questions 1, 2 and 4. The typed signature in the future (without the storage keyword) would be:

class StringDtype(StorageExtensionDtype):
    def __init__(self, nature: Literal["pyarrow", "numpy.object", "numpy.string"] = "pyarrow",
                 missing: np.nan | pd.NA = pd.NA) -> None: ...

Why nature? When you choose one of the 3 options, you are describing not just the storage backend, but also the implementation of the string methods like str.len(), str.split, etc. So it is the "nature" of the storage and the behavior that is being specified.

I'd also like to address question 6, we use a new nomenclature for the strings that represent dtypes corresponding to strings. Let's NOT use the word "numpy" to represent using np.nan for missing values, but be explicit in using the strings "pd.NA" and "np.nan". The resulting strings representing all 6 combinations could then be (with my best guess for the equivalences of today's behavior):

• "pyarrow_np.nan" (equivalent to "pyarrow_numpy")
• "pyarrow_pd.NA" (equivalent to storage="pyarrow")
• "numpy.object_np.nan" (equivalent to dtype="object")
• "numpy.object_pd.NA" (equivalent to pd.StringDtype() in 2.x ??)
• "numpy.string_pd.NA" (New)
• "numpy.string_np.nan" (New)

For the missing value indicators, let's be explicit about whether np.nan or pd.NA is being used whenever we refer to them, because the community at large is going to have to be educated about the difference in the future and using some other "code word" to represent the 2 possibilities is just masking the issue even more (no pun intended).

[simonjayhawkins]

After writing this down, I think my current preference would go to StringDtype(backend="python"|"pyarrow"), as that seems the simplest for most users (it's a bit confusing for those who already explicitly used storage, but most users have never done that)

I'd like to suggest the word nature as the keyword to represent the storage/backend

some more suggestions: memory, memory_layout or layout

• How missing values are handled:

For the missing value indicators, let's be explicit about whether np.nan or pd.NA is being used whenever we refer to them

This is maybe not explicit about the behavior in the sense that the nullable string dtypes that return numeric output will always return a nullable integer dtype, rather than either int or float dtype, depending on the presence of NA values. So this behavior gives a more consistent return type and using terms such as missing or na_value do not convey this.

[jbrockmendel]

backend="python"|"pyarrow"

is "python" an option for the dtype_backend keyword? The keywords are similar enough that it will cause confusion if they don't have matching behavior.

nullable=False (although we have used "nullable dtypes" in the past to denote the dtypes using NA, it's also confusing here because the False variant does support missing values as well)

Yah I think there is confusion as to what "nullable" means depending on the writer/context.

propagation=... would be more explicit than semantics in describing what it controls. The downside is I can never remember whether to spell it "propa" or "propo".

I lean towards storage+na_value.

[jorisvandenbossche]

It seems to me that there are 2 choices that a user could make, along with options for those choices

For the PDEP / pandas 3.0, I personally explicitly do not want typical users to make this choice. In each context (pyarrow installed or not), there is just one default string dtype, and that is all what most uses should worry about.

So while we still need some (public or internal) way to create the different variants explicitly (which is what we are discussing here, so thanks for your comments!), in context of the PDEP I would like to hide that as much as possible. For that reason, I am personally not really a fan of adding an explicit keyword to choose the missing value semantics (like na_value or missing). Or the elaborate string representations that give a lot of details. Users that didn't opt in to any of the experimental options should just see string, and IMO we could even disallow creating the new NaN-variants of the string dtype with a alias other than "string" (like, don't support "string[pyarrow_numpy]" to use the current naming in the main branch)

[jorisvandenbossche]

backend="python"|"pyarrow"

is "python" an option for the dtype_backend keyword? The keywords are similar enough that it will cause confusion if they don't have matching behavior.

No, but currently you also can't choose the "default" dtypes through dtype_backend, that keyword is only used for opting in to a set of non-default dtypes.
Now, it is a good point that backend="python" wouldn't translate to other dtypes like numeric or datetime dtypes, where we would never have the option to store such data as python objects (although in theory for date that could be an option, but not that we actually want to add it I think)

[Dr-Irv]

For the PDEP / pandas 3.0, I personally explicitly do not want typical users to make this choice. In each context (pyarrow installed or not), there is just one default string dtype, and that is all what most uses should worry about.

But at the top of this issue, you wrote:

one of the discussion points is how to name the different variants of the StringDtype that will exist with the PDEP (whether using pyarrow or numpy object dtype for the data under the hood, and whether using NA or NaN as missing value sentinel).

I agree that typical users should not make that choice, but if we use the nature and missing concept, and have default values for those parameters, then they don't have to make the choice, unless they choose to do so.

[lithomas1]

FWIW, there is also a fourth option: not have any keyword for this, and don't give users a way to control this through StringDtype(). Calling the default pd.StringDtype() will under the hood still create a dtype instance either backed by numpy object or pyarrow depending on whether pyarrow is installed, but that choice would then always be done automatically. And we can still have a private API to create the dtype with one of the specific backends for testing.

Big +1 on this.

I don't think there's a good way to resolve the ambiguity in a name like "pyarrow_numpy", and since pyarrow_numpy and the python fallback are going to be the default string dtype anyways, it's not going to be a common usecase to convert to it.

So, I would be fine making the users manually specify the whatever keywords that we decide on to create the pyarrow/python backed string array with np.nan as the missing value, and having no way to create e.g. a pyarrow_numpy array with a string alias.

I lean towards storage+na_value.

+1 on this.

There's a precedent for storage for string arrays, and na_value has history in things like read_csv.

I would be against adding a new keyword that clashes with "storage" (or changing "storage"), since it makes for messy handling internally. I also don't think it's worth the churn.

I'm less opinionated about na_value.

[WillAyd]

I realize this is for PDEP 14 which we want as a fast mover, but PDEP 13 proposed the following structure for a data type:

class BaseType:

    @property
    def dtype_backend -> Literal["pandas", "numpy", "pyarrow"]:
        """
        Library is responsible for the array implementation
        """
        ...

    @property
    def physical_type:
        """
        How does the backend physically implement this logical type? i.e. our
        logical type may be a "string" and we are using pyarrow underneath -
        is it a pa.string(), pa.large_string(), pa.string_view() or something else?
        """
        ...

    @property
    def missing_value_marker -> pd.NA|np.nan:
        """
        Sentinel used to denote missing values
        """
        ...

Which may be of interest here too (though feedback so far is that missing_value_marker is better called na_value).

It might be useful to separate the idea of the backend / provider (e.g. "pyarrow") from how that backend / provider is actually implementing things (e.g. "string", "large_string", "string_view"). @Dr-Irv I think you are going down that path with your "nature" suggestion but rather than trying to cram all that metadata into a single string having two properties might be more future proof

[Dr-Irv]

It might be useful to separate the idea of the backend / provider (e.g. "pyarrow") from how that backend / provider is actually implementing things (e.g. "string", "large_string", "string_view"). @Dr-Irv I think you are going down that path with your "nature" suggestion but rather than trying to cram all that metadata into a single string having two properties might be more future proof

Yes, the fact that a pyarrow backend could use a different physical type adds another aspect. So backend and physical_type might make sense to have instead of "nature"