Arrow string array dtype #36142

TomAugspurger · 2020-09-05T14:59:38Z

This is a continuation of #35259, updated to use a single StringDtype parameterized by storage (python or pyarrow). We do use different classes for the array though.

I wouldn't recommend reviewing this yet. In particular, I need to refactor the string ops implementation. That will likely be done in a separate PR to keep things smaller here.

In [1]: import pandas as pd

In [2]: a = pd.Series(pd.array([pd.NA, pd.NA], dtype=pd.StringDtype("python")))

In [3]: b = pd.Series(pd.array([pd.NA, pd.NA], dtype=pd.StringDtype("pyarrow")))

In [4]: a.array
Out[4]:
<StringArray>
[<NA>, <NA>]
Length: 2, dtype: StringDtype[python]

In [5]: b.array
Out[5]:
<ArrowStringArray>
[<NA>, <NA>]
Length: 2, dtype: StringDtype[pyarrow]

Closes #35169

TomAugspurger · 2020-09-05T14:59:52Z

Just FYI @xhochy.

xhochy · 2020-09-05T18:07:06Z

@TomAugspurger I would continue to work on this starting next week again. Do we want to coordinate somehow?

jorisvandenbossche

(I know you said to not yet review, but was just curious ;) The dtype dispatching to two different array classes looks good to me)

jorisvandenbossche · 2020-09-07T06:41:27Z

pandas/core/arrays/base.py

@@ -459,6 +461,7 @@ def astype(self, dtype, copy=True):
        from pandas.core.arrays.string_ import StringDtype

        dtype = pandas_dtype(dtype)
+        # FIXME: Really hard-code here?


This needs the general 'astype' machinery refactor to fix this, I think

jorisvandenbossche · 2020-09-07T06:43:27Z

pandas/core/arrays/string_.py

+            ========================== ==============
+            ``'string'``               global default
+            ``'string[python]'``       python
+            ``'StringDtype[python]'``  python


Why allow the full class name "StringDtype" as string as well? (I don't think we do that for other dtypes?)

EDIT: ah, I see you used that as name above. But so then: why not "string" as name?

jorisvandenbossche · 2020-09-07T06:44:25Z

pandas/core/arrays/string_.py

+
+    @property
+    def name(self):
+        return f"StringDtype[{self.storage}]"


Suggested change

return f"StringDtype[{self.storage}]"

return f"string[{self.storage}]"

?

Agreed, this seems better.

jorisvandenbossche · 2020-09-07T06:46:06Z

pandas/core/arrays/string_arrow.py

+
+    Examples
+    --------
+    >>> pd.array(['This is', 'some text', None, 'data.'], dtype="arrow_string")


Suggested change

>>> pd.array(['This is', 'some text', None, 'data.'], dtype="arrow_string")

>>> pd.array(['This is', 'some text', None, 'data.'], dtype="string[arrow]")

jorisvandenbossche · 2020-09-07T06:48:12Z

pandas/core/arrays/string_arrow.py

+    def _from_sequence(cls, scalars, dtype=None, copy=False):
+        # TODO(ARROW-9407): Accept pd.NA in Arrow
+        scalars_corrected = [None if isna(x) else x for x in scalars]
+        return cls(pa.array(scalars_corrected, type=pa.string()))


Suggested change

return cls(pa.array(scalars_corrected, type=pa.string()))

return cls(pa.array(scalars_corrected, type=pa.string(), from_pandas=True))

then np.nan and pd.NA will be accepted as "null" as well, in addition to None

TomAugspurger · 2020-09-25T20:38:57Z

Closing in favor of #35259 (hopefully)

simonjayhawkins · 2020-10-19T15:32:42Z

@TomAugspurger @xhochy OK to merge these changes into #35259?

TomAugspurger · 2020-10-19T15:54:47Z

I'm not sure if we have complete agreement over how this will be exposed to the user yet in #35169. This PR was taking a strong opinion on

One user-facing pd.StringDtype[storage], parametrized by storage = python/pyarrow.
One or two (or three) user-facing StringArray classes, an ArrowStringArray, PythonStringArray, and maybe a base StringArray.

Given the uncertainty around the API here, I'd prefer to just move forward with an internals-only ArrowStringArray and decide on the exact API later (unless we can agree on something quickly).

simonjayhawkins · 2020-10-19T16:01:47Z

I'd prefer to just move forward with an internals-only ArrowStringArray

OK, i've just changed #35259 so that the doctest passed. I should probably revert that change and change the doctest so that ArrowStringDtype is not a top level import.

jorisvandenbossche · 2020-10-19T18:20:40Z

Personally, I think there is enough agreement to at least start making a concrete PR, which will more easily trigger further discussion on it.
(but let's bring this up again in the issue).

jorisvandenbossche · 2020-10-19T18:30:47Z

That said, I think it's certainly fine to focus on the actual implementation in #35259, and leave the user exposing API for a separate PR. There is already more than enough to review and discuss for the internal implementation details ;)

simonjayhawkins · 2021-02-19T13:48:10Z

I don't seem to able to push to this branch. will open a new PR

simonjayhawkins · 2021-02-19T14:05:49Z

I don't seem to able to push to this branch. will open a new PR

#39908

xhochy and others added 2 commits September 3, 2020 14:35

Implement BaseDtypeTests for ArrowStringDtype

4cb60e6

Refactor to use parametrized StringDtype

d242f2d

jorisvandenbossche reviewed Sep 7, 2020

View reviewed changes

jbrockmendel added the Strings String extension data type and string data label Sep 14, 2020

TomAugspurger closed this Sep 25, 2020

jorisvandenbossche mentioned this pull request Oct 19, 2020

Plan for a native string dtype #35169

Closed

simonjayhawkins reopened this Feb 19, 2021

simonjayhawkins closed this Feb 19, 2021

simonjayhawkins mentioned this pull request Feb 19, 2021

[ArrowStringArray] API: StringDtype parameterized by storage (python or pyarrow) #39908

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Arrow string array dtype #36142

Arrow string array dtype #36142

TomAugspurger commented Sep 5, 2020

TomAugspurger commented Sep 5, 2020

xhochy commented Sep 5, 2020

jorisvandenbossche left a comment

jorisvandenbossche Sep 7, 2020

jorisvandenbossche Sep 7, 2020

jorisvandenbossche Sep 7, 2020

TomAugspurger Sep 14, 2020

jorisvandenbossche Sep 7, 2020

jorisvandenbossche Sep 7, 2020

TomAugspurger commented Sep 25, 2020

simonjayhawkins commented Oct 19, 2020

TomAugspurger commented Oct 19, 2020

simonjayhawkins commented Oct 19, 2020

jorisvandenbossche commented Oct 19, 2020

jorisvandenbossche commented Oct 19, 2020

simonjayhawkins commented Feb 19, 2021

simonjayhawkins commented Feb 19, 2021

	return f"StringDtype[{self.storage}]"
	return f"string[{self.storage}]"

	>>> pd.array(['This is', 'some text', None, 'data.'], dtype="arrow_string")
	>>> pd.array(['This is', 'some text', None, 'data.'], dtype="string[arrow]")

	return cls(pa.array(scalars_corrected, type=pa.string()))
	return cls(pa.array(scalars_corrected, type=pa.string(), from_pandas=True))

Arrow string array dtype #36142

Arrow string array dtype #36142

Conversation

TomAugspurger commented Sep 5, 2020

TomAugspurger commented Sep 5, 2020

xhochy commented Sep 5, 2020

jorisvandenbossche left a comment

Choose a reason for hiding this comment

jorisvandenbossche Sep 7, 2020

Choose a reason for hiding this comment

jorisvandenbossche Sep 7, 2020

Choose a reason for hiding this comment

jorisvandenbossche Sep 7, 2020

Choose a reason for hiding this comment

TomAugspurger Sep 14, 2020

Choose a reason for hiding this comment

jorisvandenbossche Sep 7, 2020

Choose a reason for hiding this comment

jorisvandenbossche Sep 7, 2020

Choose a reason for hiding this comment

TomAugspurger commented Sep 25, 2020

simonjayhawkins commented Oct 19, 2020

TomAugspurger commented Oct 19, 2020

simonjayhawkins commented Oct 19, 2020

jorisvandenbossche commented Oct 19, 2020

jorisvandenbossche commented Oct 19, 2020

simonjayhawkins commented Feb 19, 2021

simonjayhawkins commented Feb 19, 2021