API/ENH: dtype='string' / pd.String

[jreback]

update for 2019-10-07: We have a StringDtype extension dtype. It's memory model is the same as the old implementation, an object-dtype ndarray of strings. The next step is to store & process it natively.

---

xref #8627
xref #8643, #8350

Since we introduced Categorical in 0.15.0, I think we have found 2 main uses.

1. as a 'real' Categorical/Factor type to represent a limited of subset of values that the column can take on
2. as a memory saving representation for object dtypes.

I could see introducting a dtype='string' where String is a slightly specialized sub-class of Categroical, with 2 differences compared to a 'regular' Categorical:

• it allows unions of arbitrary other string types, currently Categorical will complain if you do this:

In [1]: df = DataFrame({'A' : Series(list('abc'),dtype='category')})
In [2]: df2 = DataFrame({'A' : Series(list('abd'),dtype='category')})
In [3]: pd.concat([df,df2])
ValueError: incompatible levels in categorical block merge

Note that this works if they are Series (and prob should raise as well, side -issue)

But, if these were both 'string' dtypes, then its a simple matter to combine (efficiently).

• you can restrict the 'sub-dtype' (e.g. the dtype of the categories) to string/unicode (iow, don't allow numbers / arbitrary objects), makes the constructor a bit simpler, but more importantly, you now have a 'real' non-object string dtype.

I don't think this would be that complicated to do. The big change here would be to essentially convert any object dtypes that are strings to dtype='string' e.g. on reading/conversion/etc. might be a perf issue for some things, but I think the memory savings greatly outweigh.

We would then have a 'real' looking object dtype (and object would be relegated to actual python object types, so would be used much less).

cc @shoyer
cc @JanSchulz
cc @jorisvandenbossche
cc @mwiebe
thoughts?

[jorisvandenbossche]

I think it would be a very nice improvement to have a real 'string' dtype in pandas.
So no longer having the confusion in pandas of object dtype being actually in most cases a string, and sometimes a 'real' object.

However, I don't know if this should be 'coupled' to categorical. Maybe that is only a technical implementation detail, but for me it should just be a string dtype, a dtype that holds string values, and has in essence nothing to do with categorical.

If I think about a string dtype, I am more thinking about numpy's strings types (but it has of course also impractialities, that is has fixed sizes), or the CHAR/VARCHAR in sql.

[shoyer]

I'm of two minds about this. This could be quite useful, but on the other hand, it would be way better if this could be done upstream in numpy or dynd. Pandas specific array types are not great for compatibility with the broader ecosystem.

I understand there are good reasons it may not be feasible to implement this upstream (#8350), but these solutions do feel very stop-gap. For example, if @teoliphant is right that dynd could be hooked up in the near future to replace numpy in pandas internals, I would be much more excited about exploring that possibility.

As for this specific proposal:

1. Would we really use this in place of object dtype for almost all string data in pandas? If so, this needs to meet a much higher standard than if it's merely an option.
2. It would be premature to call this the dtype "string" rather than "interned_string", unless we're sure interning is always a good idea. Also, libraries like dynd do implement a true variable length string type (unlike numpy), and I think it is a good long term goal to align pandas dtypes with dtypes on the ndarray used for storage.
3. The worst of the performance consequences might be avoided if we do not guarantee that the string "categories" are unique. Otherwise every str op requires a call to factorize.
4. Especially if this is the default/standard, I really think we should try to make it work for N-dimensional data (I still need to finish up my patch for categorical).

[jreback]

So I have tagged a related issue, about including integer NA support by using libdynd (#8643). This will actuall be the first thing I do. (as its new and cool, and I think a slightly more straightforward path to include dynd as an optional dep).

@mwiebe

can you maybe explain a bit about the tradeoffs involved with representing strings in 2 ways using libdynd

• as a libdynd categorical (like proposing above but using the native categorical type which DOES exist in libdynd currently)
• as vlen strings (another libdynd feature that DOES exist).

cc @teoliphant

[mwiebe]

I've had in mind an intention to tweak the string representation in dynd slightly, and have written that up now. libdynd/libdynd#158 The vlen string in dynd does work presently, but it has slightly different properties than what I'm writing here.

Properties that this vlen string has are a 16 byte representation, using the small string optimization. This means strings with size <= 15 bytes encoded as utf-8 will fit in that memory. Bigger strings will involve a dynamic memory allocation per string, a little bit like Python's string, but with the utf-8 encoding and knowledge that it is a string instead of having to go through dynamic dispatch like in numpy object arrays of strings.

Representing strings as a dynd categorical is a bit more complicated, and wouldn't be dynamically updatable in the same way. The types in dynd are immutable, so a categorical type, once created, has a fixed memory layout, etc. This allows for optimized storage, e.g. if the total number of categories is <= 256, each element can be stored as one byte in the array, but does not allow the assignment of a new string that was not already a in the array of categories.

[jankatins]

The issue mentioned in the last comment is now at libdynd/libdynd#158