DOC: add pandas 3.0 migration guide for the string dtype #61705
+273
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jorisvandenbossche
added
Docs
Strings
There was a problem hiding this comment.
@jorisvandenbossche i'll post these few now rather than doing too many in a batch, but feel free to wait until i'm done, whatever is more convenient for you.
|
|
||
| Historically, pandas has always used the NumPy ``object`` dtype as the default | ||
| to store text data. This has two primary drawbacks. First, ``object`` dtype is | ||
| not specific to strings: any Python object can be stored in an ``object```-dtype |
There was a problem hiding this comment.
Suggested change
| not specific to strings: any Python object can be stored in an ``object```-dtype | |
| not specific to strings: any Python object can be stored in an ``object``-dtype |
| not specific to strings: any Python object can be stored in an ``object```-dtype | ||
| array, not just strings, and seeing ``object`` as the dtype for a column with | ||
| strings is confusing for users. Second, this is not always very efficient (both | ||
| performance wise as for memory usage). |
There was a problem hiding this comment.
Suggested change
| performance wise as for memory usage). | |
| performance wise and for memory usage). |
| not yet been made the default, and uses the ``pd.NA`` scalar to represent | ||
| missing values. | ||
|
|
||
| Pandas 3.0 changes the default dtype for strings to a new string data type, |
There was a problem hiding this comment.
Suggested change
| Pandas 3.0 changes the default dtype for strings to a new string data type, | |
| Pandas 3.0 changes the default inferred dtype for strings to a new string data type, |
| missing values. | ||
|
|
||
| Pandas 3.0 changes the default dtype for strings to a new string data type, | ||
| a variant of the existing optional string data type but using ``NaN`` as the |
There was a problem hiding this comment.
Suggested change
| a variant of the existing optional string data type but using ``NaN`` as the | |
| a variant of the existing optional string data type but using ``np.NaN`` as the |
|
|
||
| pd.options.future.infer_string = True | ||
|
|
||
| This allows to test your code before the final 3.0 release. |
There was a problem hiding this comment.
Suggested change
| This allows to test your code before the final 3.0 release. | |
| This allows you to test your code before the final 3.0 release. |
| .. - Breaking changes: | ||
| .. - dtype is no longer object dtype | ||
| .. - None gets coerced to NaN | ||
| .. - setitem raises an error for non-string data |
There was a problem hiding this comment.
the above is not rendered?
| .. - None gets coerced to NaN | ||
| .. - setitem raises an error for non-string data | ||
|
|
||
| Brief intro to the new default string dtype |
There was a problem hiding this comment.
Suggested change
| Brief intro to the new default string dtype | |
| Brief introduction to the new default string dtype |
| 2 NaN | ||
| dtype: str | ||
|
|
||
| In contrast the the current object dtype, the new string dtype will only store |
There was a problem hiding this comment.
Suggested change
| In contrast the the current object dtype, the new string dtype will only store | |
| In contrast to the current object dtype, the new string dtype will only store |
| non-string value in it (see below for more details). | ||
|
|
||
| Missing values with the new string dtype are always represented as ``NaN``, and | ||
| the missing value behaviour is similar as for other default dtypes. |
There was a problem hiding this comment.
Suggested change
| the missing value behaviour is similar as for other default dtypes. | |
| the missing value behavior is similar as for other default dtypes. |
do we use US English in the docs?
| non-string value in it (see below for more details). | ||
|
|
||
| Missing values with the new string dtype are always represented as ``NaN``, and | ||
| the missing value behaviour is similar as for other default dtypes. |
There was a problem hiding this comment.
Suggested change
| the missing value behaviour is similar as for other default dtypes. | |
| the missing value behaviour is similar to other default dtypes. |
| Missing values with the new string dtype are always represented as ``NaN``, and | ||
| the missing value behaviour is similar as for other default dtypes. | ||
|
|
||
| For the rest, this new string dtype should work the same as how you have been |
There was a problem hiding this comment.
Suggested change
| For the rest, this new string dtype should work the same as how you have been | |
| This new string dtype should work the same as how you have been |
|
|
||
| >>> ser.dtype == "str" | ||
|
|
||
| **How to write compatible code?** |
There was a problem hiding this comment.
Suggested change
| **How to write compatible code?** | |
| **How to write compatible code** |
| >>> pd.api.types.is_string_dtype(ser.dtype) | ||
| True | ||
|
|
||
| This will return ``True`` for both the object dtype as for the string dtypes. |
There was a problem hiding this comment.
Suggested change
| This will return ``True`` for both the object dtype as for the string dtypes. | |
| This will return ``True`` for both the object dtype and the string dtypes. |
| True | ||
|
|
||
| One caveat: this function works both on scalars and on array-likes, and in the | ||
| latter case it will return an array of boolean dtype. When using it in a boolean |
There was a problem hiding this comment.
Suggested change
| latter case it will return an array of boolean dtype. When using it in a boolean | |
| latter case it will return an array of Boolean dtype. When using it in a Boolean |
not to confuse with pandas nullable type should capitalize as named after George Boole?
| .. code-block:: python | ||
|
|
||
| >>> ser = pd.Series(["a", "b", None], dtype="str") | ||
| >>> ser[1] = 2.5 |
There was a problem hiding this comment.
i notice you can do ser[1] = pd.NA so we are accepting this as a missing value. Should we disallow this or perhaps encourage it instead to perhaps make migration to the pd.NA variant simpler?
| **How to write compatible code?** | ||
|
|
||
| You can update your code to ensure you only set string values in such columns, | ||
| or otherwise you have explicitly ensure the column has object dtype first. This |
There was a problem hiding this comment.
Suggested change
| or otherwise you have explicitly ensure the column has object dtype first. This | |
| or otherwise you can explicitly ensure the column has object dtype first. This |
| >>> ser[1] = 2.5 | ||
|
|
||
| This ``astype("object")`` call will be redundant when using pandas 2.x, but | ||
| this way such code can work for all versions. |
There was a problem hiding this comment.
Suggested change
| this way such code can work for all versions. | |
| this code will work for all versions. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR starts adding a migration guide with some typical issues one might run into regarding the new string dtype when upgrading to pandas 3.0 (or when enabling it in pandas 2.3).
(for now I just added it to the user guide, which is already a long list of pages, so we might need to think about better organizing this or putting it elsewhere)