Skip to content

Fix PandasArrayExtensionArray.take coercing fill_value when not filling - #8486

Draft
aryansk wants to merge 1 commit into
huggingface:mainfrom
aryansk:fix-fill-value-coercion-8465
Draft

Fix PandasArrayExtensionArray.take coercing fill_value when not filling#8486
aryansk wants to merge 1 commit into
huggingface:mainfrom
aryansk:fix-fill-value-coercion-8465

Conversation

@aryansk

@aryansk aryansk commented Aug 15, 2026

Copy link
Copy Markdown

Fixes #8465

Problem

PandasArrayExtensionArray.take coerces fill_value to the pandas dtype before checking whether any row is actually missing. When allow_fill=True with a mask that contains no missing indices, take still needs to return a proper pandas Series — but the coercion can blow up for values that don't fit the column dtype, e.g.:

import pandas as pd
from datasets.features.features import PandasArrayExtensionArray

arr = PandasArrayExtensionArray(pd.array(["a", "b", "c"], dtype=pd.StringDtype()))
arr.take([0, 1, 2], allow_fill=True, fill_value=False)  # conversion error even though nothing is missing

Change

Coerce fill_value to the pandas dtype only when filling is actually needed (a missing index was requested), and always return a proper pandas.Series from take — previously the no-fill path could return a bare np.ndarray, breaking downstream code that expects Series semantics (indexing by boolean mask etc.).

Testing

  • Regression test: take with allow_fill=True and a complete mask no longer errors on a non-matching fill_value, and the result matches a plain take.
  • Existing suite green.

fill_value was coerced to the pandas dtype before checking whether any
row was actually missing, so calling take() with a complete mask and a
fill_value that does not fit the dtype (e.g. False for a str column)
raised a conversion error. Coerce only when filling is actually needed,
and always return a proper pandas Series.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PandasArrayExtensionArray.take coerces fill_value before checking whether anything is filled, breaking boolean masks on integer array columns

1 participant