[Python] Unable to import arrow table to pandas if it has categorical columns with index types of unsigned ints

[dweih]

Describe the bug, including details regarding any error messages, version, and platform.

Our code primarily uses polars but external tools use pandas, and when we use them to import parquet files with categorical columns that have unsigned int index types (uint16 and uint32) we get the error

ArrowTypeError: Converting unsigned ddictionary indices to pandas not yet supported, index type: uint32

Simple repro below.

import polars as pl
import pyarrow as pa

n = 100
cat_values = [f"cat_{i}" for i in range(n)]
df = pl.DataFrame({
    "cat": cat_values,
    "val": list(range(n))
})
arrow_table = df.to_arrow()

dict_type = pa.dictionary(index_type=pa.uint16(), value_type=pa.string())
arrow_table = arrow_table.set_column(
    arrow_table.schema.get_field_index("cat"),
    "cat",
    arrow_table.column("cat").cast(dict_type)
)

print("Arrow schema:", arrow_table.schema)


try:
    pdf = pl.from_table(arrow_table).to_pandas()
    pdf = arrow_table.to_pandas()
    print("Loaded into pandas successfully.")
except Exception as e:
    print("Failed to load into pandas:")
    print(e)

try:
    pol_df = pl.from_arrow(arrow_table)
    print("Loaded into Polars successfully.")
except Exception as e:
    print("Failed to load into Polars:")
    print(e)

Finally, I wasn't sure whether to make this a FR or Issue, because it's missing, not incorrect.

Component(s)

Python

[raulcd]

Hi, thanks for opening the issue!
Could you share the versions of the different libraries you are using?
Can you also share the stack trace and not just a single error line, I can't seem to find the error msg on our code base and the typo on ddictionary should be obvious to find.

I tries to reproduce and the polars API doesn't seem to be the same that's why I am asking for versions:

In [1]: import polars as pl
   ...: import pyarrow as pa
   ...: 
   ...: n = 100
   ...: cat_values = [f"cat_{i}" for i in range(n)]
   ...: df = pl.DataFrame({
   ...:     "cat": cat_values,
   ...:     "val": list(range(n))
   ...: })
   ...: arrow_table = df.to_arrow()
   ...: 
   ...: dict_type = pa.dictionary(index_type=pa.uint16(), value_type=pa.string())
   ...: arrow_table = arrow_table.set_column(
   ...:     arrow_table.schema.get_field_index("cat"),
   ...:     "cat",
   ...:     arrow_table.column("cat").cast(dict_type)
   ...: )
   ...: 
   ...: print("Arrow schema:", arrow_table.schema)
Arrow schema: cat: dictionary<values=string, indices=uint16, ordered=0>
val: int64

In [2]: pdf = pl.from_table(arrow_table).to_pandas()
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], line 1
----> 1 pdf = pl.from_table(arrow_table).to_pandas()

File ~/code/venv/lib/python3.12/site-packages/polars/__init__.py:479, in __getattr__(name)
    476     return getattr(dtgroup, name)
    478 msg = f"module {__name__!r} has no attribute {name!r}"
--> 479 raise AttributeError(msg)

AttributeError: module 'polars' has no attribute 'from_table'

[dweih]

Sorry! Typo, I was trying something else and must have copied from the wrong place...

import polars as pl
import pyarrow as pa

n = 100
cat_values = [f"cat_{i}" for i in range(n)]
df = pl.DataFrame({
    "cat": cat_values,
    "val": list(range(n))
})
arrow_table = df.to_arrow()

dict_type = pa.dictionary(index_type=pa.uint16(), value_type=pa.string())
arrow_table = arrow_table.set_column(
    arrow_table.schema.get_field_index("cat"),
    "cat",
    arrow_table.column("cat").cast(dict_type)
)

print("Arrow schema:", arrow_table.schema)


try:
    pdf = arrow_table.to_pandas()
    print("Loaded into pandas successfully.")
except Exception as e:
    print("Failed to load into pandas:")
    print(e)

try:
    pol_df = pl.from_arrow(arrow_table)
    print("Loaded into Polars successfully.")
except Exception as e:
    print("Failed to load into Polars:")
    print(e)

[antonioalegria]

Same issue

[raulcd]

Yes, I can see this is indeed not supported:

arrow/python/pyarrow/src/arrow/python/arrow_to_pandas.cc

Lines 2026 to 2032 in bb33493

	case Type::UINT8:
	case Type::UINT16:
	case Type::UINT32:
	case Type::UINT64:
	return Status::TypeError(
	"Converting unsigned dictionary indices to pandas",
	" not yet supported, index type: ", index_type.ToString());

@WillAyd do you know if there's any reason for arrow to pandas to not support this? or is just a matter of someone to do the work? I am putting the full traceback because it's easier to understand what happens.

>>> pdf = arrow_table.to_pandas()
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    pdf = arrow_table.to_pandas()
  File "pyarrow/array.pxi", line 987, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 5174, in pyarrow.lib.Table._to_pandas
  File "/home/raulcd/miniforge3/envs/py13/lib/python3.13/site-packages/pyarrow/pandas_compat.py", line 808, in table_to_dataframe
    result = pa.lib.table_to_blocks(options, table, categories,
                                    list(ext_columns_dtypes.keys()))
  File "pyarrow/table.pxi", line 4103, in pyarrow.lib.table_to_blocks
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowTypeError: Converting unsigned dictionary indices to pandas not yet supported, index type: uint16

cc @AlenkaF

[WillAyd]

I don't know the exact history of this in Arrow, but my assumption is this raises because pandas requires signed integer codes for its categorical values, as the sentinel -1 is used to denote missing values.

So it seems possible to support, but would likely not preserve the unsigned nature and in some cases may require an upcast of the index type to avoid overflows

[AlenkaF]

This is the PR that implemented this logic and it has a note about the unsigned indices: #7659. I wasn't able to find a follow-up issue though.

[raulcd]

@AlenkaF @WillAyd thanks for the investigation and context!
I suppose it requires someone with time to work on this :)

[fingoldo]

This problem is actually a real deal-breaker. Can someone work on a fix, please?