[Bug]: constructing a Categorical column from Python values raises on the PyArrow backend

[mkzung]

Tracking

• [Python] pa.array silently ignores unsigned dictionary index types apache/arrow#50474

Describe the bug

On the PyArrow backend, building a Categorical column straight from Python values raises. The same calls work on pandas and Polars, and casting an existing PyArrow column to Categorical already works, so it looks like only the construct-from-Python-values path is affected.

Steps or code to reproduce the bug

import narwhals as nw

# ArrowTypeError: Array chunks must all be same type
nw.new_series("s", ["a", "b", "a"], dtype=nw.Categorical, backend="pyarrow")

# ArrowInvalid: Column data for field 0 with type dictionary<values=string, indices=int32> ...
nw.from_dicts([{"a": "x"}, {"a": "y"}], schema={"a": nw.Categorical()}, backend="pyarrow")

For contrast, these all work:

nw.new_series("s", ["a", "b"], dtype=nw.Categorical, backend="pandas")               # ok
nw.new_series("s", ["a", "b"], dtype=nw.Categorical, backend="polars")               # ok
nw.from_dict({"a": ["x", "y"]}, schema={"a": nw.Categorical()}, backend="pyarrow")   # ok
nw.from_dict({"a": ["x", "y"]}, backend="pyarrow")["a"].cast(nw.Categorical)          # ok

Expected results

The Categorical column is built on PyArrow too, the same as on pandas and Polars.

Actual results

new_series raises ArrowTypeError: Array chunks must all be same type, and from_dicts raises ArrowInvalid: Column data for field 0 with type dictionary<...>. Categorical maps to pa.dictionary(uint32, string) and isn't in UNSUPPORTED_DTYPES; the arrow construction path (pa.chunked_array / pa.Table.from_pylist) can't build a dictionary-typed array straight from Python values.

This came up in #3757.

narwhals.show_versions()

Python dependencies:
     narwhals: 2.23.0
        numpy: 2.2.6
       pandas: 2.3.3
      pyarrow: 23.0.1
       polars: 1.42.1
         dask: 2026.7.0
       duckdb: 1.5.4
         ibis: 12.0.0
     sqlframe: 4.3.0

[dangotbanned]

I think this is a pyarrow bug and should be reported there before we start working around it.

@FBruzzesi, WDYT?

Minimal repro

import pyarrow as pa

arr = [["a", "b", "a"]]
dtype = pa.dictionary(pa.uint32(), pa.string())


print(pa.chunked_array(arr).cast(dtype))
print(pa.chunked_array(arr, dtype))

[

  -- dictionary:
    [
      "a",
      "b"
    ]
  -- indices:
    [
      0,
      1,
      0
    ]
]

---------------------------------------------------------------------------
ArrowTypeError                            Traceback (most recent call last)
Cell In[5], line 5
      1 arr = [["a", "b", "a"]]
      2 dtype = pa.dictionary(pa.uint32(), pa.string())
      3 
      4 print(pa.chunked_array(arr).cast(dtype))
----> 5 print(pa.chunked_array(arr, dtype))

File pyarrow/table.pxi:1558, in pyarrow.lib.chunked_array()
-> 1558 'Could not get source, probably due dynamically evaluated source code.'

File /pyarrow/error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()

File /pyarrow/error.pxi:92, in pyarrow.lib.check_status()

ArrowTypeError: Array chunks must all be same type

[mkzung]

Reported upstream: apache/arrow#50474

Digging into your repro, the cause looks narrower than "can't build a dictionary from Python values". pa.array does build one, it just silently ignores an unsigned index type and gives back the signed one:

>>> pa.array(["a", "b", "a"], type=pa.dictionary(pa.uint32(), pa.string())).type
dictionary<values=string, indices=int32, ordered=0>

int8/int16/int32/int64 are honoured, uint8/uint16/uint32/uint64 all come back signed. That is why chunked_array raises "Array chunks must all be same type" only for the unsigned case: the chunk it builds internally is int32-indexed, but the requested type is uint32. With a signed index type chunked_array(values, dtype) works, and .cast honours uint32 fine.

Since Categorical maps to dictionary(uint32, string), that is the specific thing tripping it here.

[mkzung]

Correction to the link above: #50474 turned out to be a duplicate of an older report, so I closed it and moved to apache/arrow#37476, which is where the fix is going. A PR against Arrow is on the way.

[mkzung]

The Arrow PR is up: apache/arrow#50475

It preserves the unsigned index type instead of quietly swapping it for the signed one, so pa.array and pa.chunked_array return the dtype you asked for. Arrow review is slow though, a comparable dict-builder PR sat for months, so I would not hold anything here waiting on it.

[mkzung]

The Arrow PR is up (apache/arrow#50475), but it won't help here for a while. narwhals supports pyarrow 13+, so even after it lands, older versions still break.

I think there's a simpler fix. We declare Categorical as dictionary(uint32, string), but pyarrow always builds int32 indices from Python values and won't reconcile the two. If we just declare int32 instead, new_series and from_dicts both work, on every pyarrow we support. No cast, so nothing breaks on old versions. chunked_array stays untouched and the schema isn't iterated, so both of your comments go away.

The only cost is 2.1B categories instead of 4.3B. Nothing else reads the index width.

I have it on a branch: one line, three tests, suite green on pyarrow 13 and 22. mkzung/narwhals@main...mkzung:narwhals:fix/categorical-int32-index

Want me to cut #3757 down to that, or would you rather wait for Arrow?