Have read_json serialize directly to Arrow arrays

[WillAyd]

Feature Type

• Adding new functionality to pandas

• Changing existing functionality in pandas

• Removing existing functionality in pandas

Problem Description

Right now if you use dtype_backend=“pyarrow” with read_json you still first serialize to numpy arrays

Feature Description

I propose we vendor nanoarrow and use that directly to create arrow arrays in JSONToObj.c

I think this is also nice because pyarrow is currently limited to reading line delimited JSON. We have a unique advantage with the formats we already cover if we can just plug in nanoarrow to build the arrays

Alternative Solutions

N/a

Additional Context

Before tackling this #55102 would be very helpful

[lithomas1]

Any idea how much improvement this would give?

I thought copying from numpy was already zero copy in some cases. I'm not too hot about vendoring nanoarrow just for this,

[WillAyd]

Not sure about performance - I'm guessing there would be some but hard to say without writing parts of it.

I think where this is significantly better in the JSON space is support for arrays and structs, which are common in JSON but within our numpy type system just become object

[WillAyd]

@lithomas1 curious what are your concerns with nanoarrow

[lithomas1]

I think my main worry is stuff getting out of sync with the vendoring.
(I think with meson, though, we can just try to add it as a subproject, similar to how CMake has FetchContent.)

[WillAyd]

That's a cool idea. I don't know I'd consider that a dependency for this though - we could just not customize the library at all and update as needed (this is what arrow-ADBC does)