Improve Arrow-IPC performance by avoiding Unsafe Unchecked IPC Read RecordBatch

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

When transferring flatbuffers between trusted sources I would like a mechanism to elide costly verification of ArrayData contents.

Describe the solution you'd like

I would like a variant of read_record_batch_unchecked that performs the function of read_record_batch but without performing validation of the ArrayData.

Describe alternatives you've considered

Additional context

[totoroyyb]

Coming from #6933.

Would you think a separate API, as you suggested, would be good, or maybe provide an option for FileReader (or FileDecoder)?

[alamb]

Coming from #6933.

Would you think a separate API, as you suggested, would be good, or maybe provide an option for FileReader (or FileDecoder)?

I think an option would be good

[alamb]

Copying @totoroyyb 's high level usecase description from #6933

They report a 100x performance improvement when disabling data validation:

Describe your question
I am using high-level API (FileReader and FileDecoder) to read IPC files via mmap. I have noticed that validate_data() in the Array building process (here) adds significant overhead.

I am targeting an ultra-low-latency scenario. With validate_data I got 290ms for reading a 2.2GB IPC file (via mmap), and 3.8ms without validate_data, which I tested locally by commenting that out. 3.8ms latency is pretty much identical to c++ arrow implementation I tested, and I suspect c++ codebase didn't do this sanity check (not entirely sure).

The functions for the "unchecked" building are here in the codebase, but they are not accessible from high-level API, where I can easily disable them without creating my own array and everything on top of it.

I wonder if there is any better way to achieve that?

Additional context
Low latency is critical in my case. Thus, I am trying to avoid any additional overhead (C++ codebase as the baseline, maybe?)