Repository navigation
ipc stream reader RecordBatchReader::schema ignores column projection #10676
Description
Describe the bug
Same as #10382 (csv) and #10613 (ipc file reader), but for the ipc stream reader.
When a projection is set, StreamReader::schema() and its RecordBatchReader::schema() impl return the full stream schema, while the batches the reader yields have the projected schema. #10627 fixed FileReader, StreamReader has the same problem.
StreamReader::try_new already computes the projected schema and stores it in the private projection field, but that value is never read back.
To Reproduce
// arrow-ipc/src/reader.rs (mirrors test_file_reader_projected_schema_matches_batch_schema)
#[test]
fn test_stream_reader_projected_schema_matches_batch_schema() {
let schema = create_test_projection_schema();
let batch = create_test_projection_batch_data(&schema);
let mut buf = Vec::new();
{
let mut writer = crate::writer::StreamWriter::try_new(&mut buf, &schema).unwrap();
writer.write(&batch).unwrap();
writer.finish().unwrap();
}
let projection = vec![3, 2, 1];
let mut reader = StreamReader::try_new(Cursor::new(buf), Some(projection)).unwrap();
let reader_schema = RecordBatchReader::schema(&reader);
let read_batch = reader.next().unwrap().unwrap();
assert_eq!(reader_schema, read_batch.schema());
}On eaf21ed this fails with a 14 column schema on the left (the full stream schema) and a 3 column schema on the right (the projected batch schema):
assertion `left == right` failed
left: Schema { fields: [f0 UInt32, f1 Utf8, f2 Boolean, f3 Union(..), f4 Null, f5 Float64,
f6 List, f7 FixedSizeBinary(3), f8 FixedSizeList, f9 Struct, f10 RunEndEncoded,
f11 Boolean, f12 Dictionary(Int8, Utf8), f13 Utf8], metadata: {} }
right: Schema { fields: [f3 Union(..), f2 Boolean, f1 Utf8], metadata: {} }
The same projection against FileReader passes, so the two IPC readers currently disagree.
Expected behavior
RecordBatchReader::schema should match the schema of the output batches
arrow-rs/arrow-array/src/record_batch.rs
Lines 27 to 36 in eaf21ed
| /// Trait for types that can read `RecordBatch`'s. | |
| /// | |
| /// To create from an iterator, see [RecordBatchIterator]. | |
| pub trait RecordBatchReader: Iterator<Item = Result<RecordBatch, ArrowError>> { | |
| /// Returns the schema of this `RecordBatchReader`. | |
| /// | |
| /// Implementation of this trait should guarantee that all `RecordBatch`'s returned by this | |
| /// reader should have the same schema as returned from this method. | |
| fn schema(&self) -> SchemaRef; | |
| } |
Additional context
Found by auditing the remaining RecordBatchReader impls after #10613 was fixed. This report was prepared with AI assistance; I ran the reproduction locally and reviewed the result.
Activity
apacheGH-10676: Fix IPC StreamReader projected schema (apache#10677)
apacheGH-10676: Fix IPC StreamReader projected schema (apache#10677)