Error "Not all children array length are the same!" when decoding rows spanning across page boundaries in parquet file when using `RowSelection`

[jonded94]

Describe the bug

I'm trying to read a parquet file containing a large number of image bytes in a columns with MapType. Unfortunately, this leads to an error "Not all children array length are the same!", but only if I use a RowSelection! If I omit the RowSelection and let the file be traversed normally, my reproduction test succeeds.

To Reproduce

mod tests {
    use parquet::arrow::arrow_reader::{ArrowReaderBuilder, RowSelection, RowSelector};
    use std::fs::File;
    use std::path::PathBuf;

    #[test]
    fn validate_issue() {
        pub fn row_selection_from_indices(indices: &[usize]) -> RowSelection {
            let mut selectors = Vec::new();
            let mut last_end = 0;

            for &idx in indices {
                if idx > last_end {
                    selectors.push(RowSelector::skip(idx - last_end));
                }
                selectors.push(RowSelector::select(1));
                last_end = idx + 1;
            }

            selectors.into()
        }
        let indices = vec![352, 955];

        let arrow_reader = ArrowReaderBuilder::try_new(
            File::open(PathBuf::from(
                "issue_file.parquet",
            ))
            .unwrap(),
        )
        .unwrap();

        let mut batch_reader_builder = arrow_reader;
        batch_reader_builder = batch_reader_builder.with_row_groups(vec![99]);
        batch_reader_builder =
            batch_reader_builder.with_row_selection(row_selection_from_indices(indices.as_slice()));  // Removing this lets the test suceed again!

        let batch_reader = batch_reader_builder.build().unwrap();

        for item in batch_reader {
            item.unwrap();
        }    
    }
}

=> (the debug statements were added my be)

[.../arrow-rs/parquet/src/arrow/array_reader/struct_array.rs:120:9] children_array_len = 3
[.../arrow-rs/parquet/src/arrow/array_reader/struct_array.rs:121:9] children_array.iter().map(|arr| arr.len()).collect::<Vec<_>>() = [
    3,
    3,
    3,
    3,
    3,
    3,
    3,
]
[.../arrow-rs/parquet/src/arrow/array_reader/struct_array.rs:120:9] children_array_len = 3
[.../arrow-rs/parquet/src/arrow/array_reader/struct_array.rs:121:9] children_array.iter().map(|arr| arr.len()).collect::<Vec<_>>() = [
    3,
    3,
]
[.../arrow-rs/parquet/src/arrow/array_reader/struct_array.rs:120:9] children_array_len = 2
[.../arrow-rs/parquet/src/arrow/array_reader/struct_array.rs:121:9] children_array.iter().map(|arr| arr.len()).collect::<Vec<_>>() = [
    2,  // <-- Only the first array seems to be of length 2, all others have length 3
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
    3,
]
called `Result::unwrap()` on an `Err` value: ParquetError("Parquet error: Not all children array length are the same!")

Expected behavior
Iteration through the parquet file should work without problem, regardless of whether somebody uses RowSelection or not.

Additional context
Happens with arrow-rs 57.1.0, 57.2.0 and in this specific report I used commit fb77501.

I unfortunately can't give you the reproduction file, as it contains tons of confidential stuff, but I shared as much parquet-viewer output as possible. Most probably this is about the image_data map, specifically the image_bytes values column?

[jonded94]

I'm sadly not too well versed around the internals of arrow-rs, but let me paste what Claude Code vibed for me (sorry if you don't like this spam 😢).

This is the debug summary:

  Root Cause                                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                          
  The bug occurs when using RowSelection with nested types (like List<String>) when:                                                                                                                                                                                                      
  1. A column has multiple pages in a row group                                                                                                                                                                                                                                           
  2. The selected rows span across page boundaries                                                                                                                                                                                                                                        
  3. The first page is entirely consumed during skip operations                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                          
  Specific Scenario                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                          
  In the test case:                                                                                                                                                                                                                                                                       
  - Row group 99 has 5120 rows                                                                                                                                                                                                                                                            
  - Column l1.element (first list column) has only 13711 values in its first page, while other element columns have 224796 values per page                                                                                                                                                
  - Selected rows are [2, 955]                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                          
  The sequence that triggers the bug:                                                                                                                                                                                                                                                     
  1. Skip 2 rows: First column consumes 60 of 13711 values                                                                                                                                                                                                                                
  2. Read 1 row (row 2): Returns 6 list elements correctly                                                                                                                                                                                                                                
  3. Skip 952 rows: First column consumes ALL remaining 13651 values (13711 total)                                                                                                                                                                                                        
  4. Read 1 row (row 955):                                                                                                                                                                                                                                                                
    - has_next() reads a new page (12834 values)                                                                                                                                                                                                                                          
    - BUT GenericColumnReader::read_records returns (1, 0, 0) - 1 record with 0 levels!                                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                          
  Why This Happens                                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                          
  The issue is in how the repetition level decoder's flush_partial() interacts with page boundaries:                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                          
  has_next [element] -> needs new page, buffered=13711, decoded=13711                                                                                                                                                                                                                     
  has_next [element] -> new page read, buffered=12834                                                                                                                                                                                                                                     
  GenericColumnReader::read_records EXIT [element] -> (1, 0, 0)                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                          
  After reading a new page, the read_rep_levels function returns 0 levels but the flush_partial() mechanism adds 1 record. This creates a situation where 1 record is "counted" without any actual data being read from the new page.                                                     
                                                                                                                                                                                                                                                                                          
  Result                                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                          
  - First list column (l1): Returns 6 values with 1 record boundary (rep_level=0)                                                                                                                                                                                                         
  - Other list columns: Return 273 values with 2 record boundaries                                                                                                                                                                                                                        
  - This mismatch causes the "Not all children array length are the same!" error at struct_array.rs:127                                                                                                                                                                                   
                                                                                                                                                                                                                                                                                          
  Affected Code Locations                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                          
  1. /parquet/src/column/reader.rs - GenericColumnReader::read_records and skip_records                                                                                                                                                                                                   
  2. /parquet/src/column/reader/decoder.rs - RepetitionLevelDecoderImpl::flush_partial                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                          
  The bug is specifically in how has_partial state is managed across page boundaries when using row selection. When a page is exhausted during skip and a new page is read for subsequent read operations, the record counting logic can incorrectly "complete" a record without actually 
  reading any data.                         

Debug output:

claude-debug-output.txt

Patch for enabling all the debug output:

claude-debug-statements.patch

[jonded94]

I opened a PR that seems to fix this issue: #9374