Skip to content

fix(arrow-avro): decode records with no fields - #10771

Merged
Jefffrey merged 2 commits into
apache:mainfrom
sqd:avro-empty-record-decode
Sep 2, 2026
Merged

Jefffrey merged 2 commits into
apache:mainfrom
sqd:avro-empty-record-decode

Conversation

@sqd

@sqd sqd commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

An Avro record with no fields is legal, and encodes to zero bytes. The reader accepts such a schema and decodes its (empty) bytes fine, but fails at flush:

Invalid argument error: use StructArray::try_new_with_length or
StructArray::new_empty_fields to create a struct array with no
fields so that the length can be set correctly

Decoder::Record flushes through StructArray::try_new, which infers its length from the first child array, but a zero-field record has no child array.

RecordDecoder::flush (the top level) already handles this: it passes its own row_count through RecordBatchOptions::with_row_count. This change gives the nested case the same treatment. Decoder::Record now carries a row count, incremented wherever a row is appended and reset at flush.

Which issue does this PR close?

Are these changes tested?

See included tests.

Are there any user-facing changes?

No

@github-actions github-actions Bot added arrow Changes to the arrow crate arrow-avro arrow-avro crate labels Aug 20, 2026
Comment thread arrow-avro/src/reader/record.rs Outdated
Vec<Decoder>,
Vec<Option<AvroLiteral>>,
Option<Projector>,
usize,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps we should name the fields, now that we have many fields

Record {
    fields: Fields,
    decoders: Vec<Decoder>,
    defaults: Vec<Option<AvroLiteral>>,
    projector: Option<Projector>,
    /// Useful when there are no fields (legal edge case)
    row_count: usize
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the review! The fields are now named.

Comment thread arrow-avro/src/reader/record.rs Outdated
@Jefffrey Jefffrey added the bug label Aug 21, 2026
Han You added 2 commits September 1, 2026 22:11
An Avro record with no fields is legal, and encodes to zero bytes. The
reader accepts such a schema and decodes its (empty) bytes fine, but
fails at flush:

    Invalid argument error: use StructArray::try_new_with_length or
    StructArray::new_empty_fields to create a struct array with no
    fields so that the length can be set correctly

`Decoder::Record` flushes through `StructArray::try_new`, which infers
its length from the first child array, but a zero-field record has no
child array.

`RecordDecoder::flush` (the top level) already handles this: it passes
its own `row_count` through `RecordBatchOptions::with_row_count`. This
change gives the nested case the same treatment.  `Decoder::Record`
now carries a row count, incremented wherever a row is appended and
reset at `flush`.
@sqd
sqd force-pushed the avro-empty-record-decode branch from b183083 to 64a6625 Compare September 1, 2026 21:14
@Jefffrey
Jefffrey merged commit bd4d32b into apache:main Sep 2, 2026
33 checks passed
@Jefffrey

Jefffrey commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

thanks @sqd

Rich-T-kid pushed a commit to Rich-T-kid/arrow-rs that referenced this pull request Sep 18, 2026
An Avro record with no fields is legal, and encodes to zero bytes. The
reader accepts such a schema and decodes its (empty) bytes fine, but
fails at flush:

    Invalid argument error: use StructArray::try_new_with_length or
    StructArray::new_empty_fields to create a struct array with no
    fields so that the length can be set correctly

`Decoder::Record` flushes through `StructArray::try_new`, which infers
its length from the first child array, but a zero-field record has no
child array.

`RecordDecoder::flush` (the top level) already handles this: it passes
its own `row_count` through `RecordBatchOptions::with_row_count`. This
change gives the nested case the same treatment. `Decoder::Record` now
carries a row count, incremented wherever a row is appended and reset at
`flush`.

# Which issue does this PR close?

- Closes apache#10770.

# Are these changes tested?

See included tests.

# Are there any user-facing changes?

No

---------

Co-authored-by: Han You <han.you@imc.com>
Rich-T-kid pushed a commit to Rich-T-kid/arrow-rs that referenced this pull request Sep 21, 2026
An Avro record with no fields is legal, and encodes to zero bytes. The
reader accepts such a schema and decodes its (empty) bytes fine, but
fails at flush:

    Invalid argument error: use StructArray::try_new_with_length or
    StructArray::new_empty_fields to create a struct array with no
    fields so that the length can be set correctly

`Decoder::Record` flushes through `StructArray::try_new`, which infers
its length from the first child array, but a zero-field record has no
child array.

`RecordDecoder::flush` (the top level) already handles this: it passes
its own `row_count` through `RecordBatchOptions::with_row_count`. This
change gives the nested case the same treatment. `Decoder::Record` now
carries a row count, incremented wherever a row is appended and reset at
`flush`.

# Which issue does this PR close?

- Closes apache#10770.

# Are these changes tested?

See included tests.

# Are there any user-facing changes?

No

---------

Co-authored-by: Han You <han.you@imc.com>
Rich-T-kid pushed a commit to Rich-T-kid/arrow-rs that referenced this pull request Sep 25, 2026
An Avro record with no fields is legal, and encodes to zero bytes. The
reader accepts such a schema and decodes its (empty) bytes fine, but
fails at flush:

    Invalid argument error: use StructArray::try_new_with_length or
    StructArray::new_empty_fields to create a struct array with no
    fields so that the length can be set correctly

`Decoder::Record` flushes through `StructArray::try_new`, which infers
its length from the first child array, but a zero-field record has no
child array.

`RecordDecoder::flush` (the top level) already handles this: it passes
its own `row_count` through `RecordBatchOptions::with_row_count`. This
change gives the nested case the same treatment. `Decoder::Record` now
carries a row count, incremented wherever a row is appended and reset at
`flush`.

# Which issue does this PR close?

- Closes apache#10770.

# Are these changes tested?

See included tests.

# Are there any user-facing changes?

No

---------

Co-authored-by: Han You <han.you@imc.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate arrow-avro arrow-avro crate bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

arrow-avro cannot flush records with no fields

2 participants