Skip to content

[rust] Wire batch statistics into the log batch write and read paths - #4074

Open
leekeiabstraction wants to merge 1 commit into
apache:mainfrom
leekeiabstraction:rust-batch-stats-wiring
Open

[rust] Wire batch statistics into the log batch write and read paths#4074
leekeiabstraction wants to merge 1 commit into
apache:mainfrom
leekeiabstraction:rust-batch-stats-wiring

Conversation

@leekeiabstraction

@leekeiabstraction leekeiabstraction commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Wires the V1 batch statistics collection ([rust] Add V1 batch statistics collection #4011) into the write and read paths. Closes [rust] Batch statistics read wiring #4009, closes [rust] Batch statistics write wiring #4010.
  • Write: tables with table.statistics.columns set emit V1 batches carrying the serialized statistics between the header and the records data, mirroring Java's MemoryLogRecordsArrowBuilder. A serialization failure degrades to an empty statistics section rather than failing the batch.
  • Read: all three magic versions decode — V1 skips the statistics section, V2 additionally reads the leader epoch and shifts the post-CRC header fields, and unknown magics are rejected. This fixes the Rust reader misparsing V1/V2 batches that Java clients and the server already produce; the writer stays on V0/V1 like Java's client-side builder.
  • Refactoring: the Fluss batch framing moved out of arrow.rs into a new log_record_batch.rs, separating the Fluss wire format from Arrow IPC concerns. The Arrow builder's long argument lists collapsed into an ArrowBatchConfig struct.

Test Plan

  • A cross-language golden fixture (fluss-common/src/test/resources/encoding/statistics_block.hex) pins the statistics block byte-for-byte across all supported types, nulls, both decimal widths, both timestamp precisions, and float NaN/signed-zero ordering (bounds follow Java's Float.compare semantics, not Arrow's IEEE totalOrder). A new Java test generates it, asserts it and parses it back verifying the decoded bounds; the Rust test reads the same file.
  • New unit tests cover V1/V2 encode/decode round trips (append-only and changelog), empty, corrupt and truncated batches, leader epoch reads, unsupported magic, per-column degradation for unsupported column types and the size reserve. cargo test -p fluss-rs --lib passes 707 tests; clippy and fmt are clean; the end-to-end integration test is left to [rust] Filter pushdown integration test #3845.

🤖 AI-assisted changes - reviewed by human developer

@leekeiabstraction
leekeiabstraction force-pushed the rust-batch-stats-wiring branch 10 times, most recently from e33bb99 to c09cb91 Compare August 23, 2026 16:19
Tables with 'table.statistics.columns' set now produce V1 batches carrying
serialized statistics, matching Java's MemoryLogRecordsArrowBuilder. The
reader now locates the records data per the batch's magic, fixing the
misparsing of V1 batches that Java clients and the server's projection
path already produce.
@leekeiabstraction

Copy link
Copy Markdown
Contributor Author

@fresh-borzoni @platinumhamburg @charlesdong1991 Appreciate a review here 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[rust] Batch statistics write wiring [rust] Batch statistics read wiring

1 participant