Repository navigation
A delimited record keeps its empty fields and its own row - #1035
Conversation
Signed-off-by: dada-yan <BinjunYann@gmail.com>
WaylandYang
left a comment
WaylandYang
left a comment
There was a problem hiding this comment.
Thanks @Maya-Kid. Reviewed at ab6273a, and the change is right: a delimited file is records, and an empty field is a missing value, not a section. The tests pin rows and columns exactly.
One thing before I merge: docs/design/sources.md still says CSV files are laid on the same grid as DOCX tables and spreadsheets (#750) and that a CSV's first row is promoted by that rule. With this change CSV and TSV go through render_records instead. Please update that sentence in this PR.
…ign doc says CSV is read as records The conflict with deeplethe#1043 is in csv_text: the record renderer from this branch and the capped-records warning from dev are both kept. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Wayland Yang <wayland0916@gmail.com>
WaylandYang
left a comment
WaylandYang
left a comment
There was a problem hiding this comment.
Thanks @Maya-Kid. To get this in without another round, I pushed one merge commit (b64adc7) to your branch:
- Merged dev and resolved the conflict with #1043 in
csv_text: your record renderer and the capped-records warning are both kept. - Updated the sentence in
docs/design/sources.md: a CSV or TSV file is read as records, its first record is the header, and an empty field is a missing value.
The ingest tests pass, including yours and the truncation tests. Landing it once CI passes.
2 participants
CSV and TSV records are passed through the financial-table classifier. A missing email in
name,email\nAlice,\nBob,bob@example.comturns Alice into a section label and renders a singleAlice › Bobrow with Bob's email. Other records can lose their field boundaries when text columns or standalone symbols are combined.Delimited files now use a record renderer: the first record supplies the header, each subsequent nonempty record keeps its own row and field positions, short records are padded, and wider records retain their extra fields. Six regression tests cover sparse records, CSV/TSV parity, single and multiple columns, empty headers, variable widths, symbols, and quoted text. The new regressions fail on the previous code and pass with this change.
Validation:
cargo fmt --all --checkcargo clippy --workspace --all-targets -- -D warningscargo test -p utopia-ingest: 129 tests passed, including the six new regressions and the existing Office/HTML fixtures.cargo test --workspace: passed without a database configured; database-backed tests skipped. Poppler was required and available.Local Rust validation used the installed 1.98.1 toolchain.