Refactor arrow-ipc: Rename ArrayReader to RecodeBatchDecoder
#7028
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Jan 27, 2025
alamb
force-pushed
the
alamb/ipc_reader_refactor2
branch
from
d3265f6 to
4838002
Compare
alamb
commented
Jan 27, 2025
| @@ -65,7 +65,7 @@ fn read_buffer( | |||
| (false, Some(decompressor)) => decompressor.decompress_to_buffer(&buf_data), | |||
| } | |||
| } | |||
| impl ArrayReader<'_> { | |||
| impl RecordBatchDecoder<'_> { | |||
There was a problem hiding this comment.
The major change is to rename this per @tustvold 's suggestion:
| @@ -83,18 +83,17 @@ impl ArrayReader<'_> { | |||
| field: &Field, | |||
| variadic_counts: &mut VecDeque<i64>, | |||
| ) -> Result<ArrayRef, ArrowError> { | |||
| let reader = self; | |||
There was a problem hiding this comment.
removed reader alias as well
tustvold
approved these changes
Jan 28, 2025
|
Thanks again @tustvold |
phillipleblanc
pushed a commit
to spiceai/arrow-rs
that referenced
this pull request
Feb 10, 2025
…che#7028) * Rename `ArrayReader` to `RecordBatchDecoder` * Remove alias for `self`
alamb
added a commit
that referenced
this pull request
Feb 12, 2025
…e32 (#7074) * Support converting large dates (i.e. +10999-12-31) from string to Date32 * Fix lint * Update arrow-cast/src/parse.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * fix: issue introduced in #6833 - less than equal check for scale in decimal conversion (#7070) * fix <= check for scale in decimal conversion * Update arrow-cast/src/cast/mod.rs name change Co-authored-by: Arttu <Blizzara@users.noreply.github.com> * remove incorrect comment --------- Co-authored-by: Arttu <Blizzara@users.noreply.github.com> * minor: re-export `OffsetBufferBuilder` in `arrow` crate (#7077) * Add another decimal cast edge test case (#7078) * Add another decimal cast edge test case Before 1019f5b this test would fail, as the cast produced 1. 0 is an edge case worth explicitly testing for. * typo/fmt Co-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com> --------- Co-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com> * Support both 0x01 and 0x02 as type for list of booleans in thrift metadata (#7052) * Support both 0x01 and 0x02 as type for list of booleans * Also support 0 for false inside boolean collections * Use hex notation in tests * Fix LocalFileSystem with range request that ends beyond end of file (#6751) * Fix LocalFileSystem with range request that ends beyond end of file * fix windows * add comment * Seek error * fix seek check * remove windows flag * Get file length from file metadata * Introduce `UnsafeFlag` to manage disabling `ArrayData` validation (#7027) * Introduce UnsafeFlag to manage disabling validation * fix docs * Refactor arrow-ipc: Rename `ArrayReader` to `RecodeBatchDecoder` (#7028) * Rename `ArrayReader` to `RecordBatchDecoder` * Remove alias for `self` * Minor: Update release schedule (#7086) * Minor: Update release schedule * realism * Refactor some decimal-related code and tests (#7062) * Refactor some decimal-related code and tests in preparation for adding Decimal32 and Decimal64 support * Fixed symbol * Apply PR feedback * Fixed format problem * Fixed logical merge conflicts * PR feedback * Refactor arrow-ipc: Move `create_*_array` methods into `RecordBatchDecoder` (#7029) * Move `create_primitive_array` into RecordBatchReader * Move `create_list-array` into RecordBatchReader * Move `create_dictionay_array` into RecordBatchReader * Print Parquet BasicTypeInfo id when present (#7094) * Print Parquet BasicTypeInfo id when present * Improve print_schema documentation * tiny cleanup * Add a custom implementation `LocalFileSystem::list_with_offset` (#7019) * Initial change from Daniel. * Upgrade unit test to be more generic. * Add comments on why we have filter * Cleanup unit tests. * Update object_store/src/local.rs Co-authored-by: Adam Reeve <adreeve@gmail.com> * Add changes suggested by Adam. * Cleanup match error. * Apply formatting changes suggested by cargo +stable fmt --all. * Apply cosmetic changes suggested by clippy. * Upgrade test_path_with_offset to create temporary directory + files for testing rather than pointing to existing dir. --------- Co-authored-by: Adam Reeve <adreeve@gmail.com> * fix: first none/empty list in `ListArray` panics in `cast_with_options` (#7065) * fix: first none in `ListArray` panics in `cast_with_options` * simplify * fix * Update arrow-cast/src/cast/list.rs Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> --------- Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> * Benchmarks for Arrow IPC writer (#7090) * Add benchmarks for Arrow IPC writer * Add benchmarks for Arrow IPC writer * reuse target buffer * rename, etc * Add compression type * update --------- Co-authored-by: Andy Grove <agrove@apache.org> * Minor: Clarify documentation on `NullBufferBuilder::allocated_size` (#7089) * Minor: Clarify documentaiton on NullBufferBuilder::allocated_size * add note about why allocations are 64 bytes * Add more tests for edge cases * Add negative test case for incorrectly formatted large dates --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> Co-authored-by: Himadri Pal <mehimu@gmail.com> Co-authored-by: Arttu <Blizzara@users.noreply.github.com> Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com> Co-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com> Co-authored-by: Jörn Horstmann <git@jhorstmann.net> Co-authored-by: Kyle Barron <kyle@developmentseed.org> Co-authored-by: Curt Hagenlocher <curt@hagenlocher.org> Co-authored-by: Devin Smith <devinsmith@deephaven.io> Co-authored-by: Corwin Joy <corwin.joy@gmail.com> Co-authored-by: Adam Reeve <adreeve@gmail.com> Co-authored-by: irenjj <renj.jiang@gmail.com> Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com> Co-authored-by: Andy Grove <agrove@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
ArrayReader#7006Rationale for this change
I did not make these changes in #7006 to minimize the diffs
What changes are included in this PR?
RecordBatchDecoderI am working on another PRs to migrate the rest of the IPC reading code to be part of RecordBatchDecoder
Are there any user-facing changes?
No, these are crate internal structures