Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support StringView / BinaryView in interleave kernel #6779

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Add support StringView / BinaryView in interleave kernel #6779

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
27ac654
add byteview specific interleave
onursatici Nov 23, 2024
2052cb8
clippy
onursatici Nov 24, 2024
8b11191
test
onursatici Nov 24, 2024
d5daedc
more clippy
onursatici Nov 24, 2024
6ff5c0a
more test coverage
onursatici Nov 25, 2024
763a792
enable assertion, remove explicit vector capacity
onursatici Nov 27, 2024
7aa95d2
add new test, address comments
onursatici Dec 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
enable assertion, remove explicit vector capacity
  • Loading branch information
@onursatici
onursatici committed Nov 27, 2024
commit 763a792280dcec30fe2417a0143eab9bdb457c38
10 changes: 5 additions & 5 deletions arrow-select/src/interleave.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,7 @@ fn interleave_views<T: ByteViewType>(
) -> Result<ArrayRef, ArrowError> {
let interleaved = Interleave::<'_, GenericByteViewArray<T>>::new(values, indices);
let mut views_builder = BufferBuilder::new(indices.len());
let mut buffers = Vec::with_capacity(values[0].len());
let mut buffers = Vec::new();

let mut buffer_lookup = HashMap::new();
Copy link
Contributor

@tustvold tustvold Nov 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This misunderstands #6780, the issue isn't not skipping buffers that aren't referenced, it is that the arrays being interleaved may contain the same actual buffers, e.g. sourced from the same parquet dictionary page. This needs to deduplicate based on the underlying Buffer pointers.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I have filed #6808 to deduplicate the buffers while building the interleaved array in the fallback implementation.

I am not sure if that would completely remove the need for this PR though, it should guarantee that no duplicate buffers should exist on the interleaved array, but the fallback implementation would still have 2 buffers in the test below, because those two buffers are unique. It feels like this PR and #6808 are complementary

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also please add some comments about what the buffer lookup represents? I think it is

// (input array_index, input buffer_index) --> output array buffer_index

for (array_idx, value_idx) in indices {
Expand All @@ -263,8 +263,9 @@ fn interleave_views<T: ByteViewType>(
views_builder.append(view.with_buffer_index(*new_buffer_idx).into());
}

let array =
GenericByteViewArray::<T>::try_new(views_builder.into(), buffers, interleaved.nulls)?;
let array = unsafe {
GenericByteViewArray::<T>::new_unchecked(views_builder.into(), buffers, interleaved.nulls)
};
Ok(Arc::new(array))
}

Expand Down Expand Up @@ -535,9 +536,8 @@ mod tests {
// Test fallback implementation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since interleave_fallback isn't pub I don't think we should be testing it directly. Intead you should arrange the paramters to call it directly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the comment, it was a bit misleading. I was not trying to test the fallback implementation here, but was trying to use it to show that it does produce the same results with the new implementation, but with more buffers.

With the new implementation I don't think it is straightforward to call the fallback implementation using the public methods now

let fallback = interleave_fallback(&[&view_a, &view_b], indices).unwrap();
let fallback_result = fallback.as_string_view();
// as of commit 97055631, assertion below, commented out to not block future improvements, passes:
// note that fallback_result has 2 buffers, but only one long enough string to warrant a buffer
// assert_eq!(fallback_result.data_buffers().len(), 2);
assert_eq!(fallback_result.data_buffers().len(), 2);

// Convert to strings for easier assertion
let collected: Vec<_> = result.iter().map(|x| x.map(|s| s.to_string())).collect();
Expand Down
Loading