-
Notifications
You must be signed in to change notification settings - Fork 130
Feature: Implement Compute Functions for FixedSizeListArray
#4495
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
b625617 to
f685c12
Compare
robert3005
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we can simplify things a little bit
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
| let Some(target_element_type) = dtype.as_list_element_opt() else { | ||
| return Ok(None); | ||
| }; | ||
|
|
||
| // The list elements could technically be from either `FixedSizeList` or `List`. | ||
| if dtype.is_list() { | ||
| return Ok(None); | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems somewhat error-prone, should I create a as_fixed_size_list_element_opt that only returns the type if it is DType::FixedSizeList and have as_list_element_opt return None in that case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think s we should maybe just make this work?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Having thought about this more I think you want to separate this two out. If FixedSizeList is a dtype then it's elements are distinct from the list elements
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I separated them out:
/// Get the inner element dtype if `self` is a [`DType::List`], otherwise returns `None`.
///
/// Note that this does _not_ return `Some` if `self` is a [`DType::FixedSizeList`].
pub fn as_list_element_opt(&self) -> Option<&Arc<DType>> {
if let List(edt, _) = self {
Some(edt)
} else {
None
}
}
/// Get the inner element dtype if `self` is a [`DType::FixedSizeList`], otherwise returns
/// `None`.
///
/// Note that this does _not_ return `Some` if `self` is a [`DType::List`].
pub fn as_fixed_size_list_element_opt(&self) -> Option<&Arc<DType>> {
if let FixedSizeList(edt, ..) = self {
Some(edt)
} else {
None
}
}
/// Get the inner element dtype if `self` is **either** a [`DType::List`] or a
/// [`DType::FixedSizeList`], otherwise returns `None`
pub fn as_any_size_list_element_opt(&self) -> Option<&Arc<DType>> {
if let FixedSizeList(edt, ..) = self {
Some(edt)
} else if let List(edt, ..) = self {
Some(edt)
} else {
None
}
}| // TODO(connor): There must be a more efficient way to do this. Each `scalar_at()` call | ||
| // makes several allocations... | ||
| for i in 1..array.len() { | ||
| let current_scalar = array.scalar_at(i); | ||
| if current_scalar != first_scalar { | ||
| return Ok(Some(false)); | ||
| } | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah this is pretty bad. I'm also realizing that this will also check the dtypes are equal every single time. Maybe a eq_value or something is needed? I'll probably leave this for later.
robert3005
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only have some nits
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
Tracking Issue: #4372
Implements the same compute functions as
ListArray, but forFixedSizeListArray.FixedSizeListArrayListArrayfilter compute function with a similar density checkThe two complicated compute functions here are
filterandtake, withtakebeing significantly more complicated. Those are accompanied by tests since the logic is a lot harder.Nice-to-have-but-I'm-too-tired:
ListArraytakecompute function