feat: add expression array_size #1122
Conversation
native/spark-expr/src/list.rs
Outdated
| } | ||
| } | ||
| let sizes_array = Int32Array::from(builder.finish()); | ||
| Ok(ColumnarValue::Array(Arc::new(sizes_array))) |
There was a problem hiding this comment.
Is there a more efficient way to do this?
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
Outdated
Show resolved
Hide resolved
viirya
changed the title
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala
Outdated
Show resolved
Hide resolved
|
Why not to use an It looks like it does the same. It may be done in a similar way, like for the |
Good input! will look into it! |
|
Note that DataFusion |
| expr.children(2)) | ||
| None | ||
| } | ||
| case Size(child, _) if child.dataType.isInstanceOf[ArrayType] => |
There was a problem hiding this comment.
This either needs to handle when legacySizeOfNull is true or pattern match to only false
|
Have to wait for this to get into the next version |
Thanks @Groennbeck. The next DataFusion release will likely be released in the next couple of weeks. |
|
@Groennbeck DataFusion 44.0.0 is now released |
Nice! will update this later today. |
|
Hi @Groennbeck are you still planning on update this PR? |
native/spark-expr/src/list.rs
Outdated
| let list_array = as_list_array(&array_value)?; | ||
| let mut builder = Int32Array::builder(list_array.len()); | ||
| for i in 0..list_array.len() { | ||
| if list_array.is_null(i) { |
There was a problem hiding this comment.
You might want to handle legacySizeOfNull here
native/spark-expr/src/list.rs
Outdated
| if list_array.is_null(i) { | ||
| builder.append_null(); | ||
| } else { | ||
| builder.append_value(list_array.value_length(i)); |
There was a problem hiding this comment.
Will value_length(i) include nulls in the array at index i?
Hey! sorry has been busy. I can have a look this week. |
|
I am closing this PR since it is no longer active. Feel free to re-open if needed. |
6 participants
Which issue does this PR close?
Closes #.
Rationale for this change
To add support for expression: array_size
What changes are included in this PR?
How are these changes tested?
Tested by adding spark plan that calculated array_size