Added support for ScalarUDFImpl::invoke_with_return_type where the invoke is passed the return type created for the udf instance

[joseph-isaacs]

Which issue does this PR close?

Closes #12819.

Rationale for this change

There is currently no way to determine the schema-defined nullability of the arrays from args: &[ColumnarValue],from the invoke_batch function. The array nullability Array::is_nullable() (which is available) only checks the array null count.

If the return type (from ScalarUDFImpl::return_type) is passed to the newly defined invoke method then this datatype can be inspected and any nullability information can be inferred.

What changes are included in this PR?

Define a new trait function invoke_batch_with_return_type which defaults to calling to invoke_batch.

Example usage can be seen here.

Are these changes tested?

Are there any user-facing changes?

There are no breaking changes, but a new optional method is added.

[jayzhan211]

How about we add InvokeUDFArg struct that includes these to avoid breaking change in the future?

[joseph-isaacs]

Do you think we should roll all the values into this struct or just number_rows and return_type?

[jayzhan211]

including args

[joseph-isaacs]

Any thoughts on the new function name, if we roll these in together. Since this subsumes invoke_batch, would we then want to deprecate that too?

[jayzhan211]

I personally prefer single invoke function, but that will be a large change so not required to roll them together.

[joseph-isaacs]

I think a better name than

 pub fn invoke_batch_with_return_type(
        &self,
        args: InvokeArgs
    ) -> Result<ColumnarValue> {

is needed, maybe invoke_with_args(), I don't like it that much, but not sure on the name.

 pub fn invoke_with_args(
        &self,
        args: InvokeArgs
    ) -> Result<ColumnarValue> {

[findepi]

In an ideal worlds, UDF would have single invoke method. This is why #13238 / #13064 / #13345 consolidation takes place.
Now, we add new invoke variant. Would we want all implementations to consolidate on this new variant, or would it feel awkward when doing so?
Passing back the return_type: &DataType might be sometimes useful, but generally feels redundant.

In #12819 (comment) i am attempting to clarify why simplify isn't useful for the use-case.

If not simplify, what if we had something like bind_return_type method that's guaranteed to be invoked before invoke / invoke_batch and which may return new UDF instance (if state changes) or nothing (per default impl). This way we would provide exactly and directly what's needed for the use-case, without changing invoke itself.

[joseph-isaacs]

I would also be happy to scope out another prepare/bind method on a UDF. But It seems like we need a consensus on the preferred solution @findepi @alamb.