Skip to content

Conversation

@waynexia
Copy link
Member

Which issue does this PR close?

Rationale for this change

Follows #15326, this patch includes Arrow, LongArrow, HashArrow, HashLongArrow, AtAt, IntegerDivide, HashMinus, AtQuestion, Question, QuestionAnd and QuestionPipe binary operators.

What changes are included in this PR?

New operators. Only XOR is actually implemented (functional) among them, by mapping to existing BitwiseXor.

Are these changes tested?

Compiler tests it

Are there any user-facing changes?

New APIs

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
@github-actions github-actions bot added sql SQL Planner logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates substrait Changes to the substrait crate labels Mar 20, 2025
Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @waynexia -- I think the idea is great. However, I think there should be sql level tests (sqllogitests) that run these operators. I think it will make extending operators easier for other systems built on DataFusion

The tests can / should probably error with unsupported

It might also be a good idea to include some documentation in the operators themselves that DataFusion doesn't have default implementations

AtArrow | ArrowAt | Arrow | LongArrow | HashArrow | HashLongArrow | AtAt
| HashMinus | AtQuestion | Question | QuestionAnd | QuestionPipe
| IntegerDivide => {
unreachable!("These operators should be rewritten to functions")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think this code is unreachable -- I got it to panic:

DataFusion CLI v46.0.1
> select 'foo' @> 'bar';

thread 'main' panicked at datafusion/physical-expr/src/expressions/binary.rs:799:17:
internal error: entered unreachable code: These operators should be rewritten to functions
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just realized <@ and @> are only supported for lists 🤣

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
@github-actions github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Mar 20, 2025
@waynexia
Copy link
Member Author

I think there should be sql level tests (sqllogitests) that run these operators

That's a good idea! I think I know them much better after writing some SQLs (though none of them can run... lol)

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
@waynexia
Copy link
Member Author

It might also be a good idea to include some documentation in the operators themselves that DataFusion doesn't have default implementations

Added in 5828cba 👍

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

THanks @waynexia

@waynexia waynexia merged commit 916ccc3 into apache:main Mar 24, 2025
27 checks passed
@waynexia waynexia deleted the extend-operator-2 branch March 24, 2025 17:25
@waynexia
Copy link
Member Author

Thank you @alamb

qstommyshu pushed a commit to qstommyshu/datafusion that referenced this pull request Mar 27, 2025
* include some BinaryOperator from sqlparser

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* slt for new operators

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add document about availability

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
waynexia added a commit to waynexia/arrow-datafusion that referenced this pull request Apr 15, 2025
* include some BinaryOperator from sqlparser

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* slt for new operators

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add document about availability

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
nirnayroy pushed a commit to nirnayroy/datafusion that referenced this pull request May 2, 2025
* include some BinaryOperator from sqlparser

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* slt for new operators

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* add document about availability

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates sql SQL Planner sqllogictest SQL Logic Tests (.slt) substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants