Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Implement the bitwise_not in NotExpr #5902

Merged
merged 5 commits into from
Apr 11, 2023
Merged

feat: Implement the bitwise_not in NotExpr #5902

merged 5 commits into from
Apr 11, 2023

Conversation

RTEnzyme
Copy link
Contributor

@RTEnzyme RTEnzyme commented Apr 7, 2023

Which issue does this PR close?

Closes #5700. Because the SQL dialect use Generic, sqlparser can't parse operator ~ as unary Bitwise_not. But we can use not to complement integer instead of operator ~ in SQL.

Rationale for this change

What changes are included in this PR?

Extending Expr::Not to work with integral types so that it can evaluate Bitwise_not expression.

Are these changes tested?

Pass Integrated tests.

Add United tests for bitwise_not integral types.

Are there any user-facing changes?

Yes. The not can NOT boolean type and Bitwise_not integer in sql.

@github-actions github-actions bot added core Core DataFusion crate physical-expr Physical Expressions labels Apr 7, 2023
@@ -622,6 +622,17 @@ async fn test_not_expressions() -> Result<()> {
];
assert_batches_eq!(expected, &actual);

let sql = "SELECT not(1), not(0)";
let actual = execute_to_batches(&ctx, sql).await;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe lets move it to scalar.slt sqllogic test?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it would be nice to move to https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/sqllogictests/test_files/scalar.slt but we can do that potentially as a follow on PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will port some tests over

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow on PR to port the tests: #5968

Ok(DataType::Boolean)
fn data_type(&self, input_schema: &Schema) -> Result<DataType> {
// Ok(DataType::Boolean)
let data_type = self.arg.data_type(input_schema)?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That looks breaking imho... I would propose to have the own BitwiseNotExpr instead of making NotExpr universal

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As @alamb suggestion, I embed bitwise_not into NotExpr. But I agree that this way will break NotExpr original struct. I think I should further discuss the implementation in the issue #5700

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, sounds good

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @RTEnzyme and for the review @comphead

@RTEnzyme please let me know if you want to polish this PR up any more (e.g. move tests) or if we should merge it as is.

@@ -622,6 +622,17 @@ async fn test_not_expressions() -> Result<()> {
];
assert_batches_eq!(expected, &actual);

let sql = "SELECT not(1), not(0)";
let actual = execute_to_batches(&ctx, sql).await;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it would be nice to move to https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/tests/sqllogictests/test_files/scalar.slt but we can do that potentially as a follow on PR

@alamb alamb merged commit 87e5adc into apache:main Apr 11, 2023
@alamb
Copy link
Contributor

alamb commented Apr 11, 2023

Thanks again @RTEnzyme and @comphead

korowa pushed a commit to korowa/arrow-datafusion that referenced this pull request Apr 13, 2023
* feat: make NotExpr support integer Bitwise_not

* feat: make NotExpr support integer Bitwise_not

---------

Co-authored-by: RT_Enzyme <52275903001@stu.ecnu.edu.cn>
jackwener added a commit to jackwener/arrow-datafusion that referenced this pull request Jun 8, 2023
alamb pushed a commit that referenced this pull request Jun 9, 2023
jayzhan211 pushed a commit to jayzhan211/datafusion that referenced this pull request Jun 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core DataFusion crate physical-expr Physical Expressions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement the bitwise complement operator
3 participants