Skip to content

Conversation

@guojidan
Copy link
Contributor

@guojidan guojidan commented Jan 9, 2024

Which issue does this PR close?

Closes #8756 .

Rationale for this change

What changes are included in this PR?

add monotonicity function, User can define fun monotonicity by themselves

@github-actions github-actions bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates labels Jan 9, 2024
Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @guojidan
Can we have tests for this change?

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with @comphead 's request for a test

I was thinking about how we could write a test for this feature.

The easiest way would probably be a unit test using create_physical_expr with a scalarUDF that shows the correct monotonicity.

A more end to end style one would be to show different explains depending on the monotonicity of a udf. For example, abs is not monotonic and sqrt is and thus they generate subtlely different plans:

For the abs (non- montonic) the sort is done after the Projection

For sqrt (monotonic) the sort is pushed below the Projection

❯ create table t as values (1), (3), (2);
0 rows in set. Query took 0.003 seconds.

❯ explain select abs(column1) as x from (SELECT * FROM t ORDER BY column1) ORDER BY x;
+---------------+---------------------------------------------------+
| plan_type     | plan                                              |
+---------------+---------------------------------------------------+
| logical_plan  | Sort: x ASC NULLS LAST                            |
|               |   Projection: abs(t.column1) AS x                 |
|               |     Sort: t.column1 ASC NULLS LAST                |
|               |       TableScan: t projection=[column1]           |
| physical_plan | SortExec: expr=[x@0 ASC NULLS LAST]               |
|               |   ProjectionExec: expr=[abs(column1@0) as x]      |
|               |     MemoryExec: partitions=1, partition_sizes=[1] |
|               |                                                   |
+---------------+---------------------------------------------------+
2 rows in set. Query took 0.004 seconds.

❯ explain select sqrt(column1) as x from (SELECT * FROM t ORDER BY column1) ORDER BY x;
+---------------+--------------------------------------------------------------+
| plan_type     | plan                                                         |
+---------------+--------------------------------------------------------------+
| logical_plan  | Sort: x ASC NULLS LAST                                       |
|               |   Projection: sqrt(CAST(t.column1 AS Float64)) AS x          |
|               |     Sort: t.column1 ASC NULLS LAST                           |
|               |       TableScan: t projection=[column1]                      |
| physical_plan | ProjectionExec: expr=[sqrt(CAST(column1@0 AS Float64)) as x] |
|               |   SortExec: expr=[column1@0 ASC NULLS LAST]                  |
|               |     MemoryExec: partitions=1, partition_sizes=[1]            |
|               |                                                              |
+---------------+--------------------------------------------------------------+
2 rows in set. Query took 0.003 seconds.

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me -- thank you again @guojidan and @comphead

@alamb alamb merged commit ecdfd02 into apache:main Jan 11, 2024
@guojidan guojidan deleted the monotonicity branch January 12, 2024 00:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement monotonicity for ScalarUDF

3 participants