Skip to content

Avoid the need to rewrite expressions when evaluating logical case nullability #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

Avoid the need to rewrite expressions when evaluating logical case nullability #2

wants to merge 2 commits into from

Conversation

@pepijnve
Copy link
Owner

@pepijnve pepijnve commented Nov 20, 2025

Temporary PR

@pepijnve pepijnve mentioned this pull request Nov 20, 2025
Merged
@alamb
Copy link

alamb commented Nov 20, 2025

🤖 ./gh_compare_branch_bench.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp apache#19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing issue_17801_norewrite (fd07383) to 7fa2a69 diff
BENCH_NAME=sql_planner
BENCH_COMMAND=cargo bench --bench sql_planner
BENCH_FILTER=
BENCH_BRANCH_NAME=issue_17801_norewrite
Results will be posted here when complete

pepijnve and others added 2 commits November 20, 2025 15:36
@pepijnve @alamb
apache#17801 Improve nullability reporting of case expressions (apach…
0bd127f 
…e#17813)

## Which issue does this PR close?

- Closes apache#17801
- Obviates (contains) and thus Closes apache#17833
- Obviates (contains) and thus Closes apache#18536

## Rationale for this change

apache#17357 introduced a change that replaces `coalesce` function calls with
`case` expressions. In the current implementation these two differ in
the way they report their nullability. `coalesce` is more precise than
`case` all will report itself as not nullable in situations where the
equivalent `case` does report being nullable.

The rest of the codebase currently does not expect the nullability
property of an expression to change as a side effect of expression
simplification. This PR is a first attempt to align the nullability of
`coalesce` and `case`.

## What changes are included in this PR?

Tweaks to the `nullable` logic for the logical and physical `case`
expression code to report `case` as being not nullable in more
situations.

- For logical `case`, a best effort const evaluation of 'when'
expressions is done to determine 'then' reachability. The code errs on
the conservative side wrt nullability.
- For physical `case`, const evaluation of 'when' expressions using a
placeholder record batch is attempted to determine 'then' reachability.
Again if const evaluation is not possible, the code errs on the
conservative side.
- The optimizer schema check has been relaxed slightly to allow
nullability to be removed by optimizer passes without having to disable
the schema check entirely
- The panic'ing benchmark has been reenabled

## Are these changes tested?

Additional unit tests have been added to test the new logic.

## Are there any user-facing changes?

No

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
@pepijnve
Avoid the need to rewrite expressions when evaluating logical case nu…
81f101c 
…llability
@pepijnve pepijnve force-pushed the issue_17801_norewrite branch from fd07383 to 81f101c Compare November 20, 2025 15:40
@alamb
Copy link

alamb commented Nov 20, 2025

🤖: Benchmark completed

Details 

group                                                 issue_17801_norewrite                  main
-----                                                 ---------------------                  ----
logical_aggregate_with_join                           1.01    645.7±6.97µs        ? ?/sec    1.00    639.7±9.23µs        ? ?/sec
logical_select_all_from_1000                          1.00     10.9±0.03ms        ? ?/sec    1.00     11.0±0.06ms        ? ?/sec
logical_select_one_from_700                           1.01    426.5±2.18µs        ? ?/sec    1.00    420.6±4.44µs        ? ?/sec
logical_trivial_join_high_numbered_columns            1.00    380.3±3.20µs        ? ?/sec    1.00    379.4±1.95µs        ? ?/sec
logical_trivial_join_low_numbered_columns             1.00    365.5±1.39µs        ? ?/sec    1.00    365.8±3.74µs        ? ?/sec
physical_intersection                                 1.02    851.5±9.45µs        ? ?/sec    1.00    837.6±5.58µs        ? ?/sec
physical_join_consider_sort                           1.01   1411.3±4.64µs        ? ?/sec    1.00   1395.0±5.97µs        ? ?/sec
physical_join_distinct                                1.01    359.8±2.30µs        ? ?/sec    1.00    357.0±6.41µs        ? ?/sec
physical_many_self_joins                              1.01      9.8±0.20ms        ? ?/sec    1.00      9.7±0.04ms        ? ?/sec
physical_plan_clickbench_all                          1.00    183.5±1.84ms        ? ?/sec    1.00    184.3±4.49ms        ? ?/sec
physical_plan_clickbench_q1                           1.01      2.4±0.02ms        ? ?/sec    1.00      2.4±0.02ms        ? ?/sec
physical_plan_clickbench_q10                          1.01      3.2±0.03ms        ? ?/sec    1.00      3.2±0.02ms        ? ?/sec
physical_plan_clickbench_q11                          1.02      3.5±0.15ms        ? ?/sec    1.00      3.4±0.03ms        ? ?/sec
physical_plan_clickbench_q12                          1.00      3.5±0.04ms        ? ?/sec    1.00      3.5±0.03ms        ? ?/sec
physical_plan_clickbench_q13                          1.00      3.2±0.04ms        ? ?/sec    1.00      3.2±0.03ms        ? ?/sec
physical_plan_clickbench_q14                          1.00      3.4±0.03ms        ? ?/sec    1.00      3.4±0.03ms        ? ?/sec
physical_plan_clickbench_q15                          1.01      3.3±0.04ms        ? ?/sec    1.00      3.3±0.03ms        ? ?/sec
physical_plan_clickbench_q16                          1.00      3.1±0.03ms        ? ?/sec    1.00      3.1±0.03ms        ? ?/sec
physical_plan_clickbench_q17                          1.01      3.2±0.02ms        ? ?/sec    1.00      3.2±0.02ms        ? ?/sec
physical_plan_clickbench_q18                          1.01      2.7±0.05ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q19                          1.01      3.6±0.02ms        ? ?/sec    1.00      3.6±0.03ms        ? ?/sec
physical_plan_clickbench_q2                           1.00      2.8±0.02ms        ? ?/sec    1.00      2.8±0.03ms        ? ?/sec
physical_plan_clickbench_q20                          1.00      2.5±0.02ms        ? ?/sec    1.00      2.5±0.02ms        ? ?/sec
physical_plan_clickbench_q21                          1.00      2.8±0.03ms        ? ?/sec    1.01      2.8±0.03ms        ? ?/sec
physical_plan_clickbench_q22                          1.01      3.4±0.03ms        ? ?/sec    1.00      3.4±0.03ms        ? ?/sec
physical_plan_clickbench_q23                          1.00      3.6±0.03ms        ? ?/sec    1.00      3.6±0.03ms        ? ?/sec
physical_plan_clickbench_q24                          1.00      4.1±0.02ms        ? ?/sec    1.00      4.1±0.04ms        ? ?/sec
physical_plan_clickbench_q25                          1.00      3.0±0.03ms        ? ?/sec    1.00      3.0±0.02ms        ? ?/sec
physical_plan_clickbench_q26                          1.00      2.7±0.02ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q27                          1.00      3.0±0.02ms        ? ?/sec    1.00      3.0±0.03ms        ? ?/sec
physical_plan_clickbench_q28                          1.00      3.7±0.03ms        ? ?/sec    1.00      3.7±0.04ms        ? ?/sec
physical_plan_clickbench_q29                          1.00      4.0±0.04ms        ? ?/sec    1.00      4.0±0.04ms        ? ?/sec
physical_plan_clickbench_q3                           1.00      2.7±0.03ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q30                          1.01     15.0±0.19ms        ? ?/sec    1.00     14.8±0.16ms        ? ?/sec
physical_plan_clickbench_q31                          1.00      3.7±0.04ms        ? ?/sec    1.00      3.7±0.04ms        ? ?/sec
physical_plan_clickbench_q32                          1.00      3.7±0.03ms        ? ?/sec    1.01      3.7±0.05ms        ? ?/sec
physical_plan_clickbench_q33                          1.00      3.3±0.03ms        ? ?/sec    1.00      3.3±0.03ms        ? ?/sec
physical_plan_clickbench_q34                          1.01      2.9±0.02ms        ? ?/sec    1.00      2.9±0.02ms        ? ?/sec
physical_plan_clickbench_q35                          1.01      3.0±0.02ms        ? ?/sec    1.00      3.0±0.03ms        ? ?/sec
physical_plan_clickbench_q36                          1.01      3.7±0.07ms        ? ?/sec    1.00      3.7±0.03ms        ? ?/sec
physical_plan_clickbench_q37                          1.00      3.8±0.04ms        ? ?/sec    1.00      3.8±0.03ms        ? ?/sec
physical_plan_clickbench_q38                          1.01      3.9±0.03ms        ? ?/sec    1.00      3.8±0.03ms        ? ?/sec
physical_plan_clickbench_q39                          1.00      3.7±0.03ms        ? ?/sec    1.00      3.7±0.04ms        ? ?/sec
physical_plan_clickbench_q4                           1.00      2.4±0.02ms        ? ?/sec    1.00      2.4±0.02ms        ? ?/sec
physical_plan_clickbench_q40                          1.01      4.5±0.04ms        ? ?/sec    1.00      4.5±0.05ms        ? ?/sec
physical_plan_clickbench_q41                          1.01      3.9±0.04ms        ? ?/sec    1.00      3.9±0.04ms        ? ?/sec
physical_plan_clickbench_q42                          1.01      3.9±0.04ms        ? ?/sec    1.00      3.8±0.02ms        ? ?/sec
physical_plan_clickbench_q43                          1.00      4.3±0.03ms        ? ?/sec    1.00      4.3±0.12ms        ? ?/sec
physical_plan_clickbench_q44                          1.01      2.6±0.02ms        ? ?/sec    1.00      2.5±0.02ms        ? ?/sec
physical_plan_clickbench_q45                          1.00      2.6±0.02ms        ? ?/sec    1.00      2.6±0.02ms        ? ?/sec
physical_plan_clickbench_q46                          1.00      3.0±0.03ms        ? ?/sec    1.00      3.0±0.05ms        ? ?/sec
physical_plan_clickbench_q47                          1.00      3.6±0.04ms        ? ?/sec    1.00      3.6±0.05ms        ? ?/sec
physical_plan_clickbench_q48                          1.00      4.3±0.06ms        ? ?/sec    1.02      4.4±0.11ms        ? ?/sec
physical_plan_clickbench_q49                          1.00      4.6±0.05ms        ? ?/sec    1.02      4.7±0.11ms        ? ?/sec
physical_plan_clickbench_q5                           1.00      2.7±0.03ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q50                          1.00      4.1±0.03ms        ? ?/sec    1.00      4.1±0.04ms        ? ?/sec
physical_plan_clickbench_q51                          1.00      3.2±0.02ms        ? ?/sec    1.00      3.2±0.03ms        ? ?/sec
physical_plan_clickbench_q6                           1.00      2.7±0.03ms        ? ?/sec    1.00      2.7±0.02ms        ? ?/sec
physical_plan_clickbench_q7                           1.00      2.4±0.02ms        ? ?/sec    1.00      2.4±0.02ms        ? ?/sec
physical_plan_clickbench_q8                           1.00      3.2±0.03ms        ? ?/sec    1.00      3.2±0.06ms        ? ?/sec
physical_plan_clickbench_q9                           1.00      3.1±0.03ms        ? ?/sec    1.00      3.1±0.02ms        ? ?/sec
physical_plan_tpcds_all                               1.03   1082.1±6.00ms        ? ?/sec    1.00   1046.6±7.60ms        ? ?/sec
physical_plan_tpch_all                                1.01     65.3±1.05ms        ? ?/sec    1.00     64.9±0.34ms        ? ?/sec
physical_plan_tpch_q1                                 1.00      2.1±0.01ms        ? ?/sec    1.00      2.1±0.01ms        ? ?/sec
physical_plan_tpch_q10                                1.00      4.0±0.02ms        ? ?/sec    1.00      4.0±0.01ms        ? ?/sec
physical_plan_tpch_q11                                1.00      3.6±0.01ms        ? ?/sec    1.00      3.6±0.02ms        ? ?/sec
physical_plan_tpch_q12                                1.00   1830.3±7.74µs        ? ?/sec    1.00   1830.0±8.58µs        ? ?/sec
physical_plan_tpch_q13                                1.00   1441.4±6.78µs        ? ?/sec    1.00   1445.3±7.58µs        ? ?/sec
physical_plan_tpch_q14                                1.00      2.0±0.01ms        ? ?/sec    1.00      2.0±0.01ms        ? ?/sec
physical_plan_tpch_q16                                1.00      2.4±0.01ms        ? ?/sec    1.00      2.4±0.01ms        ? ?/sec
physical_plan_tpch_q17                                1.01      2.7±0.05ms        ? ?/sec    1.00      2.6±0.02ms        ? ?/sec
physical_plan_tpch_q18                                1.01      2.7±0.02ms        ? ?/sec    1.00      2.7±0.01ms        ? ?/sec
physical_plan_tpch_q19                                1.00      3.2±0.01ms        ? ?/sec    1.00      3.2±0.10ms        ? ?/sec
physical_plan_tpch_q2                                 1.01      5.9±0.02ms        ? ?/sec    1.00      5.8±0.03ms        ? ?/sec
physical_plan_tpch_q20                                1.01      3.2±0.02ms        ? ?/sec    1.00      3.2±0.02ms        ? ?/sec
physical_plan_tpch_q21                                1.00      4.2±0.03ms        ? ?/sec    1.00      4.2±0.02ms        ? ?/sec
physical_plan_tpch_q22                                1.01      2.9±0.02ms        ? ?/sec    1.00      2.9±0.01ms        ? ?/sec
physical_plan_tpch_q3                                 1.00      2.7±0.01ms        ? ?/sec    1.00      2.7±0.01ms        ? ?/sec
physical_plan_tpch_q4                                 1.00  1480.1±11.94µs        ? ?/sec    1.00  1475.3±14.52µs        ? ?/sec
physical_plan_tpch_q5                                 1.00      3.3±0.01ms        ? ?/sec    1.00      3.3±0.03ms        ? ?/sec
physical_plan_tpch_q6                                 1.00    865.6±6.84µs        ? ?/sec    1.00    868.6±6.57µs        ? ?/sec
physical_plan_tpch_q7                                 1.00      4.2±0.02ms        ? ?/sec    1.00      4.2±0.05ms        ? ?/sec
physical_plan_tpch_q8                                 1.00      5.5±0.03ms        ? ?/sec    1.00      5.5±0.03ms        ? ?/sec
physical_plan_tpch_q9                                 1.00      4.0±0.02ms        ? ?/sec    1.00      4.0±0.02ms        ? ?/sec
physical_select_aggregates_from_200                   1.00     16.7±0.08ms        ? ?/sec    1.00     16.7±0.06ms        ? ?/sec
physical_select_all_from_1000                         1.01     24.0±0.17ms        ? ?/sec    1.00     23.8±0.09ms        ? ?/sec
physical_select_one_from_700                          1.01  1106.5±10.33µs        ? ?/sec    1.00   1091.3±8.59µs        ? ?/sec
physical_sorted_union_order_by_10_int64               1.02      6.0±0.05ms        ? ?/sec    1.00      6.0±0.02ms        ? ?/sec
physical_sorted_union_order_by_10_uint64              1.01     13.0±0.10ms        ? ?/sec    1.00     12.8±0.06ms        ? ?/sec
physical_sorted_union_order_by_50_int64               1.06    171.4±1.01ms        ? ?/sec    1.00    162.3±0.96ms        ? ?/sec
physical_sorted_union_order_by_50_uint64              1.04    387.4±2.22ms        ? ?/sec    1.00    374.3±2.38ms        ? ?/sec
physical_theta_join_consider_sort                     1.00   1765.4±7.75µs        ? ?/sec    1.00   1759.0±6.68µs        ? ?/sec
physical_unnest_to_join                               1.01   1856.4±6.18µs        ? ?/sec    1.00   1842.2±9.19µs        ? ?/sec
physical_window_function_partition_by_12_on_values    1.02   1130.9±4.30µs        ? ?/sec    1.00   1108.5±7.98µs        ? ?/sec
physical_window_function_partition_by_30_on_values    1.02      2.3±0.01ms        ? ?/sec    1.00      2.3±0.05ms        ? ?/sec
physical_window_function_partition_by_4_on_values     1.00    679.7±5.16µs        ? ?/sec    1.00    676.9±4.55µs        ? ?/sec
physical_window_function_partition_by_7_on_values     1.02    841.9±3.84µs        ? ?/sec    1.00    829.1±4.24µs        ? ?/sec
physical_window_function_partition_by_8_on_values     1.01   899.4±15.81µs        ? ?/sec    1.00    890.6±5.35µs        ? ?/sec
with_param_values_many_columns                        1.00    650.2±7.80µs        ? ?/sec    1.01    658.3±8.02µs        ? ?/sec

@pepijnve
Copy link
Owner Author

pepijnve commented Nov 20, 2025

Well that's surprising... Doesn't seem to be an improvement, does it?

@pepijnve pepijnve closed this Nov 20, 2025
@alamb
Copy link

alamb commented Nov 20, 2025

Well that's surprising... Doesn't seem to be an improvement, does it?

I don't think the time spent analyziing case nullability is a large part of these benchmarks

@pepijnve
Copy link
Owner Author

pepijnve commented Nov 20, 2025

Closed in favour of apache#18849

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pepijnve @alamb