feat: null aware RightAnti hash join execution + planning support by saadtajwar · Pull Request #23957 · apache/datafusion

saadtajwar · 2026-07-29T00:12:30Z

Which issue does this PR close?

Closes Support null-aware RightAnti hash join (build on the subquery side) for NOT IN #23931

Rationale for this change (copied from issue):

DataFusion plans NOT IN (subquery) as a null-aware anti join, but HashJoinExec only supports null_aware = true for LeftAnti with a single join key (validated in datafusion-physical-plan/src/joins/hash_join/exec.rs). Since HashJoinExec always builds on the left input, the build side of a null-aware anti join is the outer table, not the subquery.

This has two costs:

Memory scales with the wrong side. For SELECT ... FROM big_fact WHERE key NOT IN (SELECT k FROM small_dim), the hash table is built over the entire fact table. Memory is O(outer) when it could be O(subquery).

The operator cannot be distributed. The null-aware logic coordinates three pieces of global state across probe partitions through in-process shared memory: probe_side_has_null: AtomicBool, probe_side_non_empty: AtomicBool, and the visited-build-row bitmap, with the last probe partition to finish emitting the unmatched build rows (hash_join/stream.rs). This is correct and cheap in one process, but engines that split probe partitions across processes get independent copies of all three and produce duplicated or incorrect rows. Ballista hit exactly this (apache/datafusion-ballista#2187) and currently has to force the join into a single task (apache/datafusion-ballista#2188), losing all parallelism.

What changes are included in this PR?

This PR adds physical execution support for null-aware RightAnti hash joins using the CollectLeft partition mode.
The build side now records whether it contains a NULL join key, and the join outputs no rows when the build side contains NULL, filters probe rows with NULL keys, and outputs all probe rows when the build side is empty.

For planner support, JoinSelection was updated to swap a null-aware LeftAnti join to a null-aware RightAnti join when statistics show that the right side is smaller with the swapped join always using the CollectLeft partition mode

Are these changes tested?

Yes

Are there any user-facing changes?

NOT IN queries may now use a null-aware RightAnti physical plan when the subquery is smaller. Query results remain unchanged, but these queries can use less memory and keep the outer table partitioned.

codecov-commenter · 2026-07-29T00:42:31Z

Codecov Report

❌ Patch coverage is 97.68786% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.99%. Comparing base (d5bd10d) to head (c29d3c0).

Files with missing lines	Patch %	Lines
...fusion/physical-plan/src/joins/hash_join/stream.rs	95.91%	1 Missing and 1 partial ⚠️
...atafusion/physical-optimizer/src/join_selection.rs	94.44%	0 Missing and 1 partial ⚠️
datafusion/physical-plan/src/joins/mod.rs	85.71%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #23957    +/-   ##
========================================
  Coverage   80.98%   80.99%            
========================================
  Files        1106     1106            
  Lines      383495   383621   +126     
  Branches   383495   383621   +126     
========================================
+ Hits       310590   310722   +132     
+ Misses      54572    54561    -11     
- Partials    18333    18338     +5

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

saadtajwar · 2026-07-29T01:24:56Z

cc @andygrove - looking forward to your feedback! I'll have the second PR with the planner support up shortly :)

saadtajwar · 2026-07-29T01:26:55Z

+            if !matches!(
+                (join_type, partition_mode),
+                (JoinType::LeftAnti, _)
+                    | (JoinType::RightAnti, PartitionMode::CollectLeft) // `PartitionMode::CollectLeft` is safe because `RightAnti` is probe-driven


Do we need the partition_mode check for CollectLeft here? Or should we just count on it being enforced/never chosen as anything but CollectLeft beforehand? I suppose it's possible to explicitly create a plan with a different partition mode if you bypass the SQL -> plan frontend and create directly? Or do we make the assumption that users will only create logical plans and not physical? 👀

saadtajwar · 2026-07-30T23:14:39Z

cc @andygrove - just to update, this PR should now fully close out #23931 ! Just this one PR now with all changes instead of two :)

…htanti-hash-join # Conflicts: # datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt

kumarUjjawal

Thank you for working on this @saadtajwar

I have left some comments. Do we have benchmark results for this?

kumarUjjawal · 2026-08-07T12:45:14Z

        bounds = None;
    }

+    let build_has_null = !left_values.is_empty() && left_values[0].null_count() > 0;


This can:
- Miss a NULL in the subquery and incorrectly return outer rows.
- Return a logically NULL outer key when the subquery is non-empty.

we should use logical NULL masks on both sides.

Ah good catch - fixed to use those logical null masks!

kumarUjjawal · 2026-08-07T12:47:35Z

-                    .store(true, Ordering::Relaxed);
-            }
+            match self.join_type {
+                JoinType::RightAnti => {


filtered null-aware joins should not beswapped or filtered null-aware RightAnti should be rejected.

Thanks, done! Added that condition to can_swap_hash_join

kumarUjjawal · 2026-08-07T12:48:10Z

            self.right_side_ordered,
        )?;

+        // If null-aware RightAnti join, we don't want to emit NULL probe keys


always allocates a Vec and copies every unmatched index, even when the probe batch has no NULLs. This affects low-hit NOT IN queries that return many rows—the important target workload. Reuse the logical validity mask and skip the copy when it contains no NULLs.

Ah I see, that makes sense, thanks! Pushed a change to address, please let me know your thoughts!

saadtajwar · 2026-08-08T17:36:59Z

@kumarUjjawal thanks for the PR comments! Pushed some commits to address those, please let me know your thoughts!

I couldn't find any existing benchmarks that exercises this path, so I just ran the below on my local machine! Please let me know your thoughts - thanks again for taking the time to review!

-- large outer (nullable key)
CREATE TABLE big_outer AS
SELECT
  CASE WHEN value % 17 = 0 THEN CAST(NULL AS BIGINT) ELSE value END AS k,
  value AS payload
FROM range(5000000);

-- small subquery side (nullable key)
CREATE TABLE small_dim AS
SELECT
  CASE WHEN value % 19 = 0 THEN CAST(NULL AS BIGINT) ELSE value END AS k
FROM range(20000);

SET datafusion.optimizer.join_reordering = true;
SET datafusion.optimizer.prefer_hash_join = true;

EXPLAIN
SELECT count(*)
FROM big_outer o
WHERE o.k NOT IN (SELECT k FROM small_dim);

Query under test:

SELECT count(*)
FROM big_outer o
WHERE o.k NOT IN (SELECT k FROM small_dim);

Runner flags (same on both sides):

--iterations 3 --partitions 2 --batch-size 4096 --memory-limit 2G

Plans

Branch	Hash join
`main`	`LeftAnti`, `null_aware` (build on outer / 5M rows)
this PR	`RightAnti`, `CollectLeft`, `null_aware` (build on subquery / 20K rows)

Results

	iter 0	iter 1	iter 2	Peak pool reserved
main	58.8 ms	57.6 ms	51.9 ms	77.5 MB
this PR	36.2 ms	31.6 ms	31.3 ms	314.9 KB

So for this shape the PR cuts peak join memory by ~250x (build moves from the outer table to the subquery) and is ~1.7x faster wall-clock on this machine

kumarUjjawal

I have left few comments please let me know what you think.

kumarUjjawal · 2026-08-09T04:20:38Z

+                    // Check if probe side (RIGHT) contains NULL
+                    // Since null_aware validation ensures single column join, we only check the first column
+                    let probe_key_column = &state.values[0];
+                    if probe_key_column.logical_null_count() > 0 {


This changes filtered LeftAnti results. A dictionary/run-end logical NULL is recorded globally before the JoinFilterm runs. For example, outer (1, A), subquery (NULL, B), with outer.group = inner.group: the NULL row is rejected, so theouter row should survive, but this suppresses it. Please revert this LeftAnti change for this PR, limit it to unfiltered joins, or make NULL handling filter-aware.

Ah I see - limited to unfiltered, thank you for the catch

kumarUjjawal · 2026-08-09T04:22:40Z

        bounds = None;
    }

+    let build_has_null =


logical_null_count() now runs for every hash join, but build_side_has_null is only used by null-aware RightAnti. For dictionary/run-end keys this can add an unnecessary O(build rows) pass to unrelated joins. Can we compute it only for null-aware RightAnti?

Yes! Sounds good, done

kumarUjjawal · 2026-08-09T04:23:42Z

The optimizer no longer swaps filtered joins, which fixes the SQL planner path. However, public HashJoinExec::try_new and protobuf decoding still accept filtered null-aware RightAnti, where the build-NULL check runs before the filter. I suggest rejecting that combination?

Ah thank you, done

saadtajwar · 2026-08-09T17:18:29Z

@kumarUjjawal thank you again for taking the time to review here! Addressed your comments in the latest commit - please let me know what you think!

kumarUjjawal · 2026-08-10T03:39:34Z

@saadtajwar can you please look at the failing ci

saadtajwar · 2026-08-10T23:29:01Z

@saadtajwar can you please look at the failing ci

Apologies - this should be fixed now. Really appreciate you reviewing as always!

kumarUjjawal

Thank you @saadtajwar

LGTM!

saadtajwar added 3 commits July 28, 2026 18:43

feat: supporting null-aware RightAnti hash join for NOT IN

54be73d

modifying hash join streaming for changes

be053ca

testing added

d18f9b1

github-actions Bot added the physical-plan Changes to the physical-plan crate label Jul 29, 2026

spellchecker smh

eb3ba98

saadtajwar marked this pull request as ready for review July 29, 2026 01:24

saadtajwar mentioned this pull request Jul 29, 2026

Support null-aware RightAnti hash join (build on the subquery side) for NOT IN #23931

Open

saadtajwar commented Jul 29, 2026

View reviewed changes

saadtajwar added 4 commits July 30, 2026 18:45

feat: null aware RightAnti hash join planning support

7dd419d

planner changes for statistical join selection

9c37c68

test changes for planner support

e1a98e5

slt files changed

c034eaa

saadtajwar changed the title ~~feat: null aware RightAnti hash join execution~~ feat: null aware RightAnti hash join execution + planning support Jul 30, 2026

Merge upstream main and resolve SQL plan conflict

93f11b5

github-actions Bot added optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Jul 30, 2026

saadtajwar and others added 4 commits July 31, 2026 14:34

Merge branch 'main' into feat/null-aware-rightanti-hash-join

a91a26b

Merge branch 'main' into feat/null-aware-rightanti-hash-join

d321199

Merge remote-tracking branch 'upstream/main' into feat/null-aware-rig…

466da61

…htanti-hash-join # Conflicts: # datafusion/sqllogictest/test_files/dynamic_filter_pushdown_config.slt

Merge branch 'main' into feat/null-aware-rightanti-hash-join

71e166b

kumarUjjawal reviewed Aug 7, 2026

View reviewed changes

saadtajwar added 4 commits August 7, 2026 21:03

reusing mask for not emiting NULL probe keys

9363df4

not swapping filtered null-aware joins

d393d4c

using logical NULL masks

181cfca

updated tests

d3c49e6

Merge branch 'main' into feat/null-aware-rightanti-hash-join

41583c9

kumarUjjawal reviewed Aug 9, 2026

View reviewed changes

rejecting filtered and the compute build side null check

fff036a

Merge branch 'main' into feat/null-aware-rightanti-hash-join

8088eec

failing CI tests fixed

0e75fba

Merge branch 'main' into feat/null-aware-rightanti-hash-join

c29d3c0

saadtajwar requested a review from kumarUjjawal August 11, 2026 12:24

kumarUjjawal approved these changes Aug 13, 2026

View reviewed changes

Uh oh!

Conversation

saadtajwar commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change (copied from issue):

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

codecov-commenter commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

saadtajwar commented Jul 29, 2026

Uh oh!

saadtajwar Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saadtajwar commented Jul 30, 2026

Uh oh!

kumarUjjawal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saadtajwar commented Aug 8, 2026

Plans

Results

Uh oh!

kumarUjjawal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saadtajwar commented Aug 9, 2026

Uh oh!

kumarUjjawal commented Aug 10, 2026

Uh oh!

saadtajwar commented Aug 10, 2026

Uh oh!

kumarUjjawal left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

saadtajwar commented Jul 29, 2026 •

edited

Loading

codecov-commenter commented Jul 29, 2026 •

edited

Loading

saadtajwar Jul 29, 2026 •

edited

Loading