Milestone 24: DISTINCT and IN, and two NULLs that disagree

[YheChen]

The two things a visitor types first and could not do:

SELECT DISTINCT city FROM users;              -- "not implemented yet"
SELECT * FROM users WHERE id IN (1, 2, 3);    -- "not implemented yet"

Unrelated in the engine, shipped together because each is a place SQL's equality
is not the one a programmer brings to it — and the two disagree with each
other:

• DISTINCT treats two NULLs as the same, so one row survives, even though
  NULL = NULL is unknown everywhere else in the language.
• NOT IN treats a NULL in its list as poison, so x NOT IN (1, NULL) is
  never TRUE, however unrelated x is to either value.

IN is a union of equalities, exactly

x IN (a, b) is x = a OR x = b. Not approximately — that equivalence is
the definition, and every surprise follows from it rather than being coded:

NULL IN (1, 2)	NULL, every comparison is unknown
2 IN (2, NULL)	TRUE, because TRUE OR unknown is TRUE
x NOT IN (1, NULL)	never TRUE, it means x <> 1 AND x <> NULL

It stays an InList node rather than being desugared at parse time. The
desugaring is exact; the AST view is meant to show the query somebody wrote,
and an error span pointing at an OR nobody typed costs a reader more than the
node costs the evaluator, which is one case.

IN (SELECT …) is refused by name — it's a semi-join, and evaluating it as a
list would materialise the subquery per row.

DISTINCT streams

The lazy version is a sort that drops equal neighbours. Correct, needs no new
operator, and wrong here for one reason: a sort is blocking, so
SELECT DISTINCT city FROM t LIMIT 2 would read the whole table. Hashing
instead emits the first row immediately and the LIMIT really does stop the
scan early. What it holds instead of a buffer is a set of distinct rows — the
right trade for a LIMIT.

distinct_key carries each value's type, because Python hashes True == 1
and 1 == 1.0 alike and a plain set of row tuples would fold a BOOLEAN row
into an INTEGER one and lose it. -0.0 and 0.0 deliberately collide: they're
equal in SQL, so emitting both would be a duplicate.

One integration was free, one was not

IN inherited Milestone 20's most-common-values list for nothing — a union of
equalities is a sum of their selectivities, so on a column whose list is
complete the estimate is a count:

	estimated	actual
city IN ('london', 'ny')	334	334
age IN (7)	10	10

But Milestone 19's null-rejection analysis returns "could be anything" for a node
it doesn't recognise — safe, and it would have quietly stopped rewriting
outer joins. It had to be taught InList explicitly, four lines and a test that
fails without them.

Verification

320,000 generated query pairs agree with SQLite, with DISTINCT on ~20% of
plain selects and an IN list in ~18% of predicates, a quarter of those with a
NULL in the list on purpose. Nineteen hand-written cases, weighted at the NULL
rules.

Five guards fired, all naming these as unimplemented. The fifth was found by
CI rather than by looking: examples/milestone2_parser.py keeps a list of
deliberately-failing statements, and its own comment says this has happened
twice before.

That demo slot has now had four occupants: ORDER BY until Milestone 13,
LEFT JOIN until 18, DISTINCT until this one, LIKE now. An example of what
an engine cannot do is a claim with a shelf life.

make ci green. 1,736 Python tests, 160 frontend.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
chen-db	Ready	Preview	Aug 9, 2026 1:09am