Skip to content

fix(query): compare two lists in dictionary order - #4956

Merged
Ignition merged 8 commits into
masterfrom
relations/34-compare-two-lists
Sep 29, 2026
Merged

Ignition merged 8 commits into
masterfrom
relations/34-compare-two-lists

Conversation

@Ignition

@Ignition Ignition commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Two lists are placed in the dictionary order the specification gives: elements
pairwise from the start, settling on the first that differs, and a shorter list
first where the two agree up to its end. Where placing them reaches a null
nothing is decided, so all four ordered comparisons answer Null. That leaves
[1] < [1, 0] and [1] < [1, null] both true, the second because no element ever
meets the null, while [1, 2] >= [1, null] is undecided.

No band separates the rows such a comparison keeps from the rows it drops when a
list is the bound, and the filter a scan stands in for is gone from the plan by
then, so the scan reads the bound itself over every row carrying the property.
Each indexed property carries its own reading, and a rejection passes every
entry sharing the values that reading looked at, which the index holds in one
run. On-disk storage fences by a band alone, so it reads the bound over what the
band gathered and pays for a second pass.

@Ignition Ignition added this to the mg-v3.14.0 milestone Sep 28, 2026
@Ignition Ignition added CI -build=community -test=core Run community build and core tests on push CI -build=coverage -test=core Run coverage build and core tests on push CI -build=debug -test=core Run debug build and core tests on push CI -build=debug -test=integration Run debug build and integration tests on push CI -build=release -test=core Run release build and core tests on push CI -build=release -test=e2e Run release build and e2e tests on push CI -build=release -test=benchmark Run release build and benchmark on push CI -build=coverage -test=clang_tidy labels Sep 28, 2026
@Ignition Ignition linked an issue Sep 28, 2026 that may be closed by this pull request
@github-actions

github-actions Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

This PR has potential conflicts with the following other open pull requests which modify the same files:

@Ignition Ignition added bug bug documentation documentation labels Sep 28, 2026
@Ignition

Copy link
Copy Markdown
Contributor Author

Tracking

  • [Link to Epic/Issue]

Standard development

CI Testing Labels

  • Select the appropriate CI test labels (CI -build=build-name -test=test-suite)

Documentation checklist

  • Add the documentation label
  • Add the bug / feature label
  • Add the milestone for which this feature is intended
    • If not known, set for a later milestone
  • Write a release note, including added/changed clauses
    • Comparing two lists with <, <=, > or >= now places them in
      dictionary order instead of raising an error, and answers Null where
      reaching a null leaves the pair undecided.
      #4956
  • [ Documentation PR link memgraph/documentation#XXXX ]
    • Is back linked to this development PR

@Ignition

Ignition commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author

CompareOfLists in src/query/relations/comparability.cpp walks the two lists
in step and returns on the first element pair that decides: nothing at all where
an element pair is undecided, unordered where one is unordered, and otherwise
the element's own answer. Reaching the end of the shorter list without deciding
falls through to a.size() <=> b.size(). It recurses through Compare, so a
list nested in a list is read the same way.

On the index side, ValidFor was answering two questions at once: whether
comparability places a value at all, and whether a band over the stored order
fences exactly the rows a filter keeps. Those part company for a list, because
the stored order decides pairs comparability leaves undecided, putting
[1, null] above [1, 2] where the filter keeps neither. AnIndexCanFence is
the second question, and a bound it turns down now reaches the scan as
IsNotNull() carrying a value predicate that reads the bound over each
candidate, rather than as a band. The predicate exists because the index-lookup
rewrite deletes the filter it stands in for, so nothing downstream would catch
the extra rows a band hands back.

A composite index needed the reading per property rather than on the leading one
alone, since an equality on the leading property is what makes a trailing one
the range. leading_predicate_ is therefore a vector, and AdvanceUntilValid_
returns the property whose predicate rejected rather than a bare flag. The scan
then seeks past every entry sharing the values that predicate read: entries
carrying equal values are contiguous at any depth, because the index orders on
them ahead of everything beyond, so the seek is the existing leading-value jump
given a longer prefix. A chunked scan steps instead at every depth, since a seek
would pass a node another thread has marked and knows nothing of the chunk end.

On-disk storage read only the band out of the range, so a list bound answered
with rows the filter drops. It now gathers by the band as before and drops what
the predicate turns down in a second pass, which costs a pass and answers the
same.

Two tests carry the weight. tests/unit/query_index_differential.cpp runs the
same queries with and without an index and requires the answers to match, over a
column of lists and over a composite index with a trailing list bound.
LabelPropertyCompositeIndexPassesEveryEntrySharingARejectedValue counts
predicate calls over five vertices per trailing value, so it fails if the scan
reads a rejected run entry by entry instead of seeking past it.

tests/benchmark/storage_v2_label_property_index_predicate.cpp sweeps the rows
sharing a value over a fixed column, with a predicate that keeps nothing, so the
only work is rejecting. Its first point is the baseline: every value is distinct
there, no run exists and no seek fires. The trailing sweep tracks the leading one
the index already had, which is the claim worth checking, since the trailing case
now runs the same mechanism.

Both sweeps agree on where that mechanism pays, and it is later than the seek
assumes. On a Debug build of this branch, a run of four entries costs roughly
four times the baseline on either sweep, a run of sixteen is about level with it,
and the gain arrives from sixty-four upward. So a seek is worth its descent only
once a run is some tens of entries long, while the rule fires it as soon as two
entries share a value.

That rule is not introduced here and nothing regresses by it. It governs the
leading property just as much, where it has shipped since 3.13.0, and the
baseline those figures are read against never shipped: before a scan carried a
predicate at all, every entry in the band was resolved to a vertex and handed to
the filter above. Removing that resolution is the larger effect and it holds at
every run length, so a column whose values repeat only a few times gains less
than it could rather than less than it had. A column of distinct values never
seeks, since the rule needs a duplicate to fire.

Two things follow for this change rather than for that rule. A list bound
allocates a TypedValue per candidate, so its predicate costs far more than the
string comparisons the rule was tuned against, which makes the seek likelier to
be right here than there. And the figures above come from a trivial predicate,
the cheapest case for stepping and so the least favourable to seeking, which
makes the crossover they show an upper bound. Tuning it wants its own change and
a sweep over predicate cost, not this one.

@Ignition
Ignition force-pushed the relations/34-compare-two-lists branch 4 times, most recently from d3faed5 to 7484bab Compare September 28, 2026 18:24
@Ignition Ignition self-assigned this Sep 28, 2026
@Ignition
Ignition marked this pull request as ready for review September 28, 2026 19:18
A scan standing in for a filter can re-read an entry the band it walks cannot
separate, which is how the search-term predicates already work. The predicate
was kept for the leading property alone, so a bound on a trailing property was
left to a filter that the index-lookup rewrite had already removed.

Each indexed property now carries its own. A rejection names the property whose
predicate raised it, and the scan seeks past every entry sharing the values that
predicate read: those entries are held in one run whichever property it was,
since the index orders on them ahead of everything beyond. A chunked scan steps
instead, because a seek would pass a node another thread has marked and knows
nothing of the chunk end.
Two lists are placed in the dictionary order the specification gives: elements
pairwise from the start, a shorter list first where the two agree up to its end,
and nothing decided where placing them reaches a null. That leaves
[1] < [1, null] decided, because no element meets the null, while
[1, 2] >= [1, null] is not.

A band cannot separate the rows that follow from the rows a filter drops, and
the filter is gone by the time the scan runs, so a list bound is read by the
scan instead: every row carrying the property is handed to the same comparison
the filter would have made. On-disk storage fences by a band alone, so it reads
the bound over what the band gathered, which costs it a second pass.
…x predicates

Rewrite the comments added by this branch to name the mechanism directly.
Move AdvanceOutcome and Advance above the AdvanceUntilValid_ comment they
had separated from its function. Merge case List into the true group of
ValidFor, which clang-tidy flagged as a cloned branch.
…e scans

A list bound reaches the scan as an IS NOT NULL range with a value
predicate, and the planner has already dropped its filter.

The global vertex property index read only the bounds, so a list bound
returned every vertex carrying the property. The index now takes a
PropertyValueRange and checks its predicate per entry, as the edge
indexes do; ScanAllByVertexProperty and ScanParallelByVertexProperty
pass the range when it carries a predicate. The parallel edge scans
passed no range on the IS NOT NULL branch and now pass it.

On-disk storage in edge import mode reads from its own cache and aborted
on any range carrying a predicate, CONTAINS included. The cache now
takes the whole range, and its in-memory index applies the predicate.
The specification counts a NaN incomparable, and a pair of lists that
compares an incomparable element pair is itself incomparable, so all four
ordered comparisons answer Null. They answered false, reading the NaN as
unordered the way a scalar comparison does. A scalar NaN still compares
false.

Adds gql_behave scenarios for list comparison, each list-bound query run
with no index, a label-property, composite, global and edge index.
@Ignition Ignition added Docs - changelog only Docs - changelog only and removed CI -build=community -test=core Run community build and core tests on push CI -build=coverage -test=core Run coverage build and core tests on push CI -build=debug -test=core Run debug build and core tests on push CI -build=debug -test=integration Run debug build and integration tests on push CI -build=release -test=core Run release build and core tests on push CI -build=release -test=e2e Run release build and e2e tests on push CI -build=release -test=benchmark Run release build and benchmark on push CI -build=coverage -test=clang_tidy documentation documentation labels Sep 29, 2026
@Ignition
Ignition disabled auto-merge September 29, 2026 16:00
The cache keeps the vertices a scan loads into it, but their deltas belong to
the loading transaction, and only a transaction that commits is handed to the
cache to outlive its accessor. A scan from a later transaction walks deltas the
loading one has already released.

Both edge import mode scans now share an accessor. The second still reads the
cache rather than reloading it, so it covers the same ground.
@Ignition Ignition added CI -build=community -test=core Run community build and core tests on push CI -build=coverage -test=core Run coverage build and core tests on push CI -build=coverage -test=clang_tidy CI -build=debug -test=core Run debug build and core tests on push CI -build=debug -test=integration Run debug build and integration tests on push CI -build=release -test=core Run release build and core tests on push CI -build=release -test=e2e Run release build and e2e tests on push CI -build=release -test=benchmark Run release build and benchmark on push labels Sep 29, 2026
@sonarqubecloud

Copy link
Copy Markdown

@Ignition
Ignition added this pull request to the merge queue Sep 29, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 29, 2026
@Ignition
Ignition added this pull request to the merge queue Sep 29, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 29, 2026
@Ignition
Ignition added this pull request to the merge queue Sep 29, 2026
Merged via the queue into master with commit f79b917 Sep 29, 2026
40 of 41 checks passed
@Ignition
Ignition deleted the relations/34-compare-two-lists branch September 29, 2026 22:43
@vpavicic vpavicic mentioned this pull request Oct 2, 2026
66 of 75 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug bug CI -build=community -test=core Run community build and core tests on push CI -build=coverage -test=clang_tidy CI -build=coverage -test=core Run coverage build and core tests on push CI -build=debug -test=core Run debug build and core tests on push CI -build=debug -test=integration Run debug build and integration tests on push CI -build=release -test=benchmark Run release build and benchmark on push CI -build=release -test=core Run release build and core tests on push CI -build=release -test=e2e Run release build and e2e tests on push Docs - changelog only Docs - changelog only

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Comparing two booleans or two lists raises instead of ordering them

2 participants