Repository navigation
perf(storage,rel): anchor an indexed node lookup by ordinal so (:L {id: $x}) reads the match, not the node topology (#1983) - #2009
Conversation
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Warning Billing warning: we have not been able to collect payment for this subscription for more than 72 hours. Please update the payment method or pay any pending invoices in Billing to avoid service interruption. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…so (:L {id: $x}) reads the match, not the node topology (#1983)
A current persisted equality index names the UUIDs of the nodes an
equality selects. StoredEqualityHints now offers that equality to the
node-topology scan that feeds the property join's UUID key, and the bound
node provider resolves the index result to node ids through the topology
identity columns and reads exactly those rows by ordinal. The label and the
equality stay in the plan; with no current index the scan reads every row as
before.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…acade; document it (#1983) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…und (#1983) Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
2a68943 to
d2db561
Compare
Closes #1983
What
With a current persisted equality index,
(n:L {id: $x})(andMATCH (n:L) WHERE n.id = $x) no longer decodes the whole node topology to meet the one index result. The index returns the matching UUIDs; the topology identity columns resolve them tonode_ids and the fragments that hold them (row-group and page statistics prune everything else); each holding fragment decodes exactly those rows by dense ordinal selection. The label filter and the equality stay in the plan, so the label is checked on the row that was read and the answer is the unanchored scan's answer.EXPLAINnames it:GraphForgeParquetExec: ..., anchor=property_equality(id).graphforge-relStoredEqualityHintsalready offerscolumn = literalto the property scan. It now also offers it, as aGraphReadAnchoron theNodessource, to the node scan that feeds the join'snode_uuidkey, looking only through column projections and total predicates (so no row that could make a predicate fail is dropped early).graphforge-execbinds an anchoredNodessource toCatalog::anchored_node_table.graphforge-storageNodeAnchor/GraphForgeParquetExec: anchored only while the index is current at plan time and again at execute time; otherwise the scan reads every row exactly as before. A corrupt index is refused (GF_VALIDATION), not bypassed. The anchored scan reports its row count as an upper bound, never exact, and is not a "complete node id scan".anchor=annotations.No index, a stale index, an unindexed column or a pending-write union: today's plan and reads, unchanged. Only
filtered_parquet.rsgains an additive node-side function; the edge side is untouched.Tests
New
crates/graphforge-api/tests/indexed_node_anchor.rs(6 tests):MATCH (n:Entity {id: $ident}) RETURN n.id AS id, n.name AS nameat 12,288 and 24,576 nodes, 1/2/4 partitions, three lookups in different fragments. Random UUIDs, so the identity column cannot compress away; 1 KiB incompressible text per node, so the property route spans fragments. Gates: the plan namesanchor=andindex=; at one partition exactly one node row is selected by ordinal and none is read whole (io_stats); read bytes grow by at most 32 KiB, measured both as the calling thread's lifecycle read-path bytes (one partition) and as the process'srchar(every thread, so it holds at 2 and 4 partitions, where the plan's exchanges run on worker threads the lifecycle counters do not see);{}andWHEREforms, labelled and unlabelled;rebuild_property_indexesrestores the anchor;:A:B), nodes sharing a value, a deleted node (DETACH DELETEthen rebuild), and the same answers once the index is stale;stored_equalityunit tests cover where the anchor is placed and where it is not (computed projection, failing filter, non-node source, key not the node identity, already anchored, unsupported equality).Revert proof. With
crates/graphforge-rel/src/stored_equality.rsat main's version (nothing else changed; the rest is inert without it), the growth test fails on the plan assertion (noanchor=). With that assertion removed the byte gate fails too: one partition, observed read bytes 2,685,897 (12,288 nodes) to 2,926,069 (24,576 nodes), +240 KB; processrchar2,719,126 to 3,091,547, +372 KB, the same at 1, 2 and 4 partitions. With the change: 2,714,737 to 2,718,431 observed (+3.7 KB),rchar+2.4 KB. (Run on the pre-#2003 base; the topology path is unchanged by the rebase.)Verification at the PR head (
2a6894317, on main5ad88dd60)cargo test -p graphforge-rel --libcargo test -p graphforge-plan --libcargo test -p graphforge-storage --lib -- topology_identity parquet_scan catalog:: node_anchor property_equality property_scancargo test -p graphforge-api --test indexed_node_anchor --test property_equality_index --test property_read_pushdown --test fixed_hop_limit -- --test-threads=1fixed_hop_limit::fast_paths_survive_multi_file_node_tablescargo test -p graphforge-api --test bddmake checkAll with
CARGO_BUILD_JOBS=2 CARGO_PROFILE_DEV_DEBUG=0, throughscripts/test_environment.py.SF1 measurement
Copies (
cp -a) of/home/ubuntu/gf-sf1-1982-design/{ic,fb}(indexed), workloads/home/ubuntu/gdc-diag-1914b/inputs/{ic,fb}-workload.json,gdc-probe(from/home/ubuntu/gf-probe-1982-design) built--releaseagainst main5ad88dd60and against this branch. Warm process,execute_with_paramsonly, before and after back to back. The host was shared and busy (load average 24-36 over these runs), so absolute times are inflated; read the pairs. Raw outputs are on #1983.Anchor only (
MATCH (p:Person {id: $personId}) RETURN p.id, IC;MATCH (a:Account {id: $id1}) RETURN a.id, FB; ms per repetition, first first):Suites, first three bindings, one warm process per query (s per binding; median):
TCR4 reads slower in that one pair. Four further alternating runs at load 25-40 (main / branch, three bindings each: 9.40 3.94 5.67 / 9.20 4.24 5.01; 6.99 3.96 5.50 / 8.40 3.45 3.42; 4.72 3.03 3.38 / 5.17 2.32 3.22; 4.22 2.63 3.21 / 4.19 2.18 2.61) show no consistent sign, and its physical plan differs from main only by the two
anchor=annotations, so I read it as host noise, not a regression. On a quieter host (load about 4.5) at the previous main (e3716befd): IS1 6.63 s to 5.53 s, IC2 23.73 s to 22.80 s, TCR4 10.32 s to 10.53 s (medians).What the anchor removed: the topology scan. With an id no node holds, the query went from 0.9-1.7 s to 11-21 ms warm. For an id that matches, a warm query reads one of the 49 node fragments (968,795 bytes,
strace -ff) where the first query of the process admitted all of them (60 MB); the identity statistics prune to that fragment. IC2 and TCR4 stay bound by their enrichment and expansion stages, which this slice does not touch.Gaps
idexists (feat(storage,api): persisted property-equality index for node id anchors (#1983) #1998), so onlyidanchors.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.