Reduce low-inference memory and bound operation discovery

[forhappy]

Summary

Follow up merged #214 by moving low-inference rejection and proven query admission earlier, releasing build-only memory sooner, bounding incremental clustering, and reducing query recall/hydration work without weakening Compass's native graph contract.

This PR now:

• rejects deferred receivers, heuristic source-backed calls, and inferred external placeholders before low-resolution materialization;
• compacts duplicate test candidates plus resolver candidate/occurrence storage, and streams portable-AST cache publication;
• releases secondary resolver lookup indexes once every resolution decision is fixed;
• materializes resolved provenance-rich edges in deterministic 4,096-record batches, avoids cloning every primary edge, and drops resolver-only lookup IDs after semantic projection;
• consumes the legacy clustering/report graph projection instead of retaining complete typed and compatibility graphs together;
• reuses communities for fact-neutral updates and bounds local reclustering for small topology changes;
• publishes additive source-backed operation-role, declaration, and relationship term indexes with bounded legacy fallbacks;
• intersects exact-term node IDs before hydrating only the surviving multi-concept candidates;
• compares the full specificity rank before reporting natural-query ambiguity, while preserving equal-rank and duplicate exact-name ambiguity;
• preserves explicit path from <symbol> to <symbol> discovery when both endpoint names are exact references;
• proves absent composite identifiers before generic recall;
• defaults natural discovery to a focused 64-node/128-edge neighborhood while retaining 500/1,000 as explicit hard ceilings;
• disables mimalloc's process-wide reserved arena for one-shot builds unless the operator already configured it.
• qualifies all four public inference levels for forced-rebuild byte equality and nested graph breadth;
• reports independent-oracle precision, recall, F1, ambiguity, and scanned/parsed/unsupported coverage overall and by language/relation, while incomplete source populations remain quality-ineligible;
• adds a bounded streaming multiplicity audit proving occurrence-distinct parallel edges and reporting serialized relationship cost by relation;

Why

The pinned delta-rs comparison showed that low publication removed most inferred relationships, but the build still paid much of the allocation cost before filtering. Stage-correlated profiling isolated a resolver-index/materialized-graph lifetime overlap and a typed/legacy publication overlap. Query profiling separately found broad questions hydrating thousands of node records before intersection and then expanding a 500-node neighborhood by default. The changes move decisions and releases to the earliest ownership boundary with enough evidence while retaining bounded reads, explicit truncation, deterministic ordering, provenance, and backend parity.

Real-repository build results

Pinned input: delta-rs. A clean replay of the final admission binary established the current semantic reference: 9,982 nodes, 25,206 relationships, 532 communities, 1,212 structural test roles, and a canonical 34,116,625-byte graph with SHA-256 971d588275cbad097ed1f7b5f54e32b86a80fadd8875760e3848b9948069f573. The earlier 77c49... artifacts remain historical evidence for their recorded binaries; they are not the final-admission comparison reference.

The measured graph-producing head (cc9fa12b) additionally drops resolver-only candidate and occurrence lookup IDs after relation, occurrence rule, exact anchor, endpoints, and provenance are materialized. Three clean sequential samples using the matching default SQLite storage profile completed at 4.30/4.36/4.06 seconds with 455,966,720/455,131,136/462,143,488 bytes maximum RSS:

• median: 4.30 seconds and 434.8 MiB peak RSS;
• every graph was byte-identical at the current semantic-reference SHA-256, including all 1,212 test roles;
• relative to one clean final-admission replay at 4.12 seconds / 461,357,056 bytes, RSS decreased a diagnostic 1.2%; this is not a controlled baseline replacement;
• Compass still used about 2.91x Graphify's retained 149.47 MiB cold RSS.

The cold-build memory target is therefore not met. Dead identity pruning is valid but small; compact or streamed producer evidence and graph-record representation remain the dominant architectural gap. Clustering is not the cold-build peak. A one-file incremental clustering replay remained below its 250 MiB ceiling at 141.11 MiB.

Focused all-level delta-rs matrix

Each level reproduced its exact graph digest on a second forced build:

Level	Nodes	Edges	Communities	JSON bytes	First build	Peak RSS
low	9,982	25,206	532	34,116,625	4.30 s median	434.8 MiB median
medium	9,982	25,219	528	34,137,117	4.43 s	433.0 MiB
high	21,368	56,640	749	85,047,812	5.91 s	722.8 MiB
max	34,384	93,247	1,078	143,936,676	9.02 s	1,120.9 MiB

Only low has three measured samples. The other rows are one measured build plus an unmeasured deterministic rebuild, not promoted baselines. This is focused delta-rs evidence, not the required complete eight-repository all-level matrix.

Multiplicity and serialization evidence

The bounded delta-rs audit reports 25,206 unique edge IDs, 23,209 semantic pairs, and 1,076 parallel pairs containing 3,073 distinct source occurrences. Pair-only coalescing would lose 1,997 real events. There are zero duplicate edge IDs, zero duplicate pair/site records, and zero missing relationship sites.

Relationships serialize to 22,808,206 bytes (66.9% of the canonical graph), averaging 904.9 bytes each. This identifies shared provenance/anchor representation as the valid size target; deleting occurrence-distinct parallel edges is not an acceptable optimization.

Query quality and performance

Independent source-reviewed delta-rs evidence now produces:

• all 29/29 exact required identifiers resolved;
• all 20/20 positive natural labels pass the strict seed/source/direction/ambiguity oracle;
• all 5/5 adversarial negative controls remain no-answer;
• 25/25 total strict rows, with all 500 measured samples correctness-eligible.

The CommitProperties question was independently re-reviewed against the pinned source. Both the Rust-core and Python-facing public declarations configure commit behavior, so the unscoped oracle now requires explicit cross-language ambiguity instead of forcing an oracle-specific language preference.

The final exact-commit harness ran ten fresh CLI processes and ten persistent-MCP requests for each of all 25 labels:

• fresh-process p50 range: 0.0481–0.5500 s;
• fresh-process p95 range: 0.0483–0.5647 s; every row is below 0.59 s;
• positive persistent-MCP p50 range: 0.1613–0.4806 s;
• negative persistent-MCP p50 range: 0.00147–0.00169 s;
• fresh-process peak RSS range: 19.64–84.31 MiB;
• shared MCP server high-water: 147.48 MiB for the complete session, not per query.

The harness performs one unmeasured warmup per row. “Fresh” therefore means a new CLI process over warmed filesystem state; it is not a cold-page-cache claim. Raw evidence is retained outside the repository at /Volumes/Workspace/CrabBuild/compass-performance-pr214/runs/query-default64-final-20260811/ and is not a promoted cross-platform baseline.

Bounded declaration admission

At exact commit 2e9d1455, a complete type-declaration projection reduced decoded candidate IDs across the 25 reviewed questions from 21,543 to 897 in low (95.84%) and from 25,978 to 913 in max (96.49%). The canonical low/max graph SHA-256 values remain 971d5882... and dc87abf2...; SQLite store size grew 1.14% in low and 0.22% in max.

Every low and max matrix passed all 25 independently labeled rows in both 250-sample fresh CLI and 250-sample persistent-MCP modes. Low retained every ranked seed list. Max intentionally changed one rank-3 state-query seed from DeltaTableFactory to the more specific DeltaTable; Top-1 stayed correct, and low/max now agree instead of allowing max-only inferred relationship evidence to promote the factory.

Max fresh aggregate p95 improved from 1.067 to 0.518 seconds and peak RSS from 126.72 to 91.75 MiB; persistent p95 improved from 1.259 to 0.367 seconds and server high-water from 237.52 to 176.23 MiB. Two complete low repeats had stable p50 but alternating runner noise: fresh p95 ranged 0.290–0.609 seconds and persistent p95 0.244–0.542 seconds. Neither better run is promoted as a cross-platform latency baseline.

Compatibility

The v1 discovery request/response schemas and hard ceilings are unchanged. Normal discovery now defaults to 64 nodes and 128 edges; automation that depended on the former 500/1,000 defaults can request --max-nodes 500 --max-edges 1000 explicitly. COMPATIBILITY.md, MIGRATION.md, command/configuration references, the changelog, and contract fixtures document the behavioral default change.

The immutable query accelerators are additive capabilities in existing index roots. Older snapshots remain readable and use bounded fallbacks. Missing capability data never becomes invented semantic evidence.

Validation

All Cargo commands used RUSTC_WRAPPER= and CARGO_TARGET_DIR=/Volumes/Workspace/crabbuild-target/compass-pr214-verify.

• cargo fmt --all -- --check
• cargo clippy --workspace --lib --bins --locked -- -D warnings
• cargo test --workspace --lib --bins --locked
• cargo test -p compass-model --locked
• cargo test -p compass-graph --test store_snapshot --locked
• cargo test -p compass-resolve --locked
• cargo test -p compass-resolve --test universal_resolution --locked (181 passed)
• cargo test -p compass-core --lib --locked
• cargo test -p compass-query --lib --locked
• cargo test -p compass-query --test relevance_qualification --locked
• cargo test -p compass-cli --test code_query_cli --locked
• cargo test -p compass-cli --test compass_product --locked (all four inference levels, forced-rebuild byte equality, nested breadth)
• python3 -m unittest benchmarks.performance.tests.test_config
• python3 -m unittest discover -s benchmarks/performance/tests (205 passed)
• cargo build -p compass-cli --bin compass --release --locked
• sh scripts/check_product_boundary.sh
• ./scripts/qualify_code_graph_v1.sh --fixtures-only
• focused Delta query qualification: 25 rows × 10 fresh + 10 persistent-MCP measured samples, overall PASS
• cargo test -p compass-query --test store_engine --locked (25 passed, including declaration fast-path parity)
• cargo test -p compass-graph --lib --locked (50 passed, including additive-capability fallback)
• cargo clippy -p compass-graph -p compass-query --all-targets --locked -- -D warnings
• focused low and max Delta query qualification: each level 25 rows × 10 fresh + 10 persistent-MCP measured samples, overall PASS; low repeated once

The fixture qualification proved clean/warm/rebuild/restored/alternate-checkout byte equality and passed its enterprise in-process scale ceilings. An additional non-baseline compass-resolve --all-targets --all-features Clippy attempt reports pre-existing expect/panic and style lints in unrelated resolver integration tests; the required workspace --lib --bins Clippy baseline passes.

Pipeline worker reuse

AST extraction now runs as bounded deterministic chunks on the existing Compass pipeline pool instead of creating and destroying a second Rayon pool before resolution. This preserves the established 8/12-worker policy and --max-workers, while allowing resolver phases to reuse extraction allocator pages.

Three fresh pinned delta-rs low-inference SQLite builds completed in 4.49, 4.56, and 4.70 seconds with maximum RSS of 430,489,600, 431,833,088, and 431,194,112 bytes. The medians are 4.56 seconds and 411.2 MiB: 5.4% less RSS and 6.0% more wall time than the preceding 434.8 MiB / 4.30-second median. All three 34,116,625-byte graphs retained SHA-256 971d588275cbad097ed1f7b5f54e32b86a80fadd8875760e3848b9948069f573. Compass still uses about 2.75x Graphify’s retained 149.47 MiB, so this is an incremental lifetime fix, not closure of the memory gate.

Additional validation: cargo test -p compass-core --lib --locked (72 passed) and cargo clippy -p compass-core --all-targets --locked -- -D warnings.

Corrected PHP determinism and low-suite status

The first Laravel qualification exposed a real publication defect: cold edge counts varied between runs. Controlled serial and automatic-worker reproductions localized it to PHP exact/case-folded symbol maps. Random HashMap iteration could select a same-spelled class or method arbitrarily. The current implementation keeps colliding labels unresolved, admits only type-like declarations to the type index, and resolves duplicate qualified types only when same-file or global uniqueness is proven. Extraction semantics moved from version 8 to 9 so stale cached facts cannot mask the correction.

Six controlled Laravel builds (three serial and three automatic-worker) are byte-identical at 38,491 nodes and 58,885 edges. Graph SHA-256 is b7e793da2ae37a822cb67b043c7cc353115f7dd617a93aa1429dd775f8b7cba2; facts SHA-256 is cc8cf40fd0f974639ce0b237a2a822c7fada2d1a5f6caff41daba35826d448af.

A fresh corrected low-inference run completed every build, independent-oracle, fresh-query, persistent-MCP, and 77-query CompassQL workload for four repositories:

Repository	Nodes	Edges	Cold median	Warm median	Incremental median	Restore median	Query materialization
Django	78,772	191,002	16.70 s	1.14 s	5.80 s	8.18 s	27.70 s
Spring	149,792	416,900	61.40 s	2.67 s	2.57 s	2.50 s	89.72 s
Rails	64,814	92,973	8.41 s	0.71 s	4.38 s	4.57 s	15.07 s
Laravel	38,491	58,885	5.85 s	0.73 s	4.10 s	3.82 s	11.81 s

Bevy additionally completed three cold builds at 11.89, 11.70, and 11.92 seconds before the run was stopped at the user's requested reporting boundary. All eight owned qualification checkouts are clean. This evidence is intentionally labeled partial: the complete eight-repository low run, the medium/high/max full suites, and the separate clustered Graphify comparison are not complete and are not promoted baselines.

The latest CI matrix also exposed two under-specified language assertions that selected same-named export wrappers. Commit 51a665b now requires the intended semantic declaration kind. The full compass-languages suite and library Clippy pass locally; the new GitHub matrix is the authoritative cross-platform result.

Highest-priority next optimization

Cold-build product RSS remains the largest gap. Worker reuse reduced the pinned delta-rs low median from 434.8 MiB to 411.2 MiB, but Graphify retained 149.47 MiB, leaving Compass at about 2.75x. Query specificity/startup, bounded clustering, and independently labeled accuracy are no longer the dominant bottlenecks.

The next architectural move should stream or compact declarations, bindings, scopes, and occurrences at the producer/resolver boundary so Compass does not retain complete per-file semantic evidence while materializing full graph records and publication projections. It must be stage-profiled and accepted only with deterministic graph byte equality, multiplicity preservation, and the independent accuracy oracle. Rails/Laravel incremental invalidation is the next performance target after memory: their 4.10-4.38 second incremental medians remain far above their sub-second warm builds.

Post-merge Django low-inference verification

A separate macOS qualification used the exact detached Django checkout at 957d0cee7167757ae221ffde59d2cf0a322e89c7, Compass release binary from PR head 51a665bebe62de023471108b757148bf726f5308, and Graphify package 0.9.36 pinned to upstream tag commit 3d19463484ebcf773b399ddad9fd3363b2ab3bff.

Three serial clustered cold builds completed as follows:

Tool	Samples	Median	Relative result
Compass low	16.56 / 16.38 / 19.40 s	16.56 s	about 2.7x faster
Graphify	about 45 / 45 / 48 s	about 45 s	reference

The Compass artifact contains 78,659 nodes, 190,628 links, and 2,289 communities. Canonical graph.json is 265,959,622 bytes (253.6 MiB). Compass explicitly reported a partial publication after omitting 8,190 nodes and 521 edges and quarantining one identity collision.

Graphify produced 50,832 nodes and 158,684 edges in every cold sample, but reported 1,901 / 1,912 / 1,901 communities. Therefore the structural counts were stable while community assignment was not byte-stable. This evidence supports a real roughly 2.7x low-inference cold-build advantage on this Django checkout, but does not meet the 5x target and is not an all-level or cross-platform promoted baseline.

[forhappy]

Final-head inference-level smoke qualification on pinned delta-rs (0df5340e, release build):

Level	Nodes	Relationships	Inferred relationships	Communities	Wall time	Max RSS
low	9,982	25,206	0	532	2.44 s p50 (3 samples)	610.7 MiB p50 (3 samples)
medium	9,982	25,219	13	528	3.47 s (1 sample)	616.2 MiB (1 sample)
high	21,368	56,640	31,434	749	4.45 s (1 sample)	857.6 MiB (1 sample)
max	34,384	93,247	68,041	1,078	5.91 s (1 sample)	1,340.3 MiB (1 sample)

All builds completed validation and publication. Node/relationship/inferred counts are monotonically nested by level; max exactly retains the documented 34,384-node / 93,247-relationship compatibility counts. Only low has a three-sample median here; the other rows are explicitly single-sample smoke evidence and should not be presented as promoted performance baselines.

[forhappy]

Pushed 2720b6ca with the bounded operation-role index, subject-complete action ranking, backend parity coverage, and one-shot allocator mitigation. The full native baseline, product boundary, CLI contract, 500-query relevance suite, and fixture code-graph qualification pass. The PR body now records the exact delta-rs quality/performance evidence and explicitly leaves the 3.18x Graphify cold-RSS gap and default-neighborhood startup latency unresolved.