fix(core): add args regex capture group for 7 more languages (#1209) - #1216
Merged
Conversation
Completes #1209's Tier 1 mechanical list: csharp, java, javascript, typescript, groovy, ruby, powershell (the remaining 7 of 20, following PR #1212's first 13). Same root cause as #1199/#1212: without a capture group isolating the parameter-list span, detector.py's _calculate_block_metrics falls back to the WHOLE regex match, and a zero/one-arg signature with no comma hits a whitespace-split fallback meant for space-separated languages, overcounting by +1. These 7 are the genuinely multi-branch regexes flagged as a distinct, costlier follow-up when PR #1212 stopped -- each has 3-4 real alternatives (standard method, constructor, lambda/arrow-function, class-method shorthand) that all needed their own capture group, not just one: - csharp/java: standard method + constructor + lambda/method-ref branches. - javascript/typescript: function decl + arrow function + class-method shorthand (typescript also has a bare-identifier arrow branch). - groovy: standard method + closure (`->`), where the closure branch's params are entirely optional -- needed an empty-string alternative added inside the group so a zero-param closure (`{ ->`) still has something to capture (the shared test harness requires a non-empty captured-groups list once any group exists in the pattern; same shape as perl's bare `shift` fix in PR #1212). - ruby: def-with-parens + `do |...|` + `{ |...| }` + stabby lambda `->(...)`, four independent parameter spans. - powershell: `param(...)` block + `function name(...)` + class-method constructor-shaped call. tests/extraction/languages/test_ruby.py's test_ruby_args needed updating: its "expected" values were the WHOLE matched payload (a different, older convention than every other language's name-based expected values), which no longer holds now that the match is split across capture groups. Switched the 21 cases to check the method name (or parameter span for blocks/lambdas) instead -- this doesn't lose coverage, since assert_valid_match still separately asserts the payload matches at all regardless of what "expected" is. Extends the ARGS_COUNT_FIXTURES regression suite added in PR #1212 with these 7 languages (7 new parametrized real-pipeline cases, 21 total now). Golden master fixtures regenerated via update_golden_master.py per the Differential Scan protocol -- confirmed clean by crucible_check.py afterward (both full-precision and zero-dependency modes). #1209's remaining scope: the 5-language Tier 2 (objective-c/scheme/ matlab/haskell/fortran), structurally unlike Python's single-paren- comma-list model and each needing bespoke handling. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
5 tasks
squid-protocol
added a commit
that referenced
this pull request
Aug 10, 2026
… + Tier 2) (#1218) Fixes the 5 Tier 2 languages: objective-c, scheme, matlab, haskell, fortran. Unlike Tier 1's mechanical capture-group port, each of these has a signature shape that doesn't fit Python's single-paren-comma-list model at all, so each needed its own counting strategy, not just a regex tweak: - objective-c: keyword-message selectors (`doThing:(int)x withOther:(int)y`) scatter one parameter per repeated `label:(Type)name` segment across the signature -- there's no comma-separated list to count at all. The args regex's first branch now captures the WHOLE repeated selector span (was matching only the FIRST segment, meaning every multi-param method silently undercounted to 1 regardless of real arity, and could even false-match a `:(Type)` cast sitting inside an unrelated method's body). New `_count_colon_selector_segments` counts top-level `:(` occurrences instead of commas. Also fixed a real, pre-existing-shape ReDoS: adding an optional leading-label group to match 2nd+ segments introduced unbounded `\w*` backtracking at every string position (confirmed quadratic via scaling sweep before bounding it to `{0,80}`). - scheme: `(define (name a b) ...)` arguments are space-separated inside the SAME parens as the function name, not comma-separated. Split the name and parameter-list into separate capture groups so the existing whitespace-split fallback (already used for genuinely space-separated languages) sees just "a b", not "(define name a b)". Also fixed a real, pre-existing ReDoS in this exact regex found while verifying the fix: two adjacent unbounded quantifiers over near-identical character classes produced confirmed O(n^2) behavior (247x time for 16x input) on an unterminated `(define (xxxx...` with no closing paren -- bounding the name quantifier to `{1,100}` restores linear behavior. This ReDoS predates #1209 entirely; the capture-group refactor just meant actually re-deriving and scaling-testing this regex for the first time in a while. - matlab: `function [out1, out2] = name(in1, in2)` has an output-variable list before the `=` in addition to the real input-parameter parens. The new capture group targets ONLY the trailing input parens, so the output list is never miscounted as if it were part of the input signature (a distinct bug from the generic overcount, specific to MATLAB's syntax). - haskell: `name :: Int -> Int -> Int` signatures don't use parens or commas at all -- curried arity is the top-level arrow count. New `_count_haskell_type_arrows` counts top-level "->" after skipping any leading typeclass-constraint clause (`Show a => ...`) and ignoring an arrow nested inside a higher-order parameter's own type (`(Int -> Int) -> Int`). Gated on a new `_args_arrow_count_groups` flag in haskell's rules dict (an underscore-prefixed metadata key, matching the existing `_dependency_capture` convention) naming which SPECIFIC capture-group index is a type signature -- content-sniffing for "->" can't work here, since a real signature can have ZERO arrows (`noop :: IO ()`) and still need a correct zero count. Also hardened language_lens.py's two rule-iteration loops and test_language_standards_strict.py's global regex-integrity sweep to skip underscore-prefixed keys (matching detector.py's own `coding_analysis` convention already) -- they previously assumed every value in a language's `rules` dict was a compiled regex, which the new non-regex flag value would have broken (caught exceptions / the global integrity test, not silent wrongness, but worth aligning properly). - fortran: `SUBROUTINE Foo` (no parens at all) is valid zero-arg syntax alongside `SUBROUTINE Foo(a, b)`. The args group's capture now includes a trailing empty alternative so it participates even when the parens are genuinely absent -- without it, a bare zero-arg subroutine's only captured content would be the name itself, miscounted as 1 argument via the generic whitespace-split fallback. Two of these languages (scheme, matlab -- plus haskell/fortran for functions without an explicit signature or with expression-body syntax) aren't actually extracted by the real detector pipeline at all: they use paren-/keyword-delimited scoping, not braces, and the generic slicer dispatcher's Mode B brace-slicer never finds a body for them. This is a real, pre-existing recall gap, but a separate one from args-counting (out of scope here -- #1209 only fixes the count once a function IS found). New regression tests for these call `_calculate_block_metrics` directly with a hand-built block, bypassing the slicer, to isolate and pin the counting fix itself rather than being blocked by the unrelated gap. New tests: `objective-c` added to the existing ARGS_COUNT_FIXTURES real-pipeline suite; a new ARGS_COUNT_FIXTURES_DIRECT_BLOCK suite for scheme/matlab/haskell/fortran; direct unit coverage for both new counting helpers (`_count_colon_selector_segments`, `_count_haskell_type_arrows`). This closes #1209 -- all 25 languages in scope (20 Tier 1 + 5 Tier 2) are now fixed, across PRs #1207 (Python, the originating fix), #1212 (13 languages), #1216 (7 languages), and this one (5 languages). Golden master fixtures regenerated via update_golden_master.py per the Differential Scan protocol -- the diff is dominated by objective-c (the only Tier 2 language the real pipeline actually extracts at scale) plus a handful of real fortran corpus files, confirmed clean by crucible_check.py afterward (both full-precision and zero-dependency modes). Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2 tasks
squid-protocol
added a commit
that referenced
this pull request
Aug 10, 2026
…pass (#1223) * fix(core): critical-leak synthetic risk_vector hardcoded to stale 18-length RISK_SCHEMA galaxyscope.py's CRITICAL LEAKS synthetic-node path (forces a file flagged by the Aperture secrets scanner onto the 3D map even though it never went through normal parsing) built risk_vector as a hardcoded literal: `[0.0] * 13 + [0.0, 0.0, 0.0, 0.0, 100.0]` -- an 18-element vector with a comment claiming "Index 17 is secrets_risk". RISK_SCHEMA is actually 13 elements now (secrets_risk at index 12), so this path has been emitting a 5-element-too-long risk_vector for a while. Harmless on its own, but record_keeper.py's SQLite INSERT builds its column list from the live RISK_SCHEMA length -- any file taking this path made record_mission() raise `sqlite3.OperationalError: N values for M columns` and abort `--db-only` output entirely for the whole scan, not just that one file. Found while building a tree-sitter-based ground-truth accuracy pass for JavaScript (mirroring ast_accuracy_audit.py's Python methodology, #1200): `galaxyscope <path> --db-only` crashed on expressjs/express because its committed `.npmrc` trips the hardcoded-secrets detector. Reproduced, isolated to this exact literal via a cursor.execute proxy that dumped the column/value counts at the failing INSERT, confirmed by reverting it. Fix: size risk_vector from `len(SignalProcessor.RISK_SCHEMA)` (matching the sibling AI-MODEL-WEIGHTS synthetic-node path a few lines down, which already did this correctly) and set the secrets_risk slot by name lookup instead of a hardcoded index, mirroring the hit_vector/sec_hardcoded_secrets pattern already used right below it. tests/core_engine/test_galaxyscope.py::test_synthetic_node_generation asserted `risk_vector[17] == 100.0` -- also stale, and silently correct by coincidence since the old 18-length hardcoded vector still had a real value at index 17. Updated to assert against the live RISK_SCHEMA length and the schema-resolved secrets_risk index, so a future RISK_SCHEMA resize can't silently reintroduce this drift. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * docs: add JavaScript language-status doc, first tree-sitter-based §9 pass Follows python.md's structure (language-status skill), extending its §9 "measured accuracy" methodology beyond Python for the first time using tree-sitter-language-pack against real code -- exactly the path python.md's own §9 proposed for scaling past Python's stdlib `ast`. Measured against two corpora with different shapes (expressjs/express v5.2.1: small, mostly top-level functions and middleware callbacks; and GitGalaxy's own site/js/ WebGPU visualizer: heavily class-based). Confirms #1209/#1216's args capture-group fix works on real code (100% args-count exact match on every function found, both corpora) and surfaces: - A blocking infra bug that crashed --db-only on any repo with a flagged secret (fixed same-day in #1220, needed before this measurement could run at all -- express's own committed .npmrc trips the detector). - #1221 (open): func_start's method-shorthand branch has no trailing-`{` requirement, unlike args' own "Invocation Shield" for the same shape -- bare call statements (`next();`) get misidentified as definitions. Confirmed to also affect typescript/java/csharp/apex/dart/groovy. - #1222 (open): real, differently-named functions can be silently dropped from function_data even though func_start's regex finds them correctly -- most severe for ES6 class methods (one file lost 9 of 10 real methods). Likely the same _slice_by_braces mechanism #789 diagnosed and left unfixed for csharp. Neither new defect fixed here (docs-only pass, per the language-status skill's scope discipline) -- both need the fuller harden-language-extraction treatment given they touch shared detector.py slicing logic. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes #1209's Tier 1 mechanical list:
csharp, java, javascript, typescript, groovy, ruby, powershell(the remaining 7 of 20, following PR #1212's first 13).Same root cause as #1199/#1212: without a capture group isolating the parameter-list span,
detector.py's_calculate_block_metricsfalls back to the WHOLE regex match, and a zero/one-arg signature with no comma hits a whitespace-split fallback meant for space-separated languages, overcounting by +1.These 7 are the genuinely multi-branch regexes flagged as a distinct, costlier follow-up when PR #1212 stopped — each has 3-4 real alternatives (standard method, constructor, lambda/arrow-function, class-method shorthand) that all needed their own capture group, not just one:
csharp/java: standard method + constructor + lambda/method-ref branches.javascript/typescript: function decl + arrow function + class-method shorthand (typescript also has a bare-identifier arrow branch).groovy: standard method + closure (->), where the closure branch's params are entirely optional — needed an empty-string alternative added inside the group so a zero-param closure ({ ->) still has something to capture (the shared test harness requires a non-empty captured-groups list once any group exists in the pattern; same shape as perl's bareshiftfix in PR fix(core): add args regex capture group for 13 languages (#1209) #1212).ruby: def-with-parens +do |...|+{ |...| }+ stabby lambda->(...), four independent parameter spans.powershell:param(...)block +function name(...)+ class-method constructor-shaped call.tests/extraction/languages/test_ruby.py'stest_ruby_argsneeded updating: its "expected" values were the WHOLE matched payload (a different, older convention than every other language's name-based expected values), which no longer holds now that the match is split across capture groups. Switched the 21 cases to check the method name (or parameter span for blocks/lambdas) instead — this doesn't lose coverage, sinceassert_valid_matchstill separately asserts the payload matches at all regardless of whatexpectedis.Regression coverage: extends the
ARGS_COUNT_FIXTURESsuite added in PR #1212 with these 7 languages (7 new parametrized real-pipeline cases, 21 total now).Remaining scope on #1209: the 5-language Tier 2 (
objective-c/scheme/matlab/haskell/fortran), structurally unlike Python's single-paren-comma-list model and each needing bespoke handling.Type of change
gitgalaxy/standards/language_standards.py)CI checklist
python tests/tools/audit_check.pypassespython -m pytest tests/passes (6808 passed)python tests/tools/crucible_check.pypasses (both full-precision and zero-dependency modes)Differential Scan target
Corpus-wide fix across 7 languages, not a single-repo target.
golden_master_audit.json/golden_master_zero_dep_audit.jsonregenerated viaupdate_golden_master.pyagainst the standard ~80-repo language-crucible baseline — the diff is the real, intentional args-count correction across ruby/typescript/javascript/dart(shared corpus files)/etc., confirmed clean bycrucible_check.pyafterward.Verification
Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com