Summary
This epic exists because tests/tools/tree_sitter_accuracy_audit.py (built/rolled out in #1227, #1249, #1253) surfaced real, measured func_start/class_start extraction defects in GitGalaxy's structural-signature system.
IMPORTANT SCOPE NOTE: This tool measures ONLY function/class name extraction accuracy (the func_start/args/class_start "extraction gauntlet" pillar from epic #813) -- it does NOT measure GitGalaxy's risk-scoring rules (branch, io, safety_bypasses, concurrency, cryptography, etc.), which are hardened and tested completely separately and are unaffected by anything these measurements show.
What the audit does
- Scans each language's real-world corpus samples in
language-crucible/data/<lang> (pinned corpus snapshot, same one tests/tools/crucible_check.py uses)
- Parses each file with tree-sitter's grammar to extract the "ground truth" function/class count
- Compares against GitGalaxy's regex-based extraction on the same files
- Produces per-language baseline JSON files:
tests/tree_sitter_accuracy_baseline_<lang>.json (keys: files_scanned, real_functions, found_functions, extra_functions, real_classes, found_classes, extra_classes, args_comparable, args_exact_match)
Sub-issues filed
The following 5 issues categorize the gaps found, from highest-severity (100% miss rate) through medium (measurement validation needed) to lowest-severity (narrow but safe broadening opportunity):
- #[ruby] Ruby func_start matches 0/117 real methods in language-crucible corpus
- #[cpp] C++ func_start: 1,147 false positives against only 5/87 real functions found
- #[class-detection] Class detection is 0% across apex/csharp/fortran/solidity despite near-perfect function detection
- #[c-verify] Verify tree-sitter accuracy measurement's own ground truth for C before treating found_functions=0 as a GitGalaxy defect
- #[scala-matlab-haskell] Broaden scala/matlab/haskell func_start regexes -- safe, zero current false positives
References
Summary
This epic exists because
tests/tools/tree_sitter_accuracy_audit.py(built/rolled out in #1227, #1249, #1253) surfaced real, measuredfunc_start/class_startextraction defects in GitGalaxy's structural-signature system.IMPORTANT SCOPE NOTE: This tool measures ONLY function/class name extraction accuracy (the
func_start/args/class_start"extraction gauntlet" pillar from epic #813) -- it does NOT measure GitGalaxy's risk-scoring rules (branch,io,safety_bypasses,concurrency,cryptography, etc.), which are hardened and tested completely separately and are unaffected by anything these measurements show.What the audit does
language-crucible/data/<lang>(pinned corpus snapshot, same onetests/tools/crucible_check.pyuses)tests/tree_sitter_accuracy_baseline_<lang>.json(keys:files_scanned,real_functions,found_functions,extra_functions,real_classes,found_classes,extra_classes,args_comparable,args_exact_match)Sub-issues filed
The following 5 issues categorize the gaps found, from highest-severity (100% miss rate) through medium (measurement validation needed) to lowest-severity (narrow but safe broadening opportunity):
References
tests/tools/tree_sitter_accuracy_audit.pytests/tree_sitter_accuracy_baseline_<lang>.jsongitgalaxy/standards/how_to_add_a_language.md