Skip to content

Epic: Fix extraction accuracy gaps found by the tree-sitter accuracy audit #1261

Description

@squid-protocol

Summary

This epic exists because tests/tools/tree_sitter_accuracy_audit.py (built/rolled out in #1227, #1249, #1253) surfaced real, measured func_start/class_start extraction defects in GitGalaxy's structural-signature system.

IMPORTANT SCOPE NOTE: This tool measures ONLY function/class name extraction accuracy (the func_start/args/class_start "extraction gauntlet" pillar from epic #813) -- it does NOT measure GitGalaxy's risk-scoring rules (branch, io, safety_bypasses, concurrency, cryptography, etc.), which are hardened and tested completely separately and are unaffected by anything these measurements show.

What the audit does

  • Scans each language's real-world corpus samples in language-crucible/data/<lang> (pinned corpus snapshot, same one tests/tools/crucible_check.py uses)
  • Parses each file with tree-sitter's grammar to extract the "ground truth" function/class count
  • Compares against GitGalaxy's regex-based extraction on the same files
  • Produces per-language baseline JSON files: tests/tree_sitter_accuracy_baseline_<lang>.json (keys: files_scanned, real_functions, found_functions, extra_functions, real_classes, found_classes, extra_classes, args_comparable, args_exact_match)

Sub-issues filed

The following 5 issues categorize the gaps found, from highest-severity (100% miss rate) through medium (measurement validation needed) to lowest-severity (narrow but safe broadening opportunity):

  1. #[ruby] Ruby func_start matches 0/117 real methods in language-crucible corpus
  2. #[cpp] C++ func_start: 1,147 false positives against only 5/87 real functions found
  3. #[class-detection] Class detection is 0% across apex/csharp/fortran/solidity despite near-perfect function detection
  4. #[c-verify] Verify tree-sitter accuracy measurement's own ground truth for C before treating found_functions=0 as a GitGalaxy defect
  5. #[scala-matlab-haskell] Broaden scala/matlab/haskell func_start regexes -- safe, zero current false positives

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions