feat(tools): add an advisory pilot-implementation diagnostic differential

[devin-ai-integration]

Summary

training_examples_expected.txt answers "did we change?"; it cannot answer "are we right?" — it is a snapshot of our own behavior, so a regression re-baselines as quietly as a fix. This adds the other half: run the OMG SysML v2 Pilot Implementation (the reference) over the same corpora and bucket the differences. Advisory only — no CI gate, no touch to that snapshot, no stdlib edits, no diagnostic fixes (all findings are follow-ups F1–F7 in the doc).

• scripts/download-pilot-validator.sh — builds DeciSym's sysmlv2-validator at pinned commit 0d706e5b into build/pilot-validator/, and fails loudly if that commit's pom.xml no longer pins pilot 2026-05 / 0.60.1, i.e. the same release scripts/download-training-examples.sh pins.
• cmd/pilot-diff — runs both analyzers over examples/sysml-v2-training, testdata/, examples/ and a probes root, normalizes to (file, line, severity, coarse category), and writes build/pilot-diff/pilot-diff.{txt,json}. Message text is never compared; unclaimed messages stay unmapped and are also listed in a table of their own.
• docs/project/pilot-differential.md + a committed pilot-differential-baseline.json so a later run diffs against this one.

Wrapper choice. Fabi303/sysmlv2tool is the better fit (true directory batch) but does not build here — it builds the pilot from a submodule via Tycho and dies with No implementation for org.eclipse.tycho.core.resolver.MavenTargetLocationFactory was bound under both Maven 3.6.3 and 3.9.9. That is F6, not a faked result. DeciSym builds, but validates each file with a separate interactive.process(content, true) against one accumulating session, so the harness compensates and the doc records what leaks through:

order.go:  topologically sort each root by its imports (our parser finds decls/imports)
           → a file that imports another is processed after it; cycles fall back to sorted
pilot.go:  files sharing a basename go in separate invocations (diagnostics are keyed by basename)
leak:      one session ⇒ same-named packages in different files collide
           ⇒ 30 × "Duplicate of other owned member name" — a wrapper artifact, labelled as such

What the comparison found (122 files: 100 fully agreeing, 18 agreed diagnostics, 1 severity-only, 9 only ours, 188 only the pilot's):

• On the 100-file OMG training corpus the pilot reports nothing, and we report two — both ours are wrong: state on { ... } ... then on; trips "on" is a reserved keyword (F1). That row is the like-for-like verdict.
• The 188 pilot-only entries are concentrated in our fixtures and traced to root causes, not left as a pile: a bare import X::*; (10 files) is rejected because the pilot's grammar makes visibility mandatory — ImportPrefix : visibility = VisibilityIndicator 'import' — and its recovery then cascades into most of the no viable alternative / missing EOF / unresolved-Real entries; plus two genuine over-acceptances of ours (namespace N; and region in a state body are not in the pilot's SysML grammar at all).
• New severity-only bucket: part few subsets cap [0..10] under cap [0..3] is an error for us and a warning for the pilot — kept apart so it is neither counted as agreement nor double-counted as two disagreements.

The alias case that motivated this is now a probe (cmd/pilot-diff/testdata/alias-supertype-lru.sysml, part def AvionicsLRU :> Box { :>> length = 100 [mm]; }): the pilot accepts it; at #331's merge base we reported part cannot specialize alias (kind mismatch); on main with #331 we accept it and agree. That is the class of question this harness exists to ask.

Gate: go build ./..., go vet ./..., gofmt -l . (empty), go test ./... all green, with unit tests for the categorization, multiset/severity bucketing, import ordering (including a cycle), and the pilot line parser.

Link to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/a87d0b1126e94828b2260ff5ff58e5af
Requested by: @HuiJun

---

[devin-ai-integration]

Original prompt from Devin Bot

Build a differential diagnostics harness for JPL-Devin/OpenSysML that compares our analyzer's verdicts against the OMG SysML v2 Pilot Implementation (the reference implementation), so we can tell whether a diagnostic we emit — or fail to emit — matches the reference.

Motivation: a user hit unresolved reference: length on part def AvionicsLRU :> Box { :>> length = 100 [mm]; } because ShapeItems::Box is alias Box for RectangularCuboid and our resolver did not follow aliases through type relationships (fixed in PR #331). We had no way to check that class of behavior against the reference. Today docs/project/training-examples.md gates on internal/core/model/testdata/training_examples_expected.txt, which is a snapshot of OUR behavior, not an oracle — read that doc first; it explains exactly why that snapshot is not a substitute for what you are building.

What to build:

1. A script (put it under scripts/, following the conventions of the existing scripts there, especially scripts/download-training-examples.sh) that provisions a headless pilot-implementation validator. Do not write your own bridge into the pilot's Xtext internals; use an existing thin CLI wrapper around the pilot jars. Evaluate these and pick one, recording why in the PR:
   • https://github.com/DeciSym/sysmlv2-validator — emits GNU-format file:line:col: severity: message, single files.
   • https://github.com/Fabi303/sysmlv2tool — java -jar ... validate <path|dir>, loads a whole directory as one batch so cross-file imports resolve, text or JUnit-XML output, exit 0/-1.
     Batch/directory loading matters: several OMG corpus files only resolve when the whole corpus is in context. Pin whatever you use to a specific tag/commit and pin the pilot version, the way the training-examples download script pins 2026-05.
2. A comparison harness that runs both analyzers over a corpus — the OMG training corpus fetched by scripts/download-training-examples.sh plus our own testdata/ and `example... (2296 chars truncated...)

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

Runtime verification — pilot differential harness

Verified end to end on Linux (Java 21.0.11, Maven, Go 1.23). Gate green, harness deterministic and baseline-reproducing, doc numbers exact, failure modes clean.

Reproduces the committed baseline, twice, and after a from-scratch validator rebuild

Two consecutive go run ./cmd/pilot-diff runs are byte-identical to each other and to docs/project/pilot-differential-baseline.json (jq -S diff empty). scripts/download-pilot-validator.sh was also allowed to do a full rebuild from the pinned commit (~3 min), and the rebuilt validator reproduces the same baseline byte-for-byte — so the numbers are a property of pilot 2026-05 / 0.60.1, not of one local build.

Every number in docs/project/pilot-differential.md matches the report

totals = {files 122, fullyAgreeing 100, ours 28, pilot 207, agreement 18, severityMismatch 1, oursOnly 9, pilotOnly 188}; per-root rows training 100/98/2/0/0/0/2/0, testdata 9/0/24/67/18/1/5/48, examples 12/1/2/140/0/0/2/140, probes 1/1/0/0/0/0/0/0; the unmapped table matches verbatim (30/5/2/1/1 pilot, 1/1 ours).

Alias probe clean on both sides (pilot direct, exit code captured without a pipe)

validate-sysml cmd/pilot-diff/testdata/alias-supertype-lru.sysml > /tmp/probe.out 2>&1 → exit 0, zero lines mentioning the file. go run ./cmd/sysml -validate on the same file → ✓ no errors, exit 0. The probes root in the report is files: [] with all-zero buckets.

Gate: build / vet / gofmt / go test ./... / go test ./cmd/pilot-diff

Harness failure modes

Command	Result
-validator /nonexistent	exit 1, pilot validator not found at /nonexistent: run ./scripts/download-pilot-validator.sh
-out /tmp/pd-out	exit 0, both reports written there, baseline-identical
-repo /tmp/notrepo	exit 1, validator-not-found under that repo
-repo /tmp/notrepo -validator <abs>	previously exit 0 with files: 0; now exit 1 (no .sysml files found under ...) — fixed in 5233a12
-timeout 1s	exit 1, validate-sysml failed (signal: killed), no report written, no hang

Re-provisioning: no-op, bad pin, missing mvn

Already built → Pilot validator already built at ..., exit 0, validate-sysml mtime unchanged (no rebuild):

PILOT_TAG=9999-99 with the build moved aside → clones, then aborts before Maven (target/*.jar absent), exit 1:

env PATH=/tmp/nomvn ... (git/java/sed/dirname present, no mvn) → error: mvn is required to build the pilot validator, exit 1:

CI's make lint (gosec) also flagged the new files; fixed in 5233a12 by tightening report file/dir modes to 0600/0750 and annotating the two corpus reads, with make lint now passing locally.

Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/a87d0b1126e94828b2260ff5ff58e5af