Skip to content

docs(compliance): re-count quality gates; test(grpc): add Layer 2 conformance cases - #27

Merged
HuiJun merged 1 commit into
mainfrom
devin/1786056576-docs-drift-grpc-conformance
Aug 6, 2026
Merged

docs(compliance): re-count quality gates; test(grpc): add Layer 2 conformance cases#27
HuiJun merged 1 commit into
mainfrom
devin/1786056576-docs-drift-grpc-conformance

Conversation

@devin-ai-integration

Copy link
Copy Markdown

Summary

Two things the last batch left behind: stale numbers in the compliance docs, and a §5.2 Layer 2 row for the gRPC wrapper that had been an unmoving to-do. Both are now settled against actual runs on this branch.

Re-counted, not re-asserted. Every number below came from a run or from counting checked-in fixtures, replacing figures that predate the last seven merged PRs:

Claim Was Is
Execution conformance 39/39 43/43
Training corpus 80/100 (TRAINING_EXAMPLES.md said "80% clean") 81/100 — 19 files, 37 errors
Robustness subtests 25 27
Golden AST fixtures / negative parser subtests 23 / 17 24 / 19

The Coverage by Feature Type list also double-counted: its Evaluation: 3 conformance cases are three of the ten calc_* cases, and the per-type numbers summed to 41, not the actual 43. It now sums to 43 by fixture prefix. TRAINING_EXAMPLES.md additionally still listed implicit stdlib typing as an open cause of corpus errors — that was fixed by the implicit-typing work; the two remaining resolution causes (implicit redefinition, and members contributed by perform/references) replace it.

Added the caveat that the corpus gate needs ./scripts/download-training-examples.sh and therefore skips in CI, so a green check is not evidence it passed.

gRPC Layer 2 (❌ → ✅), by adding the cases rather than declaring a deviation. internal/grpc/testdata/conformance/ already contained evaluate_arithmetic and instantiate_part fixtures that no test read — dead data. internal/grpc/conformance_test.go now discovers every .expected.json there and drives the real handlers end-to-end:

parseResp := srv.ParseFile(ctx, &pb.ParseFileRequest{Source: content(model), ContentHash: caseName})
// any diagnostic of severity "error" fails the case
switch tc.RPC {
case "Evaluate":      srv.Evaluate(...)      // expected_result
case "Instantiate":   srv.Instantiate(...)   // expected_slots, type_symbol_id
case "ExecuteAction": srv.ExecuteAction(...) // inputs -> expected_outputs
case "ExecuteState":  srv.ExecuteState(...)  // events -> expected_states_visited
}

Value assertions compare the pb.Value oneof arm as well as the payload, so a right-looking number returned as the wrong type fails. expected_error pins in-band failure modes; execute_action_no_initial uses it to lock the AGENTS.md §4 contract that an empty action errors at initialize(), through the RPC boundary. Schema is documented in that directory's README.md, so new cases are a data-only change.

The two adopted fixtures needed import ScalarValues::*; to parse clean: unlike the runtime-tier harness, ParseFile loads the stdlib and runs the semantic passes, and there is no implicit library import yet.

The doc's gRPC handler table also pointed at line ranges that had drifted by ~40 lines; corrected.

Verification

gofmt -l .                        # empty
go build ./... && go vet ./...    # clean
go test ./...                     # pass
go test -race ./internal/grpc/... # pass
make lint                         # staticcheck + gosec, zero findings
go test ./internal/core/model/ -run TestTrainingExamples   # 81/100 clean, unchanged

Link to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/fd6e8fba638c403f836a5dcab81205e5
Requested by: @HuiJun

…formance cases

Re-counts the stale numbers in docs/SPEC_COMPLIANCE.md and docs/TRAINING_EXAMPLES.md from actual runs (execution conformance 43/43, corpus 81/100, robustness 27, golden AST 24, negative parser 19) and records that the corpus gate skips in CI.

Resolves the standing gRPC Layer 2 gap instead of leaving it as a to-do: internal/grpc/conformance_test.go drives Evaluate, Instantiate, ExecuteAction and ExecuteState from .sysml + .expected.json pairs through the ParseFile RPC, adopting the two fixtures that were checked in but never consumed by any test.

Co-Authored-By: jason.han <jason.han@jpl.nasa.gov>
@HuiJun HuiJun self-assigned this Aug 6, 2026
@devin-ai-integration

Copy link
Copy Markdown
Author
Original prompt from jason.han

@JPL-Devin/Systemica We've recently resynced with the open source branch. Reassess the branch and then continue on with theses next steps, using subsessions for each task that can be completed atomically.
ATTACHMENT:"https://nasa-jpl-demo.devinenterprise.com/attachments/b0443b98-63d4-4bf9-9eb2-6e053a420317/NEXT_SESSION%281%29.md"

@devin-ai-integration

Copy link
Copy Markdown
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@HuiJun
HuiJun merged commit a2d4e11 into main Aug 6, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant