docs(compliance): re-count quality gates; test(grpc): add Layer 2 conformance cases - #27
Merged
Merged
Conversation
…formance cases Re-counts the stale numbers in docs/SPEC_COMPLIANCE.md and docs/TRAINING_EXAMPLES.md from actual runs (execution conformance 43/43, corpus 81/100, robustness 27, golden AST 24, negative parser 19) and records that the corpus gate skips in CI. Resolves the standing gRPC Layer 2 gap instead of leaving it as a to-do: internal/grpc/conformance_test.go drives Evaluate, Instantiate, ExecuteAction and ExecuteState from .sysml + .expected.json pairs through the ParseFile RPC, adopting the two fixtures that were checked in but never consumed by any test. Co-Authored-By: jason.han <jason.han@jpl.nasa.gov>
Author
Original prompt from jason.han
|
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two things the last batch left behind: stale numbers in the compliance docs, and a §5.2 Layer 2 row for the gRPC wrapper that had been an unmoving to-do. Both are now settled against actual runs on this branch.
Re-counted, not re-asserted. Every number below came from a run or from counting checked-in fixtures, replacing figures that predate the last seven merged PRs:
TRAINING_EXAMPLES.mdsaid "80% clean")The
Coverage by Feature Typelist also double-counted: itsEvaluation: 3 conformancecases are three of the tencalc_*cases, and the per-type numbers summed to 41, not the actual 43. It now sums to 43 by fixture prefix.TRAINING_EXAMPLES.mdadditionally still listed implicit stdlib typing as an open cause of corpus errors — that was fixed by the implicit-typing work; the two remaining resolution causes (implicit redefinition, and members contributed byperform/references) replace it.Added the caveat that the corpus gate needs
./scripts/download-training-examples.shand therefore skips in CI, so a green check is not evidence it passed.gRPC Layer 2 (❌ → ✅), by adding the cases rather than declaring a deviation.
internal/grpc/testdata/conformance/already containedevaluate_arithmeticandinstantiate_partfixtures that no test read — dead data.internal/grpc/conformance_test.gonow discovers every.expected.jsonthere and drives the real handlers end-to-end:Value assertions compare the
pb.Valueoneof arm as well as the payload, so a right-looking number returned as the wrong type fails.expected_errorpins in-band failure modes;execute_action_no_initialuses it to lock the AGENTS.md §4 contract that an empty action errors atinitialize(), through the RPC boundary. Schema is documented in that directory'sREADME.md, so new cases are a data-only change.The two adopted fixtures needed
import ScalarValues::*;to parse clean: unlike the runtime-tier harness,ParseFileloads the stdlib and runs the semantic passes, and there is no implicit library import yet.The doc's gRPC handler table also pointed at line ranges that had drifted by ~40 lines; corrected.
Verification
Link to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/fd6e8fba638c403f836a5dcab81205e5
Requested by: @HuiJun