[compiler-v2] Extend existing v1/v2 comparison process description #12726
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In `tests/README.md` we had described since a while how the process of porting v1 tests into the v2 tree works. Because of continued misalginment about the process and its motivation, this PR adds some more details to the README outlining the process of test comparison and why it cannot be done with a simple textdiff.
wrwg
requested review from
vineethk,
sausagee,
rahxephon89,
fEst1ck and
brmataptos
⏱️ 1h 20m total CI duration on this PR
🚨 3 jobs on the last run were significantly faster/slower than expected
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #12726 +/- ##
===========================================
- Coverage 71.4% 59.8% -11.7%
===========================================
Files 2400 853 -1547
Lines 485208 208243 -276965
===========================================
- Hits 346557 124530 -222027
+ Misses 138651 83713 -54938 ☔ View full report in Codecov by Sentry. |
vineethk
approved these changes
Mar 28, 2024
Comment on lines
+17
to
+18
| - Any number of bytecode level checkers or transformers (currently `live-var` and `reference-safety` | ||
| and `visibility-checker`) |
There was a problem hiding this comment.
Suggested change
| - Any number of bytecode level checkers or transformers (currently `live-var` and `reference-safety` | |
| and `visibility-checker`) | |
| - Several stackless bytecode level checkers or transformers (eg., `live-var`, `reference-safety`) |
| In order to migrate a test such that the tool can keep track of it, ensure that you place it in a similar named parent directory (anywhere in the v2 test tree). For example, for a test `move-check/x/y.move`, ensure the test can be found somewhere at `x/y.move` in the v2 tree. | ||
| Because of this it is expensive to do test comparison, and essential that we follow the migration | ||
| process as outlined above. Specifically, do _not_ bulk copy tests into the v2 tree without | ||
| manual auditing them, and do _not_ fork tests, even if they are modified, so the relation |
There was a problem hiding this comment.
Just so I am on the same page, what does "forking a test" mean?
There was a problem hiding this comment.
I think it means: leaving the original test unchanged + adding a new copy which is modified. I had argued that we should also run the original test, to guarantee that we at least note some error in the cases where V1 has an error, although it may be a different error due to error shadowing (passes run in different orders, so V2 may show a different error first, then exit before the pass that would display the errors which V1 presents). Here Wolfgang is disagreeing with that approach.
brmataptos
reviewed
Mar 28, 2024
| Because of this it is expensive to do test comparison, and essential that we follow the migration | ||
| process as outlined above. Specifically, do _not_ bulk copy tests into the v2 tree without | ||
| manual auditing them, and do _not_ fork tests, even if they are modified, so the relation | ||
| between v1/v2 tests is maintained. |
There was a problem hiding this comment.
You forgot to mention:
- Review process for every PR must check that all test output checked to ensure that the user-visible
outputs (e.g., error messages and transactional test behavior) do not change.
- As the final goal is to generate errors for programs, test inputs should not be changed again without
clear documentation of why this was done. In particular, if semantics change, they must be
documented.
Perhaps a file could be added to note changes in semantics?
There was a problem hiding this comment.
The Quirks doc may suffice for the short-term, but we need to communicate subtle semantics better.
brmataptos
changed the title
sausagee
approved these changes
Apr 2, 2024
|
This issue is stale because it has been open 45 days with no activity. Remove the |
brmataptos
added
the
stale-exempt
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
✅ Forge suite
|
✅ Forge suite
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
In
tests/README.mdwe had described since a while how the process of porting v1 tests into the v2 tree works. Because of continued misalginment about the process and its motivation, this PR adds some more details to the README outlining the process of test comparison and why it cannot be done with a simple textdiff.Type of Change
Which Components or Systems Does This Change Impact?
How Has This Been Tested?
NA
Key Areas to Review
README.md