New test system #262
Merged
New test system #262
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
test-util.ss modifies the original test running code. It is isolated from the spec descriptions (maybe it should be a real library eventually?), and it provides enough information to satisfy the newer requirements for test-runner reporting (for v3 of the runner spec). This also has better usability for the local tests. By default the actual value returned by user defined functions is presented for failing tests, so you can see what is happening without re-entering everything in an repl. It could be extended to be repl friendly as well for users who want to stay in geiser for example. All exception should also be caught and converted to lists that are returned with failures (except for test-error). This should put an end to the mysterious failing tests with unhelpful exception errors. Now, so long as the user code loads it will return all errors encountered during tests as the actual value in the failure report. Maybe we should actually wrap (load solution) with an exception handler to try making better reporting on failing user code? This might be very helpful for recovering in such cases for the test-runner repo. Finished with first draft test-migration.ss modify ci.ss to run all directory tests easily. Add a procedure to redeploy test-util.ss to all exercises.
In a recent PR I approved work that reimplemented existing solutions that didn't work with our current infrastructure. Our CI used an exercise list kept statically in the Makefile, but it was already out of date. The new CI runs for all practice exercises (with a blacklist), so these exercises weren't tested in the CI for the PRs. This compound commit contains the fixes needed by these recent exercises to pass the new CI for both guile and chez. Make sum-of-multiples work for both guile and chez. Make the strain example solution work for both guile and chez. Make the trinary example solution work for both guile and chez. Make the triangle example solution work for both guile and chez.
The CI now runs on all exercises except for the deprecated list-ops and robot-name and the guile-only acronym. All of these changes only work against the tests I am about to commit, but i figured these changes would be better off isolated.
I used configlet to format the exercise configs too after the changes. At this point I believe everything is read to go except for the fact that the test-runner hasn't been updated yet. We should merge the test-runner PR before mergin this.
guygastineau
force-pushed
the
new-test-system
branch
from
5d75611 to
3400047
Compare
This was referenced Jun 11, 2022
This was referenced Jun 15, 2022
Closed
ErikSchierboom
approved these changes
Jun 16, 2022
|
Thank you for the quick review, @ErikSchierboom . Luckily, the new test-runner has already helped the old tests pass where it was crashing before when the implementation wasn't compatible with one of the scheme implementations. Now we will get real v2 reports from these new test and our test runner. |
This was referenced Jul 14, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a pretty huge PR. The new test system is more robust, it has better output for local testing, and it provides enough info for our new test runner (not merged yet check exercism/scheme-test-runner#32) to output fully v2 compatible test results.
This track was neglected for a long time. Part of that is my fault. I tried to break as few things as possible while getting things up-to-date. Some of the track documentation is out of date due to this PR, but we already had bit-rot in our docs. It sounds like some community members are interested in helping with documentation. We'll see how that goes. At the very least, I will try to get some of the nitty gritty details hammered out in some better documentation.
I squashed repetitive actions like working out dual compatibility for a few exercises, and I did some LARGE updates via automation, so these 5 commits are touching a lot of files. The
test-migration.ssprobably doesn't need to be used again exactly, but it has some useful procedures. If nothing else, it will have its useful parts absorbed intooutils.ssandtrack.ss. For now, I think it is fine to leave it in the repository.I hope this is somewhat self-explanatory. I tried to leave good commit messages, but I worked out the majority of the updated test facility, ie.
test-util.ss, while tinkering with my own forth solution, so I don't have commit message for that process here. The majority of the code is really the same. I made some tweaks for robustness and better reporting, and now the tests are all accepted as lists that get evaled with a reflected appened for test_code reporting. In future, I expect we will just use rest args to pack in the source and optional task relations, but adding this later won't be any harder than adding it now, so I am putting it off.