[WIP]Test upgrade #261
Closed
[WIP]Test upgrade #261
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
test-util.ss modifies the original test running code. It is isolated from the spec descriptions (maybe it should be a real library eventually?), and it provides enough information to satisfy the newer requirements for test-runner reporting (for v3 of the runner spec). This also has better usability for the local tests. By default the actual value returned by user defined functions is presented for failing tests, so you can see what is happening without re-entering everything in an repl. It could be extended to be repl friendly as well for users who want to stay in geiser for example. All exception should also be caught and converted to lists that are returned with failures (except for test-error). This should put an end to the mysterious failing tests with unhelpful exception errors. Now, so long as the user code loads it will return all errors encountered during tests as the actual value in the failure report. Maybe we should actually wrap (load solution) with an exception handler to try making better reporting on failing user code? This might be very helpful for recovering in such cases for the test-runner repo.
guygastineau
force-pushed
the
test-upgrade
branch
from
95f5580 to
3b5a202
Compare
|
This is now obsolete in favor of #262 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR isn't actually intended for merging at least right now. I will try to work out a tool to automatically convert existing exercises to the new test system. This new test system allows better conformance with the test runner specification, and our test runner has a new PR for handling this new test output exercism/scheme-test-runner#32 . The new runner maintains legacy compatibility, so even if the "update everything at once" script doesn't pan out or if it only works for most exercises, we can still move forward without breaking everything in the test-runner. For now, hopefully this PR shows some movement on our current problems.
NB. I worked this test system out before I read about tasks and v3 of the test runner interface spec. Even though we have no concept exercises it might be worth while including an escape hatch for that data now rather than later. Although we start getting to a point where it seems like we should just define a new record type to hold all the test data. Well, maybe someone will have a discussion with me about it here...