-
-
Notifications
You must be signed in to change notification settings - Fork 615
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor the test runner #3646
Refactor the test runner #3646
Commits on Jun 2, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 8ae375d - Browse repository at this point
Copy the full SHA 8ae375dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7d4da6e - Browse repository at this point
Copy the full SHA 7d4da6eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1875e7c - Browse repository at this point
Copy the full SHA 1875e7cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 558c330 - Browse repository at this point
Copy the full SHA 558c330View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8137b9d - Browse repository at this point
Copy the full SHA 8137b9dView commit details -
Configuration menu - View commit details
-
Copy full SHA for fc4f6b8 - Browse repository at this point
Copy the full SHA fc4f6b8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 50dffaf - Browse repository at this point
Copy the full SHA 50dffafView commit details -
Share
libc
signal definitions with more platformsI confirmed that these 3 platforms share the same signal definitions from these sources. Haiku: https://github.com/haiku/haiku/blob/master/headers/posix/signal.h OpenBSD: https://github.com/openbsd/src/blob/master/sys/sys/signal.h NetBSD: http://fxr.watson.org/fxr/source/sys/signal.h?v=NETBSD
Configuration menu - View commit details
-
Copy full SHA for 95c2e02 - Browse repository at this point
Copy the full SHA 95c2e02View commit details -
Changes - Support multi-threaded testing. - Support `set_fail_timeout` on all platforms. - Display an animated progress report. - Setup all tests with a context logger. - Give all tests their own separate custom allocators. - Support tracking test memory usage. - Display a summary of the failed tests at the end. - Let users select only specific tests to run. - Support copying failed tests to the clipboard to run again. - Support catching SIGINT (CTRL-C) to cancel early. - Record context in cleanup procs. - Write all log messages to STDERR for easy redirection. - Possibly more I've forgotten. New Options - `-define:test_threads=N`: Specify thread count. - `-define:test_thread_memory=B`: Specify initial memory block size in bytes to each thread. - `-define:test_track_memory=true`: Track the memory usage of individual tests. - `-define:test_fancy=false`: Disable animated progress report. - `-define:test_select=package.test_name,...`: Run only select tests. - `-define:test_clipboard=true`: Copy names of failed tests to the clipboard. - `-define:test_progress_width=24`: Change the width of the animated progress bars.
Configuration menu - View commit details
-
Copy full SHA for b6c4dfb - Browse repository at this point
Copy the full SHA b6c4dfbView commit details -
Configuration menu - View commit details
-
Copy full SHA for d030240 - Browse repository at this point
Copy the full SHA d030240View commit details -
Temporarily disable Windows-specific test runner
I do not have a Windows machine to test the refactored test runner, and I am unsure if it would even run correctly on Windows without this disabled.
Configuration menu - View commit details
-
Copy full SHA for 87ea4a2 - Browse repository at this point
Copy the full SHA 87ea4a2View commit details -
`T` no longer has a writer assigned to it. `test_core_cbor.odin` has global state and is run with `odin test`, so I've set it to use only one thread.
Configuration menu - View commit details
-
Copy full SHA for 852f694 - Browse repository at this point
Copy the full SHA 852f694View commit details -
Configuration menu - View commit details
-
Copy full SHA for eb3d6d7 - Browse repository at this point
Copy the full SHA eb3d6d7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 22c092f - Browse repository at this point
Copy the full SHA 22c092fView commit details -
Configuration menu - View commit details
-
Copy full SHA for e3181c1 - Browse repository at this point
Copy the full SHA e3181c1View commit details -
Configuration menu - View commit details
-
Copy full SHA for a463e28 - Browse repository at this point
Copy the full SHA a463e28View commit details -
Configuration menu - View commit details
-
Copy full SHA for a0b2ea6 - Browse repository at this point
Copy the full SHA a0b2ea6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 568b746 - Browse repository at this point
Copy the full SHA 568b746View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0f675fa - Browse repository at this point
Copy the full SHA 0f675faView commit details -
Configuration menu - View commit details
-
Copy full SHA for 09ef08f - Browse repository at this point
Copy the full SHA 09ef08fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1afc235 - Browse repository at this point
Copy the full SHA 1afc235View commit details -
Configuration menu - View commit details
-
Copy full SHA for eadfbb1 - Browse repository at this point
Copy the full SHA eadfbb1View commit details -
Remove
safe_heap_allocator
from test runnerI was under the impression that the default `context.allocator` was not thread-safe, but I've been told that this is not the case.
Configuration menu - View commit details
-
Copy full SHA for dffc3af - Browse repository at this point
Copy the full SHA dffc3afView commit details -
Configuration menu - View commit details
-
Copy full SHA for 89d8df2 - Browse repository at this point
Copy the full SHA 89d8df2View commit details -
Configuration menu - View commit details
-
Copy full SHA for a1c5beb - Browse repository at this point
Copy the full SHA a1c5bebView commit details -
Send terminal control code to
STDOUT
instead`STDERR` might be redirected, and this code signals to the terminal to show the cursor again. Otherwise, the cursor will be invisible.
Configuration menu - View commit details
-
Copy full SHA for dcfda19 - Browse repository at this point
Copy the full SHA dcfda19View commit details -
Configuration menu - View commit details
-
Copy full SHA for e11f3d2 - Browse repository at this point
Copy the full SHA e11f3d2View commit details -
Change test runner options to
SCREAMING_SNAKE_CASE
This commit also changes the name of `test_select` to `ODIN_TEST_NAMES`, to better conform with the already-existing `-test-name:<name>` option.
Configuration menu - View commit details
-
Copy full SHA for b7e1ae7 - Browse repository at this point
Copy the full SHA b7e1ae7View commit details -
Configuration menu - View commit details
-
Copy full SHA for c531427 - Browse repository at this point
Copy the full SHA c531427View commit details -
Be specific about
int
size forRollback_Stack
assertsThis should fix tests failing on 32-bit platforms.
Configuration menu - View commit details
-
Copy full SHA for bf42e39 - Browse repository at this point
Copy the full SHA bf42e39View commit details -
Configuration menu - View commit details
-
Copy full SHA for e1a3c0e - Browse repository at this point
Copy the full SHA e1a3c0eView commit details -
Report test memory usage only if there's an issue
Adds new option `ODIN_TEST_ALWAYS_REPORT_MEMORY`, for when you always want to see the memory usage report.
Configuration menu - View commit details
-
Copy full SHA for 49fa663 - Browse repository at this point
Copy the full SHA 49fa663View commit details -
Support
ODIN_TEST_PROGRESS_WIDTH=0
This will automatically calculate how wide the progress bars should be based on the package with the greatest number of tests. The progress width is now capped to 100.
Configuration menu - View commit details
-
Copy full SHA for 84ad71f - Browse repository at this point
Copy the full SHA 84ad71fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6a1649d - Browse repository at this point
Copy the full SHA 6a1649dView commit details -
Update
tests\core\encoding\cbor
to use new test runner.It was leaky and required a substantial number of `loc := #caller_location` additions to parts of the core library to make it easier to track down how and where it leaked. The tests now run fine multi-threaded.
Configuration menu - View commit details
-
Copy full SHA for a27b167 - Browse repository at this point
Copy the full SHA a27b167View commit details -
Support deterministic random seeding of tests
Add a new option `ODIN_TEST_RANDOM_SEED` which is picked from the cycle counter at startup, if it's not specified by the user. This number is sent to every test in the `T` struct and reset every test (just in case).
Configuration menu - View commit details
-
Copy full SHA for 1f6a6f2 - Browse repository at this point
Copy the full SHA 1f6a6f2View commit details -
Configuration menu - View commit details
-
Copy full SHA for b74b956 - Browse repository at this point
Copy the full SHA b74b956View commit details -
Configuration menu - View commit details
-
Copy full SHA for 40b20fb - Browse repository at this point
Copy the full SHA 40b20fbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3404dea - Browse repository at this point
Copy the full SHA 3404deaView commit details -
And fix a few leaks in `core:encoding/hxa` while at it.
Configuration menu - View commit details
-
Copy full SHA for 1b32e27 - Browse repository at this point
Copy the full SHA 1b32e27View commit details -
Configuration menu - View commit details
-
Copy full SHA for 601df0e - Browse repository at this point
Copy the full SHA 601df0eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6641a6f - Browse repository at this point
Copy the full SHA 6641a6fView commit details -
Made them run in parallel as well.
Configuration menu - View commit details
-
Copy full SHA for 9d0f483 - Browse repository at this point
Copy the full SHA 9d0f483View commit details -
Configuration menu - View commit details
-
Copy full SHA for d334b8c - Browse repository at this point
Copy the full SHA d334b8cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6f7c5a7 - Browse repository at this point
Copy the full SHA 6f7c5a7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 39fd73f - Browse repository at this point
Copy the full SHA 39fd73fView commit details -
Configuration menu - View commit details
-
Copy full SHA for d7bfbe0 - Browse repository at this point
Copy the full SHA d7bfbe0View commit details -
Configuration menu - View commit details
-
Copy full SHA for b0faab2 - Browse repository at this point
Copy the full SHA b0faab2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8383a45 - Browse repository at this point
Copy the full SHA 8383a45View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9829a02 - Browse repository at this point
Copy the full SHA 9829a02View commit details -
Configuration menu - View commit details
-
Copy full SHA for 80b1157 - Browse repository at this point
Copy the full SHA 80b1157View commit details -
Configuration menu - View commit details
-
Copy full SHA for ed0384c - Browse repository at this point
Copy the full SHA ed0384cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9ba02e8 - Browse repository at this point
Copy the full SHA 9ba02e8View commit details -
Configuration menu - View commit details
-
Copy full SHA for a406ff7 - Browse repository at this point
Copy the full SHA a406ff7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5b1ffba - Browse repository at this point
Copy the full SHA 5b1ffbaView commit details -
Configuration menu - View commit details
-
Copy full SHA for dacb0f7 - Browse repository at this point
Copy the full SHA dacb0f7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 62b7d8d - Browse repository at this point
Copy the full SHA 62b7d8dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8d93379 - Browse repository at this point
Copy the full SHA 8d93379View commit details -
Configuration menu - View commit details
-
Copy full SHA for 54dae06 - Browse repository at this point
Copy the full SHA 54dae06View commit details -
Disable benchmark on Windows for a moment.
Configuration menu - View commit details
-
Copy full SHA for 3354212 - Browse repository at this point
Copy the full SHA 3354212View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3061696 - Browse repository at this point
Copy the full SHA 3061696View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6050bc3 - Browse repository at this point
Copy the full SHA 6050bc3View commit details -
Configuration menu - View commit details
-
Copy full SHA for fb37572 - Browse repository at this point
Copy the full SHA fb37572View commit details -
Configuration menu - View commit details
-
Copy full SHA for d172366 - Browse repository at this point
Copy the full SHA d172366View commit details -
Configuration menu - View commit details
-
Copy full SHA for 433ca53 - Browse repository at this point
Copy the full SHA 433ca53View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6a5d51f - Browse repository at this point
Copy the full SHA 6a5d51fView commit details -
Remove
-test-name
in favor of test runner option`-define:ODIN_TEST_NAMES=...` is capable of selecting test by package and name or name only, with the ability to access packages included by `-all-packages`.
Configuration menu - View commit details
-
Copy full SHA for cb8faf5 - Browse repository at this point
Copy the full SHA cb8faf5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 45fa9d8 - Browse repository at this point
Copy the full SHA 45fa9d8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 21064fb - Browse repository at this point
Copy the full SHA 21064fbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9dcf345 - Browse repository at this point
Copy the full SHA 9dcf345View commit details -
Configuration menu - View commit details
-
Copy full SHA for ccdbd4b - Browse repository at this point
Copy the full SHA ccdbd4bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4875f74 - Browse repository at this point
Copy the full SHA 4875f74View commit details -
Configuration menu - View commit details
-
Copy full SHA for cb00b80 - Browse repository at this point
Copy the full SHA cb00b80View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5db65aa - Browse repository at this point
Copy the full SHA 5db65aaView commit details -
Log thread count at test run start
Provides a helpful info message about the option to change how many threads are used per run.
Configuration menu - View commit details
-
Copy full SHA for 6a92033 - Browse repository at this point
Copy the full SHA 6a92033View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5e3e958 - Browse repository at this point
Copy the full SHA 5e3e958View commit details -
Configuration menu - View commit details
-
Copy full SHA for d581dbb - Browse repository at this point
Copy the full SHA d581dbbView commit details -
Disable
FANCY_OUTPUT
in Odin test scriptsThis should tidy up the CI output logs a bit.
Configuration menu - View commit details
-
Copy full SHA for 890fe07 - Browse repository at this point
Copy the full SHA 890fe07View commit details -
Configuration menu - View commit details
-
Copy full SHA for 21a1ddf - Browse repository at this point
Copy the full SHA 21a1ddfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 9d8d864 - Browse repository at this point
Copy the full SHA 9d8d864View commit details -
Configuration menu - View commit details
-
Copy full SHA for 60d0c03 - Browse repository at this point
Copy the full SHA 60d0c03View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8d8c42e - Browse repository at this point
Copy the full SHA 8d8c42eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 3f1249c - Browse repository at this point
Copy the full SHA 3f1249cView commit details -
Be pedantic about not overwriting Odin errors
I was encountering bounds-check error messages being overwritten during a test, if the test failed for another reason and sent a log message. The original intent of having this check inside of the above `if` block was that if a test sent an error message, then it was assumed an overwrite would be safe, but it's completely possible for a test to fail for a legitimate reason, then do an unrelated bounds check somewhere else that would be buried under the animation. This change will make sure that, no matter what, the progress display will not trigger a clear if a signal was raised. There's still no guarantee that bounds-check messages will be printed properly, and it's best to redirect STDERR. The only way that can be fixed is if they get a similar hook to `context.assertion_failure_proc`.
Configuration menu - View commit details
-
Copy full SHA for f77ce35 - Browse repository at this point
Copy the full SHA f77ce35View commit details -
Fix
STDIN
,STDOUT
,STDERR
handles for BSDsTested on FreeBSD 14.0 and NetBSD 10.0 OpenBSD is untested, but link names were sourced from: https://github.com/openbsd/src/blob/master/include/stdio.h According to this, OpenBSD shares the same layout as NetBSD. FreeBSD has the same as Darwin in this regard.
Configuration menu - View commit details
-
Copy full SHA for ac94842 - Browse repository at this point
Copy the full SHA ac94842View commit details -
Revert "Disable NetBSD tests until 'undefined reference to stdout' is…
… solved." This reverts commit 21a1ddf.
Configuration menu - View commit details
-
Copy full SHA for c8539fe - Browse repository at this point
Copy the full SHA c8539feView commit details
Commits on Jun 3, 2024
-
Fix wrong
PTHREAD_CANCEL_ASYNCHRONOUS
on FreeBSD and OpenBSDThe test runner was deadlocking when a test raised a signal on FreeBSD. This is untested on OpenBSD, but I have referenced this file: https://github.com/openbsd/src/blob/master/include/pthread.h
Configuration menu - View commit details
-
Copy full SHA for 6a5633d - Browse repository at this point
Copy the full SHA 6a5633dView commit details -
Prevent test runner deadlock on NetBSD
Add `pthread_testcancel` to `core:sys/unix`
Configuration menu - View commit details
-
Copy full SHA for 7764ab2 - Browse repository at this point
Copy the full SHA 7764ab2View commit details -
Use
Warning
log level for reporting memory leaksWorks well with `-define:ODIN_TEST_LOG_LEVEL=warning`.
Configuration menu - View commit details
-
Copy full SHA for fa29974 - Browse repository at this point
Copy the full SHA fa29974View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0ff130d - Browse repository at this point
Copy the full SHA 0ff130dView commit details