Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor the test runner #3646

Merged
merged 90 commits into from
Jun 4, 2024
Merged

Refactor the test runner #3646

merged 90 commits into from
Jun 4, 2024

Commits on Jun 2, 2024

  1. Configuration menu
    Copy the full SHA
    8ae375d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7d4da6e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1875e7c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    558c330 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8137b9d View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    fc4f6b8 View commit details
    Browse the repository at this point in the history
  7. Add mem.Rollback_Stack

    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    50dffaf View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    95c2e02 View commit details
    Browse the repository at this point in the history
  9. Refactor the test runner

    Changes
    
    - Support multi-threaded testing.
    - Support `set_fail_timeout` on all platforms.
    - Display an animated progress report.
    - Setup all tests with a context logger.
    - Give all tests their own separate custom allocators.
    - Support tracking test memory usage.
    - Display a summary of the failed tests at the end.
    - Let users select only specific tests to run.
    - Support copying failed tests to the clipboard to run again.
    - Support catching SIGINT (CTRL-C) to cancel early.
    - Record context in cleanup procs.
    - Write all log messages to STDERR for easy redirection.
    - Possibly more I've forgotten.
    
    New Options
    
    - `-define:test_threads=N`: Specify thread count.
    - `-define:test_thread_memory=B`: Specify initial memory block size in bytes to each thread.
    - `-define:test_track_memory=true`: Track the memory usage of individual tests.
    - `-define:test_fancy=false`: Disable animated progress report.
    - `-define:test_select=package.test_name,...`: Run only select tests.
    - `-define:test_clipboard=true`: Copy names of failed tests to the clipboard.
    - `-define:test_progress_width=24`: Change the width of the animated progress bars.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    b6c4dfb View commit details
    Browse the repository at this point in the history
  10. Remove unneeded code

    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    d030240 View commit details
    Browse the repository at this point in the history
  11. Temporarily disable Windows-specific test runner

    I do not have a Windows machine to test the refactored test runner, and
    I am unsure if it would even run correctly on Windows without this
    disabled.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    87ea4a2 View commit details
    Browse the repository at this point in the history
  12. Get tests passing again

    `T` no longer has a writer assigned to it.
    
    `test_core_cbor.odin` has global state and is run with `odin test`,
    so I've set it to use only one thread.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    852f694 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    eb3d6d7 View commit details
    Browse the repository at this point in the history
  14. Delete duplicated flag.

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    22c092f View commit details
    Browse the repository at this point in the history
  15. Update core:compress tests

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    e3181c1 View commit details
    Browse the repository at this point in the history
  16. Update core:container tests

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    a463e28 View commit details
    Browse the repository at this point in the history
  17. Update tests\core\crypto

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    a0b2ea6 View commit details
    Browse the repository at this point in the history
  18. Fix indentation

    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    568b746 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    0f675fa View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    09ef08f View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    1afc235 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    eadfbb1 View commit details
    Browse the repository at this point in the history
  23. Remove safe_heap_allocator from test runner

    I was under the impression that the default `context.allocator` was not
    thread-safe, but I've been told that this is not the case.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    dffc3af View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    89d8df2 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    a1c5beb View commit details
    Browse the repository at this point in the history
  26. Send terminal control code to STDOUT instead

    `STDERR` might be redirected, and this code signals to the terminal to
    show the cursor again. Otherwise, the cursor will be invisible.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    dcfda19 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    e11f3d2 View commit details
    Browse the repository at this point in the history
  28. Change test runner options to SCREAMING_SNAKE_CASE

    This commit also changes the name of `test_select` to `ODIN_TEST_NAMES`,
    to better conform with the already-existing `-test-name:<name>` option.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    b7e1ae7 View commit details
    Browse the repository at this point in the history
  29. Update -define for crypto

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    c531427 View commit details
    Browse the repository at this point in the history
  30. Be specific about int size for Rollback_Stack asserts

    This should fix tests failing on 32-bit platforms.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    bf42e39 View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    e1a3c0e View commit details
    Browse the repository at this point in the history
  32. Report test memory usage only if there's an issue

    Adds new option `ODIN_TEST_ALWAYS_REPORT_MEMORY`, for when you always
    want to see the memory usage report.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    49fa663 View commit details
    Browse the repository at this point in the history
  33. Support ODIN_TEST_PROGRESS_WIDTH=0

    This will automatically calculate how wide the progress bars should be
    based on the package with the greatest number of tests.
    
    The progress width is now capped to 100.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    84ad71f View commit details
    Browse the repository at this point in the history
  34. Configuration menu
    Copy the full SHA
    6a1649d View commit details
    Browse the repository at this point in the history
  35. Update tests\core\encoding\cbor to use new test runner.

    It was leaky and required a substantial number of `loc := #caller_location` additions to parts of the core library to make it easier to track down how and where it leaked.
    
    The tests now run fine multi-threaded.
    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    a27b167 View commit details
    Browse the repository at this point in the history
  36. Support deterministic random seeding of tests

    Add a new option `ODIN_TEST_RANDOM_SEED` which is picked from the cycle
    counter at startup, if it's not specified by the user.
    
    This number is sent to every test in the `T` struct and reset every test
    (just in case).
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    1f6a6f2 View commit details
    Browse the repository at this point in the history
  37. Remove unneeded import

    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    b74b956 View commit details
    Browse the repository at this point in the history
  38. Port tests\core\c\libc

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    40b20fb View commit details
    Browse the repository at this point in the history
  39. Port tests\encoding\hex

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    3404dea View commit details
    Browse the repository at this point in the history
  40. Port tests\core\encoding\hxa

    And fix a few leaks in `core:encoding/hxa` while at it.
    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    1b32e27 View commit details
    Browse the repository at this point in the history
  41. Port tests\core\encoding\json

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    601df0e View commit details
    Browse the repository at this point in the history
  42. Port tests\core\encoding\varint

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    6641a6f View commit details
    Browse the repository at this point in the history
  43. Port tests\core\encoding\xml

    Made them run in parallel as well.
    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    9d0f483 View commit details
    Browse the repository at this point in the history
  44. Port tests\core\path\filepath

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    d334b8c View commit details
    Browse the repository at this point in the history
  45. Port tests\core\fmt

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    6f7c5a7 View commit details
    Browse the repository at this point in the history
  46. Port testing\core\hash

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    39fd73f View commit details
    Browse the repository at this point in the history
  47. Port testing\core\text\i18n

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    d7bfbe0 View commit details
    Browse the repository at this point in the history
  48. Configuration menu
    Copy the full SHA
    b0faab2 View commit details
    Browse the repository at this point in the history
  49. Port tests\core\text\match

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    8383a45 View commit details
    Browse the repository at this point in the history
  50. Port tests\core\odin

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    9829a02 View commit details
    Browse the repository at this point in the history
  51. Port tests\core\reflect

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    80b1157 View commit details
    Browse the repository at this point in the history
  52. Port tests\core\runtime

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    ed0384c View commit details
    Browse the repository at this point in the history
  53. Port tests\core\slice

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    9ba02e8 View commit details
    Browse the repository at this point in the history
  54. Port tests\core\strings

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    a406ff7 View commit details
    Browse the repository at this point in the history
  55. Port testing\core\time

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    5b1ffba View commit details
    Browse the repository at this point in the history
  56. Port tests\core\thread

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    dacb0f7 View commit details
    Browse the repository at this point in the history
  57. Port tests\core\net

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    62b7d8d View commit details
    Browse the repository at this point in the history
  58. Configuration menu
    Copy the full SHA
    8d93379 View commit details
    Browse the repository at this point in the history
  59. Update CI

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    54dae06 View commit details
    Browse the repository at this point in the history
  60. Update ci.yml

    Disable benchmark on Windows for a moment.
    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    3354212 View commit details
    Browse the repository at this point in the history
  61. Update ci.yml

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    3061696 View commit details
    Browse the repository at this point in the history
  62. Add missing benchmarks build.bat.

    Kelimion authored and Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    6050bc3 View commit details
    Browse the repository at this point in the history
  63. Configuration menu
    Copy the full SHA
    fb37572 View commit details
    Browse the repository at this point in the history
  64. Configuration menu
    Copy the full SHA
    d172366 View commit details
    Browse the repository at this point in the history
  65. Configuration menu
    Copy the full SHA
    433ca53 View commit details
    Browse the repository at this point in the history
  66. Configuration menu
    Copy the full SHA
    6a5d51f View commit details
    Browse the repository at this point in the history
  67. Remove -test-name in favor of test runner option

    `-define:ODIN_TEST_NAMES=...` is capable of selecting test by package
    and name or name only, with the ability to access packages included by
    `-all-packages`.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    cb8faf5 View commit details
    Browse the repository at this point in the history
  68. Configuration menu
    Copy the full SHA
    45fa9d8 View commit details
    Browse the repository at this point in the history
  69. Configuration menu
    Copy the full SHA
    21064fb View commit details
    Browse the repository at this point in the history
  70. Configuration menu
    Copy the full SHA
    9dcf345 View commit details
    Browse the repository at this point in the history
  71. Configuration menu
    Copy the full SHA
    ccdbd4b View commit details
    Browse the repository at this point in the history
  72. Configuration menu
    Copy the full SHA
    4875f74 View commit details
    Browse the repository at this point in the history
  73. Configuration menu
    Copy the full SHA
    cb00b80 View commit details
    Browse the repository at this point in the history
  74. Configuration menu
    Copy the full SHA
    5db65aa View commit details
    Browse the repository at this point in the history
  75. Log thread count at test run start

    Provides a helpful info message about the option to change how many
    threads are used per run.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    6a92033 View commit details
    Browse the repository at this point in the history
  76. Configuration menu
    Copy the full SHA
    5e3e958 View commit details
    Browse the repository at this point in the history
  77. Configuration menu
    Copy the full SHA
    d581dbb View commit details
    Browse the repository at this point in the history
  78. Disable FANCY_OUTPUT in Odin test scripts

    This should tidy up the CI output logs a bit.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    890fe07 View commit details
    Browse the repository at this point in the history
  79. Configuration menu
    Copy the full SHA
    21a1ddf View commit details
    Browse the repository at this point in the history
  80. Plug leak in AES tests.

    Kelimion committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    9d8d864 View commit details
    Browse the repository at this point in the history
  81. Configuration menu
    Copy the full SHA
    60d0c03 View commit details
    Browse the repository at this point in the history
  82. Configuration menu
    Copy the full SHA
    8d8c42e View commit details
    Browse the repository at this point in the history
  83. Configuration menu
    Copy the full SHA
    3f1249c View commit details
    Browse the repository at this point in the history
  84. Be pedantic about not overwriting Odin errors

    I was encountering bounds-check error messages being overwritten during
    a test, if the test failed for another reason and sent a log message.
    
    The original intent of having this check inside of the above `if` block
    was that if a test sent an error message, then it was assumed an
    overwrite would be safe, but it's completely possible for a test to fail
    for a legitimate reason, then do an unrelated bounds check somewhere
    else that would be buried under the animation.
    
    This change will make sure that, no matter what, the progress display
    will not trigger a clear if a signal was raised. There's still no
    guarantee that bounds-check messages will be printed properly, and it's
    best to redirect STDERR.
    
    The only way that can be fixed is if they get a similar hook to
    `context.assertion_failure_proc`.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    f77ce35 View commit details
    Browse the repository at this point in the history
  85. Fix STDIN, STDOUT, STDERR handles for BSDs

    Tested on FreeBSD 14.0 and NetBSD 10.0
    
    OpenBSD is untested, but link names were sourced from:
    https://github.com/openbsd/src/blob/master/include/stdio.h
    
    According to this, OpenBSD shares the same layout as NetBSD.
    
    FreeBSD has the same as Darwin in this regard.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    ac94842 View commit details
    Browse the repository at this point in the history
  86. Revert "Disable NetBSD tests until 'undefined reference to stdout' is…

    … solved."
    
    This reverts commit 21a1ddf.
    Feoramund committed Jun 2, 2024
    Configuration menu
    Copy the full SHA
    c8539fe View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. Fix wrong PTHREAD_CANCEL_ASYNCHRONOUS on FreeBSD and OpenBSD

    The test runner was deadlocking when a test raised a signal on FreeBSD.
    
    This is untested on OpenBSD, but I have referenced this file:
    https://github.com/openbsd/src/blob/master/include/pthread.h
    Feoramund committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    6a5633d View commit details
    Browse the repository at this point in the history
  2. Prevent test runner deadlock on NetBSD

    Add `pthread_testcancel` to `core:sys/unix`
    Feoramund committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    7764ab2 View commit details
    Browse the repository at this point in the history
  3. Use Warning log level for reporting memory leaks

    Works well with `-define:ODIN_TEST_LOG_LEVEL=warning`.
    Feoramund committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    fa29974 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0ff130d View commit details
    Browse the repository at this point in the history