zsh-test-runner (ztr)

Straight-forward tests and coverage reports for zsh

Features:

• short and gentle learning curve
• run one or more tests on the command line
• run one or more tests saved in a test suite file
• skip tests
• optionally give tests descriptive names
• optionally provide notes to tests, for example to list dependencies in the logged output
• access cumulative failure, pass, and skip counts as shell variables
• print coverage summaries with test count, failure count and rate, pass count and rate, and skip count

What it does not feature: its own human language-like declarative test syntax. There's no "describe", "expect", etc. Downside is the tests don't read like a story. Upside is —because the shell already has rich support for tests— there is nothing to learn, there are no artificial limits on what can be tested, the cost to migrating to zsh-test-runner (or from, if you must) is very low, and there is no risk that assertions were incorrectly implemented. Just write your [[ ]]s, your (( ))s, even your tests or [ ]s, and never again punt on testing.

Installation

Plugin (recommended)

You can install zsh-test-runner with a zsh plugin manager. This is the recommended installation because most modern plugin managers are optimized for shell load time performance.

Each has its own way of doing things. See your package manager's documentation or the zsh plugin manager plugin installation procedures gist. If you're new to zsh plugin management, at this writing zinit is a good choice for its popularity, frequent updates, and great performance.

After adding the plugin to the manager, restart zsh:

exec zsh

Package

zsh-test-runner is available on Homebrew. Run

brew install olets/tap/zsh-test-runner

and follow the post-install instructions logged to the terminal.

Manual

Clone this repo and add source path/to/ztr.zsh to your .zshrc (replace path/to/ with the correct path). Then restart zsh:

exec zsh

Commands

clear

Clear counts.

% ztr test true
PASS true
% ztr clear
% ztr test false
FAIL false
% ztr summary
1 test total
1 (100%) failed
0 were skipped
0 passed

skip [--quiet | -q] <arg> [<name> [<notes>]]

Skip <arg>.

% ztr skip my_test
SKIP my_test

See test for details about --quiet, <name>, and <notes>.

summary

Pretty-print summary of counts.

% ztr test true
PASS true
% ztr test false
FAIL false
% ztr summary
2 tests total
1 (50%) failed
0 were skipped
1 (50%) passed

test [--quiet | -q] <arg> [<name> [<notes>]]

Test <arg>. Pretty-print result and notes unless "quiet".

% ztr test true
PASS true
% ztr test false
FAIL false

In practice <arg> will most likely be a shell test expression.

% ztr test '[[ 1 == 1 ]]'
PASS [[ 1 == 1 ]]

Note that <arg> is passed to eval, so 1) don't pass something you don't want to eval and 2) watch out for quotation errors.

% ztr test [[ 1 == 1 ]]
zsh: = not found # same error you get if you run `eval [[ 1 == 1 ]]`
% ztr test '[[ 1 == 1 ]]'
PASS [[ 1 == 1 ]]

<arg> can be a value, a function, a [ ] or [[ ]] test, anything that you can pass to eval.

% ztr test 'test -f myfile.txt'
% ztr test '[ -f myfile.txt ]'
% ztr test '[[ -f myfile.txt ]]'
% ztr test my_function
# etc

Choose your quote level to control what is logged.

% my_var=1
% ztr test "[[ $my_var == 1 ]]"
PASS [[ 1 == 1 ]]
% ztr test '[[ $my_var == 1 ]]'
PASS [[ $my_var == 1 ]]

A passing test has a passing exit code; a failing test has a failing exit code:

% ztr test true 'passing exit code'
PASS passing exit code
% echo $?
0
% ztr test false 'failing exit code'
FAIL failing exit code
% echo $?
1

(--quiet | -q)

Optionally silence output.

% ztr test true
PASS true
% ztr test --quiet true
% ztr test true
PASS true

<name>

Optionally pass a name as a second parameter.

% ztr test '[[ 1 == 1 ]]' '<name> appears instead of <arg>'
PASS <name> appears instead of <arg>

<notes>

Optionally pass notes as a third parameter. For example, noting dependencies can help with troubleshooting. In the output notes are indented.

% cat my_tests.ztr
# --- snip ---
ztr test my_test_10
# --- snip ---
ztr test my_test_20 my_test_20 'Dependencies: my_test_10'
# --- snip ---
ztr test my_test_30 my_test_30 'Dependencies: my_test_10'
# --- snip ---

% ./my_tests.ztr
# --- snip ---
FAIL my_test_10
# --- snip ---
FAIL my_test_20
    'Dependencies: my_test_10'
# --- snip ---
FAIL my_test_30
    'Dependencies: my_test_10'
# Ok let's see if fixing my_test_10 fixes my_test_20 and my_test_30

Running test suites

You can run a test suite from a file. The following examples suppose the file is in the current working directory; adjust the path to meet your situation.

1. Prepare your test suite.

   % cat suite.ztr
   my_test=false
   
   ztr test true 'my first test'
   ztr test my_test 'my second test'
   ztr test 'my_test && true' 'my third test' 'depends on my second test'
   ztr skip my_other_test 'my other test' '@TODO build the api for this!'
   
   echo
   ztr summary

2. Run your test suite either by

   • sourcing it:

     % . ./suite.ztr # or the longhand `source ./suite.ztr`
     PASS my first test
     FAIL my second test
     FAIL my third test
     	depends on my second test
     SKIP my other test
     	@TODO build the api for this!
     
     4 tests total
     2 (40%) failed
     1 was skipped
     1 (20%) passed

     This method has advantage that the results are available to the parent shell. It has the potential disadvantage that any other side effects of your tests are not sandboxed.

     % ztr clear
     
     % zsh suite.ztr
     # --- snip ---
     4 tests total
     2 (40%) failed
     1 was skipped
     1 (20%) passed
     
     % ztr summary # suite's summary not available
     4 tests total
     2 (50%) failed
     1 was skipped
     1 (25%) passed
     
     % echo $my_test # suite's context available
     false

   • running it in a subshell

     In this case you must explicitly source ztr.zsh in your test suite. zsh-test-runner provides the variable ZTR_PATH to make this easy

     % cat suite.ztr
     source $ZTR_PATH
     # --- snip ---

     To run the suite in a subshell pass the file to zsh:

     % zsh suite.ztr
     PASS my first test
     FAIL my second test
     FAIL my third test
     	depends on my second test
     SKIP my other test
     	@TODO build the api for this!
     
     4 tests total
     2 (40%) failed
     1 was skipped
     1 (20%) passed

     This method has the potential advantage of sandboxing your tests. It has the potential disadvantage that the results are not available to the parent shell.

     % ztr clear
     
     % zsh suite.ztr
     # --- snip ---
     4 tests total
     2 (40%) failed
     1 was skipped
     1 (20%) passed
     
     % ztr summary # suite's summary not available
     0 tests total
     0 failed
     0 were skipped
     0 passed
     
     % echo $my_test # suite's context not available
     

Examples

zsh-abbr uses zsh-test-runner for its test suite. For real world example of ztr use, check out zsh-abbr/tests/abbr.ztr.

( --help | -h | help)

Show the manpage.

( --version | -v | version )

Print the command name and current version.

Variables

Counts

Variable	Type	Default	Use
ZTR_COUNT_FAIL	integer	0	The number of tests which have failed
ZTR_COUNT_PASS	integer	0	The number of tests which have passed
ZTR_COUNT_SKIP	integer	0	The number of tests which have been skipped

Note that "tests" in the above are not necessarily unique:

% ztr test true --quiet
% echo $ZTR_COUNT_PASS
1
% ztr test true --quiet
% echo $ZTR_COUNT_PASS
2

Use ztr clear to zero out count variables:

% ztr test true --quiet
% ztr clear
% echo $ZTR_COUNT_PASS
0

ZTR_COUNT_FAIL is a convenient way to check for 100% pass rate:

% (( ZTR_COUNT_FAIL )) || echo all tests pass

Configuration

Variable	Type	Default	Use
NO_COLOR	any		To suppress color output, set to any value or simply declare (NO_COLOR=) in .zshrc before loading zsh-test-runner. See https://no-color.org/
ZTR_DEBUG	integer	0	If non-zero, print debugging messages
ZTR_QUIET	integer	0	If non-zero, use quiet mode without passing --quiet

Other

Variable	Type	Use
ZTR_PATH	string	source $ZTR_PATH in scripts that include ztr commands

Changelog

See the CHANGELOG file.

Roadmap

See the ROADMAP file.

Contributing

Thanks for your interest. Contributions are welcome!

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

Check the Issues to see if your topic has been discussed before or if it is being worked on. You may also want to check the roadmap (see above).

Please read CONTRIBUTING.md before opening a pull request.

To test ztr run ./tests/ztr.ztr

License

zsh-test-runner by Henry Bley-Vroman is licensed under CC BY-NC-SA 4.0 with a human rights condition from Hippocratic License 2.1. Persons interested in using or adapting this work for commercial purposes should contact the author.

For the full text of the license, see the LICENSE file.