Skip to content

Updates to test execution documentation #940

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Oct 2, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
105 changes: 103 additions & 2 deletions docs/TESTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Running tests

From the terminal, change to the base directory of the problem then execute the tests with:
From the terminal, change to the base directory of the exercise then execute the tests with:

```bash
$ mix test
Expand All @@ -10,7 +10,7 @@ This will execute the test file found in the `test` subfolder -- a file ending i

## Pending tests

In the test suites, all but the first test have been skipped.
In test suites of practice exercises, all but the first test have been tagged to be skipped.

Once you get a test passing, you can unskip the next one by
commenting out the relevant `@tag :pending` with a `#` symbol.
Expand All @@ -24,13 +24,114 @@ test "shouting" do
end
```

If you wish to run all tests at once, you can include all skipped test by using the `--include` flag on the `mix test` command:

```bash
$ mix test --include pending
```

Or, you can enable all the tests by commenting out the
`ExUnit.configure` line in the test suite.

```elixir
# ExUnit.configure exclude: :pending, trace: true
```

## Additional Elixir testing features

`ExUnit` and `mix test` provide quite a few methods of grouping, tagging & executing
tests and various methods of controlling test execution, much of which is summarized
below.

### Methods for running specific tests

Documentation:

* [`mix test` documentation](https://hexdocs.pm/mix/Mix.Tasks.Test.html)

#### Running tests in a specific file

All tests in a single file may be executed via `mix test` by specifying the file:

```bash
$ mix test test/<FILE>.exs
```

> NOTE: `tagging` may impact which tests are actually executed using this method.

#### Running individual tests

Individual tests may be executed by referencing a test's line number in the file:

```bash
$ mix test test/<FILE>.exs:LINENUM
```

Multiple tests may be executed by giving multiple line numbers separated by `:`.

For example, given a file with the following content with line numbers:

```elixir
test "Test 1" do # 1
# test implementation # 2-6
end # 7
# 8
test "Test 2" do # 9
# test implementation # 10-21
end # 22
# 23
test "Test 3" do # 24
# test implementation # 25-35
end # 36
```

The 1st and 3rd tests can be executed by:

```bash
$ mix test test/FILE.exs:1:24
```

> NOTE: When specifying tests via line numbers, `tagging` is ignored.

#### Running groups of tests

Tests may be grouped using `describe`:

```elixir
describe "short test group description" do
test "test description" do
# test implementation
end

test "another test description" do
# test implementation
end
end
```

All tests in a group may be executed by referencing its line number in the file,
just like referencing and executing individual tests.

Documentation:

* [`describe`](https://hexdocs.pm/ex_unit/ExUnit.Case.html#describe/2)

#### Other useful `mix test` options

* `--include` and `--exclude` - runs or doesn't run specific tests based on their `@tag`s
* `--failed` - runs only tests that failed the last time they ran
* `--max-failures` - the suite stops evaluating tests when this number of test failures
is reached
* `--seed` - seeds the random number generator used to randomize the order of tests
`--seed 0` disables randomization so the tests in a single file will always be ran
in the same order they were defined in
* `--stale` - runs only tests which reference modules that changed since the last
time tests were ran with `--stale`

Documentation:

* [`mix test` command-line options](https://hexdocs.pm/mix/Mix.Tasks.Test.html#module-command-line-options)

## Typespecs and Dialyzer (DIscrepancy AnalYZer for ERlang programs)

Elixir exercises include a skeleton implementation file in the `lib`
Expand Down
20 changes: 18 additions & 2 deletions exercises/shared/.docs/tests.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,21 @@
# Running tests

Execute the tests with:
From the terminal, change to the base directory of the exercise then execute the tests with:

```bash
$ mix test
```

This will execute the test file found in the `test` subfolder -- a file ending in `_test.exs`

Documentation:

* [`mix test` - Elixir's test execution tool](https://hexdocs.pm/mix/Mix.Tasks.Test.html)
* [`ExUnit` - Elixir's unit test library](https://hexdocs.pm/ex_unit/ExUnit.html)

## Pending tests

In the test suites, all but the first test have been skipped.
In test suites of practice exercises, all but the first test have been tagged to be skipped.

Once you get a test passing, you can unskip the next one by commenting out the relevant `@tag :pending` with a `#` symbol.

Expand All @@ -32,3 +39,12 @@ Or, you can enable all the tests by commenting out the `ExUnit.configure` line i
```elixir
# ExUnit.configure(exclude: :pending, trace: true)
```

## Useful `mix test` options

* `test/<FILE>.exs:LINENUM` - runs only a single test, the test from `<FILE>.exs` whose definition is on line `LINENUM`
* `--failed` - runs only tests that failed the last time they ran
* `--max-failures` - the suite stops evaluating tests when this number of test failures
is reached
`--seed 0` - disables randomization so the tests in a single file will always be ran
in the same order they were defined in