Road to v1.0

[ztombol]

What is this?

This is a list containing popular feature requests, common misconceptions and long standing bugs. All of which should be addressed on the way to v1.0.

Not all of these will become part of Bats. Some may be rejected, some may become a footnote in the documentation, while some may be compiled into a best practice guide. What they have in common is that they are all popular and recurring topics, and so they must be addressed in some form.

You can contribute by:

• Owning an issue - If you want to work through one or more of these issues, just say so and you'll be assigned to the issue to let others know that work is underway.
• Scavenging for new tasks - This list is likely to be incomplete and the issue tracker may still hide gems to be found. If you find one, let us know in a comment, using the format below, and it will be added to the list.
• Being a co-pilot - Help others testing new features and fixing bugs. Another pair of eyes can be very helpful.

All in all, this is an effort to clean-up the issue tracker as well. Also see #150.

The list

Documentation

Topics in which the documentation should be expanded.

• Add an example on the proper use of run with piped commands.

  status
  assigned to
  issue	Pipe does not work in the test #10 bats script not working #69
  pull request	added example of piping commands together with run #47

• Add warning and workaround for [[ not triggering test failure on some Bash versions.

  status
  assigned to
  issue	Write a warning in big fat letters about [[ somewhere in the readme #49
  pull request

• Add warning about test descriptions (i.e. function names) having to be unique.

  status
  assigned to
  issue
  pull request

• Add link to bats-* helper libraries.

  status
  assigned to
  issue	Ability to log #67 Dump ${output} from tests by default? #105 Failing tests don't output the actual value #190 Need more information in pretty output #192
  pull request	Assertions #5 report $output content when test fails #8 bats_print_failed_command: Include $output if available. #101

• Add detailed description of how Bats processes and executes tests.

  The wiki entry is a good start, but without diving into the source, users don't understand why a for around a test case can't be used to programmatically generate tests, or what's the difference between doing something in global scope or setup. This should be a part of the official documentation, not just the wiki. ("I addressed this problem somewhat in Loop around tests? #136." -- @ztombol)

  status
  assigned to
  issue	Loop around tests? #136 Dynamic test creation #157 bats loops - only last iteration is displayed in console output #179 Nested test #185
  pull request

Fixes

Bugs that should be fixed.

• Empty output lines are missing from ${lines[@]}.

  status
  assigned to
  issue	IFS broken after first run #89
  pull request	saving $IFS in run() not altered for code using it - fix #89 #90 Revert "saving $IFS in run() not altered for code using it" #92

• Use POSIX compliant options with head for better compatibility (e.g. Busybox support).

  status
  assigned to
  issue
  pull request	Avoid undocumented usage of the head utility. #163

• Incorrect TAP format for skipped tests.

  status
  assigned to
  issue	format of TAP output for skipped tests is incorrect #141
  pull request	This commit corrects the problem of incorrect TAP format for skipped tests. #142

• set -o nounset breaks bats-exec-test.

  status
  assigned to
  issue	Support for "unofficial strict mode"? #171
  pull request

• bats-preprocess breaks when translating a @test line that uses tabs for white space, instead of plain spaces.

  status
  assigned to
  issue	@test: command not found if tab is used after test name #98
  pull request	Fixed issue #98 #122

• Bats failing on OS X 10.11.2.

  status
  assigned to
  issue	let: count+=: Syntax Error on OS X 10.11.2 #140
  pull request

Features

Popular feature requests.

• Capture standard output and standard error separately as well, in addition to the combined output.

  status
  assigned to
  issue
  pull request	Add run variant with stdout and stderr separated #55 Separate stdout and stderr in run #180

• Running only a specific set of tests.

  Tests may be selected by a regular expression matching their description.

  Using numbers may be more convenient in some cases. For example, when running tests on Travis CI, Bats automatically uses TAP format output, which includes the test numbers. Using only the number of failing tests speeds up debugging and testing fixes. Tests may be specified as a list of numbers, e.g. 6,8, number ranges 1-4,10-13, or the mixture of both, e.g. 1-4,6,8,10-13.

  status
  assigned to
  issue	Run a single test by name #36 Easy way to run just one test #164
  pull request	Option to run tests names matching a pattern. #117 Support log file and filtering tests #173

• Create and document new public variables that can be used in teardown to determine whether the test has succeeded, failed or has been skipped.

  This can be used to preserve test artefacts on failure to help debugging, e.g. what bats-file's temp_del does.

  This most likely involves making BATS_TEST_COMPLETED and BATS_TEST_SKIPPED public.

  status
  assigned to
  issue
  pull request

• Log output or TAP output to a file.

  status
  assigned to
  issue	Ability to log #67 option to let stderr show through even when test is successful? #96 Provide --log=<filename> #182
  pull request	adds ability to output TAP to file and release STDOUT from tests #149 Support log file and filtering tests #173

• Test files including other test files.

  status
  assigned to
  issue	Ability to include another test suite #99 Show real test cases' description when they are called from a test suite #169
  pull request	Add the ability to include other test files #116

• Per file setup and teardown functions that are run only once per test file.

  May look something like Perl's interface.

  • startup - before the first test
  • setup - before every test
  • teardown - after every test
  • shutdown - after the last test

  status
  assigned to
  issue
  pull request

• Stop load <path> from automatically appending .bash when <path> is relative.

  While files with arbitrary extension can be loaded with using the absolute path BATS_TEST_DIRNAME. Consistent behaviour in case of absolute and relative paths would be easier to use.

  status
  assigned to
  issue
  pull request	Make load handle free filename too #147 (similar)

[Zearin]

FANTASTIC post!

@ztombol: Keep up the excellent work—not just in the code, but in top-notch posts like this to rally the troops. Excellent technical writing like this is rare (probably more than it should be!), but powerful.

I currently don’t have time to contribute—but if and when that changes, I will certainly sign up for this.

Until then, if you’re reading this and have the time, please contribute! Bats improves life for anyone developing or using bash scripts and programs everywhere. But its benefits go beyond the borders of current bash code; all the “glue” scripts written in bash-flavored shell scripts benefit as well.

Bats also makes bash a more attractive place to program, as other languages have multiple mature testing tools and frameworks at their disposal. It’s long past time that the most popular and widespread dialect of shell scripting had something similar!

Let’s iron out the remaining wrinkles of bats and improve the coding hygiene of bash developers everywhere. ☻

[Zearin]

(P.S.: May The Source™ be with you!)

[vanosg]

As an add-on to the TAPS feature request, perhaps adding a method with which to organize the tests into subtests, particularly so that they can output a TAPS-compliant subtest format? That format is something similar to

  1..3
  ok 1 - First test
      1..2
      ok 1 - This is a subtest
      ok 2 - So is this
        1..2
        ok 1 - This is a subtest
        ok 2 - So is this
  ok 2 - An example subtest
  ok 3 - Third test

Thanks!

[Sylvain303]

@ztombol I will work on "Test files including other test files." you can assign to me. #99 #169

[Sylvain303]

@ztombol you can also assign me #36 "Running only a specific set of tests".

[btamayo]

@ztombol @vanosg @Sylvain303 @Zearin:

As per my comment here, I'm migrating @ztombol's fantastic work here to the new repo's Road to v1.0 Project. Please direct PRs there! Thank you.

I'll be turning them into Issues with assignments, etc. as well.

[btamayo]

@Sylvain303 Are you still interested in completing those issues but over at bats-core/bats-core? If you are, I can/will assign you to them once I recreate the relevant issue.

[Sylvain303]

@btamayo unfortunately I'm missing time. I'm interested, by those issue, #36 + #99 because I need it to complete and release an internal unit test of bats itself. But no change since many month. I saw your interest to wake up bats project, but didn't read your roadmap, yet. :) So yes… Assign them to me and touch me, so I will finally fix it.

That way it will progress. Thanks for your job!

[jasonkarns]

@ztombol #89 (Empty output lines are missing from ${lines[@]}) can be marked resolved.

#89 was resolved by #90 (which was merged). #92 intended to revert #90 (as #90 fixed it in a backwards incompatible way) However, #90 was never merged, so the fix (#90) is still in master. As we are prepping for 1.0, even if we decided to revert the fix, we would restore the fix for 1.0.

[jasonkarns]

@ztombol

#163 has now been merged over at bats-core as bats-core/bats-core#17

and #141/#142 has been transferred over to bats-core as bats-core/bats-core#19