This project provides addons for TestBalloon, the next generation KMP-first, coroutine-first testing framework.

This project originally started as a shim to make migration from Kotest easier, after being dissatisfied with the Kotest framework's second-class KMP support and third-class Android support.

At the same time, the Kotest libraries—its assertions, the way it models property testing, etc.—are still unrivaled and don’t suffer from the framework’s shortcomings. Paired with TestBalloon’s flexibility and small API surface, this provides the best of both worlds.

TestBalloon Addons provide:

• data-driven testing
• property testing
• per-suite and per-test fixture generation
• FreeSpec test style as known from Kotest

The real strength of TestBalloon Addons emerges when multiple features are combined—especially when FreeSpec allows entire test hierarchies to be expressed almost like natural language. By eliminating boilerplate and ceremony, this keeps the focus squarely on the tests themselves, so a test suite already tells its story at first glance. The following example demonstrates how this looks like in practice.

val combinedFeaturesSuite by testSuite {

    // Generate a fresh fixture for every test
    withFixtureGenerator { Random.nextInt() } - {

        "A FreeSpec-style suite with generated fixtures" - { freshSeed ->

            // Data-driven tests
            withData(1, 2, 3) { multiplier ->
                "works for simple data-driven cases" {
                    val result = freshSeed * multiplier
                    // assert something about result
                }
            }

            // Property-based tests
            checkAll(iterations = 50, Arb.int(0..10)) { value ->
                "also supports property testing" {
                    val result = freshSeed + value
                    // assert an invariant about result
                }
            }
        }
    }
}

Feature Overview

This project consists of the following modules:

• datatest replicates Kotest's data-driven testing features for TestBalloon
• property brings Kotest's property testing to TestBalloon
• fixturegen introduces per-test fixture generation for TestBalloon without boilerplate, and beyond TestBalloon's current fixture generation capabilities
• freespec emulates Kotest's FreeSpec test style for TestBalloon

Tip

freespec and fixturegen are modulated into the fixturegen-freespec module. This means: if you add the at.asitplus.modulator Gradle plugin to any project that uses both, you can automagically combine FreeSpec syntax and per-test fixture generation.

If you don't want to use modulator, you can add the at.asitplus.testballoon:fixturegen-freespec:$version dependency manually to your project.

Compatibility Matrix

TestBalloon Addons	TestBalloon
0.7.0 - 0.7.1	0.8.2 (Kotlin 2.3.0)
0.7.0-RC	0.8.0-RC (Kotlin 2.3.0)
0.1.1–0.6.1	0.7.1 (Kotlin 2.2.21)
0.1.0	0.7.0 (Kotlin 2.2.21)

Test Name Truncation

Caution

TestBalloon jumps through quite some hoops to avoid the shortcomings of the underlying Gradle-based test infrastructure and file system limitations eating your cat. However, deep nesting and exceptionally long test names (both of which are easily produced when using data-driven testing or property testing) can still cause errors or even crashes.

This is especially true for Android device/emulator-based test execution, which is a wondrous mess!

Because TestBalloon can only shorten test names (not suite names), truncation becomes useful.

All modules allow setting global defaults with regard to test name truncation. These properties are called:

• defaultTestNameLength
• defaultDisplayNameLength

The former generally defaults to 64 characters (15 on Android). Display names are not truncated by default.

Both properties can be set in two ways:

• globally (e.g., TestBalloonAddons.defaultTestNameLength = 15)
• per test style (e.g., FreeSpec.defaultTestNameLength = 10, PropertyTest.defaultDisplayNameLength = 100)

Per-style configuration takes precedence over global configuration. Hence, per-style configuration property setters are nullable, even though their getters will never return null, as they fall back to the global configuration properties automatically.

It is also possible to set test name length and display name length for individual tests by passing the maxLength and displayNameMaxLength parameters, respectively. Truncated names are ellipsised in the middle, not just cut off at the end.

→ Check out the full API docs for each test style for all configuration options!

By-Default Sane Test Names

TestBalloon Addons use sane default stringification for test names of collection and array types inside data-driven tests and property tests:

• All primitive arrays are correctly joined to string (i.e. [-1, 4, -643, 34310])
• All unsigned arrays are correctly joined to string (i.e. [9, 76, 145, 9365])
• ByteArray and UByteArray use hex uppercase notation (i.e. CA:FE:BA:BE)

Compacting Test Series

Data-driven testing and property testing can easily produce millions of individual cases being tested. To avoid making the test runner's heap explode in such cases, the datatest and property modules allow for compacting test series.

Just pass the compact = true parameter when creating data-driven tests or property tests (see examples in the module descriptions for data-driven testing and property testing).

The names of compacted test series consist of an uppercase sigma (Σ) followed by the test series' datatype (e.g., ΣULong, ΣByteArray, …).

To still get intelligible output about which precise data point(s) caused failing tests, the error message of the resulting failed assertion will list everything that failed and which succeeded:

java.lang.AssertionError: ΣString
Error: 1: 4: expected:<three> but was:<4>
Error: 2: one: expected:<three> but was:<one>
Error: 3: null: Expected "three" but actual was null
Error: 4: null: Expected "three" but actual was null
Error: 5: null: Expected "three" but actual was null
Error: 6: two: expected:<three> but was:<two>
OK:    7: three
Error: 8: four: expected:<three> but was:<four>
----------------------------------------

The stack trace of the thrown exception is the stack trace of the first error (which is equal to the stack traces of all failed assertions). As such, you can directly navigate to the error with the same convenience as ever!

On the JVM: the individual exceptions of all failed test series' individual tests are added to the top-level assertion error as suppressed exceptions.

To globally enable compacting test series for data-driven testing and property testing, set DataTest.compactByDefault = true and PropertyTest.compactByDefault = true, respectively.

Compacting works on test and suite level!

In addition, it is possible to specify a prefix parameter when defining data-driven tests or property tests. The prefix is prepended to generated test names (in front of the sigma), which helps navigate large test graphs.

→ Check out the full API docs for each test style for all configuration options!

Modules

Data-Driven Testing

Maven Coordinates	at.asitplus.testballoon:datatest:$version

Note

Deep nesting will produce a large number of tests, making the heap explode. Either manually compact tests as in the second example below (works for both withData and withDataSuites), or set the global DataTest.compactByDefault = true to automatically compact all data-driven tests.

TestBalloon makes it ridiculously easy to roll your own data-driven testing wrapper with just a couple of lines of code. So we did, by replicating Kotest's data-driven testing API:

import at.asitplus.testballoon.withData
import at.asitplus.testballoon.withDataSuites
import de.infix.testBalloon.framework.core.testSuite

val aDataDrivenSuite by testSuite {
    withDataSuites(1, 2, 3, 4) { number ->
        withData("one", "two", "three", "four") { word ->
            //your test logic being run 16 times
        }
    }

    //Alternative syntax for withDataSuites
    // -> NOTE the minus ↙↙↙
    withData(1, 2, 3, 4) - { number ->
        // Will create only a single test, but the error will contain all failed inputs
        withData("one", "two", "three", "four", compact = true) { word ->
            //your test logic being run 16 times
        }
    }
}

It is possible to specify a prefix parameter when defining data-driven tests and suites. The prefix is prepended to generated test names, which helps navigate large test reports.

Running individual tests from the gutter is not possible, as the test suite structure and the names of suites and tests are computed at runtime. Hence, you must run the entire suite (but you can manually filter using wildcards).

Property Testing

Maven Coordinates	at.asitplus.testballoon:property:$version

Note

Deep nesting will produce a large number of tests, making the heap explode. Either manually compact tests as in the first example below (works for both checkAll and checkAllSuites), or set the global PropertyTest.compactByDefault = true to automatically compact all data-driven tests.

Although it comes with some warts, kotest-property is still extremely helpful for generating a large corpus of test data—especially as it covers many edge cases out of the box. Again, since TestBalloon has been specifically crafted to be flexible and extensible, we did just that:

import at.asitplus.testballoon.checkAll
import at.asitplus.testballoon.checkAllSuites
import de.infix.testBalloon.framework.core.testSuite
import io.kotest.property.Arb
import io.kotest.property.arbitrary.byte
import io.kotest.property.arbitrary.byteArray
import io.kotest.property.arbitrary.int
import io.kotest.property.arbitrary.uLong

val propertySuite by testSuite {
    // DON'T generate a suite for each item. Instead: aggregate >->-->------------↘↘↘↘↘↘↘↘↘↘↘↘
    checkAllSuites(iterations = 100, Arb.byteArray(Arb.int(100, 200), Arb.byte()), compact = true) { byteArray ->
        checkAll(iterations = 10, Arb.uLong()) { number ->
            //test with byte arrays and number for fun and profit
        }
    }

    //Alternative syntax for checkAllSuites
    // --> NOTE THE MINUS HERE >->-->--------------------------------------↘↘↘
    checkAll(iterations = 100, Arb.byteArray(Arb.int(100, 200), Arb.byte())) - { byteArray ->
        checkAll(iterations = 10, Arb.uLong()) { number ->
            //test with byte arrays and number for fun and profit
        }
    }
}

It is possible to specify a prefix parameter when defining property tests and suites. The prefix is prepended to generated test names, which helps navigate large test reports.

Running individual tests from the gutter is not possible, as the test suite structure and the names of suites and tests are computed at runtime. Hence, you must run the entire suite (but you can manually filter using wildcards).

On-Demand Fixture Generation

Maven Coordinates	at.asitplus.testballoon:fixturegen:$version

TestBalloon enforces a strict separation between blue code and green code. This is a good thing—especially for deeply nested test suites—and it supports deep concurrency. Hence, ye olde JUnit4-style @Before and @After hacks mutating global state are deliberately not supported.

Sometimes, though, you really want fresh data for every test or suite—in effect, you want to generate a fresh test fixture for every test/suite.

Note

Fixture generation as provided by the addons does not use TestBalloon's native fixtures, as those only work in green code. The flavour of fixture generation provided by TestBalloon Addons works for suites (blue code) and tests (green code), as shown below.

import at.asitplus.testballoon.withFixtureGenerator //<- Look ma, only a single import!
import de.infix.testBalloon.framework.core.testSuite
import kotlin.random.Random
import kotlinx.coroutines.delay //just to get some suspending demo generator

val aGeneratingSuite by testSuite {

    //seed before the generator function, not inside!
    val byteRNG = Random(42);
    //We want to test with fresh randomness, so we generate a fresh fixture for each test
    withFixtureGenerator { byteRNG.nextBytes(32) } - {

        repeat(5) {
            test("Generated test with fresh randomness") { freshFixture ->
                //your test logic here
            }
        }

        testSuite("Generated Suite with fresh randomness") { freshFixture ->
            test("using the outer fixture") {
                //your logic based on freshFixture here
            }
        }
        repeat(5) {
            //✨ it ✨ just ✨ werks ✨
            test("Test with implicit fixture name `it`") {
                //do something with `it`, it contains fresh randomness!
            }
        }
    }

    //seed the RNG for reproducible tests
    val random = Random(42)

    //reference function to be called for each test inside withFixtureGenerator
    withFixtureGenerator(random::nextFloat) - {
        repeat(10) {
            test("Generated test with random float") {
                //test something floaty!
            }
        }
        test("And some other test that des not conform to the shema from the loop") {
            //test something different, with a fresh float
        }
    }

    //always-the-same fixtures also work, of course
    withFixtureGenerator {
        object {
            var a: Int = 1
            val b: Int = 2
        }
    } - {
        test("one") {
            it.a++ //and we can even modify them in one test
            println("a=${it.a}, b=${it.b}") //a=2, b=2
        }
        test("two") {
            //without affecting the other!
            println("a=${it.a}, b=${it.b}") //a=1, b=2
        }
    }

    //Let's test some nasty bug that shows itself only sometimes functionality
    val ageRNG = Random(seed = 26)
    withFixtureGenerator {
        class ABuggyImplementation(val age: Int) {
            fun restrictedAction(): Boolean =
                if (age < 18) false
                else if (age > 18) true
                else Random.nextBoolean() //introduce jitter to simulate a faulty implementation
        }

        //create new object for each test
        ABuggyImplementation(ageRNG.nextInt(0, 99))
    } - {
        repeat(1000) {
            test("Generated test accessing restricted resources") {
                //test `restrictedAction` across a wide age range
                //a thousand times to unveil the bug
            }
        }
    }
}

Warning

A fixture-generating scope is intended to be consumed by the scope directly below it (i.e. the outermost test suite, or directly by a test). Programmatically, you can mix this up and it will compile, but it will not run!

The following is an antipattern:

val outermostSuite by testSuite {
  withFixtureGenerator(random::nextFloat) - {
    testSuite("outer") { /*fixture implicitly available as `it`*/
      test("nested") { float -> /**`it` is not available, explicit parameter specification messes things up*/
        //This will throw a runtime error, because "nested" will be erroneously wired directly below the outermos suite
      }
    }
  }
}

FreeSpec

Maven Coordinates	at.asitplus.testballoon:freespec:$version

At A-SIT Plus, we've been using Kotest's FreeSpec for its expressiveness, as it allows modeling tests and test dependencies close to natural language.

TestBalloon is flexible enough to emulate FreeSpec with very little code, if you have context parameters enabled for your codebase:

Setting up context parameters

// build.gradle.kts
kotlin {
    compilerOptions {
        freeCompilerArgs.add("-Xcontext-parameters")
    }
}

import at.asitplus.testballoon.invoke
import at.asitplus.testballoon.minus
import de.infix.testBalloon.framework.core.TestConfig
import de.infix.testBalloon.framework.core.TestInvocation
import de.infix.testBalloon.framework.core.invocation
import de.infix.testBalloon.framework.core.singleThreaded
import de.infix.testBalloon.framework.core.testSuite

val aFreeSpecSuite by testSuite {
    //testConfigs are supported for suites
    "The outermost blue code"(testConfig = TestConfig.singleThreaded()) - {
        "contains some more blue code" - {
            ", some green code inside the lambda" {
                // your test logic here
            }
            //testConfigs are supported for Tests
            ", and some more green code inside the second lambda"(testConfig = TestConfig.invocation(TestInvocation.SEQUENTIAL)) {
                // more test logic here
            }
        }
        "And finally some more blue code" - {
            "!With some final disabled green code in this lambda" {
                //additional, disabled test logic here
            }
        }
    }
}

Running individual tests from the gutter is not (yet) possible, due to the intricacies of how code analysis works. Hence, you must run the entire suite (but you can manually filter using wildcards).
(You can, of course, just migrate off FreeSpec and use TestBalloon's native functions to create suites and tests.)

Combining with FixtureGen

Maven Coordinates (if not using modulator)	at.asitplus.testballoon:fixturegen-freespec:$version

[!WARNING]
As without FreeSpec syntax, a fixture-generating scope is intended to be consumed by the scope directly below it (i.e. the outermost test suite, or directly by a test). To disambiguate and be explicit about this, explicit parameter specification is required, starting with TestBalloon Addons 0.6.0.

import at.asitplus.testballoon.withFixtureGenerator //   <- Look ma, only regular generatingFixture import!
import at.asitplus.testballoon.invoke //              <- Look ma, only regular freespec import!
import at.asitplus.testballoon.minus  //              <- Look ma, only regular freespec import!
import de.infix.testBalloon.framework.core.testSuite
import kotlin.random.Random

val aGeneratingFreeSpecSuite by testSuite {

    //any lambda with any return type is a fixture generator. Type is reified.
    withFixtureGenerator { Random.nextBytes(32) } - {

        "A Suite with fresh randomness" - { freshFixture ->
            "Consuming outer fixture" {
                //your freshFixture-based test logic here
            }

            withFixtureGenerator { Random.nextBytes(32) } - {
                "With fresh inner fixture" { inner ->
                    //your test logic here with always fresh inner
                    //and fixed freshFixture from outer scope
                }
            }

        }

        repeat(100) {
            "Generated test with fresh randomness" { freshFixture ->
                //some more test logic; each call gets fresh randomness
            }
        }

        //parameter must be explicitly specified to disambiguate
        "Test with fixture name `it`" { it ->
            //no need for an explicit parameter name here, just use `it`
        }

        "And we can even nest!" - {
            withFixtureGenerator { Random.nextBytes(16) } - {
                repeat(10) {
                    "pure, high-octane magic going on" { it ->
                        //Woohoo! more randomness each run
                    }
                }
            }
        }
    }
}

Contributing

External contributions are greatly appreciated! Just be sure to observe the contribution guidelines (see CONTRIBUTING.md).

---

The Apache License does not apply to the logos, (including the A-SIT logo) and the project/module name(s), as these are the sole property of A-SIT/A-SIT Plus GmbH and may not be used in derivative works without explicit permission!