proposal: testing: per-subtest setup and cleanup

[unkeep]

When writing tests in BDD style using subtests it's often necessary to invoke some common code before or after a subtest. In the example below we call beforeRun in the first place of subtests 1 and 2, otherwise the subtests would affect the outcome of each other:

type CoffeeMachine struct {
	on bool
}

func (a *CoffeeMachine) TurnOn() error {
	if a.on {
		return fmt.Errorf("aready on")
	}
	a.on = true
	return nil
}

func (a *CoffeeMachine) MakeCoffee() error {
	if !a.on {
		return fmt.Errorf("power is off")
	}

	return nil
}

func TestCoffeeMachine(t *testing.T) {
	// 0
	t.Run("given a new coffee machine", func(t *testing.T) {
		var a CoffeeMachine

		beforeRun := func(t *testing.T) {
			a = CoffeeMachine{}
		}

		// 1
		t.Run("it could be turned on", func(t *testing.T) {
			beforeRun(t)

			err := a.TurnOn()
			assert.NoError(t, err)

			// 1.1
			t.Run("it could not be turned on again", func(t *testing.T) {
				err := a.TurnOn()
				assert.Error(t, err)
			})

			// 1.2
			t.Run("it makes a coffee", func(t *testing.T) {
				err := a.MakeCoffee()
				assert.NoError(t, err)
			})
		})

		// 2
		t.Run("MakeCoffee fails because it's not turned on", func(t *testing.T) {
			beforeRun(t)

			err := a.MakeCoffee()
			assert.Error(t, err)
		})
	})
}

It would be nice instead, if the testing package could invoke a specific function by itself before/after direct subtests of the parent where it's configured:

func TestCoffeeMachine(t *testing.T) {
	// 0
	t.Run("given a new coffee machine", func(t *testing.T) {
		var a CoffeeMachine

		// a new proposed method for testing.T
		t.BeforeRun(func(t *testing.T) {
			a = CoffeeMachine{}
		})

		// 1
		t.Run("it could be turned on", func(t *testing.T) {
			// the functor passed to BeforeRun is called before we reach this point

			err := a.TurnOn()
			assert.NoError(t, err)

			// 1.1
			t.Run("it could not be turned on again", func(t *testing.T) {
				err := a.TurnOn()
				assert.Error(t, err)
			})

			// 1.2
			t.Run("it makes a coffee", func(t *testing.T) {
				err := a.MakeCoffee()
				assert.NoError(t, err)
			})
		})

		// 2
		t.Run("MakeCoffee fails because it's not turned on", func(t *testing.T) {
			// the functor passed to BeforeRun is called before we reach this point

			err := a.MakeCoffee()
			assert.Error(t, err)
		})
	})
}

The points against of the explicit beforeRun(t) calls from subtests are the following:

1. it can simply be skipped by mistake
2. it can be called from a wrong level by mistake

So I propose to add func (t *T) BeforeRun(func(t *testing.T) and func (t *T) AfterRun(func(t *testing.T) methods which should be invoked by the testing package before/after direct subtests of the parent where it's configured.

Relates to #39222 #40984

[ianlancetaylor]

You can get a similar effect today by writing code like

func TestCounter(t *testing.T) {
    var a CoffeeMachine
    wrapper := func(f func(t *testing.T)) func(t *testing.T) {
        return func(t *testing.T) {
            a = CoffeeMachine{}
            f(t)
        }
    }
    t.Run("on", wrapper(func(t *testing.T) {
        err := a.TurnOn()
        ...
    }))
}

You can ensure that wrapper is always called by creating a table of test functions and then writing a loop to call t.Run for each one.

There is probably room for more control over tests and subtests, but in my opinion this proposal is too specific, adding functionality that will rarely be used.

[unkeep]

@ianlancetaylor, yes, I considered a wrapper like yours. But in fact it does not resolve the same points against direct setup in subtests: you can forget to wrap and you can wrap at wrong level.
And I do not agree that the functionality will be rarely used, since having it in the testing package would help to get rid of dependencies such Ginkgo and Goconvey which are quite popular.

[unkeep]

Closed by mistake, sorry

[willfaught]

you can forget to wrap and you can wrap at wrong level.

It seems like you could equally forget to use BeforeRun.

Can you demonstrate what you mean by wrap at the wrong level?

[unkeep]

It seems like you could equally forget to use BeforeRun.

Once you set BeforeRun, you can be sure that it will be called for a new added subtests of the same parent.

Can you demonstrate what you mean by wrap at the wrong level?

I meant to call wrap at other level of subtests than the level where it’s defined (deeper than necessary)

[nerock]

@willfaught I think your objection could also apply to defer but it's something really useful in the language, and this could cover a necessity as @unkeep mentioned currently you have to use an external library or wrap every test function as other comment suggested

[dnephin]

The functionality provided by the testing package today makes it quite easy to accomplish this style of testing. Here's an another example similar to the one above, which I think may do a better job of addressing the problems identified in the PR description.

func TestCoffeeMachine(t *testing.T) {
	givenCoffeeMachine := func(fn func(*testing.T, CoffeeMachine)) {
		m := CoffeeMachine{}
		t.Run("given a new coffee machine", func(t *testing.T) {
			// t.Cleanup( ... ) can be done here
			fn(t, m)
		})
	}

	givenCoffeeMachine(func(t *testing.T, a CoffeeMachine) {
		t.Run("it could be turned on", func(t *testing.T) {
			err := a.TurnOn()
			assert.NilError(t, err)

			t.Run("it could not be turned on again", func(t *testing.T) {
				err := a.TurnOn()
				assert.Error(t, err, "already on")
			})

			t.Run("it makes a coffee", func(t *testing.T) {
				err := a.MakeCoffee()
				assert.NilError(t, err)
			})
		})
	})

	givenCoffeeMachine(func(t *testing.T, a CoffeeMachine) {
		t.Run("MakeCoffee fails because it's not turned on", func(t *testing.T) {
			err := a.MakeCoffee()
			assert.Error(t, err, "power is off")
		})
	})
}

This approach avoids the shared a variable, and instead gives each call to givenCoffeeMachine a reference to the CoffeeMachine. This makes it impossible to forget because you wouldn't have a reference to CoffeeMachine without it. It also makes it very obvious when you're calling it at the wrong level because the function name is completely different, and you're getting a new reference that shadows the previous one.

[cespare]

What can be accomplished with t.AfterRun that cannot be accomplished today with t.Cleanup today?