proposal: testing: define naming convention for test functions

[taiidani]

This proposal is for eliciting discussion around the current absent naming convention for test functions. This should not involve any changes to the language specification itself but instead serve as a guidepost for community tooling and best practices.

Background

There is precedent within the community and within the Go packages themselves to name test functions in a specific way based on their definitions. Two primary cases demonstrate this.

gotests

The popular gotests tool defines an informal specification for naming tests and uses it to auto-generate table driven tests from a given function definition. It appears to be widely accepted and used among the IDE community.

If accepted into the main specification, the IDE integrations using this tool could be further developed with additional features such as "Jump to Test for Function" / "Jump to Function for Test". Without a specification these features would be based upon an informal specification and subject to potential breakage if the community shifts to another upstream test generation tool.

(Disclaimer -- this issue came from a discussion on gotests: cweill/gotests#106)

testing examples

In the go source for the testing package a naming convention has already been defined for "Example" test cases. Extracting the content from those docs for clarity:

The naming convention to declare examples for the package, a function F, a type T and
method M on type T are:

func Example() { ... }
func ExampleF() { ... }
func ExampleT() { ... }
func ExampleT_M() { ... }

Multiple example functions for a package/type/function/method may be provided by
appending a distinct suffix to the name. The suffix must start with a
lower-case letter.

func Example_suffix() { ... }
func ExampleF_suffix() { ... }
func ExampleT_suffix() { ... }
func ExampleT_M_suffix() { ... }

Having a naming convention for Example test functions gives a solid starting point if we decide to move forward for a standard Test function.

Proposal

I would stop short of officially recommending a naming convention for Go to adopt, but encourage that the discussion start based on the existing examples we are seeing in gotests and in the Example function naming conventions. Looking forward to what everyone thinks!

[ianlancetaylor]

Seems reasonable to me but it's not clear what would actually change in the Go project. Are you suggesting that we add some comments to the testing package doc explaining that the format used for example names should also be used for test names?

[taiidani]

Yes, that's the suggestion -- a documentation update that implies convention.

At the moment the docs are not (to me at least) indicating that they apply to both Tests and Examples. Maybe my interpretation was incorrect? Do they actually already apply to both types?

[ianlancetaylor]

The naming conventions only apply to examples, because the naming of examples affects the way that godoc presents the package documentation. Test names do not affect godoc, as godoc doesn't do anything about tests.

[taiidani]

Ah, that makes a lot of sense! The tooling for godoc required that a specification be formed around Examples in order to accurately parse the source and generate documentation from it.

That's very similar to what's happening on the community side with gotests. They were auto-generating tests from source code and needed a clear specification for their tooling to handle developers' function definitions. From the original discussion:

It was a solution to the fact that in Golang you can have a function called Foo and foo in the same package or on the same struct. So if you generated a test for both, they would have identical test names.

[kylelemons]

I have experimented with a few approaches to this:

1a. Test names with underscore suffixes

// for the primary happy path table-driven tests:
func TestF(t *Testing.T) { ... }
func TestT(t *Testing.T) { ... }
func TestM(t *Testing.T) { ... } // when the method is unambiguous, or there's only one type
func TestT_M(t *Testing.T) { ... }
// for the error or special-case testing:
func TestF_suffix(t *Testing.T) { ... }
func TestT_suffix(t *Testing.T) { ... }
func TestT_M_suffix(t *Testing.T) { ... }

Notice that even within this arguably easy-to-parse version, I tend to elide the "T_" prefix when I think it doesn't cause confusion, i.e. if there's only one major exported type in the package.

Pros:

• Easy to parse.
• Easy to write on autopilot.
• Suffixes don't actually come up a lot, except for methods.

Cons:

• Underscores stand out in Go, and distract from the more interesting things in your test.
• Using both a method and a suffix makes the name stand out a lot

1b. Test names with camelCase suffixes

// for the primary happy path table-driven tests:
func TestF(t *Testing.T) { ... }
func TestT(t *Testing.T) { ... }
func TestM(t *Testing.T) { ... } // when the method is unambiguous, or there's only one type
func TestTM(t *Testing.T) { ... }
// for the error or special-case testing:
func TestFSuffix(t *Testing.T) { ... }
func TestTSuffix(t *Testing.T) { ... }
func TestTMSuffix(t *Testing.T) { ... }

Pros:

• No underscores.
• Easy to write on autopilot.
• Not really that hard for a human to understand.

Cons:

• Potentially ambiguous for a robot to parse.

2. Additional subtests

This looks more like

func TestF(t *Testing.T) {
  tests := ...; for ... {} // happy table-driven tests

  t.Run("bad_inputs", func(t *testing.T) { ... }) // sad table-driven tests
  t.Run("dependency_failures", func(t *testing.T) { ... }) // one-off sub-sub tests, not table driven
}

Pros:

• Subtests are awesome.
• Potentially easy to tie back to a type or function.

Cons:

• Kinda leaves method tests out in the cold, with everyone doing something slightly different.
• Large functions mean you still have to dig around to find the coverage you're looking for.
• Subtests probably doesn't show up nicely in IDE outlines.

---

I don't find a whole lot to like about 1a, because underscores are so uncommon in Go that they stand out like a sore thumb. I thought I would like 2 but it also makes the Test function giant, and can result in suppressing a ton of output if one of the subtests times out.

For my part, I think I tend to mix and match these days. I'll gravitate to 1b when there's usually only one test for the "thing" in the package, and to 2 if there are a lot of interrelated tests that share setup. I do appreciate that both of these make it significantly more difficult for tooling to draw the 1:1 relationships, however.

I have been writing Go for a really long time and don't really know if I have a good idea of what the right pattern is here. In code reviews, I more-or-less let my teammates pick whatever approach makes sense to them, and I just focus on making sure their failure messages are easy to understand.

If the motivation here is to help out IDEs that are looking for tests, you could approach it from a slightly different direction, and develop heuristics based off of tests that pass table fields to the function, name the function in error messages, refer to the function in the doc comment, and/or that just call the function syntactically within the {}s of the test/case.

[rsc]

Test functions should have names that are meaningful to the user, not dictated by some inflexible tool. We shouldn't retroactively invalidate all the names people have picked.

Tools that care about matching code to test functions could look at which functions are called.

[taiidani]

Regarding the proposal, I was hoping more to see a convention in the language that tools could then build against, similar to how Examples and Benchmarks have their own [strict] conventions. The functionality the tools add wouldn't affect any existing test methods; they just would preserve current state. Features that the tools introduce based on the convention would be expected to cleanly handle (or ignore) method names which don't give the tools the pattern they need in order to introspect on the tests' target.

I think that automatically discovering that based on the functions that are called would serve the same purpose! But...I'm having a hard time seeing how that would work? How could a tool (or go test itself) determine which method is the target of the test, sifting through all the setup/teardown and other utility methods that may be called by the same test as it runs?

[rsc]

Benchmarks have no strict convention either.
Only Examples do, because we need to know which function to show them next to in the godoc output.

How could a tool (or go test itself) determine which method is the target of the test

This is not really a well-formed question. Many tests test multiple functions. Only the simplest tests focus on just one function. To the extent that that happens, it is often TestF to test the function F. But it's not a "defined" or required convention.

This seems like a likely decline (infeasible).