Testing with multiple suites has them run in parallel

[seantalts]

If use go test on all of your packages at once and there are multiple test suites, the suites will end up running in parallel with other suites.

[matryer]

Interesting - is this a bug or feature? :)

On May 2, 2014, at 2:15 PM, seantalts notifications@github.com wrote:

If use go test on all of your packages at once and there are multiple test suites, the suites will end up running in parallel with other suites.

—
Reply to this email directly or view it on GitHub.

[seantalts]

I think it's a bug, as normally go test will only execute tests in parallel if they mark themselves as parallel by calling *testing.T.Parallel().

[nelsam]

I've been running in to this lately, and have been trying to find a fix. Work on it is slow, though - I'm swamped.

[tylerstillwater]

Thanks for looking in to this, @nelsam :)

[MysticHLE]

I believe our team is running into this too today.

Basically I created a new test file in a particular package with some barebone test structure - no actual tests...just an empty struct type that embeds suite.Suite, and a function that takes in a *testing.T object and calls suite.Run() on said struct. This immediately caused all our other tests to start failing indeterministically. The nature of the failures were associated with database unique key integrity violations on inserts and deletes into a single Postgres DB. This is leading me to believe that the tests were being run concurrently without calling our setup methods to prepare the environment properly between tests. Needless to say, the moment I move this test file to another package, everything magically works!

We are not choosing to run any of the tests concurrently.

[matryer]

Would an option to run synchronously be the solution?

[nelsam]

The issue really boils down to the fact that I don't know what's causing suites to run in parallel. I've done some research (again, I'm swamped), but haven't really gotten much worthwhile about what's causing it.

Technically, tests that can be run in parallel should call t.Parallel(), but all suite methods are being run in parallel without having that called.

I'll see what I can find out today, but no promises. This is on my priority list, but it's below several other things I'm working on.

[matryer]

@tylerb how about two eyes on it?

On Oct 29, 2014, at 16:51, nelsam notifications@github.com wrote:

The issue really boils down to the fact that I don't know what's causing suites to run in parallel. I've done some research (again, I'm swamped), but haven't really gotten much worthwhile about what's causing it.

Technically, tests that can be run in parallel should call t.Parallel(), but all suite methods are being run in parallel without having that called.

I'll see what I can find out today, but no promises. This is on my priority list, but it's below several other things I'm working on.

—
Reply to this email directly or view it on GitHub.

[seantalts]

They're running in parallel because of the implementation of golang's
testing.Run that testify's suite.Run calls. It spawns a new goroutine and
communicates with it over a channel.

HTH, on a bus sorry.
On Oct 29, 2014 5:53 PM, "Mat Ryer" notifications@github.com wrote:

@tylerb how about two eyes on it?

On Oct 29, 2014, at 16:51, nelsam notifications@github.com wrote:

The issue really boils down to the fact that I don't know what's causing
suites to run in parallel. I've done some research (again, I'm swamped),
but haven't really gotten much worthwhile about what's causing it.

Technically, tests that can be run in parallel should call t.Parallel(),
but all suite methods are being run in parallel without having that called.

I'll see what I can find out today, but no promises. This is on my
priority list, but it's below several other things I'm working on.

—
Reply to this email directly or view it on GitHub.

—
Reply to this email directly or view it on GitHub
#52 (comment).

[nelsam]

Found it.

It basically looks like testing.RunTests and testing.InternalTest are not intended for use externally. I'll see what I can do. I think our original version just walked through methods calling them individually (instead of adding them to a slice and calling testing.RunTests), and I think that would be a better choice than what we currently have; however, I'd love it if I could get individual test methods to respect both testing.T.Parallel() and the -parallel flag.

For the time being, I'll probably just strip out the testing.InternalTest and testing.RunTests usage, because (as I mentioned previously) I'm swamped and don't have much time to dedicate to this. However, I'll keep an item on my todo list to figure out a way to make use of the parallel test stuff. I'm pretty sure GoCheck works with those flags, but their solution is pretty complicated, and I think I'd rather have a simple solution than a complete solution. Anyone who really needs that functionality can probably just use GoCheck.

EDIT: Actually, I think I have a better idea. Sorta. Maybe. We'll see how much it breaks things.

[nelsam]

... Argh.

Okay, so, I was pretty sure I had found the problem, but I was (to put it simply) wrong. I added channels to suite.Run() to ensure that only one instance of suite.Run() was calling testing methods at a time, and then I just removed everything so that suite.Run() called every method in sequence (without using testing.InternalTest or testing.RunTests). Same issue, both times.

My boss found this, which suggests that the issue isn't exclusive to our suites: http://stackoverflow.com/questions/15721238/go-serial-execution-of-package-tests

It looks like relying on serial execution when using go test ./... just isn't supported without tweaking. I'm going to try to find a way to swap things around so that I can use transactions, because I still want the queries to actually run against my database engine (for checking query syntax); that might require a pretty big change to my application code, but we'll see.

[MysticHLE]

As that StackOverflow thread pointed out, it's possible to use the -p=1 flag to force the maximum number of packages to be ran in parallel (with 1 effectively serializing it all). This doesn't seem to be a problem with testify - but rather, just how go test works.

Another thread for this issue is here:

http://stackoverflow.com/questions/23715302/go-how-to-run-tests-for-multiple-packages

[hectorj]

Can confirm @MysticHLE 's solution works. Go documentation matches: https://golang.org/cmd/go/#hdr-Compile_packages_and_dependencies

-p n
the number of programs, such as build commands or
test binaries, that can be run in parallel.
The default is the number of CPUs available, except
on darwin/arm which defaults to 1.

Ticket should be closed I think.

[oryband]

I guess it wouldn't hurt to document this in the godoc or README or something. Looks like some people already stumbled into this and where clueless. Me included.

[boazavital]

just -parallel 1 may not be sufficient, see golang/go#19280
Looks like in the next release there will be a parallel=-1 to make tests sequential https://go-review.googlesource.com/c/go/+/44530

[tisonkun]

How to reproduce this problem, is there a gist? I cannot reproduce by

package main

import (
	"testing"
	"time"

	"github.com/stretchr/testify/suite"
)

type ExampleTestSuite struct {
	suite.Suite
}

func TestExampleTestSuite(t *testing.T) {
	suite.Run(t, new(ExampleTestSuite))
}

func (s ExampleTestSuite) TestSimple() {
	time.Sleep(2*time.Second)
	s.True(true)
}

type ExampleTestSuite2 struct {
	suite.Suite
}

func TestExampleTestSuite2(t *testing.T) {
	suite.Run(t, new(ExampleTestSuite2))
}

func (s ExampleTestSuite2) TestSimple1() {
	time.Sleep(2*time.Second)
	s.True(true)
}

It executes in 4 seconds.

[martelskiy]

How to reproduce this problem, is there a gist? I cannot reproduce by

package main

import (
	"testing"
	"time"

	"github.com/stretchr/testify/suite"
)

type ExampleTestSuite struct {
	suite.Suite
}

func TestExampleTestSuite(t *testing.T) {
	suite.Run(t, new(ExampleTestSuite))
}

func (s ExampleTestSuite) TestSimple() {
	time.Sleep(2*time.Second)
	s.True(true)
}

type ExampleTestSuite2 struct {
	suite.Suite
}

func TestExampleTestSuite2(t *testing.T) {
	suite.Run(t, new(ExampleTestSuite2))
}

func (s ExampleTestSuite2) TestSimple1() {
	time.Sleep(2*time.Second)
	s.True(true)
}

It executes in 4 seconds.

try to put suites in different files

[jitendra-1217]

It's how the go test works. Please see this answer.

Yes, as other answers already pointed, tests inside one _test.go file are running in parallel only if you add t.Parallel() and run with -parallel tag.

BUT - by default the command go test runs in parallel tests for different packages, and default is the number of CPUs available (see go help build)

So to disable parallel execution you should run tests like this go test -p 1 ./...

[pellared]

@dolmen, shouldn't this behavior be at least mentioned in package documentation?