Broken test: TestBasicPartitionRecovery

[yaovi-a]

Description

Travis Test Errors Reference Spreadsheet for details on cases where this test has failed: https://docs.google.com/spreadsheets/d/1apjaNvZodnTSIdZaQlpMaqf0nzbKgSYdIWHqvmINHS8/edit#gid=0

Example Failure

Time: Thu 7/15/2021
Build: https://app.travis-ci.com/github/algorand/go-algorand/jobs/524585138#L1949
Travis Test: Ubuntu AMD64 Integration Test
Package: test/e2e-go/features/partitionRecovery
Test: TestBasicPartitionRecovery

==================
WARNING: DATA RACE
Read at 0x00c00013beba by goroutine 75:
  testing.tRunner.func1()
      /home/travis/.gimme/versions/go1.14.7.linux.amd64/src/testing/testing.go:960 +0xdd
  runtime.call32()
      /home/travis/.gimme/versions/go1.14.7.linux.amd64/src/runtime/asm_amd64.s:539 +0x3a
  testing.(*T).FailNow()
      <autogenerated>:1 +0x43
  github.com/[secure]/go-[secure]/test/framework/fixtures.(*synchTest).FailNow()
      /home/travis/build/[secure]/go-[secure]/test/framework/fixtures/fixture.go:115 +0x97
  github.com/stretchr/testify/require.NoError()
      /home/travis/gopath/pkg/mod/github.com/stretchr/testify@v1.6.1/require/require.go:1038 +0x102
  github.com/[secure]/go-[secure]/test/framework/fixtures.(*RestClientFixture).ClientWaitForRoundWithTimeout()
      /home/travis/build/[secure]/go-[secure]/test/framework/fixtures/restClientFixture.go:114 +0x23a
  github.com/[secure]/go-[secure]/test/e2e-go/features/partitionRecovery.TestPartitionHalfOffline()
      /home/travis/build/[secure]/go-[secure]/test/e2e-go/features/partitionRecovery/partitionRecovery_test.go:233 +0x4f9
  testing.tRunner()
      /home/travis/.gimme/versions/go1.14.7.linux.amd64/src/testing/testing.go:1039 +0x1eb
Previous write at 0x00c00013beba by goroutine 122:
  testing.(*common).FailNow()
      /home/travis/.gimme/versions/go1.14.7.linux.amd64/src/testing/testing.go:697 +0x4c
  testing.(*T).FailNow()
      <autogenerated>:1 +0x43
  github.com/[secure]/go-[secure]/test/framework/fixtures.(*synchTest).FailNow()
      /home/travis/build/[secure]/go-[secure]/test/framework/fixtures/fixture.go:115 +0x97
  github.com/stretchr/testify/require.NoError()
      /home/travis/gopath/pkg/mod/github.com/stretchr/testify@v1.6.1/require/require.go:1038 +0x102
  github.com/[secure]/go-[secure]/test/framework/fixtures.(*LibGoalFixture).nodeExitWithError()
      /home/travis/build/[secure]/go-[secure]/test/framework/fixtures/libgoalFixture.go:125 +0x2ef
  github.com/[secure]/go-[secure]/test/framework/fixtures.(*LibGoalFixture).nodeExitWithError-fm()
      /home/travis/build/[secure]/go-[secure]/test/framework/fixtures/libgoalFixture.go:107 +0x5f
  github.com/[secure]/go-[secure]/nodecontrol.(*NodeController).StartAlgod.func2()
      /home/travis/build/[secure]/go-[secure]/nodecontrol/algodControl.go:232 +0xe8
Goroutine 75 (running) created at:
  testing.(*T).Run()
      /home/travis/.gimme/versions/go1.14.7.linux.amd64/src/testing/testing.go:1090 +0x700
  testing.runTests.func1()
      /home/travis/.gimme/versions/go1.14.7.linux.amd64/src/testing/testing.go:1334 +0xa6
  testing.tRunner()
      /home/travis/.gimme/versions/go1.14.7.linux.amd64/src/testing/testing.go:1039 +0x1eb
  testing.runTests()
      /home/travis/.gimme/versions/go1.14.7.linux.amd64/src/testing/testing.go:1332 +0x527
  testing.(*M).Run()
      /home/travis/.gimme/versions/go1.14.7.linux.amd64/src/testing/testing.go:1249 +0x43f
  main.main()
      _testmain.go:52 +0x223
Goroutine 122 (finished) created at:
  github.com/[secure]/go-[secure]/nodecontrol.(*NodeController).StartAlgod()
      /home/travis/build/[secure]/go-[secure]/nodecontrol/algodControl.go:224 +0x5e9
  github.com/[secure]/go-[secure]/netdeploy.Network.startNodes()
      /home/travis/build/[secure]/go-[secure]/netdeploy/network.go:339 +0x333
  github.com/[secure]/go-[secure]/netdeploy.Network.Start()
      /home/travis/build/[secure]/go-[secure]/netdeploy/network.go:293 +0x4f2
  github.com/[secure]/go-[secure]/test/framework/fixtures.(*LibGoalFixture).Start()
      /home/travis/build/[secure]/go-[secure]/test/framework/fixtures/libgoalFixture.go:264 +0xf8
  github.com/[secure]/go-[secure]/test/framework/fixtures.(*LibGoalFixture).setup()
      /home/travis/build/[secure]/go-[secure]/test/framework/fixtures/libgoalFixture.go:101 +0x509
  github.com/[secure]/go-[secure]/test/framework/fixtures.(*LibGoalFixture).Setup()
      /home/travis/build/[secure]/go-[secure]/test/framework/fixtures/libgoalFixture.go:67 +0x91
  github.com/[secure]/go-[secure]/test/framework/fixtures.(*RestClientFixture).Setup()
      /home/travis/build/[secure]/go-[secure]/test/framework/fixtures/restClientFixture.go:43 +0x89
  github.com/[secure]/go-[secure]/test/e2e-go/features/partitionRecovery.TestPartitionHalfOffline()
      /home/travis/build/[secure]/go-[secure]/test/e2e-go/features/partitionRecovery/partitionRecovery_test.go:223 +0x24e
  testing.tRunner()
      /home/travis/.gimme/versions/go1.14.7.linux.amd64/src/testing/testing.go:1039 +0x1eb
==================
=== FAIL: test/e2e-go/features/partitionRecovery TestBasicPartitionRecovery (unknown) 
=== FAIL: test/e2e-go/features/partitionRecovery TestPartitionRecoverySwapStartup (unknown)

Doesn't seem to have anything after it specifically so I inclueded the warning that happened before it.

[algonautshant]

The log also shows TestBasicPartitionRecoveryPartOffline failing

    fixture.go:105: 
        	Error Trace:	partitionRecovery_test.go:202
        	Error:      	Not equal: 
        	            	expected: 0x3
        	            	actual  : 0x5
        	Test:       	TestBasicPartitionRecoveryPartOffline
        	Messages:   	We should not have made progress since stopping the first node
algod(11751) : Exiting on terminated
algod(11918) : Exiting on terminated
=== FAIL: test/e2e-go/features/partitionRecovery TestPartitionHalfOffline (unknown)
    fixture.go:105: 
        	Error Trace:	restClientFixture.go:114
        	            				partitionRecovery_test.go:233
        	Error:      	Received unexpected error:
        	            	Get "http://127.0.0.1:39595/v1/status": EOF
        	Test:       	TestPartitionHalfOffline
    fixture.go:105: 
        	Error Trace:	libgoalFixture.go:125
        	            				algodControl.go:232
        	            				asm_amd64.s:1373
        	Error:      	Received unexpected error:
        	            	exit status 2
        	Test:       	TestPartitionHalfOffline
        	Messages:   	Node at /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node1-3 has terminated with error code 2

[algonautshant]

DATA RACE is caused by the test.
After the test errored, the test environment has data race inside the FailNow function.
The real cause is the test failure which is masked by the data race.