Flaky test: TestBasic PartitionRecoveryPartOffline #2545
Description
Description
Travis Test Errors Reference Spreadsheet for details on cases where this test has failed: https://docs.google.com/spreadsheets/d/1apjaNvZodnTSIdZaQlpMaqf0nzbKgSYdIWHqvmINHS8/edit#gid=0
Example Failure
Time: Thu 7/15/2021
Build: https://app.travis-ci.com/github/algorand/go-algorand/jobs/524585138#L1706
Travis Test: Ubuntu AMD64 Integration Test
Package: test/e2e-go/features/partitionRecovery
Test: TestBasicPartitionRecoveryPartOffline
=== FAIL: test/e2e-go/features/partitionRecovery TestBasicPartitionRecoveryPartOffline (45.79s)
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestBasicPartitionRecoveryPartOffline/Wallet3.rootkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/10pct.rootkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/20pct.rootkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node2.rootkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node1.rootkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestBasicPartitionRecoveryPartOffline/Wallet1.rootkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestBasicPartitionRecoveryPartOffline/Wallet2.rootkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/20pct.0.3000000.partkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/10pct.0.3000000.partkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node3.rootkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node4.rootkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestBasicPartitionRecoveryPartOffline/Wallet1.0.3000000.partkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestBasicPartitionRecoveryPartOffline/Wallet2.0.3000000.partkey
https://github.com/[secure]foundation/specs/tree/65b4ab3266c52c56a0fa7d591754887d68faad0a 100000
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node1.0.3000000.partkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node2.0.3000000.partkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node5.rootkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node6.rootkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node3.0.3000000.partkey
Created new rootkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node7.rootkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node5.0.3000000.partkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node4.0.3000000.partkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node6.0.3000000.partkey
Created new partkey: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node7.0.3000000.partkey
https://github.com/[secure]foundation/specs/tree/65b4ab3266c52c56a0fa7d591754887d68faad0a 100000
algod(11751) : No REST API Token found. Generated token: 08d1939fc76f675ea99d4a78f602544e427c7773b438ea26ae0da1d8ccc94867
algod(11751) : No Admin REST API Token found. Generated token: 01fcd26614dc54dfc03d67306a4b271daaa7ed7152bc428097fb24c26f1a8771
algod(11751) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestBasicPartitionRecoveryPartOffline/Primary/node.log
algod(11758) : No REST API Token found. Generated token: c84e5581495496709a38a4887f61459fa1ba7ea2ca3c808f5955956b696b40f7
algod(11758) : No Admin REST API Token found. Generated token: ca66d3fa6becf4888c09277d9eed308c9db81884d22380dd2894793204946bf0
algod(11758) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Primary/node.log
algod(11751) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11751) : Initializing the Algorand node...
algod(11751) : Success!
algod(11751) : ⇨ http server started on 127.0.0.1:34093
algod(11751) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:34093. Press Ctrl-C to exit
algod(11758) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11758) : Initializing the Algorand node...
algod(11758) : Success!
algod(11758) : ⇨ http server started on 127.0.0.1:33627
algod(11758) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:33627. Press Ctrl-C to exit
algod(11766) : No REST API Token found. Generated token: 4ba622c950cbd042292a3c9410c44a3a1740b82bee694a64d4943ab97fd71fcd
algod(11766) : No Admin REST API Token found. Generated token: b6070d1259a6fdcac1233dff7438c1b7df021a5b35de06dd8ef58bfaebb06663
algod(11766) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestBasicPartitionRecoveryPartOffline/Node1/node.log
algod(11772) : No REST API Token found. Generated token: 0b4b155e76723b3834bef6fd5b991ba419da0eb64b243ef604a8c7723ded6430
algod(11772) : No Admin REST API Token found. Generated token: 392641dd4cb524d532a28c40519fd688e5cbbd651b447f4b1b141159b3fc1065
algod(11772) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Relay2/node.log
algod(11766) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11766) : Initializing the Algorand node...
algod(11766) : Success!
algod(11766) : ⇨ http server started on 127.0.0.1:42665
algod(11766) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:42665. Press Ctrl-C to exit
algod(11772) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11772) : Initializing the Algorand node...
algod(11772) : Success!
algod(11772) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:37461. Press Ctrl-C to exit
algod(11772) : ⇨ http server started on 127.0.0.1:37461
algod(11779) : No REST API Token found. Generated token: 668f9be0d3896d05c41eb745f8c6bacaff3689c729f0528cd86708956eed4020
algod(11779) : No Admin REST API Token found. Generated token: 60ef6bd859aadda2416d1f931ed807b822d746ee0033c29a459d45f62cad9aae
algod(11779) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestBasicPartitionRecoveryPartOffline/Node2/node.log
algod(11787) : No REST API Token found. Generated token: c40ec655ad09e9bacb19606e5deeb2b4b2bbb61c1e85d6d6d561fb6466ebf9ff
algod(11787) : No Admin REST API Token found. Generated token: 0da088badb3ac92542282827fa70877555dc327b5a7f574ebabf39de5bb5dc0a
algod(11787) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node9/node.log
algod(11779) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11779) : Initializing the Algorand node...
algod(11779) : Success!
algod(11779) : ⇨ http server started on 127.0.0.1:45965
algod(11779) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:45965. Press Ctrl-C to exit
algod(11787) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11787) : Initializing the Algorand node...
algod(11787) : Success!
algod(11787) : ⇨ http server started on 127.0.0.1:46569
algod(11787) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:46569. Press Ctrl-C to exit
algod(11808) : No REST API Token found. Generated token: e5f733fe9f8c65011851bc9496a6d304446aa6d156355e663f4534bd5db7398e
algod(11808) : No Admin REST API Token found. Generated token: 30c066416de445937f1a2ad03e8a92f4b96ea0c48d5b468e28f81cd55db22b21
algod(11808) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node10/node.log
algod(11808) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11808) : Initializing the Algorand node...
algod(11808) : Success!
algod(11808) : ⇨ http server started on 127.0.0.1:35813
algod(11808) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:35813. Press Ctrl-C to exit
algod(11820) : No REST API Token found. Generated token: 59291473a90896c6db8f0f3b59586b9731eb4170ce5fd7a9ff103ebe1fa28260
algod(11820) : No Admin REST API Token found. Generated token: ba6b045339754abc9f27ffe24c5d9b35939e445869f52f4412c0cbdac70b8ab1
algod(11820) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node1-3/node.log
algod(11820) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11820) : Initializing the Algorand node...
algod(11820) : Success!
algod(11820) : ⇨ http server started on 127.0.0.1:39595
algod(11820) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:39595. Press Ctrl-C to exit
algod(11838) : No REST API Token found. Generated token: b56d7c57747773b348b854868dc1fff3628876391ae852a8d204ce33554ec52b
algod(11838) : No Admin REST API Token found. Generated token: b8c91a9257d82c1e2a5a378b036f0f67a686789c5c07d7268a8886fc579d735c
algod(11838) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node4/node.log
algod(11838) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11838) : Initializing the Algorand node...
algod(11838) : Success!
algod(11838) : ⇨ http server started on 127.0.0.1:35849
algod(11838) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:35849. Press Ctrl-C to exit
algod(11849) : No REST API Token found. Generated token: 4154c860d9fc9458ef57f37c68b46dea2850e4c7498814e2a04a12548c1771a2
algod(11849) : No Admin REST API Token found. Generated token: 2e25d4a2485740930c369094a4aae86ed1935a9b32491bbf404f23c6c011b67c
algod(11849) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node5/node.log
algod(11849) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11849) : Initializing the Algorand node...
algod(11849) : Success!
algod(11849) : ⇨ http server started on 127.0.0.1:35461
algod(11849) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:35461. Press Ctrl-C to exit
algod(11863) : No REST API Token found. Generated token: c6b54f1e9e8f0cdda7eab400918f85bbccf0d7a3e119f24f5529d8a7bc5a8c0a
algod(11863) : No Admin REST API Token found. Generated token: 897b34b20ba5b9a2b4e4b0e1bca107cb6c45dd29a20be1192c5cacadb668d7d0
algod(11863) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node6/node.log
algod(11863) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11863) : Initializing the Algorand node...
algod(11863) : Success!
algod(11863) : ⇨ http server started on 127.0.0.1:37965
algod(11863) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:37965. Press Ctrl-C to exit
algod(11875) : No REST API Token found. Generated token: 41e405d9e882ad7a5c652f05f85d2fcf0c9ae11ba7a923c7fee70e30dcdb21b7
algod(11875) : No Admin REST API Token found. Generated token: f009c6bcdb77f3153fcad13190df359ec13486847cf13bcaea181b12faf37036
algod(11875) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node7/node.log
algod(11875) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11875) : Initializing the Algorand node...
algod(11875) : Success!
algod(11875) : ⇨ http server started on 127.0.0.1:39307
algod(11875) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:39307. Press Ctrl-C to exit
algod(11884) : No REST API Token found. Generated token: 7eb3dcd68398ecbfbe9f9a11928cb6bff1eefd47c74f05f4180257c00f373886
algod(11884) : No Admin REST API Token found. Generated token: 8069eaaf0704809e4ba1342bdf25884a9a21114d019f9b995e51973aa1c3ea70
algod(11884) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestPartitionHalfOffline/Node8/node.log
algod(11884) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11884) : Initializing the Algorand node...
algod(11884) : Success!
algod(11884) : ⇨ http server started on 127.0.0.1:41075
algod(11884) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:41075. Press Ctrl-C to exit
algod(11766) : Exiting on terminated
algod(11779) : Exiting on terminated
algod(11918) : Logging to: /home/travis/build/[secure]/go-[secure]/tmp/out/e2e/11471-1626314244810/TestBasicPartitionRecoveryPartOffline/Node1/node.log
algod(11918) : Deadlock detection is set to: enabled (Default state is 'enable')
algod(11918) : Initializing the Algorand node...
algod(11918) : Success!
algod(11918) : ⇨ http server started on 127.0.0.1:40227
algod(11918) : Node running and accepting RPC requests over HTTP on port 127.0.0.1:40227. Press Ctrl-C to exit
fixture.go:105:
Error Trace: partitionRecovery_test.go:202
Error: Not equal:
expected: 0x3
actual : 0x5
Test: TestBasicPartitionRecoveryPartOffline
Messages: We should not have made progress since stopping the first node
algod(11751) : Exiting on terminated
algod(11918) : Exiting on terminated