Flake: TestLeasingPutGetDeleteConcurrent (*serverWatchStream).recvLoop leak

[ptabor]

Please read https://etcd.io/docs/latest/reporting_bugs/

https://github.com/etcd-io/etcd/pull/12991/checks?check_run_id=2609037848

ok  	go.etcd.io/etcd/tests/v3/integration/clientv3/experimental/recipes	3.194s
--- FAIL: TestLeasingPutGetDeleteConcurrent (1.97s)
    logger.go:130: 2021-05-18T09:38:54.949Z	INFO	m0	LISTEN GRPC	{"member": "m0", "m.grpcAddr": "localhost:m0", "m.Name": "m0"}
    logger.go:130: 2021-05-18T09:38:54.949Z	INFO	m0	launching a member	{"member": "m0", "name": "m0", "advertise-peer-urls": ["unix://127.0.0.1:2114906229"], "listen-client-urls": ["unix://127.0.0.1:2115006229"], "grpc-address": "unix://localhost:m00"}
    logger.go:130: 2021-05-18T09:38:54.951Z	INFO	m0	opened backend db	{"member": "m0", "path": "/tmp/TestLeasingPutGetDeleteConcurrent37284366/002/etcd293620713/member/snap/db", "took": "1.806604ms"}
    logger.go:130: 2021-05-18T09:38:54.953Z	INFO	m0	starting local member	{"member": "m0", "local-member-id": "59d1bca8eb583e4e", "cluster-id": "41f4a03a38ee4cca"}
    logger.go:130: 2021-05-18T09:38:54.953Z	INFO	m0.raft	59d1bca8eb583e4e switched to configuration voters=()	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.953Z	INFO	m0.raft	59d1bca8eb583e4e became follower at term 0	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.953Z	INFO	m0.raft	newRaft 59d1bca8eb583e4e [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.953Z	INFO	m0.raft	59d1bca8eb583e4e became follower at term 1	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.953Z	INFO	m0.raft	59d1bca8eb583e4e switched to configuration voters=(6472161573197069902)	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.954Z	WARN	m0	simple token is not cryptographically signed	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.955Z	INFO	m0	kvstore restored	{"member": "m0", "current-rev": 1}
    logger.go:130: 2021-05-18T09:38:54.955Z	INFO	m0	starting etcd server	{"member": "m0", "local-member-id": "59d1bca8eb583e4e", "local-server-version": "3.5.0-alpha.0", "cluster-version": "to_be_decided"}
    logger.go:130: 2021-05-18T09:38:54.956Z	INFO	m0	started as single-node; fast-forwarding election ticks	{"member": "m0", "local-member-id": "59d1bca8eb583e4e", "forward-ticks": 9, "forward-duration": "90ms", "election-ticks": 10, "election-timeout": "100ms"}
    logger.go:130: 2021-05-18T09:38:54.956Z	INFO	m0.raft	59d1bca8eb583e4e switched to configuration voters=(6472161573197069902)	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.956Z	INFO	m0	added member	{"member": "m0", "cluster-id": "41f4a03a38ee4cca", "local-member-id": "59d1bca8eb583e4e", "added-peer-id": "59d1bca8eb583e4e", "added-peer-peer-urls": ["unix://127.0.0.1:2114906229"]}
    logger.go:130: 2021-05-18T09:38:54.956Z	INFO	m0	launched a member	{"member": "m0", "name": "m0", "advertise-peer-urls": ["unix://127.0.0.1:2114906229"], "listen-client-urls": ["unix://127.0.0.1:2115006229"], "grpc-address": "unix://localhost:m00"}
    logger.go:130: 2021-05-18T09:38:54.963Z	INFO	m0.raft	59d1bca8eb583e4e is starting a new election at term 1	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.963Z	INFO	m0.raft	59d1bca8eb583e4e became candidate at term 2	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.963Z	INFO	m0.raft	59d1bca8eb583e4e received MsgVoteResp from 59d1bca8eb583e4e at term 2	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.963Z	INFO	m0.raft	59d1bca8eb583e4e became leader at term 2	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.963Z	INFO	m0.raft	raft.node: 59d1bca8eb583e4e elected leader 59d1bca8eb583e4e at term 2	{"member": "m0"}
    logger.go:130: 2021-05-18T09:38:54.963Z	INFO	m0	setting up initial cluster version using v2 API	{"member": "m0", "cluster-version": "3.5"}
    logger.go:130: 2021-05-18T09:38:54.964Z	INFO	m0	set initial cluster version	{"member": "m0", "cluster-id": "41f4a03a38ee4cca", "local-member-id": "59d1bca8eb583e4e", "cluster-version": "3.5"}
    logger.go:130: 2021-05-18T09:38:54.964Z	INFO	m0	cluster version is updated	{"member": "m0", "cluster-version": "3.5"}
    logger.go:130: 2021-05-18T09:38:54.964Z	INFO	m0	published local member to cluster through raft	{"member": "m0", "local-member-id": "59d1bca8eb583e4e", "local-member-attributes": "{Name:m0 ClientURLs:[unix://127.0.0.1:2115006229]}", "request-path": "/0/members/59d1bca8eb583e4e/attributes", "cluster-id": "41f4a03a38ee4cca", "publish-timeout": "5.2s"}
    cluster.go:252:  - m0 -> 59d1bca8eb583e4e (unix://localhost:m00)
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[core] parsed scheme: "etcd-endpoints"]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[core] ccResolverWrapper: sending update to cc: {[{unix:localhost:m00 localhost <nil> 0 <nil>}] 0xc000750e40 <nil>}]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[core] ClientConn switching balancer to "round_robin"]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[core] Channel switches to new LB policy "round_robin"]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[balancer] base.baseBalancer: got new ClientConn state:  {{[{unix:localhost:m00 localhost <nil> 0 <nil>}] 0xc000750e40 <nil>} <nil>}]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[core] Subchannel Connectivity change to CONNECTING]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[core] Subchannel picks a new address "unix:localhost:m00" to connect]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[balancer] base.baseBalancer: handle SubConn state change: 0xc001a03f30, CONNECTING]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[core] Channel Connectivity change to CONNECTING]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[core] Subchannel Connectivity change to READY]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[balancer] base.baseBalancer: handle SubConn state change: 0xc001a03f30, READY]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[roundrobin] roundrobinPicker: newPicker called with info: {map[0xc001a03f30:{{unix:localhost:m00 localhost <nil> 0 <nil>}}]}]
    logger.go:130: 2021-05-18T09:38:54.968Z	INFO	grpc	[[core] Channel Connectivity change to READY]
    logger.go:130: 2021-05-18T09:38:55.888Z	INFO	grpc	[[core] Channel Connectivity change to SHUTDOWN]
    logger.go:130: 2021-05-18T09:38:55.888Z	INFO	grpc	[[core] Subchannel Connectivity change to SHUTDOWN]
    logger.go:130: 2021-05-18T09:38:55.888Z	INFO	m0	terminating a member	{"member": "m0", "name": "m0", "advertise-peer-urls": ["unix://127.0.0.1:2114906229"], "listen-client-urls": ["unix://127.0.0.1:2115006229"], "grpc-address": "unix://localhost:m00"}
    logger.go:130: 2021-05-18T09:38:55.888Z	INFO	grpc	[[transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"]
    logger.go:130: 2021-05-18T09:38:55.888Z	INFO	grpc	[[transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"]
    logger.go:130: 2021-05-18T09:38:55.889Z	INFO	m0	verification of persisted state	{"member": "m0", "data-dir": "/tmp/TestLeasingPutGetDeleteConcurrent37284366/002/etcd293620713"}
    logger.go:130: 2021-05-18T09:38:55.891Z	INFO	m0	verification: consistentIndex OK	{"member": "m0", "backend-consistent-index": 1302, "hardstate-commit": 1302}
    logger.go:130: 2021-05-18T09:38:55.892Z	INFO	m0	verification of persisted state successful	{"member": "m0", "data-dir": "/tmp/TestLeasingPutGetDeleteConcurrent37284366/002/etcd293620713"}
    logger.go:130: 2021-05-18T09:38:55.892Z	INFO	m0	terminated a member	{"member": "m0", "name": "m0", "advertise-peer-urls": ["unix://127.0.0.1:2114906229"], "listen-client-urls": ["unix://127.0.0.1:2115006229"], "grpc-address": "unix://localhost:m00"}
    leak.go:117: Test appears to have leaked :
        go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*serverWatchStream).recvLoop(0xc0003906c0, 0xc0004065f8, 0x41611d)
        	/home/runner/work/etcd/etcd/server/etcdserver/api/v3rpc/watch.go:327 +0x214
        go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*watchServer).Watch.func2(0xc0003906c0, 0x1212dc0, 0xc001051780, 0xc00334b380)
        	/home/runner/work/etcd/etcd/server/etcdserver/api/v3rpc/watch.go:189 +0x45
        created by go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*watchServer).Watch
        	/home/runner/work/etcd/etcd/server/etcdserver/api/v3rpc/watch.go:188 +0x2ec
FAIL
FAIL	go.etcd.io/etcd/tests/v3/integration/clientv3/lease	46.540s

[ptabor]

Another case:
https://github.com/etcd-io/etcd/pull/13047/checks?check_run_id=2682459069

logger.go:130: 2021-05-27T07:13:22.863Z	INFO	m0	LISTEN GRPC	{"member": "m0", "m.grpcAddr": "localhost:m0", "m.Name": "m0"}
    logger.go:130: 2021-05-27T07:13:22.863Z	INFO	m0	launching a member	{"member": "m0", "name": "m0", "advertise-peer-urls": ["unix://127.0.0.1:2114905995"], "listen-client-urls": ["unix://127.0.0.1:2115005995"], "grpc-address": "unix://localhost:m00"}
    logger.go:130: 2021-05-27T07:13:22.864Z	INFO	m0	opened backend db	{"member": "m0", "path": "/tmp/TestLeasingPutGetDeleteConcurrent4194669131/002/etcd094279543/member/snap/db", "took": "1.083162ms"}
    logger.go:130: 2021-05-27T07:13:22.865Z	INFO	m0	starting local member	{"member": "m0", "local-member-id": "d6e991359ed835ab", "cluster-id": "dffc6f446ac17b82"}
    logger.go:130: 2021-05-27T07:13:22.865Z	INFO	m0.raft	d6e991359ed835ab switched to configuration voters=()	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.865Z	INFO	m0.raft	d6e991359ed835ab became follower at term 0	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.866Z	INFO	m0.raft	newRaft d6e991359ed835ab [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.866Z	INFO	m0.raft	d6e991359ed835ab became follower at term 1	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.866Z	INFO	m0.raft	d6e991359ed835ab switched to configuration voters=(15486068453174425003)	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.866Z	WARN	m0	simple token is not cryptographically signed	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.867Z	INFO	m0	kvstore restored	{"member": "m0", "current-rev": 1}
    logger.go:130: 2021-05-27T07:13:22.867Z	INFO	m0	starting etcd server	{"member": "m0", "local-member-id": "d6e991359ed835ab", "local-server-version": "3.5.0-alpha.0", "cluster-version": "to_be_decided"}
    logger.go:130: 2021-05-27T07:13:22.868Z	INFO	m0	started as single-node; fast-forwarding election ticks	{"member": "m0", "local-member-id": "d6e991359ed835ab", "forward-ticks": 9, "forward-duration": "90ms", "election-ticks": 10, "election-timeout": "100ms"}
    logger.go:130: 2021-05-27T07:13:22.868Z	INFO	m0.raft	d6e991359ed835ab switched to configuration voters=(15486068453174425003)	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.868Z	INFO	m0	added member	{"member": "m0", "cluster-id": "dffc6f446ac17b82", "local-member-id": "d6e991359ed835ab", "added-peer-id": "d6e991359ed835ab", "added-peer-peer-urls": ["unix://127.0.0.1:2114905995"]}
    logger.go:130: 2021-05-27T07:13:22.869Z	INFO	m0	launched a member	{"member": "m0", "name": "m0", "advertise-peer-urls": ["unix://127.0.0.1:2114905995"], "listen-client-urls": ["unix://127.0.0.1:2115005995"], "grpc-address": "unix://localhost:m00"}
    logger.go:130: 2021-05-27T07:13:22.896Z	INFO	m0.raft	d6e991359ed835ab is starting a new election at term 1	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.896Z	INFO	m0.raft	d6e991359ed835ab became candidate at term 2	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.896Z	INFO	m0.raft	d6e991359ed835ab received MsgVoteResp from d6e991359ed835ab at term 2	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.896Z	INFO	m0.raft	d6e991359ed835ab became leader at term 2	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.896Z	INFO	m0.raft	raft.node: d6e991359ed835ab elected leader d6e991359ed835ab at term 2	{"member": "m0"}
    logger.go:130: 2021-05-27T07:13:22.896Z	INFO	m0	setting up initial cluster version using v2 API	{"member": "m0", "cluster-version": "3.5"}
    logger.go:130: 2021-05-27T07:13:22.896Z	INFO	m0	set initial cluster version	{"member": "m0", "cluster-id": "dffc6f446ac17b82", "local-member-id": "d6e991359ed835ab", "cluster-version": "3.5"}
    logger.go:130: 2021-05-27T07:13:22.896Z	INFO	m0	cluster version is updated	{"member": "m0", "cluster-version": "3.5"}
    logger.go:130: 2021-05-27T07:13:22.896Z	INFO	m0	published local member to cluster through raft	{"member": "m0", "local-member-id": "d6e991359ed835ab", "local-member-attributes": "{Name:m0 ClientURLs:[unix://127.0.0.1:2115005995]}", "request-path": "/0/members/d6e991359ed835ab/attributes", "cluster-id": "dffc6f446ac17b82", "publish-timeout": "5.2s"}
    cluster.go:252:  - m0 -> d6e991359ed835ab (unix://localhost:m00)
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[core] parsed scheme: "etcd-endpoints"]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[core] ccResolverWrapper: sending update to cc: {[{unix:localhost:m00 localhost <nil> 0 <nil>}] 0xc00029d580 <nil>}]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[core] ClientConn switching balancer to "round_robin"]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[core] Channel switches to new LB policy "round_robin"]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[balancer] base.baseBalancer: got new ClientConn state:  {{[{unix:localhost:m00 localhost <nil> 0 <nil>}] 0xc00029d580 <nil>} <nil>}]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[core] Subchannel Connectivity change to CONNECTING]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[core] Subchannel picks a new address "unix:localhost:m00" to connect]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[balancer] base.baseBalancer: handle SubConn state change: 0xc00312dec0, CONNECTING]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[core] Channel Connectivity change to CONNECTING]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[core] Subchannel Connectivity change to READY]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[balancer] base.baseBalancer: handle SubConn state change: 0xc00312dec0, READY]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[roundrobin] roundrobinPicker: newPicker called with info: {map[0xc00312dec0:{{unix:localhost:m00 localhost <nil> 0 <nil>}}]}]
    logger.go:130: 2021-05-27T07:13:22.905Z	INFO	grpc	[[core] Channel Connectivity change to READY]
    logger.go:130: 2021-05-27T07:13:23.704Z	INFO	grpc	[[core] Channel Connectivity change to SHUTDOWN]
    logger.go:130: 2021-05-27T07:13:23.704Z	INFO	grpc	[[core] Subchannel Connectivity change to SHUTDOWN]
    logger.go:130: 2021-05-27T07:13:23.704Z	INFO	m0	terminating a member	{"member": "m0", "name": "m0", "advertise-peer-urls": ["unix://127.0.0.1:2114905995"], "listen-client-urls": ["unix://127.0.0.1:2115005995"], "grpc-address": "unix://localhost:m00"}
    logger.go:130: 2021-05-27T07:13:23.704Z	INFO	grpc	[[transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"]
    logger.go:130: 2021-05-27T07:13:23.704Z	INFO	grpc	[[transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing"]
    logger.go:130: 2021-05-27T07:13:23.705Z	INFO	m0	verification of persisted state	{"member": "m0", "data-dir": "/tmp/TestLeasingPutGetDeleteConcurrent4194669131/002/etcd094279543"}
    logger.go:130: 2021-05-27T07:13:23.707Z	INFO	m0	verification: consistentIndex OK	{"member": "m0", "backend-consistent-index": 1316, "hardstate-commit": 1316}
    logger.go:130: 2021-05-27T07:13:23.707Z	INFO	m0	verification of persisted state successful	{"member": "m0", "data-dir": "/tmp/TestLeasingPutGetDeleteConcurrent4194669131/002/etcd094279543"}
    logger.go:130: 2021-05-27T07:13:23.707Z	INFO	m0	terminated a member	{"member": "m0", "name": "m0", "advertise-peer-urls": ["unix://127.0.0.1:2114905995"], "listen-client-urls": ["unix://127.0.0.1:2115005995"], "grpc-address": "unix://localhost:m00"}
    leak.go:117: Test appears to have leaked :
        go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*serverWatchStream).recvLoop(0xc000162cc0, 0x203000, 0x41611d)
        	/home/runner/work/etcd/etcd/server/etcdserver/api/v3rpc/watch.go:327 +0x214
        go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*watchServer).Watch.func2(0xc000162cc0, 0x1236a00, 0xc0032927c0, 0xc000983980)
        	/home/runner/work/etcd/etcd/server/etcdserver/api/v3rpc/watch.go:189 +0x45
        created by go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*watchServer).Watch
        	/home/runner/work/etcd/etcd/server/etcdserver/api/v3rpc/watch.go:188 +0x2ec
FAIL
FAIL	go.etcd.io/etcd/tests/v3/integration/clientv3/lease	45.852s

[lilic]

Seems like I see this locally as well, when running tests in docker:

--- FAIL: TestLeasingPutGetDeleteConcurrent (3.35s)
....
    logger.go:130: 2021-05-31T14:07:02.781Z	INFO	m0	terminated a member	{"member": "m0", "name": "m0", "advertise-peer-urls": ["unix://127.0.0.1:2100104303"], "listen-client-urls": ["unix://127.0.0.1:2100204303"], "grpc-address": "unix://localhost:m00"}
    leak.go:118: Test appears to have leaked :
        go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*serverWatchStream).recvLoop(0xc0007b20c0, 0x18c57d8, 0xc00032e180)
        	/go/src/go.etcd.io/etcd/server/etcdserver/api/v3rpc/watch.go:327 +0x4be
        go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*watchServer).Watch.func2(0xc0007b20c0, 0x18c3f40, 0xc00066e630, 0xc000a1c240)
        	/go/src/go.etcd.io/etcd/server/etcdserver/api/v3rpc/watch.go:189 +0x56
        created by go.etcd.io/etcd/server/v3/etcdserver/api/v3rpc.(*watchServer).Watch
        	/go/src/go.etcd.io/etcd/server/etcdserver/api/v3rpc/watch.go:188 +0x405
FAIL
FAIL	go.etcd.io/etcd/tests/v3/integration/clientv3/lease	3.464s
ok  	go.etcd.io/etcd/tests/v3/integration/clientv3/naming	0.121s [no tests to run]
ok  	go.etcd.io/etcd/tests/v3/integration/clientv3/snapshot	0.083s [no tests to run]

I tried to then rerun a couple of more times but it seems it never failed again. I wonder if we should just continue here instead of returning

etcd/client/pkg/testutil/leak.go

Line 81 in 71448cb

return nil

, otherwise, we will never pick up the goroutines shutting down?

etcd/client/pkg/testutil/leak.go

Lines 90 to 93 in 71448cb

	// Undesired goroutines found, but goroutines might just still be
	// shutting down, so give it some time.
	runtime.Gosched()
	time.Sleep(50 * time.Millisecond)

[ptabor]

I interpret

etcd/client/pkg/testutil/leak.go

Lines 79 to 82 in 71448cb

	goroutines := interestingGoroutines()
	if len(goroutines) == 0 {
	return nil
	}

as the healthy-path, i.e. situation when there are no goroutines we need to wait for, so we exit from leak detection with success. Do I miss something ?

I think there might be 'false negatives', i.e. situation when goroutine is 'RUNNING' (not waiting), so its not reported by list of goroutines, so we might exit leak detection when there is actually a goroutine leaked. But it should not be reported as failure.

[lilic]

Yes after further digging into the code, you are right!

I did notice a TODO, I wonder if that might help at all if we replace it?

TODO: Replace with https://github.com/uber-go/goleak.

[ptabor]

I think goleak is a cleaner/industry standard implementation of what we are doing here.

From my experience, a leaking transports were frequently a consequence of not calling:
"Transport.CloseIdleConnections" ()
that kept open connection being for IdleConnTimeout=90s.
that was too long for a leak detector to wait for

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.