go1.23 timers cause issues with libp2p connmgr and more

[lidel]

Version

0.30.0-rc2

Config

default `ipfs init` + custom ports to run A/B test

Description

Bug

0.30.0-rc2 switched to go1.23, and we experience an increased number of connections when running go-libp2p + go-libp2p-kad-dht server, confirmed both by me, and anecdotal evidence from discussion forums.

The problem seem to go away if running with forced go1.22 timer behavior GODEBUG=asynctimerchan=1 (Ref. https://go.dev/wiki/Go123Timer)

How to reproduce

1. Run two instances of 0.30.0-rc2 with a public IP or UPnP, so they act as DHT servers and get a lot of connections from random peers. Run them with different env variables:

• GODEBUG=asynctimerchan=0 to force Go 1.23 timers
• GODEBUG=asynctimerchan=1 to force Go 1.22 timers

2. After a few hours, the one that runs with GODEBUG=asynctimerchan=0 will have significantly more connections (e.g. 6k vs 200)
3. Stopping with ctl-C does not work reliably with GODEBUG=asynctimerchan=0, it sometimes takes a second time to correctly stop the daemon.

Next steps

Short term, I will revert #10486 and switch back to 1.22 for 0.30.0 release and keep it until go-libp2p and go-libp2p-kad-dht are compatible with 1.23.

[Jorropo]

golang/go#69186 ?

[Stebalien]

I don't think we have any cases where we copy timers.

[lidel]

One of potential causes: quic-go/quic-go#4659
Upstream bug: golang/go#69312 / golang/go#69333

We will wait for upstream go to fix it in new patch release.