Used httpx as the go library. runner.fastdialer not close and rateLimiter not stop

[yhy0]

httpx version:

1.2.9

Current Behavior:

When I used httpx as the go library, I found that there was a memory leak in fastdialer for a period of time. The study found that it was because runner.fastdialer finally did not call close. I think he should call r.fastdialer.Close() in func (r *Runner) Close()

heap profile: 295: 626206736 [195995: 37260945640] @ heap/1048576
139: 583008256 [139: 583008256] @ 0x83ad8d 0x862ba5 0x8630e5 0x853d69 0x8513bc 0x851948 0x851afd 0x8b1bc9 0x938bfc 0xa8b5b8 0x1d2b918 0x1d82618 0x1d831b3 0x1d94594 0x1d93cb3 0x1d978fe 0x1f5fc92 0x470ee1
#	0x83ad8c	github.com/syndtr/goleveldb/leveldb/memdb.New+0xec					/root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/memdb/memdb.go:474
#	0x862ba4	github.com/syndtr/goleveldb/leveldb.(*DB).mpoolGet+0xc4					/root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:90
#	0x8630e4	github.com/syndtr/goleveldb/leveldb.(*DB).newMem+0x364					/root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db_state.go:147
#	0x853d68	github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal+0x188				/root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:613
#	0x8513bb	github.com/syndtr/goleveldb/leveldb.openDB+0x43b					/root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:125
#	0x851947	github.com/syndtr/goleveldb/leveldb.Open+0x187						/root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:197
#	0x851afc	github.com/syndtr/goleveldb/leveldb.OpenFile+0x5c					/root/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.0/leveldb/db.go:219
#	0x8b1bc8	github.com/projectdiscovery/hmap/store/disk.OpenLevelDB+0x68				/root/go/pkg/mod/github.com/projectdiscovery/hmap@v0.0.11/store/disk/leveldb.go:25
#	0x938bfb	github.com/projectdiscovery/hmap/store/hybrid.New+0x33b					/root/go/pkg/mod/github.com/projectdiscovery/hmap@v0.0.11/store/hybrid/hybrid.go:157
#	0xa8b5b7	github.com/projectdiscovery/fastdialer/fastdialer.NewDialer+0x217			/root/go/pkg/mod/github.com/projectdiscovery/fastdialer@v0.0.24/fastdialer/dialer.go:47

This happens only when using httpx as a third-party library call, and this problem occurs after running for a period of time

And rateLimiter didn't stop.

goroutine profile: total 4513
2551 @ 0x43f5d6 0x44f4bc 0x12f03ee 0x470ee1
#	0x12f03ed	github.com/projectdiscovery/ratelimit.(*Limiter).run+0x16d	/root/go/pkg/mod/github.com/projectdiscovery/ratelimit@v0.0.6/ratelimit.go:27

This problem exists not only in HTTPX, but also in the Nuclei.

[anhnmt]

I had the same problem, the problem i had was with the leveldb library. My temporary fix is ​​to fork the httpx library and switch to using memory instead of disk

[Mzack9999]

@anhnmt This should be fixed in the dev branch and was caused due to a new instance of fastdialer that was not closed, which is not removed (ref: #1174). You can test it out with the following:

$ go install -v github.com/projectdiscovery/httpx/cmd/httpx@dev

[anhnmt]

goroutine profile: total 3135
2749 @ 0x439cd6 0x449f7e 0x1371dac 0x46c861
#	0x1371dab	github.com/projectdiscovery/ratelimit.(*Limiter).run+0x16b	/go/pkg/mod/github.com/projectdiscovery/ratelimit@v0.0.8/ratelimit.go:31

I have switched to using github.com/projectdiscovery/httpx/cmd/httpx@dev but still having this problem

[XTeam-Wing]

The problem still exists