Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add downloader hash, flush and complete metrics #11308

Merged
merged 17 commits into from
Jul 25, 2024

Conversation

mh0lt
Copy link
Contributor

@mh0lt mh0lt commented Jul 24, 2024

This adds metrics to indicate the downloaders internal processing for:

Flush - the amount of data currently flushed to disk
Hash - the amount of data currently hashed
Complete - the amount of data currently completed

For logging these are translated to rates in the same way that downloading and uploading are:

 progress="21.15% 261.7GB/1.2TB" time-left=1hrs:48m total-time=22m0s download=516.1MB/s flush=120.6MB/s hash=247.5MB/s complete=153.0MB/s upload=0B/s peers=49 files=527 metadata=455/527 connections=50 alloc=16.0GB sys=18.4GB

This is useful for 2 things:

  1. To get an overall idea about the overall flow
  2. At the end of the download typically the download finished before the other stages, now rather than seemingly sitting idle it is possible to see what the down loader is doing.

If all 4 metrics are 0 then the downloader has stalled.

The torrent lib update also includes a fix which should remove this http.transport panic:

panic: net/http: internal error: connCount underflow

goroutine 32675 [running]:
net/http.(*Transport).decConnsPerHost(0xc003a3a000, {{0x0, 0x0}, {0xc000f1e600, 0x5}, {0xc01a014e70, 0x2f}, 0x0})
        net/http/transport.go:1511 +0x405
net/http.(*persistConn).closeLocked(0xc171c378c0, {0x32163c0, 0x479f9a0})
        net/http/transport.go:2764 +0xe5
net/http.(*persistConn).close(0xc003a3a000?, {0x32163c0?, 0x479f9a0?})
        net/http/transport.go:2754 +0xa8
net/http.(*Transport).putOrCloseIdleConn(0xc003a3a000?, 0xc171c378c0)
        net/http/transport.go:913 +0x2d
net/http.(*Transport).dialConnFor(0xc003a3a000, 0xc05af76420)
        net/http/transport.go:1491 +0x11d
created by net/http.(*Transport).queueForDial in goroutine 30586
        net/http/transport.go:1461 +0x1cd

@AskAlexSharov AskAlexSharov merged commit ab28c04 into main Jul 25, 2024
10 of 11 checks passed
@AskAlexSharov AskAlexSharov deleted the dl_hash_and_complete_metrics branch July 25, 2024 01:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants