Peer and torrent specific statistics

[AnimeSub]

Statistics which will show how much has been transferred, how many people downloaded, how many people are downloading, graph, etc will look awesome

[babico]

I made a simple version, you can check it using this link

to save this history: https://github.com/babico/torrust-tracker-json

[5731la]

Is there a particular roadmap for this? This is a feature of particular interest to me and I'd be willing to put some work into supporting it. Many trackers that I'm aware of rely on this to stop bad actors and provide a reasonable way to sort files, so I think it would be good to have.

[josecelano]

Hi @5731la, I'm currently working on a big refactor of statistics.

I'm trying to build metrics in a way that:

• We can easily add more metadata to the events,
• So we can generate more aggregate data.

For example, if we add the socket address to the event, we can label the metric (Prometheus-style) with the socket address so we can get aggregate data per socket. Same for other labels.

And considering that we can not fulfill all users's requirements, I'm trying to make it easy to build your own by creating event listeners (that requires to use the UDP or HTTP serve packages in your software).

In the middle term we also also planning to build a new GraphQL API to expose the events. So you can also subscribe to update your metrics.

In the short-term (what I'm working right now) I will include more aggregate metrics and other basic torrents and peers metrics that are supposed to be very common.

Here you have one extended proposal. However, that's not going to be the final format. I'm working on another version that allows adding new metrics without introducing breaking changes in the JSON strcuture. The idea is to collect metrics is a format similar to Prometheus. Metrics can be labeled. It's still a proof of concept. I might go back to the previous fixed-aggregate-metrics in JSON with the most demanding groups like counter for IP version, etc.

[5731la]

Hi @5731la, I'm currently working on a big refactor of statistics.

I'm trying to build metrics in a way that:

* We can easily add more metadata to the events,

* So we can generate more aggregate data.

For example, if we add the socket address to the event, we can label the metric (Prometheus-style) with the socket address so we can get aggregate data per socket. Same for other labels.

And considering that we can not fulfill all users's requirements, I'm trying to make it easy to build your own by creating event listeners (that requires to use the UDP or HTTP serve packages in your software).

In the middle term we also also planning to build a new GraphQL API to expose the events. So you can also subscribe to update your metrics.

In the short-term (what I'm working right now) I will include more aggregate metrics and other basic torrents and peers metrics that are supposed to be very common.

Here you have one extended proposal. However, that's not going to be the final format. I'm working on another version that allows adding new metrics without introducing breaking changes in the JSON strcuture. The idea is to collect metrics is a format similar to Prometheus. Metrics can be labeled. It's still a proof of concept. I might go back to the previous fixed-aggregate-metrics in JSON with the most demanding groups like counter for IP version, etc.

Hey, thanks a lot for your reply! Looking through the Rust crates, it seems like Torrust already collects most of this data internally (https://docs.rs/torrust-tracker/latest/torrust_tracker/core/index.html#torrents and the related sections,) is it not just a matter of storing it inside of the DB (presumably under keys and torrents, or maybe instead of keys using a PeerID?) and then providing an API hook to it? Or is the longer-term intent for that to be something a user would add separately?

[josecelano]

Hi @5731la, the tracker core stores a list of peers with this data per peer. Actually, it's the latest announce requests from the peers.

Regarding you question:

1. The tracker should work without a database

There are many different use cases for the tracker. One of them is to run a public tracker not linked to any concrete private Index. In this case, users only want the tracker to be fast. They don't need to collect metrics at all or only basic metrics lie number of requests.

That would be be a problem because the feature could be enabled/disabled by configuration.

2. What type of data do you need?

The data we have in memory is only the current state. We have a list with the latest announce requests data from all peers. You can already get that info per torrent from the REST API:

https://docs.rs/torrust-tracker/3.0.0/torrust_tracker/servers/apis/v1/context/torrent/index.html#get-a-torrent

If you what to import an small subset of torrents you can use that.

There are some problems with this approach:

• You can't use it to import a huge amount of data. If you have hundreds of torrents it would be very slow.
• You can't import only torrents that have changed recently. Anyway all of them should change every two minutes at the most. At least the info about active peers. They should announce a regular interval. And new peers can be added and other can be removed.
• You don't have historical data. You only have the latest state. For example, you don't know how many requests a peer has made in the last 15 minutes.

In order to improve stats I'm working in three different levels:

1. Improve the pre-defined metrics the tracker shared via the REST API by adding more that seem to be very common.
2. Decouple events from stats and allow third-party software to write their listeners. The idea is you can write you own listener like the one we use for the pre-defined metrics. For example, you can use the axum-http-tracker-server to run an HTTP server and link it to your software (likely an Index). All you need to do is to write a listener that listens to all (or some) events and writes into your database.
3. Build a new GraphQL API to expose state changes via subscriptions. For example, you could subscribe to changes to your torrents. So you application would be notify when need peers are added, removed, changed, etc. This would take long. We have to carefully design the events and it might require other features. See Alternatives to the Tracker REST API #1307.

I think the main problem is the tracker handles a lot of requests and data and it's very hard to scale the synchronization process. For example, as far as I know UNIT3D index is building a Rust Tracker for the index. I think the tracker writes directly into the Index database to avoid synchronization. We can't do that because this is a generic tracker. However the custom listener would allow you to easily implement something like that: a listener to listen to tracker events and update your DB accordingly. For my this is the easiest and more flexible solution.

cc @da2ce7