Overhaul stats: Segregated metrics for each tracker running on a different socket address

[josecelano]

Relates to: #1341 (comment)
Depends on: #1385

UPDATED ON 2025-03-20: added sub-tasks

IMPORTANT!!: global metrics are not going to be removed (at least until the next major version)

While I was reviewing the tracker-core package documentation I've realized we have the tracker_usage_statistics config option in the core configuration:

[core]
tracker_usage_statistics = true

That value enables or disables the statistics.

I've recently moved the statistics out of the tracker core package. Therefore It does not make sense to have that config option there because there is nothing related to statistics tin the tracker-core package now.

That value is used in the app bootstrap:

// HTTP stats
let (http_stats_event_sender, http_stats_repository) =
    http_tracker_core::statistics::setup::factory(configuration.core.tracker_usage_statistics);
let http_stats_event_sender = Arc::new(http_stats_event_sender);
let http_stats_repository = Arc::new(http_stats_repository);

// UDP stats
let (udp_stats_event_sender, udp_stats_repository) =
    udp_tracker_core::statistics::setup::factory(configuration.core.tracker_usage_statistics);
let udp_stats_event_sender = Arc::new(udp_stats_event_sender);
let udp_stats_repository = Arc::new(udp_stats_repository);

The same config option enables stats in both the UDP and HTTP trackers.

The current config omitting irrelevant values:

[core]
tracker_usage_statistics = true

[[udp_trackers]]
bind_address = "0.0.0.0:6969"

  [udp_trackers.cookie_lifetime]
  secs = 120
  nanos = 0

[[http_trackers]]
bind_address = "0.0.0.0:7070"

[http_api]
bind_address = "0.0.0.0:1212"

  [http_api.access_tokens]
  admin = "***"

[health_check_api]
bind_address = "127.0.0.1:1313"

Maybe we should create two new values for UDP and HTTP trackers. However in the current configuration there is not a place to put generic configuration for all HTTP trackers or all UDP trackers.

We can only add a config option per instance.

NOTE: That you can run more than one HTTP and/or UDP trackers.

If we do that the config could be:

[core]
tracker_usage_statistics = true

[[udp_trackers]]
bind_address = "0.0.0.0:6969"
tracker_usage_statistics = true

  [udp_trackers.cookie_lifetime]
  secs = 120
  nanos = 0

[[http_trackers]]
bind_address = "0.0.0.0:7070"
tracker_usage_statistics = true

[http_api]
bind_address = "0.0.0.0:1212"

  [http_api.access_tokens]
  admin = "***"

[health_check_api]
bind_address = "127.0.0.1:1313"

Where you can enable/disable stats for each instance.

cc @da2ce7

Sub-tasks

My implementation plan is more or less like this:

• Overhaul stats: add a configuration option to enable/disable stats per server #1401
• Overhaul stats: add E2E tests for the global metrics #1407
• Overhaul stats: Start collecting stats per server instance (per socket) #1403
  • Expose seggregated metrics in the REST API in JSON format. See this comment.
  • Expose seggregated metrics also in the Prometheus format in the REST API endpoint.

[josecelano]

I've just realized all trackers of the same type use the same events sender and metrics repository. I mean, there is only one stats event sender and stats metrics repository service for all HTTP tracker or UDP tracker. That means counters for are for all trackers. There is no way to get metrics for each socket.

If we want to enable/disable stats for all server at the same time we need to introduce new section in the configuration file. That would be global configuration for all UDP or HTTP trackers. For example:

[core]
#tracker_usage_statistics = true <- remove this

[udp_trackers]
tracker_usage_statistics = true <- global config for all UDP trackers

[http_trackers]
tracker_usage_statistics = true <- global config for all HTTP trackers

[[udp_trackers]]
bind_address = "0.0.0.0:6969"
tracker_usage_statistics = true

  [udp_trackers.cookie_lifetime]
  secs = 120
  nanos = 0

[[http_trackers]]
bind_address = "0.0.0.0:7070"
tracker_usage_statistics = true

[http_api]
bind_address = "0.0.0.0:1212"

  [http_api.access_tokens]
  admin = "***"

[health_check_api]
bind_address = "127.0.0.1:1313"

We need to decide if:

• We want only aggregate statistics per tracker type (HTTP or UDP) or we want also statistics for each socket.
• If we want to enable/disable statistics for each socket.

Assuming we want both (metrics per socket and aggregate metrics per tracker tpye):

1. We can enable/disable stats in the configuration at the single service level:

[[http_trackers]]
bind_address = "0.0.0.0:7070"
tracker_usage_statistics = true

2. We can change repositories to include single service stats.

The current ones would be aggregate metrics. We can get the individualized metrics per sockets by adding new metadata (socket address) to the event. We need to change the event from:

pub enum Event {
    Connect { ip: IpAddr }
    Announce { ip: IpAddr }
    Scrape { ip: IpAddr }
}

to:

pub enum Event {
    Connect { client_ip: IpAddr, server_socket_addr: SocketAddr }
    Announce { client_ip: IpAddr, server_socket_addr: SocketAddr }
    Scrape { client_ip: IpAddr, server_socket_addr: SocketAddr }
}

cc @da2ce7

[josecelano]

In today's meeting @da2ce7 and I agreed on:

• In the long-term only generate metrics and the lowest granularity level. That means we collect metrics for socket-bound services. For example, a single UDP tracker, a single HTTP tracker. Consumers of those metrics in the future have to build their own aggregate data:
  • Metrics for services using the same IP
  • Metrics for services using the same socket
  • Metrics for services using the same IP version
  • Metrics for all UDP servers (protocol)
  • Metrics for all HTTP servers (protocol)
  • Global metrics
• We need a stats repository per socket address.
• You can enable/disable stats for each socket-address service.
• In the short term (the scope of this issue) we can refactor stats/events but continue exposing only the global metrics (for all trackers).
• In the configuration the new flag to enable stats would have a true default value.

[josecelano]

Hi @da2ce7 one question, so far we have been using events only to generate the stats. However after our discussion today I think events might be used for different purposes in the future. For example, to decouple some part of the application. I think we need to:

• Always enable events. You cannot disable them.
• The config option tracker_usage_statistics only enables the listener that listen to the events to collect metrics but it's does not disable sending the events.

I don't thing it's going to be cause performance issues because we always check if we have to send the event and I was even considering implementing an events sender that does nothing (null object pattern).

I think in general we have to decouple the stats terminology from the events. Right now events are only created for stats, but we should split the statistic modules into two modules:

• Events modules: event definition and trigger events, ...
• Statistics modules: event listeners and metrics repositories, ...

What do you thing?

Relates to: #1343

[josecelano]

Since we are going to add the segregated metrics instead of changing the aggregate metrics, we can do the following:

• Rename metrics as I described here following Prometheus conventions.
• Add metrics proposed here.