Overhaul stats: Handle lagging in event listeners channels

[josecelano]

Relates to: #1385

When I run the tracker in the demo server, stats stop working after a while, and I get this error in logs:

tracker  | 2025-04-14T08:43:30.045862Z ERROR torrust_udp_tracker_server::statistics::event::listener: Error receiving udp tracker server event: Lagged(9)

The problem was produced when I switched to Tokio broadcast channels to allow multiple event listeners.

Lagging is not handled in the event listener. For example, the UDP tracker server event listener:

use tokio::sync::broadcast;
use torrust_tracker_clock::clock::Time;

use super::handler::handle_event;
use crate::event::Event;
use crate::statistics::repository::Repository;
use crate::CurrentClock;

pub async fn dispatch_events(mut receiver: broadcast::Receiver<Event>, stats_repository: Repository) {
    loop {
        match receiver.recv().await {
            Ok(event) => handle_event(event, &stats_repository, CurrentClock::now()).await,
            Err(e) => {
                tracing::error!("Error receiving udp tracker server event: {:?}", e);
                break;
            }
        }
    }
}

The channel capacity is 1024, so when unhandled events reach the channel capacity, we receive an error in the loop that breaks the loop, stopping the listener.

Since:

• We only have one listener now to update metrics.
• I want metrics to be exact.

As a temporary solution, I will increase the channel capacity to have the same effect as when we used a mspc channel.

I will also continue the loop when the listener receives that error.

IMPORTANT: The metrics will NOT be exact if there are lagged events.

cc @da2ce7

Alternative solution 1

Restore the mspc channel for metrics.

Pros:

• Precise metrics

Cons:

• Slower request processing? The request has to wait until the metrics are updated.
• We will need two channels: one mpsc channel for metrics and a broadcast channel for generic listeners.

NOTE: The main reason I wanted to have multiple listeners is that I want third-party software to be able to build their own metrics from tracker events (by registering their event listeners).

[josecelano]

Alternative solution 2

Maybe crossbeam unbounded channel?

https://docs.rs/crossbeam/latest/crossbeam/channel/index.html

[josecelano]

The temporary solution is working, at least for the demo tracker. The metrics are working more than 2 hours after the deployment:

And there is no error in logs related to the broadcast channel.

[josecelano]

In a meeting today, @da2ce7 and I decided to leave it as it is. If we see a performance problem, we can switch to the crossbeam channel in the future. We think losing some events in a high load scenario is not a big problem. You would have other bigger problems, like timeouts for the requests, and in general, metrics are used to make sure the tracker is running fine. Even private indexes that have more advanced requirements in terms of tracker metrics would accept some missing requests in the counters when the tracker is running overloaded.

On the other hand, we don't expect more than one listener. The idea of having multiple listeners is just to allow third-party users of our crates to build their own metrics, not to add many listeners.

[josecelano]

In a meeting today, @da2ce7 and I decided to leave it as it is. If we see a performance problem, we can switch to the crossbeam channel in the future. We think losing some events in a high load scenario is not a big problem. You would have other bigger problems, like timeouts for the requests, and in general, metrics are used to make sure the tracker is running fine. Even private indexes that have more advanced requirements in terms of tracker metrics would accept some missing requests in the counters when the tracker is running overloaded.

On the other hand, we don't expect more than one listener. The idea of having multiple listeners is just to allow third-party users of our crates to build their own metrics, not to add many listeners.

Decided not to change it for now.