in-memory federation transaction transmission queues build up indefinitely for offline servers

[richvdh]

when a server is offline, any events that we would have sent to it stack up in memory indefinitely.

See, for example, this graph for matrix.org:

Apart from the memory usage concerns, this means that if a server goes offline for 1000 events, then when it comes back we will try to send it all those 1000 events in turn, rather than just sending it the most recent ones in a room.

[richvdh]

I suspect this was introduced way back in #2064.

I think we just need to clear the in-memory queues when handling a blacklisted server in _transaction_transmission_loop.

[richvdh]

I'm chucking this into the todo list as it appears an obvious precursor to #2528.

[deepbluev7]

This may be the reason for #7176 since it was reported shortly after #2064 was merged (didn't look at the year). For reference here's my federation_sender gc times graph:

[tulir]

This graph looks like the queue build-up was happening without causing any GC leak:

[reivilibre]

Will land this in a less aggressive way, but will make more aggressive when #2528 is solved because it will be safe to do so.