Pagination in a large room with restricted visibility can cause an OOM

[richvdh]

Consider a large room (thousands of members) with thousands of backward extremities.

A client makes a /messages request, which triggers us to consider whether to backfill from a remote server.

This leads to this bit of code, where we look at each of the backwards extremities and decide if they are likely to be visible to any of our users. (Actually, it's worse - we look at each of the successors of the backwards extremities, which can increase the number of events by an order of magnitude.) For each event we consider in this way, we end up pulling the event ids for the entire room state at that event into memory. (We don't actually pull the events themselves, unless they are membership events for our local server).

We can easily end up trying to hold tens of millions of event ids in memory at once for this. It completely blows the *stateGroupMembersCache*, and leads to an OOM after a minute or two.

[richvdh]

note that we do that filtering for every single /messages request - even if we don't need to backfill.

It's also worth noting that this is rather self-exacerbating. Because we can't backfill, we can never "heal" the backwards extremities, so we end up accumulating more and more.

[richvdh]

This seems to date from #4699 (Synapse 0.99.3), which, ironically, contains the review comment:

are we not likely to end up doing the filter_events_for server thing (which is quite expensive) repeatedly, every time anyone asks for some backfill?