Large rooms become unjoinable as send_join response exceeds 100MB

[deepbluev7]

Description

When trying to join a large room, like #_oftc_#oftc:matrix.org or #freenode_#gentoo:matrix.org, the response for send_join can become very large. Responses in synapse are usually limited to 100MB (https://github.com/matrix-org/synapse/blob/develop/synapse/http/matrixfederationclient.py#L91). This makes the join for rooms like oftc fail, since the room state there exceed 100MB. My patched synapse, where I increased the limit, logs the following for example:

2021-05-26 13:31:54,183 - synapse.http.matrixfederationclient - 276 - INFO - POST-62806 - {PUT-O-29440} [matrix.org] Completed request: 200 OK in 61.61 secs, got 106452938 bytes - PUT matrix://matrix.org/_matrix/federation/v2/send_join/%21ZjZUpZpLKxVfAEWyjp%3Amatrix.org/%2416220358520AGPsK%3Aneko.dev

Sometimes you don't even get that far, as the remote server can timeout while sending the response, probably because the state is too large.

(Not that this is partially because send_join returns the full room state without pagination.)

Steps to reproduce

• try to join #_oftc_#oftc:matrix.org (on a beefy homeserver, if you can)
• Watch it fail because of response size limits

Version information

• Homeserver: neko.dev

If not matrix.org:

• Version: 1.35_rc1 (and a few others)

• Install method: custom ebuild

• Platform: Gentoo/Linux, pretty beefy hardware (64GB RAM, 16 cores, Ryzen 3000, NVMe RAID)

[clokep]

Seems this should have been closed by #10093.

[babolivier]

Seems this should have been closed by #10093.

The issue wasn't closed automatically because the commit hasn't landed on the default branch (develop) yet.

[deepbluev7]

Wow, that was quick! Although this may still break on rooms, I guess that is a spec issue then and I haven't found any like that yet. Thank you!

[erikjohnston]

Yeah, its unfortunate that we have APIs that can have unbounded request/response size, so in Synapse we have to do a bit of a balancing act between keeping things working and not allowing toooo many memory exhaustion attacks :)

[reivilibre]

Extremely related: in #15127 we have a room which exceeds the 500 MiB size limit for /state, but I'm pretty sure you'd hit the same issue with a non-FRRJ /send_join (then again, faster joins solves it for /send_join, so I guess we can possibly consider it solved).