Joining rooms over federation can be very slow (SYN-293)

[matrixbot]

Edit: Currently tracking related work in the milestones:

• https://github.com/matrix-org/synapse/milestone/6
• https://github.com/matrix-org/synapse/milestone/8
• https://github.com/matrix-org/synapse/milestone/10

For an overview of the technical changes, see MSC3902.

---

Original Post:

I have Synapse/0.7.1-r2 installed from pip/github.
I am running nginx on a public server in front of it:

• https://matrix.ytnoc.net:443/ -> proxied to http 8009
• https://matrix.ytnoc.net:8448/ -> proxied to https 8448
  I have copied my key and crt in use on the nginx server down to my homeserver install and restarted (had to do this before federation worked).

I can initiate 1:1 chats or join rooms on matrix.org from matrix.ytnoc.net. If I join an active room like #matrix-dev:matrix.org, it takes well over a minute to synchronize. I only had one instance where I saw all the user avatars and the channel name show up. The rest of the time, the channel name on the left is replace my my local id (@​kb3dfz:matrix.ytnoc.net). See screenshot.

#test:matrix.org seems to be working just fine with federation.

I have attached homeserver.log

(Imported from https://matrix.org/jira/browse/SYN-293)

(Reported by John Hogemiller)

Attachments:

https://matrix.org/jira/secure/attachment/10115/icognito-1.png
https://matrix.org/jira/secure/attachment/10118/noalias.png
https://matrix.org/jira/secure/attachment/10116/redacted_homeserver.log
https://matrix.org/jira/secure/attachment/10113/Screen+Shot+2015-02-28+at+11.24.53+AM.png
https://matrix.org/jira/secure/attachment/10117/usernames+only.png

[matrixbot]

Jira watchers: @erikjohnston

[matrixbot]

Once I open an incognito browser (for different cache) I did get more, mainly avatars and images. The channel ID showed up, not the alias.

So I feel there's still some disconnect, but not a big one.

-- John Hogemiller

[matrixbot]

After 24 hours, the channel alias still hasn't shown up, just the ID (as seen in the screenshot). I can click on the cog and see the alias. I just submitting it as is, changing it to be an alias on my server, and listing both: ["#matrix-dev:matrix.org", "#matrix-dev:matrix.ytnoc.net"]. In all cases, it reverts back to the list propogated from matrix.org.

-- John Hogemiller

[matrixbot]

Can a project admin edit the description? I think some of my initial issues (no avatars) were browser cache related, and the key issue remaining is that the channel alias won't show up.

-- John Hogemiller

[matrixbot]

Did you have multiple clients open? Were the ones that did not get the complete state the ones that you didn't use to join the room?

-- @erikjohnston

[matrixbot]

I initially had one client open. It did not get the room member's list/complete state (first screenshot). It showed the room as my username. Later, upon opening an incognito client, I got the member list, and it showed the room id and member list. Refreshing my main client got the room id and member list to show. m.room.alias can be viewed by clicking on the gear, but does not appear on the left channel list.

This finicky behavior seems to be based on room size.
Joining #test:matrix.org worked just fine.
Joining #matrix:matrix.org (today) shows me usernames on the left side (screenshot "usernames only"). Hitting the gear, no m.room.alias appears. I see the same in chrome, chrome incognito, firefox.

-- John Hogemiller

[richvdh]

Having just done a join from a test server to matrix HQ, the join eventually completed with:

2018-04-18 13:55:23,582 - synapse.access.https.8447 - 93 - INFO - POST-726- None - 8447 - {@richvdh:matrixtest.sw1v.org} Processed request: 467006ms (53781ms, 4376ms) (12344977ms/16508ms/4567) 65B 200 "POST /_matrix/client/r0/join/%23matrix%3Amatrix.org? HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"

- that's 12345 seconds (3.5 hours) waiting for db connections. given the whole request completed in "only" 467 seconds, presumably it is doing a bunch of that waiting in parallel.

I suspect we're handling each server separately and in parallel from the point of view of fetching server keys and persisting said keys to the db. We should apply more intelligence here.

[richvdh]

Also related: #3120

[richvdh]

Some things which are related here:

• sometimes we use an out-of-date server for send_joins #2418
• get_keys_from_store should do one big lookup, not hundreds of tiny ones #3818
• store_server_verify_keys shouldn't need to lock the table #3819
• server-key fetching logic is slow and queue-bound #3825
• matrix.org had the wrong value for a signing key #3807
• joining #neb:matrix.org often fails #2956
• presence processing pins the cpu for hours after joining a large room #5248
• Joining a large room timed out, and subsequent attempts ISE #5705