accuracy of the nginx routing rules with the recent changes in auth media and sliding sync?

[wolfspyre]

Heya Tom!

1. HOLY HELL THIS WAS WELL PUT TOGETHER.

2. no seriously.... give yourself some credit... this was really nice.

You do some cool and elegantly trixy shit here.
Question...

how accurate is the nginx routing now that sliding-sync / pseudo-sliding-sync / authenticated media exist?

[tcpipuk]

Hey Wolf! Thanks!

From what I can tell, the ^/_matrix/((client|federation)/[^/]+/)media/ location block should cover the correct endpoints for the Content Repository section of the spec, so this should work correctly - I'd love to hear from anyone that's tested and found otherwise!

For "simplified sliding sync" (or "snake sync" as I like to call it) it looks like it's just another versioned endpoint for /sync - MSC4186 suggests its unstable endpoint is just taking over the previous sliding sync endpoint /_matrix/client/org.matrix.simplified_msc3575/sync and currently we're matching /_matrix/client/(api/v1|r0|v3|unstable)/(sync|events|initialSync|rooms/[\s\S]+/initialSync) so I suspect for safety we should have /_matrix/client/((api/)?[^/]+)/(sync|events|initialSync|rooms/[\s\S]+/initialSync) instead, so it'll capture any unstable versions of the sync endpoints!

Are you using snake sync? Do you want to give that a try and let me know how it looks?

[wolfspyre]

well, okay, admittedly, I'm sorta perverting your ruleset; so it's entirely possible (and quite likely, in fact) that the problem is ME.... That being said It's taken me a bit to get things somewhat sorted
However... ( MY ADAPTATION OF ) this location seems not to be working right:

# Federation rooms
location ~ "^/_matrix/(client|federation)/.*?(?:%21|!)[\s\S]+(?:%3A|:)[A-Za-z0-9.\-]+" {
  set $proxy_pass http://synapse_inbound_room_workers;
  include proxy.conf;
}

I'm using haproxy, and using an haproxy map file to route regex'ed paths to backend pools:

(like I said, this is likely me)

# WolfspyreLabs Synapse HAProxy Map - Thank you  Tom -  https://tcpipuk.github.io/synapse/deployment/nginx.html
#   Client: Main overrides
^/_matrix/client/(api/v1|r0|v3|unstable)/(account/3pid/|directory/list/room/|pushrules/|rooms/[\s\S]+/(forget|upgrade)|login/sso/redirect/|register) Synapse-Main-HTTPS

# Client: OpenID Connect SSO
^(/_matrix/client/(api/v1|r0|v3|unstable)/login/sso/redirect|/_synapse/client/(pick_username|(new_user_consent|oidc/callback|pick_idp|sso_register)$)) Synapse-Main-HTTPS

### FEDERATION ###
# Federation: Main overrides
^/_matrix/federation/v1/openid/userinfo$   Synapse-Main-HTTPS

# Federation rooms
^/_matrix/(client|federation)/.*?(?:%21|!)[\s\S]+(?:%3A|:)[A-Za-z0-9.\-]+ Synapse-Worker-Pool-Rooms
^/_matrix/federation/v[12]/(?:state_ids|get_missing_events)/(?:%21|!)[\s\S]+(?:%3A|:)[A-Za-z0-9.\-]+   Synapse-Worker-Pool-Rooms

# Federation readers
^/_matrix/(federation/(v1|v2)|key/v2)/   Synapse-Worker-Pool-Federation

### CLIENTS ###
# Stream: account_data
^/_matrix/client/(api/v1|r0|v3|unstable)/[\s\S]+(/tags|/account_data)   Synapse-Worker-Pool-Sync

# Stream: presence
^/_matrix/client/(api/v1|r0|v3|unstable)/presence/   Synapse-Worker-Pool-Sync

# Stream: receipts - mod
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/([\s\S]+)/(receipt|read_markers)/(.*)$  Synapse-Worker-Pool-Sync

# Stream: to_device - mod 
^/_matrix/client/(api/v1|r0|v3|unstable)/sendToDevice/(.*)$   Synapse-Worker-Pool-Sync

# Stream: typing - mod
^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/[\s\S]+/typing/(.*)$   Synapse-Worker-Pool-Sync

# Note: The following client blocks must come *after* the stream blocks above otherwise some stream requests would be incorrectly routed

# Client: User directory
^/_matrix/client/(api/v1|r0|v3|unstable)/user_directory/search   Synapse-Worker-Pool-Sync

# Client: Rooms
^/_matrix/client/.*?![\s\S]+:[A-Za-z0-9.\-]+   Synapse-Worker-Pool-Rooms

# Client: Sync
^/_matrix/client/((api/)?[^/]+)/(sync|events|initialSync|rooms/[\s\S]+/initialSync)  Synapse-Worker-Pool-Sync

#^/_matrix/client/(api/v1|r0|v3|unstable)/(sync|events|initialSync|rooms/[\s\S]+/initialSync)$   Synapse-Worker-Pool-Sync

# Client: Reader
^/_matrix/client/(api/v1|r0|v3|unstable)/(room_keys/|keys/(query|changes|claim|upload/|room_keys/)|login|register(/available|/m.login.registration_token/validity|)|password_policy|profile|rooms/[\s\S]+/(joined_members|context/[\s\S]+|members|state|hierarchy|relations/|event/|aliases|timestamp_to_event|redact|send|state/|(join|invite|leave|ban|unban|kick))|createRoom|publicRooms|account/(3pid|whoami|devices)|versions|voip/turnServer|joined_rooms|search|user/[\s\S]+/filter(/|$)|directory/room/[\s\S]+|capabilities)   Synapse-Main-HTTPS

# Media
^/_matrix/((client|federation)/[^/]+/)media/   Synapse-Worker-Pool-Media

# Matrix default location
^/_matrix/   Synapse-Main-HTTPS

so that works out to:

# Federation rooms
^/_matrix/(client|federation)/.*?(?:%21|!)[\s\S]+(?:%3A|:)[A-Za-z0-9.\-]+ Synapse-Worker-Pool-Rooms

Which, strangely, seems to be matching more than it "should":

rooms-1                | 2024-09-22 02:22:21,762 - synapse.http.server - 146 - ERROR - PUT-1102 - Failed handle request via 'RoomTypingRestServlet': <XForwardedForRequest at 0x76a129599c90 method='PUT' uri='/_matrix/client/v3/rooms/!ROOMSTRINGHERE:HOMESERVERDOMAIN/typing/@USERNAMEHERE:HOMESERVERDOMAIN' clientproto='HTTP/1.1' site='haproxy'>
rooms-1                | Traceback (most recent call last):
rooms-1                |   File "/usr/local/lib/python3.11/site-packages/synapse/http/server.py", line 332, in _async_render_wrapper
rooms-1                |     callback_return = await self._async_render(request)
rooms-1                |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
rooms-1                |   File "/usr/local/lib/python3.11/site-packages/synapse/http/server.py", line 544, in _async_render
rooms-1                |     callback_return = await raw_callback_return
rooms-1                |                       ^^^^^^^^^^^^^^^^^^^^^^^^^
rooms-1                |   File "/usr/local/lib/python3.11/site-packages/synapse/rest/client/room.py", line 1244, in on_PUT
rooms-1                |     raise Exception("Got /typing request on instance that is not typing writer")
rooms-1                | Exception: Got /typing request on instance that is not typing writer

I'm guessing that some of your grouping logic (which I've not replicated in haproxy((yet))) does further routing (based on your room-scoped-worker paradigm, which is pretty slick! but I'm tryin to walk before flying :) )

Did you happen to have a list someplace of request URIs that aught match a given rule?

https://spec.matrix.org/v1.11/client-server-api/ isn't really laid out in a fashion which makes it easy to grok what should go where. (at least, not for me)

I ended up finding this gent's notes on haproxy routing:
https://analogpixeldotblog.wordpress.com/2019/07/31/understanding-haproxys-reqrep/

and adapted his script to inspect a mangled map of regexes and emit where it haproxy would send the connection based on the path and the regex that matched.... (I can share wot I have if it'd be useful but it's fugly)

but I figured it mightn't hurt to ask if you had something you were working with as a routing validator

regardless, thanks again for sharing your work! it really helped me.

[moxid]

media uploads are still handled via /_matrix/media/v3/upload - so this currently fails with your location block

https://spec.matrix.org/v1.11/client-server-api/#post_matrixmediav3upload
https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3916-authentication-for-media.md#proposal (note below the table)

i'm currently using a second location block to enable uploads:

        location ~* ^/_matrix/media/v3/upload {
            proxy_pass http://synapse_inbound_media;
            include conf.d/matrix_proxy.conf;
        }

[tcpipuk]

Hopefully that does the job! Let me know if I've missed anything!