MSC4186: Simplified Sliding Sync #4186

erikjohnston · 2024-08-30T08:31:04Z

Gnuxie · 2024-08-30T12:18:34Z

proposals/4186-simplified-sliding-sync.md

+      "initial": true|null,
+
+      // Same as in sync v2. Indicates whether there are more events to fetch than those in the timeline.
+      "limited:" true|null,


Are the options in the request and response supposed to be in a union with null?

Isn't it the convention in the spec that the property is omitted entirely rather than given the value null?

(For clarity, this is something that jumps out at me since frequently people don't realise omitting a property and giving the value null are distinct things in JSON. Again, I'm not sure what the convention is for the CS-API though.)

Hmm, seems like limited is already optional in the timeline portion of sync? https://github.com/matrix-org/matrix-spec/blob/7f2f100420a355cf8ea50d08329e6f98989fc975/data/api/client-server/definitions/timeline_batch.yaml#L15-L36

But this does not imply a union with null (it's not explicit). It could if there is another section of the specification that says something to the effect "all unrequired properties in responses and requests should be treated as though their values are in a union with null" somewhere?

It might be better to rewrite this section using JSONSchema so that the intent is clearer in either case.

turt2live · 2024-08-30T15:11:54Z

proposals/4186-simplified-sliding-sync.md

Implementation requirements:

Client (ideally multiple)

Server

Current implementations to my knowledge:

Client implementation: matrix-rust-sdk (PR)

Server implementation: has been implemented in Synapse across 10s of PRs. (Perhaps @MadLittleMods, the implementer, can give better link(s) here.)

Synapse server implementation

Maybe it's easiest to point to the main directories where the code lives:

element-hq/synapse -> synapse/rest/client/sync.py#L782

element-hq/synapse -> synapse/handlers/sliding_sync/

element-hq/synapse -> tests/rest/client/sliding_sync/

The PRs between @MadLittleMods and @erikjohnston with the ~A-Sync label are also a pretty good encapsulation:

https://github.com/element-hq/synapse/pulls?q=+is%3Apr+author%3AMadLittleMods+label%3AA-Sync

https://github.com/element-hq/synapse/pulls?q=+is%3Apr+author%3Aerikjohnston+label%3AA-Sync

Gnuxie · 2024-08-30T15:58:57Z

proposals/4186-simplified-sliding-sync.md

+        // Flag which only returns rooms which have an `m.room.encryption` state event. If unset,
+        // both encrypted and unencrypted rooms are returned. If false, only unencrypted rooms
+        // are returned. If true, only encrypted rooms are returned.
+        "is_invite": true|false|null,


documentation is copied across from is_encrypted

Gnuxie · 2024-08-30T16:25:56Z

proposals/4186-simplified-sliding-sync.md

+The format of returned state in `required_state` is a list of events. This does now allow the server to indicate if a
+"state reset" has happened which removed an entry from the state entirely (rather than it being replaced with another
+event).
+
+This is particularly problematic if the user gets "state reset" out of the room, where the server has no mechanism to
+indicate to the client that the user has effectively left the room (the server has no leave event to return).
+
+We may want to allow special entries in the `required_state` list of the form
+`{"type": .., "state_key": .., content: null}` to indicate that the state entry has been removed.


This is pretty serious and I don't think we can leave it up to clients to figure out somehow. Since that requires client authors to both be extremely knowledgeable about how room state works and also write perfect code. The chances of both of those happening is slim. For example, I tried to write some code to track room state from legacy /sync (by also using /state) in Draupnir and it seems like I haven't even been able to do that right. https://github.com/Gnuxie/matrix-protection-suite/blob/main/src/StateTracking/StateChangeType.ts#L13-L64, https://github.com/Gnuxie/matrix-protection-suite/blob/main/src/StateTracking/StateChangeType.ts#L83-L123. The situation where a state event can be removed entirely via a state reset is not even accounted for. Partially because I was under the impression that this was not something that could happen, as I was a dumbass.

You can read more about the way Draupnir collects state here https://marewolf.me/posts/draupnir/2402.html#development-room-state-tracking. But it would be good if clients didn't need to calculate state deltas at all and there was more specialized API support for that. There'd need to be consideration for how such an API would interact with timeline gaps if it was done alongside /sync.

proposals/4186-simplified-sliding-sync.md

clokep · 2024-09-03T14:13:47Z

proposals/4186-simplified-sliding-sync.md

+
+The current `/sync` endpoint scales badly as the number of rooms on an account increases. It scales badly because all
+rooms are returned to the client, incremental syncs are unbounded and slow down based on how long the user has been
+offline, and clients cannot opt-out of a large amount of extraneous data such as receipts. On large accounts with


Can't data be removed with filters? (See RoomFilter)

Specifically, I believe you can opt-out of receipts with RoomFilter.ephemeral.not_types = [ "m.read.*" ] or similar. When I was working on filtering support in grapevine, we had a fast path for filters like this that would skip reading any of the receipt data from the db server-side. That work hasn't been finished yet, and I have no idea if synapse has a similar fast-path.

One of the difficulties with v3 sync filtering in general is that performance can vary wildly wildly depending on server implementation details. The spec doesn't indicate which fast-paths should exist, and whether or not you hit a fast-path can also affect how many events are returned in the sync window (matrix-org/matrix-spec#1887).

proposals/4186-simplified-sliding-sync.md

clokep · 2024-09-03T14:23:40Z

proposals/4186-simplified-sliding-sync.md

+The core component of a sliding sync request is “lists”, which specify what information to return about which rooms.
+Each list specifies some filters on rooms (e.g. ignore spaces), the range of filtered rooms to select (e.g. the most
+recent 20 filtered rooms), and the config for the data to return for those rooms (e.g. the required state, timeline
+limit, etc). The order of rooms is always done based on when the server received the most recent event for the room.


"Stream order" in synapse parlance?

Yes, should standardize this type of language.

Suggested change

limit, etc). The order of rooms is always done based on when the server received the most recent event for the room.

limit, etc). The order of rooms is always done based on the arrival time of the most recent event for the room on the homeserver.

clokep · 2024-09-03T14:29:58Z

proposals/4186-simplified-sliding-sync.md

+Multiple lists and subscriptions can be specified in a request. If a room matches multiple lists/subscriptions then the
+config is “combined” to be the superset of all configs (e.g. take the maximum timeline limit). See below for the exact
+algorithm.


So lists + subscriptions replace filters?

proposals/4186-simplified-sliding-sync.md

clokep · 2024-09-03T14:53:11Z

proposals/4186-simplified-sliding-sync.md

+rooms. If there are more rooms to send to the client (beyond `page_size`), then the client can request to "paginate" in
+these missed updates in subsequent updates.
+
+Since this would require client side changes, this should be explored in a separate MSC.


Doesn't all of this require client side changes?

clokep · 2024-09-03T14:54:43Z

proposals/4186-simplified-sliding-sync.md

+We could implement a bulk `/messages` endpoint, where the client would specify multiple rooms and `prev_batch` tokens.
+We can also add a flag to disable attempting to backfill over pagination (to match the behaviour of the sync timeline).
+
+## `required_state` response format


This section seems to contradict the opening that this MSC solves the issue of clients having incorrect state.

clokep · 2024-09-03T14:56:21Z

proposals/4186-simplified-sliding-sync.md

+  // The position to use as the `since` token in the next sliding sync request.
+  // c.f. Connections.
+  "pos": "<opaque string>",


I think there was an effort to standardize pagination terminology?

proposals/4186-simplified-sliding-sync.md

MadLittleMods · 2024-09-05T18:05:17Z

proposals/4186-simplified-sliding-sync.md

+      // The "heroes" for the room, if there is no room name. Only sent initially and when it changes.
+      "heroes": [
+        {"user_id":"@alice:example.com","displayname":"Alice","avatar_url":"mxc://..."},
+      ],


Do the heroes membership state events need to be included in required_state response when requesting required_state: ["m.room.member", "$LAZY"] (lazy-loading room members)?

Sync v2 says this:

When lazy-loading room members is enabled, the membership events for the heroes MUST be included in the state, unless they are redundant. When the list of users changes, the server notifies the client by sending a fresh list of heroes. If there are no changes since the last sync, this field may be omitted.

But I think that's because m.heroes in Sync v2 is only a list of user ID's. Whereas, in the sliding sync response here, we already have all of the info necessary in these stripped events.

One alternative is to match Sync v2 and only list the user ID's in heroes and include the membership events in required_state.

proposals/4186-simplified-sliding-sync.md

MadLittleMods · 2024-09-05T18:21:43Z

proposals/4186-simplified-sliding-sync.md

+      // The list of events, sorted least to most recent.
+      "timeline": [ ... ],
+      // The current state of the room as a list of events
+      "required_state": [ ... ],


We should call out that the required_state, timeline, limited, num_live, prev_batch fields will be omitted for invite/knock rooms. Only invite_state/knock_state is provided.

The reason these can't be included is because we don't have any of that information for remote invites and the user isn't participating in the room yet so we shouldn't leak anything to them. We can only rely on the information in the invite event.

Related previous conversation: #3575 (comment)

MadLittleMods · 2024-09-05T18:31:27Z

proposals/4186-simplified-sliding-sync.md

+      "heroes": [
+        {"user_id":"@alice:example.com","displayname":"Alice","avatar_url":"mxc://..."},
+      ],


One weird corner with the current spec: If the room doesn't have a name set and the user is invited/knock to the room, we don't have any heroes to provide here. This is because heroes membership isn't included in the stripped state that the server receives when they are invited/knocked over federation.

Related to matrix-org/matrix-spec#380

This was solved for partial-joins via MSC3943

Not necessarily a blocker but just calling out something that won't work completely until that spec issue is solved.

@kfiven

# Synapse 1.114.0 (2024-09-02) This release enables support for [MSC4186](matrix-org/matrix-spec-proposals#4186) — Simplified Sliding Sync. This allows using the upcoming releases of the Element X mobile apps without having to run a Sliding Sync Proxy. ### Features - Enable native sliding sync support ([MSC3575](matrix-org/matrix-spec-proposals#3575) and [MSC4186](matrix-org/matrix-spec-proposals#4186)) by default. ([\#17648](element-hq/synapse#17648)) # Synapse 1.114.0rc3 (2024-08-30) ### Bugfixes - Fix regression in v1.114.0rc2 that caused workers to fail to start. ([\#17626](element-hq/synapse#17626)) # Synapse 1.114.0rc2 (2024-08-30) ### Features - Improve cross-signing upload when using [MSC3861](matrix-org/matrix-spec-proposals#3861) to use a custom UIA flow stage, with web fallback support. ([\#17509](element-hq/synapse#17509)) - Make `hash_password` script accept password input from stdin. ([\#17608](element-hq/synapse#17608)) ### Bugfixes - Fix hierarchy returning 403 when room is accessible through federation. Contributed by Krishan (@kfiven). ([\#17194](element-hq/synapse#17194)) - Fix content-length on federation `/thumbnail` responses. ([\#17532](element-hq/synapse#17532)) - Fix authenticated media responses using a wrong limit when following redirects over federation. ([\#17543](element-hq/synapse#17543)) ### Internal Changes - MSC3861: load the issuer and account management URLs from OIDC discovery. ([\#17407](element-hq/synapse#17407)) - Refactor sliding sync class into multiple files. ([\#17595](element-hq/synapse#17595)) - Store sliding sync per-connection state in the database. ([\#17599](element-hq/synapse#17599)) - Make the sliding sync `PerConnectionState` class immutable. ([\#17600](element-hq/synapse#17600)) - Add support to `@tag_args` for standalone functions. ([\#17604](element-hq/synapse#17604)) - Speed up incremental syncs in sliding sync by adding some more caching. ([\#17606](element-hq/synapse#17606)) - Always return the user's own read receipts in sliding sync. ([\#17617](element-hq/synapse#17617)) - Replace `isort` and `black` with `ruff`. ([\#17620](element-hq/synapse#17620)) - Refactor sliding sync code to move room list logic out into a separate class. ([\#17622](element-hq/synapse#17622)) ### Updates to locked dependencies * Bump attrs from 23.2.0 to 24.2.0. ([\#17609](element-hq/synapse#17609)) * Bump cryptography from 42.0.8 to 43.0.0. ([\#17584](element-hq/synapse#17584)) * Bump phonenumbers from 8.13.43 to 8.13.44. ([\#17610](element-hq/synapse#17610)) * Bump pygithub from 2.3.0 to 2.4.0. ([\#17612](element-hq/synapse#17612)) * Bump pyyaml from 6.0.1 to 6.0.2. ([\#17611](element-hq/synapse#17611)) * Bump sentry-sdk from 2.12.0 to 2.13.0. ([\#17585](element-hq/synapse#17585)) * Bump serde from 1.0.206 to 1.0.208. ([\#17581](element-hq/synapse#17581)) * Bump serde from 1.0.208 to 1.0.209. ([\#17613](element-hq/synapse#17613)) * Bump serde_json from 1.0.124 to 1.0.125. ([\#17582](element-hq/synapse#17582)) * Bump serde_json from 1.0.125 to 1.0.127. ([\#17614](element-hq/synapse#17614)) * Bump types-jsonschema from 4.23.0.20240712 to 4.23.0.20240813. ([\#17583](element-hq/synapse#17583)) * Bump types-setuptools from 71.1.0.20240726 to 71.1.0.20240818. ([\#17586](element-hq/synapse#17586)) # Synapse 1.114.0rc1 (2024-08-20) ### Features - Add a flag to `/versions`, `org.matrix.simplified_msc3575`, to indicate whether experimental sliding sync support has been enabled. ([\#17571](element-hq/synapse#17571)) - Handle changes in `timeline_limit` in experimental sliding sync. ([\#17579](element-hq/synapse#17579)) - Correctly track read receipts that should be sent down in experimental sliding sync. ([\#17575](element-hq/synapse#17575), [\#17589](element-hq/synapse#17589), [\#17592](element-hq/synapse#17592)) ### Bugfixes - Start handlers for new media endpoints when media resource configured. ([\#17483](element-hq/synapse#17483)) - Fix timeline ordering (using `stream_ordering` instead of topological ordering) in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17510](element-hq/synapse#17510)) - Fix experimental sliding sync implementation to remember any updates in rooms that were not sent down immediately. ([\#17535](element-hq/synapse#17535)) - Better exclude partially stated rooms if we must await full state in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17538](element-hq/synapse#17538)) - Handle lower-case http headers in `_Mulitpart_Parser_Protocol`. ([\#17545](element-hq/synapse#17545)) - Fix fetching federation signing keys from servers that omit `old_verify_keys`. Contributed by @tulir @ Beeper. ([\#17568](element-hq/synapse#17568)) - Fix bug where we would respond with an error when a remote server asked for media that had a length of 0, using the new multipart federation media endpoint. ([\#17570](element-hq/synapse#17570)) ### Improved Documentation - Clarify default behaviour of the [`auto_accept_invites.worker_to_run_on`](https://element-hq.github.io/synapse/develop/usage/configuration/config_documentation.html#auto-accept-invites) option. ([\#17515](element-hq/synapse#17515)) - Improve docstrings for profile methods. ([\#17559](element-hq/synapse#17559)) ### Internal Changes - Add more tracing to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17514](element-hq/synapse#17514)) - Fixup comment in sliding sync implementation. ([\#17531](element-hq/synapse#17531)) - Replace override of deprecated method `HTTPAdapter.get_connection` with `get_connection_with_tls_context`. ([\#17536](element-hq/synapse#17536)) - Fix performance of device lists in `/key/changes` and sliding sync. ([\#17537](element-hq/synapse#17537), [\#17548](element-hq/synapse#17548)) - Bump setuptools from 67.6.0 to 72.1.0. ([\#17542](element-hq/synapse#17542)) - Add a utility function for generating random event IDs. ([\#17557](element-hq/synapse#17557)) - Speed up responding to media requests. ([\#17558](element-hq/synapse#17558), [\#17561](element-hq/synapse#17561), [\#17564](element-hq/synapse#17564), [\#17566](element-hq/synapse#17566), [\#17567](element-hq/synapse#17567), [\#17569](element-hq/synapse#17569)) - Test github token before running release script steps. ([\#17562](element-hq/synapse#17562)) - Reduce log spam of multipart files. ([\#17563](element-hq/synapse#17563)) - Refactor per-connection state in experimental sliding sync handler. ([\#17574](element-hq/synapse#17574)) - Add histogram metrics for sliding sync processing time. ([\#17593](element-hq/synapse#17593)) ### Updates to locked dependencies * Bump bytes from 1.6.1 to 1.7.1. ([\#17526](element-hq/synapse#17526)) * Bump lxml from 5.2.2 to 5.3.0. ([\#17550](element-hq/synapse#17550)) * Bump phonenumbers from 8.13.42 to 8.13.43. ([\#17551](element-hq/synapse#17551)) * Bump regex from 1.10.5 to 1.10.6. ([\#17527](element-hq/synapse#17527)) * Bump sentry-sdk from 2.10.0 to 2.12.0. ([\#17553](element-hq/synapse#17553)) * Bump serde from 1.0.204 to 1.0.206. ([\#17556](element-hq/synapse#17556)) * Bump serde_json from 1.0.122 to 1.0.124. ([\#17555](element-hq/synapse#17555)) * Bump sigstore/cosign-installer from 3.5.0 to 3.6.0. ([\#17549](element-hq/synapse#17549)) * Bump types-pyyaml from 6.0.12.20240311 to 6.0.12.20240808. ([\#17552](element-hq/synapse#17552)) * Bump types-requests from 2.31.0.20240406 to 2.32.0.20240712. ([\#17524](element-hq/synapse#17524)) # Synapse 1.113.0 (2024-08-13) No significant changes since 1.113.0rc1. # Synapse 1.113.0rc1 (2024-08-06) ### Features - Track which rooms have been sent to clients in the experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17447](element-hq/synapse#17447)) - Add Account Data extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17477](element-hq/synapse#17477)) - Add receipts extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17489](element-hq/synapse#17489)) - Add typing notification extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17505](element-hq/synapse#17505)) ### Bugfixes - Update experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint to handle invite/knock rooms when filtering. ([\#17450](element-hq/synapse#17450)) - Fix a bug introduced in v1.110.0 which caused `/keys/query` to return incomplete results, leading to high network activity and CPU usage on Matrix clients. ([\#17499](element-hq/synapse#17499)) ### Improved Documentation - Update the [`allowed_local_3pids`](https://element-hq.github.io/synapse/v1.112/usage/configuration/config_documentation.html#allowed_local_3pids) config option's msisdn address to a working example. ([\#17476](element-hq/synapse#17476)) ### Internal Changes - Change sliding sync to use their own token format in preparation for storing per-connection state. ([\#17452](element-hq/synapse#17452)) - Ensure we don't send down negative `bump_stamp` in experimental sliding sync endpoint. ([\#17478](element-hq/synapse#17478)) - Do not send down empty room entries down experimental sliding sync endpoint. ([\#17479](element-hq/synapse#17479)) - Refactor Sliding Sync tests to better utilize the `SlidingSyncBase`. ([\#17481](element-hq/synapse#17481), [\#17482](element-hq/synapse#17482)) - Add some opentracing tags and logging to the experimental sliding sync implementation. ([\#17501](element-hq/synapse#17501)) - Split and move Sliding Sync tests so we have some more sane test file sizes. ([\#17504](element-hq/synapse#17504)) - Update the `limited` field description in the Sliding Sync response to accurately describe what it actually represents. ([\#17507](element-hq/synapse#17507)) - Easier to understand `timeline` assertions in Sliding Sync tests. ([\#17511](element-hq/synapse#17511)) - Reset the sliding sync connection if we don't recognize the per-connection state position. ([\#17529](element-hq/synapse#17529)) ### Updates to locked dependencies * Bump bcrypt from 4.1.3 to 4.2.0. ([\#17495](element-hq/synapse#17495)) * Bump black from 24.4.2 to 24.8.0. ([\#17522](element-hq/synapse#17522)) * Bump phonenumbers from 8.13.39 to 8.13.42. ([\#17521](element-hq/synapse#17521)) * Bump ruff from 0.5.4 to 0.5.5. ([\#17494](element-hq/synapse#17494)) * Bump serde_json from 1.0.120 to 1.0.121. ([\#17493](element-hq/synapse#17493)) * Bump serde_json from 1.0.121 to 1.0.122. ([\#17525](element-hq/synapse#17525)) * Bump towncrier from 23.11.0 to 24.7.1. ([\#17523](element-hq/synapse#17523)) * Bump types-pyopenssl from 24.1.0.20240425 to 24.1.0.20240722. ([\#17496](element-hq/synapse#17496)) * Bump types-setuptools from 70.1.0.20240627 to 71.1.0.20240726. ([\#17497](element-hq/synapse#17497))

deepbluev7 · 2024-09-19T13:12:16Z

proposals/4186-simplified-sliding-sync.md

+        // Flag which only returns rooms which have an `m.room.encryption` state event. If unset,
+        // both encrypted and unencrypted rooms are returned. If false, only unencrypted rooms
+        // are returned. If true, only encrypted rooms are returned.
+        "is_invite": true|false|null,


How will knocks and left rooms be handled? Maybe this should be a string and be the current membership of the user?

MadLittleMods · 2024-09-19T16:08:04Z

proposals/4186-simplified-sliding-sync.md

+      // Same as in sync v2. Indicates whether there are more events to fetch than those in the timeline.
+      "limited:" true|null,


Clarify what limited means when complicating it with gaps, newly joined, newly joined a room where you can't see the history, etc

Sync v2 in Synapse is a bit crazy with the logic probably because the spec isn't clear.

The existing definition seems accurate "Indicates whether there are more events to fetch than those in the timeline." Could maybe clarify that you can be limited if there is timeline gap and you stop at the gap and say limited: true so the client paginates over the gap themselves. And maybe also need to say that being newly joined doesn't factor in.

MadLittleMods · 2024-09-19T19:34:27Z

proposals/4186-simplified-sliding-sync.md

+## Timeline event trickling
+
+If the `timeline_limit` is increased then the server will send down historic data (c.f. "Room config changes"), which
+allows the clients to easily preload more history in recent rooms.
+
+This mechanism is fiddly to implement, and ends up resending down events that we have previously sent to the client.
+
+A simpler alternative is to use `/messages` to fetch the history. This has two main problems: 1) clients generally want
+to preload history for multiple rooms at once, and 2) `/messages` can be slow if it tries to backfill over federation.
+
+We could implement a bulk `/messages` endpoint, where the client would specify multiple rooms and `prev_batch` tokens.
+We can also add a flag to disable attempting to backfill over pagination (to match the behaviour of the sync timeline).


Please kill unstable_expanded_timeline with one of these alternatives. It was implemented in Simplified Sliding Sync solely to match what the Sliding Sync proxy did. And the proxy behavior was a bug not a feature (self-described)

Previous conversation:

Sliding Sync: Handle room_subscriptions that increase the timeline_limit element-hq/synapse#17503 (comment)

Sliding Sync: Handle timeline limit changes (take 2) element-hq/synapse#17579 (comment)

MadLittleMods · 2024-09-19T19:51:04Z

proposals/4186-simplified-sliding-sync.md

+  // Aggregated rooms from lists and room subscriptions. There will be one entry per room, even if
+  // the room appears in multiple lists and/or room subscriptions.
+  "rooms": {


Clarify which rooms should be showing up.

We want to return every room where the user has membership except rooms that the user has left themselves; unless "newly left" (left since the last incremental sync pos token) because we want everything that's still relevant to the user. We include "newly left" rooms because the last event that the user should see is their own leave event.

A leave != kick (leave events where the sender is not the same user). Kicks should always be included.

To say it another way, here is a list:

join

invite

knock

ban

Kicks (leave where the sender doesn't match the state_key)

Newly left rooms (leave membership that happened since the last incremental sync pos token)

Should you only include ban if there is a previous join, invite, or knock? :) (Maybe scope creep.)

I also had that in mind but I'm not sure (could go either way). It's kinda the same as an unsolicited invite (optionally followed by a kick/ban).

Ideally, we probably wouldn't show those kind of ban

MadLittleMods · 2024-09-19T20:56:31Z

proposals/4186-simplified-sliding-sync.md

+  "room_subscriptions": {
+    // The key is the room to subscribe to.
+    "!foo:example.com": {


Re-adding this discussion from the other MSC3575:

What should happen if you try to subscribe to a room you are not part of (or not allowed)?

The easy option would be to ignore it but then clients have no feedback about what they did wrong.

Perhaps entries in the rooms map could also have an errcode/error field. We could also just blow up the whole request with an error and point out which room_id is wrong although that isn't very machine-readable.

Can you subscribe to a public world_readable room that you're not part of?

Hywan · 2024-09-24T08:12:45Z

proposals/4186-simplified-sliding-sync.md

+      // Whether the room is a DM room.
+      "is_dm": true|null,
+
+      // An opaque integer that can be used to sort the rooms by "Bump Stamp"


Suggested change

// An opaque integer that can be used to sort the rooms by "Bump Stamp"

// An opaque unsigned 64-bits integer that can be used to sort the rooms by "Bump Stamp"

We may also precise that bump_stamp has a total order.

ara4n · 2024-09-24T10:20:57Z

proposals/4186-simplified-sliding-sync.md

+This is particularly problematic if the user gets "state reset" out of the room, where the server has no mechanism to
+indicate to the client that the user has effectively left the room (the server has no leave event to return).
+
+We may want to allow special entries in the `required_state` list of the form


Surely this isn't a nice-to-have but a requirement, otherwise we end up back in matrix-org/matrix-spec#1209

Hywan · 2024-09-25T08:14:36Z

proposals/4186-simplified-sliding-sync.md

+      "bump_stamp": 1,
+
+      // These are the same as sync v2.
+      "joined_count": 1,


Should be precised that joined_count is given only if the room has been joined. For invite room, this value will be missing.

This test was failing since the migration from the sliding sync proxy to Synapse. This patch fixes the test. The failing part was: ```rust assert_eq!(notification.joined_members_count, 1); ``` This patch changes the value from 1 to 0. Indeed, Synapse doesn't share this data for the sake of privacy because the room is not joined. A comment has been made on MSC4186 to precise this behaviour: matrix-org/matrix-spec-proposals#4186 (comment). Moreover, this test was asserting a bug (which is alright), but now a bug report has been made. The patch contains the link to this bug report. The code has been a bit rewritten to make it simpler, and more comments have been added.

Hywan · 2024-09-25T12:30:16Z

proposals/4186-simplified-sliding-sync.md

+      },
+
+      // The maximum number of timeline events to return per response.
+      "timeline_limit": 10,


MSC3575 (at least its implementation inside the sliding sync proxy) allowed timeline_limit to be null. This is NOT the case with MSC4186 (as implemented inside Synapse). I believe the MSC should clarify this value is required.

This test was failing since the migration from the sliding sync proxy to Synapse. This patch fixes the test. The failing part was: ```rust assert_eq!(notification.joined_members_count, 1); ``` This patch changes the value from 1 to 0. Indeed, Synapse doesn't share this data for the sake of privacy because the room is not joined. A comment has been made on MSC4186 to precise this behaviour: matrix-org/matrix-spec-proposals#4186 (comment). Moreover, this test was asserting a bug (which is alright), but now a bug report has been made. The patch contains the link to this bug report. The code has been a bit rewritten to make it simpler, and more comments have been added.

jmartinesp · 2024-10-16T15:03:54Z

I've made a suggestion in Synapse to implement a membership field for Room objects. This would simplify the logic for the clients a lot, avoiding checks and assumptions, and make sure they'll have consistent behaviours since it'll be the server who explicitly sets this value.

jmartinesp · 2024-10-17T06:53:09Z

proposals/4186-simplified-sliding-sync.md

+      // For invites this is the stripped state of the room at the time of invite
+      "invite_state": [ .. ],
+
+      // For knocks this is the stripped state of the room at time of knock
+      "knock_state": [ .. ],


As mentioned in a comment below it would be super useful to have a separate field indicating the user's membership state in a room instead of relying on some rules around required_state and invite_state and finding the membership event in the client-side.

Also note the current simplified sliding sync version implementation in Synapse uses a single invite_state array instead of separate ones for invite and knock. I think it was discussed to rename it to stripped_state or something along the lines?

HarHarLinks · 2024-11-02T14:02:55Z

proposals/4186-simplified-sliding-sync.md

+
+# Unstable prefix
+
+The unstable URL for simplified sliding sync is `/org.matrix.simplified_msc3575/sync`. The flag in `/versions` is


Suggested change

The unstable URL for simplified sliding sync is `/org.matrix.simplified_msc3575/sync`. The flag in `/versions` is

The unstable URL for simplified sliding sync is `/_matrix/client/unstable/org.matrix.simplified_msc3575/sync`. The flag in `/_matrix/client/versions` is

HarHarLinks · 2024-11-02T14:18:45Z

proposals/4186-simplified-sliding-sync.md

+**TODO** explain how these interact with the room lists, this is the same as in
+[MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575)
+
+## Request format


which of these are required and which optional?

MadLittleMods · 2024-11-19T21:09:38Z

proposals/4186-simplified-sliding-sync.md

+      // event _except_ for @alice:example.com, and include every other state event.
+      // In addition, ["*", "*"], ["m.space.child", "*"] is an error, the m.space.child filter is not
+      // required as it would have been returned anyway.
+      "required_state": [ ... ],


Clarify that $LAZY loading room members should also include invite, ban, kick, membership targets (state_key) alongside the senders of any events in the timeline.

This ensures that we send down membership updates whenever we see it change in the timeline. This allows clients to cache the membership list for as long as it doesn't get a gappy sync, but still ensures for large gaps the server doesn't need to send down all membership changes.

Reference: element-hq/synapse#17929

turt2live · 2024-11-28T18:43:24Z

proposals/4186-simplified-sliding-sync.md

+      "invite_state": [ .. ],
+
+      // For knocks this is the stripped state of the room at time of knock
+      "knock_state": [ .. ],


There should also be a knock_servers field, copying #4233

foresto · 2024-11-30T00:46:03Z

proposals/4186-simplified-sliding-sync.md

+      // all state events are returned by default, and additional entries FILTER OUT the returned set
+      // of state events. These additional entries cannot use '*' themselves.
+      // For example, ["*", "*"], ["m.room.member", "@alice:example.com"] will _exclude_ every m.room.member
+      // event _except_ for @alice:example.com, and include every other state event.


I find this confusing.

One unique exception is when you request all state events via ["", ""]. When used,
all state events are returned by default, and additional entries FILTER OUT the returned set
of state events.

Okay, so ["*", "*"] is an include directive, and should yield all the events except those indicated by additional tuples. In other words, additional tuples will be exclude directives modifying this wildcard include. That seems fine so far, but...

For example, ["", ""], ["m.room.member", "@alice:example.com"] will exclude every m.room.member
event except for @alice:example.com,

Here we have an additional tuple. Its first member is an exclude directive modifying the wildcard tuple, just as expected. But its second member is an include directive modifying the first part of the tuple itself? Is that right?

Why overload a single tuple to both modify the preceding wildcard tuple and modify itself? This seems inconsistent to me. Why overload it to be both an exclude and an include directive? I suppose it might be a convenient way to succinctly express commonly wanted filters, but it makes the earlier statement, "additional entries FILTER OUT the returned set of state events to return," something of a contradiction.

If this really is the intent, I think some clarification and elaboration would be helpful. At the very least, rephrasing to remove the apparent contradiction. As it stands, this section leaves me wondering if a mistake was introduced in one of the drafts that led here. I'm not confident implementing a client based on it.

This behavior comes from the prior art in MSC3575.

Proposed alternative in #4186 (comment) which is a lot more straight-forward 👍

I agree with that proposal: Using state and not_state, with the same semantics as used elsewhere in the protocol. It would be easier to understand, and consistent both locally and with other parts of the spec.

(It follows the convention established by room_types / not_room_types in this proposal's room lists, and rooms / not_rooms in Matrix v1.12's RoomFilter object.)

Would it then make sense to merge room_types / not_room_types into state / not_state? The m.room.create event is a state event, after all.

They reference different sub-fields, though: type vs. state_key. So merging them would require some way to indicate which sub-field applies in any given tuple. Perhaps that would be too complex? Or even if not too complex, perhaps having room type filters separate from membership filters is a better fit for common use cases?

I suppose merging them could always be considered in a future protocol version, if real-world usage proves that doing so would be an improvement.

foresto · 2024-11-30T23:58:29Z

It would be nice if room lists had a way to filter by room tags. For example, a client might want only rooms with the m.favourite tag, or to exclude rooms with the m.lowpriority tag. (In fact, I have a non-chat client that wants the latter by default.)

Tags are stored in a room's account data, though. I see that this proposal explicitly relegates account data retrieval to an extension (MSC3959). What about account data as a filtering condition?

I don't know how widely users are tagging their rooms today, but if it's common (or ever becomes so), this would seem a good opportunity to lighten the load on the servers and make clients feel faster. And having the capability here the core protocol, rather than an extension, would make clients more likely to implement it.

olivia-fl · 2024-12-16T23:50:02Z

proposals/4186-simplified-sliding-sync.md

+If new entries are added to  `required_state` then the server must send down matching current state events.
+
+
+## Extensions


My understanding is that this MSC doesn't have any support for timeline filtering. The SS MSC called this out explicitly, and it would probably be good to do that here as well. Timeline filtering on /v3/sync is used heavily by some bot and tool clients.

olivia-fl · 2024-12-16T23:54:14Z

proposals/4186-simplified-sliding-sync.md

+These goals shaped the design of this proposal.
+
+
+# Proposal


How does SSS interact with the ignored user list? In /v3/sync, the server omits events sent by ignored users, requiring the client to perform a second initial sync if they want to retroactively see messages when unignoring. The choice to do server-side ignored user filtering in sync also doesn't really simplify client implementation, because you still need client-side filtering for events received before the user was ignored.

My preference would be to drop this requirement entirely for SSS, now that we have a chance to do it in a backwards-compatible way.

This release contains the security fixes from [v1.120.2](https://github.com/element-hq/synapse/releases/tag/v1.120.2). - Fix release process to not create duplicate releases. ([\#18025](element-hq/synapse#18025)) - Support for [MSC4190](matrix-org/matrix-spec-proposals#4190): device management for Application Services. ([\#17705](element-hq/synapse#17705)) - Update [MSC4186](matrix-org/matrix-spec-proposals#4186) Sliding Sync to include invite, ban, kick, targets when `$LAZY`-loading room members. ([\#17947](element-hq/synapse#17947)) - Use stable `M_USER_LOCKED` error code for locked accounts, as per [Matrix 1.12](https://spec.matrix.org/v1.12/client-server-api/#account-locking). ([\#17965](element-hq/synapse#17965)) - [MSC4076](matrix-org/matrix-spec-proposals#4076): Add `disable_badge_count` to pusher configuration. ([\#17975](element-hq/synapse#17975)) - Fix long-standing bug where read receipts could get overly delayed being sent over federation. ([\#17933](element-hq/synapse#17933)) - Add OIDC example configuration for Forgejo (fork of Gitea). ([\#17872](element-hq/synapse#17872)) - Link to element-docker-demo from contrib/docker*. ([\#17953](element-hq/synapse#17953)) - [MSC4108](matrix-org/matrix-spec-proposals#4108): Add a `Content-Type` header on the `PUT` response to work around a faulty behavior in some caching reverse proxies. ([\#17253](element-hq/synapse#17253)) - Fix incorrect comment in new schema delta. ([\#17936](element-hq/synapse#17936)) - Raise setuptools_rust version cap to 1.10.2. ([\#17944](element-hq/synapse#17944)) - Enable encrypted appservice related experimental features in the complement docker image. ([\#17945](element-hq/synapse#17945)) - Return whether the user is suspended when querying the user account in the Admin API. ([\#17952](element-hq/synapse#17952)) - Fix new scheduled tasks jumping the queue. ([\#17962](element-hq/synapse#17962)) - Bump pyo3 and dependencies to v0.23.2. ([\#17966](element-hq/synapse#17966)) - Update setuptools-rust and fix building abi3 wheels in latest version. ([\#17969](element-hq/synapse#17969)) - Consolidate SSO redirects through `/_matrix/client/v3/login/sso/redirect(/{idpId})`. ([\#17972](element-hq/synapse#17972)) - Fix Docker and Complement config to be able to use `public_baseurl`. ([\#17986](element-hq/synapse#17986)) - Fix building wheels for MacOS which was temporarily disabled in Synapse 1.120.2. ([\#17993](element-hq/synapse#17993)) - Fix release process to not create duplicate releases. ([\#17970](element-hq/synapse#17970), [\#17995](element-hq/synapse#17995)) * Bump bytes from 1.8.0 to 1.9.0. ([\#17982](element-hq/synapse#17982)) * Bump pysaml2 from 7.3.1 to 7.5.0. ([\#17978](element-hq/synapse#17978)) * Bump serde_json from 1.0.132 to 1.0.133. ([\#17939](element-hq/synapse#17939)) * Bump tomli from 2.0.2 to 2.1.0. ([\#17959](element-hq/synapse#17959)) * Bump tomli from 2.1.0 to 2.2.1. ([\#17979](element-hq/synapse#17979)) * Bump tornado from 6.4.1 to 6.4.2. ([\#17955](element-hq/synapse#17955))

Co-authored-by: Johannes Marbach <n0-0ne+github@mailbox.org>

Simplified sliding sync

8cc34df

erikjohnston force-pushed the erikj/sss branch from 806e455 to 8cc34df Compare August 30, 2024 09:25

erikjohnston added 2 commits August 30, 2024 10:52

Add example usage

456db3b

Add security and unstable prefix

6fe4ba7

erikjohnston marked this pull request as ready for review August 30, 2024 09:59

Gnuxie reviewed Aug 30, 2024

View reviewed changes

anoadragon453 added feature Suggestion for a significant extension which needs considerable consideration proposal A matrix spec change proposal kind:core MSC which is critical to the protocol's success labels Aug 30, 2024

uhoreg added the client-server Client-Server API label Aug 30, 2024

turt2live added the needs-implementation This MSC does not have a qualifying implementation for the SCT to review. The MSC cannot enter FCP. label Aug 30, 2024

turt2live reviewed Aug 30, 2024

View reviewed changes

Gnuxie reviewed Aug 30, 2024

View reviewed changes

erebion mentioned this pull request Sep 3, 2024

Sliding Sync does not seem to work element-hq/synapse#17651

Closed

Johennes reviewed Sep 3, 2024

View reviewed changes

proposals/4186-simplified-sliding-sync.md Outdated Show resolved Hide resolved

clokep reviewed Sep 3, 2024

View reviewed changes

MadLittleMods reviewed Sep 5, 2024

View reviewed changes

proposals/4186-simplified-sliding-sync.md Show resolved Hide resolved

MadLittleMods reviewed Sep 5, 2024

View reviewed changes

proposals/4186-simplified-sliding-sync.md Outdated Show resolved Hide resolved

MadLittleMods mentioned this pull request Sep 5, 2024

MSC3575: Sliding Sync (aka Sync v3) #3575

Open

MadLittleMods reviewed Sep 5, 2024

View reviewed changes

This was referenced Sep 9, 2024

fix(sdk): Mark tracked users as dirty when the SS connection is reset. matrix-org/matrix-rust-sdk#3965

Merged

client-api: Implement MSC4186 ruma/ruma#1907

Merged

kfiven mentioned this pull request Sep 14, 2024

Sliding sync cinnyapp/cinny#1436

Open

tcpipuk mentioned this pull request Sep 19, 2024

accuracy of the nginx routing rules with the recent changes in auth media and sliding sync? tcpipuk/tcpipuk.github.io#20

Closed

deepbluev7 reviewed Sep 19, 2024

View reviewed changes

MadLittleMods reviewed Sep 19, 2024

View reviewed changes

Hywan reviewed Sep 24, 2024

View reviewed changes

ara4n reviewed Sep 24, 2024

View reviewed changes

Hywan reviewed Sep 25, 2024

View reviewed changes

Hywan mentioned this pull request Sep 25, 2024

Sliding sync returns initial: false for a first response element-hq/synapse#17754

Closed

Hywan mentioned this pull request Sep 25, 2024

test(integration): Enable test_notification matrix-org/matrix-rust-sdk#4029

Merged

Hywan reviewed Sep 25, 2024

View reviewed changes

gnunicorn mentioned this pull request Oct 6, 2024

Implement Simplified Sliding Sync acterglobal/a3#2225

Open

MadLittleMods mentioned this pull request Oct 16, 2024

Native sliding sync: add membership field to rooms element-hq/synapse#17838

Open

jmartinesp reviewed Oct 17, 2024

View reviewed changes

Gnuxie mentioned this pull request Oct 29, 2024

MSC4222: Adding state_after to sync v2 #4222

Open

HarHarLinks reviewed Nov 2, 2024

View reviewed changes

MadLittleMods reviewed Nov 19, 2024

View reviewed changes

turt2live reviewed Nov 28, 2024

View reviewed changes

foresto reviewed Nov 30, 2024

View reviewed changes

olivia-fl reviewed Dec 16, 2024

View reviewed changes

dbkr mentioned this pull request Dec 19, 2024

[Epic] Simplified Sliding Sync in Element Web element-hq/element-web#28736

Open

10 tasks

S7evinK mentioned this pull request Jan 6, 2025

Embedded / Support sliding sync /syncv3 element-hq/dendrite#3236

Open

erikjohnston and others added 2 commits January 8, 2025 14:40

Update proposals/4186-simplified-sliding-sync.md

a5dc74b

Co-authored-by: Johannes Marbach <n0-0ne+github@mailbox.org>

Some clarifications

4c47844

	limit, etc). The order of rooms is always done based on when the server received the most recent event for the room.
	limit, etc). The order of rooms is always done based on the arrival time of the most recent event for the room on the homeserver.

		// Same as in sync v2. Indicates whether there are more events to fetch than those in the timeline.
		"limited:" true\|null,

	// An opaque integer that can be used to sort the rooms by "Bump Stamp"
	// An opaque unsigned 64-bits integer that can be used to sort the rooms by "Bump Stamp"


		# Unstable prefix

		The unstable URL for simplified sliding sync is `/org.matrix.simplified_msc3575/sync`. The flag in `/versions` is

		If new entries are added to `required_state` then the server must send down matching current state events.


		## Extensions

MSC4186: Simplified Sliding Sync #4186

Are you sure you want to change the base?

MSC4186: Simplified Sliding Sync #4186

Conversation

erikjohnston commented Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Gnuxie Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Synapse server implementation

Choose a reason for hiding this comment

Gnuxie Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Sep 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmartinesp commented Oct 16, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foresto Nov 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

foresto Nov 30, 2024 • edited Loading

Choose a reason for hiding this comment

foresto commented Nov 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikjohnston commented Aug 30, 2024 •

edited

Loading

Gnuxie Aug 30, 2024 •

edited

Loading

Gnuxie Aug 30, 2024 •

edited

Loading

MadLittleMods Sep 5, 2024 •

edited

Loading

MadLittleMods Sep 19, 2024 •

edited

Loading

MadLittleMods Nov 19, 2024 •

edited

Loading

foresto Nov 30, 2024 •

edited

Loading

foresto Nov 30, 2024 •

edited

Loading

foresto commented Nov 30, 2024 •

edited

Loading