timeline events can duplicate state events in a room's /sync response (SYN-766)

[matrixbot]

When requesting a full_state /sync with a filter limiting the number of room timeline events to 50, synapse 0.17.1 may send a response where the first timeline event exactly byte-for-byte duplicates one of the state events. The events I've observed this happening with are membership transitions, likely because those are the most common state events.

This contradicts the specification's statement that the state represents "all state up to the start of the timeline"

Two example Joined Room objects from /sync concerning #nix:matrix.org are attached.

(Imported from https://matrix.org/jira/browse/SYN-766)

(Reported by Benjamin Saunders)

Attachments:

https://matrix.org/jira/secure/attachment/10402/badsync2.json
https://matrix.org/jira/secure/attachment/10403/badsync3.json

[matrixbot]

It looks like this is also easy to reproduce on #homeowners:matrix.org

-- Benjamin Saunders

[joepie91]

I've run into this yesterday with a recent Synapse but without full_state set, so it seems that this problem is still occurring.

URL: https://pixie.town/_matrix/client/r0/sync?filter=0&timeout=30000&since=s1883850_10630347_132075_994463_164793_87_2494_121300_24

Filter 0:

{
    "room": {
        "state": {
            "lazy_load_members": true
        }
    }
}

Response:

{
	"account_data": {
		"events": []
	},
	"to_device": {
		"events": []
	},
	"device_lists": {
		"changed": [],
		"left": [
			"@freenode_emacsomancer:pixie.town"
		]
	},
	"presence": {
		"events": []
	},
	"rooms": {
		"join": {
			"!LLYTbNqprwUwxysHCE:pixie.town": {
				"timeline": {
					"events": [
						{
							"type": "m.room.member",
							"sender": "@freenode_emacsomancer:pixie.town",
							"content": {
								"membership": "leave"
							},
							"state_key": "@freenode_emacsomancer:pixie.town",
							"origin_server_ts": 1569181684759,
							"unsigned": {
								"replaces_state": "$fq9aky1Q6q6ASyWCI0ds7xNe2yjxFHNnWqPXcqAli0w",
								"prev_content": {
									"membership": "join",
									"displayname": "emacsomancer",
									"avatar_url": null
								},
								"prev_sender": "@freenode_emacsomancer:pixie.town",
								"age": 2105
							},
							"event_id": "$uTZMApm1Ky2rQNZ-KX5JDjiSCBmL03Bfy1tkXetwGZA"
						}
					],
					"prev_batch": "s1883851_10630347_132075_994463_164793_87_2494_121300_24",
					"limited": false
				},
				"state": {
					"events": [
						{
							"type": "m.room.member",
							"sender": "@freenode_emacsomancer:pixie.town",
							"content": {
								"membership": "leave"
							},
							"state_key": "@freenode_emacsomancer:pixie.town",
							"origin_server_ts": 1569181684759,
							"unsigned": {
								"replaces_state": "$fq9aky1Q6q6ASyWCI0ds7xNe2yjxFHNnWqPXcqAli0w",
								"prev_content": {
									"membership": "join",
									"displayname": "emacsomancer",
									"avatar_url": null
								},
								"prev_sender": "@freenode_emacsomancer:pixie.town",
								"age": 2105
							},
							"event_id": "$uTZMApm1Ky2rQNZ-KX5JDjiSCBmL03Bfy1tkXetwGZA"
						}
					]
				},
				"account_data": {
					"events": []
				},
				"ephemeral": {
					"events": []
				},
				"unread_notifications": {
					"notification_count": 0,
					"highlight_count": 0
				},
				"summary": {
					"m.joined_member_count": 865,
					"m.invited_member_count": 0
				}
			}
		},
		"invite": {},
		"leave": {}
	},
	"groups": {
		"join": {},
		"invite": {},
		"leave": {}
	},
	"device_one_time_keys_count": {
		"signed_curve25519": 50
	},
	"next_batch": "s1883851_10630347_132075_994463_164793_87_2494_121300_24"
}