Hard to diagnose poisoning of room message histories by malicious federation

[syldrathecat]

Description

(Edit: Changed some words because apparently the problem is limited to erasing contents, not modifying them)

It is possible for servers to maliciously send redacted event content in response to federated message history / backfill requests.

The way Synapse chooses which server to download messages from seems un-optimal in several ways:

• No weight is given to the servers belonging to users with power levels in the room.
• No weight is given to the server that holds room aliases or the primary room alias.
• The decision of which is the oldest server in the room is actually based on which user has least-recently changed their profile information.
• Only a single server is queried. If that server returns a redacted/erased event, it will not ask any other server for the content.

Additionally there is no record of which server that events were downloaded from, so it is impossible to tell which server sent you poisoned event content, unless INFO level logging was enabled on the federation client.

Because of all this, you can have users on new servers join a room and be confused as to why message content is blanked out, and room admins or even server admins have no clear way to figure out what is going on.

Even worse still, the server which downloaded those poisoned events is now at risk of relaying them to future servers who join, making it hard to determine the original malicious server.

---

One thing the admin of a homeserver can try is running the following query which copies the behavior of get_current_hosts_in_room_ordered:

SELECT substring(c.state_key FROM '@[^:]*:(.*)$') as server_domain
FROM current_state_events c
INNER JOIN events AS e USING (event_id)
WHERE c.type = 'm.room.member'
  AND c.membership = 'join'
  AND c.room_id = '<room_id>'
GROUP BY server_domain
ORDER BY min(e.depth) ASC;

This should return the list of servers from which events will be downloaded from in priority order -- with the first server returned being the server that new servers will be backfilling from, if they are running Synapse.

After verifying that a homeserver is acting maliciously, the users from that server may be kicked (and then optionally re-invited) to a room, to make sure their homeserver is no longer considered the oldest in the room.

---

Finally, its not clear if Synapse is making it easy to erase event content shared via federation in the first place.

My reading of the code seems to imply that a spam_checker module will not cause blanked-out messages to be sent via federation. I am not sure by what mechanism that a server is publishing modified events like this, but if its something that is easy to do on accident, it is maybe a bug itself?

( Adding remote users to erased_users table without their consent might be the mechanism this is achieved by? )

If a server doesn't want to serve an event's content, it should be refusing to provide that event, so that servers can attempt to download them from other servers in the room, rather than handing out illegitimately redacted event contents.

Steps to reproduce

• Create a room between Server A (room creator's server) and Server B
• Server B removes the content of messages from users on Server A in their database
• The room creator changes their display name or profile, causing Server B to now become "the oldest server in the room"
• Server C joins the room for the first time, downloading modified event content from the malicious Server B
• Neither the room creator, nor the user from Server C, can clearly understand why message contents are not what they should be, and diagnosing the issue is difficult even for the admin of Server C

[syldrathecat]

Correction: I have been informed that "malicious modification of messages" may be limited to maliciously withholding the content of events sent out over federation (causing message histories to be riddled with blank messages from users who never requested it).

So there is a problem in that any server can choose to withhold the content of users belonging to other homeservers, and Synapse will never attempt to ask any other servers for the content of that message, simply trusting that it is a legitimately redacted or GDPR-erased event.

[syldrathecat]

Homeserver-wide message retention policies also exist which might cause this in a legitimate configuration -- so I think the issue ultimately is that Synapse will only request event content from one server.

[kegsay]

I would argue the issue is that we aren't (by the sounds of it) verifying the redaction event when we get a backfilled event. If they were genuinely redacted by some synapse module or by the sender / room admin, then it's fair game to return it to the client.

[MadLittleMods]

Related issues:

• Rooms with large numbers of redactions can result in unredacted blank events being served via /messages #5950
• Large quantities of redaction events will cause lots of limited timelines, and breaks messages pagination #18492