Messages can't be decrypted by receiver (session key missing)

[matrixbot]

This issue has been migrated from #14703.

---

Description

I use a local home server (not federated) with just two users. I use two clients at different times of the day (latest Android Element from Play Store on an Android 13 phone and element-web, also running locally). The other user only uses the latest Android Element from Play Store too. We mostly talk in a DM room which is encrypted.

Some days ago, I received an encrypted message in my phone that couldn't be decrypted (it showed the error Unable to decrypt: The sender's device has not sent us the keys for this message. ) . I switched to the web client and it couldn't be decrypted either (it showed the error ** Unable to decrypt: decryption key withheld **). I tried requesting the keys from my other sessions (I have no other session than those two, but just in case...) and it didn't help. Since then all messages I received could not be decrypted, but my messages were decrypted correctly by the other user. After a while, suddenly all new messages worked fine again although the "broken" messages could never be decrypted by any of my clients.

Btw, the whole period when messages couldn't be decrypted lasted for around one hour (at least, since we just stopped trying and only several hours later tried again and it's when we noticed we could message correctly again). During that hour, the messages went like:
A: blah blah (unable to decrypt)
B: I can't read you. Please try again. (here I tried to request keys again on both clients I use)
A: blah blah (unable to decrypt)
B: Not yet. Doesn't seem to work (I tried to request keys once more)
A: blah blah (unable to decrypt)

This problem happened a few times in the last couple of weeks on both directions.

Trying to debug a bit more I focused on one time this problem happened, opened the message source and noticed that the session_id is different in the messages that couldn't be decrypted from the previous (working) messages, and then, the session_id also changed for the later messages when decryption suddenly started working again.

I then used sqlite to have a quick look at the homeserver.db file and noticed that for the session used by the sender of the problematic messages, the e2e_room_keys table doesn't contain an entry for my user.

I used select * from e2e_room_keys where room_id='!fDOxxxxxxxxxxxxxxx:myhomeserver' and session_id='rWPxxxxxxxxxxxxxxz+jxxxxxxxxv+Hxxx+/axxxxxx';

That showed only one row with the user_id of the sender, but mine was missing. When looking for the session_id used for the messages I sent during that time (and which could be seen correctly by both of us) select returns two rows as expected, one for each user.

Btw, I'm curious, who generates the room key for my user and the session that belongs to the other user? is that done by my client? or the server?

select session_id, count(*) c from e2e_room_keys group by session_id having c!=2;

gives 13 rows (all with count(*) = 1)

select user_id, e2e_room_keys.session_id from e2e_room_keys, (select session_id, count(*) c from e2e_room_keys group by session_id having c!=2) t where t.session_id=e2e_room_keys.session_id;

gives 8 rows with my user_id and 5 with the other user user_id.

I don't know how to find the last time a session was used to put those 13 times the issue happened in perspective. Btw, the sessions table is empty here.

Steps to reproduce

I'm afraid I didn't find how to reproduce the problem. It happened a few times, always after a period of not sending any message to the room (which I guess is why a new session is started).

Homeserver

unfederated homeserver

Synapse Version

1.73.0

Installation Method

pip (from PyPI)

Database

sqlite

Workers

Single process

Platform

Raspberry Pi with Raspbian (bullseye) running synapse in a virtualenv installed with pip (no vm/container involved).

Configuration

Nothing unusual about the configuration.

Relevant log output

I'm afraid I had backupCount: 3 in the log configuration so I no longer have server logs for the last time this issue happened. I increased it now and will comment adding the logs if/when it happens again. I remember seeing a message like 'key missing'.

Anything else that would be useful to know?

The Android clients some times connect from the local network and some times connect using wireguard (which is running in the same host as synapse) so they can change the connecting IP . I don't think that has anything to do with the issue, but just in case.