Messages can't be decrypted by receiver (session key missing)

[antlarr]

Description

I use a local home server (not federated) with just two users. I use two clients at different times of the day (latest Android Element from Play Store on an Android 13 phone and element-web, also running locally). The other user only uses the latest Android Element from Play Store too. We mostly talk in a DM room which is encrypted.

Some days ago, I received an encrypted message in my phone that couldn't be decrypted (it showed the error Unable to decrypt: The sender's device has not sent us the keys for this message. ) . I switched to the web client and it couldn't be decrypted either (it showed the error ** Unable to decrypt: decryption key withheld **). I tried requesting the keys from my other sessions (I have no other session than those two, but just in case...) and it didn't help. Since then all messages I received could not be decrypted, but my messages were decrypted correctly by the other user. After a while, suddenly all new messages worked fine again although the "broken" messages could never be decrypted by any of my clients.

Btw, the whole period when messages couldn't be decrypted lasted for around one hour (at least, since we just stopped trying and only several hours later tried again and it's when we noticed we could message correctly again). During that hour, the messages went like:
A: blah blah (unable to decrypt)
B: I can't read you. Please try again. (here I tried to request keys again on both clients I use)
A: blah blah (unable to decrypt)
B: Not yet. Doesn't seem to work (I tried to request keys once more)
A: blah blah (unable to decrypt)

This problem happened a few times in the last couple of weeks on both directions.

Trying to debug a bit more I focused on one time this problem happened, opened the message source and noticed that the session_id is different in the messages that couldn't be decrypted from the previous (working) messages, and then, the session_id also changed for the later messages when decryption suddenly started working again.

I then used sqlite to have a quick look at the homeserver.db file and noticed that for the session used by the sender of the problematic messages, the e2e_room_keys table doesn't contain an entry for my user.

I used select * from e2e_room_keys where room_id='!fDOxxxxxxxxxxxxxxx:myhomeserver' and session_id='rWPxxxxxxxxxxxxxxz+jxxxxxxxxv+Hxxx+/axxxxxx';

That showed only one row with the user_id of the sender, but mine was missing. When looking for the session_id used for the messages I sent during that time (and which could be seen correctly by both of us) select returns two rows as expected, one for each user.

Btw, I'm curious, who generates the room key for my user and the session that belongs to the other user? is that done by my client? or the server?

select session_id, count(*) c from e2e_room_keys group by session_id having c!=2;

gives 13 rows (all with count(*) = 1)

select user_id, e2e_room_keys.session_id from e2e_room_keys, (select session_id, count(*) c from e2e_room_keys group by session_id having c!=2) t where t.session_id=e2e_room_keys.session_id;

gives 8 rows with my user_id and 5 with the other user user_id.

I don't know how to find the last time a session was used to put those 13 times the issue happened in perspective. Btw, the sessions table is empty here.

Steps to reproduce

I'm afraid I didn't find how to reproduce the problem. It happened a few times, always after a period of not sending any message to the room (which I guess is why a new session is started).

Homeserver

unfederated homeserver

Synapse Version

1.73.0

Installation Method

pip (from PyPI)

Database

sqlite

Workers

Single process

Platform

Raspberry Pi with Raspbian (bullseye) running synapse in a virtualenv installed with pip (no vm/container involved).

Configuration

Nothing unusual about the configuration.

Relevant log output

I'm afraid I had backupCount: 3 in the log configuration so I no longer have server logs for the last time this issue happened. I increased it now and will comment adding the logs if/when it happens again. I remember seeing a message like 'key missing'.

Anything else that would be useful to know?

The Android clients some times connect from the local network and some times connect using wireguard (which is running in the same host as synapse) so they can change the connecting IP . I don't think that has anything to do with the issue, but just in case.

[feroom]

I have similar problem
To reproduce, client must have very slow or bad connection to server.
Workaround, both users leave the group and create a new one (

[ankit-pn]

For me its happen when I logout from one device and logged in same account to another device and I don't verify my recovery key ( or security phrase) in that another device . And since message are E2E encrypted i don't they there is any workaround to get the message back without having security key.

[Thatoo]

I have similar problem when a client have very slow or bad connection to server.
Then it comes back later. Sometime, this client need to do a /discardsession command to make it work again.
This issue happen time to time for a few month, I'd say since last September, maybe August but never happened before.

[stratacast]

This just started happening to me too. I installed Element desktop on Linux, it was crap so I signed out of the session and signed into web and did a PWA. Both times I verified my sessions with emojis. I cannot decrypt messages from one user now who I am verified with

[SPiRiT369]

Sorry if I'm flooding this thread. Just wanted to say that I have the same problem with 2 unfederated servers. :(
I kept getting "Unable to decrypt message" on Element Web, then I opened Element Android and new messages started to arrive normally, but older ones are lost.
With the other server I use fluffychat on Android, if that matters.It also has problems.
Also, this has happened to me all the time, but now it's really happening very often.
I'm using latest stable versions of Synapse and Element Web/Android.
Really hope that someone could find what causes it. <3

[loseirdo]

I recently ran into a similar but not quite identical problem. I didn't want to open a new issue for it, so I hope you don't mind me piggybacking on this one.

I also run an unfederated server that I use purely for a family group chat. Earlier today, one user sent a few messages, but his display name and profile picture didn't display properly. Several hours later, he posted again. This time, the display name and pfp were correct, but only users who had posted within that time could decrypt his messages. It wasn't until other users (including myself) posted that his future messages would decrypt properly, although the older ones still wouldn't. This occurred on Element on Linux and Android. It seems that there's a simple way to resolve it (I'm assuming that sending a message prompts Element to update the encryption keys), but I would really like to find a way to prevent it from happening at all, if possible.