"Unable to decrypt: The sender's device has not sent us the keys for this message." (The UISI bug)

[richvdh]

Update 2023-02-27

This issue is now superceded by element-hq/element-meta#245

Historical information follows.

---

This message (or, less often, the closely related "OLM.UNKNOWN_MESSAGE_INDEX") can be caused by a number of things. This bug serves as a reference to the reasons we know about.

• You logged into your device, or joined the room, after the message was sent. This is sort-of by design, though Hide undecryptable messages which predate your device's participation in a e2e room #2258, Let devices request keys to decrypt past history for a room. element-meta#647 and Let megolm session keys be available to devices added by invited users since the point they are invited #2713 are relevant.
• The sender has blacklisted your device. This is definitely by design, though see Improved communication to recipient that the sender didn't want you to see their message #3831 for improving visibility on this.
• The keys haven't arrived yet. Patience you must have.

Client-specific bugs:

• Olm is prone to race conditions across multiple tabs #2325 [olm is racy when riot-web is used in multiple tabs]
• vector-im/element-web#3309 (also on android/ios) [We can still throw away one-time keys which have messages in-flight]
• We should prioritise established olm sessions over half-open ones when sending outgoing messages #2783 (also on android/ios) [prioritise established olm sessions over half-open ones]
• different devices with the same curve25519 key confuse the olm layer element-meta#1143 (also on android/ios) [different devices with the same curve25519 key confuse the olm layer]
• Stuck in a loop generating and failing to upload one-time keys #4216 / Stuck in a loop generating and failing to upload one-time keys riot-android#1289 / Stuck in a loop generating and failing to upload one-time keys element-ios#1256 [We throw away all our one-time-keys and replace them with new ones]
• Stale device lists when users re-join e2e rooms #4983 / [e2e issue] Decrypt error related to new device creation  matrix-org/matrix-ios-sdk#340 / Stale device lists when users re-join e2e rooms riot-android#1603 [yet another device list sync bug]
• Race in decrypting events causes UISI #5001 (fixed riot-web 0.12.5) [race in decryption]
• Check that device list is updated correctly when encryption is turned on in a room #2672 / Stale device lists #2305 / device list update is *still* racy #3796 / e2e : Stale device lists  element-ios#955 (fixed riot-ios 0.5.0) / E2E: Stale device lists riot-android#863 (fixed riot-android 0.6.10) [racy device list sync]
• Make sure we process to-device messages before generating new one-time-keys #2782 [process incoming messages before uploading new keys]
• We sometimes fail to decrypt events which arrive in the initialsync #2273
• network blips during device key download stop megolm key-sharing working #2562
• [e2e] Corrupted OlmSessions riot-android#799 (fixed, but you may still have broken Olm sessions because of it)
• Out-of-sync olm sessions riot-android#1209 (fixed in riot-android 0.6.10, but you may still have broken Olm sessions because of it)
• We don't appear to be retrying /sendToDevice requests matrix-org/matrix-js-sdk#1866 (fixed for room keys at least)
• Might not be using available session backups for some reason, and we don't re-check them when attempting to fix UISIs Cannot decrypt old message #20461 When requesting keys from other sessions to attempt to resolve decryption errors, also double-check key backups in 4S #21026

Protocol/server things:

• possible to run out of one-time-keys on rarely-used devices #3187
• Users whose servers were unreachable when you logged in will send you undecryptable messages matrix-org/synapse#2165
• if your server goes down, when it comes back you get messages long before the keys #3754
• Restoring a phone (or browser tab) from backup will break Olm if the other device still exists. #3822
• one-time-key upload/claim is racy #3868
• sometimes the sending device doesn't seem to know about some recipients in the room #3846 [sometimes the sending device doesn't seem to know about some users in the room] (hopefully will be fixed by state resets)?
• Failures uploading one-time keys riot-android#1208 (fixed in synapse v0.21.0-rc3)
• sometimes we use an out-of-date server for send_joins matrix-org/synapse#2418 (joins can seed from servers with stale state)
• vector-im/element-web#6989 Device lists can get out of sync

Unexplained:

• to_device message went missing from vdh->matthew with megolm session data #2711

[ara4n]

For anyone reading this looking for a workaround, the best advice currently is that if you suddenly find yourself unable to decrypt messages from someone, ask them to open up your contact details on their client. This should force their Riot to resync its copy of your device list, increasing the chance that the next message sent will actually be encrypted for your current device.

If this doesn't work, you have no choice but create a new room (or worst case, export your session keys, logout, login and import your session keys), until the workaround in #3553 is implemented (or this meta-bug is finally closed up)

[lampholder]

Making forward progress on this requires either @richvdh to hop back into it, or for him to hand over to somebody else.