Signature upload fails due to missing keys

[dbkr]

Upstream bug: element-hq/element-web#12973

We are seeing signature uploads fail because synapse doesn't have the cross-signing keys for the user in question:

{
  "@davetestissdjlkfghndklsfjngjlk:dbkr.me": {
    "bO6OqlA3AjbjPOxW7kSfx8N5KHnDU9rjlBWiaCYQOlo": {
      "status": 404,
      "errcode": "M_NOT_FOUND",
      "message": "404: No master key found for @davetestissdjlkfghndklsfjngjlk:dbkr.me"
    }
  }
}

One cause of this bug was #7252 which is now fixed but the problem still persists.

A reliable way to repro this seems to be to register fresh accounts on matrix.org and a different server, start a chat between them and then verifying the other user. The matrix.org side will fail with the above error.

[dbkr]

It looks like a race between the matrix.org user doing a /keys/query and the other user being in the same room, because synapse only saves the user's keys to the cache if they share a room with a user on its own server.

As far as I can see this is the only point at which the keys would get saved to the cache: I thought this was supposed to happen via an EDU, but in the case of talking to someone you (and indeed your server) was not previously talking to, I don't see any point at which that would happen.

[anoadragon453]

Some more details from the channel:

anoa: So the problem is Synapse is not saving keys to the cache in all instances, but then when they were requested, and we got a cache miss, wouldn't we retrieve it from the db?

dbkr: yeah - it caches on /keys/query but only if the user shares a room with a user on the HS. when it gets the key to validate an uploaded signature, it expects it to be in the local cache and doesn't go & fetch from the remote server

[anoadragon453]

Testing this can be down through the following:

• ensure you're running riot.im/develop
• make two fresh accounts on matrix.org and another workerless homeserver (example.com)
• example.com: copy full user id
• matrix.org: press direct chat button, paste in user ID, wait, click 'go'
• example.com: accept invite
• matrix.org: expand member panel, click on example.com user, click verify, wait, click 'verify with emoji', wait, click 'they match'
• example.com: also say they match
• example.com: verifies OK
• matrix.org: signature upload error

[anoadragon453]

I just received this error on a non-workerized Synapse instance.

Using workerized workers.example.com and non-workers noworkers.example.com, I did the following:

• Start the direct chat from workers
• Start verification of workers user from my noworkers user
• Workers user accepts verification
• Nonworkers proposes to verify by emoji
• Nonworkers says 'They match'
• Workers says 'They match'
• Workers has no upload error
• Nonworkers has upload error

And I just tried this with two non-workers homeservers and got the same result. So this bug doesn't look to have to do with workers.

In none of these cases did the users on either homeserver share a room prior. All homeservers were running v1.12.3.

[anoadragon453]

This is mitigated in #7289, but ideally we'd still like to get to the bottom of it.