Key Backup Failing to proceed.

[FSG-Cat]

Steps to reproduce

1. Where are you starting? What can you see?
   The client has migrated to using the Rust Crypto backend and is prompting me to upgrade key backup.
2. What do you click?
   I press the upgrade prompt the client gives me. Then i upload my key and its checked out as good. I press continue and nothing seems to happen in console or on the screen for quite a bit of time after this point.

Outcome

What did you expect?

Some kind of visual feedback or atleast progress indicated in the developer console.

What happened instead?

Nothing that i could see.

Operating system

Windows 10 22H2

Application version

Element Nightly version: 0.0.1-nightly.2024030101 Crypto version: Rust SDK 0.7.0 (b1918e9), Vodozemac 0.5.1

How did you install the app?

https://element.io/download

Homeserver

feline.support

Will you send logs?

Yes

[FSG-Cat]

To add to this it seems like Element is getting confused between passphase entry and key entry after you submit the key as in the dialouge says do you want to cancel submitting your passphrase at one point when logging in and facing this issue.

[BillCarsonFr]

@FSG-Cat Is this the prompt you are seeing #26977
In english Encryption upgrade available ?

[FSG-Cat]

@BillCarsonFr yes it looks like its the same prompt as in that issue. And a similar behavior as is described.

[BillCarsonFr]

@BillCarsonFr Check if need to re-open #26977 as it can still happens

[richvdh]

@BillCarsonFr Check if need to re-open #26977 as it can still happens

Sounds like the prompt is probably expected in this case, so not the same issue?

[richvdh]

Presumably the account has a backup on the server, but one or both of 4S and Cross-Signing is not correctly provisioned.

[richvdh]

I'd argue that this is not a thing we should support at all. If clients have created key backups without setting up 4S, that sounds like the bug to be fixed. The "upgrade your encryption" flow is a relic that should be ripped out.

[richvdh]

(Though I guess we should ask how @FSG-Cat has got this far with an account in that state, given it should have been fixed when EW landed 4S four years ago)

[FSG-Cat]

Does this help clarify the true extent of this shenaniganz? @richvdh I have keybackup configured and worked up until Rust Crypto for Element Web got flipped on.

And ye after i restart the client this pops up.

The exact prompt that caused this whole story to start. And yes if i dont load in my backup key via having it come in via my element iOS session i cant upload the key as of last i tried.

Edit: First screenshot is taken essentially a few seconds before the client restart required to make screenshot 2 happen.

[richvdh]

Does this help clarify the true extent of this shenaniganz?

Well, it suggests there's a problem with your 4S, though what exactly that is is hard to say.

If you don't mind doing some digging, could you open the developer tools in Element (/devtools in the composer bar) and "Explore account data", and:

• Inspect the content of m.secret_storage.default_key. It should look something like this:

  {
    "type": "m.secret_storage.default_key",
    "content": {
      "key": "armrQEFbFstj9boiCAcRPVQgIrb4N8AU"
    }
  }

  The string of characters is not a (cryptographic) key; rather it is the "key ID" of the key for your secret storage.
• Inspect m.secret_storage.key.<key id>. It should look something like:

  {
    "type": "m.secret_storage.key.armrQEFbFstj9boiCAcRPVQgIrb4N8AU",
    "content": {
      "algorithm": "m.secret_storage.v1.aes-hmac-sha2",
      "mac": "1ccEnOd98dGIPLCgm53DxDhLfg5/HSOsZ1N73jWMvDk=",
      "iv": "bG9LEBq6Ylcmr3PQsI8ERw==",
      "signatures": {
        "@richvdh:sw1v.org": {
          "ed25519:93fB6y9SJ3183h6zpYjImjRE8om8X01O7Xjl7UkcHeo": "6widAeIjdf7OoGivTJh9b8zZ32Q0bmdGhvXA4Q08jRB3etvwrIDotZBO1ET/4opWT6rZFZd+rVPWtCriiFm+Cw"
        }
      }
    }
  }

  This stuff is not particularly sensitive: it is cryptographically derived from your 4S key, but there is no way to go back to the actual key from it. Don't worry too much about the content; just check that it is there and has the right shape.
• Inspect each of m.cross_signing.master, m.cross_signing.self_signing, m.cross_signing.user_signing and m.megolm_backup.v1. Each of them should look something like:

  {
    "type": "m.megolm_backup.v1",
    "content": {
      "encrypted": {
        "armrQEFbFstj9boiCAcRPVQgIrb4N8AU": {
          "ciphertext": "vQzyhdNVf59d8/V11ZCmOROKD4JYCygkcqheMVoQgaKv7bie/Dzczn8DT9k=",
          "mac": "UtaRjoIdIJn/+FdWSCMuzeD0JxGep3lWMA9KJwMgt2k=",
          "iv": "J7Ewn4DwG41Ull1QIKh2bw=="
        }
      }
    }
  }

  Critically, there must be an entry within the encrypted section that matches the "key ID" we found earlier.
  (The ciphertext is sensitive, but only if someone also has your 4S key. Which they shouldn't have!)

Thanks! Let me know what you find.

(It's super annoying that we don't have better tooling within the application for investigating this sort of situation.)

[richvdh]

We don't have enough information to fix this, so I'm going to close it for now. If you can give more information, let us know and we can reopen.

[FSG-Cat]

Inspecting the account data you asked me to inspect has everything look like it should except that i dont have a signatures key in the m.secret_storage.key.<key id> field on either of the entries i have. But only one of those 2 entries is the one that shows up under m.megolm_backup.v1

Tho my entry for m.secret_storage.default_key points to a separate key than all the other account data.

[richvdh]

I'm struggling to understand what you're saying. Please paste the actual names and content. (As I said, none of this is security-sensitive)

[FSG-Cat]

{
  "type": "m.secret_storage.default_key",
  "content": {
    "key": "H2VqCrNfNGaPM1JHVoaQGeN9ivyaFcR3"
  }
}

{
  "type": "m.secret_storage.key.H2VqCrNfNGaPM1JHVoaQGeN9ivyaFcR3",
  "content": {
    "iv": "JJ7MYaY+L6QpN9tbgZCYSA==",
    "mac": "ayB5xzc13tdaNFyqe9qgRgrowhzKd8HAtQWjAk8Ebnc=",
    "algorithm": "m.secret_storage.v1.aes-hmac-sha2"
  }
}

{
  "type": "m.secret_storage.key.fr7ngHzKLxmPPOOgX2tQ7eli6WOBeesL",
  "content": {
    "algorithm": "m.secret_storage.v1.aes-hmac-sha2",
    "iv": "1TAw4XANYVwZCLq1qLKwGw==",
    "mac": "h3SA+RWEi90A7o9AoRtEnUg1pZhmqzCgQBtWBXgAzws="
  }
}

{
  "type": "m.megolm_backup.v1",
  "content": {
    "encrypted": {
      "fr7ngHzKLxmPPOOgX2tQ7eli6WOBeesL": {
        "iv": "vj2ZNIuLaLtS+zKZxNFhRA==",
        "ciphertext": "DdTPqS1o/ftsz5bcI296S1lnTBqh3lBs85bXp619mqqrEs2BmMfDXQkwnoU=",
        "mac": "YErujz3IShGTp09B+O3NzaAnEvXmJTcQuI8fS2C2UPQ="
      }
    }
  }
}

Truncated iv,chipertext and mac to starting 2 characters.

{
  "type": "m.cross_signing.master",
  "content": {
    "encrypted": {
      "fr7ngHzKLxmPPOOgX2tQ7eli6WOBeesL": {
        "iv": "Sy",
        "ciphertext": "Ya",
        "mac": "2H"
      }
    }
  }
}

I hope this response is more useful.

[richvdh]

Right, thanks. So the problem is that your master secret-storage key is H2VqCrNfNGaPM1JHVoaQGeN9ivyaFcR3, but neither the backup key nor the private cross-signing key are encrypted with that key. Without them, the existing backup and cross-signing identity are essentially useless.

It seems like, at some point, one of your clients has reset the 4S key (and, for whatever reason, not stored the backup key or cross-signing keys under the new 4S key). It's probably a bit late to figure out how or when this happened, but I guess if you can tell us which clients you have connected to your account that might give us some clues.

The real problem here is that EWR is treating this as an "upgrade" situation. There is no upgrade possible here, since we have lost the keys to the key backup. The only thing to do (unless another client has those secrets and can share them over key sharing) is to reset the key backup and cross-signing. I've opened #27455 to track this.

[FSG-Cat]

I have the original keybackup key for the account and well its Element Web / Element Desktop and Element iOS and a stint with Nheko post problems started. (And a stint on Fluffy 2 years ago or something)

And considering that all the problems started when Element Web switched to Rust with stuff like the key backups suddently failing to be able to load this was probably caused by the Rust transfer especially since my Client thought it wiped its local storage when i was forced onto rust. (Edit: My client thought it wiped its Crypto stores and i relog and then i was on rust suddently. Yes my Elemen Desktop does that quite frequently when my drive is at 80% full so 100G left on a 500G)

Edit 2: I can also fetch from the keybackup so clearly my client somewhere has access to the keys for the backup. (And said backup goes back as far as one would expect if it was the original backup. Aka back to day 1)

[richvdh]

And considering that all the problems started when Element Web switched to Rust with stuff like the key backups suddently failing to be able to load this was probably caused by the Rust transfer especially since my Client thought it wiped its local storage when i was forced onto rust. (Edit: My client thought it wiped its Crypto stores and i relog and then i was on rust suddently.

Holy run-on-sentence, batman. This is incredibly hard to read. Also, there's no t in suddenly.

Anyway, please do file separate issues about your client forgetting its local data. (Technically speaking: I think you probably mean indexeddb rather than localstorage, by the way.) That shouldn't really have caused it to reset 4S though, so that remains a mystery.

Edit 2: I can also fetch from the keybackup so clearly my client somewhere has access to the keys for the backup. (And said backup goes back as far as one would expect if it was the original backup. Aka back to day 1)

I don't really understand. You previously said "stuff like the key backups suddently failing to be able to load". Can your client load key backups or not?

[FSG-Cat]

You cant use the key directly in the session but under the right circumstances you can get the keybackup to load if the key was recived over cross signing from my Element iOS session. But you couldnt even go about attempting to load the key into Element Web Directly.

Edit: When i say forgetting local data i mean the error where the client says that it thinks your E2EE stuff was removed due to low storage and then asks you to clear cache and reload and yes i can file a issue about that next time it happens.

[FSG-Cat]

I should also note that while this might be a separate issue it might be related that Trying to use the Setup Secure backup button here is broken the exact same way as the Secure Backup prompt at login is.

[FSG-Cat]

I should add that i have managed to resolve the SSSS problems and the cross-signing related problems with the help of Nico cooking up some custom script work to help me out. Helps having friends who have worked on E2EE parts of matrix before.

Element stopped asking to "upgrade" as soon as i changed the default key to the one that is tied to my recovery key (that was already used by everything EXCEPT default) and suddently everything is perfectly happy. Well everything i have checked in settings i have yet to relog to try that part as well i dont want to try that until im forced to as relogging is a PITA on my account.

[richvdh]

Glad you got it fixed. Would you be able to clarify exactly what steps you took, for future reference?

[FSG-Cat]

Fixed Default Key value

{
  "type": "m.secret_storage.default_key",
  "content": {
    "key": "fr7ngHzKLxmPPOOgX2tQ7eli6WOBeesL"
  }
}

Previous state of this was

{
  "type": "m.secret_storage.default_key",
  "content": {
    "key": "H2VqCrNfNGaPM1JHVoaQGeN9ivyaFcR3"
  }
}

And i also fixed the Signatures problem for m.secret_storage.key.fr7ngHzKLxmPPOOgX2tQ7eli6WOBeesL

{
  "type": "m.secret_storage.key.fr7ngHzKLxmPPOOgX2tQ7eli6WOBeesL",
  "content": {
    "algorithm": "m.secret_storage.v1.aes-hmac-sha2",
    "iv": "1TAw4XANYVwZCLq1qLKwGw==",
    "mac": "h3SA+RWEi90A7o9AoRtEnUg1pZhmqzCgQBtWBXgAzws=",
    "name": "",
    "signatures": {
      "@cat:feline.support": {
        "ed25519:370jrNjdAIQNvB8mXdSH5wKUmQHcV0AIFxks4KYSjY8": "eH12BBoAff7AwrDqdDema8o6Pj1VTH3e8JO+TK13IfMQZRvdcXmssTcW34DXIiMOrn9dPOinwGGhbTS6ZfVMBg"
      }
    }
  }
}

Also now that i think of it i think i need to tell Nico that we have a rouge field in the key entry. (Stuff works and thats whats the most important.)

Now Element became happy as soon as i fixed the first out of these 2. And the second thing Nico helped me fix with some custom tooling. (I had to build nheko like 10 times as part of this. Fun fact Nheko builds quite quickly on a 8 core VM)

The custom tooling was able to using my Recovery key fetch the stuff from SSSS and sign the event and if it wasnt bugged would also have set Default Key. I did that step manually using Element web before i ran Nicos tool anyways. The bug is that it runs logout before sending the request to set the default key. (That step is optional anyways)

I should also note that i have probably never had a copy of whatever key that H2 key is. I mean my SSSS was never incremented up so clearly that was never moved to a new version so no clue where that H2 key was halucinated into existance from.

[HarHarLinks]

Anyway, please do file separate issues about your client forgetting its local data.

yes i can file a issue about that next time it happens.

#4014
#14174

[FSG-Cat]

@HarHarLinks i cant see how either of these issues are related. I get the splash screen about how Element has lost the E2EE stuff and how its likely due to full disk and clear cache and reload or loggout are the options.

That is NOT triggered by disk being full so its not #4014 and 80% of 500G is 100G free so thats not even "close" to the point you start dropping shit i would imagine.

And #14174 doesnt seem applicable at all because i am not getting a timeline and that issue to me seems like you must have a timeline rendering to be applicable.

[HarHarLinks]

i think usually it's 95% full where the remaining 5% are reserved for root processes so my user element desktop process drops the crypto ball when i run into that threshold. you're right it might not be the same (but you could check if for some reason some quota is configured to 80% in your setup). it might or might not be related at least 🤷 i just wanted to give the hint.

what i don't really understand is why this issue is closed when it was only fixed by manual hacking and the bug would appear to persist in element R (not sure if anyone could reproduce).

[FSG-Cat]

You do make a fair point about if the quota is X and i wouldnt know. All i do know is that its completely unpredictable. Its not consistent what so ever if and when it happens. (Edit: not consistent from my perspective but it could in the end be consistent based on a variable unknown to me)

If you do happen to know where to check the setting for purging i would love to go and check it.

Also as for why its closed it was closed with this comment that was essentially stating not enough information at the time.