Remove the v1 API

[matrixbot]

This issue was originally created by @richvdh at matrix-org/sydent#338.

Blocked by matrix-org/synapse#9677

The deprecated v1 identity service API is a source of security issues and poor UX. We should set a date to remove it, and follow through with that.

[matrixbot]

This comment was originally posted by @richvdh at matrix-org/sydent#338 (comment).

Some notable places this is still used:

• Synapse (see Switch to Identity Service API v2 matrix-org/synapse#9677)
• Element-web (for .well-known validation or something - see MSC2499, Identity server discovery pokes v1 API, which is planned for removal matrix-org/matrix-spec#557)
• we see a lot of requests from mxisd user-agents. mxisd is unmaintained aiui, so it's hard to know how best to proceed here.

[matrixbot]

This comment was originally posted by @turt2live at matrix-org/sydent#338 (comment).

mxisd is more likely to be ma1sd: https://github.com/ma1uta/ma1sd

[matrixbot]

This comment was originally posted by @callahad at matrix-org/sydent#338 (comment).

Would be good to set a public deprecation timeline for this

[matrixbot]

This comment was originally posted by @turt2live at matrix-org/sydent#338 (comment).

related: matrix-org/matrix-spec-proposals#2713

[matrixbot]

This comment was originally posted by @erikjohnston at matrix-org/sydent#338 (comment).

@turt2live suggests that there is other work that might need to be done before we can remove the v1 APIs, but he'd need to go and spend some time figuring out what that is. (I hope that is a fair representation)

[matrixbot]

This comment was originally posted by @BillCarsonFr at matrix-org/sydent#338 (comment).

Notice that email invites are still using V1 api in order to sign the invitation.
Also we cannot just change the email template to use the v2 API, because v2 API requires consent for the sign-ed25519 endpoint, this will lead to a pretty bad invite flow.

This API is defined as helper for client that cannot do crypto, but still element clients are using it.
I tried to get more info on what clients are supposed to do but failed to find more info.

I guess that before removing this API we should:

• Update documentation to explain the invitation signing process (for client to do it without server)
• Update EW, EA, EI to do the signing themself from the invite link
• Modify the email template to not send a sign URL (as it wont be usable as is because in v2 params are in request body and not in query) but instead send the token, private_key as well as the identity server url as regular parameters

[matrixbot]

This comment was originally posted by @richvdh at matrix-org/sydent#338 (comment).

I decided to try and pull some figures on where the v1 API is being used. Over the 7 days to 24 June:

• Lookup API:
  • 53719 calls to /v1/lookup (excluding those from our internal Prometheus 🙄). These are split:
    • 52538: mxisd (Use of outdated /_matrix/identity/api/v1/lookup API ma1uta/ma1sd#114)
    • 1129: Synapse (Drop support for calling /_matrix/client/v3/rooms/{roomId}/invite without an id_access_token matrix-org/synapse#13206)
  • 10195 bulk lookups via /v1/bulk_lookup. Useragent splits:
    • 9602 Riot.im/0.9.1 on Android. Suspect this is a single user doing something weird
    • 435 Riot on iPhone, with weird version numbers. Forks/old versions? None from current versions of Riot-iOS.
    • 124 AOCIM/1.5. No real idea what this is.
• Due to Remove account_threepid_delegates option matrix-org/synapse#5881:
  • 6082 GET /_matrix/identity/api/v1/3pid/getValidated3pid (of which 468 were successful: the remainder were polls)
  • 200 POST /_matrix/identity/api/v1/validate/email/requestToken
  • 448 GET /_matrix/identity/api/v1/validate/email/submitToken
  • 604 POST /_matrix/identity/api/v1/validate/msisdn/requestToken
  • 405 POST /_matrix/identity/api/v1/validate/msisdn/submitToken
• Due to Stop using /_matrix/identity/api/v1/3pid/unbind matrix-org/synapse#13171:
  • 78 POST /_matrix/identity/api/v1/3pid/unbind
• Due to Drop support for calling /_matrix/client/v3/rooms/{roomId}/invite without an id_access_token matrix-org/synapse#13206:
  • the 1129 calls to /v1/lookup mentioned above
  • 1193 calls to POST /_matrix/identity/api/v1/store-invite from Synapse
  • 13 GET /_matrix/identity/api/v1/pubkey/isvalid
• Unclassified:

    365 POST /_matrix/identity/api/v1/sign-ed25519
     17 GET /_matrix/identity/api/v1/sign-ed25519
   288 GET /_matrix/identity/api/v1/pubkey/ephemeral/isvalid
      8 HEAD /_matrix/identity/api/v1/validate/email/submitToken
      6 OPTIONS /_matrix/identity/api/v1/account/register
      3 GET /_matrix/identity/api/v1/terms
      3 GET /_matrix/identity/api/v1/pubkey/ed25519:0
      3 GET /_matrix/identity/api/v1/hash_details
  

[matrixbot]

This comment was originally posted by @t3chguy at matrix-org/sydent#338 (comment).

/_matrix/identity/api/v1/sign-ed25519 is a strange one.

3PID (email) invites include a link like https://app.element.io/#/room/%21cCpRBTFuXwPmOCUFRL%3Amatrix.org?email=michaelt%40element.io&signurl=https%3A%2F%2Fvector.im%2F_matrix%2Fidentity%2Fapi%2Fv1%2Fsign-ed25519%3Ftoken%3DcBeQOefWJXQrcgHavWuqjMtiazBrYMCvLbpGvOSWFLkitbMbuFGyEmgBbcPtfHhHxiKKsdEbEXSJezCbTXowCuFaQIKV%26private_key%3DRkzs-2BDCjkOOnS74LwLRO_Uw&room_name=&room_avatar_url=&inviter_name=Michael%20Telatynski&guest_access_token=&guest_user_id=&room_type=

Which passes a signurl using the aforementioned API, but encodes the params in query params rather than a POST body which the v2 API docs state the API demands. This means a simple v1->v2 port isn't so simple as now clients would need to receive the parameters outside of the URL and include them in a POST body themselves, with no backwards compatibility.

The API description being

To aid clients who may not be able to perform crypto themselves, the identity server offers some crypto functionality to help in accepting invitations. This is less secure than the client doing it itself, but may be useful where this isn’t possible.

when in reality clients can't really do the signing themselves without parsing the signurl in brittle ways given the URL params don't match any known documentation means that clients are forced to blindly just call a given URL in undocumented ways.

[matrixbot]

This comment was originally posted by @richvdh at matrix-org/sydent#338 (comment).

For links, the endpoint in question used to be specced at https://matrix.org/docs/spec/identity_service/r0.3.0.html#deprecated-post-matrix-identity-api-v1-sign-ed25519; note that a GET variant was never part of the spec.

@t3chguy (and @BillCarsonFr, since I think you said the same thing): I'm failing to grok how this "signurl" is used in the 3pid flow. How is the URL generated? How is it used? I can't find it mentioned at https://spec.matrix.org/v1.5/client-server-api/#server-behaviour-7.

[matrixbot]

This comment was originally posted by @t3chguy at matrix-org/sydent#338 (comment).

It is being used as POST but with no body and all params being query params, so still doesn't match that spec

The link comes from the email templates https://github.com/search?q=repo%3Amatrix-org%2Fsydent+sign-ed25519&type=code
The client includes a third_party_signed in the /join/ request which is the object it got back from the sign-ed25519 API, the path & params to which it got via signurl in the link in the email template above

[matrixbot]

This comment was originally posted by @richvdh at matrix-org/sydent#338 (comment).

It is being used as POST but with no body and all params being query params, so still doesn't match that spec

Right. Yes.

The link comes from the email templates https://github.com/search?q=repo%3Amatrix-org%2Fsydent+sign-ed25519&type=code

Is that the link you meant? It doesn't seem to give anything useful here.

[matrixbot]

This comment was originally posted by @t3chguy at matrix-org/sydent#338 (comment).

Yup - all the templates have v1 references to the API.

[matrixbot]

This comment was originally posted by @richvdh at matrix-org/sydent#338 (comment).

that link definitely doesn't show me those results, but thanks

[matrixbot]

This comment was originally posted by @t3chguy at matrix-org/sydent#338 (comment).

Ah, might be because I'm on the Github Search Beta which makes the search quite usable