Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSC3911: Linking media to events #3911

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

richvdh
Copy link
Member

@richvdh richvdh commented Oct 16, 2022

@turt2live turt2live added proposal A matrix spec change proposal s2s Server-to-Server API (federation) client-server Client-Server API security kind:core MSC which is critical to the protocol's success hacktoberfest-accepted needs-implementation This MSC does not have a qualifying implementation for the SCT to review. The MSC cannot enter FCP. labels Oct 16, 2022

### Overview

After an item of media is uploaded, it must be linked to an event (via
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For MLS, we may need to reference media within a to-device event, but we might be able to link it to an in-room event. I'll have to look into this more, but flagging this as something that will need to be considered.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's also ephemeral-ish things like user profiles which aren't directly related to events, though are duplicated/copied to membership events in most cases.

@turt2live turt2live self-requested a review October 18, 2022 17:52
Copy link

@Munna231 Munna231 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • #

@richvdh
Copy link
Member Author

richvdh commented Oct 23, 2022

I've moved a bunch of the changes here out to MSC3716 MSC3916, in the hope of making them a little more digestible.

@richvdh richvdh force-pushed the rav/proposal/linking_media_to_events branch from 464e5a4 to cc853ef Compare October 23, 2022 23:17
@richvdh richvdh closed this Oct 23, 2022
@deepbluev7
Copy link
Contributor

deepbluev7 commented Oct 23, 2022

I've moved a bunch of the changes here out to MSC3716, in the hope of making them a little more digestible.

You meant MSC3916 (if anyone else has issues following that link :D)

@richvdh richvdh reopened this Oct 23, 2022
@richvdh
Copy link
Member Author

richvdh commented Oct 23, 2022

I've moved a bunch of the changes here out to MSC3716, in the hope of making them a little more digestible.

You meant MSC3916 (if anyone else has issues following that link :D)

Thank you, yes I did 🤦. Edited.

Comment on lines +233 to +235
In addition, this could cause duplication of media in the remote media cache,
if the implementation does not take steps to deduplicate (eg, storing media
by content hash rather than media id).
Copy link
Contributor

@reivilibre reivilibre Mar 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we spec a response header for the media download that provides a suitable hash of the object?
e.g. X-Matrix-Media-Hash: blake2,AAAAAAAAAAAAAAAAAAAAAAAAAAA....
That way, a homeserver (or client!) which is downloading a copy of the media can abort the download if it finds the hash already in its media store. This means that copies of large media do not lead to each server downloading it for each copy.
(Further, for bandwidth-sensitive clients, a HEAD could be used to get the hash before even initiating the download?)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

De-duplication is already possible at an implementation level, and would probably be handled by a different MSC.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

De-duplication is already possible at an implementation level

I think that reivilibre wasn't talking about the way clients/homeservers are deduplicating files after download, but rather before they're downloaded. I agree that former is an implementation detail that doesn't require uniformity across software, and as such doesn't need to be covered in protocol spec. But if the intention here is to avoid the very process of downloading a file multiple times (from a different linked MXC each time), can this be done without protocol declaring one well known way to discover e.g. that hash of underlying file by HEAD request?

I'm specifically thinking of a situation where you've got a user looking at sticker selector and not-so-maliciously spamming them by clicking them 100s of times. Some people do that, it isn't really a bad behavior in some rooms and contexts (slightly distant example would be custom emote spam on Twitch chats). Since the stickers sent in each event are no longer sharing the same MXC, AFAIK client (and homeserver, if sender is remote) will have to download the sticker 100s of times, and thus generate a lot of unnecessary traffic. Only then it will be able to deduplicate file for local storage/cache.

So my (and I assume reivilibre's) concern here is that with the proposal as it is right now, there's no way for client/homeserver to avoid that "traffic duplication", so to speak. Or is there? 😅

Copy link

@rom4nik rom4nik Jan 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, on the other hand, that choice of hash type used for deduplication before download, might eventually have an effect on future implementations of dedupe after download. For example, iirc currently in Synapse there's a mapping of MXC URI to local filename which is a random string. With this MSC, each server would have to store a mapping from the new generated IDs to the local file (in Synapse's case, again, a random name). If protocol adds a way to map MXC IDs to file hash early, is there a reason for servers to not store files by naming them after their hashes (as received over e.g. federation)? This would save you one more lookup: MXC IDs -> file hash -> randomized local name.

Oh, and I haven't thought how/if this "early dedupe" could realistically work with files attached to E2EE events, and whether it would add any additional metadata leak other than what's caused by this MSC as is.

Comment on lines +35 to +39
1. A new "media upload" endpoint is defined, `POST
/_matrix/client/v1/media/upload`. It is based on the existing
[`/_matrix/media/v3/upload`](https://spec.matrix.org/v1.4/client-server-api/#post_matrixmediav3upload)
endpoint, but media uploaded this way is not initially viewable (except to
the user that uploaded it). This is referred to as a "restricted" media item.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: extend this to /_matrix/media/v1/create, as added by MSC2246.

Comment on lines +145 to +146
The new media item can be attached to a new event, and generally functions
in every way the same as uploading a brand new media item.
Copy link
Member

@turt2live turt2live Sep 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can the user copy restricted media they don't have access to? (I assume not, but what is the appropriate error code?)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say no, it's the same as /download and /thumbnail. 403 and M_UNAUTHORIZED.

Comment on lines +221 to +222
removed. Currently, Synapse does not support removing the events (see also
[synapse#4720](https://github.com/matrix-org/synapse/issues/4720)); but if at
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Synapse was fixed a few days ago, apparently :)

Comment on lines +177 to +180
It is expected that servers will continue to treat such media as unrestricted
(at least for local users), but it would be legitimate for them to, for example,
return a different `mxc:` URI for each requesting user, and only allow each user
access to the corresponding `mxc:` URI.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I see the advantage of a server issuing distinct URIs for previews. Is there a motivation for putting this here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I was trying to reason about how we might apply access restrictions to media that is part of a URL preview (if we decided that was a sensible thing to do). It may well be that the extra text confuses more than it clarifies.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leaving url previews as a way to keep "uploading" unrestricted media would leave open the attack vector this MSC is trying to close.


## Unstable prefix

TODO
Copy link
Member

@turt2live turt2live Sep 18, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
TODO
While this MSC is not considered stable, implementations should use the following mapped values.
| Stable | Unstable |
|-|-|
| `/_matrix/client/v1/media/upload` | `/_matrix/client/unstable/org.matrix.msc3911/media/upload` |
| `/_matrix/client/v1/media/create` | `/_matrix/client/unstable/org.matrix.msc3911/media/create` |
| `attach_media` | `org.matrix.msc3911.attach_media` |
| `restrictions` | `org.matrix.msc3911.restrictions` (`event_id` and `profile_user_id` are not prefixed) |
| `/_matrix/client/v1/media/copy/:serverName/:mediaId` | `/_matrix/client/unstable/org.matrix.msc3911/media/copy/:serverName/:mediaId` |

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(this is what MMR will be using)

Comment on lines +55 to +58
If any of the `attach_media` parameters do not correspond to known
restricted media items, or they refer to restricted media items that have
already been attached, the server responds with a 400 error with
`M_INVALID_PARAM`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there also be a limitation that one can only attach media that they uploaded?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

erm... good question.

I think the answer is "yes, there should", to guard against the possibility that someone gets hold of the mxc: URI between it being uploaded and the media being attached to an event. (In which case, they could attach it to their own event and get access to the media.) Particularly in the case of /create rather than /upload, in which the mxc: URI would appear in access logs.

Comment on lines +148 to +149
This "copy" api is to be used by clients when forwarding events with media
attachments.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'd need to use this for m.sticker events too.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(and custom emoji, when that exists)

--gc0p4Jq0M2Yt08jU534c0p
```

5. New "media copy" API
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A downside of the media copy API is clients would no longer be able to reliably cache media. For the example of stickers or emoji, popular/busy rooms could have tens of references to the same media object, which causes tens of downloads because the client doesn't know different.

Possible alternatives (for future MSCs, imo) would be:

  1. Allow some media objects to be reference counted/linked to multiple events.
  2. Include a hash of the file in the event, so clients can locally realize that the objects are all the same content-wise.

I'm more preferential to the second.

--gc0p4Jq0M2Yt08jU534c0p
```

5. New "media copy" API
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

going down the rabbit hole of custom emoji, there's an interaction with MSC4027 we will have to figure out.

@AndrewRyanChama

This comment was marked as duplicate.


Fixes [synapse#1263](https://github.com/matrix-org/synapse/issues/1263).

## Potential issues
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AndrewRyanChama says:

How does this interact with m.replace? Will the client be expected to /copy all of the medias or will it be handled automagically?

#### Redacting events

Under this proposal, servers can determine which media is referenced by an
event that is redacted, and add that media to a list to be cleaned up.
Copy link

@AndrewRyanChama AndrewRyanChama Dec 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with this is that if the user who sent the media leaves the room, the homeserver may no longer get the reaction events. That means there will be no mechanism for those events to be removed.

Content-Type: application/json

{ "restrictions": {
"event_id": "$Rqnc-F-dvnEYJTyHq_iKxU2bZ1CI92-kuZq3a5lr5Zg"
Copy link

@AndrewRyanChama AndrewRyanChama Dec 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This means that the association between (encrypted) media and event ids will be plainly accessible to the homeserver, and federated servers. I'd imagine that isn't something we actually care about but something that others may complain about.

Given that /copy presumably can't actually decrypt the media, that also means that the homeserver can associate any copied encrypted media as well


After an item of media is uploaded, it must be linked to an event (via
parameters to the `/send` api). A given piece of media is only visible
to a user if the user can see the corresponding event.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since reference counting is required anyways, I feel like it's better if the server does it automatically. I wrote a short doc on that here: #4086

Comment on lines +90 to +92
The `/_matrix/federation/v1/media/download` and `/_matrix/federation/v1/media/thumbnail`
endpoints specified by [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916)
are extended: the returned json object may have a property `restrictions`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are several updates required to this MSC - the priority has been to get 3916 into working order, then the team will move onto this MSC afterwards.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MSC3916 was merged 🥳

Copy link
Contributor

@HarHarLinks HarHarLinks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewing with my Nordeck hat on.

The existing endpoint is deprecated. Media uploaded via the existing endpoint
is "unrestricted".

2. Attaching media
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the proposal is called "Linking media to events" and "linking" terminology is used outside of this list item. is this intentional? can we agree on a single term otherwise? "attaching" seems better suited.


If any of the `attach_media` parameters do not correspond to known
restricted media items, or they refer to restricted media items that have
already been attached, the server responds with a 400 error with
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reading this proposal top down, it is not entirely clear whether "already attached" means a different event or a duplicate query parameter.

Comment on lines +76 to +77
If the media is not attached to either an event or a profile within a reasonable period
(say, ten minutes), then the server is free to assume that the user has changed their
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for clarity: the period starts as the media upload has finished, which is particularly relevant with async uploads?


The new `/download` and `/thumbnail` endpoints added in
[MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916) are
updated the server must check if the requesting user or server has
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
updated the server must check if the requesting user or server has
updated such that the server must check if the requesting user or server has

cache the restrictions list.

If neither `event_id` nor `profile_user_id` are present, the requesting
user should assume that an unknown restriction is present, and not allow access
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
user should assume that an unknown restriction is present, and not allow access
homeserver should assume that an unknown restriction is present, and not allow access

* The only protection for media is the obscurity of the URL, and URLs are
easily leaked (eg accidental sharing, access
logs). [synapse#2150](https://github.com/matrix-org/synapse/issues/2150)
* Anybody (including non-matrix users) can cause a homeserver to copy media
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since authenticated media landed, I think

Suggested change
* Anybody (including non-matrix users) can cause a homeserver to copy media
* Any locally authenticated user can cause a homeserver to copy media

Comment on lines +177 to +180
It is expected that servers will continue to treat such media as unrestricted
(at least for local users), but it would be legitimate for them to, for example,
return a different `mxc:` URI for each requesting user, and only allow each user
access to the corresponding `mxc:` URI.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

leaving url previews as a way to keep "uploading" unrestricted media would leave open the attack vector this MSC is trying to close.

Comment on lines +200 to +201
* special-casing the bridge AS user to permit it to upload media without
expiry.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* special-casing the bridge AS user to permit it to upload media without
expiry.
* special-casing the bridge AS user to permit it to upload unrestricted media.

other ideas:

  • attaching media to messages retroactively (e.g. the pastebin copy to the original long message)
  • tracking media origin room and/or user so AS don't upload entirely unrestricted media


Fixes [synapse#1263](https://github.com/matrix-org/synapse/issues/1263).

## Potential issues
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please consider that widgets also use media, i.e. #4039

to summarize, linking media is not a big issue for events of a well-known structure referencing them - with regards to widgets, that means that clients can parse the payload and handle it automatically. however especially for widgets it is often attractive to send nonstandard events (let alone ones with some custom encoding), that prohibit this.

there can be use cases (with widgets or without), where multiple media are attached to an event that represents a certain state at a point in time and which is edited collaboratively. however editing would usually involve changing only one of the referenced media items at most, resulting in an immense overhead when having to copy all the other attached media as well when sending an update (example implementation).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
client-server Client-Server API hacktoberfest-accepted kind:core MSC which is critical to the protocol's success needs-implementation This MSC does not have a qualifying implementation for the SCT to review. The MSC cannot enter FCP. proposal A matrix spec change proposal s2s Server-to-Server API (federation) security
Projects
Status: Scheduled - v1.10
Development

Successfully merging this pull request may close these issues.