MSC4039: Access the Content repository with the Widget API #4039

dhenneke · 2023-07-24T18:01:42Z

Implementations

Implement MSC4039: Add an MSC for a new Widget API action to upload files into the media repository matrix-widget-api#86
Implement MSC4039: Add an MSC for a new Widget API action to upload files into the media repository matrix-react-sdk#11311

… repository Signed-off-by: Dominik Henneke <dominik.henneke@nordeck.net>

robintown

Thanks for the nice write-up, I think this is very reasonable as an extension to the widget API as it currently exists (and it's an extension I was already expecting to be made sooner or later)

proposals/4039-widget-api-media.md

Signed-off-by: Dominik Henneke <dominik.henneke@nordeck.net>

proposals/4039-widget-api-media.md

Co-authored-by: Johannes Marbach <n0-0ne+github@mailbox.org>

- added download API to handle MSC3916 "Authentication for media" - added e2ee aware variants for both upload and download Signed-off-by: Kim Brose <kim.brose@nordeck.net>

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

HarHarLinks · 2024-08-15T09:48:11Z

proposals/4039-widget-api-media.md

+extended such that whenever a widget includes references to files in an event, it is required to provide them to the
+client in the `send_event` request so the client knows to attach the files.
+
+## Alternatives


we could of course also pass the client's access token to the widget by URI param or with a request, but that seems to contradict the whole idea of the widget API

HarHarLinks · 2024-08-16T09:30:25Z

In line with matrix-org/matrix-spec#1700, the following disclosure applies:

I am an active member of the Matrix FOSS community, member of the Matrix Foundation governing board, and employed by Nordeck. My parts of this proposal are written and published as a Nordeck employee.

(Similar applies to the other commits to this PR as indicated by the origin branch coming from the nordeck fork.)

HarHarLinks · 2024-08-27T08:13:30Z

Implementations:

download_file (authenticated media part)
timeout_ms (async media)
- tbd
EncryptedFile (e2ee)
- tbd

HarHarLinks · 2024-08-29T17:05:39Z

proposals/4039-widget-api-media.md

+}
+```
+
+`response.file` mirrors `data.file` of the `upload_file` action and is a `XMLHttpRequestBodyInit` that is supported as a


Suggested change

`response.file` mirrors `data.file` of the `upload_file` action and is a `XMLHttpRequestBodyInit` that is supported as a

`response.file` mirrors `data.file` of the `upload_file` action and is an `XMLHttpRequestBodyInit` that is supported as a

HarHarLinks · 2024-08-29T17:05:44Z

proposals/4039-widget-api-media.md

+}
+```
+
+`data.file` is a `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API.


Suggested change

`data.file` is a `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API.

`data.file` is an `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API.

dbkr · 2024-08-30T10:30:59Z

proposals/4039-widget-api-media.md

+}
+```
+
+`data.file` is a `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API.


This seems like quite a random type? Why not just blob or something? Do we explicitly want the API to support passing, say, FormData through directly?

This is not easy to answer, since the team members originally proposing that type have left the team. We did not change the type as we updated the MSC because the upload part is already implemented. The reason seems to be "Something that can be sent to XMLHttpRequest.send (typically a File)." https://github.com/matrix-org/matrix-widget-api/pull/86/files#diff-67e6f0676307a87283ca4533526de9b3ac4ff3bafdd0d016947e40e2b3574757R693-R694. However we can follow your reasoning that probably a Blob (which is one of the types forming this union type) would make more sense.

dbkr · 2024-08-30T10:32:40Z

proposals/4039-widget-api-media.md

+  "requestid": "generated-id-1234",
+  "action": "upload_file",
+  "data": {
+    "file": "some-content"


'file' here sort of implies that this is a https://developer.mozilla.org/en-US/docs/Web/API/File which it is not. This sort of goes with my other comment below on what type this is and what it's called.

dbkr · 2024-08-30T10:35:43Z

proposals/4039-widget-api-media.md

+}
+```
+
+`response.file` mirrors `data.file` of the `upload_file` action and is a `XMLHttpRequestBodyInit` that is supported as a


As above for 'file', plus it will be the widget's job to support whatever the client decides to send it of the union of different types, which might be an easy trap for widget authors to fall into. The impl in the react-sdk PR just always sends a blob anyway.

if we decide to change the type to Blob, perhaps calling both fields blob would make the most sense.

dbkr · 2024-08-30T10:37:42Z

proposals/4039-widget-api-media.md

+the API described above to add the `copy` endpoint. Additionally, the event sending payload from MSC2762 will need to be
+extended such that whenever a widget includes references to files in an event, it is required to provide them to the
+client in the `send_event` request so the client knows to attach the files.
+


Also, large files will have to be sent over the postmessage API in one big lump, so this isn't going to scale up to anything over a few tens of MB very well.

One alternative already described could be letting widgets directly access (parts of) the C-S API in some way.

Add an MSC for a new Widget API action to upload files into the media…

f346c77

… repository Signed-off-by: Dominik Henneke <dominik.henneke@nordeck.net>

dhenneke marked this pull request as ready for review July 24, 2023 18:58

turt2live added proposal A matrix spec change proposal kind:feature MSC for not-core and not-maintenance stuff widgets anything to do with widgets labels Jul 25, 2023

robintown reviewed Jul 31, 2023

View reviewed changes

proposals/4039-widget-api-media.md Outdated Show resolved Hide resolved

Fix copy-paste errors

3e95735

Signed-off-by: Dominik Henneke <dominik.henneke@nordeck.net>

Johennes reviewed Aug 16, 2023

View reviewed changes

proposals/4039-widget-api-media.md Outdated Show resolved Hide resolved

uhoreg changed the title ~~MSC4039: Add an MSC for a new Widget API action to upload files into the media repository~~ MSC4039: Access the Content repository with the Widget API Sep 15, 2023

hegdenischay mentioned this pull request May 12, 2024

Giphy GIF search integration maunium/stickerpicker#75

Closed

Fix new interface description

1adb2e3

Co-authored-by: Johannes Marbach <n0-0ne+github@mailbox.org>

HarHarLinks mentioned this pull request Aug 14, 2024

Update MSC4039 nordeck/matrix-spec-proposals#1

Merged

1 task

Kim Brose added 14 commits August 15, 2024 10:37

WIP: Update MSC4039

1fc1831

- added download API to handle MSC3916 "Authentication for media" - added e2ee aware variants for both upload and download Signed-off-by: Kim Brose <kim.brose@nordeck.net>

define EncryptedFile object

c1c8f64

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

restructure headings

1947d2d

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

address the "e2ee in widget" alternative

26b6a7b

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

address "async uploads"

9202051

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

explain the choice of payload design

4627fc0

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

remove authenticated media issue which has been addressed, reformat

960ebc5

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

add "widget api 2.0" alternative

59386f3

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

remove matrix_base_url

09a6260

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

format diff

58f899e

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

update alternatives

a5e8065

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

consider MSC3911

569c854

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

remove all TODOs

ff1692c

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

replace flowchart with sequenceDiagram

514e8be

Signed-off-by: Kim Brose <kim.brose@nordeck.net>

HarHarLinks reviewed Aug 15, 2024

View reviewed changes

HarHarLinks mentioned this pull request Aug 16, 2024

MSC3911: Linking media to events #3911

Open

This was referenced Aug 23, 2024

Implement download_file action matrix-org/matrix-widget-api#98

Closed

Implement the download_file widget action nordeck/matrix-widget-toolkit#759

Merged

weeman1337 mentioned this pull request Aug 27, 2024

Implement MSC4039 download_file action matrix-org/matrix-widget-api#99

Merged

HarHarLinks mentioned this pull request Aug 29, 2024

Implement download_file in widget driver matrix-org/matrix-react-sdk#12931

Merged

4 tasks

HarHarLinks reviewed Aug 29, 2024

View reviewed changes

dbkr reviewed Aug 30, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MSC4039: Access the Content repository with the Widget API #4039

MSC4039: Access the Content repository with the Widget API #4039

dhenneke commented Jul 24, 2023 •

edited

Loading

robintown left a comment

HarHarLinks Aug 15, 2024 •

edited

Loading

HarHarLinks commented Aug 16, 2024

HarHarLinks commented Aug 27, 2024 •

edited

Loading

HarHarLinks Aug 29, 2024

HarHarLinks Aug 29, 2024

dbkr Aug 30, 2024

HarHarLinks Aug 30, 2024

dbkr Aug 30, 2024

dbkr Aug 30, 2024

HarHarLinks Aug 30, 2024 •

edited

Loading

dbkr Aug 30, 2024

HarHarLinks Aug 30, 2024

	`response.file` mirrors `data.file` of the `upload_file` action and is a `XMLHttpRequestBodyInit` that is supported as a
	`response.file` mirrors `data.file` of the `upload_file` action and is an `XMLHttpRequestBodyInit` that is supported as a

	`data.file` is a `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API.
	`data.file` is an `XMLHttpRequestBodyInit` that is supported as a data type in the `postMessage` API.

MSC4039: Access the Content repository with the Widget API #4039

Are you sure you want to change the base?

MSC4039: Access the Content repository with the Widget API #4039

Conversation

dhenneke commented Jul 24, 2023 • edited Loading

robintown left a comment

Choose a reason for hiding this comment

HarHarLinks Aug 15, 2024 • edited Loading

Choose a reason for hiding this comment

HarHarLinks commented Aug 16, 2024

HarHarLinks commented Aug 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HarHarLinks Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhenneke commented Jul 24, 2023 •

edited

Loading

HarHarLinks Aug 15, 2024 •

edited

Loading

HarHarLinks commented Aug 27, 2024 •

edited

Loading

HarHarLinks Aug 30, 2024 •

edited

Loading