if remote media cache is deleted on the fs, the server should notice and redownload (SYN-709)

[matrixbot]

Submitted by @​matthew:matrix.org

(Imported from https://matrix.org/jira/browse/SYN-709)

[Ezwen]

Such feature would be great, I lost part of my media folder by mistake, and now riot is confused for some avatars and room icons.

Is there any workaround for such case, like a postgre script/query to properly remove missing (or even all) remote media and make synapse consequently re-download them?

[ordinarygulp]

Is this still an issue? Because we recently moved servers, while cloning the data, and now some avatars not registered with our homeserver are missing. Seems related to #3479 as well.

[aaronraimist]

Yes it is still an issue (that's why it is open). If you didn't move all the data the then you essentially deleted it manually which Synapse doesn't like. You can use the purge remote media API to get avatars back. https://github.com/matrix-org/synapse/blob/master/docs/admin_api/purge_remote_media.rst

Use a time in the future.

#synapse:matrix.org is a better place for support.

[anoadragon453]

You can use the purge remote media API to get avatars back. https://github.com/matrix-org/synapse/blob/master/docs/admin_api/purge_remote_media.rst

Note that this doc file has moved to https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/media_admin_api.md#purge-remote-media-api.

[PC-Admin]

Noticed this bug myself recently when using the purge API. We purged everything on perthchat.org until 30 days ago, then deleted all the empty folders in the media repo.

We then noticed most of the remote avatars for users/rooms were not rendering. Upon attempting to load them postgresql produces this error:

Apr 27 06:30:32 perthchat matrix-postgres[29502]: 2021-04-27 06:30:32.208 UTC [29] DETAIL:  Key (media_origin, media_id)=(gomatrixhosting.com, SCCSVfsybyCRyYvtFdRnMjmh) already exists.
Apr 27 06:30:32 perthchat matrix-postgres[29502]: 2021-04-27 06:30:32.208 UTC [29] STATEMENT:  INSERT INTO remote_media_cache (media_origin, media_id, media_type, media_length, created_ts, upload_name, filesystem_id, last_access_ts) VALUES('gomatrixhosting.com', 'SCCSVfsybyCRyYvtFdRnMjmh', 'image/png', 248309, 1619505032207, 'PNG_File_trimmed.png', 'bQqYeuCbiMGwIOzviBJUAwMO', 1619505032207)

Can confirm that purging to a future date does in fact fix this. :)

[gbolcer]

Purge command appears to succeed. Returns deleted:0. Element.io client still does not return corrected images.
I need to confirm what server Synapse server versions have this fixed.

{
"deleted": 0
}

[aaronraimist]

@gbolcer deleted 0 means you didn't delete any. The most common reason for this is providing a timestamp in seconds rather than milliseconds.

[gbolcer]

https://myservername.com/_synapse/admin/v1/purge_media_cache?before_ts=1634329818
Timestamp is current date/tz.

[aaronraimist]

@gbolcer yes but in the wrong format. The API wants milliseconds. In milliseconds 1634329818 is Jan 19, 1970 which is probably not the day you were trying to use.

[gbolcer]

Thank you @aaronraimist Nice catch.

{
"deleted": 3759
}

Confirmed, all icons repopulated!