duplicate key value violates unique constraint "remote_media_cache_media_origin_media_id_key" when getting remote thumbnails

[michaelkaye]

on matrix-org-hotfixes:

psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "remote_media_cache_media_origin_media_id_key"
DETAIL:  Key (media_origin, media_id)=(XXX.net, XXXXXXXXXXXXscjwZ) already exists.

This fires when attempting to get a thumbnail for some remote media:

GET /_matrix/media/r0/thumbnail/XXX.net/XXXXXXXXXXXXscjwZ?width=30&height=30&method=crop

There were multiple requests to this URL from different IPs near-simultaneously - presumably someone just posted this to a large channel.

The request to the remote server to download the media looks just fine, I assume there's a race somewhere. If this duplicate error is OK can we suppress the logging of it, as it's a ~200line stacktrace, but ideally the additional connections block and don't process the upload.

[michaelkaye]

May be related to #1335 which was marked fixed due to not being replicable for a period.

[richvdh]

yeah looks like #1335 was never fixed. It will just be a race between the two requests.

If this duplicate error is OK can we suppress the logging of it

It's not OK, because one of the requesters will have got a 500. It'll need to be fixed properly.

[njouanin]

Hi,
I've just migrated to synapse 1.11. I generated a new homeserver.yml because of env variables deprecation. Now I get the same kind of errors in my log file:

psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "remote_media_cache_media_origin_media_id_key"```

[richvdh]

to be clear, this issue affects only concurrent requests for thumbnails of the same piece of media.

[njouanin]

my mistake, I had an access denied error to the docker volume holding media_store.

[varac]

I'm still getting the same error msg for latest helm chart (docker.io/bitnami/postgresql:11.11.0-debian-10-r71 + matrixdotorg/synapse:v1.49.0)