This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
500 errors with "duplicate key value" on remote_media_cache_thumbnails when dynamic_thumbnails enabled #2182
Comments
|
This is the reason I made pull request #1946 |
|
This continues to bite me all the time on riot-android and the toasts that keep popping up are getting annoying. |
|
I can confirm that I experience the same bug! Seems to only apply to specific remote thumnails. EDIT: DELETE FROM remote_media_cache_thumbnails;during stopped state, then restarting did the trick for me! |
|
Still experiencing this. It's a pretty awful experience for some avatars to load and some to fail. |
turt2live
changed the title
|
Same in 0.31.1 |
|
The issue can be solved without doing full table migrations - at least on PostgreSQL, technically without even requiring downtime. ALTER TABLE remote_media_cache_thumbnails DROP CONSTRAINT remote_media_cache_thumbnails_media_origin_media_id_thumbna_key;
CREATE INDEX remote_media_cache_thumbnails_media_origin_media_id_thumbna_key ON remote_media_cache_thumbnails (media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type, thumbnail_method); |
|
Just had the same after enabling dynamic thumbnails on my Synapse. Can confirm that the SQL posted by ananace helps, but what is the consequence? That now multiple thumbs of the same size are allowed, right? But why does Synapse even try to create its own dynamically if there's already one present for the desired resolution? |
|
Dropping by to confirm that @ananace workaround did the trick! |
|
This remains an issue as of the 0.33.4 Release. The @ananace workaround provides the relief others have mentioned though. |
|
Could some dev comment on @ananace workaround consequences? Should we just do that? |
|
Still an issue on 0.99.1.1, @ananace's workaround still works. Would be nice to know how that affects my server though. Does that mean I store a new thumbnail every time a client asks for one? |
This was referenced Mar 22, 2019
Closed
|
Another report of this at #4915. Logs are: |
|
thanks alot for sql statement. helped instantly! |
|
I have a similar error here #4389 ( Thank you |
|
Just came across this issue and the psql statements above helped, but can we get an official fix for this? |
|
Same for local thumbnails failing sometimes with this error: I believe this should fix it: |
|
@rkfg please create a pr with the fix |
|
It feels more like a workaround than as a fix tbh |
4 tasks
richvdh
changed the title
|
For links: #7124 is a work-in-progress in fixing this. |
|
I got: for my userpic on my homeserver: Is this the same issue, or another one? |
|
@532910 I don't think so. this issue affects remote thumbnails only, afaik. Suggest you file a separate issue with a full stacktrace. |
|
I believe @532910 you're seeing a regression of #1816. I also experienced this, my database schema did not match what was in the PR. Altering the CONSTRAINT to include the method resolved the database error for me. (I know I really ought to open a new issue, but I'm entirely too busy right now, I'm sorry.) |
|
fixed by #7124. |
I'm still getting these errors in v1.21.0 which afaik includes that commit? |
|
sigh. Please could you share the logs, with the stack trace? |
|
|
It seems like the migrations didn't delete the duplicate entries. The following two queries should fix that, if you run into those issues with the background update: DELETE FROM local_media_repository_thumbnails a USING (
SELECT MIN(ctid) as ctid, media_id, thumbnail_width, thumbnail_height, thumbnail_type, thumbnail_method
FROM local_media_repository_thumbnails
GROUP BY media_id, thumbnail_width, thumbnail_height, thumbnail_type, thumbnail_method HAVING COUNT(*) > 1
) b
WHERE a.media_id = b.media_id
AND a.thumbnail_width = b.thumbnail_width
AND a.thumbnail_height = b.thumbnail_height
AND a.thumbnail_type = b.thumbnail_type
AND a.thumbnail_method = b.thumbnail_method
AND a.ctid <> b.ctid;
DELETE FROM remote_media_cache_thumbnails a USING (
SELECT MIN(ctid) as ctid, media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type, thumbnail_method
FROM remote_media_cache_thumbnails
GROUP BY media_origin, media_id, thumbnail_width, thumbnail_height, thumbnail_type, thumbnail_method HAVING COUNT(*) > 1
) b
WHERE a.media_id = b.media_id
AND a.media_origin = b.media_origin
AND a.thumbnail_width = b.thumbnail_width
AND a.thumbnail_height = b.thumbnail_height
AND a.thumbnail_type = b.thumbnail_type
AND a.thumbnail_method = b.thumbnail_method
AND a.ctid <> b.ctid;They delete the duplicates and should allow the migrations to pass, although I'm not sure, if that has any consequences. |
|
@deepbluev7 I'm confused. I thought these exceptions were happening because the existing index ( That doesn't seem to be @f0x52's problem, anyway, because he's reporting an error with the new constraint. @f0x52: is it possible that you are just seeing two requests for the same thumbnail at the same time, and they are racing? I think that would be #6755, which is a somewhat different problem :/ |
|
@richvdh It seems their databases contains duplicates, which is why the migration fails. They may have deleted the index manually or something, not sure, but I did verify, that the db contains duplicates and deleting them fixes the issue. At least running the above query seemed to fix it for both people reporting the issue. But yeah, that seems to not apply to @f0x52's issue. No idea about that one. |
|
@deepbluev7 @richvdh |
|
right, yes, people who have manually modified their database schema will now need to manually fix the resulting damage :). |
|
It looks like this is fixed. Anyway I don't see this error more. |
|
Hi there, I've just noticed I have errors in my synapse 1.29.0 logs, such as: Is it related to this issue? |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
and others
Error is at the end of this issue (it's a long one).
The following URLs produce similar errors to the one included here:
There may be more, however those 3 are easily noticed.
Changing the method to
scaledoes not result in an error. This does not appear to affect local media.I tried to delete the files and records from the database in varying combinations (delete files & delete records, only delete records, only delete files) in various running states of synapse (running, shut down, restart after). I tried various combinations of deleting data from the two tables (from both
remote_media_cacheandremote_media_cache_thumbnails, only_thumbnails, only_cache). No luck.The error continues and does not go away. If I'm missing a step, please let me know. Otherwise, this appears to be a very specific and strange bug.
Synapse version: v0.20.0
The text was updated successfully, but these errors were encountered: