[DeviantArt] Duplicates in database

[kattjevfel]

When downloading a deviation directly, the entry in gallery-dl's database is like deviantart415470071.png, but when when for example downloading via a gallery, you get deviantartg_fluffytheneko_415470071.png, this easily leads to duplicates being downloaded, and completely defeats the purpose of the database. My solution would be to only keep the first format, as it would also skip re-downloading images when accounts change name.

I also have countless of entries missing the filename, so really I think the format should just be deviantart{index}

[mikf]

Each deviantart sub-extractor uses its own archive id scheme for reasons: g_{_username}_{index}.{extension} for galleries, {index}.{extension} for single posts, etc (it kind of makes sense for favorites and such).
You are absolutely right that {index}.{extension} in general would be the best, but changing the internal defaults would invalidate someone else's archive entries and I'd rather avoid that if possible.
For your own setup, you can use the archive-format option to change the default value to something more reasonable.

[Hrxn]

I take a part of the blame here, because I've argued in favor of those reasons here in the past 😄

The rationale here is basically the principle of least surprise, but this is somewhat subjective and ultimately a question of personal priorities (in effect, not having avoiding duplicates as the highest priority), but I'll admit it's debatable whether this is really the most reasonable choice.

DeviantArt is kind of extreme here, with 10 different archive_fmt used as defaults for the different subcategories.

But this also depends on the cooperation of the site, to some extent. The identifier used here is apparently the {index} field, as a good reference for the user side archive we depend on the site to actually have working globally unique IDs for their hosted content. I think DeviantArt passes this test, as far as I know, but some sites had issues with this in ye olden days.

[kattjevfel]

I wasn't aware of archive-format, I've set that now and I guess that'll be good enough for now, but it sure has caused a lot of frustration in the past, especially since I have a post-processor that converts pngs to webp, but the tool wont overwrite, so several times I've ended up with duplicates that are hard to deal with :p

Anyway, can we at least set the same for galleries and direct link to posts? Aka set g_{_username}_{index}.{extension} for that one too, this would get rid of duplicate entries and I don't think it'd mess anything up.

[rautamiekka]

g_{_username}_{index}.{extension}

This leads to the problem where replacing the upload with a modified one, one which could be a better one, is skipped.

Conversely, having the upload's name leads to potential dupes, but pretty sure you can't upload anything so massive it'll be a problem if downloaded multiple times, and it's always better to have a potential dupe than replace with a potentially damaged, let alone inferior, one.

I'd recommend:

{category}_{author[username]}_{index}_{date:%Y-%m-%d_%H_%M_%S}_{title}.{extension}

^ Example:

deviantart_Olivergriffiths_892443878_2021-09-20_13_34_31_MLP_ Watercolor commission.jpg

^ That's a 24h time cuz I didn't find a direct formatting code in Python docs.

[Hrxn]

[..] especially since I have a post-processor that converts pngs to webp, but the tool wont overwrite, so several times I've ended up with duplicates that are hard to deal with :p

Well, there are tools specifically made to deal with duplicate files. These here should even support similar image search, which might help with manually converted images etc.

• https://github.com/qarmin/czkawka
• https://github.com/arsenetar/dupeguru

Anyway, can we at least set the same for galleries and direct link to posts? Aka set g_{_username}_{index}.{extension} for that one too, this would get rid of duplicate entries and I don't think it'd mess anything up.

Fine with me...

[kattjevfel]

Well, there are tools specifically made to deal with duplicate files. These here should even support similar image search, which might help with manually converted images etc.

I think we misunderstood each other, what ends up happening is that the file has been downloaded by gallery-dl, then converted to another format. Then gallery-dl downloads that file again, suddenly I have one version in png, and one in webp. But the safety measure to not overwrite files kicks in and I am stuck with both files.

All in all, external problem, but it gets triggered by the database not doing its job :P

[mikf]

Anyway, can we at least set the same for galleries and direct link to posts? Aka set g_{_username}_{index}.{extension} for that one too, this would get rid of duplicate entries and I don't think it'd mess anything up.

Done (ada36c2)