Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[redgifs] 404 not found, but still available in browser #1962

Closed
Hrxn opened this issue Oct 19, 2021 · 5 comments
Closed

[redgifs] 404 not found, but still available in browser #1962

Hrxn opened this issue Oct 19, 2021 · 5 comments
Labels
bug enhancement

Comments

@Hrxn
Copy link
Contributor

Hrxn commented Oct 19, 2021 

PS E:\Etc> gallery-dl --ignore-config "https://www.redgifs.com/watch/blankstrongiberianlynx"
[downloader.http][warning] '404 Not Found' for 'https://thumbs2.redgifs.com/BlankStrongIberianlynx.mp4'
[download][error] Failed to download redgifs_BlankStrongIberianlynx.mp4
PS E:\Etc>

PS E:\Etc> gallery-dl --ignore-config --verbose "https://www.redgifs.com/watch/blankstrongiberianlynx"
[gallery-dl][debug] Version 1.19.0
[gallery-dl][debug] Python 3.10.0 - Windows-10-10.0.19042-SP0
[gallery-dl][debug] requests 2.26.0 - urllib3 1.26.7
[gallery-dl][debug] Starting DownloadJob for 'https://www.redgifs.com/watch/blankstrongiberianlynx'
[redgifs][debug] Using RedgifsImageExtractor for 'https://www.redgifs.com/watch/blankstrongiberianlynx'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.redgifs.com:443
[urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v1/gfycats/blankstrongiberianlynx HTTP/1.1" 200 676
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): thumbs2.redgifs.com:443
[urllib3.connectionpool][debug] https://thumbs2.redgifs.com:443 "GET /BlankStrongIberianlynx.mp4 HTTP/1.1" 404 1097
[downloader.http][warning] '404 Not Found' for 'https://thumbs2.redgifs.com/BlankStrongIberianlynx.mp4'
[download][error] Failed to download redgifs_BlankStrongIberianlynx.mp4
PS E:\Etc>

First thought, API acting up again?! Site continuing to disintegrate before our very eyes again?!

But maybe not?

PS E:\Etc> http "https://api.redgifs.com:443/v1/gfycats/blankstrongiberianlynx"
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Headers: Authorization,DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type
Access-Control-Allow-Methods: GET, PUT, POST, DELETE, HEAD, OPTIONS, PATCH
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 86400
Age: 3498
CF-Cache-Status: HIT
CF-RAY: 6a06e5e0e9f14a86-FRA
Cache-Control: public, max-age=3600
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 676
Content-Type: application/json
Date: Tue, 19 Oct 2021 03:25:13 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
Vary: Accept-Encoding
X-Handler-Duration: 0.014147043228149
X-Init-Duration: 0.0025351047515869
X-PMU: 1832408
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

{
    "gfyItem": {
        "avgColor": "#000000",
        "captionsUrl": null,
        "content_urls": {
            "100pxGif": {
                "height": 250,
                "size": 12890257,
                "url": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-small.gif",
                "width": 444
            },
            "mobile": {
                "height": 480,
                "size": 3189884,
                "url": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-mobile.mp4",
                "width": 854
            },
            "mobilePoster": {
                "height": 250,
                "size": 9879,
                "url": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-mobile.jpg",
                "width": 444
            },
            "poster": {
                "height": 250,
                "size": 9879,
                "url": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-poster.jpg",
                "width": 444
            }
        },
        "createDate": 1622289374,
        "dislikes": 0,
        "domainWhitelist": [],
        "duration": 43,
        "encoding": false,
        "finished": false,
        "frameRate": 0,
        "gatekeeper": 12,
        "geoWhitelist": [],
        "gfyId": "blankstrongiberianlynx",
        "gfyName": "BlankStrongIberianlynx",
        "gifUrl": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-small.gif",
        "hasAudio": true,
        "hasTransparency": false,
        "height": 250,
        "languageCategories": [],
        "likes": 1,
        "max2mbGif": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-small.gif",
        "max5mbGif": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-small.gif",
        "miniPosterUrl": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-mobile.jpg",
        "mobileHeight": 480,
        "mobilePosterUrl": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-mobile.jpg",
        "mobileUrl": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-mobile.mp4",
        "mobileWidth": 854,
        "mp4Url": "https://thumbs2.redgifs.com/BlankStrongIberianlynx.mp4",
        "nsfw": true,
        "numFrames": 0,
        "posterUrl": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-poster.jpg",
        "published": 1,
        "ratio": 1,
        "source": 1,
        "tags": [],
        "thumb100PosterUrl": "https://thumbs2.redgifs.com/BlankStrongIberianlynx-mobile.jpg",
        "type": 1,
        "userData": {
            "description": null,
            "followers": 91,
            "following": 4,
            "name": "Ratsono",
            "profileImageUrl": null,
            "profileUrl": "https://www.redgifs.com/users/ratsono",
            "publishedGfycats": 18,
            "subscription": 0,
            "url": "https://www.redgifs.com/users/ratsono",
            "username": "ratsono",
            "verified": false,
            "views": 893073
        },
        "userName": "ratsono",
        "views": 45,
        "views5": 2,
        "width": 444
    }
}


PS E:\Etc>

It seems to work in the browser, because it's apparently using the <id>-mobile.mp4 variant..
I tried to find any visible differences first by checking the profile page and testing the neighboring clips, but they work without throwing any issues.
The only thing that's also unusual in the browser: There's this options cog wheel symbol in the player, where you can toggle SD and HD, and that is also not working with that specific video.

Seems we can still blame the site, nonetheless.
Anyway, what do do? Also do some kind of fallback in such cases?
@mikf mikf closed this as completed in 8bb442f Oct 23, 2021
@Hrxn
Copy link
Contributor Author

Hrxn commented Oct 24, 2021 •
edited
Loading

Just tested latest dev version from master, can confirm the URL fallback now works! 😄

PS D:\Temp> gallery-dl --verbose --ignore-config 'https://www.redgifs.com/users/ratsono'
[gallery-dl][debug] Version 1.19.1-dev
[gallery-dl][debug] Python 3.10.0 - Windows-10-10.0.19042-SP0
[gallery-dl][debug] requests 2.26.0 - urllib3 1.26.7
[gallery-dl][debug] Starting DownloadJob for 'https://www.redgifs.com/users/ratsono'
[redgifs][debug] Using RedgifsUserExtractor for 'https://www.redgifs.com/users/ratsono'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): api.redgifs.com:443
[urllib3.connectionpool][debug] https://api.redgifs.com:443 "GET /v1/users/ratsono/gfycats?count=100 HTTP/1.1" 200 3884
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): thumbs2.redgifs.com:443
[urllib3.connectionpool][debug] https://thumbs2.redgifs.com:443 "GET /SubmissiveRashWeasel.mp4 HTTP/1.1" 200 19332498
* .\gallery-dl\redgifs\ratsono\redgifs_SubmissiveRashWeasel.mp4
        (( snipped a bit here))
[urllib3.connectionpool][debug] https://thumbs2.redgifs.com:443 "GET /ImaginaryAcrobaticHuemul.mp4 HTTP/1.1" 200 47016233
* .\gallery-dl\redgifs\ratsono\redgifs_ImaginaryAcrobaticHuemul.mp4
[urllib3.connectionpool][debug] https://thumbs2.redgifs.com:443 "GET /BlankStrongIberianlynx.mp4 HTTP/1.1" 404 1097
[downloader.http][warning] '404 Not Found' for 'https://thumbs2.redgifs.com/BlankStrongIberianlynx.mp4'
[download][info] Trying fallback URL #1
[urllib3.connectionpool][debug] Starting new HTTPS connection (2): thumbs2.redgifs.com:443
[urllib3.connectionpool][debug] https://thumbs2.redgifs.com:443 "GET /BlankStrongIberianlynx-mobile.mp4 HTTP/1.1" 200 3189884
* .\gallery-dl\redgifs\ratsono\redgifs_BlankStrongIberianlynx.mp4
[urllib3.connectionpool][debug] https://thumbs2.redgifs.com:443 "GET /ComfortableLightheartedPlainsqueaker.mp4 HTTP/1.1" 404 1097
[downloader.http][warning] '404 Not Found' for 'https://thumbs2.redgifs.com/ComfortableLightheartedPlainsqueaker.mp4'
[download][info] Trying fallback URL #1
[urllib3.connectionpool][debug] Starting new HTTPS connection (3): thumbs2.redgifs.com:443
[urllib3.connectionpool][debug] https://thumbs2.redgifs.com:443 "GET /ComfortableLightheartedPlainsqueaker-mobile.mp4 HTTP/1.1" 200 3189884
* .\gallery-dl\redgifs\ratsono\redgifs_ComfortableLightheartedPlainsqueaker.mp4
[urllib3.connectionpool][debug] https://thumbs2.redgifs.com:443 "GET /BigheartedFormalAfghanhound.mp4 HTTP/1.1" 200 24037918
* .\gallery-dl\redgifs\anonymous\redgifs_BigheartedFormalAfghanhound.mp4
[urllib3.connectionpool][debug] https://thumbs2.redgifs.com:443 "GET /FearlessGlamorousArmadillo.mp4 HTTP/1.1" 200 37690382
* .\gallery-dl\redgifs\ratsono\redgifs_FearlessGlamorousArmadillo.mp4
        (( and here ))
PS D:\Temp>

@mikf
Lucky me, just noticed something else here.. probably not a real bug so I'm reluctant to file a new "official" issue right now..
Nonetheless, might be a question of sane defaults, so I'll do it here:

Again, the same example URL (https://www.redgifs.com/users/ratsono), I found this:

PS D:\Temp\gallery-dl\redgifs> ls

    Directory: D:\Temp\gallery-dl\redgifs

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d----          24.10.2021    16:46                anonymous
d----          24.10.2021    16:47                ratsono
d----          24.10.2021    16:47                saltshaker32

PS D:\Temp\gallery-dl\redgifs>

One is the correct user directory, but there are two more (containing one file each, so in total we have 18 of 18 files just as visible when viewed in the browser)

Note, this is definitely not some recent change, it was not just this test run, I checked my normal download directory, and the result was exactly the same as some days ago.

Confirming here again:

PS D:\Temp\gallery-dl\redgifs> gallery-dl --ignore-config --no-download 'https://www.redgifs.com/users/ratsono'
* .\gallery-dl\redgifs\ratsono\redgifs_SubmissiveRashWeasel.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_NormalMellowNinebandedarmadillo.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_PlumMarvelousAfricanwilddog.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_SilverBuoyantLiger.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_PoliticalBlissfulAmericanbulldog.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_UtterVigorousAmericansaddlebred.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_ConstantHilariousPuma.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_UnluckyGaseousAcouchi.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_ImaginaryAcrobaticHuemul.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_BlankStrongIberianlynx.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_ComfortableLightheartedPlainsqueaker.mp4
* .\gallery-dl\redgifs\anonymous\redgifs_BigheartedFormalAfghanhound.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_FearlessGlamorousArmadillo.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_IndianredRotatingGlowworm.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_QueasyBrilliantNighthawk.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_DifficultShamefulGar.mp4
* .\gallery-dl\redgifs\ratsono\redgifs_SpanishPhonyBabirusa.mp4
* .\gallery-dl\redgifs\saltshaker32\redgifs_FarflungResponsibleAzurevasesponge.mp4
PS D:\Temp\gallery-dl\redgifs>

This is the default config used:

PS D:\Temp\gallery-dl\redgifs> gallery-dl --ignore-config -E 'https://www.redgifs.com/users/ratsono'
Category / Subcategory
  "redgifs" / "user"
Filename format (default):
  "{category}_{gfyName}{title:?_//}.{extension}"
Directory format (default):
  ["{category}", "{userName}"]
Archive format (default):
  "{gfyName}"
PS D:\Temp\gallery-dl\redgifs>

https://www.redgifs.com/watch/farflungresponsibleazurevasesponge
https://www.redgifs.com/watch/bigheartedformalafghanhound

The Problem is the {userName} here, which works for all existing profiles/users, but as can be seen for the two files who ended up in the "wrong" location, while the video content is there and displays just fine, the user profiles do not exist anymore.
(They link to https://www.redgifs.com/users/undefined)
The {userName} as used by gallery-dl is probably the former profile name, as the profile still existed, I'd guess.

@mikf
Copy link
Owner

mikf commented Oct 24, 2021

FarflungResponsibleAzurevasesponge, the one file from saltshaker32, is listed under https://www.redgifs.com/users/ratsono, so it does "belong" there, but none of its API metadata entries mention anything about ratsono.

When downloading from a user URL (/users/ratsono), we could take the name from there and replace any {userName} entries with that, but I don't see a way to fix this when downloading directly from https://www.redgifs.com/watch/farflungresponsibleazurevasesponge. That's just an unfixable problem with redgifs, I guess.

@Hrxn
Copy link
Contributor Author

Hrxn commented Oct 25, 2021 •
edited
Loading

FarflungResponsibleAzurevasesponge, the one file from saltshaker32, is listed under https://www.redgifs.com/users/ratsono, so it does "belong" there, but none of its API metadata entries mention anything about ratsono.

Yes, that is basically what I meant, sorry if I did not make myself clear here.. 😅

When downloading from a user URL (/users/ratsono), we could take the name from there and replace any {userName} entries with that, [..

Exactly, that is what I would be suggesting. Principle of least surprise etc. pp., and this also mimics the behaviour when using the website, i.e. you visit the profile page you desire, and see all the clips there, which should also end up in the same directory structure when downloading with gallery-dl, in my opinion.
(Like, you want to download the profile of "xyz", then everything should end up in ../xyz/)

..] but I don't see a way to fix this when downloading directly from https://www.redgifs.com/watch/farflungresponsibleazurevasesponge. That's just an unfixable problem with redgifs, I guess.

True, and I wasn't even suggesting anything here, to be honest. I was basically just pasting these two URLs into the comment for testing, so that I can find them immediately. I mean, everything works here as expected I'd say, you want to download that specific clip, you get that specific clip. And as a bonus, you can get the old {userName} of a profile that doesn't even exist anymore.

PS D:\> gallery-dl -K "https://www.redgifs.com/watch/farflungresponsibleazurevasesponge" | sls -NoEmphasis -Context 0,1 "userName"

> userName
    saltshaker32
> userName
    saltshaker32

PS D:\>

What is supposed to be missing here?

mikf added a commit that referenced this issue Oct 26, 2021
@mikf
[gfycat] consistent 'userName' values for 'user' downloads (#1962)
e436a26 
by using the name from the input URL and not relying on possibly faulty
or incomplete API results.

'userData[username]', if available, will still have the original name.
@mikf
Copy link
Owner

mikf commented Oct 26, 2021 •
edited
Loading

The username issue is fixed with e436a26.

What is supposed to be missing here?

Any reference to the user profile name this video is listed in, i.e. ratsono. It could have been the case that {userName} has a wrong value, but some other field has the correct one.

@Hrxn
Copy link
Contributor Author

Hrxn commented Oct 27, 2021

This is great, thanks a lot! :)

What is supposed to be missing here?

Any reference to the user profile name gets listed in, i.e. ratsono. It could have been the case that {userName} has a wrong value, but some other field has the correct one.

That's true. What I meant with "missing" was that, given the context of average Joe end-user attempting to download a single gfy clip, the profile name info etc, probably does not matter at all, even if the values are corrrect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mikf @Hrxn