Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deviantart: Is it possible to avoid these API calls when skipping downloads? #4767

Open
tcmbackwards opened this issue Nov 4, 2023 · 2 comments

Comments

@tcmbackwards
Copy link

I'm hitting the API limit quickly despite downloads being skipped because gallery-dl seems to still be making API calls for every skipped download. It doesn't happen with every gallery but here's an example. Is it possible to avoid these calls?

[gallery-dl][debug] Version 1.26.1 - Executable
[gallery-dl][debug] Python 3.8.10 - Windows-10-10.0.19045
[gallery-dl][debug] requests 2.31.0 - urllib3 1.26.17
[gallery-dl][debug] Configuration Files ['G:\\gallery-dl\\gallery-dl.conf']
[gallery-dl][debug] Starting DownloadJob for 'https://www.deviantart.com/vermillianno/gallery'
[deviantart][debug] Using DeviantartGalleryExtractor for 'https://www.deviantart.com/vermillianno/gallery'
[deviantart][debug] Using custom API credentials (client-id ****)
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.deviantart.com:443
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/user/profile/vermillianno HTTP/1.1" 200 1328
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/gallery/all?username=vermillianno&offset=0&limit=24&mature_content=true HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/metadata?deviationids%5B0%5D=D38DB198-CEFD-BBA9-B721-EFFD742B24D4&deviationids%5B1%5D=9E965F12-5DE0-8EB1-5E3C-69C9D49080EB&deviationids%5B2%5D=B209F0C6-9F23-B9E8-ED21-DC84B61A48FA&deviationids%5B3%5D=CF3424FE-0162-7FE2-FD1E-ED506799DA28&deviationids%5B4%5D=D53ACFA5-03E4-CB0A-326A-1B4F61A8D3F2&deviationids%5B5%5D=553072EA-18EF-569B-0D9C-F465F87C4438&deviationids%5B6%5D=011E3F22-4852-D240-BEC5-357373C09095&deviationids%5B7%5D=AA93E7A4-07E6-5C90-B242-5000C6B998CF&deviationids%5B8%5D=F02D6D0A-908C-9EE2-AC6F-C861784DFCBA&deviationids%5B9%5D=3E6998F4-DD20-367B-3727-A839CFE35EA4&deviationids%5B10%5D=0D6F6714-191D-CB40-8C45-66BD519E5AE9&deviationids%5B11%5D=28CD06A4-2863-BCA1-D4B4-CB466C09D926&deviationids%5B12%5D=6D15E6FA-6C80-82CC-49B6-04657529C544&deviationids%5B13%5D=3F552B86-F587-754E-922A-B494B3D8C653&deviationids%5B14%5D=82F4F5B1-E8D6-66C2-B06E-C66E165AE31B&deviationids%5B15%5D=98EFF86A-91B8-70D0-7DC8-ED29A90C8B5E&deviationids%5B16%5D=553124DB-CC83-5CA6-BA63-7D36C96940EC&deviationids%5B17%5D=5721E7E3-C3FF-A10B-BC35-7EC0489EF12F&deviationids%5B18%5D=D5454D4A-7BD6-8744-798C-3FB674CADC2E&deviationids%5B19%5D=6F623E23-9237-07CA-8EF3-58A1A6FAA3A8&deviationids%5B20%5D=73043376-C1D0-A992-F2CC-F4792EA05FF4&deviationids%5B21%5D=30159B45-5915-73BE-429D-D925DF11FD3B&deviationids%5B22%5D=E11EE3B3-4D78-C4E0-9FB4-283A26D3E959&deviationids%5B23%5D=E39E2695-BD51-01E2-34C5-59F0CFB7026E&mature_content=true HTTP/1.1" 200 1982
[deviantart][debug] Using download archive 'deviantart-vermillianno-archive.sqlite3'
[postprocessor.metadata][debug] Using metadata archive 'deviantart-vermillianno-metaarchive.sqlite3'
[deviantart][debug] Active postprocessor modules: [MetadataPP]
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/D38DB198-CEFD-BBA9-B721-EFFD742B24D4?mature_content=true HTTP/1.1" 200 594
# .\gallery-dl\deviantart\vermillianno\deviantart_992202208_old man.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/9E965F12-5DE0-8EB1-5E3C-69C9D49080EB?mature_content=true HTTP/1.1" 200 592
# .\gallery-dl\deviantart\vermillianno\deviantart_991901359_Howl.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/B209F0C6-9F23-B9E8-ED21-DC84B61A48FA?mature_content=true HTTP/1.1" 200 596
# .\gallery-dl\deviantart\vermillianno\deviantart_984270208_ghosts.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/CF3424FE-0162-7FE2-FD1E-ED506799DA28?mature_content=true HTTP/1.1" 200 597
# .\gallery-dl\deviantart\vermillianno\deviantart_973726104_More Athan.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/D53ACFA5-03E4-CB0A-326A-1B4F61A8D3F2?mature_content=true HTTP/1.1" 200 593
# .\gallery-dl\deviantart\vermillianno\deviantart_973725937_Athan.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/553072EA-18EF-569B-0D9C-F465F87C4438?mature_content=true HTTP/1.1" 200 614
# .\gallery-dl\deviantart\vermillianno\deviantart_973725446_yet another drawing of lucky.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/011E3F22-4852-D240-BEC5-357373C09095?mature_content=true HTTP/1.1" 200 599
# .\gallery-dl\deviantart\vermillianno\deviantart_973725184_Miles Morales.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/AA93E7A4-07E6-5C90-B242-5000C6B998CF?mature_content=true HTTP/1.1" 200 601
# .\gallery-dl\deviantart\vermillianno\deviantart_973725060_A building.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/F02D6D0A-908C-9EE2-AC6F-C861784DFCBA?mature_content=true HTTP/1.1" 200 610
# .\gallery-dl\deviantart\vermillianno\deviantart_973724817_Chroma corps cadet_ Calico.jpg
[urllib3.connectionpool][debug] https://www.deviantart.com:443 "GET /api/v1/oauth2/deviation/download/3E6998F4-DD20-367B-3727-A839CFE35EA4?mature_content=true HTTP/1.1" 200 595```

Here's my config file, I'm making a metadata archive but I can't see that it's doing anything. Setting "extra" to false didn't help.

```{
    "extractor": {
        "archive": "{category}-{username}-archive.sqlite3",
        "archive-pragma": ["journal_mode=WAL", "synchronous=NORMAL"],

        "deviantart": {
            "client-id": "****",
            "client-secret": "****",
            "auto-watch": true,
            "extra": true,
            "include": "gallery,scraps,journals",
            "public": false,
  	    "sleep": 2,
            "postprocessors": [{
                    "name": "metadata",
			"skip": true,
			"archive": "{category}-{username}-metaarchive.sqlite3",
			"archive-pragma": ["journal_mode=WAL", "synchronous=NORMAL"]
                }
            ]
        }
    },
    "downloader": {},
    "output": {},
    "postprocessor": {}
}```
@mikf
Copy link
Owner

mikf commented Nov 4, 2023

You can use --range 123- to skip that many posts ahead without using any extra API calls.
Avoiding API calls for skipped downloads is currently not possible.

@mikf
Copy link
Owner

mikf commented Dec 10, 2023

Duplicate of #317

@mikf mikf marked this as a duplicate of #317 Dec 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants