[Fanbox] Cloudflare challenges #5791

biggestsonicfan · 2024-06-26T13:22:39Z

Looks like Fanbox recently updated to add cloudflare challenges? I'm using my cookies from browser with the same browser type and user agent.

EDIT: Only way I've gotten anything to work is by a Japanese VPN

drunk-moe · 2024-06-26T21:48:04Z

Refresh your cookies and double check your user agent. Just happened to me, got my UA from the requested headers in the network panel and rexported my cookies and it works fine now. A bit annoying but what can you do?

Also - not using any sort of VPN or anything. It worked fine for me once I grabbed my cookies again and used the same User Agent my browser was using.

biggestsonicfan · 2024-06-27T01:40:49Z

Strangely, it was the "browser": "firefox" key that was the culprit. Removing that resolved my issue.

mikf · 2024-06-27T13:24:10Z

Removing that resolved my issue.

browser overrides user-agent when present.

bugreporter2334 · 2024-06-27T22:11:54Z

drunk-moe's method is not quite working for me. spoofing my browser's user agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 and using the same FANBOXSESSID from the cookies still gives me the error
[fanbox][warning] Cloudflare challenge
[fanbox][error] HttpError: '403 Forbidden' for 'https://api.fanbox.cc/post.listCreator?creatorId=[creator]&limit=10'
refreshing the page on my browser does not give a cloudfare challenge.

edit:
gallery-dl version for posterity (windows):

$ gallery-dl --version
1.27.1

relevant portions of my config:

    "extractor":
    {
        "base-directory": "./gallery-dl/",
        "parent-directory": false,
        "postprocessors": null,
        "archive": null,
        "cookies": null,
        "cookies-update": true,
        "proxy": null,
        "skip": true,

        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
        "retries": 4,
        "timeout": 30.0,
        "verify": true,
        "fallback": true,

        "sleep": 0,
        "sleep-request": 0,
        "sleep-extractor": 0,

        "path-restrict": "auto",
        "path-replace": "_",
        "path-remove": "\\u0000-\\u001f\\u007f",
        "path-strip": "auto",
        "extension-map": {
            "jpeg": "jpg",
            "jpe" : "jpg",
            "jfif": "jpg",
            "jif" : "jpg",
            "jfi" : "jpg"
        },

        "fanbox":
        {
            "cookies": 
            {
              "FANBOXSESSID": "<sessionid>"
            },
            "postprocessors": 
            [{
              "name": "metadata",
              "event": "post",
              "mode": "json",
              "filename": "{id}.json",
              "directory": "posts"
            }]
        },

ZenoArrows · 2024-06-28T20:42:53Z

I've also suddenly started encountering these none of the methods described so far resolve it unfortunately.

biggestsonicfan · 2024-06-28T22:25:05Z

"FANBOXSESSID": "<sessionid>"

You are missing both the cf_clearance and __cf_bm cloudflare cookies.

ZenoArrows · 2024-06-28T22:35:34Z

You are missing both the cf_clearance and __cf_bm cloudflare cookies.

I've tried that also with the user agent set to be the same as my browser, but it still seems to be able to distinguish between my browser and gallery-dl.

biggestsonicfan · 2024-06-29T00:03:01Z

My reply was for @bugreporter2334, I can't see your config file if you don't post it.

bugreporter2334 · 2024-06-29T00:13:04Z

"FANBOXSESSID": "<sessionid>"

You are missing both the cf_clearance and __cf_bm cloudflare cookies.

Yep, works now with these two cookies. Here's the syntax:

"fanbox":
        {
            "cookies": 
            {
              "FANBOXSESSID": "fanbox cookie",
              "cf_clearance": "cf cookie1",
              "__cf_bm": "cf cookie 2"
            },
            "postprocessors": 
            [{
              "name": "metadata",
              "event": "post",
              "mode": "json",
              "filename": "{id}.json",
              "directory": "posts"
            }]
        },

On Chrome you can extract them from the same Application tab that you normally get your Fanbox cookies from.

bossanovaorca · 2024-06-30T06:46:47Z

"fanbox": {
  "cookies": ["firefox", null, null, null, ".fanbox.cc"],
},

I currently retrieve my cookies from my browser. do I need to change this method for it to retrieve the cloudflare cookies properly or am I using it incorrectly?

mikf · 2024-06-30T13:33:25Z

You are using it correctly, but you need to use the user-agent of your browser as well.

"fanbox": {
  "cookies": ["firefox", null, null, null, ".fanbox.cc"],
  "user-agent": "..."
},

Extracting cookies from a browser profile can only get cookies that have been written to disk. Sometimes it takes a while for your browser to update its cookies database file.

tryoutuser · 2024-07-02T16:16:32Z

i've got this error instead :

[2024-07-02 23:09:03][warning] cookies: not enough values to unpack (expected 7, got 6)
[2024-07-02 23:09:03][warning] no 'FANBOXSESSID' cookie set

i'm using gallery-dl latest v1.27.1.
my config for fanbox :

        "fanbox":
        {
            "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
            "cookies": "%APPDATA%\\gallery-dl\\fanbox.txt",
            "cookies-update": false,
            "base-directory": "I:\\Server\\Server 1\\home\\user\\gallery-dl\\kemonoparty\\",
            "verify": false,
            "archive": "%APPDATA%\\gallery-dl\\fanbox.sqlite3",
            "archive-format": "{user['userId']}_{id}_{num:02d}_{filename}",
            "embeds": false,
            "filename": "{id}_{title}_{num:02d}_{filename}.{extension}",
            "directory": ["fanbox", "{user['userId']}"]
        },

inside my cookies text file :

# Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This file was generated by EditThisCookie
.fanbox.cc	TRUE	/	TRUE	1719936563	__cf_bm <redacted>
.fanbox.cc        TRUE       /       TRUE        1751445568    cf_clearance <redacted>
.fanbox.cc	TRUE	/	TRUE	1722524401	agreement_master_terms_of_use_202408	1
.fanbox.cc	TRUE	/	TRUE	1722526763	FANBOXSESSID	<redacted>
.fanbox.cc	TRUE	/	TRUE	1725825006	p_ab_d_id	187055155
.fanbox.cc	TRUE	/	TRUE	1725825006	p_ab_id	5
.fanbox.cc	TRUE	/	TRUE	1725825006	p_ab_id_2	3
.fanbox.cc	TRUE	/	TRUE	1751481237	privacy_policy_agreement	7
.fanbox.cc	TRUE	/	TRUE	1754494763	privacy_policy_notification	0
www.fanbox.cc	FALSE	/	TRUE	0	d_prefs	<redacted>

and i'm connected to softether VPN volunteer servers that is not the public-vpn.*number*.ad.ne.jp ones.

mikf · 2024-07-02T17:21:07Z

@tryoutuser
Values in a cookies.txt file must be separated by tab characters and not spaces like it is the case for your cf_clearance line. Replace it with the following and it should load:

.fanbox.cc	TRUE	/	TRUE	1751445568	cf_clearance	<redacted>

tryoutuser · 2024-07-03T05:51:06Z

@mikf i did edit with tab after realizing that and it still says same error those 2 lines above.

updated cookie :

# Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This file was generated by EditThisCookie
.fanbox.cc	TRUE	/	TRUE	1719941452	__cf_bm	<redacted>
.fanbox.cc	TRUE	/	TRUE	1751475670	cf_clearance	<redacted>
.fanbox.cc	TRUE	/	TRUE	1722524401	agreement_master_terms_of_use_202408	1
.fanbox.cc	TRUE	/	TRUE	1722531668	FANBOXSESSID	<redacted>
.fanbox.cc	TRUE	/	TRUE	1725825006	p_ab_d_id	187055155
.fanbox.cc	TRUE	/	TRUE	1725825006	p_ab_id	5
.fanbox.cc	TRUE	/	TRUE	1725825006	p_ab_id_2	3
.fanbox.cc	TRUE	/	TRUE	1751481237	privacy_policy_agreement	7
.fanbox.cc	TRUE	/	TRUE	1754499668	privacy_policy_notification	0
www.fanbox.cc	FALSE	/	TRUE	0	d_prefs	<redacted>

Hrxn · 2024-07-04T14:41:57Z

You sure those are real tabs, and not soft tabs, elastic tabs etc. or something else?

Coolchilion · 2024-07-04T23:26:13Z

Where do you get cf_clearance from? I keep clearing my cookies and solving the verification but it never appears in my Get Cookies.txt

komoreshi · 2024-07-09T03:01:46Z

Where do you get cf_clearance from? I keep clearing my cookies and solving the verification but it never appears in my Get Cookies.txt

Same issue on my end, but I suspect it just takes a while for it to be updated. Worst case you just extract it manually using developer tools and add it to your config:

USBhost · 2024-07-11T04:43:24Z

Okay lads I need some help understanding what to do.
./gallery-dl --cookies-from-browser firefox --user-agent browser URL does not work
./gallery-dl --cookies cookies-fanbox-cc.txt --user-agent browser URL does not work.
cookie file gotten from the export cookie addon.
I keep getting 403 Forbidden

edit: ./gallery-dl --cookies-from-browser firefox --user-agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:128.0) Gecko/20100101 Firefox/128.0' URL

for the dumb lads like me make sure firefox is your primary or manually get your useragent.

Coolchilion · 2024-07-12T03:51:32Z

bugreporter2334

I tried following your config but I get

Extra data: line 1 column 16 (char 15)
and
no 'FANBOXSESSID' cookie set

TheCrypticOne · 2024-08-23T21:25:29Z

Having this issue as well. Using a cookies.txt file and getting

[fanbox][warning] Cloudflare challenge
[fanbox][error] HttpError: '403 Forbidden' for 'https://api.fanbox.cc/post.info?postId=8400185'

as the output

amackenzie · 2024-09-04T22:23:05Z

Same here, using cookies.txt method that worked for a while:

[fanbox][warning] Cloudflare challenge
[fanbox][error] HttpError: '403 Forbidden' for 'https://api.fanbox.cc/post.paginateCreator?creatorId=<id removed>'

JacobDreiling · 2024-09-11T22:03:31Z

I started getting the same error. Here are my fanbox configs. I'm not sure if I need to specify a user agent or a cookie file, because this used to work before I added them.

"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15",
"cookies": {
    "agreement_master_terms_of_use_202408": "1",
    "p_ab_id": "...",
    "p_ab_id_2": "...",
    "p_ab_d_id": "...",
    "FANBOXSESSID": "...",
    "cf_clearance": "...",
    "__cf_bm": "...",
    "privacy_policy_agreement": "...",
    "privacy_policy_notification": "..."
    },
"cookies-update": "test-fanbox-cookies.txt"

mikf · 2024-09-12T07:48:39Z

Visit the URL causing the 403 error in a browser
Let it solve the challenge
Export its cookies (cf_clearance is important)
Run gallery-dl with those cookies and the browser's User-Agent header

$ gallery-dl https://xub.fanbox.cc --cookies cookies-fanbox-cc.txt --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"
./fanbox/xub/7288162_0.jpg
./fanbox/xub/6566008_0.jpg
./fanbox/xub/6561218_0.jpg
...

(I can personally access fanbox without any of this)

JacobDreiling · 2024-09-12T23:45:34Z

Please excuse all my comments, but I'm getting an error when trying to export cookies. Here's what I have to automatically get cookies from safari.

"fanbox": {
    "user-agent": "browser",
    "cookies": ["safari"]
}

This gives me an error 1.

[fanbox][warning] cookies: [Errno 1] Operation not permitted: '/Users/quasarbooster/Library/Cookies/Cookies.binarycookies'
[fanbox][warning] no 'FANBOXSESSID' cookie set
[fanbox][warning] Cloudflare challenge

cloudywings2 · 2024-09-19T21:21:50Z

Hello! The process I have used is:

using the Get Cookies.txt LOCALLY, mentioned on the main GitHub page
config file contains the path to the cookies.txt file, and the user agent

When running Gallery-DL as an .exe file I get the "cloudflare challenge" error and "403 Forbidden." My cookies do not have the cf_clearance tag, even if I export all the cookies from the browser. I have exported the cookies from the Fanbox home page as well as an individual artist's page, being sure I am logged in, of course.

Syilumi · 2024-09-27T10:17:02Z

can anyone simply explain how to fix it because i dont know how, i dont even know how iv been using gallery-dl since iv just been using .bat files with the gallery-dl hyperlinks and hoping for the best

Taboganas · 2024-10-02T23:49:50Z

i keep getting cloudfare challenges i tried getting the cf clearance but it doesnt pop up on my developer tools at all only cf bm does

Can someone show me how to get cf clearance please

cloudywings2 · 2024-10-04T08:08:16Z

Visit the URL causing the 403 error in a browser

Let it solve the challenge

Export its cookies (cf_clearance is important)

Run gallery-dl with those cookies and the browser's User-Agent header
$ gallery-dl https://xub.fanbox.cc --cookies cookies-fanbox-cc.txt --user-agent "Mozilla/5.0 (X11; Linux x86_64; rv:128.0) Gecko/20100101 Firefox/128.0"
./fanbox/xub/7288162_0.jpg
./fanbox/xub/6566008_0.jpg
./fanbox/xub/6561218_0.jpg
...
(I can personally access fanbox without any of this)

Hello! I mentioned above that I was unable to get the cf_clearance cookie to appear. By switching from Chrome to Firefox I was able to get this cookie, however, I still get the 403 error for all attempts to download from Fanbox. As far as I can tell, it seems that Fanbox support is completely broken. Is anyone able to download from Fanbox?

My settings are as follows:

`
"extractor":
{
"base-directory": "C:/gallery-dl",
"postprocessors": null,
"archive": null,
"cookies": null,
"proxy": null,
"skip": true,
"sleep": 0,
"user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0",

"fanbox":
{
"cookies": "www.fanbox.cc_cookies.txt",
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
},`

Rezboa · 2024-10-05T22:01:22Z

I am having this issue too. I tried all of the suggestions people listed here, but I still get that darn "[fanbox][warning] Cloudflare challenge" error when I try to pull the images from someone I'm subscribed too.

biggestsonicfan · 2024-10-07T02:33:14Z

If what I'm seeing is correct, Fanbox is now setting cf_clearance cookies to expire immediately.

USBhost · 2024-10-07T03:38:17Z

lads all you need to do is this

./gallery-dl --cookies-from-browser firefox --user-agent 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0' "https://someone.fanbox.cc/posts"
[cookies][info] Extracted 139 cookies from Firefox
etc.....

use firefox, login like normal check and make sure your user agent is the same on your command. then run it.

biggestsonicfan · 2024-10-07T11:36:57Z

~~@USBhost That method no longer works with Firefox 131.~~

~~EDIT: As a temporary fix, I have installed Firefox Portable 130 and point to the Data\Profile folder to extract cookies from and it's working. This is a band-aid solution, however.~~

Was using the incorrect user agent, oops. Firefox works with 131, but you seemingly can't gain access to the cf_clearance through the browser. I can access it in the cookies.sqlite file directly with an sqlite browser, but I realize this is not ideal.

USBhost · 2024-10-08T04:44:27Z

but you seemingly can't gain access to the cf_clearance through the browser. I can access it in the cookies.sqlite

Open debugging tools, network, go somewhere... copy request headers and the cf_clearance is right there.

biggestsonicfan · 2024-10-08T06:37:38Z

Open debugging tools, network, go somewhere... copy request headers and the cf_clearance is right there.

...but you seemingly can't gain access to the cf_clearance as a cookie through the browser

Excuse me for not being specific

mikf · 2024-10-08T13:57:20Z

Firefox 131 introduced support for "partitioned" cookies: CHIPS. This prevents addons like Export Cookies from including the cf_clearance cookie in their output, but --cookies-from-browser will still grab it. This feature can be controlled with Firefox's network.cookie.CHIPS.enabled setting in about:config.

mikf added the cloudflare label Jun 27, 2024

matsumo0922 mentioned this issue Jul 1, 2024

ログインできなくなりました matsumo0922/PixiView-KMP#4

Closed

komoreshi mentioned this issue Jul 9, 2024

fanbox HttpError: '403 Forbidden' #5835

Closed

biggestsonicfan mentioned this issue Oct 9, 2024

CF_Clearance Cookie Missing Nandaka/PixivUtil2#1379

Open

mikf mentioned this issue Oct 27, 2024

[Fanbox] Cookies not Working #6380

Closed

[Fanbox] Cloudflare challenges #5791

[Fanbox] Cloudflare challenges #5791

Comments

biggestsonicfan commented Jun 26, 2024 • edited Loading

drunk-moe commented Jun 26, 2024 • edited Loading

biggestsonicfan commented Jun 27, 2024

mikf commented Jun 27, 2024

bugreporter2334 commented Jun 27, 2024 • edited Loading

ZenoArrows commented Jun 28, 2024

biggestsonicfan commented Jun 28, 2024

ZenoArrows commented Jun 28, 2024

biggestsonicfan commented Jun 29, 2024

bugreporter2334 commented Jun 29, 2024

bossanovaorca commented Jun 30, 2024

mikf commented Jun 30, 2024 • edited Loading

tryoutuser commented Jul 2, 2024 • edited Loading

mikf commented Jul 2, 2024

tryoutuser commented Jul 3, 2024

Hrxn commented Jul 4, 2024

Coolchilion commented Jul 4, 2024

komoreshi commented Jul 9, 2024 • edited Loading

USBhost commented Jul 11, 2024 • edited Loading

Coolchilion commented Jul 12, 2024

TheCrypticOne commented Aug 23, 2024

amackenzie commented Sep 4, 2024

JacobDreiling commented Sep 11, 2024

mikf commented Sep 12, 2024

JacobDreiling commented Sep 12, 2024

cloudywings2 commented Sep 19, 2024

Syilumi commented Sep 27, 2024

Taboganas commented Oct 2, 2024 • edited Loading

cloudywings2 commented Oct 4, 2024 • edited Loading

Rezboa commented Oct 5, 2024

biggestsonicfan commented Oct 7, 2024

USBhost commented Oct 7, 2024 • edited Loading

biggestsonicfan commented Oct 7, 2024 • edited Loading

USBhost commented Oct 8, 2024 • edited Loading

biggestsonicfan commented Oct 8, 2024

mikf commented Oct 8, 2024

biggestsonicfan commented Jun 26, 2024 •

edited

Loading

drunk-moe commented Jun 26, 2024 •

edited

Loading

bugreporter2334 commented Jun 27, 2024 •

edited

Loading

mikf commented Jun 30, 2024 •

edited

Loading

tryoutuser commented Jul 2, 2024 •

edited

Loading

komoreshi commented Jul 9, 2024 •

edited

Loading

USBhost commented Jul 11, 2024 •

edited

Loading

Taboganas commented Oct 2, 2024 •

edited

Loading

cloudywings2 commented Oct 4, 2024 •

edited

Loading

USBhost commented Oct 7, 2024 •

edited

Loading

biggestsonicfan commented Oct 7, 2024 •

edited

Loading

USBhost commented Oct 8, 2024 •

edited

Loading