[Bug]: Cannot bypass automated crawler detection by creating User Data Directory

[Hnam29]

crawl4ai version

v0.5.x

Expected Behavior

I follow the instruction in "Creating a User Data Directory (Command-Line Approach via Playwright)" to save the profile data in user_data_dir.
browser_config = BrowserConfig(
headless=False,
verbose=True,
use_managed_browser=True,
user_data_dir="/Users/{my_name}/my_chrome_profile",
browser_type="chromium"
)

Current Behavior

The cloudflare still displays the checkbox captcha

Is this reproducible?

Yes

Inputs Causing the Bug

Steps to Reproduce

Code snippets

OS

macOS

Python version

3.11

Browser

Chrome

Browser version

No response

Error logs & Screenshots (if applicable)

URL: https://ahrefs.com/website-authority-checker/

[aravindkarnam]

@Hnam29 The browser profile can help you with authentication on websites but I believe you'd still be subject to the bot detection measures of different services. Did you try with magic mode turned on. It mimics the behaviour of real user. Try that once and see if it works.

[Hnam29]

@aravindkarnam Thank you for your advice. I tried with the magic mode but it also failed to handle the captcha.

[aravindkarnam]

Ok @Hnam29 . I'll play around with this and see if I can come up with some solution.

[Hnam29]

Ok @Hnam29 . I'll play around with this and see if I can come up with some solution.
@aravindkarnam Thanks a ton. Btw, could you please also look at my previous issue https://github.com/unclecode/crawl4ai/issues/737?