Bug with "rotating proxies"

[jiezi4ai]

Rotating proxy is a very good feature when dealing with multiple urls to avoid rate limit or even ip ban. However, there a bug with sample code in "document - advanced - proxy & security". The sample code on "Rotating Proxies" suggest:

from crawl4ai.async_configs import BrowserConfig

async def get_next_proxy():
    # Your proxy rotation logic here
    return {"server": "http://next.proxy.com:8080"}

browser_config = BrowserConfig()
async with AsyncWebCrawler(config=browser_config) as crawler:
    # Update proxy for each request
    for url in urls:
        proxy = await get_next_proxy()
        browser_config.proxy_config = proxy
        result = await crawler.arun(url=url, config=browser_config)

The config in crawler.arun should be CrawlerRunConfig which is not compatible with BrowserConfig object.
Since there is no proxy parameter in CrawlerRunConfig, one quick fix for the above code could be:

import random
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig

browser_config = BrowserConfig()
results = []
# Update proxy for each request
for url in urls:
    proxy = await get_next_proxy()
    browser_config.proxy_config = proxy
    crawler = AsyncWebCrawler(config=browser_config)
    await crawler.start()
    result = await crawler.arun(url=url, config=CrawlerRunConfig())
    results.append(result)
    await crawler.close()

This works but may not be the best approach, since the crawler is initiated each time when fetching the urls. It would be even better to add update_proxy function in BrowserConfig object or add proxy parameter in CrawlerRunConfig initilization.

[Umpire2018]

@jiezi4ai Thank you for pointing out the issue so carefully!

@aravindkarnam @unclecode Suggest to assign this issue to me. I already have some preliminary ideas for a fix in mind and will be able to dive into it once the refactoring for the current version is completed.

[aravindkarnam]

@Umpire2018 Thanks a bunch. Take a stab at this.

Also outline your ideas for the fix, so the community also has some visibility on how you are going to proceed about this.

[unclecode]

@jiezi4ai Thank you for pointing this out. That's a mistake in the documentation, so I will remove it quickly.

Your solution is workable, but it has one important concern. Imagine crawling multiple URLs; when you create an instance of the AsyncWebCrawler, it creates a new fresh browser object, which is very expensive in terms of memory. The cost is really high.

Currently, in Crawl4ai, when we want to crawl thousands of URLs from the same configuration, we always use one browser object. There are two additional layers according to the Playwright structure: one is the context and the other is the page object. The context is essentially a logical concept that consumes memory and refers to a set of configurations. From that configuration, we create multiple instances. I have tricked the library in various ways to ensure that I don't consume unnecessary memory. Even when we crawl many URLs from the same configuration, Crawl4ai creates only one context and understands the configuration signature. This way, we consume memory for each page and one memory for a single browser object.

You can use hooks and manipulate the context object to achieve this. However, not a good habit haha.

Anyway this is fantastic features that we can easily implement in the current library. @Umpire2018 I appreciate your enthusiasm and your offer to help. You're most welcome. As @aravindkarnam also mentioned, let me know your ideas. For parts of the library that are very close to the core, I listen to suggestions and create requirement documents for contributors to ensure alignment with the current state and future plans. I will share that with you, and then you can have fun helping and supporting. I look forward to your contribution.

[jiezi4ai]

@jiezi4ai Thank you for pointing this out. That's a mistake in the documentation, so I will remove it quickly.

Your solution is workable, but it has one important concern. Imagine crawling multiple URLs; when you create an instance of the AsyncWebCrawler, it creates a new fresh browser object, which is very expensive in terms of memory. The cost is really high.

Currently, in Crawl4ai, when we want to crawl thousands of URLs from the same configuration, we always use one browser object. There are two additional layers according to the Playwright structure: one is the context and the other is the page object. The context is essentially a logical concept that consumes memory and refers to a set of configurations. From that configuration, we create multiple instances. I have tricked the library in various ways to ensure that I don't consume unnecessary memory. Even when we crawl many URLs from the same configuration, Crawl4ai creates only one context and understands the configuration signature. This way, we consume memory for each page and one memory for a single browser object.

You can use hooks and manipulate the context object to achieve this. However, not a good habit haha.

Anyway this is fantastic features that we can easily implement in the current library. @Umpire2018 I appreciate your enthusiasm and your offer to help. You're most welcome. As @aravindkarnam also mentioned, let me know your ideas. For parts of the library that are very close to the core, I listen to suggestions and create requirement documents for contributors to ensure alignment with the current state and future plans. I will share that with you, and then you can have fun helping and supporting. I look forward to your contribution.

Thanks for your detailed explanations! Really appreciate you and all other contributors' work in craw4ai.

[Umpire2018]

I fully concur that

That's a mistake in the documentation.

as the conflation of BrowserConfig and CrawlerRunConfig serve different purposes:

1. BrowserConfig Responsibility:

   • Governs browser/context initialization (e.g., browser type, headless mode, proxy settings).

2. CrawlerRunConfig Responsibility:

   • Manages runtime behaviors (e.g., content extraction rules, waiting conditions).
   • Proxy settings are irrelevant to these operational workflows.

---

Proposed Resolution

1. Documentation Fix:

   • Clarify that proxies must be configured via BrowserConfig during crawler initialization.
   • Remove the flawed example of passing BrowserConfig to arun()/arun_many().

2. Temporary On-Hold Decision:
   Given recent issues (e.g., Filtered markdown result is always empty #461, How modify page_timeout in crawler.arun_many mode #455, fix: "config" for arun_many() is not passing to arun() #472) highlighting inconsistencies in config parameter handling and @unclecode ‘s announcement of:

arun_many() has changed drastically. I made tons of optimizations for much faster and better parallel crawling. I tested it, and I will release it as a beta and will release it by Mondy(UTC+8).

It is prudent to:

• Await the beta release to avoid redundant work or conflicting changes.
• Reassess proxy rotation strategies after verifying the new arun_many() implementation, which may natively support dynamic context/proxy management.

---

Next Steps

• Monitor the new release for changes to configuration handling (especially in BrowserConfig/CrawlerRunConfig) and update user guidance accordingly.
• Re-evaluate proxy rotation implementation once the optimized architecture is stable.

Appreciation: A massive shout-out to the maintainers for their heroic effort in refactoring the core crawling logic – your work is advancing the future of web scraping! 🎉

[Umpire2018]

@jiezi4ai Hi there, You can checkout here.

crawl4ai/docs/examples/v0_4_3b2_features_demo.py

Lines 280 to 331 in 7b7fe84

	async def demo_proxy_rotation():
	"""
	8. Proxy Rotation Demo
	===================
	Demonstrates how to rotate proxies for each request using Crawl4ai.
	"""
	print("\n=== 8. Proxy Rotation Demo ===")
	async def get_next_proxy(proxy_file: str = f"proxies.txt") -> Optional[Dict]:
	"""Get next proxy from local file"""
	try:
	proxies = os.getenv("PROXIES", "").split(",")
	ip, port, username, password = random.choice(proxies).split(":")
	return {
	"server": f"http://{ip}:{port}",
	"username": username,
	"password": password,
	"ip": ip # Store original IP for verification
	}
	except Exception as e:
	print(f"Error loading proxy: {e}")
	return None
	# Create 10 test requests to httpbin
	urls = ["https://httpbin.org/ip"] * 2
	browser_config = BrowserConfig(headless=True, verbose=False)
	run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
	async with AsyncWebCrawler(config=browser_config) as crawler:
	for url in urls:
	proxy = await get_next_proxy()
	if not proxy:
	print("No proxy available, skipping...")
	continue
	# Create new config with proxy
	current_config = run_config.clone(proxy_config=proxy)
	result = await crawler.arun(url=url, config=current_config)
	if result.success:
	ip_match = re.search(r'(?:[0-9]{1,3}\.){3}[0-9]{1,3}', result.html)
	print(f"Proxy {proxy['ip']} -> Response IP: {ip_match.group(0) if ip_match else 'Not found'}")
	verified = ip_match.group(0) == proxy['ip']
	if verified:
	print(f"✅ Proxy working! IP matches: {proxy['ip']}")
	else:
	print(f"❌ Proxy failed or IP mismatch!")
	else:
	print(f"Failed with proxy {proxy['ip']}")

@aravindkarnam I think we can remove on-hold tag now.

[aravindkarnam]

@Umpire2018 Can we close this issue too?

[Umpire2018]

@aravindkarnam Yes, we can close this issue as it pertains to a previous version. If the user encounters any issues after using the latest version, they can open a new issue, and we can observe and address it accordingly.