docs: clarify tieredProxyUrls behavior for standalone newUrl() calls#3666
Closed
okwn wants to merge 1 commit into
Closed
docs: clarify tieredProxyUrls behavior for standalone newUrl() calls#3666okwn wants to merge 1 commit into
okwn wants to merge 1 commit into
Conversation
The existing warning said 'newUrl calls will not yield the expected results' without explaining why. Now explicitly states: - Tier logic is bypassed when calling newUrl/newProxyInfo directly - Behavior falls back to flat round-robin through ALL tier URLs - Root cause: tier tracking depends on session state and per-domain error tracking that only exist within a running crawler - Provides explicit alternatives: proxyUrls for simple rotation, newUrlFunction for custom logic
Contributor
|
I think @barjin should be able to evaluate this. On the other hand, we removed the feature for v4, so maybe it doesn't deserve much attention. |
barjin
reviewed
May 22, 2026
Member
barjin
left a comment
barjin
left a comment
There was a problem hiding this comment.
Thank you @okwn for your contribution.
As @janbuchar mentions, the feature is scheduled for removal in the upcoming major, so we imo do not need to expand on the documentation here.
For this reason, I'll close this PR now. Thank you anyway!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Expands the warning for
tieredProxyUrlsin the proxy management docs to explain exactly what happens whennewUrl()/newProxyInfo()are called outside a crawler context — and why.Why this matters
The existing warning was vague: "Using this configuration through the
newUrlcalls will not yield the expected results." This leaves users confused about what "unexpected" actually means. The expanded warning now explains:proxyUrls(simple round-robin) ornewUrlFunction(custom logic)Changes
docs/guides/proxy_management.mdx: Expanded the warning block in the "Tiered proxies" section (lines 106-114) to explain the technical mechanism and provide actionable alternativesValidation
npx tsc --noEmitonpackages/core/passes (no code changes)#crawler-integration) are preserved