enhancement(connectors): add built-in HTTP retry with exponential backoff to AbstractConnector #8144

Problem

AbstractConnector has no retry mechanism in its base class. Each connector that makes HTTP requests (web crawler, Notion, audio, etc.) must implement its own retry logic or silently drop data on transient failures (429, 502, 503, 504). This leads to:

• Inconsistent retry behavior across connectors
• Silent data loss when a source returns a rate-limit or transient error
• Connector authors having to reinvent backoff logic every time

Proposed Solution

Add fetch_with_retry() to AbstractConnector in knowledge/connectors/base.py:

async def fetch_with_retry(
    self,
    fetch_fn: Callable[[], Awaitable[T]],
    max_attempts: int = 5,
    backoff_base: float = 2.0,
    retryable_status: tuple[int, ...] = (429, 500, 502, 503, 504),
) -> T:
    ...

Subclasses override should_retry(exc) -> bool and backoff_time(attempt) -> float to customize. The default implementation handles network errors and standard HTTP transient status codes.

Also add two hook methods so individual connectors can opt out or customize:

def should_retry(self, exception: Exception) -> bool: ...
def backoff_time(self, attempt: int) -> float: ...

Files

• autobot-backend/knowledge/connectors/base.py — add fetch_with_retry(), should_retry(), backoff_time()
• autobot-backend/knowledge/connectors/web_crawler.py — migrate existing retry to base class call
• autobot-backend/knowledge/connectors/models.py — add retry_attempts to SyncResult.errors entries for observability

Acceptance Criteria

• AbstractConnector exposes fetch_with_retry() with configurable max attempts and base delay
• Default behavior: exponential backoff on 429/5xx, up to 5 attempts
• should_retry() and backoff_time() are overridable hooks
• web_crawler.py migrated to use base class retry
• Unit tests cover: success on first try, success after retry, exhausted retries, non-retryable errors