Skip to content

feat: Implement Human-in-the-Loop CAPTCHA Handling for Web Research #206

Closed
Closed
feat: Implement Human-in-the-Loop CAPTCHA Handling for Web Research#206
Labels
enhancementNew feature or requestfeature

Description

@mrveiss
mrveiss
opened on Nov 26, 2025

Summary

Implement a human-in-the-loop mechanism for CAPTCHA handling during web research operations.

Background

From Issue #201 TODO analysis:

  • Location: autobot-user-backend/agents/advanced_web_research.py:596
  • Current behavior: Returns False to skip sources requiring CAPTCHA
  • Impact: Some valuable web sources are inaccessible during research

Current Code

async def _handle_captcha(self, url: str) -> bool:
    """Handle CAPTCHA challenges during web research."""
    # TODO: Implement human-in-the-loop mechanism for CAPTCHA handling
    logger.warning(f"⚠️ CAPTCHA detected at {url}, skipping source")
    return False

Requirements

  1. Detection: Reliably detect CAPTCHA challenges on web pages
  2. User Notification: Notify user via WebSocket when CAPTCHA encountered
  3. Manual Resolution: Allow user to solve CAPTCHA in browser
  4. Continuation: Resume research after CAPTCHA solved
  5. Timeout Handling: Implement timeout for user response

Technical Approach

  1. Integrate with existing VNC desktop streaming for visual CAPTCHA display
  2. Add WebSocket endpoint for CAPTCHA notifications
  3. Create Vue component for CAPTCHA interaction UI
  4. Implement async wait mechanism with configurable timeout
  5. Add configuration for auto-skip vs prompt modes

Acceptance Criteria

  • CAPTCHA challenges detected and flagged
  • User notified via UI when CAPTCHA encountered
  • User can view and solve CAPTCHA manually
  • Research continues after CAPTCHA resolution
  • Timeout handling with graceful fallback

Related: Issue #201 (parent tracking issue)

Activity

  1. mrveiss
    added
    enhancementNew feature or request
    feature
    on Nov 26, 2025
  2. mrveiss
    added a commit that references this issue on Nov 26, 2025

    feat(orchestrator): Implement custom success criteria checking (#201)

    ae1c34e

  3. mrveiss commented on Nov 29, 2025

    @mrveiss
    mrveiss
    on Nov 29, 2025
    OwnerAuthor

    Starting work on Issue #206 - Human-in-the-Loop CAPTCHA Handling for Web Research.

  4. mrveiss commented on Nov 29, 2025

    @mrveiss
    mrveiss
    on Nov 29, 2025
    OwnerAuthor

    Implementation Complete ✅

    Implemented the human-in-the-loop CAPTCHA handling system for web research (commit c10731a).

    Components Created:

    1. Backend Service - src/services/captcha_human_loop.py

    • CaptchaHumanLoop class with singleton pattern
    • WebSocket notifications via event_manager.publish()
    • Screenshot capture for CAPTCHA display in UI
    • Async wait mechanism with 120s configurable timeout
    • Resolution statuses: SOLVED, SKIPPED, TIMEOUT, ERROR

    2. REST API - backend/api/captcha.py

    • POST /api/captcha/{id}/resolve - Mark CAPTCHA as solved
    • POST /api/captcha/{id}/skip - Skip and exclude source
    • GET /api/captcha/pending - List pending CAPTCHAs
    • GET /api/captcha/health - Service health check

    3. Updated Web Research Agent - src/agents/advanced_web_research.py

    • Added CAPTCHA type detection (reCAPTCHA, hCaptcha, Cloudflare)
    • Integrated CaptchaHumanLoop service for human fallback
    • Attempts automated solving first, then prompts user

    4. Vue Component - autobot-vue/src/components/research/CaptchaNotification.vue

    • Fixed-position notification card (bottom-right)
    • Screenshot preview of CAPTCHA
    • Countdown timer with visual progress bar
    • "Open VNC Desktop" button for manual resolution
    • "I've Solved It" / "Skip Source" action buttons
    • WebSocket event subscriptions via useGlobalWebSocket

    Acceptance Criteria Status:

    • ✅ CAPTCHA challenges detected and flagged
    • ✅ User notified via UI when CAPTCHA encountered
    • ✅ User can view and solve CAPTCHA manually (via VNC)
    • ✅ Research continues after CAPTCHA resolution
    • ✅ Timeout handling with graceful fallback

    Ready for testing and final review.

  5. mrveiss commented on Nov 29, 2025

    @mrveiss
    mrveiss
    on Nov 29, 2025
    OwnerAuthor

    Enhancement: Automatic CAPTCHA Solving Added (commit a207443)

    Added OCR-based automatic CAPTCHA solving as a first-attempt before human fallback:

    New: src/services/captcha_solver.py

    • Text CAPTCHA solving using pytesseract OCR
    • Math CAPTCHA solving (e.g., "5 + 3 = ?")
    • Multiple preprocessing techniques for better OCR accuracy:
      • Grayscale conversion + thresholding
      • Contrast enhancement
      • Sharpening and denoising
      • Inversion for dark backgrounds
      • Scaling for small CAPTCHAs
    • Confidence scoring: high (>90%), medium (70-90%), low (50-70%)
    • Auto-detection of CAPTCHA type based on dimensions and content

    Updated: src/services/captcha_human_loop.py

    • Hybrid approach: Attempts auto-solve before human intervention
    • Complex CAPTCHAs (reCAPTCHA, hCaptcha, Cloudflare) → immediate human fallback
    • New AUTO_SOLVED status for tracking automatic solutions
    • Optional captcha_input_selector for automatic form filling

    Flow:

    CAPTCHA Detected
     │
     ▼
┌────────────────────┐
│ Is it reCAPTCHA/   │───YES───▶ Human Intervention
│ hCaptcha/CF?       │
└────────────────────┘
     │NO
     ▼
┌────────────────────┐
│ Attempt OCR Solve  │
│ (pytesseract)      │
└────────────────────┘
     │
     ▼
┌────────────────────┐
│ Confidence > 70%?  │───YES───▶ Return AUTO_SOLVED
└────────────────────┘
     │NO
     ▼
Human Intervention (WebSocket notification)

    This reduces manual intervention for simple text/math CAPTCHAs while maintaining reliability for complex challenges.

  6. mrveiss
    added 2 commits that reference this issue on Nov 29, 2025

    feat(captcha): Add human-in-the-loop CAPTCHA handling for web research (

    c10731a

    feat(captcha): Add automatic OCR-based CAPTCHA solver with human fall…

    a207443
  7. mrveiss
    added 3 commits that reference this issue on Dec 31, 2025

    feat(orchestrator): Implement custom success criteria checking (#201)

    102a66f

    feat(captcha): Add human-in-the-loop CAPTCHA handling for web research (

    e8c79a1

    feat(captcha): Add automatic OCR-based CAPTCHA solver with human fall…

    a1d9261

  8. mrveiss commented on Jan 6, 2026

    @mrveiss
    mrveiss
    on Jan 6, 2026
    OwnerAuthor

    Verification Complete ✅

    Verified the full CAPTCHA Human-in-the-Loop implementation. All components are properly integrated and working:

    Acceptance Criteria Verification:

    Criterion Status Implementation
    ✅ CAPTCHA challenges detected and flagged PASS _detect_captcha() in advanced_web_research.py with multiple selectors
    ✅ User notified via UI when CAPTCHA encountered PASS WebSocket captcha_detected event → CaptchaNotification.vue
    ✅ User can view and solve CAPTCHA manually PASS VNC desktop link + screenshot preview in notification
    ✅ Research continues after CAPTCHA resolution PASS mark_captcha_resolved() triggers async event continuation
    ✅ Timeout handling with graceful fallback PASS 120s timeout with captcha_timeout WebSocket event

    Components Verified:

    1. Backend Service (src/services/captcha_human_loop.py)

      • Hybrid approach: OCR auto-solve first, human fallback
      • Thread-safe singleton pattern
      • Proper WebSocket event publishing

    2. OCR Solver (src/services/captcha_solver.py)

      • Text/math CAPTCHA auto-solving with pytesseract
      • Multiple preprocessing techniques for accuracy
      • Graceful degradation when Tesseract unavailable

    3. API Endpoints (backend/api/captcha.py)

      • POST /api/captcha/{id}/resolve - Mark solved
      • POST /api/captcha/{id}/skip - Skip source
      • GET /api/captcha/pending - List pending
      • GET /api/captcha/health - Service health
      • Router registered in feature_routers.py

    4. Vue Component (autobot-vue/src/components/research/CaptchaNotification.vue)

      • Integrated in App.vue
      • WebSocket event subscriptions working
      • Timer, progress bar, VNC button functional

    5. WebSocket Flow

      • event_manager.publish() → WebSocket broadcast → Frontend on() listener
      • Events: captcha_detected, captcha_timeout, captcha_resolved

    Code Review Summary:

    • Overall: GOOD - Well-structured implementation
    • Minor fix applied: Added timestamp field to CaptchaResolutionResponse model
    • Low-priority items noted for future cleanup (late imports, media_type on remaining endpoints)

    Ready for Testing

    The implementation is complete and ready for end-to-end testing with actual CAPTCHA scenarios.

  9. mrveiss
    added a commit that references this issue on Jan 19, 2026

    fix(captcha): Add timestamp to response model and improve error loggi…

    e1a901c
  10. mrveiss
    added a commit that references this issue on Feb 17, 2026

    fix(captcha): move threading imports to top of file (#206)

    2a60fe9

  11. mrveiss commented on Feb 17, 2026

    @mrveiss
    mrveiss
    on Feb 17, 2026
    OwnerAuthor

    Final Verification & Closure ✅

    All implementation from prior commits (e8c79a1, a1d9261, e1a901c) verified complete.

    Linting Fix Applied (commit 2a60fe9):

    • Moved import threading to top of captcha_human_loop.py and captcha_solver.py (E402 violations)
    • All ruff/flake8/isort checks passing cleanly

    Acceptance Criteria - All Met:

    Criterion Status
    CAPTCHA challenges detected and flagged ✅ _detect_captcha() with 8 selectors + keyword matching
    User notified via UI when CAPTCHA encountered ✅ WebSocket captcha_detected → CaptchaNotification.vue
    User can view and solve CAPTCHA manually ✅ VNC link + screenshot preview in notification
    Research continues after CAPTCHA resolution ✅ mark_captcha_resolved() triggers async continuation
    Timeout handling with graceful fallback ✅ 120s timeout with captcha_timeout WebSocket event

    Files Delivered:

    • autobot-user-backend/services/captcha_human_loop.py - Human-in-the-loop service
    • autobot-user-backend/services/captcha_solver.py - OCR auto-solver
    • autobot-user-backend/api/captcha.py - REST API (resolve/skip/pending/health)
    • autobot-user-frontend/src/components/research/CaptchaNotification.vue - Vue UI component
    • Router registered in initialization/router_registry/feature_routers.py
  12. mrveiss
    closed this as completedon Feb 17, 2026
  13. mrveiss
    added a commit that references this issue on Feb 19, 2026

    feat(ansible): add missing system packages to backend and npu-worker …

    b3030db
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestfeature

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions