Alternatives to Queue System for Sequential CV Processing

[EddyDavies]

Problem Analysis

The current issue is that the database trigger sync_ashby_candidate_to_applicant() fires concurrent webhooks via net.http_post() for every high-priority candidate with a resume file handle.

This results in 224+ simultaneous requests to /api/ashby/files, which overwhelms the system and causes 500 errors.

⸻

Alternative Solutions (No Queue Required)

Option 1: Disable Database Webhooks, Use Application-Level Processing (Recommended)
• Remove the direct net.http_post call from sync_ashby_candidate_to_applicant() function
• Keep the sequential processing we already added to the candidates route
• Let the application handle CV processing after candidate sync completes
• ✅ Pros: Simple, no queue system needed, uses existing sequential logic
• ❌ Cons: CVs only processed during sync operations

⸻

Option 2: Database-Level Delay/Throttling
• Add random delays in the database trigger using pg_sleep(random() * 5)
• Stagger webhook firing so they don’t all hit at once
• ✅ Pros: Minimal changes, keeps database-driven architecture
• ❌ Cons: Still concurrent requests, just spread out; unpredictable timing

⸻

Option 3: Batch Processing with Single Webhook
• Collect all candidates needing CV processing in database trigger
• Fire a single webhook with multiple candidate IDs
• Process batch sequentially in the API endpoint
• ✅ Pros: One request instead of 224, maintains webhook architecture
• ❌ Cons: Requires changing webhook payload format and API

⸻

Option 4: Conditional CV Processing
• Only download CVs for candidates with score 50+ (instead of 30+)
• Reduce the number of concurrent requests significantly
• ✅ Pros: Simplest fix, reduces concurrent load
• ❌ Cons: May miss some candidates that need CV processing

⸻

✅ Recommended Solution: Option 1 — Application-Level Processing

Implementation
1. Modify sync_ashby_candidate_to_applicant() — remove lines 435-462 (the direct webhook call).
2. Keep sequential processing in candidates route (already implemented).
3. Store resume file handles in database for later processing.
4. Process CVs sequentially after candidate sync completes.

Benefits
• 🚫 Zero concurrent database webhooks
• ⏱ Uses existing AshbyClient rate limiting (2s, 4s, 8s)
• 🛠 No new infrastructure needed
• 🔍 Simple to implement and debug
• 📊 Predictable sequential processing