-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
architectureArchitecture-level changes or design debtArchitecture-level changes or design debtbackendBackend API/data-layer scopeBackend API/data-layer scopeenhancementNew feature or requestNew feature or requesthigh-priorityHigh-impact item to schedule soonHigh-impact item to schedule soonreliabilityAvailability, performance, or operational resilienceAvailability, performance, or operational resilience
Description
Summary
The backend currently performs write-heavy import/correlation work directly in request handlers against SQLite. This creates contention under concurrent usage and degrades API latency/reliability.
Why this matters
- SQLite has a single-writer bottleneck for this workload pattern.
/api/imports/*and correlation routines can hold transactions long enough to impact other requests.- As data volume/concurrency grow, this becomes an availability issue, not just a performance issue.
Repo evidence
- Import write path:
backend/routers/imports.py - Correlation merge path:
backend/services/correlation.py - DB setup/migrations:
backend/database.py
Scope
- Add a background execution path (or equivalent chunked transactional strategy) for heavy ingest/correlation operations.
- Keep request handlers short-lived and return job status for long-running work.
- Add lock/latency instrumentation around write-heavy endpoints.
Acceptance criteria
- Import/correlation no longer run as long blocking DB writes on critical request paths.
- Concurrent import test demonstrates improved p95/p99 response times.
- Operational docs define expected limits and migration trigger points (SQLite -> Postgres).
Out of scope
- Full DB engine migration in this issue.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
architectureArchitecture-level changes or design debtArchitecture-level changes or design debtbackendBackend API/data-layer scopeBackend API/data-layer scopeenhancementNew feature or requestNew feature or requesthigh-priorityHigh-impact item to schedule soonHigh-impact item to schedule soonreliabilityAvailability, performance, or operational resilienceAvailability, performance, or operational resilience