Skip to content

Mitigate SQLite write contention on import/correlation paths #21

@BadgerOps

Description

@BadgerOps

Summary

The backend currently performs write-heavy import/correlation work directly in request handlers against SQLite. This creates contention under concurrent usage and degrades API latency/reliability.

Why this matters

  • SQLite has a single-writer bottleneck for this workload pattern.
  • /api/imports/* and correlation routines can hold transactions long enough to impact other requests.
  • As data volume/concurrency grow, this becomes an availability issue, not just a performance issue.

Repo evidence

  • Import write path: backend/routers/imports.py
  • Correlation merge path: backend/services/correlation.py
  • DB setup/migrations: backend/database.py

Scope

  • Add a background execution path (or equivalent chunked transactional strategy) for heavy ingest/correlation operations.
  • Keep request handlers short-lived and return job status for long-running work.
  • Add lock/latency instrumentation around write-heavy endpoints.

Acceptance criteria

  • Import/correlation no longer run as long blocking DB writes on critical request paths.
  • Concurrent import test demonstrates improved p95/p99 response times.
  • Operational docs define expected limits and migration trigger points (SQLite -> Postgres).

Out of scope

  • Full DB engine migration in this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    architectureArchitecture-level changes or design debtbackendBackend API/data-layer scopeenhancementNew feature or requesthigh-priorityHigh-impact item to schedule soonreliabilityAvailability, performance, or operational resilience

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions