Duplicate incidents + stuck state when komodo runs more than one instance (monitor isn't singleton) #64

What we observed

A single Fly machine stop (fluentcards/long-wildflower-1566) produced two incidents, both resolved:

• incident_opened cf752b16 from app machine d8d9231a674028 → "Restarted … came back healthy" (the one approved in the UI)
• incident_opened 9d783666 from app machine 287423da7e23d8 → "Recovered on its own before I acted"

And the Now card's diagnosis sat on "pending" and never filled in.

Root cause

komodo runs with Fly's default 2 machines (auto_start_machines = true), and ServiceMonitor is a per-process singleton — so each instance runs its own poll loop, detects the same outage, and opens its own incident. Worse for the UI: active-incident state lives in that per-process object, and the fly-proxy routes requests to either instance, so the card you're viewing (owned by instance A) can get its "what's the diagnosis" answer from instance B, which never diagnosed it → permanent "pending."

In auto_fix this is more than cosmetic: both instances would have issued a Machines-API restart for the same machine.

Fix

Immediate (mitigation): run komodo as a single instance — fly scale count 1 -a komodo. min_machines_running = 1 keeps the sole machine up so the monitor runs continuously. Confirm auto_stop_machines never pauses the only machine (the monitor must run 24/7, not just under HTTP traffic).

Proper fix (so HA is even possible):

• Coordinate the monitor across instances — a Postgres advisory lock / leader election so exactly one instance runs the poll loop and opens incidents.
• Serve active-incident state from the shared DB, not per-process memory, so the UI is consistent regardless of which instance answers.
• Guard incident creation (unique partial index on "one open incident per service") so a race can't double-open.

Until then, single-instance is the correct topology for this controller.