Skip to content

fix: add private locations incident trigger - #2503

Merged
thibaultleouay merged 21 commits into
openstatusHQ:mainfrom
zapteryx:fix/private-location-incident-support-clean
Jul 30, 2026
Merged

fix: add private locations incident trigger#2503
thibaultleouay merged 21 commits into
openstatusHQ:mainfrom
zapteryx:fix/private-location-incident-support-clean

Conversation

@zapteryx

@zapteryx zapteryx commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Resolves #2502

Review in cubic

@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 29, 2026 17:35 Inactive
@vercel

vercel Bot commented Jul 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated (UTC)
openstatus-dashboard Skipped Skipped Jul 30, 2026 8:09am
openstatus-status-page Skipped Skipped Jul 30, 2026 8:09am
openstatus-web Skipped Skipped Jul 30, 2026 8:09am

Request Review

@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 29, 2026 17:35 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 29, 2026 17:35 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 29, 2026 17:36 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 29, 2026 17:36 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 29, 2026 17:36 Inactive

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed against the latest diff

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/workflows/src/checker/private-location.ts
Comment thread apps/workflows/src/checker/private-location.ts
Comment thread apps/workflows/src/checker/private-location.ts
- Add resolveIncident helper matching cloud checker pattern
- Fix degraded case: resolve incidents when transitioning from error
- Fix error case: capture and pass incident ID to notifications
- Fix active case: capture resolved incident and pass to notifications

Resolves violations P1 and P2 (incident data in notifications)
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 29, 2026 17:57 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 29, 2026 17:57 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 29, 2026 17:57 Inactive
- Detect constraint violations from concurrent incident creation
- Fetch existing incident when conflict occurs instead of failing
- Prevents duplicate incident errors in notifications
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 29, 2026 18:02 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 29, 2026 18:02 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 29, 2026 18:02 Inactive

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread apps/workflows/src/checker/private-location.ts Outdated
Comment thread apps/workflows/src/checker/private-location.ts Outdated
V1 (P3): Extract duplicate incident resolution logic
- Created shared incident-utils.ts with findOpenIncident and resolveIncident
- Removed duplicate code from index.ts (cloud checker)
- Removed duplicate code from private-location.ts (private checker)
- Single source of truth for incident resolution

V2 (P2): Fix concurrent recovery race condition
- Changed resolveIncident to use conditional update with resolvedAt IS NULL
- Only publish audit log if update succeeded (.returning() check)
- Prevents duplicate audits and notifications from concurrent recoveries
- Atomic check-and-update pattern
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 05:10 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 05:10 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 05:10 Inactive
Fixed syntax error where template string was corrupted during file creation.
Changed 'id: monitor:,' to 'id: \monitor:\\,'

Fixes CI build failure.
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 05:13 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 05:13 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 05:13 Inactive
Moved 'let incident = null' outside switch to fix redeclaration error.
Switch cases share scope, so declaring in each case caused conflicts.

Fixes oxlint error: Identifier 'incident' has already been declared
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 05:15 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 07:35 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 07:35 Inactive
Resolved conflict in private-location.ts by keeping both threshold logic from our branch and triggeredNotifications declaration from upstream.
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 07:37 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 07:37 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 07:37 Inactive
P1 openstatusHQ#1: Removed dead code condition in degraded case that checked monitor.status === error. Private location handler never writes monitor.status, so this condition was always false, preventing incident resolution on degraded transitions.

P1 openstatusHQ#2: Replaced inline incident resolution in active case with shared resolveIncident() helper (matching degraded case). The inline code only resolved one incident and lacked atomic IS NULL check, reintroducing race conditions and multiple-incident bugs.

Benefits: Incidents now resolve correctly on degraded transitions, all open incidents resolved (not just first), atomic resolution prevents duplicate notifications, consistent code between active/degraded cases, eliminated 41 lines of duplicate code.
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 07:44 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 07:44 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 07:44 Inactive

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread apps/workflows/src/checker/private-location.ts Outdated
Added try-catch around resolveIncident() calls in degraded and active cases to prevent transient failures from aborting recovery handling.

Without error handling, if resolveIncident() throws after status is upserted, the handler aborts and retries exit early at status === priorStatus check, leaving incident open and recovery notifications unsent.

Now logs warnings but continues with notifications, ensuring recovery completes even if incident resolution fails temporarily.
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 07:52 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 07:52 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 07:52 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 07:56 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 07:56 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 07:56 Inactive

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread apps/workflows/src/checker/private-location.ts
Changed catch blocks in degraded and active cases to extract error message explicitly (error_message: error instanceof Error ? error.message : String(error)) instead of passing raw error object.

Raw Error objects have non-enumerable properties (message, stack) that don't serialize properly in structured logging. The extracted pattern ensures error messages are always captured, matching the existing convention at line 422.
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 08:09 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 08:09 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 08:09 Inactive
@thibaultleouay

Copy link
Copy Markdown
Member

@claude review this

@claude

claude Bot commented Jul 30, 2026

Copy link
Copy Markdown

Claude finished @thibaultleouay's task in 2m 10s —— View job


Review: private location incident triggering

  • Gather context (diff vs origin/main, prior review rounds)
  • Read changed files in full (incident-utils.ts, private-location.ts, index.ts)
  • Review for bugs, race conditions, style/consistency
  • Post review feedback

Re-reviewed the current state (a833b41f) against origin/main. Good news: the two P1 blockers from my last pass on this PR are now fixed.

Resolved since last review

  • The degraded and active branches in private-location.ts (lines 326–344, 370–385) both now call the shared resolveIncident() unconditionally when shouldTriggerIncident is true, instead of gating on monitor.status === "error" (a field this file never writes) or duplicating logic with a manual findOpenIncident + update. Both paths are consistent now.
  • resolveIncident() in incident-utils.ts does a single atomic bulk UPDATE ... WHERE id IN (...) AND resolved_at IS NULL RETURNING *, so it can't leave a second open incident behind and can't double-resolve under concurrent reports.
  • index.ts now imports the same findOpenIncident/resolveIncident from incident-utils.ts instead of keeping a second near-identical copy — addresses cubic's duplication comment.
  • Error logging in the new catch blocks is now consistent with the rest of the file (error_message: error instanceof Error ? error.message : String(error)).

Still open — test coverage gap (P2)

apps/workflows/src/checker/private-location.test.ts:97TEST_MONITOR_ID is created with regions: "ams", a non-empty cloud region. In private-location.ts, hasCloudRegions is derived from that field and shouldTriggerIncident = !hasCloudRegions && (...). With this fixture, shouldTriggerIncident is always false, so none of the incident create/resolve code this PR adds (private-location.ts:237-299 error, :325-343 degraded, :368-385 active) is exercised by any test in this file — including "recovery after error" (:282) and "degraded report" (:304), which only assert on notification calls, not on incidentTable rows. This is exactly the gap that let the earlier monitor.status === "error" dead-code bug ship unnoticed for several review rounds.

Suggest adding a private-only monitor fixture (empty regions) and asserting incidentTable rows across: error → creates incident, degraded-with-majority → resolves it, active → resolves it.
Fix this →

Minor, non-blocking (P3)

packages/db/src/schema/incidents/incident.ts:65unique(monitorId, startedAt) only collides if two concurrent inserts share the exact same cronTimestamp; different-second reports can still create two open incidents for one monitor. This is mitigated (not eliminated) by resolveIncident() now resolving all open rows rather than assuming one, so it's not a correctness bug in practice, just a residual gap noted in earlier reviews. Worth a follow-up (partial unique index on monitorId WHERE resolved_at IS NULL) if duplicate incident.created audit rows/notifications on the same outage matter.


Summary: The core incident create/resolve logic is now correct and consistent between the degraded and active paths, and the atomic bulk resolve in incident-utils.ts handles the race conditions cubic flagged earlier. The one thing I'd still want before merge is test coverage for the actual feature (private-only monitor fixture), since right now the whole incident path is untested and that's precisely how the earlier P1 regression slipped through several review rounds.

@thibaultleouay
thibaultleouay merged commit 0a69aaf into openstatusHQ:main Jul 30, 2026
21 of 23 checks passed
@zapteryx
zapteryx deleted the fix/private-location-incident-support-clean branch July 30, 2026 08:32
thibaultleouay pushed a commit that referenced this pull request Aug 1, 2026
* docs: update private location documentation to reflect fixed limitations

Both previously documented limitations have been resolved:

1. Automatic incidents and notifications (fixed in #2503, July 30 2026)
   - Private location monitors now trigger incidents on failures
   - Includes intelligent threshold logic for multi-location setups
   - Notifications fire same as public locations

2. Status page display (fixed in 24b0e99, July 31 2026)
   - Private location monitors now appear on public status pages
   - Includes proper status tracking and uptime graphs
   - Full feature parity with public locations

Changes:
- Updated concept page comparison table (Yes for both features)
- Changed 'Current limitations' to 'Recent improvements'
- Added feature parity note to reference page

Refs: commits 0a69aaf, 51fbc63, 24b0e99

* docs: remove recent improvements

* docs: clarify monitors field returns full objects not IDs

Updated API surface documentation to reflect that the monitors field
returns full monitor objects rather than just monitor IDs.

* docs: correct token field API documentation

The token field is returned by all API endpoints including list.
Removed incorrect claim that 'list responses omit it'.

Verified by inspecting listPrivateLocations service which spreads
all row fields (...row) without filtering the token.

* docs: correct private location API field descriptions

Fix two inaccuracies in the API reference:

1. Token field: List responses deliberately omit the token for security.
   Changed from 'returned by all API endpoints' to 'Returned by create,
   get, and update endpoints; list responses omit it for security.'

2. Monitors field: API returns monitorIds (array of ID strings), not
   full monitor objects. Changed field name from 'monitors' to 'monitorIds'
   and description to match actual API behavior.

These corrections align the docs with the actual implementation in
apps/server/src/routes/rpc/handlers/private-location/converters.ts

* fix: apply suggestion from @cubic-dev-ai[bot]

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dashboard: Private locations do not create incidents

2 participants