Skip to content

fix: update monitor.status for private location monitors - #2496

Merged
thibaultleouay merged 18 commits into
openstatusHQ:mainfrom
zapteryx:fix/private-location-monitor-status
Jul 30, 2026
Merged

fix: update monitor.status for private location monitors#2496
thibaultleouay merged 18 commits into
openstatusHQ:mainfrom
zapteryx:fix/private-location-monitor-status

Conversation

@zapteryx

@zapteryx zapteryx commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Private location monitors were not updating the monitor.status field when checks failed/recovered, causing the dashboard to always show them as 'active' even when failing.

The cloud checker updates monitor.status on status transitions, but the private location handler (updateStatusPrivate) only updated the privateLocationMonitorStatus table without touching monitor.status.

This fix adds the same monitor.status update logic to all three status transition cases (error, degraded, active) to match the cloud checker behavior.

Fixes dashboard status display and 'Failing'/'Normal' counts for private location monitors.

Resolves #2495
image
image
image

Review in cubic

Private location monitors were not updating the monitor.status field
when checks failed/recovered, causing the dashboard to always show
them as 'active' even when failing.

The cloud checker updates monitor.status on status transitions, but
the private location handler (updateStatusPrivate) only updated the
privateLocationMonitorStatus table without touching monitor.status.

This fix adds the same monitor.status update logic to all three
status transition cases (error, degraded, active) to match the
cloud checker behavior.

Fixes dashboard status display and 'Failing'/'Normal' counts for
private location monitors.
@vercel

vercel Bot commented Jul 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated (UTC)
openstatus-dashboard Skipped Skipped Jul 30, 2026 8:33am
openstatus-status-page Skipped Skipped Jul 30, 2026 8:33am
openstatus-web Skipped Skipped Jul 30, 2026 8:33am

Request Review

@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 29, 2026 08:07 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 29, 2026 08:07 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 29, 2026 08:07 Inactive

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 1 file

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread apps/workflows/src/checker/private-location.ts Outdated
@zapteryx
zapteryx force-pushed the fix/private-location-monitor-status branch from f4ecc29 to 535341c Compare July 29, 2026 08:28
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 29, 2026 08:28 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 29, 2026 08:28 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 29, 2026 08:28 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 29, 2026 08:29 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 29, 2026 08:29 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 29, 2026 08:29 Inactive
@zapteryx
zapteryx force-pushed the fix/private-location-monitor-status branch from a0a5dea to 689e1de Compare July 29, 2026 08:33
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 29, 2026 08:33 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 29, 2026 08:34 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 29, 2026 08:34 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 29, 2026 08:34 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 29, 2026 08:34 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 29, 2026 08:34 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 29, 2026 14:15 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 29, 2026 14:15 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 29, 2026 14:15 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 29, 2026 14:16 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 29, 2026 14:16 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 29, 2026 14:16 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 03:12 Inactive
V1 (P1): Filter status query to only attached locations
- Prevents detached location stale status from counting toward threshold
- Added inArray filter on attachedLocationIds

V2 (P1): Gate incident creation with threshold check
- Only create incidents when threshold met for private-only monitors
- Prevents premature incident creation (1/3 errors)

V3 (P2): Pass incident ID to notifications
- Capture incident ID in error and recovery cases
- Pass to triggerNotifications for provider context

V4 (P1): Gate incident resolution with threshold check
- Only resolve when threshold met and monitor recovered
- Prevents premature incident closure (1/3 recovery)

Test (P2): Add PRIVATE_ONLY_MONITOR_ID cleanup to afterEach
- Prevents test data leakage between tests
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 04:57 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 04:57 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 04:57 Inactive
- Move incidentId declaration outside switch to fix redeclaration error
- Filter null values from attachedLocationIds for inArray type safety
- Fixes TS2451 (redeclaration) and TS2769 (inArray type mismatch)
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 05:02 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 05:02 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 05:02 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 05:03 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 05:03 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 05:03 Inactive
@thibaultleouay

Copy link
Copy Markdown
Member

@claude review this

@claude

claude Bot commented Jul 30, 2026

Copy link
Copy Markdown

Claude finished @thibaultleouay's task in 3m 31s —— View job


Reviewing PR #2496

  • Gather context (diff, related files, cloud checker for comparison)
  • Review code changes
  • Post review feedback

Good progress overall — the previously flagged P1s (detached locations counting toward the threshold, incidents not gated on threshold, incidentId missing from notifications) all look correctly addressed in apps/workflows/src/checker/private-location.ts, and CI is green (Tests, Check, DX Check, autofix.ci all passing).

One remaining issue I found by tracing through the multi-location threshold logic:

Duplicate notifications when a location's report doesn't change the aggregate monitor status

In all three cases (error at apps/workflows/src/checker/private-location.ts:300, degraded at :342, active at :425), the notification trigger is gated only by:

if (hasCloudRegions || shouldUpdateMonitorStatus) {
  await triggerNotifications({ ... });
}

shouldUpdateMonitorStatus reflects whether the threshold is currently met — it does not check whether this report actually changed monitor.status. Compare this to the DB-update guards a few lines above each block, which correctly add monitor.status !== "error" / "degraded" / "active" (and the incident create/resolve blocks, which are idempotent via findOpenIncident).

Concretely, with 3 attached private locations on a private-only monitor:

  1. Location A → error (1/3, threshold not met, no notify)
  2. Location B → error (2/3, threshold met, monitor flips to error, alert feat: coming soon #1 sent — correct)
  3. Location C → error (3/3, threshold still met, but monitor is already error) — shouldUpdateMonitorStatus is still true, so triggerNotifications fires again, sending a duplicate alert even though nothing changed.

The same pattern applies symmetrically to degraded and active/recovery — a straggling location catching up to an already-reflected aggregate status re-sends the notification.

The fix is to gate the notification the same way the DB update is gated, e.g. capture the transition once before the switch:

const isStatusTransition = monitor.status !== status;
...
if (hasCloudRegions || (shouldUpdateMonitorStatus && isStatusTransition)) {
  await triggerNotifications({ ... });
}

(monitor is fetched once and never locally mutated after the db.update, so monitor.status reliably reflects the pre-update value throughout the handler — same pattern the DB-update guards already rely on.)

This isn't covered by the new tests either — private-location.test.ts:487 ("updates status when threshold met (2/3 locations agree)") stops after the second post; a third post with the same status from PRIVATE_LOCATION_3_ID would currently double-send mockEmailSendAlert.

Fix this →

Minor

  • Since monitor is never locally mutated after the DB update, the repeated monitor.status !== "..." checks scattered across the update / incident blocks could be consolidated into a single isStatusTransition computed once per status — would also make the notification-gating fix above a one-line change.
    (branch: fix/private-location-monitor-status)

zapteryx added 2 commits July 30, 2026 16:19
Resolved conflicts in private-location.ts by combining threshold logic and monitor status updates with upstream's triggeredNotifications tracking for OTel logging.
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 08:19 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 08:19 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 08:20 Inactive
Resolved conflicts by combining upstream's incident handling improvements with monitor status updates.

Changes:

- Kept upstream's incident-utils.ts with atomic resolveIncident helper

- Kept upstream's race condition detection in incident creation

- Added monitor status updates for private-only monitors (error/degraded/active)

- Made notifications conditional: cloud monitors always, private-only when threshold met
@vercel
vercel Bot temporarily deployed to Preview – openstatus-status-page July 30, 2026 08:33 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-dashboard July 30, 2026 08:33 Inactive
@vercel
vercel Bot temporarily deployed to Preview – openstatus-web July 30, 2026 08:33 Inactive
@thibaultleouay

Copy link
Copy Markdown
Member

@claude review this

@claude

claude Bot commented Jul 30, 2026

Copy link
Copy Markdown

Claude finished @thibaultleouay's task in 3m 18s —— View job


Reviewing PR #2496

  • Gather context (diff vs origin/main, related files, cloud checker for comparison)
  • Review latest changes since last review
  • Post review feedback

CI is green (Tests, Check, DX Check, autofix.ci all passing). Note: because main already merged a related PR (#2503, "add private locations incident trigger") and this branch merged main back in, the effective diff of this PR against current main is now just two things: adding the monitor.status DB update for private-only monitors in the error/degraded/active cases, and gating the existing triggerNotifications calls with hasCloudRegions || shouldTriggerIncident.

Still-open issue from the previous review: duplicate notifications on a repeated (non-transitioning) report

This is the same finding from the 08:14 review — it hasn't been addressed by any commit since (the only commits after that review are merges of main, which don't touch this logic).

In all three cases — apps/workflows/src/checker/private-location.ts:330 (error), :393 (degraded), :453 (active) — the notification gate is:

if (hasCloudRegions || shouldTriggerIncident) {
  triggeredNotifications = await triggerNotifications({ ... });
}

shouldTriggerIncident only reflects whether the ≥50% threshold is currently met for status; it says nothing about whether this report actually changed monitor.status. The DB-update block a few lines above each case does add that check (monitor.status !== "error" etc.), but the notification gate doesn't reuse it.

Concretely, with 3 attached private locations on a private-only monitor:

  1. Location A → error (1/3, no notify)
  2. Location B → error (2/3, threshold met, monitor flips to error, alert sent — correct)
  3. Location C → error (3/3, threshold still met, monitor already error) — shouldTriggerIncident is still true, so a duplicate alert fires even though nothing changed.

The cloud checker (apps/workflows/src/checker/index.ts:168,182,201,237) avoids exactly this by breaking out early when monitor.status === status before it ever reaches triggerNotifications. private-location.ts doesn't mirror that.

Fix (mirrors the DB-update guard, which already uses the pre-update monitor.status since monitor is never locally mutated):

const isStatusTransition = monitor.status !== status;
...
if (hasCloudRegions || (shouldTriggerIncident && isStatusTransition)) {
  await triggerNotifications({ ... });
}

Still not covered by tests: private-location.test.ts:487 ("updates status when threshold met (2/3 locations agree)") stops after the 2nd post; a 3rd post from PRIVATE_LOCATION_3_ID with the same error status would currently double-send mockEmailSendAlert.

Fix this →

Minor (unchanged from earlier review, low priority)

  • private-location.ts:185-186: monitor.regions && monitor.regions.trim().length > 0regions is .notNull().default(""), so monitor.regions && is redundant (empty string is already falsy). Harmless, can simplify to monitor.regions.trim().length > 0.

What's correctly handled now

  • Threshold now counts only currently-attached, non-deleted locations (fixes the earlier "detached locations count toward threshold" P1).
  • Incident creation/resolution is idempotent (findOpenIncident / conditional resolvedAt IS NULL update), so repeated reports at a steady-state status don't create duplicate incidents or audit rows — this class of bug doesn't affect incidents, only notifications (see above).
  • incidentId is now correctly threaded into triggerNotifications for all three cases.
  • Monitors with cloud regions (hasCloudRegions) are excluded from the private-only monitor.status write, and their private-location notification behavior is preserved as designed (fires on any private-location transition, independent of threshold — this matches an earlier explicit decision to preserve prior behavior for hybrid monitors, not a bug).
    (branch: fix/private-location-monitor-status)

@thibaultleouay
thibaultleouay merged commit 51fbc63 into openstatusHQ:main Jul 30, 2026
16 checks passed
@zapteryx
zapteryx deleted the fix/private-location-monitor-status branch July 30, 2026 08:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dashboard: private location monitors are always shown as active

2 participants