Skip to content

MemoryPressureEventsCollector: arithmetic overflow after #2751's millisecond-precision fix #2755

Closed

Description

@erikdarlingdata

What's happening

Found via a live production health check while installing tonight's nightly (3.6.0-nightly.20260901) on the use2 monitoring host. memory_pressure_events went from 447/447 SUCCESS in the hour before the restart to 100% ERROR immediately after, across many servers, every cycle:

Arithmetic overflow error converting expression to data type int.

Rolled the box back to the prior build immediately (confirmed pre-upgrade DLL timestamp restored, service healthy) — this build must not go to any other box until fixed.

Root cause (from the #2751 diff)

MemoryPressureEventsCollector was changed from:

DATEADD(SECOND, -((@ms_ticks - t.timestamp) / 1000), SYSDATETIME())

to:

DATEADD(MILLISECOND, -(@ms_ticks - t.timestamp), SYSDATETIME())

to fix the sub-second truncation that caused #2749's duplicate-row bug. But DATEADD(MILLISECOND, ...)'s number argument is an int, capped at ±2,147,483,647 — about ±24.8 days worth of milliseconds. The old /1000 division kept the value safely inside int range (24.8 days worth of seconds is ~68 years). Removing it without also widening the arithmetic (or the parameter type passed to DATEADD) means any (@ms_ticks - t.timestamp) delta larger than ~24.8 days in milliseconds now overflows.

CpuUtilizationCollector got the identical millisecond-precision fix in the same PR — needs checking for the same exposure (its own health check on use2 showed 0 errors and 0 duplicate rows in the ~14-minute post-restart window, but that's not long enough to rule this out if its @ms_ticks/timestamp values can have the same wide range).

Fix direction

Cast to a wider type before the DATEADD call, e.g. compute the millisecond delta as a bigint, or use DATEADD(MILLISECOND, ...) chained with a DATEADD(SECOND, ...) remainder split (seconds via /1000 on the bigint, then add the sub-second MILLISECOND remainder via a second DATEADD call) so no single arithmetic step needs to fit a >24.8-day span into an int. Needs a real fix, not just widening the cap — verify against actual ring-buffer retention/tick-count ranges on a live box before considering it closed, the way #2751 itself was verified by reproduction rather than by reading the error text alone.

Verified

  • Pre-restart (11:57:59–12:57:59 UTC): memory_pressure_events 447/447 SUCCESS.
  • Post-restart (12:57:59–13:12 UTC): 24/24 ERROR, same message, across multiple server_ids.
  • Rollback to the pre-nightly build (DLL timestamp 2026-09-01 06:23:58) restores the service to Running; confirming SUCCESS resumes post-rollback separately.

This blocks the rest of tonight's fleet install (the use1 monitoring host, the PostgreSQL monitoring host were not touched, still on the prior build).

Activity

  1. erikdarlingdata commented on Sep 1, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed and merged in PR #2756 (d82c0a62), verified live in production during tonight's fleet install — closing manually for the same dev-vs-main reason as above.

  2. added 2 commits that reference this issue on Sep 10, 2026
    7d04ad3
    a98c206
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions