Skip to content

Fix duplicate Nightscout treatment uploads by serializing per-pipeline runs - #1290

Merged
dnzxy merged 3 commits into
nightscout:devfrom
bjorkert:fix-nightscout-duplicate-uploads
Jul 15, 2026
Merged

Fix duplicate Nightscout treatment uploads by serializing per-pipeline runs#1290
dnzxy merged 3 commits into
nightscout:devfrom
bjorkert:fix-nightscout-duplicate-uploads

Conversation

@bjorkert

Copy link
Copy Markdown
Member

Problem

Trio occasionally uploads the same treatment to Nightscout twice — most visibly Exercise entries for overrides enacted via Shortcuts or remote commands. Reproduced with an APNS remote command producing two documents with millisecond-identical created_at and consecutive Mongo _ids, i.e. two near-simultaneous POSTs:

  • Enacting an override via App Intents / remote control triggers uploads twice: the Core Data save fires the upload pipeline, and .willUpdateOverrideConfiguration fires a direct, unthrottled upload. Both fetch the same rows still marked isUploadedToNS == false before either has marked them uploaded.
  • Nightscout's treatment upsert is a non-atomic find-then-insert (no unique index), so concurrent identical POSTs both insert. Sequential re-posts are deduped server-side; only concurrent ones slip through. (API v3 has the same race, so switching API versions wouldn't help.)
  • The 2s throttle didn't prevent this: it only spaced out task starts within one trigger path, never waited for the previous upload to finish, and with latest: false silently dropped triggers arriving inside its window. Carb edits had a third, entirely unthrottled path (HistoryStateModel calls uploadCarbs() directly).

Solution

Replace the per-pipeline subject/throttle machinery with NightscoutUploadSerializer, an actor that coalesces and serializes upload runs per pipeline:

  • An idle pipeline starts a run immediately; while a run is in flight, exactly one follow-up is queued and further requests coalesce into it.
  • No two runs of a pipeline ever overlap: a follow-up re-fetches only after the previous run marked its rows uploaded, so it finds nothing left to send. Queued runs double as retry attempts after network failures.
  • The upload routine is provided once at init; request(_:)/run(_:) take only the pipeline, so every entry point (Core Data triggers, App Intents, remote control, direct calls) is covered with no call-site changes.
  • App Intents semantics are preserved: uploadOverrides()/uploadTempTargets() are awaitable, so .didUpdateOverrideConfiguration still posts only after the upload attempt finished.

Benefits

  • No more duplicate treatments, regardless of how many trigger paths fire for the same change.
  • Uploads start immediately — the fixed 2-second throttle delay is gone.
  • No dropped triggers (the old throttle discarded a second change arriving within 2 s; it stayed unuploaded until the next unrelated trigger).
  • uploadDeviceStatus() no longer throws; its only caller was the deleted pipeline dispatch, and errors are logged in performUpload(for:).

Unit tests cover the serializer: no overlap under concurrency, queued-run ordering, burst coalescing into a single follow-up, completion semantics, and pipeline independence.

…e runs

Concurrent upload runs of the same pipeline could both fetch the same rows
still marked isUploadedToNS == false and POST them twice. Nightscout's upsert
is a non-atomic find-then-insert, so near-simultaneous identical POSTs create
duplicate documents - most visibly Exercise entries for overrides enacted via
Shortcuts or remote commands, which fire both the notification path and the
Core Data path at once.

Replace the per-pipeline subject/throttle machinery with
NightscoutUploadSerializer, an actor that starts a run immediately when the
pipeline is idle and otherwise queues exactly one follow-up run; further
requests coalesce into it. No two runs of a pipeline can overlap, no request
is ever dropped (the old 2s throttle with latest: false silently discarded
triggers arriving inside its window), and there is no fixed delay anymore.
The awaitable upload*() entry points keep their semantics, so App Intents
still post their completion notification only after their upload finished.

uploadDeviceStatus() no longer throws; its only caller was the deleted
pipeline dispatch, and errors are logged in performUpload(for:).
MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 13, 2026
An actor that runs enqueued upload operations strictly one at a time, in
order: each operation starts only after the previous one has fully
completed, including its network round-trip. Serialization is global
across pipelines because concurrent uploads -- even to different
endpoints -- can each trigger a Tidepool session refresh that reuses the
single-use rotating refresh token, corrupting the session (nightscout#1235).

Design points, several adopted from the Nightscout upload serializer in
nightscout#1290:

- Requests for a pipeline that gathers its pending data at run time
  coalesce: while a run is enqueued but not started, further requests
  are dropped (the waiting run picks up their data when it fetches),
  and a request arriving mid-run chains exactly one follow-up, which
  doubles as a retry after a failed run. A trigger burst therefore
  costs at most one waiting run per pipeline. Delete pipelines carry
  unique payloads and never coalesce.
- Chain tasks run at .utility QoS: the inherited .background QoS is
  not scheduled during background execution windows, freezing uploads
  until the app is foregrounded.
- A wall-clock watchdog (recoverIfWedged) abandons the chain when its
  head operation exceeds 10 minutes, e.g. because the app was
  suspended mid-request and the completion was lost. Wall clock is
  deliberate: it keeps advancing while the process is suspended.
  Abandoning bumps a generation counter; operations re-check
  isCurrent(generation) between network calls so an orphaned chain
  cannot fire requests alongside its replacement.
- awaitUpload bridges TidepoolKit's completion-based calls into
  async/await with a 120s timeout backstop: a call that never invokes
  its completion resolves to a logged .timedOut failure instead of
  suspending the chain forever. SingleResumer guards the continuation
  so whichever of completion/timeout fires second is a no-op instead
  of a double-resume crash.

Not wired up yet; groundwork for serializing all Tidepool uploads
(nightscout#1235).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 13, 2026
…1235)

Trio fired Tidepool uploads concurrently and unserialized: each CGM
reading triggers uploadGlucose() from BOTH glucoseStorage.updatePublisher
AND the GlucoseStored Core Data notification, alongside independent
dose/carbs triggers. Every trigger spawned a Task into the shared
TidepoolKit `TAPI` actor. When the OAuth access token expired, multiple
in-flight requests each passed session.shouldRefresh() and called
refreshSession() in parallel, reusing the single-use rotating refresh
token. That race corrupted the session; subsequent upload completions
never fired, so uploads stopped silently with nothing logged. Nightscout
was unaffected (separate manager), and Loop avoids the bug by funneling
uploads through one serialized RemoteDataServicesManager cycle.

Confirmed from the issue logs: last Tidepool success 01:50:06, zero
Tidepool errors after, app running continuously to 07:33; at 07:31 a
dose reached the "TIDEPOOL DOSE ENTRIES" log but its upload completion
never returned.

Route every upload (glucose, carbs, dose, pump events, settings,
deletes) through TidepoolUploadSerializer so only one network request is
in flight at a time. This guarantees at most one session refresh,
eliminating the concurrent-refresh race, and a wedged request resolves
to a logged .timedOut failure instead of silently stalling every later
upload forever.

Each pipeline's operation now fetches its pending rows (or rebuilds its
payload) inside the serialized run rather than at trigger time, the
pattern used by the Nightscout serializer in nightscout#1290. Two
triggers firing close together -- e.g. the twin glucose triggers above --
previously both snapshotted the same unmarked rows and uploaded them
twice; now the second request coalesces into a follow-up that re-fetches
after the first run marked its rows, finding only genuinely new data.
Chunked uploads also now mark only the chunk that succeeded as uploaded,
instead of marking the full batch after the first chunk. In the dose
pipeline, the backgroundContext.perform block just builds the upload
payloads and returns them; the network calls happen outside the Core
Data context, inside the serialized run.

The serializer watchdog is checked at each enqueue and on app foreground
(didBecomeActive) -- the reliable recovery points while the pipeline
itself is stalled.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 13, 2026
An actor that runs enqueued upload operations strictly one at a time, in
order: each operation starts only after the previous one has fully
completed, including its network round-trip. Serialization is global
across pipelines because concurrent uploads -- even to different
endpoints -- can each trigger a Tidepool session refresh that reuses the
single-use rotating refresh token, corrupting the session (nightscout#1235).

Design points, several adopted from the Nightscout upload serializer in
nightscout#1290:

- Requests for a pipeline that gathers its pending data at run time
  coalesce: while a run is enqueued but not started, further requests
  are dropped (the waiting run picks up their data when it fetches),
  and a request arriving mid-run chains exactly one follow-up, which
  doubles as a retry after a failed run. A trigger burst therefore
  costs at most one waiting run per pipeline. Delete pipelines carry
  unique payloads and never coalesce.
- Chain tasks run at .utility QoS: the inherited .background QoS is
  not scheduled during background execution windows, freezing uploads
  until the app is foregrounded.
- A wall-clock watchdog (recoverIfWedged) abandons the chain when its
  head operation exceeds 10 minutes, e.g. because the app was
  suspended mid-request and the completion was lost. Wall clock is
  deliberate: it keeps advancing while the process is suspended.
  Abandoning bumps a generation counter; operations re-check
  isCurrent(generation) between network calls so an orphaned chain
  cannot fire requests alongside its replacement.
- awaitUpload bridges TidepoolKit's completion-based calls into
  async/await with a 120s timeout backstop: a call that never invokes
  its completion resolves to a logged .timedOut failure instead of
  suspending the chain forever. SingleResumer guards the continuation
  so whichever of completion/timeout fires second is a no-op instead
  of a double-resume crash.

Not wired up yet; groundwork for serializing all Tidepool uploads
(nightscout#1235).
MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 13, 2026
…1235)

Trio fired Tidepool uploads concurrently and unserialized: each CGM
reading triggers uploadGlucose() from BOTH glucoseStorage.updatePublisher
AND the GlucoseStored Core Data notification, alongside independent
dose/carbs triggers. Every trigger spawned a Task into the shared
TidepoolKit `TAPI` actor. When the OAuth access token expired, multiple
in-flight requests each passed session.shouldRefresh() and called
refreshSession() in parallel, reusing the single-use rotating refresh
token. That race corrupted the session; subsequent upload completions
never fired, so uploads stopped silently with nothing logged. Nightscout
was unaffected (separate manager), and Loop avoids the bug by funneling
uploads through one serialized RemoteDataServicesManager cycle.

Confirmed from the issue logs: last Tidepool success 01:50:06, zero
Tidepool errors after, app running continuously to 07:33; at 07:31 a
dose reached the "TIDEPOOL DOSE ENTRIES" log but its upload completion
never returned.

Route every upload (glucose, carbs, dose, pump events, settings,
deletes) through TidepoolUploadSerializer so only one network request is
in flight at a time. This guarantees at most one session refresh,
eliminating the concurrent-refresh race, and a wedged request resolves
to a logged .timedOut failure instead of silently stalling every later
upload forever.

Each pipeline's operation now fetches its pending rows (or rebuilds its
payload) inside the serialized run rather than at trigger time, the
pattern used by the Nightscout serializer in nightscout#1290. Two
triggers firing close together -- e.g. the twin glucose triggers above --
previously both snapshotted the same unmarked rows and uploaded them
twice; now the second request coalesces into a follow-up that re-fetches
after the first run marked its rows, finding only genuinely new data.
Chunked uploads also now mark only the chunk that succeeded as uploaded,
instead of marking the full batch after the first chunk. In the dose
pipeline, the backgroundContext.perform block just builds the upload
payloads and returns them; the network calls happen outside the Core
Data context, inside the serialized run.

The serializer watchdog is checked at each enqueue and on app foreground
(didBecomeActive) -- the reliable recovery points while the pipeline
itself is stalled.
MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 13, 2026
An actor that runs enqueued upload operations strictly one at a time, in
order: each operation starts only after the previous one has fully
completed, including its network round-trip. Serialization is global
across pipelines because concurrent uploads -- even to different
endpoints -- can each trigger a Tidepool session refresh that reuses the
single-use rotating refresh token, corrupting the session (nightscout#1235).

Design points, several adopted from the Nightscout upload serializer in
nightscout#1290:

- Requests for a pipeline that gathers its pending data at run time
  coalesce: while a run is enqueued but not started, further requests
  are dropped (the waiting run picks up their data when it fetches),
  and a request arriving mid-run chains exactly one follow-up, which
  doubles as a retry after a failed run. A trigger burst therefore
  costs at most one waiting run per pipeline. Delete pipelines carry
  unique payloads and never coalesce.
- Chain tasks run at .utility QoS: the inherited .background QoS is
  not scheduled during background execution windows, freezing uploads
  until the app is foregrounded.
- A wall-clock watchdog (recoverIfWedged) abandons the chain when its
  head operation exceeds 10 minutes, e.g. because the app was
  suspended mid-request and the completion was lost. Wall clock is
  deliberate: it keeps advancing while the process is suspended.
  Abandoning bumps a generation counter; operations re-check
  isCurrent(generation) between network calls so an orphaned chain
  cannot fire requests alongside its replacement.
- awaitUpload bridges TidepoolKit's completion-based calls into
  async/await with a 120s timeout backstop: a call that never invokes
  its completion resolves to a logged .timedOut failure instead of
  suspending the chain forever. SingleResumer guards the continuation
  so whichever of completion/timeout fires second is a no-op instead
  of a double-resume crash.

Not wired up yet; groundwork for serializing all Tidepool uploads
(nightscout#1235).
MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 13, 2026
…1235)

Trio fired Tidepool uploads concurrently and unserialized: each CGM
reading triggers uploadGlucose() from BOTH glucoseStorage.updatePublisher
AND the GlucoseStored Core Data notification, alongside independent
dose/carbs triggers. Every trigger spawned a Task into the shared
TidepoolKit `TAPI` actor. When the OAuth access token expired, multiple
in-flight requests each passed session.shouldRefresh() and called
refreshSession() in parallel, reusing the single-use rotating refresh
token. That race corrupted the session; subsequent upload completions
never fired, so uploads stopped silently with nothing logged. Nightscout
was unaffected (separate manager), and Loop avoids the bug by funneling
uploads through one serialized RemoteDataServicesManager cycle.

Confirmed from the issue logs: last Tidepool success 01:50:06, zero
Tidepool errors after, app running continuously to 07:33; at 07:31 a
dose reached the "TIDEPOOL DOSE ENTRIES" log but its upload completion
never returned.

Route every upload (glucose, carbs, dose, pump events, settings,
deletes) through TidepoolUploadSerializer so only one network request is
in flight at a time. This guarantees at most one session refresh,
eliminating the concurrent-refresh race, and a wedged request resolves
to a logged .timedOut failure instead of silently stalling every later
upload forever.

Each pipeline's operation now fetches its pending rows (or rebuilds its
payload) inside the serialized run rather than at trigger time, the
pattern used by the Nightscout serializer in nightscout#1290. Two
triggers firing close together -- e.g. the twin glucose triggers above --
previously both snapshotted the same unmarked rows and uploaded them
twice; now the second request coalesces into a follow-up that re-fetches
after the first run marked its rows, finding only genuinely new data.
Chunked uploads also now mark only the chunk that succeeded as uploaded,
instead of marking the full batch after the first chunk. In the dose
pipeline, the backgroundContext.perform block just builds the upload
payloads and returns them; the network calls happen outside the Core
Data context, inside the serialized run.

The serializer watchdog is checked at each enqueue and on app foreground
(didBecomeActive) -- the reliable recovery points while the pipeline
itself is stalled.
…cate-uploads

# Conflicts:
#	Trio/Sources/Services/Network/Nightscout/BaseNightscoutManager+Subscribers.swift
#	Trio/Sources/Services/Network/Nightscout/NightscoutManager.swift

@bjornoleh bjornoleh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving based on live testing for a few days. No more duplicate Exercise events from LF remote overrides.

Thx @bjorkert for fixing this!

dnzxy
dnzxy previously approved these changes Jul 14, 2026
@dnzxy
dnzxy dismissed their stale review July 14, 2026 09:35

Need to type up a longer comment. I think we need to potentially bring changes into this with the massive core data rewrite around handling of the NSFetchedResultsController (FRC), as I think it may potentially cause empty upload runs.

MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 14, 2026
An actor that runs enqueued upload operations strictly one at a time, in
order: each operation starts only after the previous one has fully
completed, including its network round-trip. Serialization is global
across pipelines because concurrent uploads -- even to different
endpoints -- can each trigger a Tidepool session refresh that reuses the
single-use rotating refresh token, corrupting the session (nightscout#1235).

Design points, several adopted from the Nightscout upload serializer in
nightscout#1290:

- Requests for a pipeline that gathers its pending data at run time
  coalesce: while a run is enqueued but not started, further requests
  are dropped (the waiting run picks up their data when it fetches),
  and a request arriving mid-run chains exactly one follow-up, which
  doubles as a retry after a failed run. A trigger burst therefore
  costs at most one waiting run per pipeline. Delete pipelines carry
  unique payloads and never coalesce.
- Chain tasks run at .utility QoS: the inherited .background QoS is
  not scheduled during background execution windows, freezing uploads
  until the app is foregrounded.
- A wall-clock watchdog (recoverIfWedged) abandons the chain when its
  head operation exceeds 10 minutes, e.g. because the app was
  suspended mid-request and the completion was lost. Wall clock is
  deliberate: it keeps advancing while the process is suspended.
  Abandoning bumps a generation counter; operations re-check
  isCurrent(generation) between network calls so an orphaned chain
  cannot fire requests alongside its replacement.
- awaitUpload bridges TidepoolKit's completion-based calls into
  async/await with a 120s timeout backstop: a call that never invokes
  its completion resolves to a logged .timedOut failure instead of
  suspending the chain forever. SingleResumer guards the continuation
  so whichever of completion/timeout fires second is a no-op instead
  of a double-resume crash.

Not wired up yet; groundwork for serializing all Tidepool uploads
(nightscout#1235).
MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 14, 2026
…1235)

Trio fired Tidepool uploads concurrently and unserialized: each CGM
reading triggers uploadGlucose() from BOTH glucoseStorage.updatePublisher
AND the GlucoseStored Core Data notification, alongside independent
dose/carbs triggers. Every trigger spawned a Task into the shared
TidepoolKit `TAPI` actor. When the OAuth access token expired, multiple
in-flight requests each passed session.shouldRefresh() and called
refreshSession() in parallel, reusing the single-use rotating refresh
token. That race corrupted the session; subsequent upload completions
never fired, so uploads stopped silently with nothing logged. Nightscout
was unaffected (separate manager), and Loop avoids the bug by funneling
uploads through one serialized RemoteDataServicesManager cycle.

Confirmed from the issue logs: last Tidepool success 01:50:06, zero
Tidepool errors after, app running continuously to 07:33; at 07:31 a
dose reached the "TIDEPOOL DOSE ENTRIES" log but its upload completion
never returned.

Route every upload (glucose, carbs, dose, pump events, settings,
deletes) through TidepoolUploadSerializer so only one network request is
in flight at a time. This guarantees at most one session refresh,
eliminating the concurrent-refresh race, and a wedged request resolves
to a logged .timedOut failure instead of silently stalling every later
upload forever.

Each pipeline's operation now fetches its pending rows (or rebuilds its
payload) inside the serialized run rather than at trigger time, the
pattern used by the Nightscout serializer in nightscout#1290. Two
triggers firing close together -- e.g. the twin glucose triggers above --
previously both snapshotted the same unmarked rows and uploaded them
twice; now the second request coalesces into a follow-up that re-fetches
after the first run marked its rows, finding only genuinely new data.
Chunked uploads also now mark only the chunk that succeeded as uploaded,
instead of marking the full batch after the first chunk. In the dose
pipeline, the backgroundContext.perform block just builds the upload
payloads and returns them; the network calls happen outside the Core
Data context, inside the serialized run.

The serializer watchdog is checked at each enqueue and on app foreground
(didBecomeActive) -- the reliable recovery points while the pipeline
itself is stalled.
MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 14, 2026
An actor that runs enqueued upload operations strictly one at a time, in
order: each operation starts only after the previous one has fully
completed, including its network round-trip. Serialization is global
across pipelines because concurrent uploads -- even to different
endpoints -- can each trigger a Tidepool session refresh that reuses the
single-use rotating refresh token, corrupting the session (nightscout#1235).

Design points, several adopted from the Nightscout upload serializer in
nightscout#1290:

- Requests for a pipeline that gathers its pending data at run time
  coalesce: while a run is enqueued but not started, further requests
  are dropped (the waiting run picks up their data when it fetches),
  and a request arriving mid-run chains exactly one follow-up, which
  doubles as a retry after a failed run. A trigger burst therefore
  costs at most one waiting run per pipeline. Delete pipelines carry
  unique payloads and never coalesce.
- Chain tasks run at .utility QoS: the inherited .background QoS is
  not scheduled during background execution windows, freezing uploads
  until the app is foregrounded.
- A wall-clock watchdog (recoverIfWedged) abandons the chain when its
  head operation exceeds 10 minutes, e.g. because the app was
  suspended mid-request and the completion was lost. Wall clock is
  deliberate: it keeps advancing while the process is suspended.
  Abandoning bumps a generation counter; operations re-check
  isCurrent(generation) between network calls so an orphaned chain
  cannot fire requests alongside its replacement.
- awaitUpload bridges TidepoolKit's completion-based calls into
  async/await with a 120s timeout backstop: a call that never invokes
  its completion resolves to a logged .timedOut failure instead of
  suspending the chain forever. SingleResumer guards the continuation
  so whichever of completion/timeout fires second is a no-op instead
  of a double-resume crash.

Not wired up yet; groundwork for serializing all Tidepool uploads
(nightscout#1235).
MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 14, 2026
…1235)

Trio fired Tidepool uploads concurrently and unserialized: each CGM
reading triggers uploadGlucose() from BOTH glucoseStorage.updatePublisher
AND the GlucoseStored Core Data notification, alongside independent
dose/carbs triggers. Every trigger spawned a Task into the shared
TidepoolKit `TAPI` actor. When the OAuth access token expired, multiple
in-flight requests each passed session.shouldRefresh() and called
refreshSession() in parallel, reusing the single-use rotating refresh
token. That race corrupted the session; subsequent upload completions
never fired, so uploads stopped silently with nothing logged. Nightscout
was unaffected (separate manager), and Loop avoids the bug by funneling
uploads through one serialized RemoteDataServicesManager cycle.

Confirmed from the issue logs: last Tidepool success 01:50:06, zero
Tidepool errors after, app running continuously to 07:33; at 07:31 a
dose reached the "TIDEPOOL DOSE ENTRIES" log but its upload completion
never returned.

Route every upload (glucose, carbs, dose, pump events, settings,
deletes) through TidepoolUploadSerializer so only one network request is
in flight at a time. This guarantees at most one session refresh,
eliminating the concurrent-refresh race, and a wedged request resolves
to a logged .timedOut failure instead of silently stalling every later
upload forever.

Each pipeline's operation now fetches its pending rows (or rebuilds its
payload) inside the serialized run rather than at trigger time, the
pattern used by the Nightscout serializer in nightscout#1290. Two
triggers firing close together -- e.g. the twin glucose triggers above --
previously both snapshotted the same unmarked rows and uploaded them
twice; now the second request coalesces into a follow-up that re-fetches
after the first run marked its rows, finding only genuinely new data.
Chunked uploads also now mark only the chunk that succeeded as uploaded,
instead of marking the full batch after the first chunk. In the dose
pipeline, the backgroundContext.perform block just builds the upload
payloads and returns them; the network calls happen outside the Core
Data context, inside the serialized run.

The serializer watchdog is checked at each enqueue and on app foreground
(didBecomeActive) -- the reliable recovery points while the pipeline
itself is stalled.
MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 15, 2026
An actor that runs enqueued upload operations strictly one at a time, in
order: each operation starts only after the previous one has fully
completed, including its network round-trip. Serialization is global
across pipelines because concurrent uploads -- even to different
endpoints -- can each trigger a Tidepool session refresh that reuses the
single-use rotating refresh token, corrupting the session (nightscout#1235).

Design points, several adopted from the Nightscout upload serializer in
nightscout#1290:

- Requests for a pipeline that gathers its pending data at run time
  coalesce: while a run is enqueued but not started, further requests
  are dropped (the waiting run picks up their data when it fetches),
  and a request arriving mid-run chains exactly one follow-up, which
  doubles as a retry after a failed run. A trigger burst therefore
  costs at most one waiting run per pipeline. Delete pipelines carry
  unique payloads and never coalesce.
- Chain tasks run at .utility QoS: the inherited .background QoS is
  not scheduled during background execution windows, freezing uploads
  until the app is foregrounded.
- A wall-clock watchdog (recoverIfWedged) abandons the chain when its
  head operation exceeds 10 minutes, e.g. because the app was
  suspended mid-request and the completion was lost. Wall clock is
  deliberate: it keeps advancing while the process is suspended.
  Abandoning bumps a generation counter; operations re-check
  isCurrent(generation) between network calls so an orphaned chain
  cannot fire requests alongside its replacement.
- awaitUpload bridges TidepoolKit's completion-based calls into
  async/await with a 120s timeout backstop: a call that never invokes
  its completion resolves to a logged .timedOut failure instead of
  suspending the chain forever. SingleResumer guards the continuation
  so whichever of completion/timeout fires second is a no-op instead
  of a double-resume crash.

Not wired up yet; groundwork for serializing all Tidepool uploads
(nightscout#1235).
MikePlante1 added a commit to MikePlante1/Trio that referenced this pull request Jul 15, 2026
…1235)

Trio fired Tidepool uploads concurrently and unserialized: each CGM
reading triggers uploadGlucose() from BOTH glucoseStorage.updatePublisher
AND the GlucoseStored Core Data notification, alongside independent
dose/carbs triggers. Every trigger spawned a Task into the shared
TidepoolKit `TAPI` actor. When the OAuth access token expired, multiple
in-flight requests each passed session.shouldRefresh() and called
refreshSession() in parallel, reusing the single-use rotating refresh
token. That race corrupted the session; subsequent upload completions
never fired, so uploads stopped silently with nothing logged. Nightscout
was unaffected (separate manager), and Loop avoids the bug by funneling
uploads through one serialized RemoteDataServicesManager cycle.

Confirmed from the issue logs: last Tidepool success 01:50:06, zero
Tidepool errors after, app running continuously to 07:33; at 07:31 a
dose reached the "TIDEPOOL DOSE ENTRIES" log but its upload completion
never returned.

Route every upload (glucose, carbs, dose, pump events, settings,
deletes) through TidepoolUploadSerializer so only one network request is
in flight at a time. This guarantees at most one session refresh,
eliminating the concurrent-refresh race, and a wedged request resolves
to a logged .timedOut failure instead of silently stalling every later
upload forever.

Each pipeline's operation now fetches its pending rows (or rebuilds its
payload) inside the serialized run rather than at trigger time, the
pattern used by the Nightscout serializer in nightscout#1290. Two
triggers firing close together -- e.g. the twin glucose triggers above --
previously both snapshotted the same unmarked rows and uploaded them
twice; now the second request coalesces into a follow-up that re-fetches
after the first run marked its rows, finding only genuinely new data.
Chunked uploads also now mark only the chunk that succeeded as uploaded,
instead of marking the full batch after the first chunk. In the dose
pipeline, the backgroundContext.perform block just builds the upload
payloads and returns them; the network calls happen outside the Core
Data context, inside the serialized run.

The serializer watchdog is checked at each enqueue and on app foreground
(didBecomeActive) -- the reliable recovery points while the pipeline
itself is stalled.

@dnzxy dnzxy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looking awesome. Few nit comments / questions.

Comment thread Trio/Sources/Services/Network/Nightscout/NightscoutManager.swift Outdated
Comment thread Trio/Sources/Services/Network/Nightscout/NightscoutManager.swift
Comment thread TrioTests/NightscoutUploadSerializerTests.swift Outdated

@dnzxy dnzxy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few nit comments, overall great.

@dnzxy
dnzxy merged commit 4bc16ca into nightscout:dev Jul 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants