Skip to content

chore: Fix flaky FlagSynchronizerSpec tests (SDK-2042) - #483

Merged
aaron-zeisler merged 2 commits into
v11from
aaronz/SDK-2042/flaky-unit-test-fixes
Mar 18, 2026
Merged

chore: Fix flaky FlagSynchronizerSpec tests (SDK-2042)#483
aaron-zeisler merged 2 commits into
v11from
aaronz/SDK-2042/flaky-unit-test-fixes

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Mar 17, 2026

Copy link
Copy Markdown
Contributor

BEGIN_COMMIT_OVERRIDE
chore: Fix flaky FlagSynchronizerSpec tests
END_COMMIT_OVERRIDE

Requirements

  • I have added test coverage for new or changed functionality
  • I have followed the repository's pull request submission guidelines
  • I have validated my changes against all supported platform versions

Related issues

Describe the solution you've provided

Fixes two flaky tests in FlagSynchronizerSpec.swift, each in a separate commit:

  1. change_isOnline__online_to_offline__stops_polling — Moved isOnline = false inside the onSyncComplete callback, before signaling the semaphore. This prevents a second timer tick from firing between the callback and the assertion, which caused getFeatureFlagsCallCount to be 2 instead of 1 on slow CI runners. This is the same pattern already applied to pollingTimerFiresSpec in PR chore: Parallelize builds to cut down CI wall clock time #481.

  2. streaming_events__event_reported_while_polling__reports_an_event_error — Added an explicit 5-second timeout to the first waitUntil block. The default 1-second timeout was insufficient for the multi-hop async callback chain (main RunLoop → syncQueue → getFeatureFlags → reportSyncComplete → DispatchQueue.main.async) on loaded CI runners. The 5-second timeout is consistent with other async tests in the codebase (e.g., DarklyServiceSpec, EventReporterSpec).

Both changes are test-code-only; no application code is modified.

Describe alternatives you've considered

For fix 2, an alternative was converting to the semaphore + RunLoop pump pattern used elsewhere in FlagSynchronizerSpec. However, waitUntil(timeout:) is the dominant async pattern in the test suite (16 call sites across 4 files), so an explicit timeout is more consistent and a smaller change.

Additional context

For reviewers: The second waitUntil in the event_reported_while_polling test (the one waiting on testStreamOnMessage) still uses the default 1-second timeout. That chain is simpler (no timer hop — just a direct DispatchQueue.main.async dispatch), so 1 second should remain sufficient. Worth confirming this matches your understanding.

Link to Devin session: https://app.devin.ai/sessions/1db84bb989a24b319cbc345be77efaac


Note

Low Risk
Test-only changes that adjust async coordination and timeouts; low risk aside from potentially masking real timing regressions if timeouts are too generous.

Overview
Fixes flakiness in FlagSynchronizerSpec polling-related tests by stopping polling inside the onSyncComplete callback (before unblocking assertions) to avoid extra timer ticks racing the expectations.

Also increases the first waitUntil timeout to 5 seconds for the “event reported while polling” case so the multi-hop async setup reliably completes on slower CI runners.

Written by Cursor Bugbot for commit 2be732e. This will update automatically on new commits. Configure here.

devin-ai-integration Bot and others added 2 commits March 17, 2026 17:27
Stop polling inside the onSyncComplete callback to prevent the timer
from firing a second tick before the test can set isOnline = false.
This is the same pattern applied in pollingTimerFiresSpec (PR #481).

Fixes the race condition where getFeatureFlagsCallCount == 2 instead
of 1 on slow CI runners.

SDK-2042

Co-Authored-By: Aaron Zeisler <azeisler@launchdarkly.com>
The first waitUntil block used Nimble's default 1-second timeout, but
the callback delivery chain (main RunLoop → syncQueue → getFeatureFlags
→ reportSyncComplete → DispatchQueue.main.async → onSyncComplete) can
exceed 1 second on loaded CI runners. Increase to 5 seconds, consistent
with other async tests in the codebase.

SDK-2042

Co-Authored-By: Aaron Zeisler <azeisler@launchdarkly.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@aaron-zeisler
aaron-zeisler marked this pull request as ready for review March 17, 2026 17:33
@aaron-zeisler
aaron-zeisler requested a review from a team as a code owner March 17, 2026 17:33
@aaron-zeisler
aaron-zeisler merged commit cd38c0f into v11 Mar 18, 2026
21 checks passed
@aaron-zeisler
aaron-zeisler deleted the aaronz/SDK-2042/flaky-unit-test-fixes branch March 18, 2026 15:16
@kinyoklion kinyoklion changed the title fix: Fix flaky FlagSynchronizerSpec tests (SDK-2042) chore: Fix flaky FlagSynchronizerSpec tests (SDK-2042) Mar 19, 2026
kinyoklion added a commit that referenced this pull request Mar 23, 2026
**Requirements**

- [ ] I have added test coverage for new or changed functionality
- [x] I have followed the repository's [pull request submission
guidelines](../blob/v11/CONTRIBUTING.md#submitting-pull-requests)
- [ ] I have validated my changes against all supported platform
versions

**Related issues**

- Follows up on #483 and #480 which addressed other flaky tests in this
file
- Fixes the `build-ios (15.4.0, macos-14)` CI failure:
`polling_timer_fires__one_second_interval__stops_polling, failed -
expected to equal <2>, got <6>`
- Fixes the `build-ios (16.4.0, macos-15)` CI failure:
`streaming_events__event_reported_while_polling__reports_an_event_error,
failed - waitUntil(..) expects its completion closure to be only called
once`

**Describe the solution you've provided**

Adds a `didSignal` guard to prevent `onSyncComplete` callbacks from
re-entering and double-signaling the semaphore. For tests that verify
polling stopped, captures the call count after stopping and uses
`Thread.sleep` to confirm no further requests occur. For tests that only
need to confirm polling started, relaxes exact-count assertions to `>=
1`. Five tests are fixed:

1. **`changeIsOnlineSpec` ("stops polling")** — Added `didSignal` guard.
The `isOnline = false` + `semaphore.signal()` inside the callback was
already present in v11. Changed `== 1` to `>= 1`, then captures
`countAfterStop`, sleeps 1.5s, and asserts count unchanged — proving
polling actually stopped.

2. **`changeIsOnlineSpec` ("starts polling")** — Added `didSignal` guard
to prevent double-signaling. No other changes to callback body; original
test structure preserved (`isOnline` stays `true`, cleanup at the end).
Changed `== 1` to `>= 1`.

3. **`changeIsOnlineSpec` ("does not stop polling")** — Added
`didSignal` guard. No `isOnline = false` in callback; original test
structure preserved. Changed `== 1` to `>= 1`.

4. **`streamingProcessingSpec` ("event reported while polling")** —
Replaced the first `waitUntil` block with the semaphore + `didSignal`
guard pattern. The original `waitUntil { done in ... { _ in done() } ...
isOnline = true }` failed when polling timer ticks called `done()` more
than once.

5. **`pollingTimerFiresSpec` ("stops polling")** — After stopping
polling inside the callback at `requestCount == 2`, the test now
captures the call count, waits 1.5 seconds, and asserts the count hasn't
changed — directly proving polling actually stopped. Uses `>= 2` instead
of `== 2` to tolerate in-flight callbacks.

All changes are test-code-only; no application code is modified.

**Describe alternatives you've considered**

An earlier revision stopped polling (`isOnline = false`) inside the
callbacks for tests #2 and #3 as well, but this introduced a timing
dependency: the count staying at exactly 1 relied on the main queue
processing the callback before the next 1-second timer tick. The current
approach avoids that dependency entirely — the guard prevents
re-signaling and `>= 1` tolerates any number of extra ticks.

**Additional context**

> **For reviewers — human review checklist:**
> - **`>= 1` in "does not stop polling"**: This is weaker than the
original `== 1`. The original assertion caught a restart regression
(second `isOnline = true` triggering another immediate flag request).
With `>= 1`, both normal polling and a restart regression satisfy the
assertion. The `startPolling()` guard (`flagRequestTimer == nil`) in
production code is what prevents restarts; this test no longer
independently verifies that. If this is a concern, an alternative would
be exposing timer state for direct inspection.
> - **`didSignal` is not explicitly synchronized**, matching the
existing `didCallDone` pattern in `LDTimerSpec`. The guard only needs to
prevent the callback *body* from running a second time — it does not
need to be atomic to accomplish this.
> - **`Thread.sleep(forTimeInterval: 1.5)`** is used in both "stops
polling" tests. This is generous relative to the 1-second polling
interval but could theoretically be tight on very slow CI runners. Let
me know if you'd prefer a longer wait or a different verification
approach.
> - **`pollingTimerFiresSpec` and `changeIsOnlineSpec` "stops polling"
both use `>= N` then verify count stability**: There is a small window
where an in-flight `getFeatureFlags` call could land between `isOnline =
false` and the `countAfterStop` read. The `>= N` tolerates this, and the
1.5-second sleep assertion proves no *further* polling occurs regardless
of the exact count at stop time.

Link to Devin session:
https://app.devin.ai/sessions/120fb9abea7743249e11b5bcbbcd1d8a
Requested by: @kinyoklion

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants