Skip to content

Harden Android DevFlow integration tests#287

Merged
jfversluis merged 5 commits into
mainfrom
redth/android-tests
May 20, 2026
Merged

Harden Android DevFlow integration tests#287
jfversluis merged 5 commits into
mainfrom
redth/android-tests

Conversation

@Redth

@Redth Redth commented May 20, 2026

Copy link
Copy Markdown
Member

The Android integration CI failure showed the app process still resumed while localhost agent calls started returning connection refused, which points at transient ADB/forward instability rather than a deterministic app crash. This hardens the test transport so hosted-runner blips do not cascade into dozens of false failures.

Changes

  • Add opt-in transient transport retries to AgentClient and enable them only for Android integration fixture calls.
  • Re-establish missing adb forward entries while Android tests are running.
  • Debounce process-loss detection and fall back from pidof to ps -A before treating the app as gone.
  • Capture forward-list and agent-status diagnostics when app-loss diagnostics are collected.
  • Let DEVFLOW_TEST_ANDROID_SERIAL target connected physical devices for local debugging.

Validation

  • dotnet test src/DevFlow/Microsoft.Maui.DevFlow.Tests/Microsoft.Maui.DevFlow.Tests.csproj -c Debug --logger "trx;LogFileName=devflow-unit-after-fix.trx"
  • DEVFLOW_TEST_PLATFORM=android DEVFLOW_TEST_ANDROID_SERIAL=38020DLJG004ZG dotnet test src/DevFlow/Microsoft.Maui.DevFlow.Agent.IntegrationTests/Microsoft.Maui.DevFlow.Agent.IntegrationTests.csproj -c Debug --logger "trx;LogFileName=android-full-after-fix.trx" --results-directory artifacts/TestResults/devflow-integration/android-full-after-fix

@github-actions

This comment has been minimized.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Code Review: 3 findings posted inline (2 moderate, 1 minor) plus 1 overflow finding. See the lean summary comment for full details.

Generated by Expert Code Review (auto) for issue #287 · ● 20.1M

Comment thread src/DevFlow/Microsoft.Maui.DevFlow.Driver/AgentClient.cs Outdated
Comment thread src/DevFlow/Microsoft.Maui.DevFlow.Tests/WebViewDriverTests.cs Outdated
Redth added a commit that referenced this pull request May 20, 2026
- AgentClient: narrow IsTransientTransportException so HttpClient timeout
  TaskCanceledException no longer triggers retries; only retry HttpRequestException
  with SocketException inner, IOException, or non-timeout TaskCanceledException.
- AndroidEmulatorFixture: parse ps -A PID via positional column 1 instead of the
  first int-parseable column (custom ROMs may use numeric UIDs in USER).
- AndroidEmulatorFixture: short-circuit AVD provisioning when DEVFLOW_TEST_ANDROID_SERIAL
  points at an online physical device, so the physical-device path no longer
  requires cmdline-tools to be installed.
- WebViewDriverTests: add a 5s CancellationTokenSource and finally-block accept
  guard to GetStatus_WithTransientRetry_RetriesConnectionRefused so a missed
  accept can no longer hang the test, and set ExclusiveAddressUse=false on both
  listeners to reduce port-rebind flakiness.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Redth

Redth commented May 20, 2026

Copy link
Copy Markdown
Member Author

Thanks for the cross-cutting summary. Addressed all three inline findings (PID parsing, transient retry narrowing, test hang guard) plus the overflow finding: InitializePlatformAsync now calls a new IsPhysicalDeviceReadyAsync() helper first, and when DEVFLOW_TEST_ANDROID_SERIAL points at a non-emulator serial that adb get-state reports as device, AVD provisioning is skipped entirely so the physical-device path works without cmdline-tools installed. The discarded findings look reasonable to leave as-is for now; the pre-existing iOS CI failure is unrelated to this PR.

@Redth Redth marked this pull request as ready for review May 20, 2026 16:24
Copilot AI review requested due to automatic review settings May 20, 2026 16:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens DevFlow’s Android integration test infrastructure against transient transport instability (notably ADB port-forward drops) to reduce CI flakiness that manifests as cascaded “connection refused” failures while the app is still running.

Changes:

  • Added opt-in transient transport retry support to AgentClient and enabled it for Android integration tests.
  • Improved Android integration fixture resiliency by re-checking/recreating ADB forwards, debouncing process-loss detection, and adding richer diagnostics capture.
  • Added a unit test covering the new retry behavior in AgentClient.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/DevFlow/Microsoft.Maui.DevFlow.Tests/WebViewDriverTests.cs Adds a unit test intended to validate transient retry behavior for connection-refused scenarios.
src/DevFlow/Microsoft.Maui.DevFlow.Driver/AgentClient.cs Introduces configurable transient transport retries and applies them across many HTTP operations.
src/DevFlow/Microsoft.Maui.DevFlow.Agent.IntegrationTests/Fixtures/IntegrationTestBase.cs Wraps raw HTTP helpers with Android-only transport retry logic for integration tests.
src/DevFlow/Microsoft.Maui.DevFlow.Agent.IntegrationTests/Fixtures/AppFixtureBase.cs Enables the new AgentClient retry knobs for Android fixtures.
src/DevFlow/Microsoft.Maui.DevFlow.Agent.IntegrationTests/Fixtures/AndroidEmulatorFixture.cs Adds forward re-establishment, debounced process probes, ps -A fallback, and extra diagnostics; supports targeting a physical device via env var.

Comment thread src/DevFlow/Microsoft.Maui.DevFlow.Driver/AgentClient.cs Outdated
Comment thread src/DevFlow/Microsoft.Maui.DevFlow.Tests/WebViewDriverTests.cs
Redth added a commit that referenced this pull request May 20, 2026
- AgentClient: split SendWithTransientRetriesAsync into GET vs explicit-method
  overloads and add a new RetryMutatingRequests property (default true) gating
  whether transient retries apply to POST/PUT/DELETE. XML docs spell out the
  duplicate-side-effect risk on TransientFailureRetryCount and the new
  RetryMutatingRequests so production callers can opt out cleanly without
  changing existing integration-test behavior.
- IntegrationTestBase: narrow IsTransientTransportException to mirror the
  AgentClient predicate (HttpRequestException with SocketException inner,
  IOException, or non-timeout TaskCanceledException) so HttpClient timeouts
  are no longer amplified into ~8x60s of retries.
- WebViewDriverTests: bump the server startup delay to 500ms and wrap the
  client call in a Stopwatch with an elapsed >= retry-delay assertion so the
  retry test fails loudly if the retry path is ever skipped.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Redth

Redth commented May 20, 2026

Copy link
Copy Markdown
Member Author

Follow-up: round 2 of review feedback also addressed two findings this workflow had discarded - the non-idempotent POST/PUT/DELETE retry concern (added a RetryMutatingRequests opt-out, default true to preserve existing behavior, with XML docs spelling out the duplicate-side-effect risk) and the retry test's reliance on a fixed delay (bumped server startup to 500ms and added an elapsed >= retryDelay Stopwatch assertion that fails loudly if the retry path is ever skipped). The remaining discarded findings (ObjectDisposedException propagation, duplicate retry constants, misleading get-state error, monitor tick frequency) still look reasonable to leave as-is - the iOS CI failure is unrelated to this PR.

@github-actions

Copy link
Copy Markdown
Contributor

Analysis of Finding: Missing serial validation in EnsureAgentPortForwardAsync

Verdict: Disagree — this is not a real issue in the actual code.

The reviewer's finding describes a pattern where parts[1] and parts[2] are read without validating parts[0]. However, the actual code in this PR does not use that pattern. Here's what EnsureAgentPortForwardAsync() actually does:

var hasForward = exitCode == 0 && output
    .Split('\n', StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)
    .Any(line => line.StartsWith(_serialNumber, StringComparison.OrdinalIgnoreCase)
        && line.Contains(expectedForward, StringComparison.Ordinal));

It explicitly checks line.StartsWith(_serialNumber, ...) — this does validate that the line belongs to this device's serial number. A forward rule from another device would not match because its line would start with a different serial.

Addressing the three considerations:

  1. Does adb -s <serial> forward --list scope output? — The code uses adb forward --list (without -s), so it gets all devices' forwards. However, the line.StartsWith(_serialNumber, ...) check correctly filters to only this device's lines.

  2. Multi-device CI environments — Even if multiple devices are connected, the StartsWith(_serialNumber) check prevents false positives from other devices' forward rules.

  3. Severity of a false positive — A false positive (thinking the forward exists when it doesn't) would mean the fixture skips re-establishing the forward, potentially causing test connectivity failures. However, this scenario cannot occur with the current code because the serial filtering is correct.

The finding appears to be based on a code pattern that doesn't exist in the actual implementation.

Generated by Expert Code Review (auto) for issue #287 · ● 27.5M ·

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Expert Code Review: 3 findings posted inline. See the summary comment for full methodology and discarded findings.

Generated by Expert Code Review (auto) for issue #287 · ● 27.5M

Comment thread src/DevFlow/Microsoft.Maui.DevFlow.Tests/WebViewDriverTests.cs
Comment thread src/DevFlow/Microsoft.Maui.DevFlow.Driver/AgentClient.cs Outdated
Redth and others added 4 commits May 20, 2026 13:53
Add opt-in transient transport retries for AgentClient and enable them for Android integration tests to absorb short-lived ADB forward interruptions.

Debounce Android app process monitoring, restore missing adb forwards, and capture richer diagnostics when the app appears to disappear.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- AgentClient: narrow IsTransientTransportException so HttpClient timeout
  TaskCanceledException no longer triggers retries; only retry HttpRequestException
  with SocketException inner, IOException, or non-timeout TaskCanceledException.
- AndroidEmulatorFixture: parse ps -A PID via positional column 1 instead of the
  first int-parseable column (custom ROMs may use numeric UIDs in USER).
- AndroidEmulatorFixture: short-circuit AVD provisioning when DEVFLOW_TEST_ANDROID_SERIAL
  points at an online physical device, so the physical-device path no longer
  requires cmdline-tools to be installed.
- WebViewDriverTests: add a 5s CancellationTokenSource and finally-block accept
  guard to GetStatus_WithTransientRetry_RetriesConnectionRefused so a missed
  accept can no longer hang the test, and set ExclusiveAddressUse=false on both
  listeners to reduce port-rebind flakiness.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Carry over the more defensive adb forwarding patterns from PR #273:

- Scope `adb forward --list` to the active device with `-s <serial>` so
  parsing tolerates both the 2-column `<local> <remote>` and 3-column
  `<serial> <local> <remote>` output shapes regardless of how many
  devices are attached.
- Pre-emptively run `adb forward --remove tcp:<agent>` before re-adding
  the mapping so a stale entry left by a prior adb-server crash cannot
  block recovery.
- After re-establishing the forward, re-list and verify the mapping is
  visible. `adb forward` can return 0 while the daemon is in an
  inconsistent state, and silently passing through amplified into a
  ~4 minute opaque retry storm in the agent client.
- Capture both `adb -s <serial> forward --list` and `reverse --list`
  in the crash diagnostics bundle so port-forwarding state is visible
  when triaging app-loss artifacts.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- AgentClient: split SendWithTransientRetriesAsync into GET vs explicit-method
  overloads and add a new RetryMutatingRequests property (default true) gating
  whether transient retries apply to POST/PUT/DELETE. XML docs spell out the
  duplicate-side-effect risk on TransientFailureRetryCount and the new
  RetryMutatingRequests so production callers can opt out cleanly without
  changing existing integration-test behavior.
- IntegrationTestBase: narrow IsTransientTransportException to mirror the
  AgentClient predicate (HttpRequestException with SocketException inner,
  IOException, or non-timeout TaskCanceledException) so HttpClient timeouts
  are no longer amplified into ~8x60s of retries.
- WebViewDriverTests: bump the server startup delay to 500ms and wrap the
  client call in a Stopwatch with an elapsed >= retry-delay assertion so the
  retry test fails loudly if the retry path is ever skipped.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Redth Redth force-pushed the redth/android-tests branch from 7a54bec to 85bbcce Compare May 20, 2026 17:53
- AndroidEmulatorFixture: in the 3-column `adb forward --list` fallback,
  skip rows whose serial does not match the active device. Defensive guard
  against unexpected multi-device output from older adb releases.
- AgentClient / IntegrationTestBase: tighten transient-exception predicates
  so a bare `TaskCanceledException` with no inner exception is no longer
  classified as transient. Caller-initiated CancellationToken cancellations
  must not be retried; only TCEs wrapping a real (non-timeout) transport
  failure should now flow through the retry loop.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Redth

Redth commented May 20, 2026

Copy link
Copy Markdown
Member Author

https://github.com/dotnet/maui-labs/actions/runs/26180361507 Integration tests ran for android on this one too (manually)

@jfversluis jfversluis merged commit f88a58f into main May 20, 2026
4 checks passed
@jfversluis jfversluis deleted the redth/android-tests branch May 20, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants