Maestro test reliability.

[jg210]

Temporarily run tests multiple times to assess and then improve reliability.

[jg210]

2 out of 5 ios-test jobs failed despite setting MAESTRO_DRIVER_STARTUP_TIMEOUT=60000, with:

xcuitest.installer.LocalXCTestInstaller$IOSDriverTimeoutException: iOS driver not ready in time, consider increasing timeout by configuring MAESTRO_DRIVER_STARTUP_TIMEOUT env variable

Others have had this issue too: mobile-dev-inc/Maestro#1585.

I reduced the timeout from the default! Oops.

[jg210]

Maestro will be increasing default to 180s here: mobile-dev-inc/Maestro#2389

[jg210]

https://github.com/mobile-dev-inc/Maestro/blame/b550181f9cee3cc96f034bc600ecf23bdedf28c1/.github/workflows/test-e2e.yaml#L204 uses 240s.

[jg210]

No timeouts with five runs of 180s nor 5 runs of 240s. Instead, for iOS only, got 2 cases of this out of 10 runs. E.g.:

https://productionresultssa15.blob.core.windows.net/actions-results/053db6fe-15c2-4767-a326-154c16adf3bf/workflow-job-run-9d3ef779-75c4-52cb-ab23-bcc2a3f3cafa/logs/job/job-logs.txt?rsct=text%2Fplain&se=2025-03-23T20%3A43%3A35Z&sig=DwmtZwhpTIHDHNvfJPYPqNFTMES7fjR%2BCDj286CfZFE%3D&ske=2025-03-24T07%3A27%3A02Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2025-03-23T19%3A27%3A02Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&st=2025-03-23T20%3A33%3A30Z&sv=2025-01-05

2025-03-23T14:29:56.1635890Z Launch app "uk.me.jeremygreen.expoexperiments"... COMPLETED
2025-03-23T14:30:26.9678900Z Assert that "Barnet" is visible... FAILED

...despite it being visible in commands json and screenshot:

         }, {
                                  "attributes" : {
                                    "accessibilityText" : "Barnet",
                                    "resource-id" : "authorityListItem",
                                    "bounds" : "[64,272][329,289]",
                                    "enabled" : "true",
                                    "focused" : "false",
                                    "selected" : "false",
                                    "checked" : "false"
                                  },
                                  "children" : [ {
                                    "attributes" : {
                                      "accessibilityText" : "Barnet",
                                      "bounds" : "[64,272][329,289]",
                                      "enabled" : "true",
                                      "focused" : "false",
                                      "selected" : "false",
                                      "checked" : "false"
                                    },
                                    "enabled" : true,
                                    "focused" : false,
                                    "checked" : false,
                                    "selected" : false
                                  } ],
                                  "enabled" : true,
                                  "focused" : false,
                                  "checked" : false,
                                  "selected" : false
                                }, {

[jg210]

2025-03-23T14:38:45.9930870Z Launch app "uk.me.jeremygreen.expoexperiments"... COMPLETED
2025-03-23T14:39:35.8270780Z Assert that "Barnet" is visible... COMPLETED

2025-03-23T14:28:56.1192830Z Launch app "uk.me.jeremygreen.expoexperiments"... COMPLETED
2025-03-23T14:29:10.3763960Z Assert that "Barnet" is visible... COMPLETED

2025-03-23T14:30:09.2927330Z Launch app "uk.me.jeremygreen.expoexperiments"... COMPLETED
2025-03-23T14:30:31.9435920Z Assert that "Barnet" is visible... COMPLETED

2025-03-23T14:29:56.1635890Z Launch app "uk.me.jeremygreen.expoexperiments"... COMPLETED
2025-03-23T14:30:26.9678900Z Assert that "Barnet" is visible... FAILED

2025-03-23T14:30:24.0950090Z Launch app "uk.me.jeremygreen.expoexperiments"... COMPLETED
2025-03-23T14:30:42.1287350Z Assert that "Barnet" is visible... COMPLETED

There is a lot of variability in time for Barnet assertion to complete. Up to 50s (oddly, that case passed). Often 15s.

The App is connecting to a real API, and it can be slow to start up. It's spring boot hosted by lambda functions, which take >10s to start up. There's concurrency here while running 5 jobs in parallel, so maybe test is starting multiple lambda functions? Although, have seen same error twice on master branch, where don't have concurrency. Maybe it's from slow start of lambda function for first request? Maybe both things?

Possible solutions:

• Use mocked API.
• Attempt to pre-warm the lambda functions in test before starting app? There's no guarantee that will pre-warm enough lambda functions or that requests will go to the pre-warmed endpoints. Real users don't get pre-warmed lambda functions, so why should tests.
• Larger timeouts. This seems to need https://docs.maestro.dev/advanced/wait / https://docs.maestro.dev/api-reference/commands/extendedwaituntil? Timeout docs are unclear: Default timeout of commands should be documented in API reference mobile-dev-inc/Maestro#1475

[jg210]

Two cases out of twenty, android only:

...of:

https://github.com/jg210/expo-experiments/actions/runs/14043603252/job/39322020881
https://github.com/jg210/expo-experiments/actions/runs/14043592996/job/39319781178

Launch app "uk.me.jeremygreen.expoexperiments"...Exception in thread "pool-4-thread-1" java.io.IOException: Command failed (host:transport:emulator-5554): device offline
...
 [2025-03-24 19:48:44 with_emulator_status] emulator running: true

I.e. app launch failed since device offline, but later the adb shell true works, so don't get a retry.