[Bug]: aiTap intermittently fails on Android WebView (first tap after page navigation not registered)

[H-M-Master]

Version

Version:
  System:
    OS: macOS 26.6
    CPU: (12) arm64 Apple M4 Pro
    Memory: 264.39 MB / 24.00 GB
    Shell: 5.9 - /bin/zsh
  npmPackages:
    @midscene/android: 1.10.1 => 1.10.1

Details

Details:

Description

When running multiple test cases sequentially on a remote Android device (WeTest cloud),
the first aiTap after a page navigation sometimes fails to register — the AI correctly
identifies the element position, the ADB command is sent to the correct coordinates,
but the tap has no effect on the app. A second tap on the same coordinates always succeeds.

Environment

• Device: Samsung SM-S9480, Android 16 (API 36)
• App: Native Android app with embedded WebView pages
• Connection: Remote device via WeTest (ADB over network)
• Model: qwen3-vl

Reproduce link

This issue requires a remote Android device with network-forwarded ADB and a WebView-based app to reproduce. A minimal public reproduction is ▎ not feasible due to infrastructure requirements. Happy to provide CI logs and Midscene reports if needed.

Reproduce Steps

Steps to Reproduce

1. Use agentFromAdbDevice to connect
2. Navigate to a page via deeplink (am start -a android.intent.action.VIEW -d ...)
3. Wait for page to fully load (10s sleep, page confirmed loaded in screenshot)
4. Call agent.aiTap('click some button')
5. The tap does NOT register (page state unchanged)
6. Call the same aiTap again → succeeds immediately

This only happens:

• When running multiple tests sequentially (single test always passes)
• On remote/cloud devices (local device rarely reproduces)
• After deeplink-based page navigation

Workaround

Sending input keyevent 0 (KEYCODE_UNKNOWN) after page navigation and before the first
aiTap resolves the issue — it appears to "wake up" the WebView's touch event handling.

await agent.runAdbShell('input keyevent 0');
await agent.aiTap('...');  // now works on first try

Analysis

From the Midscene report, the AI localization is correct and the tap coordinates are accurate.
The issue seems to be at the ADB input delivery level.

I noticed Midscene uses input swipe x y x y 150 to simulate taps.
Could this 150ms swipe-at-same-point approach be handled differently by WebViews
that haven't received input focus yet? A raw input tap x y or the input keyevent 0
pre-warming might behave differently at the OS level for focus acquisition.

Expected Behavior

aiTap should reliably trigger the tap on the first call when the element is correctly
identified and the page is fully loaded.

[quanru]

Thanks for the detailed report. I investigated this on a physical Samsung device and want to share the results before we change the default tap behavior.

Test environment

• Samsung SM-N9860
• Android 13 / API 33
• Android System WebView 150.0.7871.181
• The same WebView fixture was exercised over both USB ADB and TCP/network ADB after repeated deeplink navigation.

Results

Transport / mode	Legacy input swipe x y x y 150	Native input tap x y
USB ADB, natural WebView, 10 iterations	First click succeeded 10/10	First click succeeded 10/10
TCP ADB, natural WebView, 20 iterations	First click succeeded 20/20	First click succeeded 20/20
Synthetic MOVE-cancel guard, 3 iterations	First click failed 3/3	First click succeeded 3/3

The natural Android 13 device therefore does not reproduce the reported failure, even over network ADB.

I also inspected the actual WebView events through Chrome DevTools:

• input tap produced pointerdown/touchstart -> pointerup/touchend -> click, with DOWN and UP using the same timestamp.
• The 150 ms zero-distance swipe produced DOWN and UP about 152 ms apart.
• The Android View layer did receive MOVE events for the swipe, but Chromium did not expose any pointermove or touchmove for the zero-displacement gesture and still generated a click.
• On this device, zero-distance swipes of 0–2 ms produced no MOVE; durations of 5 ms and above consistently produced MOVE events at the Android View layer.

The deterministic reproduction works only because its native OnTouchListener explicitly sends ACTION_CANCEL to the WebView and consumes the first gesture when it sees MOVE. It is useful for demonstrating the semantic difference between the commands, but it is not evidence that an ordinary WebView naturally drops the first swipe.

I also compared focus state before and after input keyevent 0:

• mCurrentFocus already pointed to the fixture activity.
• InputDispatcher reported the same focused application/window with FocusRequest result='OK'.
• The DOM remained document.hasFocus() === true, visible, with the same active element.
• keyevent 0 did not change any observable focus state on this device.

Current conclusion

The available evidence does not support network ADB alone or generic WebView focus acquisition as the root cause. The failure still appears specific to the Android 16 / WeTest / target-app input pipeline—possibly a native touch interceptor, an Android 16 input-channel state, or a cloud-device input proxy.

Because swipe was intentionally introduced in #876 to improve responsiveness on some devices, globally replacing it with input tap may introduce a compatibility regression. A potentially safer low-level direction is a single injected ACTION_DOWN -> short hold -> ACTION_UP sequence without MOVE, but it must be validated on the affected Android 16 device and against the original responsiveness scenario.

To identify the exact failing layer, could you provide the following from the affected run?

1. Android System WebView/Chrome version.
2. Native dispatchTouchEvent / WebView OnTouchListener logs for the failed first gesture and successful second gesture, including action, downTime, eventTime, coordinates, and window-focus state.
3. JavaScript pointerdown/pointermove/pointerup/touchstart/touchmove/touchend/click logs.
4. The focused application/window sections from dumpsys input and mCurrentFocus from dumpsys window, both before and after the keyevent 0 workaround.
5. A direct comparison, after the same deeplink navigation, of:
   • input swipe x y x y 150
   • input tap x y

Those logs will tell us whether the gesture never reaches the app, is cancelled in the native View layer, or reaches Chromium but fails to generate a click.