Confirm commands for W3C actions

[KazuCocoa]

This is a

• Bug report
• Question
• Freature Request

Summary

• https://github.com/jlipps/simple-wd-spec#perform-actions
• For
  • XCUITest
  • UIA2

---

W3CActionBuilder: https://seleniumhq.github.io/selenium/docs/api/rb/Selenium/WebDriver/W3CActionBuilder.html

Environment

• Appium version (or git revision):
• ruby_lib version:
• Mobile platform/version/device under test:

Actual behaviour and steps to reproduce

Expected behaviour

Link to Appium/Ruby logs

Create a GIST which is a paste of your full Appium logs, and link them here.

Any additional comments

[KazuCocoa]

1. Check https://github.com/appium/ruby_lib_core first.
2. Wrap commands for current TouchAction if we need for the compatibility.
   • Hint for current implementation
     • https://github.com/SeleniumHQ/selenium/blob/master/rb/lib/selenium/webdriver/common/touch_action_builder.rb#L18

[KazuCocoa]

UIA2

Easy action. Just tapping a particular element.

[debug] [JSONWP Proxy] Proxying [POST /element] to [POST http://localhost:8200/wd/hub/session/b1b98b5a-c70e-4e1e-9365-6377ac922561/element] with body: {"strategy":"accessibility id","selector":"App","context":"","multiple":false}
[debug] [JSONWP Proxy] Got response with status 200: {"sessionId":"b1b98b5a-c70e-4e1e-9365-6377ac922561","status":0,"value":{"ELEMENT":"993b2c92-9a36-490a-be84-32a23b7b7f4f"}}
[debug] [MJSONWP] Responding to client with driver.findElement() result: {"ELEMENT":"993b2c92-9a36-490a-be84-32a23b7b7f4f"}
[HTTP] <-- POST /wd/hub/session/d7968638-90be-4306-b4e0-d15ac0fbf284/element 200 31 ms - 122
[HTTP] --> POST /wd/hub/session/d7968638-90be-4306-b4e0-d15ac0fbf284/actions {"actions":[{"type":"pointer","id":"mouse","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"993b2c92-9a36-490a-be84-32a23b7b7f4f"}},{"type":"pointerDown","button":0},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"mouse"}}]}
[MJSONWP] Driver proxy active, passing request on via HTTP proxy
[debug] [JSONWP Proxy] Proxying [POST /wd/hub/session/d7968638-90be-4306-b4e0-d15ac0fbf284/actions] to [POST http://localhost:8200/wd/hub/session/b1b98b5a-c70e-4e1e-9365-6377ac922561/actions] with body: {"actions":[{"type":"pointer","id":"mouse","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"993b2c92-9a36-490a-be84-32a23b7b7f4f"}},{"type":"pointerDown","button":0},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"mouse"}}]}
[debug] [JSONWP Proxy] Got response with status 200: {"sessionId":"b1b98b5a-c70e-4e1e-9365-6377ac922561","status":0,"value":"OK"}
[JSONWP Proxy] Replacing sessionId b1b98b5a-c70e-4e1e-9365-6377ac922561 with d7968638-90be-4306-b4e0-d15ac0fbf284
[HTTP] <-- POST /wd/hub/session/d7968638-90be-4306-b4e0-d15ac0fbf284/actions 200 306 ms - 76

[KazuCocoa]

UIA2

And I also can do the following command:

        el = @@driver.find_element(:accessibility_id, 'Views')
        @@driver.action.click(el).perform

        el = @@driver.find_element(:accessibility_id, 'Custom')
        @@driver.action.click_and_hold(el).move_to_location(0, 500).release.perform

[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, class name, accessibility id, -android uiautomator
[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, class name, accessibility id, -android uiautomator
[debug] [BaseDriver] Waiting up to 0 ms for condition
[debug] [JSONWP Proxy] Proxying [POST /element] to [POST http://localhost:8200/wd/hub/session/022a3c95-7ae6-4dcc-b7ec-bc9469ad6d3a/element] with body: {"strategy":"accessibility id","selector":"Custom","context":"","multiple":false}
[debug] [JSONWP Proxy] Got response with status 200: {"sessionId":"022a3c95-7ae6-4dcc-b7ec-bc9469ad6d3a","status":0,"value":{"ELEMENT":"900590bc-1fae-450e-afc4-be714283cf67"}}
[debug] [MJSONWP] Responding to client with driver.findElement() result: {"ELEMENT":"900590bc-1fae-450e-afc4-be714283cf67"}
[HTTP] <-- POST /wd/hub/session/1b67cce8-2654-4315-afcd-c5fa9fa21438/element 200 32 ms - 122
[HTTP] --> POST /wd/hub/session/1b67cce8-2654-4315-afcd-c5fa9fa21438/actions {"actions":[{"type":"pointer","id":"mouse","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"900590bc-1fae-450e-afc4-be714283cf67"}},{"type":"pointerDown","button":0},{"type":"pointerMove","duration":250,"x":0,"y":500,"origin":"viewport"},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"mouse"}}]}
[MJSONWP] Driver proxy active, passing request on via HTTP proxy
[debug] [JSONWP Proxy] Proxying [POST /wd/hub/session/1b67cce8-2654-4315-afcd-c5fa9fa21438/actions] to [POST http://localhost:8200/wd/hub/session/022a3c95-7ae6-4dcc-b7ec-bc9469ad6d3a/actions] with body: {"actions":[{"type":"pointer","id":"mouse","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"900590bc-1fae-450e-afc4-be714283cf67"}},{"type":"pointerDown","button":0},{"type":"pointerMove","duration":250,"x":0,"y":500,"origin":"viewport"},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"mouse"}}]}
[debug] [JSONWP Proxy] Got response with status 200: {"sessionId":"022a3c95-7ae6-4dcc-b7ec-bc9469ad6d3a","status":0,"value":"OK"}
[JSONWP Proxy] Replacing sessionId 022a3c95-7ae6-4dcc-b7ec-bc9469ad6d3a with 1b67cce8-2654-4315-afcd-c5fa9fa21438
[HTTP] <-- POST /wd/hub/session/1b67cce8-2654-4315-afcd-c5fa9fa21438/actions 200 1750 ms - 76
[BaseDriver] Shutting down because we waited 60 seconds for a command
[debug] [UiAutomator2] Deleting UiAutomator2 session
[Appium] Closing session, cause was 'New Command Timeout of 60 seconds expired. Try customizing the timeout using the 'newCommandTimeout' desired capability'
[Appium] Removing session 1b67cce8-2654-4315-afcd-c5fa9fa21438 from our master session list
[debug] [UiAutomator2] Deleting UiAutomator2 server session

Documentation:

• http://www.rubydoc.info/gems/selenium-webdriver/Selenium/WebDriver/W3CActionBuilder
• http://www.rubydoc.info/gems/selenium-webdriver/Selenium/WebDriver/PointerActions

---

For client, we can call previous touchactions.
Thus, I put some documentation and examples in test case, but I keep the touch methods.

[KazuCocoa]

WDA

• By default, Ruby client "parameters":{"pointerType":"mouse"} and WDA return error about:

  Selenium::WebDriver::Error::UnknownError: Error Domain=com.facebook.WebDriverAgent Code=1 "Only pointer type 'touch' is supported. 'mouse' is given instead for action with id 'mouse'" UserInfo={NSLocalizedDescription=Only pointer type 'touch' is supported. 'mouse' is given instead for action with id 'mouse'}
  

  • UIA2 can work with mouse. Can WDA relax to allow mouse?

[debug] [JSONWP Proxy] Proxying [POST /wd/hub/session/96e1f06a-8dc9-4a83-9f07-1322c7ae63ac/actions] to [POST http://localhost:8100/session/24CD3298-9992-4B15-AB5A-B1DF025C2621/actions] with body: {"actions":[{"type":"pointer","id":"mouse","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"F02C3E0A-F230-42B5-8ECD-88E4A381075D"}},{"type":"pointerDown","button":0},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"mouse"}}]}
[debug] [JSONWP Proxy] Got response with status 200: {"value":"Error Domain=com.facebook.WebDriverAgent Code=1 \"Only pointer type 'touch' is supported. 'mouse' is given instead for action with id 'mouse'\" UserInfo={NSLocalizedDescription=Only pointer type 'touch' is supported. 'mouse' is given instead for action with id 'mouse'}","sessionId":"24CD3298-9992-4B15-AB5A-B1DF025C2621","status":13}
[JSONWP Proxy] Replacing sessionId

I could do with touch like the below:

        action = ::Selenium::WebDriver::W3CActionBuilder.new @@driver.send(:bridge), ::Selenium::WebDriver::Interactions::PointerInput.new(:touch, name: 'touch'), Selenium::WebDriver::Interactions::KeyInput.new('keyboard'), false

        el = @@driver.find_element(:accessibility_id, 'Buttons')
        action.click(el).perform

        el = @@driver.find_element(:id, 'XCUIElementTypeApplication')
        action.click_and_hold(el).move_to_location(0, 700).release.perform

        el = @@driver.find_element(:accessibility_id, 'ImageButton')
        assert_equal 'ImageButton', el.name

[KazuCocoa]

@mykola-mokhnach
Can you relax the pointerType for WDA?
Currently, 'mouse' is used by default for the parameter because of W3C spec.

To set touch in the action, Ruby client need to the below line and only for WDA, not UIA2, so far.

::Selenium::WebDriver::W3CActionBuilder.new @@driver.send(:bridge), ::Selenium::WebDriver::Interactions::PointerInput.new(:touch, name: 'touch'), Selenium::WebDriver::Interactions::KeyInput.new('keyboard'), false

[mykola-mokhnach]

@KazuCocoa It's by design. W3C spec makes a clear distinction between different pointer types (currently 3 types are defined: mouse, pen and touch). Only touch pointers are supported on iOS and Android has support of all types. That is why you see the error below. We could think about additional wrappers/helpers on appium Ruby client side to make the creation on Actions object with touch pointer type easier

[KazuCocoa]

https://github.com/jlipps/simple-wd-spec#location-strategies

CSS selector	css selector
Link text selector	link text
Partial link text selector	partial link text
Tag name	tag name
XPath selector	xpath

---

Ruby client converts some elements to css celector like:

        def convert_locators(how, what)
          case how
          when 'class name'
            how = 'css selector'
            what = ".#{escape_css(what)}"
          when 'id'
            how = 'css selector'
            what = "##{escape_css(what)}"
          when 'name'
            how = 'css selector'
            what = "*[name='#{escape_css(what)}']"
          when 'tag name'
            how = 'css selector'
          end
          [how, what]
        end

Don't convert to Appium, so far.

[KazuCocoa]

📝
Use 'touch' by default

[HTTP] --> POST /wd/hub/session/ddb1d5c6-affa-4b47-aeef-97eb95777093/actions {"actions":[{"type":"pointer","id":"touch","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"3041AA5D-53EF-48D0-80B3-E30EE2012EB8"}},{"type":"pointerDown","button":0},{"type":"pointerMove","duration":250,"x":0,"y":300,"origin":"viewport"},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"touch"}}]}
[MJSONWP] Driver proxy active, passing request on via HTTP proxy
[debug] [XCUITest] Executing command 'proxyReqRes'
[debug] [JSONWP Proxy] Proxying [POST /wd/hub/session/ddb1d5c6-affa-4b47-aeef-97eb95777093/actions] to [POST http://localhost:8100/session/780847CE-1494-4EF5-A2B9-12C2C84F4E42/actions] with body: {"actions":[{"type":"pointer","id":"touch","actions":[{"type":"pointerMove","duration":250,"x":0,"y":0,"origin":{"element-6066-11e4-a52e-4f735466cecf":"3041AA5D-53EF-48D0-80B3-E30EE2012EB8"}},{"type":"pointerDown","button":0},{"type":"pointerMove","duration":250,"x":0,"y":300,"origin":"viewport"},{"type":"pointerUp","button":0}],"parameters":{"pointerType":"touch"}}]}
[debug] [JSONWP Proxy] Got response with status 200: {"value":{},"sessionId":"780847CE-1494-4EF5-A2B9-12C2C84F4E42","status":0}
[JSONWP Proxy] Replacing sessionId 780847CE-1494-4EF5-A2B9-12C2C84F4E42 with ddb1d5c6-affa-4b47-aeef-97eb95777093
[HTTP] <-- POST /wd/hub/session/ddb1d5c6-affa-4b47-aeef-97eb95777093/actions 200 990 ms - 74

[KazuCocoa]

✅ appium/ruby_lib_core@2868488

[KazuCocoa]

📝
post timeout's parameters differ between OSS and W3C.
=> https://github.com/jlipps/simple-wd-spec#set-timeouts