Test failures because of Saucelab tests on macOS, Safari 11

[ffMathy]

The build for all PRs is failing for Safari Selenium tests on Linux macOS.

[ffMathy]

This is quite critical, as it prevents any PRs from being merged.

[janpio]

Some context:

Example for test failure:
https://travis-ci.org/apache/cordova-plugin-media-capture/builds/418337045
Node 4.2, platform=browser-safari
https://travis-ci.org/apache/cordova-plugin-media-capture/jobs/418337048

...
cordova-paramedic: connecting webdriver
................
cordova-paramedic: connecting to app
cordova-paramedic: waiting for test results
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
cordova-paramedic: Tests failed to complete; ending appium session. The error is:
Error: Seems like device not connected to local server in 300 secs
    at null._onTimeout (/tmp/paramedic/lib/paramedic.js:467:24)
    at Timer.listOnTimeout (timers.js:92:15)
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
cordova-paramedic: Closing Sauce Connect process...
cordova-paramedic: Successfully closed Sauce Connect process
Unsupported platform for Appium test run: browser
Completed tests at 5:39:48 PM
Getting saucelabs jobs details...
============================================================================================
Job name: travis-plugin-media-capture-211.3
Job ID: 05bf4db92bf44e2287e737dd47306eca
Job URL: https://saucelabs.com/beta/tests/05bf4db92bf44e2287e737dd47306eca
Video: https://assets.saucelabs.com/jobs/05bf4db92bf44e2287e737dd47306eca/video.flv
Appium logs: https://assets.saucelabs.com/jobs/05bf4db92bf44e2287e737dd47306eca/selenium-server.log
============================================================================================
local-server: killing local file transfer server if it's up...
cordova-paramedic: Deleting the application: /tmp/tmp-2965IU12VGU4tPCj
/home/travis/build/apache/cordova-plugin-media-capture
Finished with exit code 1
The command "node /tmp/paramedic/main.js --config pr/$PLATFORM --plugin $(pwd) --shouldUseSauce --buildName travis-plugin-media-capture-$TRAVIS_JOB_NUMBER" exited with 1.
Done. Your build exited with 1.

[janpio]

cordova-plugin-media-capture/.travis.yml

Lines 19 to 22 in 5815be4

	- env: PLATFORM=browser-safari
	os: linux
	language: node_js
	node_js: '4.2'

• PLATFORM is used in the test command:
  node /tmp/paramedic/main.js --config pr/$PLATFORM --plugin $(pwd) --shouldUseSauce --buildName travis-plugin-media-capture-$TRAVIS_JOB_NUMBER
• os has meaning for Travis: https://docs.travis-ci.com/user/multi-os/#example-multi-os-build-matrix
• language and node_js as well: https://docs.travis-ci.com/user/languages/javascript-with-nodejs/

Full setup and test command:

cordova-plugin-media-capture/.travis.yml

Lines 69 to 86 in 5815be4

	before_install:
	- rm -rf ~/.nvm && git clone https://github.com/creationix/nvm.git ~/.nvm && (cd ~/.nvm
	&& git checkout `git describe --abbrev=0 --tags`) && source ~/.nvm/nvm.sh && nvm
	install $TRAVIS_NODE_VERSION
	- node --version
	- if [[ "$PLATFORM" =~ android ]]; then gradle --version; fi
	- if [[ "$PLATFORM" =~ ios ]]; then npm install -g ios-deploy; fi
	- if [[ "$PLATFORM" =~ android ]]; then echo y | android update sdk -u --filter android-22,android-23,android-24,android-25,android-26;
	fi
	- git clone https://github.com/apache/cordova-paramedic /tmp/paramedic && pushd /tmp/paramedic
	&& npm install && popd
	- npm install -g cordova
	install:
	- npm install
	script:
	- npm test
	- node /tmp/paramedic/main.js --config pr/$PLATFORM --plugin $(pwd) --shouldUseSauce
	--buildName travis-plugin-media-capture-$TRAVIS_JOB_NUMBER

[janpio]

For comparison, this is a successful run (with Firefox): https://travis-ci.org/apache/cordova-plugin-media-capture/jobs/421191978
And again, here the failing one: https://travis-ci.org/apache/cordova-plugin-media-capture/jobs/421191979

[janpio]

So something is going wrong on Saucelab where it can't connect to Safari or other way around.

The tests report a test URL on saucelab:
Success with FF: https://saucelabs.com/beta/tests/a50cb73525164226a1e3adbae6e721f1
Failure with Safari: https://saucelabs.com/beta/tests/f8cc93a41cd242a18835e5345c203df0

[janpio]

Looking at those, we see that on FF it is a short session of 4 steps:

POST /session
POST url
POST execute
DELETE /session/a50cb73525164226a1e3adbae6e721f1

On Safari, it is an endless:

POST url
POST execute
POST url
POST execute
POST url
POST execute
POST execute
POST url
POST execute
POST url
POST execute
POST execute
...

The screenshots/video of Safari also show that it doesn't really load the page as it does in FF.

[janpio]

Looking at the selenium-server.log of the both and comparing, it seems to go all wrong for Safari here:

10:12:57.177 INFO - Executing: [set window position])
10:12:57.204 WARN - Exception thrown
org.openqa.selenium.NoSuchWindowException: A request to use a window could not be satisfied because the window could not be found. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 13 milliseconds
Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
System info: host: 'itako290.prod.miso', ip: 'fe80:0:0:0:c3f:8934:b675:8b9c%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.6', java.version: '9.0.4'
Driver info: org.openqa.selenium.safari.SafariDriver
Capabilities [{applicationCacheEnabled=true, rotatable=false, databaseEnabled=true, handlesAlerts=true, version=12604.5.6.1.1, cleanSession=true, platform=MAC, nativeEvents=true, locationContextEnabled=false, webStorageEnabled=true, browserName=safari, javascriptEnabled=true, platformName=macOS, cssSelectorsEnabled=true}]
Session ID: F3C5C069-19D7-4436-B699-B705F59B57E5
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
	at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:215)
	at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:167)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:671)
	at org.openqa.selenium.remote.RemoteWebDriver$RemoteWebDriverOptions$RemoteWindow.setPosition(RemoteWebDriver.java:895)
	at org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWindow.setPosition(EventFiringWebDriver.java:629)
	at org.openqa.selenium.remote.server.handler.SetWindowPosition.call(SetWindowPosition.java:53)
	at org.openqa.selenium.remote.server.handler.SetWindowPosition.call(SetWindowPosition.java:27)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.openqa.selenium.remote.server.DefaultSession$1.run(DefaultSession.java:176)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.base/java.lang.Thread.run(Thread.java:844)
10:12:57.208 WARN - Exception: A request to use a window could not be satisfied because the window could not be found. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 13 milliseconds
Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
System info: host: 'itako290.prod.miso', ip: 'fe80:0:0:0:c3f:8934:b675:8b9c%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.6', java.version: '9.0.4'
Driver info: org.openqa.selenium.safari.SafariDriver
Capabilities [{applicationCacheEnabled=true, rotatable=false, databaseEnabled=true, handlesAlerts=true, version=12604.5.6.1.1, cleanSession=true, platform=MAC, nativeEvents=true, locationContextEnabled=false, webStorageEnabled=true, browserName=safari, javascriptEnabled=true, platformName=macOS, cssSelectorsEnabled=true}]
Session ID: F3C5C069-19D7-4436-B699-B705F59B57E5
...

After that the test goes into an endless loop.

[janpio]

As I know nothing about Saucelabs and Appium etc, I went through the builds and commits to find the last one that did not fail:
https://travis-ci.org/apache/cordova-plugin-media-capture/builds/367501510
https://travis-ci.org/apache/cordova-plugin-media-capture/jobs/367501513

But I also noticed that those tests were failing before sometimes:
https://travis-ci.org/apache/cordova-plugin-media-capture/builds/317203713
https://travis-ci.org/apache/cordova-plugin-media-capture/builds/298245732

[janpio]

Comparing the logs of the successful run vs. the current failing one only show minimal differences, mostly in npm install and similar.

[janpio]

The error message that is spammed in the log:

appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null

comes from paramedic:
https://github.com/apache/cordova-paramedic/blob/31155dfd9287e9c53b034bc00dba4473f7c9bf5c/lib/appium/helpers/wdHelper.js#L255

[janpio]

The code seems to read something from window and go into error handling if it doesn't find it:
https://github.com/apache/cordova-paramedic/blob/31155dfd9287e9c53b034bc00dba4473f7c9bf5c/lib/appium/helpers/wdHelper.js#L229-L255

[janpio]

The orange url bar seems to be normal for Safari in automation
https://saucelabs.com/beta/tests/f8cc93a41cd242a18835e5345c203df0/commands#2.

Somehow it just can't load the site fully.
selenium-server.log indicates that something goes wrong even before that, with:

10:12:57.177 INFO - Executing: [set window position])
10:12:57.204 WARN - Exception thrown
org.openqa.selenium.NoSuchWindowException: A request to use a window could not be satisfied because the window could not be found. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 13 milliseconds
Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
System info: host: 'itako290.prod.miso', ip: 'fe80:0:0:0:c3f:8934:b675:8b9c%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.6', java.version: '9.0.4'
Driver info: org.openqa.selenium.safari.SafariDriver
Capabilities [{applicationCacheEnabled=true, rotatable=false, databaseEnabled=true, handlesAlerts=true, version=12604.5.6.1.1, cleanSession=true, platform=MAC, nativeEvents=true, locationContextEnabled=false, webStorageEnabled=true, browserName=safari, javascriptEnabled=true, platformName=macOS, cssSelectorsEnabled=true}]
Session ID: F3C5C069-19D7-4436-B699-B705F59B57E5
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
	at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:215)
	at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:167)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:671)
	at org.openqa.selenium.remote.RemoteWebDriver$RemoteWebDriverOptions$RemoteWindow.setPosition(RemoteWebDriver.java:895)
	at org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWindow.setPosition(EventFiringWebDriver.java:629)
	at org.openqa.selenium.remote.server.handler.SetWindowPosition.call(SetWindowPosition.java:53)
	at org.openqa.selenium.remote.server.handler.SetWindowPosition.call(SetWindowPosition.java:27)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.openqa.selenium.remote.server.DefaultSession$1.run(DefaultSession.java:176)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.base/java.lang.Thread.run(Thread.java:844)