Functional test runner just stops in certain cases

[liza-mae]

Recently I am seeing the functional test runner just stop and there is no indication as to why. It is difficult to reproduce and it is blocking tests from running to completion. There is no summary of what already ran, it just stops and nothing else is printed. Other errors are thrown before and tests continued, so not sure what is going on - it occurred for me locally a couple times too but is not consistent.

@spalger can you help?

See:
https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-kibana+master+multijob-selenium/1/consoleText

Tests are being run with command: node scripts/functional_test_runner with --debug option, not sure if there is more logging we can obtain, if it occurs again.

[elasticmachine]

Pinging @elastic/kibana-operations

[liza-mae]

This problem was opened around the time of 6.6 release back in Jan, then it went away and I have not seen again until now when testing 7.3. Looks like we have more debug messages available, so far I have been able to reproduce this after several attempts.

│ debg TestSubjects.exists(dashboardPanelHeading-APie)
           │ debg Find.existsByDisplayedByCssSelector('[data-test-subj~="dashboardPanelHeading-APie"]') with timeout=120000
           │ERROR UNHANDLED PROMISE REJECTION
           │ERROR Error: retry.try timeout: TimeoutError: Waiting for element to be located By(css selector, .euiTableRow)
           │      Wait timed out after 122234ms
           │          at /home/liza/TESTING/k73/node_modules/selenium-webdriver/lib/webdriver.js:834:17
           │          at process._tickCallback (internal/process/next_tick.js:68:7)
           │          at lastError (/home/liza/TESTING/k73/test/common/services/retry/retry_for_success.ts:28:9)
           │          at onFailure (/home/liza/TESTING/k73/test/common/services/retry/retry_for_success.ts:68:13)

[elasticmachine]

Pinging @elastic/kibana-test-triage

[tylersmalley]

How often is this happening?

[liza-mae]

Hard to say, some releases it would happen more often than others. It is not cloud specific though, I was able to reproduce it running node functional_tests_server and in another terminal running node functional_test_runner.

[liza-mae]

This started happening again 7.8, 7.9 and master test runs.

05:56:27          └-> can navigate to app
05:56:27            └-> "before each" hook: global before each
05:56:40 Unhandled Promise rejection detected:
05:56:40 
05:56:40 { NoSuchElementError: no such element: Unable to locate element: {"method":"css selector","selector":"[data-test-subj=collapsibleNav] > button"}
05:56:40   (Session info: headless chrome=81.0.4044.122)
05:56:40     at Object.throwDecodedError (/var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/xpackGrp1/TASK/saas_run_kibana_tests/node/linux-immutable/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/error.js:550:15)
05:56:40     at parseHttpResponse (/var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/xpackGrp1/TASK/saas_run_kibana_tests/node/linux-immutable/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/http.js:565:13)
05:56:40     at Executor.execute (/var/lib/jenkins/workspace/elastic+estf-cloud-kibana-tests/JOB/xpackGrp1/TASK/saas_run_kibana_tests/node/linux-immutable/ci/cloud/common/build/kibana/node_modules/selenium-webdriver/lib/http.js:491:26)
05:56:40     at process._tickCallback (internal/process/next_tick.js:68:7)
05:56:40   name: 'NoSuchElementError',
05:56:40   remoteStacktrace: '#0 0x55aa770385b9 <unknown>\n' }
05:56:40 
05:56:40 Terminating process...

[liza-mae]

I consider this high priority since it is preventing the test suites from completing.