-
Notifications
You must be signed in to change notification settings - Fork 559
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
e2e tests: improve reliability of login test #1913
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The code looks good, nice and clean. async/await soo useful here. Ran tests, they all passed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The tests pass even if you don't add a TEST_USERNAME and TEST_PASSWORD. They should fail.
@belcherj updated so that tests now properly fail |
c0f0c96
to
2ec6392
Compare
Looks like you have a lint error. |
2ec6392
to
52fe404
Compare
Previously the tests were a bit flakey, working some times and not others. In this patch we've added the `--runInBand` option to `jest` so that the tests run sequentially in the same process as the test runny. It may have been that `jest` was attempting to run the tests in parallel even though the nature of testing Electron/Spectron doesn't support this and so we would sometimes fail based on concurrency conflicts. A few new tests have been added in order to verify that the new updates are working and to demonstrate how we can start adding testable behaviors into the app. We still need to improve how the tests are organized and setup to minimize the noise in the code and isolate the tests.
This patch fixes two bugs related to `waitForExist()` that led to the tests passing when they should have failed. The first bug was a type error whereas `waitForExist()` returns a promise-like object and `expect(...waitForExist()).toBeTruthy()` would therefore always be true. This meant that we not only didn't wait for the elements to appear but also that our check never failed. This has been fixed by awaiting the return. This has been handled inside of a new test helper function to cut down on incidental noise. The second bug was that, despite the type signaures in the webdriverio type definitions, `app.client` and `app.client.$()` return different types, whereas `app.client` returns a `Client` and `$()` returns a "WebElement JSON Object" or something that _isn't_ the same as the docs suggest. This bug was an infinite hang when trying to wait on the elements to appear since the returned value never resolved. Now all calls to wait on an element are handled by `app.client.waitForExist(selector, ...)` instead of using `$()`
8bb7aaf
to
b10762a
Compare
@belcherj doesn't look like it anymore |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ship it!
Previously the tests were a bit flakey, working some times and not others. In
this patch we've added the
--runInBand
option tojest
so that the tests runsequentially in the same process as the test runny. It may have been that
jest
was attempting to run the tests in parallel even though the nature oftesting Electron/Spectron doesn't support this and so we would sometimes fail
based on concurrency conflicts.
A few new tests have been added in order to verify that the new updates are
working and to demonstrate how we can start adding testable behaviors into the
app. We still need to improve how the tests are organized and setup to minimize
the noise in the code and isolate the tests.
Review: Signaling from the app to the tests
We would ideally like to avoid timeouts as a way of progressing through our
tests. Timeouts are inherently flakey because they represent race conditions
on uncontrollable factors. That means we prefer to wait for specific events
and send those events from the running app to the test runner.
In this PR I've created a system that pushes new entries to a global array in
order to signal an event.
This looks similar to our Redux actions though may occur at different times
and circumstances. These are to be used when we can't wait for the presence
of specific DOM nodes via the builtin
webdriverio
functionwaitForExist()
.Through the use of a webpack
DefinePlugin
entry we can do this with a simpleconditional which will be stripped entirely from
development
andproduction
builds, purely because it's discernible at compile-time if the condition is true.
Testing
Since this only affects the in-progress e2e tests it should have zero impact on
the resultant app code. Test by running the e2e test suite.
It is necessary to build the app with
NODE_ENV=test
before running these tests.I hope to cure this at some point but it's just an inconvenience at the moment.