Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

e2e tests: improve reliability of login test #1913

Merged
merged 8 commits into from
Feb 25, 2020

Conversation

dmsnell
Copy link
Member

@dmsnell dmsnell commented Feb 18, 2020

Previously the tests were a bit flakey, working some times and not others. In
this patch we've added the --runInBand option to jest so that the tests run
sequentially in the same process as the test runny. It may have been that
jest was attempting to run the tests in parallel even though the nature of
testing Electron/Spectron doesn't support this and so we would sometimes fail
based on concurrency conflicts.

A few new tests have been added in order to verify that the new updates are
working and to demonstrate how we can start adding testable behaviors into the
app. We still need to improve how the tests are organized and setup to minimize
the noise in the code and isolate the tests.

Review: Signaling from the app to the tests

We would ideally like to avoid timeouts as a way of progressing through our
tests. Timeouts are inherently flakey because they represent race conditions
on uncontrollable factors. That means we prefer to wait for specific events
and send those events from the running app to the test runner.

In this PR I've created a system that pushes new entries to a global array in
order to signal an event.

// when we only need to indicate that an event occurred
window.testEvents.push('notesLoaded');

// when we want to send data out as well
window.testEvents.push(['noteSelected', noteId, note.content]);

This looks similar to our Redux actions though may occur at different times
and circumstances. These are to be used when we can't wait for the presence
of specific DOM nodes via the builtin webdriverio function waitForExist().

Through the use of a webpack DefinePlugin entry we can do this with a simple
conditional which will be stripped entirely from development and production
builds, purely because it's discernible at compile-time if the condition is true.

__TEST__ && window.testEvents.push('someEvent');

Testing

Since this only affects the in-progress e2e tests it should have zero impact on
the resultant app code. Test by running the e2e test suite.

It is necessary to build the app with NODE_ENV=test before running these tests.
I hope to cure this at some point but it's just an inconvenience at the moment.

NODE_ENV=test npm run build
npm run test-e2e

@dmsnell dmsnell requested a review from a team February 18, 2020 06:03
Copy link
Contributor

@belcherj belcherj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code looks good, nice and clean. async/await soo useful here. Ran tests, they all passed.

Copy link
Contributor

@belcherj belcherj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests pass even if you don't add a TEST_USERNAME and TEST_PASSWORD. They should fail.

@dmsnell
Copy link
Member Author

dmsnell commented Feb 21, 2020

@belcherj updated so that tests now properly fail

@dmsnell dmsnell requested review from belcherj and a team February 21, 2020 01:55
@dmsnell dmsnell force-pushed the tests/bump-e2e-test-reliability branch from c0f0c96 to 2ec6392 Compare February 21, 2020 06:59
@belcherj
Copy link
Contributor

Looks like you have a lint error.

@dmsnell dmsnell force-pushed the tests/bump-e2e-test-reliability branch from 2ec6392 to 52fe404 Compare February 22, 2020 01:37
Previously the tests were a bit flakey, working some times and not others.  In
this patch we've added the `--runInBand` option to `jest` so that the tests run
sequentially in the same process as the test runny. It may have been that
`jest` was attempting to run the tests in parallel even though the nature of
testing Electron/Spectron doesn't support this and so we would sometimes fail
based on concurrency conflicts.

A few new tests have been added in order to verify that the new updates are
working and to demonstrate how we can start adding testable behaviors into the
app. We still need to improve how the tests are organized and setup to minimize
the noise in the code and isolate the tests.
This patch fixes two bugs related to `waitForExist()` that led to the
tests passing when they should have failed.

The first bug was a type error whereas `waitForExist()` returns a
promise-like object and `expect(...waitForExist()).toBeTruthy()`
would therefore always be true. This meant that we not only didn't
wait for the elements to appear but also that our check never failed.

This has been fixed by awaiting the return. This has been handled
inside of a new test helper function to cut down on incidental noise.

The second bug was that, despite the type signaures in the webdriverio
type definitions, `app.client` and `app.client.$()` return different
types, whereas `app.client` returns a `Client` and `$()` returns a
"WebElement JSON Object" or something that _isn't_ the same as the docs
suggest.

This bug was an infinite hang when trying to wait on the elements to
appear since the returned value never resolved.

Now all calls to wait on an element are handled by
`app.client.waitForExist(selector, ...)` instead of using `$()`
@dmsnell dmsnell force-pushed the tests/bump-e2e-test-reliability branch from 8bb7aaf to b10762a Compare February 22, 2020 02:29
@dmsnell
Copy link
Member Author

dmsnell commented Feb 24, 2020

Looks like you have a lint error.

@belcherj doesn't look like it anymore

Copy link
Contributor

@belcherj belcherj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ship it!

@dmsnell dmsnell merged commit 4964ffd into develop Feb 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants