Automated UI Testing

[hobinjk]

The UI is currently untested. Approaches this could take include the following:

• Manual testing checklist: https://github.com/mozilla-iot/wiki/wiki/Manual-UI-Testing-Criteria
• Automated in-browser tests (Add very simple in-browser tests #844)

[hobinjk]

From initial investigation I'm pretty torn between Casper and Selenium. Selenium has a slight edge due to its long history and use of the WebDriver spec so I'm going to try writing a simple test using it.

Nightmare

• Promises, built on Phantom
• https://segment.com/blog/ui-testing-with-nightmare/

Phantom

• Chrome headless

Slimer

• Phantom API with firefox
• Supports promises

Casper

• Layer on top of Phantom and Slimer
• http://casperjs.org/
• A little weird, not natively promise-based
• built-in testing support

Puppeteer

• Chrome headless, slightly different API
• https://medium.com/@ankit_m/ui-testing-with-puppeteer-and-mocha-part-1-getting-started-b141b2f9e21
• https://facebook.github.io/jest/docs/en/puppeteer.html
• https://github.com/xfumihiro/jest-puppeteer-example

Selenium/WebDriverJS

• Pretty okay
• https://blog.testproject.io/2017/06/07/javascript-testing-with-selenium-webdriver-mocha/

Marionette

• Python
• Lots of firefox-specific testing functions like switching to chrome context

Mochitest

• Not really documented for use outside of firefox

Firefox Puppeteer

• Literally recommended against in the docs

[sogaani]

If #812 is approval, we can use jsdom.
I want to port tests for JSON SCHEMA form from here.

[sogaani]

@hobinjk
I wrote thing-view test with jsdom on #935.
Which do you think is better testing with jsdom or browser in this case.

Testing with jsdom is faster than browser, but we should write polyfills and can not view screen.

Testing with brower is slower, but we can take screen and test end to end.

I think testing on browser is better and can rewrite, because thing-view dose not have many test per thingType.

[hobinjk]

@sogaani I think for the thing-view it makes more sense to use the browser since it's less tightly coupled with the implementation. There are plans for us to refactor the thing views based on a capabilities model (as opposed to the current type-based model) which would significantly change the class structure but ideally not change the displayed UI.

The jsdom tests are very cool though and definitely something I like for testing components that are less likely to change dramatically.

[sogaani]

@hobinjk I heard good things. I was wondering which way to implement, but if there is a major change in the structure I will test using browser.

[hobinjk]

While we don't have 100% coverage of the UI the initial implementation is very much done.