Automatic QUality Assurance
For detailed instructions, particularly how it is running on bayes.colorado.edu, see doc/continuous-testing-management.md. Otherwise:
- Check out all active-repos
npm install
in aquanode js/continuous-server.js
in aqua (in Administrator mode in Windows, so it can make symbolic links for node_modules)- Visit
aqua/html/continuous-loop.html?id={{SOME_IDENTIFIER_HERE}}
in various test browsers (they will run tests continuously) - Visit
aqua/html/continuous-report.html
to view current status
continuous-server.js kicks off:
- A loop that continuously checks whether newer SHAs are available upstream. When a newer SHA is detected, a "stale" flag is set, and repos are pulled. It continues looping until ALL repos are up-to-date (to ensure we catch all pushes if many repos are being pushed to), then creates a snapshot. A snapshot is a full copy of all repository contents at specific SHAs, that can be tested from without being updated.
- Some number of build threads. They look for any unbuilt repo from the last snapshot and built it (so we can get lint results and test the built sims).
- A server on port 45366, which takes requests:
- /next-test: returns a browser-based test (contains information on the snapshot, test, and URL), for consumption by continuous-loop.html
- /test-result: accepts test results to be recorded (sent by continuous-loop.js)
- /results: returns a summary of test results, for consumption by continuous-report.html
A browser visiting continuous-loop.html will begin an infinite loop of requesting a test (/next-test request to the server), running that test in an iframe, forwarding pass/fail results to /test-result, and moving to the next test.
- Snapshot time may not be accurate. Should change to the time when the last-clean "SHA-check" pass started.
- Allow display of SHAs (in dependencies.json format) so devs can check out exact snapshots.
- Prune old results (only display ~20 snapshots)
- Allow tagging browsers with a name (or display user-agent) so tests can be traced back to specific devices to reproduce.
- Handle cloning of missing repos.
- Persistence between server restarts (currently need to wipe snapshot directories after the server process exits)
- Don't require administrator mode for Windows (that's scary!)
- phet-io tests
grunt lint-everything
- Launch examples/doc pages/playgrounds for scenery/kite/dot, and assorted other files to ensure we don't break them with sherpa/lib changes.
- Color profile pages (e.g. molecule-shapes-colors_en.html)
- Sim-specific unit tests
- file: protocol
- Basic iframe tests
- Touch fuzzing (with one or more 'fingers')
- Other locales (in unbuilt and built)
- Other builds, tesing locales/brand/mangle/uglify/allHTML.
- Runtime flags, such as:
- accessibility
- brand
- eall
- phet-app
- phet-android-app
- locale
- profiler
- rootRenderer (canvas, svg, dom, webgl)
- showCanvasNodeBounds/showFittedBlockBounds/showPointerAreas/showPointers/showVisibleBounds/dev
- stringTest
- webgl=false
- Test with screenshot references (to determine if we have visual breaks)
- Test with phet-io recorded references (to determine if we have model breaks)