robot testing of games

[vsoch]

@boomsbloom this isn't priority, but when we have some games established let's talk about if there is a way to automatically test them. I built an experiment robot to go through the jspsych experiments, and this is pretty easy to do because you can extract the acceptable responses (keyboard, click, etc) from the jspsych object. Success for the test means making it to the end of the experiment without any javascript errors, 404, etc. Games are likely different, but there might be a way.

[vsoch]

To start, I'll just validate the config.json files for the continuous integration.

[cbattista]

this would definitely be a great feature - would be really excellent for
our screener as well and i'd love to hear more about your robot

On Fri, Apr 15, 2016 at 10:59 AM, Vanessa Sochat notifications@github.com
wrote:

To start, I'll just validate the config.json files for the continuous
integration.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#6 (comment)

[vsoch]

If you cd into an experiment folder, you can just type

  expfactory --test

and the robot will run through it, and you can watch. It's the same robot that runs on circle, for example, if you poke around circle:

https://circleci.com/gh/expfactory/expfactory-experiments

The continuous integration typically has a few tests. First to validate the config.json files, again can be done on command line in a folder with:

  expfactory --validate

then to check that the circle.yml includes all experiments/surveys etc that should be tested, and then it clones the current master, compares to the PR being done, and tests the experiments that were changed using selenium. It's pretty cool, definitely useful to find errors you don't anticipate.

[jonathanicholas]

This should be a fairly straightforward port from what you have written for jspsych since I think the only difference will be taking a click from each sprite rather than from the jspsych object, right?

[vsoch]

Right - if you just tell me the object in the console to look at to get the current state (and calculate what to click / do next) I can probably get a robot up and running for it!