Description
I've been tinkering with a set of scripts that posts feedback to github. Here's a quick outline on how it works:
- There's a boolean called
POST_STATUS_TO_PR
in thenode-test-pull-request
job. Make sure that's checked and enter your PR. - Each worker as part of the
node-test-commit
will ping github and say they've started building/testing (as of now, see Unfoldmake-ci
in jenkins #229 for improvements in this regard). These workers are currently:ci/linter
,ci/freebsd
,ci/smartos
,ci/linux
,ci/plinux
,ci/windows-fanned
,ci/osx
ci/arm
andci/arm-fanned
. - Once the test suite completes we're checking if the test suite returned either unstable or success status. If that's the case we post a success and a text mentioning total tests run plus any skips (linter accts differently plus needs more work since we need tap output; see this example pr for work in progress).
- If the suite has any other status (currently only supports failed tests) we post the number of failed tests, skipped and total.
In order to achieve above I had to install a new jenkins plugin that gives us access to execute logic as part of the post-build phase. I'm using the xml api and parse output in python (for portability) when needed.
Have a look at a few of the above jobs (edit in jenkins) for how it currently works. There's a few things that still needs to be done, where at least one would be up for discussion:
- port bash stuff to windows
- unfold build steps (Unfold
make-ci
in jenkins #229) so we can distinguish between a compile and test error. - land changes to linter can output tap (tools: add tap output to cpplint node#3448 and wip PR for eslint)
- figure out how to safely store github credentials (currently using the personal token)
So, the last one is a bit of a tricky problem. Since we can't really trust input from a PR this is an issue. Jenkins has a few ways of storing passwords (credential store and global passwords) and tries to mask it in some cases; but that sandbox was pretty easy to escape from.