Allow Skylark test rules to emit test/benchmark details for summary

[schroederc]

It'd be nice if Skylark test rules could emit details about the tests it ran and have them appear in Bazel's test summary.

For instance, a Skylark test rule for Go could parse the test binary's output for which exact tests passed/failed, how long each took, and any log output. It could then be relayed back to Bazel through a special file (possibly JSON) which can be used to populate each TestResultData proto and be displayed to the user.

Likewise, I think Bazel should support benchmarks in a similar way (perhaps through bazel benchmark instead of bazel test).

[hanwen]

Most of this already exists. Your test should emit a .xml file in a schema that

https://github.com/google/bazel/blob/master/src/main/java/com/google/devtools/build/lib/rules/test/TestXmlOutputParser.java

understands.

[schroederc]

Neat. Is this documented/stable? How does a Skylark rule mark a file to be parsed by Bazel?

[hanwen]

AFAICR, the test framework should just emit the .XML file, and it should work automatically. The support has been there for years, so it shoul dbe pretty stable.

[schroederc]

Where should the file be emitted and what should it be named? Tests are run in their respective runfiles directory so I'm assuming the file should be relative to that.

Can sh_tests do the same thing?

[hanwen]

https://github.com/google/bazel/blob/master/src/main/java/com/google/devtools/build/lib/rules/test/StandaloneTestStrategy.java#L178

it should have the same name as the test log file, but with xml as an extension. Come to think of it, the file name should probably be passed as an environment variable somewhere; I will have a look at the internal variant.

[damienmg]

in sh_test the good variable is XML_OUTPUT_FILE (see https://github.com/google/bazel/blob/master/src/test/shell/unittest.bash#L450)

It should be the same for other test runner I believe.

[schroederc]

XML_OUTPUT_FILE is not set in a sh_test. These are the only variables set at the beginning of a sh_test:

BASH
BASHOPTS
BASH_ALIASES
BASH_ARGC
BASH_ARGV
BASH_CMDS
BASH_LINENO
BASH_SOURCE
BASH_VERSINFO
BASH_VERSION
DIRSTACK
EUID
GROUPS
HOSTNAME
HOSTTYPE
IFS
MACHTYPE
OPTERR
OPTIND
OSTYPE
PATH
PIPESTATUS
PPID
PS4
PWD
SHELL
SHELLOPTS
SHLVL
TERM
TEST_SIZE
TEST_SRCDIR
TEST_TIMEOUT
TEST_TMPDIR
TMPDIR
TZ
UID
_

[hanwen]

b6f557b