runtimetest: Granular TAP output by wking · Pull Request #1 · mrunalp/runtime-tools

wking · 2017-01-11T21:20:31Z

Granular TAP output makes it easier to see what's being tested (and what's going wrong). You could feed this into any TAP harness you like, but I've chosen prove (it seems popular). You need a TAP harness looking for test failures, because runtimetest now returns zero when it successfully runs the tests (regardless of whether the tests all pass).

The OPTIND shift is out of getopts(1p).

I've dropped the “VALIDATING RUNTIME” header since prove displays the command it's running in its usual output (e.g. /usr/bin/runc .. ok).

Full example output:

/usr/bin/runc .. 
TAP version 13
ok 1 - # SKIP root.readonly falsy
ok 2 - hostname matches expected value
ok 3 - mount "/proc" found
ok 4 - mount "/dev" found
ok 5 - mount "/dev/pts" found
ok 6 - mount "/dev/shm" found
ok 7 - mount "/dev/mqueue" found
ok 8 - mount "/sys" found
ok 9 - expected capability CAP_CHOWN set
ok 10 - expected capability CAP_DAC_OVERRIDE set
ok 11 - unexpected capability CAP_DAC_READ_SEARCH not set
ok 12 - expected capability CAP_FOWNER set
ok 13 - expected capability CAP_FSETID set
ok 14 - expected capability CAP_KILL set
ok 15 - expected capability CAP_SETGID set
ok 16 - expected capability CAP_SETUID set
ok 17 - expected capability CAP_SETPCAP set
ok 18 - unexpected capability CAP_LINUX_IMMUTABLE not set
ok 19 - expected capability CAP_NET_BIND_SERVICE set
ok 20 - unexpected capability CAP_NET_BROADCAST not set
ok 21 - unexpected capability CAP_NET_ADMIN not set
ok 22 - expected capability CAP_NET_RAW set
ok 23 - unexpected capability CAP_IPC_LOCK not set
ok 24 - unexpected capability CAP_IPC_OWNER not set
ok 25 - unexpected capability CAP_SYS_MODULE not set
ok 26 - unexpected capability CAP_SYS_RAWIO not set
ok 27 - expected capability CAP_SYS_CHROOT set
ok 28 - unexpected capability CAP_SYS_PTRACE not set
ok 29 - unexpected capability CAP_SYS_PACCT not set
ok 30 - unexpected capability CAP_SYS_ADMIN not set
ok 31 - unexpected capability CAP_SYS_BOOT not set
ok 32 - unexpected capability CAP_SYS_NICE not set
ok 33 - unexpected capability CAP_SYS_RESOURCE not set
ok 34 - unexpected capability CAP_SYS_TIME not set
ok 35 - unexpected capability CAP_SYS_TTY_CONFIG not set
ok 36 - expected capability CAP_MKNOD set
ok 37 - unexpected capability CAP_LEASE not set
ok 38 - expected capability CAP_AUDIT_WRITE set
ok 39 - unexpected capability CAP_AUDIT_CONTROL not set
ok 40 - expected capability CAP_SETFCAP set
ok 41 - unexpected capability CAP_MAC_OVERRIDE not set
ok 42 - unexpected capability CAP_MAC_ADMIN not set
ok 43 - unexpected capability CAP_SYSLOG not set
ok 44 - unexpected capability CAP_WAKE_ALARM not set
ok 45 - unexpected capability CAP_BLOCK_SUSPEND not set
ok 46 - unexpected capability CAP_AUDIT_READ not set
ok 47 - lstat default symlink /dev/fd
ok 48 - default symlink /dev/fd is a symlink
ok 49 - default symlink /dev/fd has expected target
ok 50 - lstat default symlink /dev/stdin
ok 51 - default symlink /dev/stdin is a symlink
ok 52 - default symlink /dev/stdin has expected target
ok 53 - lstat default symlink /dev/stdout
ok 54 - default symlink /dev/stdout is a symlink
ok 55 - default symlink /dev/stdout has expected target
ok 56 - lstat default symlink /dev/stderr
ok 57 - default symlink /dev/stderr is a symlink
ok 58 - default symlink /dev/stderr has expected target
ok 59 - mount /proc has expected type
ok 60 - mount /sys has expected type
ok 61 - mount /dev/pts has expected type
ok 62 - mount /dev/shm has expected type
ok 63 - stat default device /dev/null
ok 64 - default device /dev/null is a device
ok 65 - stat default device /dev/zero
ok 66 - default device /dev/zero is a device
ok 67 - stat default device /dev/full
ok 68 - default device /dev/full is a device
ok 69 - stat default device /dev/random
ok 70 - default device /dev/random is a device
ok 71 - stat default device /dev/urandom
ok 72 - default device /dev/urandom is a device
ok 73 - stat default device /dev/tty
ok 74 - default device /dev/tty is a device
ok 75 - stat default device /dev/ptmx
ok 76 - default device /dev/ptmx is a device
ok 77 - # SKIP linux.devices (no devices configured)
ok 78 - has expected working directory
ok 79 - has expected environment variable PATH
ok 80 - has expected environment variable TERM
ok 81 - has expected user ID
ok 82 - has expected group ID
ok 83 - has expected number of process arguments
ok 84 - has expected process argument 0
ok 85 - has expected noNewPrivileges
ok 86 - # SKIP linux.resources.oomScoreAdj falsy
ok 87 - has expected soft RLIMIT_NOFILE
ok 88 - has expected hard RLIMIT_NOFILE
ok 89 - # SKIP linux.uidMappings checks (no mappings specified)
ok 90 - # SKIP linux.gidMappings checks (no mappings specified)
1..90
ok
All tests successful.
Files=1, Tests=90,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.02 cusr  0.00 csys =  0.04 CPU)
Result: PASS

And failing (if I just run runtimetest in my host namespace):

$ oci-runtime-tool generate >config.json
$ ./runtimetest
TAP version 13
ok 1 - # SKIP root.readonly falsy
not ok 2 - hostname matches expected value
# expected mrsdalloway, actual myhost
ok 3 - mount "/proc" found
not ok 4 - expected mount "/dev" found
ok 5 - mount "/dev/pts" found
ok 6 - mount "/dev/shm" found
ok 7 - mount "/dev/mqueue" found
ok 8 - mount "/sys" found
not ok 9 - expected capability CAP_CHOWN set
not ok 10 - expected capability CAP_DAC_OVERRIDE set
ok 11 - unexpected capability CAP_DAC_READ_SEARCH not set
not ok 12 - expected capability CAP_FOWNER set
not ok 13 - expected capability CAP_FSETID set
not ok 14 - expected capability CAP_KILL set
not ok 15 - expected capability CAP_SETGID set
not ok 16 - expected capability CAP_SETUID set
not ok 17 - expected capability CAP_SETPCAP set
ok 18 - unexpected capability CAP_LINUX_IMMUTABLE not set
not ok 19 - expected capability CAP_NET_BIND_SERVICE set
ok 20 - unexpected capability CAP_NET_BROADCAST not set
ok 21 - unexpected capability CAP_NET_ADMIN not set
not ok 22 - expected capability CAP_NET_RAW set
ok 23 - unexpected capability CAP_IPC_LOCK not set
ok 24 - unexpected capability CAP_IPC_OWNER not set
ok 25 - unexpected capability CAP_SYS_MODULE not set
ok 26 - unexpected capability CAP_SYS_RAWIO not set
not ok 27 - expected capability CAP_SYS_CHROOT set
ok 28 - unexpected capability CAP_SYS_PTRACE not set
ok 29 - unexpected capability CAP_SYS_PACCT not set
ok 30 - unexpected capability CAP_SYS_ADMIN not set
ok 31 - unexpected capability CAP_SYS_BOOT not set
ok 32 - unexpected capability CAP_SYS_NICE not set
ok 33 - unexpected capability CAP_SYS_RESOURCE not set
ok 34 - unexpected capability CAP_SYS_TIME not set
ok 35 - unexpected capability CAP_SYS_TTY_CONFIG not set
not ok 36 - expected capability CAP_MKNOD set
ok 37 - unexpected capability CAP_LEASE not set
not ok 38 - expected capability CAP_AUDIT_WRITE set
ok 39 - unexpected capability CAP_AUDIT_CONTROL not set
not ok 40 - expected capability CAP_SETFCAP set
ok 41 - unexpected capability CAP_MAC_OVERRIDE not set
ok 42 - unexpected capability CAP_MAC_ADMIN not set
ok 43 - unexpected capability CAP_SYSLOG not set
ok 44 - unexpected capability CAP_WAKE_ALARM not set
ok 45 - unexpected capability CAP_BLOCK_SUSPEND not set
ok 46 - unexpected capability CAP_AUDIT_READ not set
ok 47 - lstat default symlink /dev/fd
ok 48 - default symlink /dev/fd is a symlink
ok 49 - default symlink /dev/fd has expected target
ok 50 - lstat default symlink /dev/stdin
ok 51 - default symlink /dev/stdin is a symlink
ok 52 - default symlink /dev/stdin has expected target
ok 53 - lstat default symlink /dev/stdout
ok 54 - default symlink /dev/stdout is a symlink
ok 55 - default symlink /dev/stdout has expected target
ok 56 - lstat default symlink /dev/stderr
ok 57 - default symlink /dev/stderr is a symlink
ok 58 - default symlink /dev/stderr has expected target
ok 59 - mount /dev/pts has expected type
ok 60 - mount /dev/shm has expected type
ok 61 - mount /proc has expected type
ok 62 - mount /sys has expected type
ok 63 - stat default device /dev/null
ok 64 - default device /dev/null is a device
ok 65 - stat default device /dev/zero
ok 66 - default device /dev/zero is a device
ok 67 - stat default device /dev/full
ok 68 - default device /dev/full is a device
ok 69 - stat default device /dev/random
ok 70 - default device /dev/random is a device
ok 71 - stat default device /dev/urandom
ok 72 - default device /dev/urandom is a device
ok 73 - stat default device /dev/tty
ok 74 - default device /dev/tty is a device
ok 75 - stat default device /dev/ptmx
ok 76 - default device /dev/ptmx is a device
ok 77 - # SKIP linux.devices (no devices configured)
not ok 78 - has expected working directory
# working directory expected: /, actual: /home/wking/.local/lib/go/src/github.com/opencontainers/runtime-tools
not ok 79 - has expected environment variable PATH
# environment variable PATH expected: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin, actual: /home/wking/bin:/home/wking/.local/bin:/home/wking/.gem/ruby/2.1.0/bin:/home/wking/.gem/ruby/2.0.0/bin:/usr/local/bin:/usr/bin:/bin:/opt/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.9.3:/usr/sbin:/sbin
ok 80 - has expected environment variable TERM
not ok 81 - has expected user ID
# user ID expected: 0, actual 1000
not ok 82 - has expected group ID
# group ID expected: 0, actual 1000
ok 83 - has expected number of process arguments
not ok 84 - has expected process argument 0
# expected process argument 0: sh, actual: init [3]
ok 85 - has expected noNewPrivileges
ok 86 - # SKIP linux.resources.oomScoreAdj falsy
ok 87 - has expected soft RLIMIT_NOFILE
not ok 88 - has expected hard RLIMIT_NOFILE
# hard RLIMIT_NOFILE expected: 1024, actual: 4096
ok 89 - # SKIP linux.uidMappings checks (no mappings specified)
ok 90 - # SKIP linux.gidMappings checks (no mappings specified)
1..90

wking · 2017-01-11T21:42:35Z

You could also pull the TAP harness outside of test_runtime.sh, and suggest folks run:

$ prove sudo test_runtime.sh

or similar. That way folks can easily swap in their own preferred harness. Let me know if you want me to reroll taking that approach.

wking · 2017-01-13T05:17:00Z

Better skip support is on the way with mndrix/tap-go#6, so it's probably best to hold off here until that lands and I reroll this to take advantage of it.

wking · 2017-01-14T15:19:21Z

I've pushed b462307 → f3c0d27 with .Skip handling and prove pulled out of test_runtime.sh.

This reverts commit 67daea2. We won't need these with granular TAP handling. Conflicts: Godeps/Godeps.json Signed-off-by: W. Trevor King <wking@tremily.us>

I couldn't figure out how to do this with 'godep update', so I just copied the files over and bumped the hash by hand. Signed-off-by: W. Trevor King <wking@tremily.us>

Granular TAP output makes it easier to see what's being tested (and what's going wrong). You could feed this into any TAP harness you like, but I've chosen prove (it seems popular). You need a TAP harness looking for test failures, because runtimetest now returns zero when it successfully runs the tests (regardless of whether the tests all pass). With this commit, runtimetest will exit zero when it can successfully *run the tests* (regardless of whether all the tests pass). A TAP harness like prove should be used to consume the output TAP to decide if validation succeeded or not. To avoid confusing the TAP consumer, all of the non-TAP logging goes to stderr (and not stdout). Signed-off-by: W. Trevor King <wking@tremily.us>

wking · 2017-01-17T04:57:01Z

Shifted to opencontainers#308.

wking force-pushed the granular-tap branch from 0e66689 to 87bd1e3 Compare January 11, 2017 21:24

wking mentioned this pull request Jan 11, 2017

Tap output from runtime test opencontainers/runtime-tools#302

Merged

wking added 3 commits January 16, 2017 20:50

Revert "godep: Add go-multierror dependency"

168230f

This reverts commit 67daea2. We won't need these with granular TAP handling. Conflicts: Godeps/Godeps.json Signed-off-by: W. Trevor King <wking@tremily.us>

godep: Bump tap-go to pick up the Skip method

7d409aa

I couldn't figure out how to do this with 'godep update', so I just copied the files over and bumped the hash by hand. Signed-off-by: W. Trevor King <wking@tremily.us>

wking force-pushed the granular-tap branch from f3c0d27 to cf0d014 Compare January 17, 2017 04:51

wking closed this Jan 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtimetest: Granular TAP output#1

runtimetest: Granular TAP output#1
wking wants to merge 3 commits intomrunalp:tap_runtime_testfrom
wking:granular-tap

wking commented Jan 11, 2017 •

edited

Loading

Uh oh!

wking commented Jan 11, 2017

Uh oh!

wking commented Jan 13, 2017

Uh oh!

wking commented Jan 14, 2017

Uh oh!

wking commented Jan 17, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

wking commented Jan 11, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wking commented Jan 11, 2017

Uh oh!

wking commented Jan 13, 2017

Uh oh!

wking commented Jan 14, 2017

Uh oh!

wking commented Jan 17, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

wking commented Jan 11, 2017 •

edited

Loading