Skip to content

Commit

Permalink
docs: Address review comments
Browse files Browse the repository at this point in the history
Address review comments written by @wking.

* Add a blank line between non-manpage refs and manpage refs.
* Add backticks for `git status`.
* Append a punctuation mark to each list item.
* Use `pull request` instead of `PR`.
* Specify `console` format to the example commands
* Use relative links instead of static URIs
* Remove a trailing blank
* Remove trailing whitespaces
* Run with sudo whenever it's possible
* Use correct relative links
* Move sections for running runtime validation one level up to the
  section `Running the runtime validation suite`.
* Use `command -v` instead of `which`.
* Elaborate why we should specify the TAP variable.
  • Loading branch information
Dongsu Park committed Aug 22, 2018
1 parent ea0a3b2 commit 7203f73
Show file tree
Hide file tree
Showing 3 changed files with 55 additions and 53 deletions.
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,6 @@ Use the [runtime validation suite](doc/runtime-compliance-testing.md).
[config.json]: https://github.com/opencontainers/runtime-spec/blob/master/config.md
[runC]: https://github.com/opencontainers/runc
[runtime-spec]: https://github.com/opencontainers/runtime-spec

[generate.1]: man/oci-runtime-tool-generate.1.md
[validate.1]: man/oci-runtime-tool-validate.1.md
51 changes: 25 additions & 26 deletions docs/devel/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,57 +8,56 @@ The following example assumes we're going from version 0.5.0 (`v0.5.0`) to 0.6.0
Let's get started:

- Start at the relevant milestone on GitHub (e.g. https://github.com/opencontainers/runtime-tools/milestones/v0.6.0): ensure all referenced issues are closed (or moved elsewhere, if they're not done). Close the milestone.
- runtime-tools does not use a [roadmap file](https://github.com/opencontainers/runtime-tools/issues/465) but GitHub milestones. Update the [other milestones](https://github.com/opencontainers/runtime-tools/milestones), if necessary
- Branch from the latest master, make sure your git status is clean
- Update the [VERSION](https://github.com/opencontainers/runtime-tools/blob/master/VERSION)
- runtime-tools does not use a [roadmap file](https://github.com/opencontainers/runtime-tools/issues/465) but GitHub milestones. Update the [other milestones](https://github.com/opencontainers/runtime-tools/milestones), if necessary.
- Branch from the latest master, make sure your `git status` is clean.
- Update the [VERSION](https://github.com/opencontainers/runtime-tools/blob/master/VERSION).
- Update the [release notes][changelog].
Try to capture most of the salient changes since the last release, but don't go into unnecessary detail (better to link/reference the documentation wherever possible).

Ensure the branch is correct:

- Ensure the build is clean!
- `git clean -ffdx && make && make test` should work
- Integration tests on CI should be green
- `git clean -ffdx && make && make test` should work.
- Integration tests on CI should be green.
- Check the version of the binaries:
- Check `./oci-runtime-tool --version`
- Check `./runtimetest --version`
- Check `./oci-runtime-tool --version`.
- Check `./runtimetest --version`.

Once everything is fine:

- File a PR
- Ensure the CI on the release PR is green
- File a pull request.
- Ensure the CI on the release PR is green.
- Send an email to the [mailing list][mailinglist] ([example for v0.5.0](https://groups.google.com/a/opencontainers.org/forum/#!topic/dev/iuWpWUai4_I)) and get reviews from other [maintainers][maintainers].
- Once the maintainers agree, merge the PR
- Once the maintainers agree, merge the PR.

Sign a tagged release and push it to GitHub:

- Add a signed tag: `git tag -s v0.6.0 -m "release v0.6.0"`
- Push the tag to GitHub: `git push origin v0.6.0`
- Add a signed tag: `git tag -s v0.6.0 -m "release v0.6.0"`.
- Push the tag to GitHub: `git push origin v0.6.0`.

Now we switch to the GitHub web UI to conduct the release:

- Start a [new release][gh-new-release] on Github
- Tag "v0.6.0", release title "v0.6.0"
- Copy-paste the release notes you added earlier in [CHANGELOG.md][changelog]
- Start a [new release][gh-new-release] on Github.
- Tag "v0.6.0", release title "v0.6.0".
- Copy-paste the release notes you added earlier in [CHANGELOG.md][changelog].
- Attach the release.
This is a simple tarball:

```
export VER="1.2.0"
export NAME="runtime-tools-v$VER"
mkdir -p $NAME/validation
cp oci-runtime-tool $NAME/
cp validation/*.t $NAME/validation/
sudo chown -R root:root $NAME/
tar czvf $NAME.tar.gz --numeric-owner $NAME/
```console
$ export VER="1.2.0"
$ export NAME="runtime-tools-v$VER"
$ mkdir -p $NAME/validation
$ cp oci-runtime-tool $NAME/
$ cp validation/*.t $NAME/validation/
$ sudo chown -R root:root $NAME/
$ tar czvf $NAME.tar.gz --numeric-owner $NAME/
```

- Publish the release!

- Clean your git tree: `sudo git clean -ffdx`.

[changelog]: https://github.com/opencontainers/runtime-tools/blob/master/CHANGELOG.md
[maintainers]: https://github.com/opencontainers/runtime-tools/blob/master/MAINTAINERS
[changelog]: ../../CHANGELOG.md
[maintainers]: ../../MAINTAINERS
[mailinglist]: https://groups.google.com/a/opencontainers.org/forum/#!forum/dev
[gh-new-release]: https://github.com/opencontainers/runtime-tools/releases/new

56 changes: 29 additions & 27 deletions docs/runtime-compliance-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,29 +14,15 @@ If your distribution does not package node-tap, you can install [npm][] (for exa
```console
$ npm install tap
```
### From a release

Check if your release has pre-compiled tests on the [release page][releases] page.

```
$ tar xf runtime-tools-v0.6.0.tar.gz
$ RUNTIME=runc tap ./runtime-tools-v0.6.0/validation/*.t
```

### From source

Build the validation executables:

```console
$ make runtimetest validation-executables
```

Runtime validation currently [only supports](docs/runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](doc/command-line-interface.md).
Runtime validation currently [only supports](runtime-compliance-testing.md) the [OCI Runtime Command Line Interface](command-line-interface.md).
If we add support for alternative APIs in the future, runtime validation will gain an option to select the desired runtime API.
For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](doc/command-line-interface.md)).
For the command line interface, the `RUNTIME` option selects the runtime command (`funC` in the [OCI Runtime Command Line Interface](command-line-interface.md)).

Note that we should specify the `TAP` variable to the normal user's tap command, because usually the normal user had a newer version of `tap` installed locally via npm, for example, under `$HOME/node_modules/.bin`.

```
$ sudo TAP="$(which tap)" RUNTIME=runc make localvalidation
$ sudo TAP="$(command -v tap)" RUNTIME=runc make localvalidation
RUNTIME=runc /home/alban/.nvm/versions/node/v9.7.1/bin/tap validation/pidfile.t validation/linux_cgroups_memory.t validation/linux_rootfs_propagation_shared.t validation/kill.t validation/linux_readonly_paths.t validation/hostname.t validation/hooks_stdin.t validation/create.t validation/poststart.t validation/linux_cgroups_network.t validation/poststop_fail.t validation/prestart_fail.t validation/linux_cgroups_relative_blkio.t validation/default.t validation/poststop.t validation/linux_seccomp.t validation/prestart.t validation/process_rlimits.t validation/linux_masked_paths.t validation/killsig.t validation/process.t validation/linux_cgroups_relative_pids.t validation/hooks.t validation/linux_rootfs_propagation_unbindable.t validation/linux_cgroups_relative_cpus.t validation/misc_props.t validation/linux_sysctl.t validation/process_oom_score_adj.t validation/linux_devices.t validation/process_capabilities_fail.t validation/start.t validation/linux_cgroups_pids.t validation/process_capabilities.t validation/poststart_fail.t validation/linux_cgroups_relative_hugetlb.t validation/mounts.t validation/linux_cgroups_hugetlb.t validation/linux_cgroups_relative_memory.t validation/state.t validation/root_readonly_true.t validation/linux_cgroups_blkio.t validation/delete.t validation/linux_cgroups_relative_network.t validation/process_rlimits_fail.t validation/linux_cgroups_cpus.t validation/linux_uid_mappings.t
validation/pidfile.t .................................. 1/1 455ms
validation/linux_cgroups_memory.t ..................... 9/9
Expand Down Expand Up @@ -101,7 +87,7 @@ validation/linux_seccomp.t ............................ 0/1
not ok validation/linux_seccomp.t
error: >-
Pre-start hooks MUST be called after the `start` operation is called
Refer to:
https://github.com/opencontainers/runtime-spec/blob/v1.0.0/config.md#prestart
Expand Down Expand Up @@ -208,10 +194,10 @@ validation/process_capabilities_fail.t .............. 20/27
validation/start.t .....exit status 2
validation/start.t .................................... 6/7
not ok test count !== plan
+++ found
--- wanted
-1
+6
+++ found
--- wanted
-1
+6
results:
ok: false
count: 6
Expand Down Expand Up @@ -266,7 +252,7 @@ validation/linux_cgroups_blkio.t .................... 15/15
validation/delete.t ................................... 3/5 22s
not ok attempting to `delete` a container that is not `stopped` MUST generate an error
reference: 'https://github.com/opencontainers/runtime-spec/blob/v1.0.0/runtime.md#delete'
not ok attempting to `delete` a container that is not `stopped` MUST have no effect on the container
error: exit status 1
reference: 'https://github.com/opencontainers/runtime-spec/blob/v1.0.0/runtime.md#delete'
Expand Down Expand Up @@ -294,7 +280,7 @@ validation/linux_uid_mappings.t ..................... 20/26
syscall action SCMP_ACT_ALLOW
total ............................................. 420/517
420 passing (1m)
88 pending
Expand Down Expand Up @@ -340,7 +326,7 @@ ok 26 - gid mappings
or with the environment variable `VALIDATION_TESTS`:

```console
$ sudo make TAP=$(which tap) RUNTIME=runc VALIDATION_TESTS=validation/default.t localvalidation
$ sudo make TAP=$(command -v tap) RUNTIME=runc VALIDATION_TESTS=validation/default.t localvalidation
RUNTIME=runc /home/alban/.nvm/versions/node/v9.7.1/bin/tap validation/default.t
validation/default.t ................................ 20/26
Skipped: 6
Expand Down Expand Up @@ -368,6 +354,22 @@ Files=2, Tests=35, 1 wallclock secs ( 0.03 usr 0.00 sys + 0.12 cusr 0.12 csy
Result: PASS
```

### From a release

Check if your release has pre-compiled tests on the [release page][releases] page.

```console
$ tar xf runtime-tools-v0.6.0.tar.gz
$ sudo RUNTIME=runc tap ./runtime-tools-v0.6.0/validation/*.t
```

### From source

Build the validation executables:

```console
$ make runtimetest validation-executables
```

[compliance]: https://github.com/opencontainers/runtime-spec/blob/v1.0.1/spec.md
[debian-node-tap]: https://packages.debian.org/stretch/node-tap
Expand Down

0 comments on commit 7203f73

Please sign in to comment.