Using cri-tools for testing CRI-enabled runtimes

[alban]

Following discussion on opencontainers/oci-conformance#36 and discussions with @crosbymichael and @caniszczyk, I am exploring if cri-tools could help to run OCI runtime tests on containerd, docker and other container engines supporting CRI.

The runtime operations in CRI (defined by the service RuntimeService) and the operations in the OCI runtime spec have some similarities:

OCI's runtime operations	CRI's RuntimeService methods
state	ContainerStatus
create	CreateContainer
start	StartContainer
kill	StopContainer
delete	RemoveContainer

But it does not perfectly match in the semantics and in the parameters. I will list some discrepancies below.

[alban]

• OCI's create must have the first pid in the container running and waiting for the start before exec'ing the command (see state). CRI's CreateContainer returns the id of the new container but the process is not running yet. At least with the dockershim CRI.
• OCI's delete MUST error if the container is running. But CRI's RemoveContainer must forcibly remove the container if running.
• The parameters for creating a container are completely different: in CRI, CreateContainerRequest contains a ContainerConfig which does not match OCI's config.json.
  • As an example, NamespaceOption is a subset of possible OCI namespaces configurations (it can only configure network, pid, ipc namespaces) but it also offer configuration about pods that are not exposed in OCI because it's at a different layer.
• Tests in runtime-tools generate a config.json and the process runtimetest in the container reads the config.json and checks if it matches its current configuration. If we use cri-tools to start a container with a specific ContainerConfig configuration instead of generating the config.json, runtimetest does not know how to read the CRI ContainerConfig. So, runtimetest would need to be rewritten for a lot of tests.
• cri-tools has its own validation tests (example with mount propagation which has similarities with tests in OCI runtime-tools' runtimetest) but it's testing the CRI interface and not the OCI runtime spec.

[caniszczyk]

RFC @opencontainers/runtime-spec-maintainers @opencontainers/runc-maintainers @opencontainers/runtime-tools-maintainers