runtimespec VS ocitools

[liangchenye]

Filesystem bundle

• code This REQUIRED file MUST reside in the root of the bundle directory.
• code config.json : This REQUIRED file, which MUST be named config.json.
• code While these artifacts MUST all be present in a single directory on the local filesystem.
• code This directory MUST be referenced from within the config.json file.

Runtime and Lifecycle

• 🌓 code The container's runtime environment MUST be created according to the configuration in config.json.
• code Any updates to config.json after container is running MUST not affect the container.
• WIP The user specified process MUST be executed in the container.
• The container MUST be destroyed by undoing the steps performed during create phase (step 2)
• WIP Query State: state
• WIP This operation MUST generate an error if it is not provided the ID of a container.
• WIP This operation MUST return the state of a container as specified in the State section. In particular, the state MUST be serialized as JSON.

Start 🔒 discuss

• WIP This operation MUST generate an error if it is not provided a path to the bundle and the container ID to associate with the container.
• WIP If the ID provided is not unique across all containers within the scope of the runtime, or is not valid in any other way, the implementation MUST generate an error.
• code Using the data in config.json, that are in the bundle's directory, this operation MUST create a new container.This includes creating the relevant namespaces, resource limits, etc and configuring the appropriate capabilities for the container.
• code A new process within the scope of the container MUST be created as specified by the config.json file otherwise an error MUST be generated.
• WIP Attempting to start an already running container MUST have no effect on the container and MUST generate an error.

Kill

• This operation MUST generate an error if it is not provided the container ID.
• Attempting to send a signal to a container that is not running MUST have no effect on the container and MUST generate an error.
• This operation MUST send the specified signal to the process in the container.
• When the process in the container is stopped, irrespective of it being as a result of a kill operation or any other reason, the status property of this container MUST be stopped.

Delete

• This operation MUST generate an error if it is not provided the container ID.
• Attempting to delete a container that does not exist MUST generate an error.
• Attempting to delete a container whose process is still running MUST generate an error.
• Deleting a container MUST delete the resources that were created during the create step.
• Note that resources associated with the container, but not created by this container, MUST NOT be deleted.
• Once a container is deleted its ID MAY be used by a subsequent container.

Configuration

Specification version

• code ociVersion (string, required) must be in SemVer v2.0.0 format.

Root

• code On Windows, for Windows Server Containers, this field is REQUIRED and MUST be specified as a volume GUID path. For Hyper-V Containers, this field MUST be omitted.
• code On all other platforms, this field is REQUIRED. The value SHOULD be the conventional rootfs.
• code path (string, required) A directory MUST exist at the relative path declared by the field.
• code readonly (bool, optional) If true then the root filesystem MUST be read-only inside the container.
• code readonly Defaults to false.
• [code] On Windows, this field MUST be omitted or false.

Mounts

• WIP The runtime MUST mount entries in the listed order.
• Destination of mount point: path inside container. For the Windows operating system, one mount destination MUST NOT be nested within another mount. (Ex: c:\foo and c:\foo\bar).
• code destination (required)
• code mounts type (string, required)
• code source (string, required)

Process

• code process cwd (string, required) is the working directory that will be set for the executable. This value MUST be an absolute path.

• code Runtimes MUST ignore consoleSize if terminal is false or unset.

• code process env (array of strings, optional) Elements in the array are specified as Strings in the form "KEY=value". The left hand side must consist solely of letters, digits, and underscores _ .

• code 'capabilities` Any value which cannot be mapped to a relevant kernel interface MUST cause an error.

• code If oomScoreAdj is set, the runtime MUST set oom_score_adj to the given value.

• code If oomScoreAdj is not set, the runtime MUST NOT change the value of oom_score_adj.

• code If rlimits contains duplicated entries with same type, the runtime MUST error out.

Platform-specific configuration

 NOT IN RC5
 windows (object, OPTIONAL) Windows-specific configuration. This MUST be set if the target platform of this spec is windows.

• code Platform-specific configuration os (string, required) $GOOS.
• code Platform-specific configuration arch (string, required) $GOARCH.
• The runtime MUST generate an error if it does not support the configured os.
• The runtime MUST generate an error if it does not support the configured arch.
• Hooks Hooks MUST be called in the listed order.
• code Hooks path is required for a hook, args and env are optional.
• code Hooks paths are absolute
• [go map] Annotations MUST be a key-value map
• [go map] Keys MUST be strings. Keys MUST NOT be an empty string.
• [go map] Values MUST be strings.
• code Keys using the org.opencontainers namespace are reserved and MUST NOT be used by subsequent specifications.
• code Implementations that are reading/processing this configuration file MUST NOT generate an error if they encounter an unknown annotation key.
• code Runtimes that are reading or processing this configuration file MUST NOT generate an error if they encounter an unknown property. Instead they MUST ignore unknown properties.
• code Runtimes that are reading or processing this configuration file MUST generate an error when invalid or unsupported values are encountered.

Linux-specific Container Configuration

Default Filesystems

• code The following filesystems SHOULD be made available in each container's filesystem:

Path    Type
/proc   procfs
/sys    sysfs
/dev/pts    devpts
/dev/shm    tmpfs

Namespaces

• code path (string, OPTIONAL) - an absolute path to namespace file in the runtime mount namespace.
• The runtime MUST place the container process in the namespace associated with that path. The runtime MUST generate an error if path is not associated with a namespace of type.
• If path is not specified, the runtime MUST create a new container namespace of type type.
• If a namespace type is not specified in the namespaces array, the container MUST inherit the runtime namespace of that type.
• code If a namespaces field contains duplicated namespaces with same type, the runtime MUST generate an error.
• code hostname (optional) on Linux , you can only set this if your bundle creates a new UTS namespace
• code The runtime SHOULD NOT modify the ownership of referenced filesystems to realize the mapping.

Device Devices

• code In addition to any devices configured with this setting, the runtime MUST also supply 'dev/nul' .....

• code lists devices that MUST be available in the container.

• code If a file already exists at path that does not match the requested device, the runtime MUST generate an error.

• code The same type, major and minor SHOULD NOT be used for multiple devices.

• code capabilities (array of strings, optional) Valid values are the strings for capabilities defined in the man page

• code rlimits (array of rlimits, optional) Valid values for the 'type' field are the resources defined in the man page.

• Control groups The runtime MUST apply entries in the listed order.

• Control groups You must specify at least one of weight or leafWeight in a given entry, and can specify both. Planed

• code JSON All configuration JSON MUST be encoded in UTF-8.

[liangchenye]

The original chart is done by @RobDolinMS, I just convert it to markdown format.
It helps ocitools to follow up spec, now only runtime spec.

There are still missing items. for some of them, I add [WIP] link after toggle button. (sorry, I just list mine)

[wking]

This list is lovely :). Initial notes:

On Fri, May 13, 2016 at 02:14:55AM -0700, 梁辰晔 (Liang Chenye) wrote:

• When the bundle is packaged up for distribution, this directory MUST be included.

You have this checked off now, but I'm not sure how we enforce that
without talking about distribution packaging. Can you point at the
enforcing code?

• WIP This operation MUST generate an error if it is not
  provided the ID of a container.

These are going to be impossible to test until we specify how errors
should be reported 1. See earlier discussion around 2 on why that
was punted from #225, but we'll still want to address it at some
point.

• Stopping a container MUST stop all of the processes running
  within the scope of the container.

This is going to be impossible to test until we specify “within the
scope of the container” 3. Instead of getting agreement on that,
I'm pushing for spec wording that narrows this to the container
process (singular) spawned by the runtime 4.

Exec: exec

This section is obsolete since 5.

 Subject: Definining "container" and "container processes" on Linux
 Date: Thu, 8 Oct 2015 09:47:08 -0700
 Message-ID: <20151008164708.GM28418@odin.tremily.us>

[initlove]

You have this checked off now,

sorry, it is not checked off in my local document. I'll correct it.
I'll update the check list to a check table which includes code links.

These are going to be impossible to test until we specify how errors
should be reported

In my WIP pr, I simply check if there is 'an' error.
Do/Are you mean/suggestion that there will/should be an 'error code' or 'error message list' in the runtime spec?

[wking]

On Sun, May 15, 2016 at 07:35:25PM -0700, David Liang wrote:

I'll update the check list to a check table which includes code
links.

Code links are going to go stale quickly, but I can't think of a
better way. At least links will give folks a lead for further ‘git
log -p -G …’ and stuff if they want to trace the check into the
future.

These are going to be impossible to test until we specify how
errors should be reported

In my WIP pr, I simply check if there is 'an' error. Do/Are you
mean/suggestion that there will/should be an 'error code' or 'error
message list' in the runtime spec?

There is at least some support for standardized error codes 1, and
that sounds like a good idea to me. But the problem here is bigger
than that. For example, in the spec section I referenced earlier 2:

In cases where the specified operation generates an error, this
specification does not mandate how, or even if, that error is
returned or exposed to the user of an implementation.

In the cases where the runtime does not expose the error to the user
(a compliant approach with that spec wording), how does the test suite
know an error “was generated”? This is part of the “we don't have an
API-spec” problem, because the interface between the test suite (in
this repo) and the runtime being tested is currently not covered by
any OCI spec.

[liangchenye]

Thanks @wking , I updated this issue by:

1. add 🔒 sign if an item is blocked by runtime spec
2. add ❓ if it is not clear (at least to me) in runtime spec
3. add 🌓 since cmd/runtimetest is huge.
4. remove 'exec'
5. add code links, this helps to set the correct implemented status.

[wking]

On Wed, May 18, 2016 at 01:53:26AM -0700, 梁辰晔 (Liang Chenye) wrote:

1. add code links, this helps to set the correct implemented status.

Code links should probably use permanent URLs (e.g. 1) and not float
with master (e.g. 2), because with the latter approach the line
number (or even file path) may go stale.

[liangchenye]

updated:

1. use permanent URLs as @wking suggested
2. mark utf8/cwd/env tasks to done

updated (5.24th)

1. uts hostname validation

updated (5.26th)

1. hook path should be absolute

updated (5.31st)

1. rootfs readonly

updated (6.26)

1. terminal changed to not fixed.
   Testing process.terminal? #109
2. remove 'stop', add 'delete' and 'kill'

[liangchenye]

Change task mount order to WIP

[liangchenye]

updated (7.26)

1. remove pre/post hook task
2. default file system / default device change to Done
3. add new task "While these artifacts MUST all be present in a single directory on the local filesystem"

[liangchenye]

Hi @wking, @mrunalp,
Most remain tasks could be solve by a test frame . @wking and I have different preferences. Can we make a decision of it, or maybe we can have a better solution?

[mrunalp]

Let's summarize what we have remaining and who is going to work on what so we can move forward :)

• Lifecycle tests
• Support for test cases using runtimetest

Are we saying that python is a good choice for these two? Or just for rewriting the validate code?

[wking]

On Wed, Jul 27, 2016 at 04:14:58PM -0700, Mrunal Patel wrote:

• Lifecycle tests
• Support for test cases using runtimetest

Are we saying that python is a good choice for these two? Or just
for rewriting the validate code?

I think Python and/or a shell harness are reasonable choices for
runtime validation [1](I think that's what you mean by “Lifecycle
tests”). I don't have a theoretical problem with using Go for this,
but I haven't found a Go test harness that you can build and install.

runtimetest currently tries to be a one-stop shop for “this container
is setup according to its config”, but I'm not sure that's a
sustainable approach 2. With a test harness of any kind, we can
keep runtimetest when it makes sense, but plug in other container-side
test code when we need to look at something else. Having the thing
launched inside the container to look around be a static binary is
nice (and runtimetest is that), but I'm also open to having more test
binaries, or using BusyBox, or whatever to handle a particular test
case.

I think Python is a good choice for the config validation (#98), since
handling JSON in shell would not be fun, and I haven't found a Go test
harness that you can build and install.

Generation is simple enough that I don't think it matters. The
current Go generation code works well enough that I doubt it's useful
to rewrite in another language.

[mrunalp]

@liangchenye WDYT about using python for the lifecycle tests? I am fine with anything as long as those planning to implement it are comfortable with it.

[liangchenye]

Similar work with #16, just two sides.

[liangchenye]

Can we vote on this? Golang VS Python (or others)
@wking @opencontainers/runtime-tools-maintainers

[wking]

On Thu, Nov 17, 2016 at 08:55:42AM -0800, 梁辰晔 (Liang Chenye) wrote:

Can we vote on this? Golang VS Python (or others)
@wking @opencontainers/runtime-tools-maintainers

For voting in general, I think we still need to merge project-template
into this repository. I've floated #274 to address that. Once you
have that PR's GOVERNANCE.md in place, you could vote on the path
forward, but I'm not sure how effective that would be. Voting is less
about consensus building and more about formally documenting an
already-achieved consensus. The up/down voting in GOVERNANCE.md is
also not good at chosing between multiple possible paths.

I think it's better to have maintainers chime in on what sort of
runtime testing harness they'd like to maintain. Then folks who are
excited enough about that approach can work up PRs ;). Requirements
that have been floated before include:

a. Having a test harness that can be installed on the host (vs. being
run from a runtime-tools checkout) 1.
b. Having a test harness that can ship as part of a single
oci-runtime-tool binary 2.

I think (a) is reasonable. While the stronger (b) might be nice to
have, I don't think it's worth the trouble [3,4].

Proposed runtime testing harnesses include:

i. Python's unittest
ii. Bats
iii. Sharness
iv. An in-house Go harness (#61)

And if you're really excited about (b), tap-go 5 is a fairly basic,
functional harness for generating TAP output. You'd still have to
build your own skip, etc., logic in by hand.

[liangchenye]

The previous PR #61 is too big and is not well designed, I closed that.

I think we may start to discuss how to design a suitable OCI runtime test case first.

[liangchenye]

The test harness includes: test cases, test flow, test tool.

Now we use runtimetest as the test tool and use test_runtime.sh as the test flow, we miss:

• different cases with bad configs, not just the default good one
• life cycle test, test_runtime.sh just covers 'start' now.
  We need to extend the runtime-tool:

1. wrap the 'test_runtime.sh' into a test unit
   --- this is the test case we need to discuss first.
2. add more test cases
3. add a simple test flow to run all the cases.

I think the case is something like this.

{
    "config": {"arch=x64"}
    "lifecycle": {
        "create":  "success",
        "start": "fail"
      }
}

config is the args which will be used by 'ocitool-runtime-tool generate'.
create/start is the lifecycle operation, 'success/fail' is the expected running result.

[wking]

On Tue, Nov 22, 2016 at 04:38:51AM -0800, 梁辰晔 (Liang Chenye) wrote:

{
"config": {"arch=x64"}
"lifecycle": {
"create": "success",
"start": "fail"
}
}

I don't see a need to define a domain-specific language for our test
cases. In sharness, the test you outline above would be:

test_expect_success 'Test arch=x64 fails to start' "
create_default_bundle --arch=x64
${RUNTIME} create arch-x64 &&
test_must_fail ${RUNTIME} start arch-x64 &&
${RUNTIME} kill --signal KILL arch-x64 &&
${RUNTIME} delete arch-x64
"

which seems compact enough to me without OCI-specific helpers beyond
create_default_bundle.

[liangchenye]

It will be much easier to maintain this process list after #451 been merged.
I'll rework on this after that.

[liangchenye]

close
tracked it in #572