test: Stub in sharness framework #180

wking · 2016-08-03T06:16:57Z

Spun off from this discussion. This currently ignores most of the generate options and doesn't touch validate at all, but I thought I'd float it early for some feedback before sinking too much time into it. If the current work looks solid, I'm happy to land it now and follow up with additional PRs to fill it out, or to keep this PR open until the coverage is more complete.

Details on the individual changes in the commit messages.

mrunalp · 2016-08-03T23:06:27Z

test/t1001-generate-template.t

+	ocitools generate --template template | jq . >actual &&
+	cat <<-EOF >expected &&
+		{
+		  \"ociVersion\": \"1.0.0-rc1\",


I don't like the escaping that we do here :/

On Wed, Aug 03, 2016 at 04:06:28PM -0700, Mrunal Patel wrote:

@@ -0,0 +1,59 @@
+#!/bin/sh
+
+test_description='Test generate template'
+
+. ./sharness.sh
+
+test_expect_failure CAT,ECHO,HEAD,JQ 'Test ocitools generate --template with an empty template' "

echo '{}' >template &&

ocitools generate --template template | jq . >actual &&

cat <<-EOF >expected &&

{

\"ociVersion\": \"1.0.0-rc1\",

I don't like the escaping that we do here :/

There are only so many quotes we can use ;). Would you rather me wrap
the test in single quotes and use double quotes inside it? That makes
using variables in the test tricky, but we don't do any of that yet.

liangchenye · 2016-08-04T12:34:28Z

The sharness framework looks cool, I need time to absorb it.

It is different test frame topic with #66, right? It is more like a unit test. If so, I think we go test is a suitable and more nature choice since generate/validate are all writing in go.

wking · 2016-08-04T17:24:33Z

On Thu, Aug 04, 2016 at 05:34:29AM -0700, 梁辰晔 (Liang Chenye) wrote:

It is different test frame topic with
#66, right?

Yes. That is about “how do we validate a runtime?” and this is about
“how do we validate ocitools?”. I think they both benefit from an
off-the-shelf test harness, but you can use different harnesses for
each.

It is more like a unit test. If so, I think we go test is a
suitable and more nature choice since generate/validate are all
writing in go.

The goal of this PR is to test the ocitools command line as a black
box. I'd prefer ‘go test’ for testing the generation library API, but
that's not what I'm aiming at here.

liangchenye · 2016-08-08T11:06:51Z

+1 for the black box test

wking · 2016-10-12T14:45:33Z

Is this dead, or worth a rebase?

Mashimiao · 2016-10-13T06:04:23Z

Compared with importing sharness, how about importing a go based framework, like go-check?

wking · 2016-10-13T06:17:41Z

On Wed, Oct 12, 2016 at 11:04:24PM -0700, Ma Shimiao wrote:

Compared with importing sharness, how about importing a go based
framework, like go-check?

A shell-based test harness lets us use a very convenient API for
executing subprocesses: shell syntax. go-check [1,2] looks like a
nice, generic test framework, but you'd have to use os/exec 3 to
execute a command, manage its standard streams, and collect its
exit status. With sharness, you can take care of all of that with a single
line like 4:

ocitools --help | head -n2 >actual &&

Mashimiao · 2017-07-19T07:19:27Z

Please contribute to master branch

wking · 2017-07-19T14:54:42Z

Please contribute to master branch.

I can rebase onto master and change the target branch without filing a new PR and fragmenting discussion. Can you re-open this PR so I can do that?

wking · 2017-07-19T17:26:52Z

Redirected the PR and rebased onto master with d8f9c62 → b8d8357.

Mashimiao · 2017-08-02T03:19:41Z

.travis.yml

@@ -15,3 +15,4 @@ script:
  - git-validation -run DCO,short-subject -v -range ${TRAVIS_COMMIT_RANGE}
  - make
  - make test
+  - PATH="${PATH}:${PWD}" make -C test


I think this will be failed. As from test/README.md, we need do 'git submodule update --init' first. But you didn't do such preparation in travis.yml

... we need do 'git submodule update --init' first.

Travis does that for us.

OK, got it.

wking · 2017-08-02T03:30:25Z

test/README.md

+# oci-runtime-tool integration tests
+
+This project uses the [Sharness][] test harness, installed as a [Git
+submodule]submodule].  To setup the test installation after a clone,


I'm missing a [ here. Will update later tonight.

Mashimiao · 2017-08-02T03:33:47Z

I'm OK with this patchset, but generate related tests are all failed. Would you mind update these tests based on current generate output?

wking · 2017-08-02T03:41:52Z

.. generate related tests are all failed...

They look like they passed to me. Can you link to what you'd like fixed?

Mashimiao · 2017-08-02T03:56:22Z

My local test result:

not ok 1 - Test oci-runtime-tool generate writing to stdout
#	
#		oci-runtime-tool generate | head -n2 >actual &&
#		cat <<EOF >expected &&
#	{
#		"ociVersion": "1.0.0-rc6",
#	EOF
#		test_cmp expected actual
#	
not ok 2 - Test oci-runtime-tool generate --output
#	
#		oci-runtime-tool generate --output config.json &&
#		head -n2 config.json >actual &&
#		cat <<EOF >expected &&
#	{
#		"ociVersion": "1.0.0-rc6",
#	EOF
#		test_cmp expected actual
#	
# failed 2 among 2 test(s)
1..2```

wking · 2017-08-02T04:04:29Z

Ah, I probably need to rebase around #431.

It's GPLv2+ [1], so using a submodule keeps the separation from our Apache 2.0 source clear. And it's an isolated enough dependency that I prefer to keep the Git histories separate. Generated with: $ mkdir test $ git submodule add git://github.com/mlafeldt/sharness.git test/sharness $ (cd test/sharness && git checkout v1.0.0) $ git add test/sharness $ cd test $ ln -s sharness/aggregate-results.sh $ ln -s sharness/sharness.sh $ ln -s sharness/test/Makefile $ emacs README.md $ git add README.md Makefile sharness.sh aggregate-results.sh [1]: https://github.com/mlafeldt/sharness/tree/v1.0.0#license Signed-off-by: W. Trevor King <wking@tremily.us>

Signed-off-by: W. Trevor King <wking@tremily.us>

Using jq to format the output so we don't have to worry about oci-runtime-tool's default tab indents or lack of trailing newlines, neither of which play nicely with <<-EOF here documents. Signed-off-by: W. Trevor King <wking@tremily.us>

Signed-off-by: W. Trevor King <wking@tremily.us>

wking · 2017-08-02T04:36:05Z

On Wed, Aug 02, 2017 at 04:04:30AM +0000, W. Trevor King wrote: Ah, I probably need to rebase around #431.

I've done this and addressed the missing brace [1] with b8d8357 → 11deb9d. [1]: #180 (comment)

Mashimiao · 2017-08-02T05:15:01Z

LGTM

mrunalp reviewed Aug 3, 2016
View reviewed changes

wking mentioned this pull request Aug 9, 2016

add validate_test.go to do unit test #188

Merged

wking mentioned this pull request Dec 25, 2016

cmd: generate: add --linux-namespace-* family of flags #288

Merged

wking mentioned this pull request Feb 15, 2017

add device-remove and device-remove-all option #325

Merged

Mashimiao closed this Jul 19, 2017

Mashimiao reopened this Jul 19, 2017

wking changed the base branch from v1.0.0.rc1 to master July 19, 2017 15:34

wking force-pushed the sharness branch from d8f9c62 to b8d8357 Compare July 19, 2017 17:23

Mashimiao reviewed Aug 2, 2017

View reviewed changes

wking commented Aug 2, 2017

View reviewed changes

wking added 5 commits August 1, 2017 21:30

test/t0001-oci-runtime-tool.t: Test --help and --version

aa46267

Signed-off-by: W. Trevor King <wking@tremily.us>

test/t1000-generate-output.t: Tests generate and --output

4f96bb3

Signed-off-by: W. Trevor King <wking@tremily.us>

.travis.yml: Run the sharness tests

11deb9d

Signed-off-by: W. Trevor King <wking@tremily.us>

wking force-pushed the sharness branch from b8d8357 to 11deb9d Compare August 2, 2017 04:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: Stub in sharness framework #180

test: Stub in sharness framework #180

wking commented Aug 3, 2016

mrunalp Aug 3, 2016

wking Aug 3, 2016

liangchenye commented Aug 4, 2016

wking commented Aug 4, 2016

liangchenye commented Aug 8, 2016

wking commented Oct 12, 2016

Mashimiao commented Oct 13, 2016

wking commented Oct 13, 2016 •

edited

Loading

Mashimiao commented Jul 19, 2017

wking commented Jul 19, 2017

wking commented Jul 19, 2017 via email

Mashimiao Aug 2, 2017

wking Aug 2, 2017

Mashimiao Aug 2, 2017

wking Aug 2, 2017 •

edited

Loading

Mashimiao commented Aug 2, 2017

wking commented Aug 2, 2017

Mashimiao commented Aug 2, 2017 •

edited

Loading

wking commented Aug 2, 2017

wking commented Aug 2, 2017 via email

Mashimiao commented Aug 2, 2017 •

edited by caniszczyk

Loading

test: Stub in sharness framework #180

Are you sure you want to change the base?

test: Stub in sharness framework #180

Conversation

wking commented Aug 3, 2016

mrunalp Aug 3, 2016

Choose a reason for hiding this comment

wking Aug 3, 2016

Choose a reason for hiding this comment

liangchenye commented Aug 4, 2016

wking commented Aug 4, 2016

liangchenye commented Aug 8, 2016

wking commented Oct 12, 2016

Mashimiao commented Oct 13, 2016

wking commented Oct 13, 2016 • edited Loading

Mashimiao commented Jul 19, 2017

wking commented Jul 19, 2017

wking commented Jul 19, 2017 via email

Mashimiao Aug 2, 2017

Choose a reason for hiding this comment

wking Aug 2, 2017

Choose a reason for hiding this comment

Mashimiao Aug 2, 2017

Choose a reason for hiding this comment

wking Aug 2, 2017 • edited Loading

Choose a reason for hiding this comment

Mashimiao commented Aug 2, 2017

wking commented Aug 2, 2017

Mashimiao commented Aug 2, 2017 • edited Loading

wking commented Aug 2, 2017

wking commented Aug 2, 2017 via email

Mashimiao commented Aug 2, 2017 • edited by caniszczyk Loading

wking commented Oct 13, 2016 •

edited

Loading

wking Aug 2, 2017 •

edited

Loading

Mashimiao commented Aug 2, 2017 •

edited

Loading

Mashimiao commented Aug 2, 2017 •

edited by caniszczyk

Loading