test: Stub in sharness framework by wking · Pull Request #180 · opencontainers/runtime-tools

wking · 2016-08-03T06:16:57Z

Spun off from this discussion. This currently ignores most of the generate options and doesn't touch validate at all, but I thought I'd float it early for some feedback before sinking too much time into it. If the current work looks solid, I'm happy to land it now and follow up with additional PRs to fill it out, or to keep this PR open until the coverage is more complete.

Details on the individual changes in the commit messages.

mrunalp · 2016-08-03T23:06:27Z

+	ocitools generate --template template | jq . >actual &&
+	cat <<-EOF >expected &&
+ {
+ \"ociVersion\": \"1.0.0-rc1\",


I don't like the escaping that we do here :/

On Wed, Aug 03, 2016 at 04:06:28PM -0700, Mrunal Patel wrote:
@@ -0,0 +1,59 @@
+#!/bin/sh
+
+test_description='Test generate template'
+
+. ./sharness.sh
+
+test_expect_failure CAT,ECHO,HEAD,JQ 'Test ocitools generate --template with an empty template' "
echo '{}' >template &&
ocitools generate --template template | jq . >actual &&
cat <<-EOF >expected &&
{
\"ociVersion\": \"1.0.0-rc1\",
I don't like the escaping that we do here :/
There are only so many quotes we can use ;). Would you rather me wrap
the test in single quotes and use double quotes inside it? That makes
using variables in the test tricky, but we don't do any of that yet.

liangchenye · 2016-08-04T12:34:28Z

The sharness framework looks cool, I need time to absorb it.

It is different test frame topic with #66, right? It is more like a unit test. If so, I think we go test is a suitable and more nature choice since generate/validate are all writing in go.

wking · 2016-08-04T17:24:33Z

On Thu, Aug 04, 2016 at 05:34:29AM -0700, 梁辰晔 (Liang Chenye) wrote:

It is different test frame topic with
#66, right?

Yes. That is about “how do we validate a runtime?” and this is about
“how do we validate ocitools?”. I think they both benefit from an
off-the-shelf test harness, but you can use different harnesses for
each.

It is more like a unit test. If so, I think we go test is a
suitable and more nature choice since generate/validate are all
writing in go.

The goal of this PR is to test the ocitools command line as a black
box. I'd prefer ‘go test’ for testing the generation library API, but
that's not what I'm aiming at here.

liangchenye · 2016-08-08T11:06:51Z

+1 for the black box test

wking · 2016-10-12T14:45:33Z

Is this dead, or worth a rebase?

Mashimiao · 2016-10-13T06:04:23Z

Compared with importing sharness, how about importing a go based framework, like go-check?

wking · 2016-10-13T06:17:41Z

On Wed, Oct 12, 2016 at 11:04:24PM -0700, Ma Shimiao wrote:

Compared with importing sharness, how about importing a go based
framework, like go-check?

A shell-based test harness lets us use a very convenient API for
executing subprocesses: shell syntax. go-check [1,2] looks like a
nice, generic test framework, but you'd have to use os/exec 3 to
execute a command, manage its standard streams, and collect its
exit status. With sharness, you can take care of all of that with a single
line like 4:

ocitools --help | head -n2 >actual &&

Mashimiao · 2017-07-19T07:19:27Z

Please contribute to master branch

wking · 2017-07-19T14:54:42Z

Please contribute to master branch.

I can rebase onto master and change the target branch without filing a new PR and fragmenting discussion. Can you re-open this PR so I can do that?

wking · 2017-07-19T17:26:52Z

Redirected the PR and rebased onto master with d8f9c62 → b8d8357.

Mashimiao · 2017-08-02T03:19:41Z

 - git-validation -run DCO,short-subject -v -range ${TRAVIS_COMMIT_RANGE}
 - make
 - make test
+ - PATH="${PATH}:${PWD}" make -C test


I think this will be failed. As from test/README.md, we need do 'git submodule update --init' first. But you didn't do such preparation in travis.yml

... we need do 'git submodule update --init' first.
Travis does that for us.

OK, got it.

wking · 2017-08-02T03:30:25Z

+# oci-runtime-tool integration tests
+
+This project uses the [Sharness][] test harness, installed as a [Git
+submodule]submodule]. To setup the test installation after a clone,


I'm missing a [ here. Will update later tonight.

Mashimiao · 2017-08-02T03:33:47Z

I'm OK with this patchset, but generate related tests are all failed. Would you mind update these tests based on current generate output?

wking · 2017-08-02T03:41:52Z

.. generate related tests are all failed...

They look like they passed to me. Can you link to what you'd like fixed?

Mashimiao · 2017-08-02T03:56:22Z

My local test result:

not ok 1 - Test oci-runtime-tool generate writing to stdout
# # oci-runtime-tool generate | head -n2 >actual &&
# cat <<EOF >expected &&
#	{
# "ociVersion": "1.0.0-rc6",
#	EOF
# test_cmp expected actual
# not ok 2 - Test oci-runtime-tool generate --output
# # oci-runtime-tool generate --output config.json &&
# head -n2 config.json >actual &&
# cat <<EOF >expected &&
#	{
# "ociVersion": "1.0.0-rc6",
#	EOF
# test_cmp expected actual
# # failed 2 among 2 test(s)
1..2```

wking · 2017-08-02T04:04:29Z

Ah, I probably need to rebase around #431.

It's GPLv2+ [1], so using a submodule keeps the separation from our Apache 2.0 source clear. And it's an isolated enough dependency that I prefer to keep the Git histories separate. Generated with: $ mkdir test $ git submodule add git://github.com/mlafeldt/sharness.git test/sharness $ (cd test/sharness && git checkout v1.0.0) $ git add test/sharness $ cd test $ ln -s sharness/aggregate-results.sh $ ln -s sharness/sharness.sh $ ln -s sharness/test/Makefile $ emacs README.md $ git add README.md Makefile sharness.sh aggregate-results.sh [1]: https://github.com/mlafeldt/sharness/tree/v1.0.0#license Signed-off-by: W. Trevor King <wking@tremily.us>

Signed-off-by: W. Trevor King <wking@tremily.us>

Using jq to format the output so we don't have to worry about oci-runtime-tool's default tab indents or lack of trailing newlines, neither of which play nicely with <<-EOF here documents. Signed-off-by: W. Trevor King <wking@tremily.us>

Signed-off-by: W. Trevor King <wking@tremily.us>

wking · 2017-08-02T04:36:05Z

On Wed, Aug 02, 2017 at 04:04:30AM +0000, W. Trevor King wrote: Ah, I probably need to rebase around #431.

I've done this and addressed the missing brace [1] with b8d8357 → 11deb9d. [1]: #180 (comment)

Mashimiao · 2017-08-02T05:15:01Z

LGTM

mrunalp reviewed Aug 3, 2016
View reviewed changes

wking mentioned this pull request Aug 9, 2016
add validate_test.go to do unit test #188
Merged

wking mentioned this pull request Dec 25, 2016
cmd: generate: add --linux-namespace-* family of flags #288
Merged

wking mentioned this pull request Feb 15, 2017
add device-remove and device-remove-all option #325
Merged

Mashimiao closed this Jul 19, 2017

Mashimiao reopened this Jul 19, 2017

wking changed the base branch from v1.0.0.rc1 to masterJuly 19, 2017 15:34

wking force-pushed the sharness branch from d8f9c62 to b8d8357Compare July 19, 2017 17:23

Mashimiao reviewed Aug 2, 2017
View reviewed changes

wking commented Aug 2, 2017
View reviewed changes

wking added 5 commits August 1, 2017 21:30

test/t0001-oci-runtime-tool.t: Test --help and --version
aa46267
Signed-off-by: W. Trevor King <wking@tremily.us>

test/t1000-generate-output.t: Tests generate and --output
4f96bb3
Signed-off-by: W. Trevor King <wking@tremily.us>

.travis.yml: Run the sharness tests
11deb9d
Signed-off-by: W. Trevor King <wking@tremily.us>

wking force-pushed the sharness branch from b8d8357 to 11deb9dCompare August 2, 2017 04:34

Uh oh!

Conversation

wking commented Aug 3, 2016

Uh oh!

mrunalpAug 3, 2016

Choose a reason for hiding this comment

Uh oh!

wkingAug 3, 2016

Choose a reason for hiding this comment

Uh oh!

liangchenye commented Aug 4, 2016

Uh oh!

wking commented Aug 4, 2016

Uh oh!

liangchenye commented Aug 8, 2016

Uh oh!

wking commented Oct 12, 2016

Uh oh!

Mashimiao commented Oct 13, 2016

Uh oh!

wking commented Oct 13, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mashimiao commented Jul 19, 2017

Uh oh!

wking commented Jul 19, 2017

Uh oh!

wking commented Jul 19, 2017 via email

Uh oh!

MashimiaoAug 2, 2017

Choose a reason for hiding this comment

Uh oh!

wkingAug 2, 2017

Choose a reason for hiding this comment

Uh oh!

MashimiaoAug 2, 2017

Choose a reason for hiding this comment

Uh oh!

wkingAug 2, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mashimiao commented Aug 2, 2017

Uh oh!

wking commented Aug 2, 2017

Uh oh!

Mashimiao commented Aug 2, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wking commented Aug 2, 2017

Uh oh!

wking commented Aug 2, 2017 via email

Uh oh!

Mashimiao commented Aug 2, 2017• edited by caniszczyk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wking commented Oct 13, 2016•
edited
Loading

wkingAug 2, 2017•
edited
Loading

Mashimiao commented Aug 2, 2017•
edited
Loading

Mashimiao commented Aug 2, 2017•
edited by caniszczyk
Loading