[WIP] update initruntimetest by linzhinan

[liangchenye]

This pr is still under development.
Since we have 'state' and 'lifecircle' defined now (comparing to zhinan's previous PR #9)
This PR not only test the main configs (verified by cmd/runtimetest), but also test the state and life circle.

[liangchenye]

any suggestions on state test and lifecirle test? @zenlinTechnofreak

[wking]

Travis doesn't like this import, and you don't seem to use config.… in unit.go anyway, so you can probably drop this line.

[wking]

Ah, you dropped the config package in cb986be and just forgot this line.

[liangchenye]

Thanks for correction, I'll change that add post a working version.

[wking]

There was some consensus in #47 about making runtime testing a
first-class command (1 and later). That would be good, but I'm not
sure how to get there. One problem with this Go-based approach is
that it's a lot more work to test command-line invocation
(e.g. testState 2) than it would be in a shell-based test framework
(e.g. 3). So as much as I'd like an easily-installable command for
testing, my current preference is still to exercise the runtime's
command-line API through a command-line test framework 4. RunC's
test suite is likely a good starting point for this, we'd just have to
trim off the out-of-spec components.

[liangchenye]

The purpose of using golang in runtime test (or runtime certification) is we need to test more cases.
For example, the state test and lifecircle test are different with cmd/runtime test (I called it main config test in these commits). So the command-line script will become bigger and more complex if we add these. TestUnit is a wrap of a container, once there are more test cases, it is easier to apply and maintain them.

@wking@mrunalp PTAL

[wking]

On Tue, May 10, 2016 at 07:16:54AM -0700, 梁辰晔 (Liang Chenye) wrote:

For example, the state test and lifecircle test are different with
cmd/runtime test (I called it main config test in these commits). So
the command-line script will become bigger and more complex if we
add these.

I'm not recommending a single command-line script, I'm recommending
using a script-based framework (like runC already does 1). The bats
equivalent to TestUnit is [2,3], although the latter is killing the
container instead of just waiting for it to exit. In ccon I'm using
sharness, and tests look like 4. I expect that approach to scale
pretty well, and ‘test_expect_code 1 …’ (e.g. 5) is a very compact
way to check non-zero exit codes.

[wking]

It seems like the resistance to using bats, sharness, Python (#98),
etc. is the desire to ship a single binary to cover all ocitools
functionality 1. I've been looking around for a Go analog to
Python's unittest that can be built into the installed binary, but so
far haven't turned up anything useful. It's possible that we could
use Go's testing library and create our own runner with testing.Main
[2](since ‘go test’ expects *_test.go, and those are excluded from
the regular package [3]). But while it seems like it should be
possible, I haven't been able to figure out how to do it.

I'm not excited about writing and maintaining our own Go test harness,
so I'd appreciate if folks who prefer a single compiled binary to a
scriptier Bash / POSIX shell / Python solution could lay out their
reasoning for wanting the binary. Do we really expect folks to want
to run ocitools on systems without Python?

[Mashimiao]

any updates?

[liangchenye]

Close and split it into small issues/PRs.