runtimetest:add platform validation

[wangkirin]

add platform validation function in runtimetest
compare user defined in config.json with runtime parameters to determine if the test pass

Signed-off-by: Wang Qilin qilin.wang@huawei.com

[wking]

On Mon, Jan 25, 2016 at 03:25:46AM -0800, Wang Qilin wrote:

add platform validation function in runtimetest

It looks like this test (as of 7be5036) is raising an error if the
configured platform doesn't match the current platform. However,
‘runtimetest’ isn't about validating bundles, it's about validating
runtimes (as I understand it anyway). So the test should actually be
ensuring:

When passed a config specifying a different platform (e.g. an amd64
config on an x86 system) that the runtime exits with an error.

Although neither 1 nor the inflight opencontainers/runtime-spec#2252 has
wording specified behavior in this situation. Maybe we need a specs
PR first to clarify that?

[mrunalp]

@wangkirin Yep, agree with @wking

[zenlint]

@wking +1.

[mrunalp]

@wangkirin Could you please update the PR per the feedback?

[wangkirin]

@wking@mrunalp @zenlinTechnofreak Sorry for being late !
Inspired by discussions above , I think we can expand this issue to two aspects.

Firtst, When passed a config specifying a platform different from the runtime(in other words container engine like docker,rkt) is running on ,whether the runtime should exit with an error . If the answer is yes ,may be it's out of the scope of runtimetest because it didn't actually began to run. And we should find a way to capture the error info because it should come from runtime itself ,not the runtimetest program.

Second, If we allow the container begin to run even the platform specified in the config.json is different from the runtime( like current runc) . I think the runtime should give a hint and should be captured by related runtime test program.

And I also agree with @wking , we should have further definition in specs, may be not only in the platform-specific-configuration section , but also in the lifecycle section.

So I suggest pending this PR

[wking]

On Tue, Mar 08, 2016 at 05:13:36AM -0800, Wang Qilin wrote:

When passed a config specifying a platform different from the
runtime``(in other wordscontainer enginelike docker,rkt) is running on ,whether the runtime should exit with an error . If the answer is yes ,may be it's out of the scope ofruntimetestbecause it didn't actually began to run. And we should find a way to capture the error info because it should come fromruntime itself ,not the
runtimetest program.

+1 to this. There are a number of places where the runtime can die
before the user-specified process is executed (e.g. pre-start hooks
failing). Using a shell-based test harness (runC may be using Bats
[1,2]) would make testing that sort of thing easy.

However, the spec doesn't currently get into how errors are exposed at
all 3, and until we address that there's no grounds for compliance
testing this behavior.

 Subject: Re: runc integration testcases
Date: Thu, 21 Jan 2016 17:00:20 -0800
Message-ID: <CAEu77a3rY_SyyNk1_rA0tngA0=E0L1FF_-iG=TEkNwsMYJ0EHw@mail.gmail.com>

[Mashimiao]

The SPEC says "The runtime MUST generate an error if it does not support the specified os."
If os or arch doesn't match we expect in configuration and platform validation is executed, then we can confirm the runtime did not generate an error as expected.
Though error message is not suitable, I think this validation is generally correct. we can make it move on.
How do you think about it? @wangkirin@mrunalp@wking@liangchenye@hqhq

[wking]

On Tue, Feb 28, 2017 at 06:54:59PM -0800, Ma Shimiao wrote: The SPEC says "The runtime MUST generate an error if it does not support the specified os." If os or arch doesn't match we expect in configuration and platform validation is executed, then we can confirm the runtime did not generate an error as expected.

I still [1] think it's better to handle this in the validation suite (e.g. #326) and not in runtimetest. As it stands with 7be5036, runtimetest would error if an x86 config ran on an amd64 host, but depending on the host kernel, and amd64 host may be capable of running 386 bundles as well. And x32 bundles, if someone were to add an x32 entry to our arch list (there does not seem to be a GOARCH for x32 [2]). So strict equality seems like an overly specific arch check. [1]: #15 (comment) [2]: https://golang.org/doc/install/source#environment

[Mashimiao]

@wking Agree with you, putting platform validation into the validation suite will be better.