json-schema: initial pass at schema and validator by vbatts · Pull Request #313 · opencontainers/runtime-spec

vbatts · 2016-01-19T17:43:27Z

Conforming to https://tools.ietf.org/html/draft-zyp-json-schema-03
and http://json-schema.org/latest/json-schema-core.html

This work would ideally pair with separating the current reference source effort #276

provide validation tool to validate a configuration file
full structures
define basic primitives
account for all the pointer usage
discuss versioned schema-$ver.json

The gist of the config.json being tested with (may change periodically): https://gist.github.com/vbatts/5ac2723c9598875ffb0e

Signed-off-by: Vincent Batts vbatts@hashbangbash.com

vbatts · 2016-01-19T17:44:12Z

I am working through making usable definitions like has been done for devices

philips · 2016-01-19T20:12:31Z

looks good but makes my head hurt, oy.

vbatts · 2016-01-19T21:27:46Z

It is something to behold, but it is allowing decently flexible validation.

kofalt · 2016-01-19T23:20:57Z

Food for thought: I see you're already using schema references ("$ref"); perhaps if you broke this up into several files and combined them at the end? For example, the cpu key could be its own schema, etc.

vbatts · 2016-01-20T02:53:50Z

I wondered about that, but didn't see a way to "source" or "include", and
really don't think that just concatenated them together felt smooth

On Tue, Jan 19, 2016 at 6:20 PM, Nathaniel Kofalt notifications@github.com
wrote:

Food for thought: I see you're already using schema references ("$ref");
perhaps if you broke this up into several files and combined them at the
end? For example, the cpu key could be its own schema, etc.
—
Reply to this email directly or view it on GitHub
#313 (comment).

wking · 2016-01-20T05:04:49Z

On Tue, Jan 19, 2016 at 06:53:53PM -0800, Vincent Batts wrote:
On Tue, Jan 19, 2016 at 6:20 PM, Nathaniel Kofalt:
Food for thought: I see you're already using schema references
("$ref"); perhaps if you broke this up into several files and
combined them at the end? For example, the cpu key could be its
own schema, etc.
I wondered about that, but didn't see a way to "source" or
"include", and really don't think that just concatenated them
together felt smooth

$ref is the way to source/include other content 1.

vbatts · 2016-01-20T15:14:52Z

how quaint. let me revisit this.

wking · 2016-01-27T06:42:45Z

I don't think you need oneOf for arrays. Is there a reason:
"items": { "$ref": "/definitions/GID" }
doesn't work?

it is a typed array. Otherwise it would allow ["foo", 1, false]

On Thu, Mar 03, 2016 at 12:27:46PM -0800, Vincent Batts wrote:
"items": {
"oneOf": [ { "$ref": "#/definitions/GID" } ]
}
it is a typed array. Otherwise it would allow ["foo", 1, false]
I'm pretty sure that's what the $ref-only phrasing 1 means.

tetron · 2016-01-27T21:34:47Z

This is just a drive-by suggestion, but you might want to look at https://github.com/common-workflow-language/schema_salad as a possible alternative to JSON Schema.

vbatts · 2016-02-10T17:42:39Z

@tetron thanks, but that seems too niche and less of a standard even than JSON-schema is.

vbatts · 2016-03-03T20:04:13Z

rebased again

vbatts · 2016-03-03T20:05:52Z

I feel this is about ready for review too.

wking · 2016-03-03T20:12:09Z

On Thu, Mar 03, 2016 at 12:05:54PM -0800, Vincent Batts wrote:

I feel this is about ready for review too.

Any comments on my three still-applicable comments:

Unnecessary oneOf 1.
Missing indent (and a suggestion to wash through jq) 2.
Avoid the generic ‘schema/schema.json’ in favor of
‘schema/config.json’, so the schema/*.json filename matches the
config file it covers 3.

vbatts · 2016-03-03T22:06:55Z

Done. PTAL

wking · 2016-03-03T22:15:52Z

On Thu, Mar 03, 2016 at 02:06:57PM -0800, Vincent Batts wrote:

Done. PTAL

Indent and oneOfs look good with 5e656f4. I'd still like
schema/schema.json renamed to schema/config.json [1,2].

wking · 2016-03-03T22:18:32Z

+ {
+ "$ref": "#/definitions/SyscallArg"
+ }
+ ]


Remaining unnecessary oneOf (see 5e656f4).

wking · 2016-03-09T16:23:22Z

+ }
+ },
+ "CapabilityString": {
+ "description": "File permissions mode (typically an octal value)",


This is not the right description for CapabilityString.
And we can probably use Capability instead of CapabilityString (like we don't use MajorUInt); I don't foresee other singular capability types.

vbatts · 2016-03-09T18:24:24Z

rebased and updated

crosbymichael · 2016-03-09T19:01:36Z

LGTM

mrunalp · 2016-03-09T19:14:52Z

Can we squash or re-order the commits?

vbatts · 2016-03-09T19:29:42Z

@mrunalp ah yeah. the commits were just as I made progress. Let me clean them up.

Conforming to https://tools.ietf.org/html/draft-zyp-json-schema-03 and http://json-schema.org/latest/json-schema-core.html * Utilizes a number of JSON schema features, including 'pattern' * Defined primitives, like integers, that we'll use * Split out definitions for primitives and platform-specific * Provide a Makefile for: - "fmt" target for *.json - "validate" target for building the validation tool Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>

vbatts · 2016-03-09T19:38:28Z

@mrunalp fully squashed

vbatts · 2016-03-09T21:58:35Z

@philips @mrunalp @vishh @hqhq @tianon PTAL

mrunalp · 2016-03-10T18:26:11Z

LGTM

json-schema: initial pass at schema and validator

Also: * Update the link to Go bindings after 7bf06d5 (source and schema: differentiate with examples, 2015-12-18, opencontainers#276). * Add a reference to the JSON Schema after cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313). It's pretty clear that the Go bindings cannot be canonical on their own, because they do not define limits (e.g. the 0 through 512 range for FileMode). The JSON Schema is closer, but still does not cover everything (e.g. "a directory must exist at root.path"). Both the Go bindings and the JSON Schema could grow to cover the full spec by adding that sort of thing to comments and descriptions, but that's not how things seem to be working now. Signed-off-by: W. Trevor King <wking@tremily.us>

Validation is partly covered by cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313) and subequent JSON Schema work. The remainder of these targets are handled by ocitools [1]. [1]: https://github.com/opencontainers/ocitools Signed-off-by: W. Trevor King <wking@tremily.us>

# digest/hashing target Most of this has spun off with [1], and I haven't heard of anyone talking about verifying the on-disk filesystem in a while. My personal take is on-disk verification doesn't add much over serialized verification unless you have a local attacker (or unreliable disk), and you'll need some careful threat modeling if you want to do anything productive about the local attacker case. For some more on-disk verification discussion, see the thread starting with [2]. # distributable-format target This spun off with [1]. # lifecycle target I think this is resolved since 7713efc (Add lifecycle for containers, 2015-10-22, opencontainers#231), which was committed on the same day as the ROADMAP entry (4859f6d, Add initial roadmap, 2015-10-22, opencontainers#230). # container-action target Addressed by 7117ede (Expand on the definition of our ops, 2015-10-13, opencontainers#225), although there has been additional discussion in a7a366b (Remove exec from required runtime functionalities, 2016-04-19, opencontainers#388) and 0430aaf (Split create and start, 2016-04-01, opencontainers#384). # validation and testing targets Validation is partly covered by cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313) and subequent JSON Schema work. The remainder of these targets are handled by ocitools [3]. # printable/compiled-spec target The bulk of this was addressed by 4ee036f (*: printable documents, 2015-12-09, opencontainers#263). Any remaining polishing of that workflow seems like a GitHub-issue thing and not a ROADMAP thing. And publishing these to opencontainers.org certainly seems like it's outside the scope of this repository (although I think that such publishing is a good idea). [1]: https://github.com/opencontainers/image-spec [2]: https://groups.google.com/a/opencontainers.org/d/msg/dev/xo4SQ92aWJ8/NHpSQ19KCAAJ Subject: OCI Bundle Digests Summary Date: Wed, 14 Oct 2015 17:09:15 +0000 Message-ID: <CAD2oYtN-9yLLhG_STO3F1h58Bn5QovK+u3wOBa=t+7TQi-hP1Q@mail.gmail.com> [3]: https://github.com/opencontainers/ocitools Signed-off-by: W. Trevor King <wking@tremily.us>

The JSON Schema requirement dates back to cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313), but the property has been explicitly optional in the Markdown spec since 7ac41c6 (config.md: reformat into a standard style, 2015-06-30). Signed-off-by: W. Trevor King <wking@tremily.us>

Also: * Update the link to Go bindings after 7bf06d5 (source and schema: differentiate with examples, 2015-12-18, opencontainers#276). * Add a reference to the JSON Schema after cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313). It's pretty clear that the Go bindings cannot be canonical on their own, because they do not define limits (e.g. the 0 through 512 range for FileMode). The JSON Schema is closer, but still does not cover everything (e.g. "a directory must exist at root.path"). Both the Go bindings and the JSON Schema could grow to cover the full spec by adding that sort of thing to comments and descriptions, but that's not how things seem to be working now. Signed-off-by: W. Trevor King <wking@tremily.us>

# digest/hashing target Most of this has spun off with [1], and I haven't heard of anyone talking about verifying the on-disk filesystem in a while. My personal take is on-disk verification doesn't add much over serialized verification unless you have a local attacker (or unreliable disk), and you'll need some careful threat modeling if you want to do anything productive about the local attacker case. For some more on-disk verification discussion, see the thread starting with [2]. # distributable-format target This spun off with [1]. # lifecycle target I think this is resolved since 7713efc (Add lifecycle for containers, 2015-10-22, opencontainers#231), which was committed on the same day as the ROADMAP entry (4859f6d, Add initial roadmap, 2015-10-22, opencontainers#230). # container-action target Addressed by 7117ede (Expand on the definition of our ops, 2015-10-13, opencontainers#225), although there has been additional discussion in a7a366b (Remove exec from required runtime functionalities, 2016-04-19, opencontainers#388) and 0430aaf (Split create and start, 2016-04-01, opencontainers#384). # validation and testing targets Validation is partly covered by cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313) and subequent JSON Schema work. The remainder of these targets are handled by ocitools [3]. # printable/compiled-spec target The bulk of this was addressed by 4ee036f (*: printable documents, 2015-12-09, opencontainers#263). Any remaining polishing of that workflow seems like a GitHub-issue thing and not a ROADMAP thing. And publishing these to opencontainers.org certainly seems like it's outside the scope of this repository (although I think that such publishing is a good idea). [1]: https://github.com/opencontainers/image-spec [2]: https://groups.google.com/a/opencontainers.org/d/msg/dev/xo4SQ92aWJ8/NHpSQ19KCAAJ Subject: OCI Bundle Digests Summary Date: Wed, 14 Oct 2015 17:09:15 +0000 Message-ID: <CAD2oYtN-9yLLhG_STO3F1h58Bn5QovK+u3wOBa=t+7TQi-hP1Q@mail.gmail.com> [3]: https://github.com/opencontainers/ocitools Signed-off-by: W. Trevor King <wking@tremily.us>

The JSON Schema requirement dates back to cdcabde (schema: JSON Schema and validator for `config.json`, 2016-01-19, opencontainers#313), but the property has been explicitly optional in the Markdown spec since 7ac41c6 (config.md: reformat into a standard style, 2015-06-30). Signed-off-by: W. Trevor King <wking@tremily.us>

vbatts mentioned this pull request Jan 19, 2016
[Tracker] Platform/Compiler agnostic schema #166
Closed

vbatts force-pushed the json-schema branch 3 times, most recently from bda07e0 to c5521a5Compare January 19, 2016 21:26

vbatts changed the title ~~json-schema: initial pass at schema and validator~~[WIP] json-schema: initial pass at schema and validatorJan 19, 2016

vbatts force-pushed the json-schema branch from 15f8917 to 449073aCompare January 25, 2016 19:12

vbatts mentioned this pull request Jan 26, 2016
config: Single, unified config file #284
Merged

wking reviewed Jan 27, 2016
View reviewed changes

vbatts force-pushed the json-schema branch from 3d58754 to 8daa770Compare March 3, 2016 20:01

wking reviewed Mar 3, 2016
View reviewed changes

wking reviewed Mar 9, 2016
View reviewed changes

vbatts force-pushed the json-schema branch from 0bc8e19 to e974600Compare March 9, 2016 18:22

vbatts changed the title ~~[WIP] json-schema: initial pass at schema and validator~~json-schema: initial pass at schema and validatorMar 9, 2016

crosbymichael added this to the v0.4.0 milestone Mar 9, 2016

vbatts force-pushed the json-schema branch from e974600 to 084ff67Compare March 9, 2016 19:22

vbatts force-pushed the json-schema branch from 084ff67 to cdcabdeCompare March 9, 2016 19:38

vbatts mentioned this pull request Mar 9, 2016
[Tracker] Support standard validation and defaulting #219
Closed

mrunalp pushed a commit that referenced this pull request Mar 10, 2016
Merge pull request#313from vbatts/json-schema
9b19e76
json-schema: initial pass at schema and validator

mrunalp merged commit 9b19e76 into opencontainers:masterMar 10, 2016

wking mentioned this pull request Mar 10, 2016
config: Make Markdown canonical #342
Merged

vbatts deleted the json-schema branch April 12, 2016 03:50

wking mentioned this pull request Jun 2, 2016
schema/state.json: Add a JSON Schema for the state JSON #481
Merged

wking mentioned this pull request Jun 6, 2016
schema/schema.json: Make 'hostname' optional #491
Merged

thaJeztah mentioned this pull request Jul 10, 2020
JSON Schema definitions of 'int64' and 'uint64' types have ranges not representable in 64 bits #1051
Closed

Uh oh!

Conversation

vbatts commented Jan 19, 2016

Uh oh!

vbatts commented Jan 19, 2016

Uh oh!

philips commented Jan 19, 2016

Uh oh!

vbatts commented Jan 19, 2016

Uh oh!

kofalt commented Jan 19, 2016

Uh oh!

vbatts commented Jan 20, 2016

Uh oh!

wking commented Jan 20, 2016

Uh oh!

vbatts commented Jan 20, 2016

Uh oh!

wkingJan 27, 2016

Choose a reason for hiding this comment

Uh oh!

vbattsMar 3, 2016

Choose a reason for hiding this comment

Uh oh!

wkingMar 3, 2016

Choose a reason for hiding this comment

Uh oh!

tetron commented Jan 27, 2016

Uh oh!

vbatts commented Feb 10, 2016

Uh oh!

vbatts commented Mar 3, 2016

Uh oh!

vbatts commented Mar 3, 2016

Uh oh!

wking commented Mar 3, 2016

Uh oh!

vbatts commented Mar 3, 2016

Uh oh!

wking commented Mar 3, 2016

Uh oh!

wkingMar 3, 2016

Choose a reason for hiding this comment

Uh oh!

wkingMar 9, 2016

Choose a reason for hiding this comment

Uh oh!

vbattsMar 9, 2016

Choose a reason for hiding this comment

Uh oh!

vbatts commented Mar 9, 2016

Uh oh!

crosbymichael commented Mar 9, 2016

Uh oh!

mrunalp commented Mar 9, 2016

Uh oh!

vbatts commented Mar 9, 2016

Uh oh!

vbatts commented Mar 9, 2016

Uh oh!

vbatts commented Mar 9, 2016

Uh oh!

mrunalp commented Mar 10, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants