Uh oh!
There was an error while loading. Please reload this page.
test(evaluate): pin the CLI's OPA/Rego contract - #1073
Conversation
Claude finished @mbevc1's task in 2m 15s —— View job Review: pin the CLI's OPA/Rego contract
Static analysis
(I couldn't execute What this PR does wellThis is a genuinely thoughtful characterisation suite, and it earns its
Minor observations (non-blocking)
Nothing here blocks. This is exactly the kind of contract coverage that turns a customer-facing dependency-bump regression into a red CI run. Nice work. |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Policies passed to `kosli evaluate` are written by users, so a change in Rego parsing, safety checking, or the built-in surface is a breaking change for us even when our own Go code still compiles. The OPA 1.18.2 -> 1.19.0 bump in #1071 is the case in point: stricter safety checking for `:=` invalidates a class of previously working customer policies, and nothing in the suite would have caught it. Add the coverage that would have. These are characterisation tests split by layer: internal/evaluate owns Rego semantics, cmd/kosli proves only that diagnostics and verdicts reach the terminal. internal/evaluate/opa_contract_test.go: - a realistic default-deny policy end to end, with --params-driven required attestations against trail-shaped input - the built-in surface real policies reach for (regex.match, semver.compare, time.parse_rfc3339_ns, every, some ... in, walk) - Rego v1 as the default parser version - the 1.19.0 `:=` strictness change, paired with an over-strictness guard so a release that breaks correct `:=` usage also goes red - unsafe-var reporting, and a broken `violations` rule failing before any verdict is produced cmd/kosli: - the same boundary through `evaluate input` - TestPolicyFixturesStillCompile, a corpus-wide canary over testdata/policies that checks every fixture against each future bump Two sharp edges are pinned as characterisation, not endorsement: an undefined `allow` errors instead of denying, and with `--params` omitted `data.params` is undefined, so object.get(data.params, k, fallback) yields undefined rather than the fallback.
The policy existed twice — a const in the internal test and a near-identical .rego under cmd/kosli/testdata. Both copies passed independently, so drift would have produced two layers disagreeing about what the policy means with nothing going red. Keep one copy, in internal/evaluate/testdata, and consume it from both layers: the internal test embeds it with //go:embed, and the CLI test points --policy at the same file. That direction is with the grain of the dependency graph — cmd/kosli already imports internal/evaluate in production — and embedding means a rename or deletion breaks the build rather than one test. Verified by mutating the fixture's default `required` list: the CLI's ALLOWED case flips to DENIED, confirming both layers read the one file. `internal/evaluate` gains its own canary over its testdata, mirroring the cmd/kosli one, so each package covers the fixtures it owns.
2690dca to
a1587ddCompareinternal/evaluate's fixture canary had no exclusion set, unlike the cmd/kosli copy. It passes today because this package's testdata holds only valid fixtures, but internal/evaluate is exactly where an intentionally uncompilable fixture would naturally land, and that would have turned the canary red spuriously. The set is empty for now — its value is being the obvious home for such a fixture, and keeping the two mirrors structurally identical.
The cmd/kosli mirror names the OPA version in its parse and compile failure messages; this copy did not. An upgrade-triggered failure is the whole reason these canaries exist, so the message that names the version that broke the fixture should read the same from either package. Verified both messages render: a malformed fixture reports "no longer parses under OPA 1.19.0", and one that parses but fails safety checking reports "no longer compiles under OPA 1.19.0".
a1587dd to
da76d4bCompareUh oh!
There was an error while loading. Please reload this page.
Policies passed to
kosli evaluateare written by users, so a change in Rego parsing, safety checking, or the built-in surface is a breaking change for us even when our own Go code still compiles. The OPA 1.18.2 -> 1.19.0 bump in #1071 is the case in point: stricter safety checking for:=invalidates a class of previously working customer policies, and nothing in the suite would have caught it.Add the coverage that would have. These are characterisation tests split by layer: internal/evaluate owns Rego semantics, cmd/kosli proves only that diagnostics and verdicts reach the terminal.
internal/evaluate/opa_contract_test.go:
:=strictness change, paired with an over-strictness guard so a release that breaks correct:=usage also goes redviolationsrule failing before any verdict is producedcmd/kosli:
evaluate inputTwo sharp edges are pinned as characterisation, not endorsement: an undefined
allowerrors instead of denying, and with--paramsomitteddata.paramsis undefined, so object.get(data.params, k, fallback) yields undefined rather than the fallback.Checklist
charts/k8s-reporter/) updated, if needed. Note: these changes live in a separate PR