Uh oh!
There was an error while loading. Please reload this page.
acc: mutate-based invariant fuzzing for bundle configs - #5686
Open
radakam wants to merge 116 commits into
Open
Conversation
Collaborator
Integration test reportCommit: fb8d79a
8 interesting tests: 4 SKIP, 3 KNOWN, 1 RECOVERED
Top 6 slowest tests (at least 2 minutes):
|
Contributor
Approval status: pending
|
denik
reviewed
Jun 26, 2026
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Keep the how; cut glosses that duplicate the targets and "rather than / not a" framing.
Rotate seeds by day-of-epoch so PR runs of push.yml do not open gaps, raise test-fuzz's package timeout to cover four budgeted variants, restore set +e around invariant_deploy so a panicking reject is still asserted, and drop the unused prologue.sh that still called emit_fuzz_config.py.
Additive mutate now picks from a hand-curated INJECT table of deploy-proven optionals (including the fields behind past drift findings) instead of walking bundle schema. Drop gen_fuzz_config and the schema dump, and add app and experiment to MUTATE_BASES so those types stay in reach under mutate-only.
Keep the non-obvious reasons; drop restatements and contrastive framing across the harness, Taskfile, and nightly job.
Also diversify the selftest sample seeds and gate nightlies through test-result with a failure summary for triage.
State what the harness does and why; cut "rather than / not a / no X" asides.
Drop the custom YAML dumper for json.dumps (valid YAML 1.2) and check every INJECT field is a settable schema input, which also caught a nonexistent experiments.description entry.
Keep inject and destroy on separate seeds, harden the loader contract check, and keep LOG.repro reachable after a red nightly.
Treat validate type-mismatch warnings as seed rejections so configs like group_name: [] do not deploy and fail mid-migrate. Point the nightly triage upload at the same TMPDIR root the summary step uses.
Drop the hand-rolled YAML loader for committed JSON fixtures, reject schema warnings via validate --strict, and run each seed through seed.sh instead of export -f.
Sibling layout had to copy curated [[Server]] stubs and source helpers by hand. As invariant/fuzz/, parent prepare and stubs come from the directory chain; the leaf only clears INPUT_CONFIG, forces Cloud=false, and adds TESTSERVER_GAP catch-alls.
…inheritance" This reverts commit e1246fb.
The fuzzer kept its own JSON snapshot of each invariant config because the acceptance Python helpers are stdlib-only and cannot parse YAML. That made the curated YAML and its snapshot drift apart silently whenever one was edited. Parse the invariant YAML directly via a new hidden `bundle debug yaml-to-json`, so there is one copy of every base and it is parsed the way the bundle parses it. Output is byte-identical to the snapshots this removes. Also drop the unrelated comment rewording in invariant/script.prepare, leaving only the INVARIANT_DIR indirection and the panic scan on a rejected deploy.
bundle/debug's help output is a golden, and the other tooling-only subcommands (terraform, render-template-schema, list-targets) are hidden so they stay out of it. Hide this one too rather than adding a test helper to the listing.
Fall through to destructive mutate when INJECT has nothing to add, fail windows with zero deploys unless every seed is a gap, and drop incomplete INJECT entries that only produced rejects.
gotestsum places custom flags before the package list, so -keeptmp was rejected before any fuzz seed ran.
Bare `task` is not on PATH in a normal checkout.
The mutator runs in stdlib-only python and cannot read the invariant YAML, so it went through a hidden `bundle debug yaml-to-json`. That put test-only tooling in the product CLI for data the CLI has no privileged knowledge of. Snapshot the configs to fuzz/bases/*.json.tmpl with a generator wired into generate-check, the way generate-refschema owns its acceptance artifact: the validate-generated CI job regenerates and diffs, so editing a config without a regen fails there. The snapshots are byte-identical to the CLI's output, and load_base is a plain open + json.loads again.
Acceptance runs uv offline, so adding PyYAML at runtime cannot replace the snapshot generator. Store the 13 shared invariant bases in JSON syntax instead: JSON is valid YAML 1.2 for the bundle, while the stdlib-only mutator can parse the same source files with json.loads. This removes the generated copies, generator task, and drift machinery without bringing test-only YAML parsing back into the product CLI.
Keep idiomatic YAML fixtures and avoid shipping a test-only command in the product CLI; stdlib Python shells out to a harness-built helper that uses the same yamlloader as the bundle.
Remove the "Otherwise …" triage justification and rewrite nearby comments to state the why without the negative case.
Windows go build treated the flag as an import path and broke every TestAccept run.
INPUT_CONFIG_OK is printed only after a deploy succeeds, so a seed whose config passed validate --strict and then failed to deploy fell through to rejected and kept the run green. That hid the validate gaps and deploy failures mutate is meant to find (empty grant principal, secret scope ACL reject).
Migrate seeds terraform first, so catalog / external_location (direct-only) and sql_warehouse (known post-migrate drift) cannot succeed there. Mirror invariant/migrate's EnvMatrixExclude; otherwise the new deploy-fail classifier flags those seeds as bugs on every PR smoke run.
The nightly job recomputed FUZZ_SEED_COUNT and the day-of-epoch FUZZ_SEED_START that test-fuzz already defaults to, so the two had to stay in sync for nightly windows to stay disjoint. Let the task own them. INVARIANT_DIR needed no export: fuzz/script.prepare sets it before seed.sh dereferences it, so the export only leaked the variable into every CLI subprocess of the invariant targets. Comment fixes: the INJECT schema check fails the run rather than warning, and the stub check compares Pattern lines, not whole stubs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Reuse the existing invariant targets, but feed them mutated configs instead of curated ones. Everything new lives in
acceptance/bundle/fuzz/; the target scripts are untouched — fuzz overridesinvariant_renderto generate the config, and on PRs swaps the drift check for plan-determinism.What one seed does:
run_fuzz.pyrunsseed.shin a freshseed-N/with its own bundle name, so seeds cannot see each other's state.mutate_fuzz_config.pypicks a deploy-verified base frombundle/invariant/configs/and either injects one optional from a curatedINJECTtable or applies a few destructive edits (drop a field, swap in a hostile scalar, empty a container).bundle validate --strictgates the result, so a config the CLI already rejects never reaches deploy.no_drift,migrate,delete_idempotentordestroy_idempotent.run_fuzz.pyclassifies the seed.The classifier is the oracle, so it is the main thing to review:
validate --strictrefused the config. Not a finding.FUZZ_SEED_TIMEOUT.Only
hangandbugfail the run, and both write aLOG.reprowith a single-seed repro command.Two ways it runs:
acceptance/bundle/fuzz): a fixed 25-seed window per target, plan-determinism instead of exact drift (FUZZ_CHECK_DRIFT=0). Expected green, ~35s locally../task test-fuzz, schedule-only job): a day-rotated window, exactno_drift, 15 min budget per target. A red nightly is a finding to triage rather than a broken build, so the job puts eachLOG.reproin the step summary and uploads the seed workdirs.Two things reach outside the new directory:
acceptance/cmd/yaml2jsonis a new test-only binary. The mutator is stdlib-only Python and cannot read YAML, so it shells out to the sameyamlloaderthe bundle uses instead of keeping a second copy of every base.acceptance/bundle/invariant/script.prepareaffects the five existing targets:INVARIANT_DIRmakesconfigs/anddata/reachable from outside the subtree, andinvariant_deploynow scans a failed deploy for panics before re-raising its exit code.acceptance/selftest/mutate_fuzz_configpins the mutator contract: deterministic per seed, everyINJECTfield settable in the schema, and sample configs golden.Why
Invariant tests cover curated happy paths. This explores hostile and optional-field configs on the same deploy/plan/destroy scripts to catch reconcile, validate, and migrate bugs without cloud round-trips. Exact drift stays nightly because mutated configs often false-positive on incomplete testserver stubs; PRs keep a stable weaker oracle. Already surfaced real CLI/testserver issues during development, see the table below.
Tests
go test ./acceptance -run TestAccept/selftest/mutate_fuzz_configgo test ./acceptance -run TestAccept/bundle/fuzzFUZZ_SEED_COUNT=5 FUZZ_CHECK_DRIFT=1 ENVFILTER=FUZZ_TARGET=no_drift ./task test-fuzzBugs found
CLI
Test server
Upstream