Uh oh!
There was an error while loading. Please reload this page.
test(e2e): declarative Scenario layer, project files under testdata/<TestName>/ - #14087
Merged
Conversation
…trator Introduce a small declarative layer over the e2e framework so a test reads as its intent: an inline compose.yaml, a sequence of steps (command + expected observables), and nothing else. The framework owns project naming (derived from the test name), cleanup, and failure reporting: on the first unmet expectation the test fails with a step transcript, the failing command output, the project's container state, the engine events since the scenario started and the container logs. Checks observe real state (container IDs across steps, labels, image manifest platform) instead of grepping the progress writer output, which stays available as an explicit last-resort check. The e2e tests introduced by #14041 are converted as a demonstrator; their dedicated fixture files are inlined in the scenarios and removed. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Add Eventually(check, timeout) to the scenario layer: it retries a state-based check, re-observing the project containers between attempts. ServiceState(service, state) joins the check library, backed by a State field in the per-step snapshots. TestRestart is converted as the demonstrator: it replaces a WaitForCmdResult polling `ps --format json` for a JSON fragment and a bare time.Sleep with two declarative expectations, and now also locks the real invariant — restart must reuse the same container — via NotRecreated, which the lock-file trick of the original fixture was proving only indirectly. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
…alive On failure, a scenario now writes the untruncated diagnostic material to a stable per-project directory (compose.yaml, each step's full command and output, project containers, engine events, full container logs, per-step state snapshots), announced at the top of the failure report. The inline report stays truncated for readability; the artifacts let a failure be diagnosed without re-running the scenario. E2E_KEEP_FAILED=1 skips the teardown of a failed scenario so the live project state remains inspectable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
… the contract Checks are the shared vocabulary between scenarios: move them to a dedicated checks.go so the whole vocabulary is read — and reviewed — as one file, independently of the tests using it. SCENARIO.md codifies the rules that so far only lived in the PR description: one intent = one invariant, step names as behavior sentences, inline compose model, state-based checks first with OutputContains as an explicit last resort, new checks generic and named after their observable — plus how to exploit the failure artifacts and E2E_KEEP_FAILED. AGENTS.md points new e2e work at it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Each scenario resolves its project files from testdata/<TestName>/ by convention: standalone compose files in their native format, directly runnable with docker compose -f, copied to a temporary directory so the committed files are never mutated. Subtests map to nested directories. Ownership is strictly one test per directory: a check fails the suite on any testdata directory no test owns, which is what keeps this from degrading into a catch-all fixtures directory — no sharing, no leftovers after a rename. The inline Compose() declaration is removed; the demonstrator scenarios move their models to testdata, including the multi-file build context of TestUpBuildUnchangedContext (compose.yaml + Dockerfile + marker). Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ndeloofforce-pushed
the
e2e-scenario-testdata
branch
from
August 18, 2026 12:08
a139917 to
d81c764Compare…cks error paths Migrate the four pause tests to the Scenario DSL: the invariant is the container state (paused/running), not an HTTP timeout, so the scenarios observe state and drop the port-mapping helpers and the pause fixture. ExitCode joins the check vocabulary for error-path steps, paired with MayFail. TestPauseServiceDoesNotExist now locks the current behavior — pause of an unknown service is rejected with 'no such service' — which resolves the TODO the legacy test carried: it asserted exit 0 while wishing for an error, and the CLI has since started validating selected services. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
…bservable Migrate the volumes_from and network_mode:service donor tests to the Scenario DSL, inlining the compose model and dropping the no-deps fixtures. Each scenario removes the donor container behind compose's back and expects up --no-deps to be rejected with the 'container missing' error — the CLI's reported decision being the observable. ServiceNotCreated joins the check vocabulary: it asserts a service has no container at all, here proving the donor really is gone before the failing step. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Migrate the state-friendly start/stop tests to the Scenario DSL: - TestStartStop: stop halts containers in place, start brings the same ones back (NotRecreated); the ls checks match the project name exactly in the JSON output, immune to the prefix collisions the other start-stop project names would cause under parallel runs. - TestStartStopWithDependencies: dependency start/stop propagation read from container state instead of 'Container X Started' messages. - TestUpNoDeps replaces the 'Up no-deps links' subtest as its own scenario: up --no-deps must not create the linked dependency. - TestStartSingleServiceAndDependency: create/start walk the dependency chain and nothing else — now asserting the unrelated services have no container at all, stronger than the legacy message matching. The one-off and multi-file tests stay legacy: the vocabulary has no one-off awareness yet and a scenario holds a single compose file. The links and start-stop-deps fixtures become unused and are removed. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Snapshots now record whether a container is a one-off (run) container, plus its start time. Service-facing checks (ServiceState, NotRecreated, label checks) observe only long-lived containers; three checks join the vocabulary for the one-off side: OneOffState, OneOffsUntouched (same containers, same state, same start time — catches an unwanted restart, which message matching never could) and OneOffsRemoved (errors if there was nothing to remove, rather than pass vacuously). TestStartStopWithOneOffs becomes the demonstrator: stop/start/restart act on services and leave the one-off untouched, down --remove-orphans takes it away. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
…ORPHANS from .env The legacy test read as 'up removes leftover one-offs', but plain up only warns about them: the removal it asserted came from the fixture's .env declaring COMPOSE_REMOVE_ORPHANS=true (added by 'restore setEnvWithDotEnv', d3d378b) — an invariant the fixture kept invisible. The scenario declares both files inline as a txtar archive, so the mechanism under test is in plain sight, and observes the one-off's removal from container state rather than ps output. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
- TestExposeRange keeps its regression link (#13378), the invariant being simply that up accepts a port range in expose. - TestUpContainerNameConflict interpolates a project-scoped container_name instead of the fixture's global 'test', so parallel runs cannot collide on it. - TestIPC demonstrates the CLI() escape hatch: the external container the 'container:' mode points at is created eagerly before the steps, so its id is available to the declarative checks. (Runs only in CI: ipc container: mode is rejected by Docker Desktop and dind daemons — 'restricted host mount' — for the legacy test as well.) - wait scenarios observe which services exited instead of trusting the command's silence; TestWaitOnInfinity stays legacy, the DSL has no notion of a still-running command. The wait fixture stays for it; ipc-test and container_name fixtures are removed. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
…eate Vocabulary grows with Recreated (the mirror of NotRecreated), ServiceHealthy (healthcheck status observed from inspect) and a Within action modifier bounding blocking commands like up --wait, replacing the hand-rolled goroutine+timer of the legacy TestStartInterval. - cascade: abort-on-container-exit/failure scenarios inline the model; exit-code propagation (111) is locked with ExitCode. The one-off variant stays legacy — it drives a long-running up in the background. - profiles: the four scenarios observe which services materialize instead of grepping ps output; the dotenv variant declares its .env inline, making the activation mechanism visible. - TestRecreateWithNoDeps now asserts both sides: the service is replaced (Recreated) and the healthy dependency is left alone. - Fixtures start_interval/, profiles/compose.yaml+test-profile.env and recreate-no-deps.yaml are removed; cascade/ stays for the legacy one-off test. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ServiceScale and ReplicaNumbers join the vocabulary: replica counts and numbering read from container labels replace counting 'Started' lines in the CLI output. - TestScaleDownNoRecreate's final assertion now actually runs: the legacy test built the scale-down command without executing it, so 'scale down removes obsolete replicas' was never exercised; the scenario locks the real behavior (stale replicas dropped, up-to-date ones kept, whatever their numbers). - secrets scenarios inline the include-based project (compose.yaml, child project and env_file as one txtar archive). - TestRawEnvFile and TestRunEnvFile inline their env files; TestUnusedMissingEnvFile stays legacy for its completion assertions, which drive the plugin/standalone binaries directly. - Fixtures scale/, env-secret/ and dotenv/raw are removed. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
ImageExists and FileExists join the vocabulary, giving commit and export a real observable: the legacy tests only asserted the commands exited 0, the scenarios verify the committed image and the exported archive actually exist. Both also drop their fixed image tags and cwd-relative tar outputs for project-scoped tags and temp files, so parallel runs cannot collide and nothing is left behind in the repo. TestLocalComposeLogs waits for the echo service to exit before reading logs (Eventually), closing the race the legacy test tolerated. The follow and large-logs tests stay legacy: both drive a long-running command in the background. ps_test.go stays legacy as well — it verifies ps table/JSON formatting, which is inherently output-shaped. Fixtures configs/, commit/ and export/ are removed. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
- pull: the CLI's Pulled/Skipped report is the observable, so the five scenarios (always-pull, present-locally, build-only, failure, --ignore-pull-failures) legitimately match output; models are inline and the whole compose-pull fixture tree — including the orphaned duplicate-images directory no test referenced — is removed. - networks: aliases, links, ipam, network_mode: service, subnet change, mac_address and interface_name become scenarios. The subnet-change scenario locks what actually happens: the network is replaced but the container is reconnected, not recreated (NotRecreated), which the legacy test never distinguished. TestNetworks (fixed host ports, HTTP fixture services) and TestNetworkRecreate (OR-shaped output assertion) stay legacy. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
- exec: exit-code propagation, -e env forwarding and one-off targeting
become two scenarios with an inline model.
- hooks: the thirteen post_start/pre_stop/pre_start tests migrate. The
volume probe is a named action (probeVolume); wc -l counts are locked
by matching the exact '1 /mnt/tokens.txt' line instead of parsing.
The spec-change scenario replaces its two fixture files with one
model interpolating ${HOOK_VERSION}, so the changed spec — and the
Recreated expectation it implies — is visible in the test source.
pre_stop scenarios drop the fixture's fixed 'sample-data' volume
name, which parallel projects used to share.
Fixtures hooks/ and pre_start/ are removed.
Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>- cp: the twelve copy directions become one scenario; Dir() joins the Scenario API for actions that exchange files with the host, and FileContains observes what landed there. The files copied in and out live in the txtar archive instead of writing into the fixture tree. - environment: TestEnvPriority splits into three scenarios, one per compose model (environment section, bare, env_file section), each declaring its .env variants inline; StdoutContains joins the vocabulary so container output cannot collide with progress noise (the unused framework predicate of the same name is removed). The fixed 'env-compose-priority' image name becomes project-scoped. - config: rendering assertions move to scenario steps; the config output is the observable here by definition. TestConfigHashMatchesContainerLabel stays legacy: it cross-references hashes captured at runtime, which checks cannot hold. - Fixtures cp-test/, environment/ and config/ are removed. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The dry-run publish checks split into nine scenarios, one per model and refusal reason; the prompt text, refusal exit code (130) and publication report are the observables. WithStdin joins the Action modifiers to feed prompt answers. TestPublish stays legacy: it drives a real registry container and loads oci:// projects with multiple -f files. Only the oci/ fixture subtree it needs survives. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
- compose run splits into seven scenarios (one-off basics, ports, deps, optional deps, quiet pull, --pull always, chained build contexts). The orphan-warning cases fold into the basic scenario: an exited one-off IS the orphan the next run warns about, no second compose file needed. Piped-stdin and stop-signal tests stay legacy. - up/compose_up: unhealthy service, dependency exit (both now also locking that dependents stay in created state), build dependencies (project-scoped image name instead of the fixture's global one), optional dependency, --all-resources, profile targeting (dropping the fixture's global container_names), image-by-ID, exit-code-from (42 and 143), port ranges, stdout/stderr separation (StderrContains joins the vocabulary) and logging-driver reconfiguration (now locking the Recreated expectation). The Ctrl-C dependency test stays legacy. - Fixtures start-fail/, resources/, stop/, port-range/, stdout-stderr/, logging-driver/, profiles/ and most of dependencies/ and run-test/ are removed. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
- volumes: external-volume switching, volume recreation on definition change and image volumes become scenarios; the two-compose-file fixtures collapse into one model with an interpolated volume name or label, so the changed definition is visible in the test. The rebuild scenario locks the Recreated expectation and scopes the source image to the project. TestLocalComposeVolume (fixed host port, HTTP) and TestProjectVolumeBind (host file written mid-test) stay legacy. - build-once: the three run --build regression cases each get a scenario; OutputMatchesCount joins the vocabulary to count the build reports, replacing the hand-rolled regex counter, and image names are project-scoped instead of random suffixes. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
- TestExec locks the container-selection order (service container over one-off) with one-off state observed at each step. - providers: a shared providerScenario helper wires the example-provider binary into PATH; the env assertions anchor on the 'test-1 | ' log prefix, replacing the hand-rolled env parser, and the stop-hook sentinel file is observed with FileExists. - Fixtures exec/ and providers/ are removed. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
…han scoping - TestRm, TestRemoveOrphaned and TestUnnecessaryResources observe which containers exist instead of grepping docker ps. - TestCompatibility drops the heavyweight sentences fixture: underscore naming shows on any model. - TestConfig/TestConfigInterpolate build their expected rendering from the scenario's own project name and directory; OutputMatches joins the vocabulary for TestInitContainer's ordering assertion. - TestStopWithDependenciesAttached inlines its model. - Kept legacy: TestLocalComposeUp (fixed port, HTTP, shared sentences fixture), TestDownComposefileInParentFolder (project name inferred from a fixture-relative directory), TestAttachRestart (long-running attached up), the COMPOSE_FILE-from-.env family and TestNestedDotEnv (they exercise invocation without -f, which ComposeCmd always sets). - Fixtures init-container/, external/ and dependencies/ are removed. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
TestBuildDependsOn, Subset, DependentImage, SubDependencies, LongOutputLine, DependentImageWithProfile, Escaped, Tags and Secrets become scenarios: models inline (the long-line Dockerfile generated with strings.Repeat instead of 34 copy-pasted lines), image names project-scoped with Defer cleanup, and the secrets build asserts ImageExists on top of the Dockerfile's own diff checks. Kept legacy: TestLocalComposeBuild (DOCKER_BUILDKIT=0/1 matrix over a shared fixture with fixed names and host port), the buildx-builder family (SSH, platforms, privileged, entitlements, builder, TLS) which provisions builders and dind daemons, and TestBuildImageDependencies (classic/bake matrix). Their fixtures under build-test/ stay; the single-purpose subdirectories are removed. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
- TestUpIdempotentContainerdStore joins the scenario style of its corner-case siblings: Serial + ContainerdImageStore requirement, the regression story in a comment, NotRecreated as the invariant. The requireContainerdStore helper and the last identity fixture go away. - TestImagesAfterImageRemoved scopes its throwaway tag to the project and observes container state before pulling the image record away. - TestComposeModel inlines its model; verified in CI where the docker-model plugin is installed (the local runner cannot pull the model, as with the legacy test). Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
The plugin/stable/containerd CI job caught a race the DSL had: hook helper containers carry the project and service labels but no replica number, and can be destroyed between the snapshot's ps listing and its batch inspect — which then failed and blanked the entire snapshot, making ServiceState report 'service has no container' while the container was right there. Containers are now inspected one by one, tolerating disappearances, and helper containers (neither numbered as a replica nor one-off) are excluded from snapshots altogether so they can't transiently pollute service checks either. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
…it suite timeout The per-container inspect of the previous fix doubled the suite's wall-clock on the containerd CI job — one docker CLI process per container per snapshot — pushing it past go test's default 10m and killing the run mid-flight. Snapshots go back to one batch inspect, falling back to individual inspects only when the batch fails because a helper container vanished. The e2e make targets also pass an explicit -timeout 20m so a slow job fails on its own results, not on the default timeout. TestUpStopWithLogsMixed becomes deterministic: with a bounded ping the service could exit on its own before the abort reached it and the pre_stop hook never ran; it now pings forever so the abort always interrupts it. Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
This was referenced Aug 18, 2026
glours
requested changes
Aug 18, 2026
glours
left a comment
Contributor
There was a problem hiding this comment.
Few changes to apply but overall look pretty good 👍
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
…x compilation - projectNameFor maps the subtest separator to a double dash so TestFoo/bar and TestFooBar cannot resolve to the same compose project and interfere when running in parallel - copyDir preserves the source file mode instead of forcing 0644, so an executable testdata script stays executable in the scenario's project directory - OutputMatches and OutputMatchesCount compile their pattern once at construction: a malformed pattern panics where the check is declared, and Eventually loops no longer recompile it on every tick Signed-off-by: Nicolas De Loof <nicolas.deloof@gmail.com>
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alternative to #14047 + #14054, as a single PR: the declarative Scenario layer plus the full migration of the e2e suite to it, with each scenario's project files under
testdata/<TestName>/as standalone compose files instead of inline YAML/txtar in the test source.docker compose -f pkg/e2e/testdata/TestRestart/compose.yaml upworks as-is. Multi-file projects (Dockerfile, .env, includes) need no special format.testdata/<t.Name()>/by convention (subtests map to nested directories) and copies it to a temp dir, so committed files are never mutated.testdatadirectory — no shared fixtures, no leftovers after a rename, no catch-all. Models previously shared between tests through Go constants are materialized per test.fixtures/shrinks from 70 to 28 entries; the remainder belongs to tests still in legacy style).Only one of #14047/#14054 and this PR should be merged.
🤖 Generated with Claude Code