e2e failure propagation by solsson · Pull Request #1 · Yolean/buckety-controller

solsson · 2026-07-13T11:14:35Z

No description provided.

run_scenario is invoked in an if condition, where bash ignores set -e inside the function body. Every failure inside it, including a non-zero assert.sh, fell through to the PASS logging and returned 0, so the harness has never been able to report FAIL. The latest green main run (27607054780) shows six scenario executions with explicit [assert][FAIL] or wait timeouts, all recorded PASS. Each step now propagates failure explicitly with || return 1. Three more defects the same run exposed: - kafka_topic_exists/s3_bucket_exists use kubectl run -i, which forwards and consumes the caller's stdin. The harness fed its scenario list on stdin of a while read loop, so the first assert that ran an -i helper swallowed the remaining list: only 4 of 11 kadm scenarios ever executed. The list is now materialised with mapfile and assert.sh runs with stdin from /dev/null. - The helpers piped kubectl run into grep -q under pipefail, where an early grep exit SIGPIPEs kubectl and fails the pipeline even on a present topic, with stderr discarded. Output is now captured and printed on failure so assertion failures are diagnosable. - teardown between implementations deleted the CRDs while scenario CRs with finalizers could still be terminating, leaving the CRD stuck and the next implementation's applies rejected with "create not allowed while custom resource definition is terminating" (the entire versitygw block in that run). Overlays now converge in place; nothing deletes CRDs mid-run. Scenario-level fixes entailed by unmasking: - driver-version defaulted E2E_IMAGE_* to a ghcr :dev tag that is never published, wedging the Deployment for two rollout timeouts per run. The vars are now pass-through inputs and the scenario logs SKIPPED when they are unset; CI provides real rotation images (separate commit). - oob-drift waits 90-300s for periodic drift repair, but the controller default recheck is 5m. e2e overlays now pass --periodic-recheck=15s. Failed scenarios leave their namespace standing per SPEC "E2E harness and parity" #3; KEEP_FAILED now additionally preserves namespaces of passing scenarios. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n CI The reconciler gated the driver-major comparison on status.driverMajor != 0, treating 0 as "not yet stamped". Both shipped drivers are 0.x, so every existing resource is stamped major=0 and the check never ran: a 0.x to 1.0 driver bump would silently auto-apply instead of surfacing DriverVersionIncompatible, contradicting SPEC "Driver versioning". Stampedness is now signalled by status.backend, which is set in the same first reconcile pass. The driver-version e2e scenario (SPEC scenario #9) needs controller images built with rotated driver versions. CI never built them, so the scenario has never actually run. The workflow now builds patch (0.1.1) and major (1.0.0) driver builds per SCAFFOLDING "Driver version stamping" and pushes them to the run's local registry; the harness receives them via E2E_IMAGE_*/E2E_VERSION_*. These images are rotation fixtures only, not digest-pinned and not published. Release pin bumped for the reconciler change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

First honest run (29073773069): 15 PASS, 6 FAIL. The six, with root causes: - driver-version: status.driverMajor read back empty. The field was int with omitempty and the stamp used a merge patch, so a stamped 0 (any 0.x driver) neither serializes nor diffs and can never reach the API server. omitempty dropped; the first-reconcile stamp is a Status().Update() since a merge diff of 0 -> 0 is still empty. - misconfigured-startup: the scenario is config-only (resources: []) and plain kubectl apply -k errors with "no objects passed to apply". The harness now renders first and skips apply when the output is empty. - s3/happy-path on versitygw AND minio: the roundtrip Job upload and download both succeed, then the final comparison calls diff, which does not exist in the aws-cli image. Compare via the shell instead. - kadm/oob-drift: assert bug. awk /retention.ms/ also matches retention.local.target.ms, so the check compared a two-line string against 3600000. The controller had in fact restored the value. Exact-match awk. - kadm/parameter-mutation: same awk bug, plus the broker check was a single read immediately after Ready; the alter is acked before the config is necessarily visible to DescribeConfigs. Exact-match awk and a 60s poll like the oob-drift scenario already uses. Release pin bumped for the status stamping change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

kadm/retention-policy flaked in one of two runs of the same sha: topic creation is acked before a fresh rpk metadata request necessarily lists the topic, so single-shot existence checks race. kafka_topic_exists now polls (6x5s), and the deleted-topic check moves into a matching kafka_topic_absent helper. The same-sha double run came from push + pull_request both triggering on PR branches; push is now main-only. No binary change; release pin untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

solsson · 2026-07-13T11:14:35Z

No description provided.

run_scenario is invoked in an if condition, where bash ignores set -e inside the function body. Every failure inside it, including a non-zero assert.sh, fell through to the PASS logging and returned 0, so the harness has never been able to report FAIL. The latest green main run (27607054780) shows six scenario executions with explicit [assert][FAIL] or wait timeouts, all recorded PASS. Each step now propagates failure explicitly with || return 1. Three more defects the same run exposed: - kafka_topic_exists/s3_bucket_exists use kubectl run -i, which forwards and consumes the caller's stdin. The harness fed its scenario list on stdin of a while read loop, so the first assert that ran an -i helper swallowed the remaining list: only 4 of 11 kadm scenarios ever executed. The list is now materialised with mapfile and assert.sh runs with stdin from /dev/null. - The helpers piped kubectl run into grep -q under pipefail, where an early grep exit SIGPIPEs kubectl and fails the pipeline even on a present topic, with stderr discarded. Output is now captured and printed on failure so assertion failures are diagnosable. - teardown between implementations deleted the CRDs while scenario CRs with finalizers could still be terminating, leaving the CRD stuck and the next implementation's applies rejected with "create not allowed while custom resource definition is terminating" (the entire versitygw block in that run). Overlays now converge in place; nothing deletes CRDs mid-run. Scenario-level fixes entailed by unmasking: - driver-version defaulted E2E_IMAGE_* to a ghcr :dev tag that is never published, wedging the Deployment for two rollout timeouts per run. The vars are now pass-through inputs and the scenario logs SKIPPED when they are unset; CI provides real rotation images (separate commit). - oob-drift waits 90-300s for periodic drift repair, but the controller default recheck is 5m. e2e overlays now pass --periodic-recheck=15s. Failed scenarios leave their namespace standing per SPEC "E2E harness and parity" #3; KEEP_FAILED now additionally preserves namespaces of passing scenarios. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n CI The reconciler gated the driver-major comparison on status.driverMajor != 0, treating 0 as "not yet stamped". Both shipped drivers are 0.x, so every existing resource is stamped major=0 and the check never ran: a 0.x to 1.0 driver bump would silently auto-apply instead of surfacing DriverVersionIncompatible, contradicting SPEC "Driver versioning". Stampedness is now signalled by status.backend, which is set in the same first reconcile pass. The driver-version e2e scenario (SPEC scenario #9) needs controller images built with rotated driver versions. CI never built them, so the scenario has never actually run. The workflow now builds patch (0.1.1) and major (1.0.0) driver builds per SCAFFOLDING "Driver version stamping" and pushes them to the run's local registry; the harness receives them via E2E_IMAGE_*/E2E_VERSION_*. These images are rotation fixtures only, not digest-pinned and not published. Release pin bumped for the reconciler change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

First honest run (29073773069): 15 PASS, 6 FAIL. The six, with root causes: - driver-version: status.driverMajor read back empty. The field was int with omitempty and the stamp used a merge patch, so a stamped 0 (any 0.x driver) neither serializes nor diffs and can never reach the API server. omitempty dropped; the first-reconcile stamp is a Status().Update() since a merge diff of 0 -> 0 is still empty. - misconfigured-startup: the scenario is config-only (resources: []) and plain kubectl apply -k errors with "no objects passed to apply". The harness now renders first and skips apply when the output is empty. - s3/happy-path on versitygw AND minio: the roundtrip Job upload and download both succeed, then the final comparison calls diff, which does not exist in the aws-cli image. Compare via the shell instead. - kadm/oob-drift: assert bug. awk /retention.ms/ also matches retention.local.target.ms, so the check compared a two-line string against 3600000. The controller had in fact restored the value. Exact-match awk. - kadm/parameter-mutation: same awk bug, plus the broker check was a single read immediately after Ready; the alter is acked before the config is necessarily visible to DescribeConfigs. Exact-match awk and a 60s poll like the oob-drift scenario already uses. Release pin bumped for the status stamping change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

kadm/retention-policy flaked in one of two runs of the same sha: topic creation is acked before a fresh rpk metadata request necessarily lists the topic, so single-shot existence checks race. kafka_topic_exists now polls (6x5s), and the deleted-topic check moves into a matching kafka_topic_absent helper. The same-sha double run came from push + pull_request both triggering on PR branches; push is now main-only. No binary change; release pin untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

solsson · 2026-07-13T11:14:35Z

No description provided.

run_scenario is invoked in an if condition, where bash ignores set -e inside the function body. Every failure inside it, including a non-zero assert.sh, fell through to the PASS logging and returned 0, so the harness has never been able to report FAIL. The latest green main run (27607054780) shows six scenario executions with explicit [assert][FAIL] or wait timeouts, all recorded PASS. Each step now propagates failure explicitly with || return 1. Three more defects the same run exposed: - kafka_topic_exists/s3_bucket_exists use kubectl run -i, which forwards and consumes the caller's stdin. The harness fed its scenario list on stdin of a while read loop, so the first assert that ran an -i helper swallowed the remaining list: only 4 of 11 kadm scenarios ever executed. The list is now materialised with mapfile and assert.sh runs with stdin from /dev/null. - The helpers piped kubectl run into grep -q under pipefail, where an early grep exit SIGPIPEs kubectl and fails the pipeline even on a present topic, with stderr discarded. Output is now captured and printed on failure so assertion failures are diagnosable. - teardown between implementations deleted the CRDs while scenario CRs with finalizers could still be terminating, leaving the CRD stuck and the next implementation's applies rejected with "create not allowed while custom resource definition is terminating" (the entire versitygw block in that run). Overlays now converge in place; nothing deletes CRDs mid-run. Scenario-level fixes entailed by unmasking: - driver-version defaulted E2E_IMAGE_* to a ghcr :dev tag that is never published, wedging the Deployment for two rollout timeouts per run. The vars are now pass-through inputs and the scenario logs SKIPPED when they are unset; CI provides real rotation images (separate commit). - oob-drift waits 90-300s for periodic drift repair, but the controller default recheck is 5m. e2e overlays now pass --periodic-recheck=15s. Failed scenarios leave their namespace standing per SPEC "E2E harness and parity" #3; KEEP_FAILED now additionally preserves namespaces of passing scenarios. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n CI The reconciler gated the driver-major comparison on status.driverMajor != 0, treating 0 as "not yet stamped". Both shipped drivers are 0.x, so every existing resource is stamped major=0 and the check never ran: a 0.x to 1.0 driver bump would silently auto-apply instead of surfacing DriverVersionIncompatible, contradicting SPEC "Driver versioning". Stampedness is now signalled by status.backend, which is set in the same first reconcile pass. The driver-version e2e scenario (SPEC scenario #9) needs controller images built with rotated driver versions. CI never built them, so the scenario has never actually run. The workflow now builds patch (0.1.1) and major (1.0.0) driver builds per SCAFFOLDING "Driver version stamping" and pushes them to the run's local registry; the harness receives them via E2E_IMAGE_*/E2E_VERSION_*. These images are rotation fixtures only, not digest-pinned and not published. Release pin bumped for the reconciler change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

First honest run (29073773069): 15 PASS, 6 FAIL. The six, with root causes: - driver-version: status.driverMajor read back empty. The field was int with omitempty and the stamp used a merge patch, so a stamped 0 (any 0.x driver) neither serializes nor diffs and can never reach the API server. omitempty dropped; the first-reconcile stamp is a Status().Update() since a merge diff of 0 -> 0 is still empty. - misconfigured-startup: the scenario is config-only (resources: []) and plain kubectl apply -k errors with "no objects passed to apply". The harness now renders first and skips apply when the output is empty. - s3/happy-path on versitygw AND minio: the roundtrip Job upload and download both succeed, then the final comparison calls diff, which does not exist in the aws-cli image. Compare via the shell instead. - kadm/oob-drift: assert bug. awk /retention.ms/ also matches retention.local.target.ms, so the check compared a two-line string against 3600000. The controller had in fact restored the value. Exact-match awk. - kadm/parameter-mutation: same awk bug, plus the broker check was a single read immediately after Ready; the alter is acked before the config is necessarily visible to DescribeConfigs. Exact-match awk and a 60s poll like the oob-drift scenario already uses. Release pin bumped for the status stamping change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

kadm/retention-policy flaked in one of two runs of the same sha: topic creation is acked before a fresh rpk metadata request necessarily lists the topic, so single-shot existence checks race. kafka_topic_exists now polls (6x5s), and the deleted-topic check moves into a matching kafka_topic_absent helper. The same-sha double run came from push + pull_request both triggering on PR branches; push is now main-only. No binary change; release pin untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

solsson · 2026-07-13T11:14:35Z

No description provided.

run_scenario is invoked in an if condition, where bash ignores set -e inside the function body. Every failure inside it, including a non-zero assert.sh, fell through to the PASS logging and returned 0, so the harness has never been able to report FAIL. The latest green main run (27607054780) shows six scenario executions with explicit [assert][FAIL] or wait timeouts, all recorded PASS. Each step now propagates failure explicitly with || return 1. Three more defects the same run exposed: - kafka_topic_exists/s3_bucket_exists use kubectl run -i, which forwards and consumes the caller's stdin. The harness fed its scenario list on stdin of a while read loop, so the first assert that ran an -i helper swallowed the remaining list: only 4 of 11 kadm scenarios ever executed. The list is now materialised with mapfile and assert.sh runs with stdin from /dev/null. - The helpers piped kubectl run into grep -q under pipefail, where an early grep exit SIGPIPEs kubectl and fails the pipeline even on a present topic, with stderr discarded. Output is now captured and printed on failure so assertion failures are diagnosable. - teardown between implementations deleted the CRDs while scenario CRs with finalizers could still be terminating, leaving the CRD stuck and the next implementation's applies rejected with "create not allowed while custom resource definition is terminating" (the entire versitygw block in that run). Overlays now converge in place; nothing deletes CRDs mid-run. Scenario-level fixes entailed by unmasking: - driver-version defaulted E2E_IMAGE_* to a ghcr :dev tag that is never published, wedging the Deployment for two rollout timeouts per run. The vars are now pass-through inputs and the scenario logs SKIPPED when they are unset; CI provides real rotation images (separate commit). - oob-drift waits 90-300s for periodic drift repair, but the controller default recheck is 5m. e2e overlays now pass --periodic-recheck=15s. Failed scenarios leave their namespace standing per SPEC "E2E harness and parity" #3; KEEP_FAILED now additionally preserves namespaces of passing scenarios. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n CI The reconciler gated the driver-major comparison on status.driverMajor != 0, treating 0 as "not yet stamped". Both shipped drivers are 0.x, so every existing resource is stamped major=0 and the check never ran: a 0.x to 1.0 driver bump would silently auto-apply instead of surfacing DriverVersionIncompatible, contradicting SPEC "Driver versioning". Stampedness is now signalled by status.backend, which is set in the same first reconcile pass. The driver-version e2e scenario (SPEC scenario #9) needs controller images built with rotated driver versions. CI never built them, so the scenario has never actually run. The workflow now builds patch (0.1.1) and major (1.0.0) driver builds per SCAFFOLDING "Driver version stamping" and pushes them to the run's local registry; the harness receives them via E2E_IMAGE_*/E2E_VERSION_*. These images are rotation fixtures only, not digest-pinned and not published. Release pin bumped for the reconciler change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

First honest run (29073773069): 15 PASS, 6 FAIL. The six, with root causes: - driver-version: status.driverMajor read back empty. The field was int with omitempty and the stamp used a merge patch, so a stamped 0 (any 0.x driver) neither serializes nor diffs and can never reach the API server. omitempty dropped; the first-reconcile stamp is a Status().Update() since a merge diff of 0 -> 0 is still empty. - misconfigured-startup: the scenario is config-only (resources: []) and plain kubectl apply -k errors with "no objects passed to apply". The harness now renders first and skips apply when the output is empty. - s3/happy-path on versitygw AND minio: the roundtrip Job upload and download both succeed, then the final comparison calls diff, which does not exist in the aws-cli image. Compare via the shell instead. - kadm/oob-drift: assert bug. awk /retention.ms/ also matches retention.local.target.ms, so the check compared a two-line string against 3600000. The controller had in fact restored the value. Exact-match awk. - kadm/parameter-mutation: same awk bug, plus the broker check was a single read immediately after Ready; the alter is acked before the config is necessarily visible to DescribeConfigs. Exact-match awk and a 60s poll like the oob-drift scenario already uses. Release pin bumped for the status stamping change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

kadm/retention-policy flaked in one of two runs of the same sha: topic creation is acked before a fresh rpk metadata request necessarily lists the topic, so single-shot existence checks race. kafka_topic_exists now polls (6x5s), and the deleted-topic check moves into a matching kafka_topic_absent helper. The same-sha double run came from push + pull_request both triggering on PR branches; push is now main-only. No binary change; release pin untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

solsson · 2026-07-13T11:14:35Z

No description provided.

run_scenario is invoked in an if condition, where bash ignores set -e inside the function body. Every failure inside it, including a non-zero assert.sh, fell through to the PASS logging and returned 0, so the harness has never been able to report FAIL. The latest green main run (27607054780) shows six scenario executions with explicit [assert][FAIL] or wait timeouts, all recorded PASS. Each step now propagates failure explicitly with || return 1. Three more defects the same run exposed: - kafka_topic_exists/s3_bucket_exists use kubectl run -i, which forwards and consumes the caller's stdin. The harness fed its scenario list on stdin of a while read loop, so the first assert that ran an -i helper swallowed the remaining list: only 4 of 11 kadm scenarios ever executed. The list is now materialised with mapfile and assert.sh runs with stdin from /dev/null. - The helpers piped kubectl run into grep -q under pipefail, where an early grep exit SIGPIPEs kubectl and fails the pipeline even on a present topic, with stderr discarded. Output is now captured and printed on failure so assertion failures are diagnosable. - teardown between implementations deleted the CRDs while scenario CRs with finalizers could still be terminating, leaving the CRD stuck and the next implementation's applies rejected with "create not allowed while custom resource definition is terminating" (the entire versitygw block in that run). Overlays now converge in place; nothing deletes CRDs mid-run. Scenario-level fixes entailed by unmasking: - driver-version defaulted E2E_IMAGE_* to a ghcr :dev tag that is never published, wedging the Deployment for two rollout timeouts per run. The vars are now pass-through inputs and the scenario logs SKIPPED when they are unset; CI provides real rotation images (separate commit). - oob-drift waits 90-300s for periodic drift repair, but the controller default recheck is 5m. e2e overlays now pass --periodic-recheck=15s. Failed scenarios leave their namespace standing per SPEC "E2E harness and parity" #3; KEEP_FAILED now additionally preserves namespaces of passing scenarios. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n CI The reconciler gated the driver-major comparison on status.driverMajor != 0, treating 0 as "not yet stamped". Both shipped drivers are 0.x, so every existing resource is stamped major=0 and the check never ran: a 0.x to 1.0 driver bump would silently auto-apply instead of surfacing DriverVersionIncompatible, contradicting SPEC "Driver versioning". Stampedness is now signalled by status.backend, which is set in the same first reconcile pass. The driver-version e2e scenario (SPEC scenario #9) needs controller images built with rotated driver versions. CI never built them, so the scenario has never actually run. The workflow now builds patch (0.1.1) and major (1.0.0) driver builds per SCAFFOLDING "Driver version stamping" and pushes them to the run's local registry; the harness receives them via E2E_IMAGE_*/E2E_VERSION_*. These images are rotation fixtures only, not digest-pinned and not published. Release pin bumped for the reconciler change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

First honest run (29073773069): 15 PASS, 6 FAIL. The six, with root causes: - driver-version: status.driverMajor read back empty. The field was int with omitempty and the stamp used a merge patch, so a stamped 0 (any 0.x driver) neither serializes nor diffs and can never reach the API server. omitempty dropped; the first-reconcile stamp is a Status().Update() since a merge diff of 0 -> 0 is still empty. - misconfigured-startup: the scenario is config-only (resources: []) and plain kubectl apply -k errors with "no objects passed to apply". The harness now renders first and skips apply when the output is empty. - s3/happy-path on versitygw AND minio: the roundtrip Job upload and download both succeed, then the final comparison calls diff, which does not exist in the aws-cli image. Compare via the shell instead. - kadm/oob-drift: assert bug. awk /retention.ms/ also matches retention.local.target.ms, so the check compared a two-line string against 3600000. The controller had in fact restored the value. Exact-match awk. - kadm/parameter-mutation: same awk bug, plus the broker check was a single read immediately after Ready; the alter is acked before the config is necessarily visible to DescribeConfigs. Exact-match awk and a 60s poll like the oob-drift scenario already uses. Release pin bumped for the status stamping change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

kadm/retention-policy flaked in one of two runs of the same sha: topic creation is acked before a fresh rpk metadata request necessarily lists the topic, so single-shot existence checks race. kafka_topic_exists now polls (6x5s), and the deleted-topic check moves into a matching kafka_topic_absent helper. The same-sha double run came from push + pull_request both triggering on PR branches; push is now main-only. No binary change; release pin untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

solsson · 2026-07-13T11:14:35Z

No description provided.

run_scenario is invoked in an if condition, where bash ignores set -e inside the function body. Every failure inside it, including a non-zero assert.sh, fell through to the PASS logging and returned 0, so the harness has never been able to report FAIL. The latest green main run (27607054780) shows six scenario executions with explicit [assert][FAIL] or wait timeouts, all recorded PASS. Each step now propagates failure explicitly with || return 1. Three more defects the same run exposed: - kafka_topic_exists/s3_bucket_exists use kubectl run -i, which forwards and consumes the caller's stdin. The harness fed its scenario list on stdin of a while read loop, so the first assert that ran an -i helper swallowed the remaining list: only 4 of 11 kadm scenarios ever executed. The list is now materialised with mapfile and assert.sh runs with stdin from /dev/null. - The helpers piped kubectl run into grep -q under pipefail, where an early grep exit SIGPIPEs kubectl and fails the pipeline even on a present topic, with stderr discarded. Output is now captured and printed on failure so assertion failures are diagnosable. - teardown between implementations deleted the CRDs while scenario CRs with finalizers could still be terminating, leaving the CRD stuck and the next implementation's applies rejected with "create not allowed while custom resource definition is terminating" (the entire versitygw block in that run). Overlays now converge in place; nothing deletes CRDs mid-run. Scenario-level fixes entailed by unmasking: - driver-version defaulted E2E_IMAGE_* to a ghcr :dev tag that is never published, wedging the Deployment for two rollout timeouts per run. The vars are now pass-through inputs and the scenario logs SKIPPED when they are unset; CI provides real rotation images (separate commit). - oob-drift waits 90-300s for periodic drift repair, but the controller default recheck is 5m. e2e overlays now pass --periodic-recheck=15s. Failed scenarios leave their namespace standing per SPEC "E2E harness and parity" #3; KEEP_FAILED now additionally preserves namespaces of passing scenarios. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n CI The reconciler gated the driver-major comparison on status.driverMajor != 0, treating 0 as "not yet stamped". Both shipped drivers are 0.x, so every existing resource is stamped major=0 and the check never ran: a 0.x to 1.0 driver bump would silently auto-apply instead of surfacing DriverVersionIncompatible, contradicting SPEC "Driver versioning". Stampedness is now signalled by status.backend, which is set in the same first reconcile pass. The driver-version e2e scenario (SPEC scenario #9) needs controller images built with rotated driver versions. CI never built them, so the scenario has never actually run. The workflow now builds patch (0.1.1) and major (1.0.0) driver builds per SCAFFOLDING "Driver version stamping" and pushes them to the run's local registry; the harness receives them via E2E_IMAGE_*/E2E_VERSION_*. These images are rotation fixtures only, not digest-pinned and not published. Release pin bumped for the reconciler change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

First honest run (29073773069): 15 PASS, 6 FAIL. The six, with root causes: - driver-version: status.driverMajor read back empty. The field was int with omitempty and the stamp used a merge patch, so a stamped 0 (any 0.x driver) neither serializes nor diffs and can never reach the API server. omitempty dropped; the first-reconcile stamp is a Status().Update() since a merge diff of 0 -> 0 is still empty. - misconfigured-startup: the scenario is config-only (resources: []) and plain kubectl apply -k errors with "no objects passed to apply". The harness now renders first and skips apply when the output is empty. - s3/happy-path on versitygw AND minio: the roundtrip Job upload and download both succeed, then the final comparison calls diff, which does not exist in the aws-cli image. Compare via the shell instead. - kadm/oob-drift: assert bug. awk /retention.ms/ also matches retention.local.target.ms, so the check compared a two-line string against 3600000. The controller had in fact restored the value. Exact-match awk. - kadm/parameter-mutation: same awk bug, plus the broker check was a single read immediately after Ready; the alter is acked before the config is necessarily visible to DescribeConfigs. Exact-match awk and a 60s poll like the oob-drift scenario already uses. Release pin bumped for the status stamping change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

kadm/retention-policy flaked in one of two runs of the same sha: topic creation is acked before a fresh rpk metadata request necessarily lists the topic, so single-shot existence checks race. kafka_topic_exists now polls (6x5s), and the deleted-topic check moves into a matching kafka_topic_absent helper. The same-sha double run came from push + pull_request both triggering on PR branches; push is now main-only. No binary change; release pin untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

solsson · 2026-07-13T11:14:35Z

No description provided.

run_scenario is invoked in an if condition, where bash ignores set -e inside the function body. Every failure inside it, including a non-zero assert.sh, fell through to the PASS logging and returned 0, so the harness has never been able to report FAIL. The latest green main run (27607054780) shows six scenario executions with explicit [assert][FAIL] or wait timeouts, all recorded PASS. Each step now propagates failure explicitly with || return 1. Three more defects the same run exposed: - kafka_topic_exists/s3_bucket_exists use kubectl run -i, which forwards and consumes the caller's stdin. The harness fed its scenario list on stdin of a while read loop, so the first assert that ran an -i helper swallowed the remaining list: only 4 of 11 kadm scenarios ever executed. The list is now materialised with mapfile and assert.sh runs with stdin from /dev/null. - The helpers piped kubectl run into grep -q under pipefail, where an early grep exit SIGPIPEs kubectl and fails the pipeline even on a present topic, with stderr discarded. Output is now captured and printed on failure so assertion failures are diagnosable. - teardown between implementations deleted the CRDs while scenario CRs with finalizers could still be terminating, leaving the CRD stuck and the next implementation's applies rejected with "create not allowed while custom resource definition is terminating" (the entire versitygw block in that run). Overlays now converge in place; nothing deletes CRDs mid-run. Scenario-level fixes entailed by unmasking: - driver-version defaulted E2E_IMAGE_* to a ghcr :dev tag that is never published, wedging the Deployment for two rollout timeouts per run. The vars are now pass-through inputs and the scenario logs SKIPPED when they are unset; CI provides real rotation images (separate commit). - oob-drift waits 90-300s for periodic drift repair, but the controller default recheck is 5m. e2e overlays now pass --periodic-recheck=15s. Failed scenarios leave their namespace standing per SPEC "E2E harness and parity" #3; KEEP_FAILED now additionally preserves namespaces of passing scenarios. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n CI The reconciler gated the driver-major comparison on status.driverMajor != 0, treating 0 as "not yet stamped". Both shipped drivers are 0.x, so every existing resource is stamped major=0 and the check never ran: a 0.x to 1.0 driver bump would silently auto-apply instead of surfacing DriverVersionIncompatible, contradicting SPEC "Driver versioning". Stampedness is now signalled by status.backend, which is set in the same first reconcile pass. The driver-version e2e scenario (SPEC scenario #9) needs controller images built with rotated driver versions. CI never built them, so the scenario has never actually run. The workflow now builds patch (0.1.1) and major (1.0.0) driver builds per SCAFFOLDING "Driver version stamping" and pushes them to the run's local registry; the harness receives them via E2E_IMAGE_*/E2E_VERSION_*. These images are rotation fixtures only, not digest-pinned and not published. Release pin bumped for the reconciler change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

First honest run (29073773069): 15 PASS, 6 FAIL. The six, with root causes: - driver-version: status.driverMajor read back empty. The field was int with omitempty and the stamp used a merge patch, so a stamped 0 (any 0.x driver) neither serializes nor diffs and can never reach the API server. omitempty dropped; the first-reconcile stamp is a Status().Update() since a merge diff of 0 -> 0 is still empty. - misconfigured-startup: the scenario is config-only (resources: []) and plain kubectl apply -k errors with "no objects passed to apply". The harness now renders first and skips apply when the output is empty. - s3/happy-path on versitygw AND minio: the roundtrip Job upload and download both succeed, then the final comparison calls diff, which does not exist in the aws-cli image. Compare via the shell instead. - kadm/oob-drift: assert bug. awk /retention.ms/ also matches retention.local.target.ms, so the check compared a two-line string against 3600000. The controller had in fact restored the value. Exact-match awk. - kadm/parameter-mutation: same awk bug, plus the broker check was a single read immediately after Ready; the alter is acked before the config is necessarily visible to DescribeConfigs. Exact-match awk and a 60s poll like the oob-drift scenario already uses. Release pin bumped for the status stamping change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

kadm/retention-policy flaked in one of two runs of the same sha: topic creation is acked before a fresh rpk metadata request necessarily lists the topic, so single-shot existence checks race. kafka_topic_exists now polls (6x5s), and the deleted-topic check moves into a matching kafka_topic_absent helper. The same-sha double run came from push + pull_request both triggering on PR branches; push is now main-only. No binary change; release pin untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

solsson · 2026-07-13T11:14:35Z

No description provided.

run_scenario is invoked in an if condition, where bash ignores set -e inside the function body. Every failure inside it, including a non-zero assert.sh, fell through to the PASS logging and returned 0, so the harness has never been able to report FAIL. The latest green main run (27607054780) shows six scenario executions with explicit [assert][FAIL] or wait timeouts, all recorded PASS. Each step now propagates failure explicitly with || return 1. Three more defects the same run exposed: - kafka_topic_exists/s3_bucket_exists use kubectl run -i, which forwards and consumes the caller's stdin. The harness fed its scenario list on stdin of a while read loop, so the first assert that ran an -i helper swallowed the remaining list: only 4 of 11 kadm scenarios ever executed. The list is now materialised with mapfile and assert.sh runs with stdin from /dev/null. - The helpers piped kubectl run into grep -q under pipefail, where an early grep exit SIGPIPEs kubectl and fails the pipeline even on a present topic, with stderr discarded. Output is now captured and printed on failure so assertion failures are diagnosable. - teardown between implementations deleted the CRDs while scenario CRs with finalizers could still be terminating, leaving the CRD stuck and the next implementation's applies rejected with "create not allowed while custom resource definition is terminating" (the entire versitygw block in that run). Overlays now converge in place; nothing deletes CRDs mid-run. Scenario-level fixes entailed by unmasking: - driver-version defaulted E2E_IMAGE_* to a ghcr :dev tag that is never published, wedging the Deployment for two rollout timeouts per run. The vars are now pass-through inputs and the scenario logs SKIPPED when they are unset; CI provides real rotation images (separate commit). - oob-drift waits 90-300s for periodic drift repair, but the controller default recheck is 5m. e2e overlays now pass --periodic-recheck=15s. Failed scenarios leave their namespace standing per SPEC "E2E harness and parity" #3; KEEP_FAILED now additionally preserves namespaces of passing scenarios. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…n CI The reconciler gated the driver-major comparison on status.driverMajor != 0, treating 0 as "not yet stamped". Both shipped drivers are 0.x, so every existing resource is stamped major=0 and the check never ran: a 0.x to 1.0 driver bump would silently auto-apply instead of surfacing DriverVersionIncompatible, contradicting SPEC "Driver versioning". Stampedness is now signalled by status.backend, which is set in the same first reconcile pass. The driver-version e2e scenario (SPEC scenario #9) needs controller images built with rotated driver versions. CI never built them, so the scenario has never actually run. The workflow now builds patch (0.1.1) and major (1.0.0) driver builds per SCAFFOLDING "Driver version stamping" and pushes them to the run's local registry; the harness receives them via E2E_IMAGE_*/E2E_VERSION_*. These images are rotation fixtures only, not digest-pinned and not published. Release pin bumped for the reconciler change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

First honest run (29073773069): 15 PASS, 6 FAIL. The six, with root causes: - driver-version: status.driverMajor read back empty. The field was int with omitempty and the stamp used a merge patch, so a stamped 0 (any 0.x driver) neither serializes nor diffs and can never reach the API server. omitempty dropped; the first-reconcile stamp is a Status().Update() since a merge diff of 0 -> 0 is still empty. - misconfigured-startup: the scenario is config-only (resources: []) and plain kubectl apply -k errors with "no objects passed to apply". The harness now renders first and skips apply when the output is empty. - s3/happy-path on versitygw AND minio: the roundtrip Job upload and download both succeed, then the final comparison calls diff, which does not exist in the aws-cli image. Compare via the shell instead. - kadm/oob-drift: assert bug. awk /retention.ms/ also matches retention.local.target.ms, so the check compared a two-line string against 3600000. The controller had in fact restored the value. Exact-match awk. - kadm/parameter-mutation: same awk bug, plus the broker check was a single read immediately after Ready; the alter is acked before the config is necessarily visible to DescribeConfigs. Exact-match awk and a 60s poll like the oob-drift scenario already uses. Release pin bumped for the status stamping change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

kadm/retention-policy flaked in one of two runs of the same sha: topic creation is acked before a fresh rpk metadata request necessarily lists the topic, so single-shot existence checks race. kafka_topic_exists now polls (6x5s), and the deleted-topic check moves into a matching kafka_topic_absent helper. The same-sha double run came from push + pull_request both triggering on PR branches; push is now main-only. No binary change; release pin untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Yolean k8s-qa (buckety maintainer)and others added 4 commits July 10, 2026 06:24