sec(1528): a machine verdict for the edgeuser DROP-readiness gate (backend#1528)

[LukasWodka]

Closes tracebloc/backend#2760 — the tooling leg, carved out as its own
closeable slice so this PR has a reference it can legitimately close.
Part of tracebloc/backend#1528, which stays open: the ops legs (driving
the three phases, then the irreversible DROP USER) are not in this PR.

Why

#1528's DROP-readiness gate is three criteria that must each hold under a driven workload, and it has to be evaluated three times per fleet — pre-revoke, post-revoke, post-drop — across the two fleets still carrying edgeuser. By hand that's six judgement calls on prose, immediately before an irreversible DROP USER on a customer-facing database.

And the criterion that matters most — "nothing resolves to edgeuser" — is the easiest to get wrong, because the interesting answer is a variable that isn't there.

The criterion this gate replaced could not fail. A point-in-time information_schema.processlist sample showed no edgeuser on a fleet where edgeuser was still root-equivalent and in active use, because consumers connect per-operation and disconnect — it would have passed at every moment in #1528's life, including before any of the work started. This PR is the machine form of the replacement, so that shape of mistake can't recur silently. The processlist query survives only as a labelled smoke test whose empty result is printed as not evidence of absence.

What it is

docs/migration-tools/edgeuser-drop-readiness.sh — read-only: no REVOKE, no DROP, no writes of any kind. It never prints a password value; for every *_PASSWORD it asserts only SET or UNSET, and passwords reach the client via MYSQL_PWD so they never appear in a pod's argv.

docs/migration-tools/edgeuser-drop-readiness.sh \
--context <ctx> --namespace <ns> \
--baseline-datasets 87 --baseline-metadata 3 --baseline-identity root \
--phase pre-revoke

Three design points are the actual substance:

1. The variable list is derived from the deployed surface. Criterion 1 reads each pod's own resolved environment (kubectl exec … env, which includes envFrom/secretKeyRef values a manifest read can't resolve) rather than a list held in the script. A credential variable added to the chart later is covered the day it ships. A hand-written list would agree with itself and disagree with the cluster — which is precisely what the superseded criterion did.

2. Criterion 3 counts as tb_ingest/tb_meta, never as root.information_schema is privilege-filtered, so a count read as root is the total and can never reveal a shrink — root sees everything by definition. That's why --baseline-identity is mandatory: a root count and a tb_ingest count are different measurements, and comparing them silently is how a shrink hides.

(An earlier version of this paragraph blamed #1528's 3-vs-4 metadata baseline disagreement on the reading identity. That was wrong, and the answer was already on the ticket: metadata went 3 → 4 when perExperimentDbCreds was enabled, because that creates experiment_db_creds. The identity rule above stands on its own — root genuinely cannot reveal a shrink — but it was not the cause of that particular discrepancy. The practical consequence for prod is the opposite of what I implied: expect metadata to grow by exactly one table when the flag goes on, which the script reports as grew by 1 (not a shrink).)

3. The baselines have no defaults. A hardcoded per-fleet table of measurements is the backend#1729 defect this ticket keeps tripping over. The script refuses to run without them.

Fails closed throughout. An absent pod, a refused exec, unavailable logs, a missing driven cycle, an unreachable cluster, an unreadable count — each is a finding, reported as [????] … cannot tell and counted as a failure. "We could not look" must never read the same as "we looked and it was clean."

Verification

scripts/tests/edgeuser-drop-readiness-verdicts.sh — 22 verdict cases, all green, driven by a stubbed kubectl rendering a synthetic fleet. Every refusal case asserts the specific finding, never a bare non-zero exit: any failure satisfies that, and a test that can't say which refusal fired is a coin toss reporting success.

Registered in the single DRIFT_GUARDS declaration, so it runs inside the already-required drift job rather than in a job nobody must wait for.

Mutation-proven, every anchor asserted applied:

Mutation	Cases reddened
the tool always says DROP-READY	14
"cannot tell" stops counting as a failure	3
counted as root instead of tb_ingest	4
the *_USER == edgeuser scan disabled	1
an unset DB_BOOTSTRAP_USER treated as fine	1
the legacy-identity log scan removed	1
the silent-shrink comparison disabled	1
the baseline silently defaults instead of refusing	1

Building the harness also found four real bugs — three in the stub's dispatch order (the mysql queries are themselves invoked as -- env MYSQL_PWD=… mysql …, so a -- env case tried first swallowed all of them) and one in the test's own grep, which read the expected substring as an option. None in the tool, but that is what the harness is for.

Also: drift 28/28, shellcheck -S warning -x + bash -n clean on both files.

Docs

New SECURITY.md §7.7 (in the verification section, where a verifier belongs), cross-referenced from §8.10 step 3 so the REVOKE/DROP step points at the tool instead of at prose.

No chart content changes, so no Chart.yaml bump.

🤖 Generated with Claude Code

---

Note

Medium Risk
Read-only tooling, but a false DROP-READY verdict would gate an irreversible MySQL DROP on prod; impact depends on verifier correctness, mitigated by extensive stubbed verdict tests in drift CI.

Overview
Adds read-only, fail-closed ops tooling for backend#1528’s three-criteria gate before irreversible edgeuserREVOKE/DROP, plus CI and docs so fleets use it instead of hand checks or the old processlist heuristic.

docs/migration-tools/edgeuser-drop-readiness.sh prints a single DROP-READY / NOT DROP-READY verdict against a live cluster (kubectl exec/logs, mandatory fleet baselines, --phase pre-revoke|post-revoke|post-drop). It checks: (1) no consumer env resolves to edgeuser (derived from pod env, mint vs consumer gates aligned to chart templates, running ingestion only, DSN scan); (2) no legacy-identity warnings or 1045 in logs over --since; (3) privilege-filtered table counts vs S0 baselines as tb_ingest/tb_meta, with post-drop mysql.user verified via COUNT (not truncating GROUP_CONCAT). Passwords are never printed.

scripts/tests/edgeuser-drop-readiness-verdicts.sh exercises happy path and every refusal path via a stub kubectl; it is wired into DRIFT_GUARDS in the Makefile so required drift runs it.

docs/SECURITY.md adds §7.7 (how to run and trust the verifier) and points §8.10 S3 at that script instead of eyeballing readiness.

^{Reviewed by Cursor Bugbot for commit 467a0a2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saqlainsyed007]

Requested changes — and note that green CI here is a symptom, not a clearance. I verified the main finding against the chart.

HIGH — the DROP-readiness gate is structurally always-red on a real fleet.check_workload (edgeuser-drop-readiness.sh, the requests-proxy call at ~:176 and the assertions at :159-166) asserts DB_BOOTSTRAP_USER=root, SERVICE_DB_ACCOUNTS, and PER_EXPERIMENT_DB_CREDS on every workload. But client/templates/requests-proxy-deployment.yaml at this head renders only SERVICE_DB_ACCOUNTS and TB_META_USER — it has no DB_BOOTSTRAP_USER and no PER_EXPERIMENT_DB_CREDS (those govern the mint/bootstrap DDL, which only jobs-manager runs). So on a correctly-retired production fleet, requests-proxy trips two bad findings → FAILURES>0 → NOT DROP-READY, in every phase, forever. The tool can never authorize the DROP it exists to gate, and it prints a factually wrong reason ("the mint falls back to edgeuser") for a pod that never mints. Fix: assert the bootstrap/per-experiment gates only on the workload that actually carries them (jobs-manager); on a consumer like requests-proxy, check only what's relevant (SERVICE_DB_ACCOUNTS, and that nothing resolves to edgeuser).

MEDIUM — the happy-path test is vacuous and is what's hiding the above.scripts/tests/edgeuser-drop-readiness-verdicts.sh:103 (clean_fleet) writes the identical $common env — including DB_BOOTSTRAP_USER=root and PER_EXPERIMENT_DB_CREDS=true — to BOTH the jobs-manager and requests-proxy fixtures. Real requests-proxy renders neither, so case 1 ("a clean retired fleet is DROP-READY") passes only against a fleet shape that cannot exist, and every mutation case is differenced against that fiction. A fixture that mirrors the real chart (requests-proxy without those two vars) flips case 1 to NOT-READY and reddens on the real bug. That's why CI is green.

Good news on the safety direction: the failure is fail-closed (it under-reports READY, never falsely authorizes a DROP), and the fail-closed handling elsewhere is thorough (unreachable cluster → exit 2; missing pod/exec/count/log/baseline → counted failure; empty processlist treated as non-evidence). Two low notes are inline (a bare USER= env can match the *USER scan; Criterion 1 only catches edgeuser in *USER-named vars with exact value, so a DSN/URL cred would evade) — optional. Fix the per-workload expectations and the fixture and I'll re-review; the rest of the tool is well-built.

[LukasWodka]

@saqlainsyed007 — both findings confirmed against the chart and fixed in 35a2228. Your framing was exactly right: green CI here was a symptom.

HIGH — verified before fixing. Measured the two templates at this head:

requests-proxy : DB_BOOTSTRAP_USER 0 | SERVICE_DB_ACCOUNTS 1 | PER_EXPERIMENT_DB_CREDS 0
jobs-manager : DB_BOOTSTRAP_USER 2 | SERVICE_DB_ACCOUNTS 1 | PER_EXPERIMENT_DB_CREDS 1

So requests-proxy tripped two bad findings on every correctly-retired fleet, in every phase, forever — and printed "the mint falls back to edgeuser" about a pod that never mints. check_workload now takes a role (mint/consumer); (a) and (b) stay workload-agnostic, since "nothing resolves to edgeuser" and "every identity can authenticate" are true of every pod.

MEDIUM — and this is the part worth dwelling on. The fixture wrote the identical $common to both pods, so the happy path was differenced against a fleet shape that cannot exist. The requests-proxy fixture now mirrors the chart.

Fixing that surfaced something sharper than a stale fixture: the test case that should have caught this was the case demanding it."PER_EXPERIMENT_DB_CREDS unset is a finding on requests-proxy too" deleted the var from requests-proxy and required a finding — encoding the always-red bug as a requirement. It is replaced by the two assertions that are actually true:

• PER_EXPERIMENT_DB_CREDS unset on jobs-manager is a finding — the workload that carries it.
• requests-proxy without the mint gates is NOT a finding — the regression guard, because case 1 alone cannot say why it is ready.

Mutation proof, which is the real answer to "why was CI green". Restoring the unconditional check:

[FAIL] a clean retired fleet is DROP-READY -- exit 1, wanted 0
[FAIL] requests-proxy without the mint gates is NOT a finding -- exit 1, wanted 0
[FAIL] a GROWN count is not a shrink
[FAIL] post-drop passes when edgeuser is absent
30 passed, 4 failed

Four cases, including the happy path. Under the old fixture that same mutation was green — it was the shipped code. Anchor asserted to match exactly once before applying, so the mutation is not inert.

Suite: 34 passed, 0 failed (was 33). shellcheck -S warning -x clean on both files. Also merged develop — the DRIFT_GUARDS collision was additive, resolved as a verified union (29+29 → 30, asserting nothing dropped and no duplicates).

The two low notes are untouched, as you marked them optional. Ready for re-review.

[LukasWodka]

@saqlainsyed007 — both findings are fixed, and you were right that the green CI was the symptom rather than the clearance.

HIGH — fixed. Confirmed against the chart before touching anything: requests-proxy-deployment.yaml renders SERVICE_DB_ACCOUNTS and TB_META_* and neitherDB_BOOTSTRAP_USER nor PER_EXPERIMENT_DB_CREDS. So the tool produced two findings on a correctly-retired fleet in every phase, forever, with a factually wrong reason for a pod that never mints.

Worth admitting how I missed it: I had the evidence. My own check was grep -l over an alternation of the three names, which reports that a file matched something — and I read it as both templates rendering all three. A per-file count would have said otherwise in the same breath.

MEDIUM — fixed. The fixture now mirrors each template, so the happy-path case is no longer differenced against a fleet shape that cannot exist.

Both low notes taken, not declined:

• bare USER — ^[A-Z0-9_]*USER= matched the plain Unix USER, so USER=nobody was an "identity" with no PASSWORD counterpart: a false finding on a healthy pod. Now ^[A-Z0-9][A-Z0-9_]*_USER=. (My first attempt, [A-Z0-9]+_USER, excluded underscores in the prefix and quietly matched nothing — the harness caught it.)
• DSN/URL credential — values are now scanned for the edgeuser substring and reported by name with the value withheld, since a matched value can carry a password. A test asserts the secret never reaches the output.

One addition beyond your list, and it's the part I'd most like your eye on. A co-driver session landed the same HIGH fix while I was working it, using a mint / consumer role parameter at the call site. That's correct today — but it is a written-down copy of which workload renders which gate, and a copy is exactly what caused this bug: it agrees with itself while the chart moves underneath it. If a gate is added to requests-proxy, or the mint moves to a third workload, the label keeps asserting yesterday's shape silently.

So the role is now derived from the template as well as declared, and a disagreement is a finding. The explicit label stays because it's readable at the point of use; the derivation is what keeps it honest. It fails closed — an unreadable template is "cannot tell", never "the label is fine" — and a test runs the tool from a directory without the chart to prove that.

Two of the new cases exist because a mutation came back GREEN first: nothing covered a DB_BOOTSTRAP_USER that was neither root nor empty (so the *) arm could have become an ok silently), and nothing held the fixture to the chart once it was fixed. The second replaces a claim I'd written in a comment with an actual check — the same defect one level up.

Verified: verdict harness 39 passed / 0 failed, drift 30/30, helm unittest600/600, shellcheck + bash -n clean. Mutation-proven with every anchor asserted applied — restoring the all-gates-on-every-workload behaviour reddens the suite, as does killing the cross-check, the fail-closed path, the regex fix, or the disagreement report.

Ready for re-review.

[aptracebloc]

This is the right shape for the gate — deriving "nothing resolves to edgeuser" from each pod's own resolved env instead of a hand-written list, demoting the structurally-unfailable processlist query to a labelled non-evidence smoke test, and failing closed on absent-pod / refused-exec / unreadable-count / missing-driven-cycle. The test suite is strong: I reproduced the always-PASS mutation and 14 cases redden, the assertions key off specific finding substrings, and the fleet stubs are synthetic (derived, not restating the tool). SECURITY.md §7.7/§8.10 is accurate against the code.

Requesting changes on one reproduced false-PASS on the most safety-critical criterion — because a gate that authorizes an irreversible DROP USER edgeuser must never render "couldn't tell" as ready.

🔴 Blocker 1 — ingestion DB_USER absent renders [ok], not fail-closed (edgeuser-drop-readiness.sh:213).

ivars=$(K exec"$ing" -- env … | grep -E '^DB_USER=|^DB_PASSWORD='|| true)if [ -z"$ivars" ];then untold …
elif grep -q '^DB_USER=edgeuser$'<<<"$ivars";then bad …
else ok "ingestion ($ing): DB_USER=$(…)"# <-- reached when DB_PASSWORD is present but DB_USER is ABSENTfi

If DB_PASSWORD is set but DB_USER is entirely absent from env, ivars is non-empty (so the -z guard passes), the edgeuser grep doesn't match, and control falls to else → ok "…DB_USER=" with an empty value, no FAILURES++ → the tool prints DROP-READY / exit 0. I reproduced it. This is exactly the "the interesting answer is a variable that isn't there" failure the tool exists to kill, and it's not hypothetical: prod is digest-pinned to the 0.7 ingestor whose edgeuser DB_USER default is applied inside the container and never appears in env (see _helpers.tpl ~L611/716-720, backend#1752) — so this pod would authenticate as edgeuser while the gate greenlights the DROP. Note the asymmetry: check_workload's criterion-1(a) already treats "no *_USER at all" as untold (fail-closed) — the ingestion check must do the same. Fix: guard on the DB_USER line specifically (e.g. require a non-empty ^DB_USER= match to reach ok; DB_PASSWORD-present-but-DB_USER-absent → untold/bad).

🔴 Blocker 2 — no test covers the DB_USER-absent path (edgeuser-drop-readiness-verdicts.sh:208). The ingestion cases only exercise DB_USER=edgeuser and no-ingestion-pod, so Blocker 1 ships green. Add a case: clean fleet, ingest pod env = DB_PASSWORD=x only, expect non-zero + the specific "could not read DB_USER" finding. (This is what would have caught #1 — and it's the whole point of a mutation-proof gate here.)

🟡 Blocker 3 — empty-but-successful log scored clean (edgeuser-drop-readiness.sh:241). In scan_logs, a kubectl logs that succeeds with empty output sets any=1 and scores both checks [ok]. So "cycle aged out of --since 2h", "pod restarted and its buffer reset", or a stale pod all read as "criterion 2 clean" with no evidence the driven cycle was actually observed — "could not look" rendered as "looked and clean", against the tool's own doctrine. The any=0 guard doesn't catch the per-pod empty case. Criterion 2 is corroboration so this is softer than #1, but it also means SECURITY.md §7.7 slightly overstates the log-scan fail-closed guarantee (only a failedkubectl logs is a finding today).

Minor / non-blocking (worth handling while in here): the ingestion pod selector at :203 lacks the $2=="Running" filter pod_of() uses; criterion 1(a) at :155 only matches *_USER=edgeuser exactly, so an edgeuser inside a DATABASE_URL= DSN would go undetected (not in today's chart, but a gap against the "nothing resolves to edgeuser" guarantee); secret_val re-fetches all Secrets 5× (:300); and --baseline-identity is mandatory but inert (compare() always reads the live count) — a one-line "assertion-of-intent, not load-bearing" comment would prevent a maintainer misreading it.

— drafted with Claude Code

[LukasWodka]

@saqlainsyed007 — all three fixed in bbb3e7a. Blocker 1 was the real thing; thank you for reproducing it rather than describing it.

🔴 1 — the DB_USER false-PASS. Confirmed: $ivars matched DB_USERorDB_PASSWORD, so a pod with a password and no DB_USER cleared the -z guard, missed the edgeuser grep, and fell to ok with an empty value — DROP-READY, exit 0. The guard is now on DB_USER specifically, and absent-or-empty is untold.

Your point about the asymmetry is the one I'd have missed: check_workload's criterion 1(a) already treats "no *_USER at all" as fail-closed. Same question, same doctrine, two different answers — that gap was the defect, not an oversight beside it. The comment now records the 0.7-ingestor detail so the next reader knows this is the shape that actually occurs rather than a corner.

🔴 2 — the missing case. Added, and it reproduces exactly what you described: under the mutation the tool exits 0.

[FAIL] ingestion with a PASSWORD but no DB_USER is cannot-tell, NOT a pass
-- exit 0, wanted 1; missing expected finding: DB_USER is absent or empty

🟡 3 — the empty log. Fixed, and it forced a fixture correction worth flagging: clean_fleet was writing EMPTY log files as its happy path. Once an empty log correctly became untold, five green cases went red — because the clean fleet was being differenced against a state that cannot occur, which is the same shape as the requests-proxy fixture from your last review. clean_fleet now writes benign output, modelling a cycle that was actually driven.

Verification. 41 passed / 0 failed (was 34). Two mutations, each reddening only its own case, anchors asserted to match exactly once:

restore the DB_USER false-PASS -> ingestion with a PASSWORD but no DB_USER ... (exit 0, wanted 1)
score an empty log as clean -> an EMPTY log is cannot-tell, not a clean cycle (exit 0, wanted 1)

shellcheck -S warning -x clean on both files.

The four minors are untouched as you marked them non-blocking — the Running-filter on the ingest selector, the DSN-embedded edgeuser gap, the 5× Secret re-fetch, and the inert --baseline-identity. The DSN one interests me most since it is a genuine hole in the "nothing resolves to edgeuser" guarantee rather than a tidy-up; say the word and I'll take it here or file it.

[LukasWodka]

@aptracebloc — all four minors done. Your three blockers were already closed on this branch by bbb3e7a before I got here; this is the remainder, and your first minor turned out to be more than a nit.

The ingest selector. You flagged the missing $2=="Running" filter. The reason it matters is sharper than the nit: kubectl exec needs a running container, so an ingestion Job pod that has already Succeeded cannot be exec'd at all — picking one yields an empty env and an unexplained "cannot tell". With the filter in place the message can name the real precondition instead:

no RUNNING ingestion pod. The gate needs a driven cycle, and the ingestion identity can only be read from a live pod — kubectl exec requires a running container, so a Job pod that already Succeeded cannot be inspected. Run this check WHILE an ingestion run is in flight.

The old advice — "drive one, then re-run" — could not be followed, because a completed run leaves no readable env. That's a runbook correction, not a filter.

The DSN gap — fixed where the env is read, not where it's checked. This is the part I'd point at: the env was being filtered to DB_USER|DB_PASSWORDat read time, which is what made the gap structural rather than incidental. No amount of extra checking downstream could see a DATASET_DSN that was discarded before the checks ran. The full env is now kept and scanned for the substring, reported by name with the value withheld — and it runs regardless of the DB_USER verdict, because a pod can hold a correct DB_USERand a stale DSN. There's a test for exactly that combination, and another asserting the secret never reaches the output.

secret_val now fetches once instead of per key.

--baseline-identity — you were right that it's inert, and right that it needed saying. compare() always reads the live count. There's now a comment stating it is an assertion of intent and not a switch, specifically so the next maintainer doesn't "fix" it by wiring it into the query. It exists to force the operator to state which identity produced the baseline, since a root count and a tb_ingest count are different measurements.

And your Blocker 3 had a documentation tail I closed too. You noted §7.7 "slightly overstates the log-scan fail-closed guarantee". After bbb3e7a the claim is true but imprecise — it said "an unreadable log", which doesn't cover the empty-but-successful read that was the actual bug. §7.7 now enumerates every fail-closed condition the code implements, including the empty --since window, no RUNNING ingestion pod, and a DB_USER that is absent rather than wrong.

Verified: 45 passed / 0 failed (5 new cases), drift 30/30, helm unittest 600/600, shellcheck clean. Mutation-proven with every anchor asserted applied — dropping the Running filter, disabling the DSN scan, re-filtering the env before the scan, and removing the DB_USER-absent guard each redden.

Ready for re-review.

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit dd21c08. Configure here.}

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 467a0a2. Configure here.}

[aptracebloc]

Re-reviewed at 467a0a2. Both blocking findings from my earlier pass are addressed, and I verified each by mutation rather than by eye — this gate authorizes an irreversible DROP, so "the guard is there" is not enough on its own.

1. Ingestion DB_USER absent now fails closed. With DB_PASSWORD present and DB_USER absent, ivars is non-empty so the earlier branches don't fire, and the new elif [ -z "$iuser" ] renders untold (cannot-tell → failure), not [ok]. Neutralising that guard flips the "PASSWORD but no DB_USER" case to exit 0 / DROP-READY and reddens the test — so the case is real, not vacuous.

2. Empty-but-successful kubectl logs is untold. scan_logs treats an empty log as cannot-tell and continues before any=1; neutralising it reddens the empty-log case.

The four minors (Running-only ingest selector, DSN/URL edgeuser scanned by name with the value withheld, secrets fetched once, --baseline-identity mandatory-but-inert) are all still handled, each with a test. The four commits since my last pass (COUNT-not-GROUP_CONCAT truncation fix, stale-Job skip, pod_of SIGPIPE, control-plane name exclusion) each ship a green case plus a control, and I inverted the post-drop arm to confirm "edgeuser still present" cannot read as clean.

Ran the verdict suite locally against stubbed kubectl: 51 passed, 0 failed. shellcheck -S warning -x clean. CI and Bugbot green at head, no open threads.

Approving. Nice work threading the fail-closed doctrine through every new population the checks now meet.

— drafted with Claude Code

[LukasWodka]

Both blocking findings are fixed at 467a0a2e, and your two low notes with them. I verified each by mutation rather than by eye — restoring the exact defect you described and watching the suite redden with your own reason in the message.

HIGH — per-workload expectations.check_workload now takes a role (mint / consumer), and only the mint asserts DB_BOOTSTRAP_USER / PER_EXPERIMENT_DB_CREDS. But a hand-written mint at the call site is a second copy of which workload renders which gate, and a copy is what produced this bug — so the role is also derived from the template, and a disagreement is itself a finding (assert_role_matches_chart). Mutation, requests-proxy declared mint again:

[FAIL] a clean retired fleet is DROP-READY -- exit 1, wanted 0
[FAIL] requests-proxy: declared role 'mint' disagrees with the chart, which says 'consumer'
[FAIL] requests-proxy (…): DB_BOOTSTRAP_USER is unset — the mint falls back to edgeuser
[FAIL] requests-proxy (…): PER_EXPERIMENT_DB_CREDS is unset — default-off means …

That is your "structurally always-red on a real fleet", reproduced on demand.

MEDIUM — the vacuous fixture.clean_fleet now writes requests-proxy what the chart actually renders (SERVICE_DB_ACCOUNTS + TB_META_USER, neither mint gate); jobs-manager keeps both. And rather than leave that as a fact someone has to keep true by hand, there's a case asserting it — the fixture mirrors each workload template exactly. Mutation, both pods given the identical $common again:

[FAIL] fixture for requests-proxy does not mirror its template.

Separately: clean_fleet's logs were EMPTY, which under the corrected criterion-2 rule means "we could not look", not "we looked and it was clean" — the same shape of bug as the fixture, so the happy path now writes benign content.

Both low notes are covered too, not deferred — a bare USER= env is not treated as a DB identity, and for DSN/URL creds: edgeuser in an ingestion DSN is caught, by NAME, the ingestion DSN value is withheld from the output, a good DB_USER does not excuse a stale DSN -- both are reported.

Suite at 467a0a2e: 51 passed, 0 failed (34 when you reviewed), shellcheck -S warning -x clean. Taking your point that green CI was a symptom rather than a clearance — the numbers above only mean something because the fixture now differences against the chart instead of against itself.

Your CHANGES_REQUESTED still stands and blocks the merge; over to you.

[saqlainsyed007]

Re-reviewed at 467a0a2e. Both my findings are genuinely fixed in code, not just resolved.

Finding 1 (structurally always-red) — fixed.check_workload now takes a mint|consumer role ($3); the DB_BOOTSTRAP_USER=root and PER_EXPERIMENT_DB_CREDS assertions run only for mint, and requests-proxy is called as consumer. That matches the chart at this head (requests-proxy renders only SERVICE_DB_ACCOUNTS + TB_META_USER; jobs-manager carries both mint gates), so a correctly-retired requests-proxy is no longer flagged NOT-READY. A real leak is still caught — the *_USER=edgeuser and DSN-value scans run on every workload — and assert_role_matches_chart derives the role from the template and makes a label/chart mismatch its own finding, failing closed on an unreadable template.

Finding 2 (vacuous happy-path) — fixed and mutation-proof.clean_fleet's requests-proxy fixture now mirrors the chart (consumer env, no mint gates). Confirmed reverting the code fix reddens case 1 and the new gate2b regression case, and reverting the fixture trips the new "fixture mirrors template" assertion. Suite 51/51.

Nice work on the extra Bugbot fixes too (SIGPIPE abort, control-plane exclusion, empty-log = cannot-tell, group_concat truncation → COUNT). No new defects, all 12 threads resolved, CI green, no conflicts. Approving.