Uh oh!
There was an error while loading. Please reload this page.
docs(migration): state the chart floor the root-rotation runbook needs - #887
Conversation
…s (backend#2591) The rotate-mysql-root runbook's precondition 1 verifies the gate by reading MYSQL_ROOT_PASSWORD out of the release Secret, and tells the operator to "enable the gate first" when that read comes back zero. On a chart below 1.9.71 that instruction cannot succeed and does not say so. `rotateMysqlRoot` landed in client-1.9.71 (client#822). values.schema.json does not close additionalProperties, so on an older chart `--set rotateMysqlRoot=true` is accepted and exits 0 while rendering no MYSQL_ROOT_PASSWORD. Measured on client-1.9.63 via `helm template --set rotateMysqlRoot=true`: exit 0, zero occurrences of the key. The operator then re-reads zero and is pointed back at the step they just ran, with nothing in the loop naming the chart version as the cause. Add the version check as the first thing in that precondition, and a MIGRATION.md section for 1.9.71 covering what crossing it actually touches: the gate itself (false for dev/stg/prod, so a no-op on upgrade) and the Collector token Role/RoleBinding, which lost their `enabled` gate and now render unconditionally in the node-agents namespace — a namespace that is not the release namespace by default, so the upgrading identity needs reach there. Measured by rendering 1.9.63 and 1.9.71 offline with CLIENT_ENV=prod: the object set gains exactly those two objects and loses nothing. Also records what the upgrade does NOT change: serviceDbAccountsByEnv still resolves false for prod, and requests-proxy renders byte-identically across the two versions apart from chart labels. Docs only — no chart content, so the version-bump guard is N/A. Refs backend#2591
Uh oh!
There was an error while loading. Please reload this page.
saqlainsyed007
left a comment
There was a problem hiding this comment.
Reviewed at 2612a77c. The docs are accurate — I checked every version claim against the chart history and they hold.
The floor of 1.9.71 is right: rotateMysqlRoot/rotateMysqlRootByEnv/mysqlRootPassword first appear in #822 (09bb86c), which landed while Chart.yaml was at 1.9.71, and on older charts values.schema.json doesn't close additionalProperties, so --set rotateMysqlRoot=true is silently accepted and renders nothing — exactly the "enable the gate" trap the runbook now heads off. The env matrices (rotateMysqlRootByEnv all false; serviceDbAccountsByEnv dev/stg true, prod false), the telemetry-token-rbac ungating at 1.9.71, and the prodDigest/INSTALL.md cross-refs all check out, and the two files are internally consistent on 1.9.71.
One non-blocking accuracy nit (inline): MIGRATION.md:43 lists six "new values keys" under Upgrading to 1.9.71, but three (bootstrapDbPassword, bootstrapDbReparent, bootstrapDbReparentByEnv) were actually introduced in 1.9.67 (#785). Correct for the 1.9.63-fleet jump the section targets, but an operator already on 1.9.67–1.9.70 has those keys — worth splitting them out or noting the list spans 1.9.67–1.9.71.
Not approving this pass: CI isn't green yet — set-status / closing-ref is queued/pending, and the body is a deliberate "Refs backend#2591 — partial, does not close it" (no Closes), so that gate may need the same title/exemption handling we've hit before. Sort the nit (optional) and get CI green, and I'll approve on the next pass.
…ed at (backend#2591) @saqlainsyed007, verified independently against the chart rather than taken on trust: `bootstrapDbPassword`, `bootstrapDbReparent` and `bootstrapDbReparentByEnv` first appear in e03edbb (#785) at Chart 1.9.67, while `mysqlRootPassword`, `rotateMysqlRoot` and `rotateMysqlRootByEnv` first appear in 09bb86c (#822) at 1.9.71. All six are genuinely new to the 1.9.63 fleet this section targets, so the list was correct for that jump and wrong for anyone crossing 1.9.67 to 1.9.70 -- who would find the bootstrapDb* keys already present and reasonably wonder what else the note had got wrong. Split by introducing version, with the method written down so the next person derives it instead of recalling it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
saqlainsyed007
left a comment
There was a problem hiding this comment.
Approving at 76b4b0ff. My one note is fully addressed and CI is green.
The attribution nit is fixed: the flat six-key list is now a two-row table split by introducing version — bootstrapDbPassword/bootstrapDbReparent/bootstrapDbReparentByEnv attributed to 1.9.67 (#785) and mysqlRootPassword/rotateMysqlRoot/rotateMysqlRootByEnv to 1.9.71 (#822) — with an annotation explaining all six still belong in a 1.9.71 section (they target the 1.9.63 fleet). I re-verified each key's provenance against the chart history and it matches, and the 1.9.71 floor holds. No new inaccuracies.
(The body is deliberately "Refs backend#2591 — partial, does not close it" with no Closes, which is correct for partial work — and the closing-ref gate is green, so nothing mechanical is blocking.) LGTM.
Uh oh!
There was an error while loading. Please reload this page.
Summary
Found while preparing the chart upgrade for a fleet still on
client-1.9.63(backend#2591). Docs only — no chart content.The defect.
docs/migration-tools/rotate-mysql-root.mdprecondition 1 verifies the rotation gate by readingMYSQL_ROOT_PASSWORDout of the release Secret, and tells the operator to "enable the gate first" if that read returns zero. On a chart below1.9.71that instruction cannot succeed, and nothing in the loop says why.rotateMysqlRootlanded inclient-1.9.71(client#822).values.schema.jsondoes not closeadditionalProperties, so on an older charthelm upgrade ... --set rotateMysqlRoot=trueis accepted and exits 0 while rendering noMYSQL_ROOT_PASSWORDat all. Measured onclient-1.9.63:So the operator reads zero, follows the remedy, reads zero again, and is pointed back at the step they just ran. The real cause — the chart version — is named nowhere.
What this changes
docs/migration-tools/rotate-mysql-root.md— the chart-version check becomes the first thing in precondition 1, with the reason it fails silently, plus a pointer to upgrade the chart as its own change in its own window.client/MIGRATION.md— a section for1.9.71, since there was none, covering what crossing that version from below actually touches.Verification
Both
MIGRATION.mdclaims are measured, not restated:rotateMysqlRootByEnvisdev: false,stg: false,prod: falseat1.9.71.1.9.63all oftemplates/telemetry-token-rbac.yamlsat behind the Collector'senabledflag; in1.9.71the file carries noifat all. Rendering both versions offline withCLIENT_ENV: prodand otherwise-default values, the object set gains exactly the two objects from that file and loses nothing — innodeAgents.namespace.name(defaulttracebloc-node-agents), which is not the release namespace. Consistent with the cluster-scope requirement already indocs/INSTALL.md.serviceDbAccountsByEnvis stillprod: false, andrequests-proxyrenders byte-for-byte identically across the two versions apart from the chart-version labels.Test plan
make checkgreen on this branch (parse, shellcheck, drift, helm-lint, helm-vocab — 50/50 chart-env-vocabulary checks).client/MIGRATION.mdis classified docs-only byscripts/tests/chart-version-guard.bats("docs-only change is N/A"), so noChart.yamlbump is required and none is made.Refs backend#2591 — partial, does not close it: the ticket's remaining work is the operational upgrade itself.
Note
Low Risk
Documentation-only changes with no chart, template, or runtime behavior modifications.
Overview
Docs-only fix for operators stuck on charts below
client-1.9.71who follow the MySQL root rotation runbook:--set rotateMysqlRoot=truecan succeed with noMYSQL_ROOT_PASSWORDin the release Secret, so the old precondition loop never names the real blocker.docs/migration-tools/rotate-mysql-root.mdnow leads precondition 1 with a Helm chart version check (>= 1.9.71), explains that silent no-op behavior, and tells operators to upgrade the chart in its own window (not bundled with rotation), with a link to the new migration section. The Secret gate check and “enable the gate” guidance only apply after the version check passes.client/MIGRATION.mdadds Upgrading to 1.9.71, documenting what crossing that line from below actually does: the newrotateMysqlRootgate (defaults off), unconditional Collector token Role/RoleBinding intracebloc-node-agents(cluster-scope upgrade identity), what the upgrade does not change (serviceDbAccountsByEnv/ requests-proxy), and a table of new values keys with first-shipped chart versions.Reviewed by Cursor Bugbot for commit 76b4b0f. Bugbot is set up for automated code reviews on this repo. Configure here.