Skip to content

docs(migration): state the chart floor the root-rotation runbook needs - #887

Merged
LukasWodka merged 2 commits into
developfrom
docs/2591-prod-chart-upgrade-preconditions
Aug 27, 2026
Merged

docs(migration): state the chart floor the root-rotation runbook needs#887
LukasWodka merged 2 commits into
developfrom
docs/2591-prod-chart-upgrade-preconditions

Conversation

@LukasWodka

@LukasWodkaLukasWodka commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Found while preparing the chart upgrade for a fleet still on client-1.9.63 (backend#2591). Docs only — no chart content.

The defect.docs/migration-tools/rotate-mysql-root.md precondition 1 verifies the rotation gate by reading MYSQL_ROOT_PASSWORD out of the release Secret, and tells the operator to "enable the gate first" if that read returns zero. On a chart below 1.9.71 that instruction cannot succeed, and nothing in the loop says why.

rotateMysqlRoot landed in client-1.9.71 (client#822). values.schema.json does not close additionalProperties, so on an older chart helm upgrade ... --set rotateMysqlRoot=true is accepted and exits 0 while rendering no MYSQL_ROOT_PASSWORD at all. Measured on client-1.9.63:

$ helm template rel ./client-1.9.63 -f prod-values.yaml --set rotateMysqlRoot=true
exit=0
MYSQL_ROOT_PASSWORD occurrences: 0
$ python3 -c "import json;s=json.load(open('values.schema.json'));print(s.get('additionalProperties','<absent>'), 'rotateMysqlRoot' in s['properties'])"
<absent> False

So the operator reads zero, follows the remedy, reads zero again, and is pointed back at the step they just ran. The real cause — the chart version — is named nowhere.

What this changes

  1. docs/migration-tools/rotate-mysql-root.md — the chart-version check becomes the first thing in precondition 1, with the reason it fails silently, plus a pointer to upgrade the chart as its own change in its own window.
  2. client/MIGRATION.md — a section for 1.9.71, since there was none, covering what crossing that version from below actually touches.

Verification

Both MIGRATION.md claims are measured, not restated:

  • The gate is a no-op on upgrade.rotateMysqlRootByEnv is dev: false, stg: false, prod: false at 1.9.71.
  • One new object pair, outside the release namespace. In 1.9.63 all of templates/telemetry-token-rbac.yaml sat behind the Collector's enabled flag; in 1.9.71 the file carries no if at all. Rendering both versions offline with CLIENT_ENV: prod and otherwise-default values, the object set gains exactly the two objects from that file and loses nothing — in nodeAgents.namespace.name (default tracebloc-node-agents), which is not the release namespace. Consistent with the cluster-scope requirement already in docs/INSTALL.md.
  • What the upgrade does not do.serviceDbAccountsByEnv is still prod: false, and requests-proxy renders byte-for-byte identically across the two versions apart from the chart-version labels.

Test plan

make check green on this branch (parse, shellcheck, drift, helm-lint, helm-vocab — 50/50 chart-env-vocabulary checks). client/MIGRATION.md is classified docs-only by scripts/tests/chart-version-guard.bats ("docs-only change is N/A"), so no Chart.yaml bump is required and none is made.

Refs backend#2591 — partial, does not close it: the ticket's remaining work is the operational upgrade itself.


Note

Low Risk
Documentation-only changes with no chart, template, or runtime behavior modifications.

Overview
Docs-only fix for operators stuck on charts below client-1.9.71 who follow the MySQL root rotation runbook: --set rotateMysqlRoot=true can succeed with no MYSQL_ROOT_PASSWORD in the release Secret, so the old precondition loop never names the real blocker.

docs/migration-tools/rotate-mysql-root.md now leads precondition 1 with a Helm chart version check (>= 1.9.71), explains that silent no-op behavior, and tells operators to upgrade the chart in its own window (not bundled with rotation), with a link to the new migration section. The Secret gate check and “enable the gate” guidance only apply after the version check passes.

client/MIGRATION.md adds Upgrading to 1.9.71, documenting what crossing that line from below actually does: the new rotateMysqlRoot gate (defaults off), unconditional Collector token Role/RoleBinding in tracebloc-node-agents (cluster-scope upgrade identity), what the upgrade does not change (serviceDbAccountsByEnv / requests-proxy), and a table of new values keys with first-shipped chart versions.

Reviewed by Cursor Bugbot for commit 76b4b0f. Bugbot is set up for automated code reviews on this repo. Configure here.

…s (backend#2591)
The rotate-mysql-root runbook's precondition 1 verifies the gate by reading
MYSQL_ROOT_PASSWORD out of the release Secret, and tells the operator to
"enable the gate first" when that read comes back zero. On a chart below
1.9.71 that instruction cannot succeed and does not say so.
`rotateMysqlRoot` landed in client-1.9.71 (client#822). values.schema.json
does not close additionalProperties, so on an older chart
`--set rotateMysqlRoot=true` is accepted and exits 0 while rendering no
MYSQL_ROOT_PASSWORD. Measured on client-1.9.63 via `helm template
--set rotateMysqlRoot=true`: exit 0, zero occurrences of the key. The
operator then re-reads zero and is pointed back at the step they just ran,
with nothing in the loop naming the chart version as the cause.
Add the version check as the first thing in that precondition, and a
MIGRATION.md section for 1.9.71 covering what crossing it actually touches:
the gate itself (false for dev/stg/prod, so a no-op on upgrade) and the
Collector token Role/RoleBinding, which lost their `enabled` gate and now
render unconditionally in the node-agents namespace — a namespace that is
not the release namespace by default, so the upgrading identity needs reach
there. Measured by rendering 1.9.63 and 1.9.71 offline with CLIENT_ENV=prod:
the object set gains exactly those two objects and loses nothing.
Also records what the upgrade does NOT change: serviceDbAccountsByEnv still
resolves false for prod, and requests-proxy renders byte-identically across
the two versions apart from chart labels.
Docs only — no chart content, so the version-bump guard is N/A.
Refs backend#2591
@LukasWodkaLukasWodka self-assigned this Aug 27, 2026
@LukasWodkaLukasWodka changed the title docs(migration): state the chart floor the root-rotation runbook needs (backend#2591)docs(migration): state the chart floor the root-rotation runbook needsAug 27, 2026
Comment threadclient/MIGRATION.md Outdated

@saqlainsyed007saqlainsyed007 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 2612a77c. The docs are accurate — I checked every version claim against the chart history and they hold.

The floor of 1.9.71 is right: rotateMysqlRoot/rotateMysqlRootByEnv/mysqlRootPassword first appear in #822 (09bb86c), which landed while Chart.yaml was at 1.9.71, and on older charts values.schema.json doesn't close additionalProperties, so --set rotateMysqlRoot=true is silently accepted and renders nothing — exactly the "enable the gate" trap the runbook now heads off. The env matrices (rotateMysqlRootByEnv all false; serviceDbAccountsByEnv dev/stg true, prod false), the telemetry-token-rbac ungating at 1.9.71, and the prodDigest/INSTALL.md cross-refs all check out, and the two files are internally consistent on 1.9.71.

One non-blocking accuracy nit (inline): MIGRATION.md:43 lists six "new values keys" under Upgrading to 1.9.71, but three (bootstrapDbPassword, bootstrapDbReparent, bootstrapDbReparentByEnv) were actually introduced in 1.9.67 (#785). Correct for the 1.9.63-fleet jump the section targets, but an operator already on 1.9.67–1.9.70 has those keys — worth splitting them out or noting the list spans 1.9.67–1.9.71.

Not approving this pass: CI isn't green yet — set-status / closing-ref is queued/pending, and the body is a deliberate "Refs backend#2591 — partial, does not close it" (no Closes), so that gate may need the same title/exemption handling we've hit before. Sort the nit (optional) and get CI green, and I'll approve on the next pass.

…ed at (backend#2591)
@saqlainsyed007, verified independently against the chart rather than taken on
trust: `bootstrapDbPassword`, `bootstrapDbReparent` and
`bootstrapDbReparentByEnv` first appear in e03edbb (#785) at Chart 1.9.67,
while `mysqlRootPassword`, `rotateMysqlRoot` and `rotateMysqlRootByEnv` first
appear in 09bb86c (#822) at 1.9.71.
All six are genuinely new to the 1.9.63 fleet this section targets, so the list
was correct for that jump and wrong for anyone crossing 1.9.67 to 1.9.70 -- who
would find the bootstrapDb* keys already present and reasonably wonder what
else the note had got wrong.
Split by introducing version, with the method written down so the next person
derives it instead of recalling it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@saqlainsyed007saqlainsyed007 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 76b4b0ff. My one note is fully addressed and CI is green.

The attribution nit is fixed: the flat six-key list is now a two-row table split by introducing version — bootstrapDbPassword/bootstrapDbReparent/bootstrapDbReparentByEnv attributed to 1.9.67 (#785) and mysqlRootPassword/rotateMysqlRoot/rotateMysqlRootByEnv to 1.9.71 (#822) — with an annotation explaining all six still belong in a 1.9.71 section (they target the 1.9.63 fleet). I re-verified each key's provenance against the chart history and it matches, and the 1.9.71 floor holds. No new inaccuracies.

(The body is deliberately "Refs backend#2591 — partial, does not close it" with no Closes, which is correct for partial work — and the closing-ref gate is green, so nothing mechanical is blocking.) LGTM.

@LukasWodka
LukasWodka merged commit a8aea0c into developAug 27, 2026
25 checks passed
@LukasWodka
LukasWodka deleted the docs/2591-prod-chart-upgrade-preconditions branch August 27, 2026 15:39
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@LukasWodka@saqlainsyed007