Uh oh!
There was an error while loading. Please reload this page.
sec(telemetry): release-scope the Collector token Secret, migration-safe (backend#2625) - #865
Conversation
…afe (backend#2625) telemetryCollector.tokenSecret.name defaulted to a fixed `tracebloc-telemetry-token` in the SHARED node-agents namespace. Two edges on one cluster collide: two jobs-managers write one Secret, last writer wins, and the loser's Collector authenticates as the wrong tenant — cross-tenant telemetry misattribution. Latent today only because a single Collector is enabled fleet- wide; the collision arrives the moment #1906's prod half enables a second. Option A (migration-safe rename): - Resolve the name in one helper, tracebloc.telemetryTokenSecretName, behind all four consumers (writer env, reader guard + volume, RBAC resourceName), so they can never disagree. Default is release-scoped `<release>-telemetry-token`. - The legacy fixed name is a MIGRATE-ME sentinel, not an override: `helm upgrade --reuse-values` bakes the old default into existing releases' stored values, so only rewriting the sentinel actually migrates them. A genuinely custom name is honoured verbatim. - jobs-manager writes the release-scoped name; the daemonset pre-flight ALSO accepts the legacy name while it exists, so an edge already collecting under the legacy name is not wedged on upgrade (the #2400 deadlock in a new costume). The legacy acceptance and its helper carry, at their declaration, the condition for their own removal. Acceptance: (a) two releases resolve to distinct, release-scoped Secrets — asserted across two renders in telemetry_collector_test.yaml and telemetry-token-agreement.sh; (b) an edge on the legacy name upgrades without the pre-flight tripping — the guard is a lookup-backed `fail`, invisible to `helm template`, so it is exercised live in scripts/tests/telemetry-token-migration.sh (run from the k3d auto-upgrade e2e; self-skips with no cluster); (c) the legacy acceptance declares its own removal condition at the declaration. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Chart content changed (telemetry token rename), so the chart-version-guard requires a new Chart.yaml version — a Helm repo only publishes on version change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
… guard check The pipefail early-close house-rule flags `printf | grep -q` under errexit+pipefail (the reader closes early and SIGPIPEs the producer). Feed the captured output via a here-string instead — same match, no pipe. (backend#2625) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7a65049. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
…able in e2e (backend#2625) Cursor Bugbot (2 findings) on PR #865: - Unbounded `kubectl cluster-info` hangs the k3d job on a wedged API instead of failing fast. Every kubectl call now goes through a `--request-timeout=15s` wrapper. - A skip exited 0, so from e2e-auto-upgrade.sh a skip counted as acceptance (b) passing while the PASS line claimed it was verified. New `--require` flag turns every skip condition into a hard failure; the e2e passes it (a cluster is guaranteed there). Standalone/drift runs still self-skip. Verified live on k3d: three cases green under --require; teardown clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LukasWodka
left a comment
There was a problem hiding this comment.
Approving d8c3a43e — 44 passing, 4 path-skipped, nothing pending or failing, no open threads, mergeable=MERGEABLE.
I checked the three load-bearing claims against the chart rather than the description, because on a cross-tenant issue the description being right isn't the same as the templates being right.
"One resolver behind all four consumers" is literally true. All four include "tracebloc.telemetryTokenSecretName":
jobs-manager-deployment.yaml writer env
telemetry-token-rbac.yaml RBAC resourceName
telemetry-collector-daemonset.yaml reader guard + volume
and every remaining mention of tracebloc-telemetry-token in the chart is either prose or the sentinel's own definition. The name exists in exactly one place, so the four cannot drift — which is the property, not a comment about the property.
The sentinel reasoning is the part I'd have got wrong. Defaulting only an absent name looks correct and would have stranded every installed edge forever, because --reuse-values bakes the old default into stored values — so the legacy name is present, not absent, on exactly the releases that need migrating. Rewriting it is what actually moves them. And the comparison uses include "tracebloc.telemetryTokenLegacyName" rather than the literal, so even the sentinel check derives from one definition. A different explicit name is honoured verbatim; the trade-off (an operator who deliberately chose the old default gets renamed) is the right way round and you said so out loud.
The removal condition is the best part of this PR. A temporary compatibility shim usually becomes permanent because nobody writes down when it can go. This one names both things that must be deleted together, states what premature removal re-breaks (the #2400 deadlock, in its new costume), and gives an observable exit criterion:
kubectl get secret -A --field-selector metadata.name=tracebloc-telemetry-token
Empty output means the shim can go. That's a machine-checkable condition rather than a promise to remember.
And the test proves the security property, not the code shape. I ran it:
ok: writer, reader, key and RBAC all name the same Secret
release tenant-alpha -> tenant-alpha-telemetry-token
release tenant-beta -> tenant-beta-telemetry-token
ok: two releases resolve to distinct, release-scoped Secrets
telemetry token agreement: green
It pulls the resolved names out of two real helm template renders — the writer's own env and the reader's own volume — rather than grepping the templates for the helper. That's the difference between asserting the four agree and asserting the four render in agreement, and it's why the guard will still mean something after someone refactors the helper. set -euo pipefail plus the fail-closed-on-unlocatable block means a field it can't find is a failure, not a quiet pass.
Value: removes a cross-tenant telemetry misattribution that is latent only because a single Collector is enabled fleet-wide — and would have become real the moment #1906's prod half enabled a second. It does it without wedging the one edge already collecting, which is the trap the obvious fix falls into.
Uh oh!
There was an error while loading. Please reload this page.

Fixes tracebloc/backend#2625.
The bug
telemetryCollector.tokenSecret.namedefaulted to a fixedtracebloc-telemetry-tokenin the shared node-agents namespace. Two edges on one cluster collide: two jobs-managers write one Secret, last writer wins, and the loser's Collector authenticates as the wrong tenant — cross-tenant telemetry misattribution. Latent today only because a single Collector is enabled fleet-wide; the collision arrives the moment #1906's prod half enables a second.The naive fix (template the default to
{{ .Release.Name }}-telemetry-token) wedges the working edge: the daemonset's own pre-flight (fail: … Secret %q does not exist) refuses the upgrade on the one edge already collecting, because jobs-manager hasn't written the new name yet — the #2400 deadlock in a new costume.The fix — Option A (migration-safe rename)
tracebloc.telemetryTokenSecretName— behind all four consumers (writer envjobs-manager-deployment.yaml, reader guard + volumetelemetry-collector-daemonset.yaml, RBAC resourceNametelemetry-token-rbac.yaml), so they can never disagree. Default is release-scoped<release>-telemetry-token.helm upgrade --reuse-valuesbakes the old default into existing releases' stored values, so defaulting only an absent name would strand every installed edge on the colliding fixed name forever. A genuinely custom name is honoured verbatim.Acceptance
telemetry_collector_test.yaml(per-releaserelease.name) and inscripts/tests/telemetry-token-agreement.sh(renders two releases, asserts the resolved names differ and each scopes to its release).lookup-backedfail, invisible tohelm template(the suite says as much), so it's exercised live inscripts/tests/telemetry-token-migration.sh, invoked from the k3d auto-upgrade e2e (e2e-auto-upgrade.sh) and self-skipping with no cluster. Three cases: no Secret → refused; legacy only → accepted; release-scoped only → accepted.Verification (run locally against a throwaway k3d cluster)
helm unittest ./client— telemetry suite green, 56 tests (4 unrelated suites fail identically on cleandevelop— a local helm-4 schema quirk on imagetagnotconstraints; CI pins helm 3.15.4).telemetry-token-agreement.sh/telemetry-token-bootstrap.shand the other 8 collector derived checks — green.telemetry-token-migration.shagainst live k3d — all three cases pass; verified the guard accepts the legacy name (acceptance b) and still refuses when nothing exists.helm lint,check-style.sh,shellcheck— clean.Ran
/code-reviewon the diff: one robustness finding fixed (name extraction now reads from the ungated RBAC template so it can't itself trip the guard), two low-severity non-issues.🤖 Generated with Claude Code
Note
Medium Risk
Changes credential Secret naming and RBAC scoping for telemetry ingest—security-sensitive—but migration paths and live e2e reduce wedge risk; main residual is fleet timing until release-scoped Secrets are populated on every collecting edge.
Overview
Fixes cross-tenant telemetry misattribution when multiple Helm releases share the node-agents namespace: the ingest token Secret no longer defaults to a single fixed
tracebloc-telemetry-tokenname that every jobs-manager could overwrite.Central change:
tracebloc.telemetryTokenSecretNameresolves the effective name for jobs-manager (writer env), the Collector DaemonSet (volume + pre-flight), and token RBAC (get/patchon one Secret). Default is<release>-telemetry-token. EmptytokenSecret.nameor the legacy sentineltracebloc-telemetry-token(including values replayed byhelm upgrade --reuse-values) is rewritten to that release-scoped name; genuinely custom names stay verbatim.Upgrade safety: The DaemonSet pre-flight still refuses installs with no token Secret, but now accepts either the release-scoped name or the legacy fixed name so an edge already collecting under the old Secret is not wedged before jobs-manager writes the new one. Chart 1.9.75.
Tests: Helm unittest cases for per-release names, legacy migration, and custom overrides;
telemetry-token-agreement.shasserts two releases resolve to distinct Secrets; newtelemetry-token-migration.shexercises the lookup-based guard on a live cluster (wired intoe2e-auto-upgrade.shwith--requireand helm-ci path triggers).Reviewed by Cursor Bugbot for commit d8c3a43. Bugbot is set up for automated code reviews on this repo. Configure here.