Skip to content

fix(installer): stop on a failed kubeconfig merge, and release the record before a recreate - #737

Merged
LukasWodka merged 3 commits into
developfrom
fix/732-kubeconfig-merge-and-delete-hints
Aug 17, 2026
Merged

fix(installer): stop on a failed kubeconfig merge, and release the record before a recreate#737
LukasWodka merged 3 commits into
developfrom
fix/732-kubeconfig-merge-and-delete-hints

Conversation

@LukasWodka

@LukasWodkaLukasWodka commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Closes#732 (main finding). Implements the client-side half of tracebloc/backend#2077.

Two ways the installer quietly pointed a secure environment at the wrong place.

1. The kubeconfig merge failed open (#732)

scripts/lib/cluster.sh ran the merge with output and exit code discarded:

k3d kubeconfig merge "$CLUSTER_NAME" --kubeconfig-merge-default --kubeconfig-switch-context >/dev/null 2>&1

The installer passes no --kubeconfig/--context to tracebloc client create and the CLI
follows plain kubectl precedence — so a silently failed merge anchors this machine to whatever
cluster was current (a corporate EKS, a stale context).

Now:

  • the status is captured and the install stops, surfacing k3d's own words;
  • the merge is bounded (TB_KUBECONFIG_MERGE_TIMEOUT, default 60s) — k3d reads the
    kubeconfig out of the node through the Docker daemon, so a wedged daemon used to hang the
    install here with no output at all. rc 124 gets its own message;
  • the anchor is verified, not inferred: after the merge, kubectl config current-context
    must be k3d-<cluster>. A zero exit is evidence, not proof. A context that cannot be read
    is treated as a failure, not as agreement.

The new failure message (merge path):

 ⚠ Couldn't point kubectl at the 'tracebloc' cluster (k3d kubeconfig merge exited 1).
k3d said: FATA[0000] open /home/u/.kube/config: permission denied
Stopping here on purpose: this machine's secure environment is registered against whichever
cluster kubectl currently points at, so continuing would connect it to the wrong cluster.
Common causes: /home/u/.kube/config isn't writable, the disk is full, or KUBECONFIG points somewhere unexpected.
KUBECONFIG=/home/u/.kube/config
Fix that (or merge it yourself with the command below), then re-run this installer:
k3d kubeconfig merge tracebloc --kubeconfig-merge-default --kubeconfig-switch-context
✖ kubectl was not pointed at 'tracebloc' — refusing to continue against an unknown cluster.

and the anchor path:

 ⚠ k3d merged the 'tracebloc' kubeconfig, but kubectl's current context is
'arn:aws:eks:eu-central-1:111122223333:cluster/corp-prod', not 'k3d-tracebloc'.
…
kubectl config use-context k3d-tracebloc
✖ kubectl is not pointed at 'tracebloc' — refusing to continue against an unknown cluster.

--kubeconfig/--context on client create — not done here, and why

client create does accept both flags (cli/internal/cli/flags.go::addKubeconfigFlags, wired in
client.go:88), and passing them would state the anchor instead of inheriting it. But the call
site is scripts/lib/provision.sh:314, which another session owns this round — so it is out of
this PR's territory rather than out of scope on the merits. It is a small follow-up: add
--kubeconfig "$KUBECONFIG" --context "k3d-$CLUSTER_NAME" to _create_args. The check added
here makes the inherited anchor correct at the moment of provisioning; the flags would make it
stated. Both are worth having.

2. Every recreate hint stranded a dashboard record (tracebloc/backend#2077)

The backend record is anchored to the cluster's identity (the kube-system namespace UID),
which dies with the k3d cluster. k3d cluster delete never calls the API, so the record keeps a
cluster_id that will never exist again — the next run correctly registers a new secure
environment and the old one is stranded on the dashboard for good.

Every k3d cluster delete … && re-run remedy now routes through one helper
(_recreate_cluster_hint / Write-RecreateClusterHint) that names the release step first:

 Release this machine's secure environment BEFORE deleting the cluster — it is anchored to the
cluster's identity, so deleting the cluster first strands it on your dashboard for good:
tracebloc delete --keep-data (releases this secure environment; keeps your local data)
k3d cluster delete tracebloc && re-run this installer.
(nothing installed on this machine yet? then just the k3d line.)

12 sites — 7 in scripts/lib/cluster.sh (proxy, CA, bind, dataset-mount, storage-mode ×2,
k3s-version drift) and 5 in scripts/install-k8s.ps1 (k3s drift, 0.0.0.0 bind, CA drift,
dataset mount, and the two GPU-capability remedies, which are the same defect and have no bash
counterpart). The Windows GPU_SKIP_REASON summary string got the same treatment inline.

tracebloc delete was verified against cli/internal/cli/delete.go before being recommended:

  • it revokes the machine credential server-side; the record is kept as history, never
    hard-destroyed;
  • the revoke is an API call, so it still works when the cluster itself is broken — the state
    at most of these call sites;
  • it also helm uninstalls and tears down its own local cluster;
  • --keep-data is not optional. The plain form wipes ~/.tracebloc, which is
    HOST_DATA_DIR by default — precisely the data these sites promise a recreate keeps. It also
    spares the stored login, so the re-run doesn't sign in again;
  • the k3d line stays: nodeboot.ClusterName is hardcoded "tracebloc", so a custom
    CLUSTER_NAME still needs it (and on the default name it is a harmless no-op);
  • the last line covers the "broken cluster, never provisioned" call sites — with no active
    client, tracebloc delete refuses with "no active client on this machine", so the user is
    told they can skip it.

No mint-time guard was added. One-client-per-CLUSTER is the intended model; one-per-account
explicitly is not (a second machine legitimately gets its own).

Sites deliberately NOT changed

  • cluster.sh:1079/1081 (create timeout). Listed in #2077, but it is not a stranding site:
    k3d cluster create only runs when the cluster is absent, so the partial cluster the
    installer removes never had a secure environment registered against it. Adding a
    tracebloc delete hint there would fail with "no active client on this machine". The
    derived guard is scoped to hint lines for this reason, documented at the test.
  • scripts/lib/provision.sh:230 and its PowerShell peer (install-k8s.ps1:4639), plus
    install-client-helm.sh:1171, summary.sh:240, setup-linux.sh:1376 — other sessions own
    those files this round. They carry the same (wipes this client + its local data) form and
    need the same edit.
  • The cosign Windows-on-ARM note in The kubeconfig merge discards its exit code — a silent failure anchors the client to whatever cluster was current #732 is scripts/install.ps1 and is being handled on
    fix/cosign-windows-arm64-asset.

Tests + mutation evidence

New coverage in scripts/tests/cluster.bats (13 tests) and
scripts/tests/install-k8s.Tests.ps1 (11). Each guard was broken and watched to redden:

mutationresult
|| merge_rc=$?|| true (the original defect)bats 93, 94 fail
context check → if falsebats 95(now 96), 96(97) fail
one call site back to a bare hint " k3d cluster delete …"the derived guard fails and names the line: 926: hint " k3d cluster delete $CLUSTER_NAME …"
a guidance line deleted from the merge-failure branchthe no-output test fails
PS merge back to | Out-Null + a PS site back to a bare Hint4 Pester source guards fail

The derived guards parse the source rather than restating a list, so a new site added later
without the release step reddens too.

One pre-existing Pester assertion (Test-HealthyClusterGpuConsistent … "warns with the recreate
remedy") asserted the literal k3d cluster delete $CLUSTER_NAME inside that function; it now
asserts the call to the helper that owns the wording, with the wording asserted in the helper's
own Describe.

make check green
make bats 1077 ok / 0 not ok
Invoke-Pester 686 passed, 0 failed, 13 skipped
scripts/gen-manifest.sh regenerated (cluster.sh + install-k8s.ps1 are both in the bootstrap's integrity surface)

🤖 Generated with Claude Code


Note

Medium Risk
Changes the installer's fail-closed behavior on kubeconfig/context and user-facing recreate steps; mistakes could block installs or mis-document offboarding, but the change reduces wrong-cluster registration risk rather than weakening security.

Overview
Kubeconfig merge (#732) — Bash and PowerShell installers no longer ignore k3d kubeconfig merge failures. The merge runs with a timeout, uses --kubeconfig-merge-default, and the install stops if merge fails or if kubectl config current-context is not k3d-<cluster> (with guidance to fix or use-context). On Windows, context is parsed from bounded kubectl output so stderr warnings don't false-fail a good install.

Recreate guidance (backend#2077) — Shared helpers (_recreate_cluster_hint / Write-RecreateClusterHint) replace scattered k3d cluster delete hints at many drift sites (k3s version, proxy/CA/bind/dataset/storage, GPU). Users are told to tracebloc delete --keep-data first so the dashboard secure-environment record isn't stranded when the cluster identity dies with the cluster.

Tests — New bats/Pester coverage for merge failure, timeout, wrong context, recreate hint wording, and source guards so bare recreate hints can't creep back in. manifest.sha256 updated for touched scripts.

Reviewed by Cursor Bugbot for commit 1e84c84. Bugbot is set up for automated code reviews on this repo. Configure here.

…cord before a recreate
Two ways the installer quietly pointed a secure environment at the wrong
place. Both land in cluster.sh and its Windows peer.
client#732 — `_merge_kubeconfig` ran `k3d kubeconfig merge` under
`>/dev/null 2>&1` with no `||`, so a failed merge left the previous
current-context selected and the install carried straight on. The
installer passes no --kubeconfig/--context to `tracebloc client create`
and the CLI follows plain kubectl precedence, so that anchors this
machine to whatever cluster happened to be current — a corporate EKS, a
colleague's kind cluster. It failed open, silently, exactly in the
environments that have other clusters configured.
Now the status is captured (bounded, so a wedged Docker daemon can't hang
it), k3d's own words are surfaced, and the install stops. The anchor
itself is then verified rather than inferred: kubectl's current-context
must be `k3d-<cluster>`, and a context we cannot READ counts as a
failure, not as agreement.
tracebloc/backend#2077 (client half) — the backend record is anchored to
the CLUSTER's identity (the kube-system namespace UID), which dies with
the k3d cluster. `k3d cluster delete` never calls the API, so every
recreate strands the dashboard record for good — and the installer
printed exactly that advice at seven drift remedies here (proxy, CA,
bind, dataset mount, storage mode ×2, k3s version) plus five on Windows.
Each now routes through one helper that names `tracebloc delete
--keep-data` first: it revokes the machine credential (keeping the
history as a record) and works over the API even when the cluster is
broken, and --keep-data is required because the plain form wipes the very
data these sites promise a recreate keeps.
Verified `tracebloc delete`'s behaviour in cli/internal/cli/delete.go
before recommending it. The k3d line stays because the CLI only tears
down a cluster literally named `tracebloc`, so a custom CLUSTER_NAME
still needs it.
No mint-time guard was added: one-client-per-CLUSTER is the model, and a
second machine legitimately gets its own.
A derived guard (bash and PowerShell) reddens on any future recreate hint
that prints `k3d cluster delete` on its own instead of going through the
helper.
Refs client#732, tracebloc/backend#2077
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@LukasWodkaLukasWodka self-assigned this Aug 17, 2026
Comment threadscripts/install-k8s.ps1 Outdated
Comment threadscripts/install-k8s.ps1 Outdated
… and be bounded
Two Bugbot findings on the PowerShell peer, both real.
HIGH — the merge omitted `--kubeconfig-merge-default`, so k3d wrote a
standalone ~/.k3d/kubeconfig-<cluster>.yaml and never touched the file
kubectl reads. The exit-code gate added in the previous commit therefore
proved nothing about the anchor on Windows, and the remedy it printed
could not have repaired it either. Add the flag to the call and to the
printed command.
Because the flag makes k3d honour $KUBECONFIG, the 0.0.0.0 normalization
that follows now targets the file k3d actually wrote (first entry of the
';'-separated list, else %USERPROFILE%\.kube\config) instead of a
hardcoded profile path — the peer of bash's ${KUBECONFIG%%:*}.
MEDIUM — both external calls ran with no deadline while the bash peer
bounds both. They now go through Invoke-BoundedProcess (60s for the
merge, which reaches the Docker daemon; 10s for the context read), which
also gives the timeout its own message instead of a bare exit code.
Refs client#732
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit a1f052c. Configure here.

Comment threadscripts/install-k8s.ps1
… anchored install
Invoke-BoundedProcess returns stdout and stderr concatenated, and the
anchor check compared that whole blob against `k3d-<cluster>`. So a
kubectl deprecation or plugin warning — written to stderr on an otherwise
perfect run — would have hard-stopped an install whose context was
exactly right. The bash peer never had this: it reads the probe under
`2>/dev/null`.
Get-CurrentContextFromOutput is the Windows equivalent, kept pure so it
is unit-testable: kubectl prints the context as one line on stdout and
Invoke-BoundedProcess puts stdout first, so the first non-empty line is
the context and what follows is noise. Empty stays empty — "couldn't
tell" is still a failure at the call site, never a pass — and a
stderr-only line that merely mentions the context name does not read as
anchored.
Refs client#732
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@saadqbalsaadqbal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, careful PR 👍 Both directions of the merge fix read right: fail-closed keys on the merge command's own exit code (bash || merge_rc=$?, PowerShell $merge.Code, and Err does exit 1), a good merge proceeds, and adding --kubeconfig-merge-default is what makes that zero-exit actually mean something. The stderr refinement is the correct shape too — signal is the exit code plus the first stdout line via Get-CurrentContextFromOutput, never stderr-nonempty, so a kubectl deprecation warning on a healthy install can't false-abort (bash gets it free with 2>/dev/null). Release-before-delete is advice-only text, so no orphan-window risk. bash/PowerShell at parity, manifest hashes both match, and 107 bats green here including the 15 new ones (Pester not run — no pwsh locally).

One out-of-scope thought, not for this PR: the image_pull_ca diagnosis block (~L5111) still prints a bare k3d cluster delete + re-run-with-CA remedy without the tracebloc delete --keep-data release — same stranding this PR fixes, and by the time an x509 in-cluster-pull failure surfaces a record has likely already been registered. It's a Write-Host block in the failure summary rather than a reuse-path Hint site, and there's no bash peer, so leaving it doesn't break parity. Might be worth a follow-up ticket.

@LukasWodka
LukasWodka merged commit b5bbcf6 into developAug 17, 2026
47 checks passed
@LukasWodka
LukasWodka deleted the fix/732-kubeconfig-merge-and-delete-hints branch August 17, 2026 11:29
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

/fr-pass

Best-effort functional review passed (triage: merged clean, non-interactive; behavioral evidence limited while e2e journey is red — backend#2206). Advancing to Ready for prod.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The kubeconfig merge discards its exit code — a silent failure anchors the client to whatever cluster was current

2 participants

@LukasWodka@saadqbal