feat(#420): resume after reboot + install state file (fast idempotent re-runs) by shujaatTracebloc · Pull Request #501 · tracebloc/client

shujaatTracebloc · 2026-07-30T13:54:06Z

#420 — Resume after reboot + install state file (fast idempotent re-runs)

Two legitimate reboots (Windows feature enablement; Docker/WSL first boot) can interrupt the install. Before, every interruption ended with "re-find and re-paste the one-liner", and each re-run re-walked the whole install (up to ~90s of wsl --update, winget checks, full preflight) — which reads as "the installer is slow/broken".

What's in

Schema-versioned state file%USERPROFILE%\.tracebloc\install-state.json. Pure, unit-tested helpers (New-InstallState, ConvertTo-InstallState, Add-CompletedStage, Test-StateHasStage, Test-InstallStateCurrent) + thin I/O wrappers (Read/Save/Set-StageComplete/Set-InstallComplete). Corrupt or incompatible-schema file → a fresh state, never a throw. The state is advisory: every stage still self-verifies (tools re-checked, cluster re-derived), so a stale checkpoint can never skip real work.
Resume-after-reboot via RunOnce. On a reboot the installer checkpoints features-reboot-pending and registers an HKCU RunOnce continuation (Get-ResumeCommand reuses the feat(installer): self-elevate and guide wrong-terminal users (cmd / unelevated PowerShell) #421 elevation arg-builder + -Resume) so the install resumes at next sign-in with no re-pasting — for both auto-reboot and manual -NoReboot. -Resume is forwarded through the admin-gate self-elevation. Only the durable -File form carries -Resume (the irm|iex shim has no param block, per feat(installer): self-elevate and guide wrong-terminal users (cmd / unelevated PowerShell) #421; the state file drives the one-liner path anyway). Cleared on success so it never fires spuriously.
Fast idempotent re-runs. Each of the 6 steps checkpoints on completion; on a fresh run where a prior install completed and the tools + cluster are still present, the installer prints already installed -- nothing to do and exits 0. The gate verifies real presence, not just the checkpoint, so the "nothing to do" claim is honest.

Acceptance criteria

Reboot mid-install → install resumes at next logon without re-pasting anything (RunOnce continuation + -Resume; live reboot is host-gated, unit-tested here).
Full re-run on a healthy install completes quickly with a clear "nothing to do" summary (fast-path gated on completed + tools + cluster).

Tests

Pure state helpers (round-trip; corrupt/empty/wrong-schema → fresh; append/dedup/order), Get-ResumeCommand (durable -File form carries -File + switches + -Resume; one-liner omits -Resume), Save/Read I/O round-trip + corrupt-file degradation (mocked path — no writes to the real home dir), and source guards for the -Resume plumbing, reboot-exit RunOnce, per-step checkpoints, success clear+complete, and the honest fast-path gate. Full suite: 315 passed / 0 failed / 9 skipped; check-style + check-drift clean; scripts/manifest.sha256 regenerated (R8).

Host-gated (noted, not in this PR)

The literal end-to-end reboot→resume needs a real Windows host — same class as the scheduled Windows e2e leg (#436). The state-file + RunOnce + fast-path logic is unit-tested here.

Closes#420

Note

Medium Risk
Changes first-run and reboot behavior for the Windows installer (RunOnce, elevation, exit codes) and gates “nothing to do” on cluster/client health; mistakes could skip repair or leave stale resume entries, but logic is advisory with live probes and extensive unit tests.

Overview
Adds resume-after-reboot and a schema-versioned install state file under HOST_DATA_DIR so interrupted Windows install-k8s.ps1 runs can continue without re-pasting the one-liner, and healthy machines can exit quickly on repeat runs.

When Windows features need a reboot, the installer registers an HKCU RunOnce continuation (via Get-ResumeCommand / Register-ResumeAfterReboot) and forwards a new -Resume switch through self-elevation (Get-ElevationCommand / Invoke-SelfElevate). The resumed run skips the “nothing to do” fast path and shows a resume banner. RunOnce is cleared on a full successful walk; completion is written to install-state.jsononly when the client is connected (not merely starting), and a stale completed flag is cleared on failure so remediation isn’t skipped.

The fast path exits 0 when state says completed and live checks pass: all tools on PATH, k3d cluster running (bounded k3d cluster list job), and client deployments Ready (Test-ClientHealthy / shared Get-ClientDeploymentNames). Corrupt or wrong-schema state degrades to fresh state without throwing.

Pester coverage for state helpers, cluster list parsing, resume command wiring, completion vs exit predicates, and manifest hash update for install-k8s.ps1.

^{Reviewed by Cursor Bugbot for commit 792aec3. Bugbot is set up for automated code reviews on this repo. Configure here.}

Two legitimate reboots (Windows feature enablement; Docker/WSL first boot) can interrupt the install. Before, every interruption ended with "re-find and re-paste the one-liner", and each re-run re-walked the whole install. Now: - Schema-versioned state file under %USERPROFILE%\.tracebloc\install-state.json. Pure, unit-tested helpers (New-InstallState, ConvertTo-InstallState, Add-CompletedStage, Test-StateHasStage, Test-InstallStateCurrent) + thin I/O wrappers (Read/Save/Set-StageComplete/Set-InstallComplete). A corrupt or incompatible-schema file degrades to a fresh state -- never a throw. The state is ADVISORY: every stage still self-verifies (tools re-checked, cluster re-derived), so a stale checkpoint can never skip real work. - Resume-after-reboot via a RunOnce continuation. On a reboot the installer checkpoints 'features-reboot-pending' and registers HKCU RunOnce (Get-ResumeCommand reuses the #421 elevation arg-builder + -Resume) so the install resumes at next sign-in with no re-pasting -- for both auto-reboot and manual -NoReboot. -Resume is forwarded through the admin-gate self-elevation. Only the durable -File form carries -Resume; the irm|iex shim has no param block (#421), and the state file drives the one-liner path anyway. Cleared on success so it never fires spuriously. - Fast idempotent re-runs. Each of the 6 steps checkpoints on completion; on a fresh run where a prior install completed AND the tools + cluster are still present, the installer prints "already installed -- nothing to do" and exits 0. The gate verifies real presence, not just the checkpoint, so the claim is honest. Tests: pure state helpers (round-trip, corrupt/empty/wrong-schema -> fresh, dedup), Get-ResumeCommand (File form carries -File+switches+-Resume; one-liner omits -Resume), Save/Read I/O round-trip + corrupt-file degradation (mocked path, no home writes), and source guards for the -Resume plumbing, reboot-exit RunOnce, per-step checkpoints, success clear+complete, and the honest fast-path gate. Closes#420 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

LukasWodka · 2026-07-30T13:54:27Z

👋 Heads-up — Code review queue is at 36 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

averaging-service#250 — release-train: develop -> staging · author: @LukasWodka · no reviewer assigned
backend#1146 — docs(rfc): platform cost & autoscaling architecture · author: @saadqbal · reviewer: @LukasWodka, @saqlainsyed007, @aptracebloc
backend#1201 — docs(rfc): Data spaces & federated dataset composition (draft) · author: @LukasWodka · reviewer: @saadqbal
backend#1217 — docs(rfc): Dataset integrity & score reproducibility (draft) · author: @LukasWodka · reviewer: @saadqbal
backend#1261 — RFC-BACKEND-0008: board & process reset (decision RFC — for markup) · author: @LukasWodka · reviewer: @saadqbal
backend#1284 — docs(deploy): backend deploy runbook skeleton (#1263) · author: @LukasWodka · no reviewer assigned
backend#1332 — feat(check-model): allow KNN/GaussianProcess on a single-edge owned dataset (#923 follow-up) · author: @divyasinghds · no reviewer assigned
backend#1353 — chore(ci): baseline historical gitleaks findings + wire caller · author: @LukasWodka · no reviewer assigned
backend#1361 — feat(tasks): bound ExperimentQueue growth for PAUSED experiments (backend#1351) · author: @saqlainsyed007 · reviewer: @saadqbal
backend#1367 — feat(blob_store): migrate check_model_utils (CheckModelMixin) to blob_store (#1162) · author: @saadqbal · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

LukasWodka · 2026-07-30T13:58:17Z

👋 Heads-up — Code review queue is at 37 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

averaging-service#250 — release-train: develop -> staging · author: @LukasWodka · no reviewer assigned
averaging-service#253 — feat(blob_store): write averaged global weights through blob_store (#1162) · author: @saadqbal · no reviewer assigned
backend#1146 — docs(rfc): platform cost & autoscaling architecture · author: @saadqbal · reviewer: @LukasWodka, @saqlainsyed007, @aptracebloc
backend#1201 — docs(rfc): Data spaces & federated dataset composition (draft) · author: @LukasWodka · reviewer: @saadqbal
backend#1217 — docs(rfc): Dataset integrity & score reproducibility (draft) · author: @LukasWodka · reviewer: @saadqbal
backend#1261 — RFC-BACKEND-0008: board & process reset (decision RFC — for markup) · author: @LukasWodka · reviewer: @saadqbal
backend#1284 — docs(deploy): backend deploy runbook skeleton (#1263) · author: @LukasWodka · no reviewer assigned
backend#1332 — feat(check-model): allow KNN/GaussianProcess on a single-edge owned dataset (#923 follow-up) · author: @divyasinghds · no reviewer assigned
backend#1353 — chore(ci): baseline historical gitleaks findings + wire caller · author: @LukasWodka · no reviewer assigned
backend#1361 — feat(tasks): bound ExperimentQueue growth for PAUSED experiments (backend#1351) · author: @saqlainsyed007 · reviewer: @saadqbal

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

Bugbot (High): Set-InstallComplete ran unconditionally before the exit-code check, so a failed client state (bad_creds/crash/image_pull/image_pull_ca) still persisted completed=true. A re-run then hit the tools+cluster fast path, printed "already installed -- nothing to do" and exited 0 -- blocking the documented remediation (the summary tells the operator to re-run). Add Test-InstallSucceeded (connected/starting = success) as the SINGLE source of truth shared by the completion checkpoint and the exit code, so they can't drift. Set-InstallComplete now runs only when Test-InstallSucceeded; the exit code uses the same predicate. RunOnce is still cleared unconditionally (no reboot is pending once the walk finishes). Unit test across connected/starting + failure states, plus source guards for the gated completion and shared-predicate exit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Bugbot round 2 on the new fast path: - High: the probe only checked the cluster NAME appeared in `k3d cluster list`, not that it was running -- so a completed install with a STOPPED cluster (or dead node) printed "nothing to do" and exited 0, skipping New-K3dCluster's start/repair. - Medium: it ran a bare `k3d cluster list` with no deadline, so a wedged Docker engine hung the fast path at the start of every re-run (violates the bounded- external-command rule). Replace Test-ClusterPresent with Test-ClusterRunning: BOUNDED via Start-Job + Wait-JobWithProgress (15s) + Remove-Job, parsing `-o json` through a pure Test-ClusterRunningInList that requires serversRunning >= 1 for the named cluster (present-but-stopped -> false -> fall through to the repair walk). Fast-path message now states what was verified ("installed and the cluster is running"). Tests: Test-ClusterRunningInList (running/stopped/absent/corrupt/no-count), a source guard that the probe is job+deadline bounded, and the updated fast-path gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

saadqbal

Solid, well-tested state machine and the honest self-verify framing is the right call. Three things I'd want resolved before merge — two are correctness, one is dead weight. Details inline; happy to be wrong on the first if the split-account flow is out of scope for resume.

… stages, split-account note Asad's review on #501: - Completion counted `starting`, but that's Get-NotReadyState's catch-all for a client that isn't Ready yet -- so a client that never comes up got completed=true and armed the fast path, skipping remediation. Completion now requires the client to be CONNECTED (new Test-InstallConnected); the exit code keeps the more lenient connected||starting (Test-InstallSucceeded) so a slow-but-starting client doesn't hard-fail the run. The two predicates differ on purpose and say why. - The per-stage checkpoints were written but never read (the six steps set shared $script: state, so a resume must re-walk them; speed comes from each step's own self-skip). Dropped the dead machinery -- stages array, Add-CompletedStage, Test-StateHasStage, Set-StageComplete, Test-StageComplete, per-step calls, and the features-reboot-pending checkpoint. State is now just {schema, completed}. - Resume scope clarified: the reboot happens in Step 1 during the elevating account's session, so HKCU RunOnce is correct for that account. Added a Hint for the split -DailyUser case (a different user signing in after the reboot must re-run). Tests updated: pure helpers (no stages), Set-InstallComplete round-trip, Test-InstallConnected vs Test-InstallSucceeded across states, split-account Hint guard, and a guard that the stage machinery stays gone. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1be7c57. Configure here.}

… failure Bugbot round on the review rework: - High: the fast path claimed "nothing to do" from completed + tools + a running cluster WITHOUT checking the client workloads are Ready (the bash assess path requires Ready workloads), and `completed` was never cleared when a later walk failed. So a re-run started because the client is down could print "already installed" and skip the remediation. Now the fast path also requires Test-ClientHealthy (finds the release namespace via Get-InstalledClientInfo, then a SHORT bounded `kubectl rollout status --timeout=5s` per client deployment), and a walk that doesn't end connected calls Clear-InstallCompleted so a stale flag can't keep the fast path armed. Extracted Get-ClientDeploymentNames as the shared source of truth for the readiness gate and the health check (no duplication). - Low: the force-reinstall hint hard-coded ~\.tracebloc\install-state.json; it now interpolates Get-InstallStatePath so it's correct under an overridden HOST_DATA_DIR. Tests: Test-ClientHealthy (unknown/no-ns/all-ready/not-ready via mocks), Get-ClientDeploymentNames, Clear-InstallCompleted round-trip, and updated fast-path + hint-path + completion source guards. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

LukasWodka · 2026-08-02T09:02:36Z

/fr-pass

shujaatTracebloc self-assigned this Jul 30, 2026

shujaatTracebloc marked this pull request as ready for review July 30, 2026 13:57

shujaatTracebloc requested a review from saadqbal as a code owner July 30, 2026 13:57

cursorBot reviewed Jul 30, 2026
View reviewed changes

Comment threadscripts/install-k8s.ps1

cursorBot reviewed Jul 30, 2026
View reviewed changes

Comment threadscripts/install-k8s.ps1
Comment threadscripts/install-k8s.ps1

saadqbal reviewed Jul 30, 2026
View reviewed changes

Comment threadscripts/install-k8s.ps1
Comment threadscripts/install-k8s.ps1 Outdated
Comment threadscripts/install-k8s.ps1 Outdated

cursorBot reviewed Jul 30, 2026
View reviewed changes

Comment threadscripts/install-k8s.ps1
Comment threadscripts/install-k8s.ps1 Outdated

shujaatTracebloc requested a review from aptracebloc July 30, 2026 15:08

aptracebloc approved these changes Jul 30, 2026
View reviewed changes

shujaatTracebloc merged commit ebf0ebf into developJul 30, 2026
35 checks passed

shujaatTracebloc deleted the fix/420-resume-after-reboot branch July 30, 2026 15:14

shujaatTracebloc mentioned this pull request Jul 30, 2026
feat(installer): resume after reboot + install state file (fast idempotent re-runs) #420
Closed

Navigation Menu

Uh oh!

feat(#420): resume after reboot + install state file (fast idempotent re-runs) - #501

feat(#420): resume after reboot + install state file (fast idempotent re-runs)#501
shujaatTracebloc merged 5 commits into
developfrom
fix/420-resume-after-reboot

shujaatTracebloc commented Jul 30, 2026•
edited by cursor Bot
Loading

Uh oh!

LukasWodka commented Jul 30, 2026

Uh oh!

LukasWodka commented Jul 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saadqbal left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursorBot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LukasWodka commented Aug 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

shujaatTracebloc commented Jul 30, 2026• edited by cursor BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

#420 — Resume after reboot + install state file (fast idempotent re-runs)

What's in

Acceptance criteria

Tests

Host-gated (noted, not in this PR)

Uh oh!

LukasWodka commented Jul 30, 2026

Uh oh!

LukasWodka commented Jul 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saadqbal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cursorBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LukasWodka commented Aug 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shujaatTracebloc commented Jul 30, 2026•
edited by cursor Bot
Loading