fix(installer): pre-create the hostPath PV dirs on Windows so the first ingest works

[shujaatTracebloc]

Closes#653.

The bug

A fresh Windows install completes clean, then the firstdata ingest fails:

remote tar stderr: mkdir: can't create directory '/data/shared/.tracebloc-staging/': Permission denied

Nothing in the install output hints at it. The cluster is healthy, the release is deployed, and the
failure only surfaces at first use.

Root cause — a bash/PowerShell asymmetry, not a chart problem

The chart's hostPath PVs bind /tracebloc/<release>/data and /tracebloc/<release>/logs (in the
pod: /data/shared, /data/logs). When those host paths don't exist, kubelet's
DirectoryOrCreate creates them root:root 0755, and kubelet ignores fsGroup on hostPath
volumes (kubernetes#138411). uid 1000 can't create anything inside, so the staging tar dies on
its first mkdir.

The bash installer has always pre-created these — _ensure_release_dirs in
scripts/lib/cluster.sh, whose own comment names this exact failure:

Without pre-creating these as the host user, kubelet's DirectoryOrCreate makes them root:root
0755 and the host user can't drop training data into /data/shared.

install-k8s.ps1 had no equivalent — it creates HOST_DATA_DIR but never the per-release
data/logs subdirs. That asymmetry is the entire reason this was Windows-only.

Current charts also ship an init-writable-data init container that fixes the same two paths at
pod start, which masks the gap — but it isn't in the currently published chart, so every fresh
Windows install lands on the broken combination.

The fix

• Get-ReleaseDirsPrepCommand (pure) builds the sh -c body; Initialize-ReleaseDataDirs runs it
  in the server node and reports honestly.
• Runs before Helm, keyed on the release name Helm is about to touch (adopted vs fresh) — the
  PV path embeds it, so preparing the wrong release would look successful and fix nothing.
• Matches init-writable-data's end state (chown 1000:1000, chmod 3777), so a cluster on an
  older published chart ends up in the same state as one on a current chart.
• Also repairs on the nothing-to-do fast path. That path exit 0s before Helm, so a cluster
  installed before this fix is healthy, shortcuts every re-run, and would keep failing at first
  ingest. "Re-run the installer" has to be a real remedy, not advice that quietly does nothing.
• Never fatal. A non-k3d cluster, a docker exec timeout (code 124), or a mount that can't
  represent POSIX ownership all degrade to a warning plus a copy-pasteable repair command; the
  install completes either way, and on a current chart the init container fixes the same dirs anyway.
• mysql's PV is deliberately out of scope: it has its own init container and its datadir
  permissions are the database's business.

A bug this code had during development, and what it changes about the tests

The first version read the mode with stat -c. That's a GNU/coreutils flag BSD stat rejects, and
it fails silently — no output, empty mode string, so a correctly-chmodded directory reported
FAIL and the installer would warn on a perfectly healthy install. Same family as the
sha256sum --check trap (#429).

Asserting on the command string could never have caught that. The new bats tests execute the
generated command against temp dirs, which is what found it — so the fix now uses POSIX ls -ldn,
which behaves identically on busybox (rancher/k3s), coreutils (the CUDA node image) and BSD.

Verification

• Pester: 608 passed, 0 failed (+8 new).
• bats: 5 new tests in hostpath-prep.bats, all passing — POSIX/dash syntax check, OK path,
  FAIL path (mode 0755 not owned by 1000, i.e. what kubelet leaves), idempotence on re-run, and a
  stat -c regression guard.
• Proved the tests bite: reintroducing stat -c fails bats 2, 3 and 5. (1 still passes — the
  syntax is still valid; 4 fails either way — that's the correct discrimination.)
• bats-hygiene.bats clean (18 ok), so the new assertions are || return 1-hardened.
• check-style.sh and check-facts.sh clean; scripts/manifest.sha256 regenerated (the .ps1 is
  manifested — 1 line changed).
• No client/templates|values|schema changes, so no Chart.yaml bump is required.
• Full scripts/tests/*.bats: 428 ok, 1 failure — install-bootstrap.bats "early bailout: healthy
  tracebloc doctor". Pre-existing: I reproduced it on a pristine origin/develop worktree, so
  it is untouched by this change and left alone.

What I could not verify locally

I develop on macOS, so the docker exec into a k3d node isn't exercised here — the shell it sends
is executed and tested, but not through Docker on Windows. Worth one confirmation on a Windows box:
a fresh install should ingest with no manual chmod, and a re-run on an existing cluster should
repair it.

Scope note

The durable fix for users on a stale published chart is publishing a chart that contains
init-writable-data. This PR makes the installer correct on its own terms regardless of which
chart version it lands on, which is the part that belongs in code.

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches install-time host permissions and release PV paths on Windows/k3d; failures are warned-only but wrong modes could still break ingest or delete until repaired.

Overview
Parity with the bash installer:install-k8s.ps1 now pre-creates and chmods the chart’s data and logs hostPath dirs inside the k3d node before Helm (and again on the healthy “nothing to do” fast path), so kubelet doesn’t leave them root-owned 0755 and the first data ingest doesn’t hit Permission denied on /data/shared.

The prep uses per-dir modes aligned with chart init-writable-data (#667) — 2777 on shared data (no sticky, so cross-uid data delete works) and 3777 on logs — with data rooted at /tracebloc-data when the node has a dataset bind mount (Get-NodeDataBase via docker inspect, not ephemeral HOST_DATASET_DIR). The shell script is fed on stdin to docker exec … sh (avoids Windows argv quoting bugs), verifies writability with POSIX ls -ldn (not stat -c), requires an OK line per expected path, and on failure emits a copy-paste repair hint without aborting install.

Tests: large new Pester coverage plus hostpath-prep.bats that actually runs the extracted prep script; scripts/manifest.sha256 updated for install-k8s.ps1.

^{Reviewed by Cursor Bugbot for commit 431a607. Bugbot is set up for automated code reviews on this repo. Configure here.}

[aptracebloc]

@shujaatTracebloc — Approved. Verified against a fresh clone of the head.

• R8 manifestshasum -a 256 -c scripts/manifest.sha256 → all 18 OK incl. scripts/install-k8s.ps1 (head 3fbbb32).
• batshostpath-prep.bats 6/6 (validates the POSIX ls -ldn cross-platform path and that the tests execute the generated shell incl. the FAIL path). Pester ran green in CI on windows + ubuntu (no pwsh locally to re-run).
• Logic: prep runs before the adopt/fresh split with the release key the PV paths embed, guarded/idempotent/non-fatal, node paths POSIX (in-node hostPath, not Windows), no check-facts drift. The root cause is permissions (kubelet's DirectoryOrCreate makes the dir root:root 0755 and ignores fsGroup on hostPath — kubernetes#138411), and the in-node chown 1000:1000 + chmod 3777 matching the chart's init-writable-data is the correct fix. Both Bugbot findings (docker-exec quoting → stdin delivery; HOST_DATASET_DIR path) resolved.

Code-owner @saadqbal already approved the current head. The PR is still BLOCKED but not on anything in the diff — code-owner met, mergeable, 36 checks green; it looks like an org-level develop ruleset requiring a context that's stuck queued with 0 runs (claude/github-pages ghost suites). Worth an admin checking the develop ruleset's required-checks list — nothing to fix in this PR.

This is the correct/complete fix for #653; I've recommended closing the competing #659 in its favor (they conflict on the R8 manifest).

🤖 Generated with Claude Code

[LukasWodka]

@aptracebloc@divyasinghds@shujaatTracebloc@saadqbal — #659 has since merged, so the collision Arturo predicted is now live and needs reconciling. I dug into the mechanism to work out which fix is actually load-bearing. Posting the evidence rather than an opinion.

The premise that decides it: HOST_DATA_DIR is bind-mounted into the node

scripts/install-k8s.ps1:3192 passes:

-v ${HOST_DATA_DIR}:/tracebloc@all

and the chart's hostPath resolves to /tracebloc/<release>/data (client/templates/_helpers.tpl:113, datasetPath defaulting to /tracebloc).

So on this layout the Windows directory is the in-node path — creating HOST_DATA_DIR\<release>\data on the host makes /tracebloc/<release>/data exist inside the k3d node, through the same bind mount. kubelet never reaches DirectoryOrCreate, because the path is already there.

That means the DirectoryOrCreate → root:root 0755 + fsGroup-ignored reasoning (correct for a native hostPath) doesn't apply to a Docker Desktop bind mount, which supplies its own ownership/mode. Divya's note — "Docker Desktop handles Windows mount ownership" — matches that. It's also, tellingly, what this PR's own script already assumes: chown 1000:1000 … 2>/dev/null is non-fatal and the verifier accepts eithero = 1000or an other-writable mode:

case "$m" in ????????w*) w=1;; *) w=0;; esac; [ "$o" = 1000 ] && w=1

A Docker Desktop mount lands on the second branch. So the in-node step is written to tolerate exactly the situation where the host-side pre-create was already sufficient.

On the HOST_DATASET_DIR gap

That one is covered by #659: Ensure-ReleaseDirs branches on it ($dataBase = HOST_DATASET_DIR\<release>), and the installer both mounts it (install-k8s.ps1:3202, -v ${HOST_DATASET_DIR}:/tracebloc-data@all) and points the chart at it (:4389, datasetPath: /tracebloc-data).

What this PR still adds that #659 does not

Genuinely useful, and I would not want it dropped:

• In-node verification of the resulting owner/mode, with the exact docker exec … chmod remediation hint on failure.
• Graceful degradation for a cluster that isn't k3d-shaped, or a docker exec that times out.
• Tests — Pester + scripts/tests/hostpath-prep.bats. fix(installer): pre-create per-release hostPath dirs on Windows (#653) #659 shipped with none.

And one honest caveat: whether a Docker Desktop Windows bind mount really does present writable to uid 1000 is an empirical question, and I have no Windows host to settle it. If it does not, this PR's in-node step isn't a belt — it's the actual fix. That argues for keeping it either way.

Suggested reconciliation

Rather than closing either one:

1. Rebase this PR onto develop (where fix(installer): pre-create per-release hostPath dirs on Windows (#653) #659's Ensure-ReleaseDirs + regenerated manifest.sha256 now live).
2. Keep only what's additive — Get-ReleaseDirsPrepCommand / Initialize-ReleaseDataDirs, the verification + hint, and both test files — layered after the existing host-side pre-create.
3. Regenerate scripts/manifest.sha256 once on top of develop. That dissolves the mutually-exclusive-hash problem by construction.

End state: host-side pre-create (fixes the common Docker-Desktop path) plus in-node ownership enforcement with verification (covers any layout where the mount doesn't present writable) plus the test coverage. Neither review gets overruled.

Happy to do the rebase/reduction myself if @aptracebloc / @shujaatTracebloc prefer — it's your PR, so I haven't touched it.

🤖 Generated with Claude Code

[divyasinghds]

Requesting changes — the implementation is strong (pure Get-ReleaseDirsPrepCommand, ls -ldn for BSD/busybox portability, ground-truth mount detection, honest per-dir OK/FAIL, thorough Pester+bats coverage, CI green) and it correctly complements the merged #659 rather than duplicating it. Two things block merge:

1. It reintroduces the bug client#667 is removing. Get-ReleaseDirsPrepCommand runs chmod 3777 on every release dir, including the data dir that maps to /data/shared. That's setgid + sticky — the mode #667 deletes from /data/shared so the data delete teardown (a different uid) can clean up. In the scenarios this PR targets — the currently-published chart has no init-writable-data (per this PR's own body), and the 'nothing-to-do fast path' that exit 0s before Helm — nothing runs afterward to correct it, so /data/shared is left 3777 sticky and data delete breaks on Windows, re-opening #476. Please adopt #667's per-directory split: 2777 (setgid, no sticky) for the shared/data dir, 3777 for logs.

2. Branch is diverged / behind_by=3 from develop (which now includes the merged #659). Per our 'merge develop back in before review / never build on a stale checkout' rule, please rebase — and regenerate the manifest.sha256 line for install-k8s.ps1 against the post-#659 file, or it'll clobber #659's hash.

Neither is a code-quality issue — the mode constant just predates #667's sticky split and the branch went stale. Adopt 2777 on the shared dir + rebase and I'll approve.

[shujaatTracebloc]

@divyasinghds — both points were right, and the first one is a contradiction inside my own work. Fixed in a672e68.

1. Sticky bit reintroduced. I identified the sticky bit as what makes data delete impossible, removed it from the chart in #667 — and left this installer setting chmod 3777 on the data dir. Your scenario analysis is exactly right about why that isn't cosmetic: the published chart has no init-writable-data, and the nothing-to-do fast path returns before Helm runs, so in both cases nothing comes along afterwards to correct a sticky bit the installer set. /data/shared would be left 3777 and the teardown — uid 65532 removing a tree the ingest wrote as 65534 — cannot unlink it. Table dropped, files stranded, re-opening the delete bug.

Now on #667's per-directory split:

Dir	Mode
/data/shared	2777	setgid + world-write, no sticky
/data/logs	3777	setgid + sticky — nothing deletes another writer's logs

Dirs are emitted as path:mode pairs and split with ${e%:*} / ${e#*:} — the same idiom the chart's init container uses, so the installer and the chart can be diffed by eye instead of drifting. The modes are named constants for the same reason.

Reviewing that also turned up a flaw in my first pass: the loop used $m for both the desired mode and the ls-derived one, which only worked because chmod happened to run first. The wanted mode is now $want, with a test pinning it so a reorder can't quietly break it.

2. Behind develop. Merged origin/develop (now includes #659 and the merged #667). The manifest was regenerated against the post-merge file specifically so it can't clobber #659's hash — verified with shasum -a 256 -c scripts/manifest.sha256: 18/18 OK.

Verification: 625 Pester, 24 bats, check-style + check-facts clean. I executed the generated shell rather than only asserting on its text — data comes out drwxrwsrwx, logs drwxrwsrwt — and confirmed the guard bites: putting the shared dir back to 3777 takes the suite to 624/1. One more real bug surfaced while writing that: a bats substitution silently expanded $want, leaving the chmod in place so the "non-writable" case was quietly writable and the test proved nothing. Escaped and re-verified.

On your note about the earlier approval mentioning 3777 as correct — that was accurate against the chart at the time; #667 changed the target state for /data/shared, and this PR now follows it.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a672e68. Configure here.}

[shujaatTracebloc]

@divyasinghds — both of your conditions are met; requesting re-review.

1. Per-dir sticky split adopted (a672e68). /data/shared → 2777 (setgid + world-write, no sticky), /data/logs → 3777. Dirs are emitted as path:mode pairs and split with ${e%:*} / ${e#*:} — the same idiom init-writable-data uses, so the installer and the chart can be diffed by eye rather than drifting. Verified by executing the generated shell: data comes out drwxrwsrwx, logs drwxrwsrwt.

2. Branch current + manifest regenerated (2acd00e). Merged origin/develop — now includes #659 and the merged #667 — and regenerated scripts/manifest.sha256 against the post-merge file so it can't clobber develop's hash. shasum -a 256 -c: 18/18 OK.

Your review also led to two further fixes worth flagging:

• Bugbot then caught that the repair hint still printed chmod -R 3777 for both dirs — a user following the installer's own copy-paste would have put the sticky bit back on /data/shared and broken data delete. Rather than fix it in place, Get-ReleaseDirsSpec is now the single source of path+mode and the prep command, the verification list and the hint all derive from it, so they can't disagree by construction. The hint test asserted paths but not modes, which is how it slipped; it now asserts modes per dir. Dropped -R too — the directory's mode governs unlink, and recursing would stamp setgid onto every data file.
• The shell loop used $m for both the desired and the observed mode, working only because chmod ran first. Now $want, with a test pinning it.

Current state: 626 Pester, 24 bats, check-style + check-facts clean, 34 CI checks green, Bugbot success on the current head, 0 unresolved threads. @saadqbal approved at 14:02 after these changes.

One thing I could not verify and don't want to imply I did: the prep path has never executed against a real k3d node — no CI runner drives Docker on Windows. CI proves the generated shell is correct and that argv/stdin delivery holds; the end-to-end confirmation is a fresh Windows install that ingests with no manual chmod, and then a data delete.

[LukasWodka]

Approved, with follow-ups — none blocking.

The diagnosis is correct, the mechanism claims hold up under independent checking, and the tests are the good kind: they execute the shell instead of asserting on its string. Posting what I verified rather than just a verdict.

Verified independently at head 2acd00e

Check	Result
shasum -a 256 -c scripts/manifest.sha256	18/18 OK, incl. the .ps1
bats scripts/tests/hostpath-prep.bats	6/6 pass
bats scripts/tests/bats-hygiene.bats	18 ok
Generated shell on busybox (the k3s node's shell), ubuntu 24.04 (CUDA node image) and macOS/BSD — piped on stdin exactly as the installer delivers it	Identical output on all three: data → drwxrwsrwx (2777), logs → drwxrwsrwt (3777)
Mode alignment vs merged #667	Byte-for-byte the same idiom and modes as the chart's init-writable-data (for e in /data/shared:2777 /data/logs:3777 … chown 1000:1000)
Path split vs chart	tracebloc.clientDataHostPath = <datasetPath|/tracebloc>/<release>/data; logs-pvc.yaml hardcodes /tracebloc/<release>/logs — so parameterising only data is right
Call ordering	New-K3dCluster (step 3) precedes Install-ClientHelm (step 6), so the node exists when prep runs
Invoke-BoundedProcess	Output is a single concatenated string, so the (?m)^… anchors work; checked the mount regex and the OK regex including CRLF. Stdin plumbing (RedirectStandardInput/Write/Close) is correct

The ls -ldn portability fix is the real substance here and it holds up. I also chased a suspicion that the Should -Invoke Warn -Times 0 assertions were vacuous — they are not; Pester special-cases -Times 0 as exactly-zero even without -Exactly.

Findings

1. The writability check can't see the sticky bit it was tightened to prevent. The pass condition is other-write only (????????w*). Setting a data dir to 1777 and disabling the chmod yields OK … drwxrwxrwt — silently — and sticky on /data/shared is exactly what #667 removed because it makes cross-uid data delete impossible. Narrow trigger (chmod must fail while sticky is set; root_squash NFS is the realistic one), but by this PR's own standard — "absence of failure cannot stand in for success" — the verdict should also assert the sticky expectation derived from $want.

2. if (-not $Release) { return } in Initialize-ReleaseDataDirs is unreachable, and contradicts "Never fatal".[Parameter(Mandatory)][string] throws ParameterBindingValidationException on "" before the body ever runs. Not currently reachable — ConvertTo-WorkspaceName cannot return empty and $pvRelease falls back to $TB_NAMESPACE — but the sibling Ensure-ReleaseDirs($release) no-ops on empty while this one would abort the install. Pick one: drop Mandatory, or drop the guard.

3. The repair hint is chmod-only. If prep failed at mkdir (unwritable parent), the hint's chmod fails the same way — the copy-pasteable remedy doesn't remedy the case that produced it. Worth adding mkdir -p and the chown.

4. Out of scope, but worth its own issue: the merged chart's init-writable-data is chown … && chmod …. chmod is skipped whenever chown fails — precisely the root_squash / pre-provisioned mount its own error message names. This PR's PowerShell gets it right with two independent best-effort statements. That makes the installer prep more load-bearing than the description claims, not redundant.

5. bash is now the odd one out._ensure_release_dirs still does chmod -R 777 — no setgid, and recursive, which the new PowerShell comment argues against. No user-visible bug (777 has no sticky, so delete works), but a change framed as "fix the bash/PowerShell asymmetry" exits with a new one pointing the other way. Follow-up to align bash on the path:mode split.

6. Closes #653 is stale.#653 was closed by #659 at 07:52:59 on 2026-08-11, so that card is already terminal and this PR closes nothing. Retarget the description at a fresh issue, or state plainly "follow-up to #659, no separate ticket", so the board reflects reality.

On the "could not verify locally" note

Most of that gap is now closed — the shell is proven on busybox, coreutils and BSD, delivered on stdin. What's left is genuinely Windows-specific and worth the one confirmation you asked for: on a Docker Desktop bind mount from a Windows path, the in-node chown/chmod will most likely be a silent no-op (9p/virtiofs), so whether the dirs read back as world-writable decides whether this prints nothing or emits "Couldn't confirm the data directories are writable" on a perfectly healthy install — with a hint whose chmod also cannot work there. UX risk rather than a correctness one (the path is non-fatal, and #659 already creates the dirs Windows-side), but it is the same class as the stat -c bug: a scary warning on a good install.

Merge state

Zero unresolved threads and @saadqbal has approved at head, so the only thing still blocking is @divyasinghds's CHANGES_REQUESTED from b96d3327 — GitHub's stale-dismissal clears approvals but not change requests, so it stands until she re-reviews. Both her conditions look met to me (per-dir mode split in a672e68, branch current + manifest in 2acd00e).