fix: hermes PVC ownership on Linux k3d + bootstrap ingress URL by nickh-obol · Pull Request #446 · ObolNetwork/obol-stack

nickh-obol · 2026-05-07T20:27:04Z

Summary

Two Linux-specific bugs in fresh obol stack up flows that don't surface on macOS.

1. Hermes pod CrashLoopBackOff on fresh PVC (`internal/hermes/hermes.go`)

The Stack's local-path-provisioner setup script chowns each new PV to 1000:1000, and KubeletInUserNamespace=true (set in internal/embed/k3d-config.yaml) silently skips the fsGroup recursive-chown that would otherwise correct it. Hermes pods run as UID 10000 and crashloop trying to mkdir /data/.hermes/home with Permission denied.

macOS Docker Desktop hides this — its bind-mount filesystem driver fakes file ownership to whoever's asking, so the mismatch never materializes. OpenClaw doesn't hit it either: it pins fsGroup: 1000, accidentally matching the host UID.

Fix: prepend an init-hermes-perms container that runs as root and chowns /data to 10000:10000. Idempotent — also self-heals existing broken PVCs on upgrade.

2. Bootstrap probe + browser URL hardcoded to `:8080` (`cmd/obol/bootstrap.go`)

When ports 80/8080 are in use at start time, k3d remaps the loadbalancer to a random high port and the rest of obol stack up correctly surfaces it via stack.LocalIngressURL(cfg). Three other places in cmd/obol/bootstrap.go were hardcoded to :8080: the readiness probe (hung until timeout), the browser-open URL, and the "view stack interface at" hint.

Fix: reuse stack.LocalIngressURL(cfg) in all three places.

Test plan

go test ./internal/hermes/... ./cmd/obol/... passes
go build ./... clean
Verified end-to-end on Linux: fresh obol agent init lands the agent in Running without manual chown
Verified end-to-end on Linux: obol bootstrap with port 8080 occupied completes without hang; all four URL surfaces (warning / visit / browser-open / next-steps) agree on the alternate port
Smoke-test on macOS to confirm no regression (existing fsGroup path still works; new init container is a no-op on already-correct ownership)

…eps URL

bussyjd

Validated against latest origin/main in an isolated worktree.

Checks run locally:

git merge --no-commit --no-ff refs/remotes/pr/446/head onto origin/main
git diff --check --cached
go test ./internal/hermes ./internal/stack ./cmd/obol
go build ./cmd/obol
go test ./...

All passed. The Hermes PVC ownership fix and bootstrap LocalIngressURL wiring look legitimate.

The in-pod `init-hermes-perms` init container from #446 (c066baa) is neutered on Linux k3d because the embedded k3d config sets `KubeletInUserNamespace=true` (internal/embed/k3d-config.yaml). With user-namespacing, the pod's "root" maps to a host subuid that lacks chown authority over the host bind-mount path, so the in-pod `chown -R 10000:10000 /data` silently no-ops. The next init container (`init-hermes-data`) then fails with `mkdir /data/.hermes/home: Permission denied` and the pod CrashLoopBackOffs. local-path-provisioner's helper-pod sets the volume to 1000:1000 (internal/embed/infrastructure/base/templates/local-path.yaml), which happens to suit OpenClaw but not Hermes (containerUID = 10000). Fix: after `helmfile sync`, host-side chown the PVC backing dirs to containerUID:containerGID by exec-ing into the k3d server container via `docker exec`. That runs at the Docker daemon's real root and is not subject to the user-namespacing that silently breaks the in-pod attempt. The existing `fixRuntimeVolumeOwnership` helper already does exactly this for wallet keystore paths; we wire it into the agent deploy path via a new `ensureHermesPVCOwnership` that: 1. Waits up to 60s for each PVC (`hermes-data`, `remote-signer- keystores`) to be Bound — local-path is WaitForFirstConsumer so the host dir doesn't exist until the pod is scheduled. 2. Chowns each backing dir. 3. If a Hermes pod is currently stuck in Init:CrashLoopBackOff, deletes it so kubelet recreates immediately rather than after exponential backoff (~5 min worst case). Skips the delete when no pod is stuck so routine syncs (e.g. `obol model sync` after `obol model prefer`) do not gratuitously restart a healthy agent. Called from `hermes.Sync` after `helmfile sync` succeeds, so every Onboard / Setup / Sync call exercises it. Validated on spark2 (Linux ARM64, Ubuntu 24.04, NVIDIA GB10) by reverting the PVC dirs to 1000:1000, deleting the Hermes pod, and running `obol model sync`. Before: pod stuck in Init:CrashLoopBackOff with "Permission denied" in init-hermes-data logs. After: PVC dirs flip to 10000:10000 within the sync, pod reaches `Running 2/2` in ~30s with zero CrashLoopBackOff cycles. Test: - `TestHermesPVCPaths` pins the two host paths the helper chowns, so renaming `hermes-data`/`remote-signer-keystores` or relocating the namespace prefix can't silently regress the fix. - Full chown side-effect needs a live k3d cluster and is exercised by the spark2 validation above plus all existing integration flows; no unit-mocked k3d test added.

nickh-obol · 2026-05-07T20:27:04Z

Summary

Two Linux-specific bugs in fresh obol stack up flows that don't surface on macOS.

1. Hermes pod CrashLoopBackOff on fresh PVC (`internal/hermes/hermes.go`)

The Stack's local-path-provisioner setup script chowns each new PV to 1000:1000, and KubeletInUserNamespace=true (set in internal/embed/k3d-config.yaml) silently skips the fsGroup recursive-chown that would otherwise correct it. Hermes pods run as UID 10000 and crashloop trying to mkdir /data/.hermes/home with Permission denied.

macOS Docker Desktop hides this — its bind-mount filesystem driver fakes file ownership to whoever's asking, so the mismatch never materializes. OpenClaw doesn't hit it either: it pins fsGroup: 1000, accidentally matching the host UID.

Fix: prepend an init-hermes-perms container that runs as root and chowns /data to 10000:10000. Idempotent — also self-heals existing broken PVCs on upgrade.

2. Bootstrap probe + browser URL hardcoded to `:8080` (`cmd/obol/bootstrap.go`)

When ports 80/8080 are in use at start time, k3d remaps the loadbalancer to a random high port and the rest of obol stack up correctly surfaces it via stack.LocalIngressURL(cfg). Three other places in cmd/obol/bootstrap.go were hardcoded to :8080: the readiness probe (hung until timeout), the browser-open URL, and the "view stack interface at" hint.

Fix: reuse stack.LocalIngressURL(cfg) in all three places.

Test plan

go test ./internal/hermes/... ./cmd/obol/... passes
go build ./... clean
Verified end-to-end on Linux: fresh obol agent init lands the agent in Running without manual chown
Verified end-to-end on Linux: obol bootstrap with port 8080 occupied completes without hang; all four URL surfaces (warning / visit / browser-open / next-steps) agree on the alternate port
Smoke-test on macOS to confirm no regression (existing fsGroup path still works; new init container is a no-op on already-correct ownership)

…eps URL

bussyjd

Validated against latest origin/main in an isolated worktree.

Checks run locally:

git merge --no-commit --no-ff refs/remotes/pr/446/head onto origin/main
git diff --check --cached
go test ./internal/hermes ./internal/stack ./cmd/obol
go build ./cmd/obol
go test ./...

All passed. The Hermes PVC ownership fix and bootstrap LocalIngressURL wiring look legitimate.

The in-pod `init-hermes-perms` init container from #446 (c066baa) is neutered on Linux k3d because the embedded k3d config sets `KubeletInUserNamespace=true` (internal/embed/k3d-config.yaml). With user-namespacing, the pod's "root" maps to a host subuid that lacks chown authority over the host bind-mount path, so the in-pod `chown -R 10000:10000 /data` silently no-ops. The next init container (`init-hermes-data`) then fails with `mkdir /data/.hermes/home: Permission denied` and the pod CrashLoopBackOffs. local-path-provisioner's helper-pod sets the volume to 1000:1000 (internal/embed/infrastructure/base/templates/local-path.yaml), which happens to suit OpenClaw but not Hermes (containerUID = 10000). Fix: after `helmfile sync`, host-side chown the PVC backing dirs to containerUID:containerGID by exec-ing into the k3d server container via `docker exec`. That runs at the Docker daemon's real root and is not subject to the user-namespacing that silently breaks the in-pod attempt. The existing `fixRuntimeVolumeOwnership` helper already does exactly this for wallet keystore paths; we wire it into the agent deploy path via a new `ensureHermesPVCOwnership` that: 1. Waits up to 60s for each PVC (`hermes-data`, `remote-signer- keystores`) to be Bound — local-path is WaitForFirstConsumer so the host dir doesn't exist until the pod is scheduled. 2. Chowns each backing dir. 3. If a Hermes pod is currently stuck in Init:CrashLoopBackOff, deletes it so kubelet recreates immediately rather than after exponential backoff (~5 min worst case). Skips the delete when no pod is stuck so routine syncs (e.g. `obol model sync` after `obol model prefer`) do not gratuitously restart a healthy agent. Called from `hermes.Sync` after `helmfile sync` succeeds, so every Onboard / Setup / Sync call exercises it. Validated on spark2 (Linux ARM64, Ubuntu 24.04, NVIDIA GB10) by reverting the PVC dirs to 1000:1000, deleting the Hermes pod, and running `obol model sync`. Before: pod stuck in Init:CrashLoopBackOff with "Permission denied" in init-hermes-data logs. After: PVC dirs flip to 10000:10000 within the sync, pod reaches `Running 2/2` in ~30s with zero CrashLoopBackOff cycles. Test: - `TestHermesPVCPaths` pins the two host paths the helper chowns, so renaming `hermes-data`/`remote-signer-keystores` or relocating the namespace prefix can't silently regress the fix. - Full chown side-effect needs a live k3d cluster and is exercised by the spark2 validation above plus all existing integration flows; no unit-mocked k3d test added.

nickh-obol · 2026-05-07T20:27:04Z

Summary

Two Linux-specific bugs in fresh obol stack up flows that don't surface on macOS.

1. Hermes pod CrashLoopBackOff on fresh PVC (`internal/hermes/hermes.go`)

The Stack's local-path-provisioner setup script chowns each new PV to 1000:1000, and KubeletInUserNamespace=true (set in internal/embed/k3d-config.yaml) silently skips the fsGroup recursive-chown that would otherwise correct it. Hermes pods run as UID 10000 and crashloop trying to mkdir /data/.hermes/home with Permission denied.

macOS Docker Desktop hides this — its bind-mount filesystem driver fakes file ownership to whoever's asking, so the mismatch never materializes. OpenClaw doesn't hit it either: it pins fsGroup: 1000, accidentally matching the host UID.

Fix: prepend an init-hermes-perms container that runs as root and chowns /data to 10000:10000. Idempotent — also self-heals existing broken PVCs on upgrade.

2. Bootstrap probe + browser URL hardcoded to `:8080` (`cmd/obol/bootstrap.go`)

When ports 80/8080 are in use at start time, k3d remaps the loadbalancer to a random high port and the rest of obol stack up correctly surfaces it via stack.LocalIngressURL(cfg). Three other places in cmd/obol/bootstrap.go were hardcoded to :8080: the readiness probe (hung until timeout), the browser-open URL, and the "view stack interface at" hint.

Fix: reuse stack.LocalIngressURL(cfg) in all three places.

Test plan

go test ./internal/hermes/... ./cmd/obol/... passes
go build ./... clean
Verified end-to-end on Linux: fresh obol agent init lands the agent in Running without manual chown
Verified end-to-end on Linux: obol bootstrap with port 8080 occupied completes without hang; all four URL surfaces (warning / visit / browser-open / next-steps) agree on the alternate port
Smoke-test on macOS to confirm no regression (existing fsGroup path still works; new init container is a no-op on already-correct ownership)

…eps URL

bussyjd

Validated against latest origin/main in an isolated worktree.

Checks run locally:

git merge --no-commit --no-ff refs/remotes/pr/446/head onto origin/main
git diff --check --cached
go test ./internal/hermes ./internal/stack ./cmd/obol
go build ./cmd/obol
go test ./...

All passed. The Hermes PVC ownership fix and bootstrap LocalIngressURL wiring look legitimate.

The in-pod `init-hermes-perms` init container from #446 (c066baa) is neutered on Linux k3d because the embedded k3d config sets `KubeletInUserNamespace=true` (internal/embed/k3d-config.yaml). With user-namespacing, the pod's "root" maps to a host subuid that lacks chown authority over the host bind-mount path, so the in-pod `chown -R 10000:10000 /data` silently no-ops. The next init container (`init-hermes-data`) then fails with `mkdir /data/.hermes/home: Permission denied` and the pod CrashLoopBackOffs. local-path-provisioner's helper-pod sets the volume to 1000:1000 (internal/embed/infrastructure/base/templates/local-path.yaml), which happens to suit OpenClaw but not Hermes (containerUID = 10000). Fix: after `helmfile sync`, host-side chown the PVC backing dirs to containerUID:containerGID by exec-ing into the k3d server container via `docker exec`. That runs at the Docker daemon's real root and is not subject to the user-namespacing that silently breaks the in-pod attempt. The existing `fixRuntimeVolumeOwnership` helper already does exactly this for wallet keystore paths; we wire it into the agent deploy path via a new `ensureHermesPVCOwnership` that: 1. Waits up to 60s for each PVC (`hermes-data`, `remote-signer- keystores`) to be Bound — local-path is WaitForFirstConsumer so the host dir doesn't exist until the pod is scheduled. 2. Chowns each backing dir. 3. If a Hermes pod is currently stuck in Init:CrashLoopBackOff, deletes it so kubelet recreates immediately rather than after exponential backoff (~5 min worst case). Skips the delete when no pod is stuck so routine syncs (e.g. `obol model sync` after `obol model prefer`) do not gratuitously restart a healthy agent. Called from `hermes.Sync` after `helmfile sync` succeeds, so every Onboard / Setup / Sync call exercises it. Validated on spark2 (Linux ARM64, Ubuntu 24.04, NVIDIA GB10) by reverting the PVC dirs to 1000:1000, deleting the Hermes pod, and running `obol model sync`. Before: pod stuck in Init:CrashLoopBackOff with "Permission denied" in init-hermes-data logs. After: PVC dirs flip to 10000:10000 within the sync, pod reaches `Running 2/2` in ~30s with zero CrashLoopBackOff cycles. Test: - `TestHermesPVCPaths` pins the two host paths the helper chowns, so renaming `hermes-data`/`remote-signer-keystores` or relocating the namespace prefix can't silently regress the fix. - Full chown side-effect needs a live k3d cluster and is exercised by the spark2 validation above plus all existing integration flows; no unit-mocked k3d test added.

nickh-obol · 2026-05-07T20:27:04Z

Summary

Two Linux-specific bugs in fresh obol stack up flows that don't surface on macOS.

1. Hermes pod CrashLoopBackOff on fresh PVC (`internal/hermes/hermes.go`)

The Stack's local-path-provisioner setup script chowns each new PV to 1000:1000, and KubeletInUserNamespace=true (set in internal/embed/k3d-config.yaml) silently skips the fsGroup recursive-chown that would otherwise correct it. Hermes pods run as UID 10000 and crashloop trying to mkdir /data/.hermes/home with Permission denied.

macOS Docker Desktop hides this — its bind-mount filesystem driver fakes file ownership to whoever's asking, so the mismatch never materializes. OpenClaw doesn't hit it either: it pins fsGroup: 1000, accidentally matching the host UID.

Fix: prepend an init-hermes-perms container that runs as root and chowns /data to 10000:10000. Idempotent — also self-heals existing broken PVCs on upgrade.

2. Bootstrap probe + browser URL hardcoded to `:8080` (`cmd/obol/bootstrap.go`)

When ports 80/8080 are in use at start time, k3d remaps the loadbalancer to a random high port and the rest of obol stack up correctly surfaces it via stack.LocalIngressURL(cfg). Three other places in cmd/obol/bootstrap.go were hardcoded to :8080: the readiness probe (hung until timeout), the browser-open URL, and the "view stack interface at" hint.

Fix: reuse stack.LocalIngressURL(cfg) in all three places.

Test plan

go test ./internal/hermes/... ./cmd/obol/... passes
go build ./... clean
Verified end-to-end on Linux: fresh obol agent init lands the agent in Running without manual chown
Verified end-to-end on Linux: obol bootstrap with port 8080 occupied completes without hang; all four URL surfaces (warning / visit / browser-open / next-steps) agree on the alternate port
Smoke-test on macOS to confirm no regression (existing fsGroup path still works; new init container is a no-op on already-correct ownership)

…eps URL

bussyjd

Validated against latest origin/main in an isolated worktree.

Checks run locally:

git merge --no-commit --no-ff refs/remotes/pr/446/head onto origin/main
git diff --check --cached
go test ./internal/hermes ./internal/stack ./cmd/obol
go build ./cmd/obol
go test ./...

All passed. The Hermes PVC ownership fix and bootstrap LocalIngressURL wiring look legitimate.

The in-pod `init-hermes-perms` init container from #446 (c066baa) is neutered on Linux k3d because the embedded k3d config sets `KubeletInUserNamespace=true` (internal/embed/k3d-config.yaml). With user-namespacing, the pod's "root" maps to a host subuid that lacks chown authority over the host bind-mount path, so the in-pod `chown -R 10000:10000 /data` silently no-ops. The next init container (`init-hermes-data`) then fails with `mkdir /data/.hermes/home: Permission denied` and the pod CrashLoopBackOffs. local-path-provisioner's helper-pod sets the volume to 1000:1000 (internal/embed/infrastructure/base/templates/local-path.yaml), which happens to suit OpenClaw but not Hermes (containerUID = 10000). Fix: after `helmfile sync`, host-side chown the PVC backing dirs to containerUID:containerGID by exec-ing into the k3d server container via `docker exec`. That runs at the Docker daemon's real root and is not subject to the user-namespacing that silently breaks the in-pod attempt. The existing `fixRuntimeVolumeOwnership` helper already does exactly this for wallet keystore paths; we wire it into the agent deploy path via a new `ensureHermesPVCOwnership` that: 1. Waits up to 60s for each PVC (`hermes-data`, `remote-signer- keystores`) to be Bound — local-path is WaitForFirstConsumer so the host dir doesn't exist until the pod is scheduled. 2. Chowns each backing dir. 3. If a Hermes pod is currently stuck in Init:CrashLoopBackOff, deletes it so kubelet recreates immediately rather than after exponential backoff (~5 min worst case). Skips the delete when no pod is stuck so routine syncs (e.g. `obol model sync` after `obol model prefer`) do not gratuitously restart a healthy agent. Called from `hermes.Sync` after `helmfile sync` succeeds, so every Onboard / Setup / Sync call exercises it. Validated on spark2 (Linux ARM64, Ubuntu 24.04, NVIDIA GB10) by reverting the PVC dirs to 1000:1000, deleting the Hermes pod, and running `obol model sync`. Before: pod stuck in Init:CrashLoopBackOff with "Permission denied" in init-hermes-data logs. After: PVC dirs flip to 10000:10000 within the sync, pod reaches `Running 2/2` in ~30s with zero CrashLoopBackOff cycles. Test: - `TestHermesPVCPaths` pins the two host paths the helper chowns, so renaming `hermes-data`/`remote-signer-keystores` or relocating the namespace prefix can't silently regress the fix. - Full chown side-effect needs a live k3d cluster and is exercised by the spark2 validation above plus all existing integration flows; no unit-mocked k3d test added.

nickh-obol · 2026-05-07T20:27:04Z

Summary

Two Linux-specific bugs in fresh obol stack up flows that don't surface on macOS.

1. Hermes pod CrashLoopBackOff on fresh PVC (`internal/hermes/hermes.go`)

The Stack's local-path-provisioner setup script chowns each new PV to 1000:1000, and KubeletInUserNamespace=true (set in internal/embed/k3d-config.yaml) silently skips the fsGroup recursive-chown that would otherwise correct it. Hermes pods run as UID 10000 and crashloop trying to mkdir /data/.hermes/home with Permission denied.

macOS Docker Desktop hides this — its bind-mount filesystem driver fakes file ownership to whoever's asking, so the mismatch never materializes. OpenClaw doesn't hit it either: it pins fsGroup: 1000, accidentally matching the host UID.

Fix: prepend an init-hermes-perms container that runs as root and chowns /data to 10000:10000. Idempotent — also self-heals existing broken PVCs on upgrade.

2. Bootstrap probe + browser URL hardcoded to `:8080` (`cmd/obol/bootstrap.go`)

When ports 80/8080 are in use at start time, k3d remaps the loadbalancer to a random high port and the rest of obol stack up correctly surfaces it via stack.LocalIngressURL(cfg). Three other places in cmd/obol/bootstrap.go were hardcoded to :8080: the readiness probe (hung until timeout), the browser-open URL, and the "view stack interface at" hint.

Fix: reuse stack.LocalIngressURL(cfg) in all three places.

Test plan

go test ./internal/hermes/... ./cmd/obol/... passes
go build ./... clean
Verified end-to-end on Linux: fresh obol agent init lands the agent in Running without manual chown
Verified end-to-end on Linux: obol bootstrap with port 8080 occupied completes without hang; all four URL surfaces (warning / visit / browser-open / next-steps) agree on the alternate port
Smoke-test on macOS to confirm no regression (existing fsGroup path still works; new init container is a no-op on already-correct ownership)

…eps URL

bussyjd

Validated against latest origin/main in an isolated worktree.

Checks run locally:

git merge --no-commit --no-ff refs/remotes/pr/446/head onto origin/main
git diff --check --cached
go test ./internal/hermes ./internal/stack ./cmd/obol
go build ./cmd/obol
go test ./...

All passed. The Hermes PVC ownership fix and bootstrap LocalIngressURL wiring look legitimate.

The in-pod `init-hermes-perms` init container from #446 (c066baa) is neutered on Linux k3d because the embedded k3d config sets `KubeletInUserNamespace=true` (internal/embed/k3d-config.yaml). With user-namespacing, the pod's "root" maps to a host subuid that lacks chown authority over the host bind-mount path, so the in-pod `chown -R 10000:10000 /data` silently no-ops. The next init container (`init-hermes-data`) then fails with `mkdir /data/.hermes/home: Permission denied` and the pod CrashLoopBackOffs. local-path-provisioner's helper-pod sets the volume to 1000:1000 (internal/embed/infrastructure/base/templates/local-path.yaml), which happens to suit OpenClaw but not Hermes (containerUID = 10000). Fix: after `helmfile sync`, host-side chown the PVC backing dirs to containerUID:containerGID by exec-ing into the k3d server container via `docker exec`. That runs at the Docker daemon's real root and is not subject to the user-namespacing that silently breaks the in-pod attempt. The existing `fixRuntimeVolumeOwnership` helper already does exactly this for wallet keystore paths; we wire it into the agent deploy path via a new `ensureHermesPVCOwnership` that: 1. Waits up to 60s for each PVC (`hermes-data`, `remote-signer- keystores`) to be Bound — local-path is WaitForFirstConsumer so the host dir doesn't exist until the pod is scheduled. 2. Chowns each backing dir. 3. If a Hermes pod is currently stuck in Init:CrashLoopBackOff, deletes it so kubelet recreates immediately rather than after exponential backoff (~5 min worst case). Skips the delete when no pod is stuck so routine syncs (e.g. `obol model sync` after `obol model prefer`) do not gratuitously restart a healthy agent. Called from `hermes.Sync` after `helmfile sync` succeeds, so every Onboard / Setup / Sync call exercises it. Validated on spark2 (Linux ARM64, Ubuntu 24.04, NVIDIA GB10) by reverting the PVC dirs to 1000:1000, deleting the Hermes pod, and running `obol model sync`. Before: pod stuck in Init:CrashLoopBackOff with "Permission denied" in init-hermes-data logs. After: PVC dirs flip to 10000:10000 within the sync, pod reaches `Running 2/2` in ~30s with zero CrashLoopBackOff cycles. Test: - `TestHermesPVCPaths` pins the two host paths the helper chowns, so renaming `hermes-data`/`remote-signer-keystores` or relocating the namespace prefix can't silently regress the fix. - Full chown side-effect needs a live k3d cluster and is exercised by the spark2 validation above plus all existing integration flows; no unit-mocked k3d test added.

nickh-obol · 2026-05-07T20:27:04Z

Summary

Two Linux-specific bugs in fresh obol stack up flows that don't surface on macOS.

1. Hermes pod CrashLoopBackOff on fresh PVC (`internal/hermes/hermes.go`)

The Stack's local-path-provisioner setup script chowns each new PV to 1000:1000, and KubeletInUserNamespace=true (set in internal/embed/k3d-config.yaml) silently skips the fsGroup recursive-chown that would otherwise correct it. Hermes pods run as UID 10000 and crashloop trying to mkdir /data/.hermes/home with Permission denied.

macOS Docker Desktop hides this — its bind-mount filesystem driver fakes file ownership to whoever's asking, so the mismatch never materializes. OpenClaw doesn't hit it either: it pins fsGroup: 1000, accidentally matching the host UID.

Fix: prepend an init-hermes-perms container that runs as root and chowns /data to 10000:10000. Idempotent — also self-heals existing broken PVCs on upgrade.

2. Bootstrap probe + browser URL hardcoded to `:8080` (`cmd/obol/bootstrap.go`)

When ports 80/8080 are in use at start time, k3d remaps the loadbalancer to a random high port and the rest of obol stack up correctly surfaces it via stack.LocalIngressURL(cfg). Three other places in cmd/obol/bootstrap.go were hardcoded to :8080: the readiness probe (hung until timeout), the browser-open URL, and the "view stack interface at" hint.

Fix: reuse stack.LocalIngressURL(cfg) in all three places.

Test plan

go test ./internal/hermes/... ./cmd/obol/... passes
go build ./... clean
Verified end-to-end on Linux: fresh obol agent init lands the agent in Running without manual chown
Verified end-to-end on Linux: obol bootstrap with port 8080 occupied completes without hang; all four URL surfaces (warning / visit / browser-open / next-steps) agree on the alternate port
Smoke-test on macOS to confirm no regression (existing fsGroup path still works; new init container is a no-op on already-correct ownership)

…eps URL

bussyjd

Validated against latest origin/main in an isolated worktree.

Checks run locally:

git merge --no-commit --no-ff refs/remotes/pr/446/head onto origin/main
git diff --check --cached
go test ./internal/hermes ./internal/stack ./cmd/obol
go build ./cmd/obol
go test ./...

All passed. The Hermes PVC ownership fix and bootstrap LocalIngressURL wiring look legitimate.

The in-pod `init-hermes-perms` init container from #446 (c066baa) is neutered on Linux k3d because the embedded k3d config sets `KubeletInUserNamespace=true` (internal/embed/k3d-config.yaml). With user-namespacing, the pod's "root" maps to a host subuid that lacks chown authority over the host bind-mount path, so the in-pod `chown -R 10000:10000 /data` silently no-ops. The next init container (`init-hermes-data`) then fails with `mkdir /data/.hermes/home: Permission denied` and the pod CrashLoopBackOffs. local-path-provisioner's helper-pod sets the volume to 1000:1000 (internal/embed/infrastructure/base/templates/local-path.yaml), which happens to suit OpenClaw but not Hermes (containerUID = 10000). Fix: after `helmfile sync`, host-side chown the PVC backing dirs to containerUID:containerGID by exec-ing into the k3d server container via `docker exec`. That runs at the Docker daemon's real root and is not subject to the user-namespacing that silently breaks the in-pod attempt. The existing `fixRuntimeVolumeOwnership` helper already does exactly this for wallet keystore paths; we wire it into the agent deploy path via a new `ensureHermesPVCOwnership` that: 1. Waits up to 60s for each PVC (`hermes-data`, `remote-signer- keystores`) to be Bound — local-path is WaitForFirstConsumer so the host dir doesn't exist until the pod is scheduled. 2. Chowns each backing dir. 3. If a Hermes pod is currently stuck in Init:CrashLoopBackOff, deletes it so kubelet recreates immediately rather than after exponential backoff (~5 min worst case). Skips the delete when no pod is stuck so routine syncs (e.g. `obol model sync` after `obol model prefer`) do not gratuitously restart a healthy agent. Called from `hermes.Sync` after `helmfile sync` succeeds, so every Onboard / Setup / Sync call exercises it. Validated on spark2 (Linux ARM64, Ubuntu 24.04, NVIDIA GB10) by reverting the PVC dirs to 1000:1000, deleting the Hermes pod, and running `obol model sync`. Before: pod stuck in Init:CrashLoopBackOff with "Permission denied" in init-hermes-data logs. After: PVC dirs flip to 10000:10000 within the sync, pod reaches `Running 2/2` in ~30s with zero CrashLoopBackOff cycles. Test: - `TestHermesPVCPaths` pins the two host paths the helper chowns, so renaming `hermes-data`/`remote-signer-keystores` or relocating the namespace prefix can't silently regress the fix. - Full chown side-effect needs a live k3d cluster and is exercised by the spark2 validation above plus all existing integration flows; no unit-mocked k3d test added.

nickh-obol · 2026-05-07T20:27:04Z

Summary

Two Linux-specific bugs in fresh obol stack up flows that don't surface on macOS.

1. Hermes pod CrashLoopBackOff on fresh PVC (`internal/hermes/hermes.go`)

The Stack's local-path-provisioner setup script chowns each new PV to 1000:1000, and KubeletInUserNamespace=true (set in internal/embed/k3d-config.yaml) silently skips the fsGroup recursive-chown that would otherwise correct it. Hermes pods run as UID 10000 and crashloop trying to mkdir /data/.hermes/home with Permission denied.

macOS Docker Desktop hides this — its bind-mount filesystem driver fakes file ownership to whoever's asking, so the mismatch never materializes. OpenClaw doesn't hit it either: it pins fsGroup: 1000, accidentally matching the host UID.

Fix: prepend an init-hermes-perms container that runs as root and chowns /data to 10000:10000. Idempotent — also self-heals existing broken PVCs on upgrade.

2. Bootstrap probe + browser URL hardcoded to `:8080` (`cmd/obol/bootstrap.go`)

When ports 80/8080 are in use at start time, k3d remaps the loadbalancer to a random high port and the rest of obol stack up correctly surfaces it via stack.LocalIngressURL(cfg). Three other places in cmd/obol/bootstrap.go were hardcoded to :8080: the readiness probe (hung until timeout), the browser-open URL, and the "view stack interface at" hint.

Fix: reuse stack.LocalIngressURL(cfg) in all three places.

Test plan

go test ./internal/hermes/... ./cmd/obol/... passes
go build ./... clean
Verified end-to-end on Linux: fresh obol agent init lands the agent in Running without manual chown
Verified end-to-end on Linux: obol bootstrap with port 8080 occupied completes without hang; all four URL surfaces (warning / visit / browser-open / next-steps) agree on the alternate port
Smoke-test on macOS to confirm no regression (existing fsGroup path still works; new init container is a no-op on already-correct ownership)

…eps URL

bussyjd

Validated against latest origin/main in an isolated worktree.

Checks run locally:

git merge --no-commit --no-ff refs/remotes/pr/446/head onto origin/main
git diff --check --cached
go test ./internal/hermes ./internal/stack ./cmd/obol
go build ./cmd/obol
go test ./...

All passed. The Hermes PVC ownership fix and bootstrap LocalIngressURL wiring look legitimate.

The in-pod `init-hermes-perms` init container from #446 (c066baa) is neutered on Linux k3d because the embedded k3d config sets `KubeletInUserNamespace=true` (internal/embed/k3d-config.yaml). With user-namespacing, the pod's "root" maps to a host subuid that lacks chown authority over the host bind-mount path, so the in-pod `chown -R 10000:10000 /data` silently no-ops. The next init container (`init-hermes-data`) then fails with `mkdir /data/.hermes/home: Permission denied` and the pod CrashLoopBackOffs. local-path-provisioner's helper-pod sets the volume to 1000:1000 (internal/embed/infrastructure/base/templates/local-path.yaml), which happens to suit OpenClaw but not Hermes (containerUID = 10000). Fix: after `helmfile sync`, host-side chown the PVC backing dirs to containerUID:containerGID by exec-ing into the k3d server container via `docker exec`. That runs at the Docker daemon's real root and is not subject to the user-namespacing that silently breaks the in-pod attempt. The existing `fixRuntimeVolumeOwnership` helper already does exactly this for wallet keystore paths; we wire it into the agent deploy path via a new `ensureHermesPVCOwnership` that: 1. Waits up to 60s for each PVC (`hermes-data`, `remote-signer- keystores`) to be Bound — local-path is WaitForFirstConsumer so the host dir doesn't exist until the pod is scheduled. 2. Chowns each backing dir. 3. If a Hermes pod is currently stuck in Init:CrashLoopBackOff, deletes it so kubelet recreates immediately rather than after exponential backoff (~5 min worst case). Skips the delete when no pod is stuck so routine syncs (e.g. `obol model sync` after `obol model prefer`) do not gratuitously restart a healthy agent. Called from `hermes.Sync` after `helmfile sync` succeeds, so every Onboard / Setup / Sync call exercises it. Validated on spark2 (Linux ARM64, Ubuntu 24.04, NVIDIA GB10) by reverting the PVC dirs to 1000:1000, deleting the Hermes pod, and running `obol model sync`. Before: pod stuck in Init:CrashLoopBackOff with "Permission denied" in init-hermes-data logs. After: PVC dirs flip to 10000:10000 within the sync, pod reaches `Running 2/2` in ~30s with zero CrashLoopBackOff cycles. Test: - `TestHermesPVCPaths` pins the two host paths the helper chowns, so renaming `hermes-data`/`remote-signer-keystores` or relocating the namespace prefix can't silently regress the fix. - Full chown side-effect needs a live k3d cluster and is exercised by the spark2 validation above plus all existing integration flows; no unit-mocked k3d test added.

nickh-obol · 2026-05-07T20:27:04Z

Summary

Two Linux-specific bugs in fresh obol stack up flows that don't surface on macOS.

1. Hermes pod CrashLoopBackOff on fresh PVC (`internal/hermes/hermes.go`)

The Stack's local-path-provisioner setup script chowns each new PV to 1000:1000, and KubeletInUserNamespace=true (set in internal/embed/k3d-config.yaml) silently skips the fsGroup recursive-chown that would otherwise correct it. Hermes pods run as UID 10000 and crashloop trying to mkdir /data/.hermes/home with Permission denied.

macOS Docker Desktop hides this — its bind-mount filesystem driver fakes file ownership to whoever's asking, so the mismatch never materializes. OpenClaw doesn't hit it either: it pins fsGroup: 1000, accidentally matching the host UID.

Fix: prepend an init-hermes-perms container that runs as root and chowns /data to 10000:10000. Idempotent — also self-heals existing broken PVCs on upgrade.

2. Bootstrap probe + browser URL hardcoded to `:8080` (`cmd/obol/bootstrap.go`)

When ports 80/8080 are in use at start time, k3d remaps the loadbalancer to a random high port and the rest of obol stack up correctly surfaces it via stack.LocalIngressURL(cfg). Three other places in cmd/obol/bootstrap.go were hardcoded to :8080: the readiness probe (hung until timeout), the browser-open URL, and the "view stack interface at" hint.

Fix: reuse stack.LocalIngressURL(cfg) in all three places.

Test plan

go test ./internal/hermes/... ./cmd/obol/... passes
go build ./... clean
Verified end-to-end on Linux: fresh obol agent init lands the agent in Running without manual chown
Verified end-to-end on Linux: obol bootstrap with port 8080 occupied completes without hang; all four URL surfaces (warning / visit / browser-open / next-steps) agree on the alternate port
Smoke-test on macOS to confirm no regression (existing fsGroup path still works; new init container is a no-op on already-correct ownership)

…eps URL

bussyjd

Validated against latest origin/main in an isolated worktree.

Checks run locally:

git merge --no-commit --no-ff refs/remotes/pr/446/head onto origin/main
git diff --check --cached
go test ./internal/hermes ./internal/stack ./cmd/obol
go build ./cmd/obol
go test ./...

All passed. The Hermes PVC ownership fix and bootstrap LocalIngressURL wiring look legitimate.

The in-pod `init-hermes-perms` init container from #446 (c066baa) is neutered on Linux k3d because the embedded k3d config sets `KubeletInUserNamespace=true` (internal/embed/k3d-config.yaml). With user-namespacing, the pod's "root" maps to a host subuid that lacks chown authority over the host bind-mount path, so the in-pod `chown -R 10000:10000 /data` silently no-ops. The next init container (`init-hermes-data`) then fails with `mkdir /data/.hermes/home: Permission denied` and the pod CrashLoopBackOffs. local-path-provisioner's helper-pod sets the volume to 1000:1000 (internal/embed/infrastructure/base/templates/local-path.yaml), which happens to suit OpenClaw but not Hermes (containerUID = 10000). Fix: after `helmfile sync`, host-side chown the PVC backing dirs to containerUID:containerGID by exec-ing into the k3d server container via `docker exec`. That runs at the Docker daemon's real root and is not subject to the user-namespacing that silently breaks the in-pod attempt. The existing `fixRuntimeVolumeOwnership` helper already does exactly this for wallet keystore paths; we wire it into the agent deploy path via a new `ensureHermesPVCOwnership` that: 1. Waits up to 60s for each PVC (`hermes-data`, `remote-signer- keystores`) to be Bound — local-path is WaitForFirstConsumer so the host dir doesn't exist until the pod is scheduled. 2. Chowns each backing dir. 3. If a Hermes pod is currently stuck in Init:CrashLoopBackOff, deletes it so kubelet recreates immediately rather than after exponential backoff (~5 min worst case). Skips the delete when no pod is stuck so routine syncs (e.g. `obol model sync` after `obol model prefer`) do not gratuitously restart a healthy agent. Called from `hermes.Sync` after `helmfile sync` succeeds, so every Onboard / Setup / Sync call exercises it. Validated on spark2 (Linux ARM64, Ubuntu 24.04, NVIDIA GB10) by reverting the PVC dirs to 1000:1000, deleting the Hermes pod, and running `obol model sync`. Before: pod stuck in Init:CrashLoopBackOff with "Permission denied" in init-hermes-data logs. After: PVC dirs flip to 10000:10000 within the sync, pod reaches `Running 2/2` in ~30s with zero CrashLoopBackOff cycles. Test: - `TestHermesPVCPaths` pins the two host paths the helper chowns, so renaming `hermes-data`/`remote-signer-keystores` or relocating the namespace prefix can't silently regress the fix. - Full chown side-effect needs a live k3d cluster and is exercised by the spark2 validation above plus all existing integration flows; no unit-mocked k3d test added.

nickh-obol added 3 commits May 7, 2026 16:02

fix(hermes): chown PVC volume in init container for user-namespaced k3s
7cdb56d

fix(bootstrap): use LocalIngressURL for ingress readiness probe
20d43bd

fix(bootstrap): also use LocalIngressURL for browser-open and next-st…
4eb7ef1
…eps URL

bussyjd approved these changes May 8, 2026
View reviewed changes

Merge branch 'main' into fix/hermes-pvc-perms-and-ingress-port
2ebc614

bussyjd merged commit c066baa into mainMay 8, 2026
6 checks passed

bussyjd deleted the fix/hermes-pvc-perms-and-ingress-port branch May 8, 2026 03:03

This was referenced May 12, 2026
hermes init PVC perm-denied regression survives #446 (k3d local-path on Linux, both fresh and dev-mode) #475
Closed
fix(hermes): host-side chown PVC backing dirs after sync (#475) #481
Merged