fix: harden PVC permissions and paid smoke flows by bussyjd · Pull Request #615 · ObolNetwork/obol-stack

bussyjd · 2026-06-09T21:06:25Z

Summary

replace host-side chown/chmod init handling with UID/GID 1000 runtime ownership for Hermes and spawned agents
keep LiteLLM/x402 buyer routing in sync when model ConfigMaps change so paid routes roll correctly
harden release smoke flows around qwen36-deep, post-stack Bob wallet seeding, PurchaseRequest readiness, and paid inference settlement checks
update the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces

Validation

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
full release smoke passed with OBOL_LLM_MODEL=qwen36-deep, including USDC, live OBOL, and fork OBOL paid inference flows

Notes

This is stacked on #610's branch so reviewers can inspect only the follow-up fixes here.

bussyjd · 2026-06-09T21:57:29Z

Summary

What changed:

Replaced host-side chown/chmod init handling with UID/GID/fsGroup 1000 runtime ownership for Hermes and spawned agent workloads.
Ensured LiteLLM rolls when model or buyer ConfigMaps change, so paid routes are refreshed when the config changes without an env var change.
Hardened paid smoke flows around qwen36-deep, inference ServiceOffers, PurchaseRequest auth counts, LiteLLM readiness, and exact settlement deltas.
Moved Bob wallet setup in dual-stack flows to post-stack obol wallet import, then asserts the signer matches deterministic Bob.
Updated the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces.

Why it matters:

Solves the PVC permission problem without making config read-only and without host-side ownership mutation.
Reduces fragile permission scaffolding while keeping runtime writes available.
Keeps release-smoke aligned with the supported obol CLI, stack, and obol kubectl surfaces.

Risk level: medium

Commit under test: 1f3088b

Base branch: research/hermes-configmap-skills-delivery

Scope

Validation

CI checks:

Check	Status	Link
CI	not checked in this comment

Unit tests:

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
Result: PASS
Commit/SHA: 1f3088b

Integration tests:

No separate integration suite was run outside the release smoke.

Flow tests:

Flow	Network	QA machine label	Worktree	Result	Artifacts
`flow-01-prerequisites`	local	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-02-stack-init-up`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-03-inference`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-04-agent`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-05-network`	Base Sepolia RPC	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-06-sell-setup`	Base Sepolia x402 pricing	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-07-sell-verify`	x402 tunnel route	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-10-anvil-facilitator`	Base Sepolia Anvil fork	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-08-buy`	Base Sepolia Anvil fork / USDC	remote ARM64 QA host	redacted	PASS	receipt retained on QA host
`flow-09-lifecycle`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-11-dual-stack`	Base Sepolia / USDC	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-14-live-obol-base-sepolia`	live Base Sepolia / OBOL	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-13-dual-stack-obol`	Base Sepolia Anvil fork / OBOL	remote ARM64 QA host	redacted	PASS	receipt summary retained on QA host

Release smoke:

Command: flows/release-smoke.sh
Flags/env: OBOL_DEVELOPMENT=true, OBOL_NONINTERACTIVE=true, OBOL_FORCE_REBUILD_LOCAL_DEV_IMAGES=serviceoffer-controller, OBOL_LLM_MODEL=qwen36-deep, RELEASE_SMOKE_INCLUDE_OBOL=true, RELEASE_SMOKE_INCLUDE_OBOL_FORK=true
Endpoint/RPC/worktree details: redacted
Result: PASS

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

Base Sepolia
Base Sepolia Anvil fork

RPC/provider:

Paid Base Sepolia RPC, redacted
Local Anvil fork for forked flows

Facilitator:

Public Obol facilitator for live Base Sepolia flows, hostname omitted
Local x402-rs facilitator for fork flow

Contracts and tokens:

Name	Address	Version / notes
Base Sepolia OBOL	`0x0a09371a8b011d5110656ceBCc70603e53FD2c78`	live token
Base Sepolia USDC	`0x036CbD53842c5426634e7929541eC2318f3dCF7e`	live/fork token
ERC-8004 identity registry	`0x8004A818BFB912233c491871b3d84c89A494BD9e`	Base Sepolia
Fork OBOL	`0x210BBd033630e5e611B7922D70b0Caabe64636d9`	flow-13 deployed token

Wallet roles:

Role	Address	Source
Alice / seller / register	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	runtime-derived from configured signer
Bob / buyer / payer	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	deterministic second-derived buyer
Single-stack flow-08 seller	`0x70997970C51812dc3A010C7d01b50e0d17dc79C8`	local flow seller

Balances:

Token	Address	Before	After	Expected delta	Actual delta
Live OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`30000000000000000`	`31000000000000000`	`+1000000000000000`	`+1000000000000000`
Live OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`4947000000000000000`	`4946000000000000000`	`-1000000000000000`	`-1000000000000000`
Fork OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`10000000000000000000`	`10001000000000000000`	`+1000000000000000`	`+1000000000000000`
Fork OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`10000000000000000000`	`9999000000000000000`	`-1000000000000000`	`-1000000000000000`

Transaction receipts:

Purpose	Tx hash	From	To	Amount / event	Status
ERC-8004 registration	`0x8bc7bf9c5daaaa3c3df889ce7ee44f483593e0b79fd9126a34838314280e1310`	Alice	registry	Agent ID reflected in ServiceOffer	PASS
Metadata / service offer				Metadata tx unavailable, ServiceOffer status reflected Agent ID	SKIP
Live OBOL approval / permit	`0x51841c0cfcfd1b5e0b23d2976a755f8ed7d11489015e885a962bf3b1c7cd878c`	Bob	Permit2	Max allowance confirmed	PASS
Live OBOL settlement transfer	`0xb78fa7b9f1097fb88fd6e5fb6d04ffbfc03337f478691b0905eb4d70f0dc2981`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Fork OBOL buyer funding	`0x733a277dd44da427bd3749b955ab4ab77fe9dadbe94358c130afab2ccd4600de`	deployer	Bob	`10` OBOL minted	PASS
Fork OBOL approval / permit	`0x40dc8bc61cb2f4dbd31d532141ef1885f3efb29144ee2fc50f712dd265ffa3c8`	Bob	Permit2	Max allowance confirmed	PASS
Fork OBOL settlement transfer	`0xb0d8974636102f1be97383c4fb1ba8aa860ef7850084ce5d425e37c1016b114d`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Flow-08 USDC settlement transfer	`0x1e7fccd95dfd47e2720896406108f94d391d4f8d5824c37f9d3cdfdc1ca7f9c2`	buyer	seller	`1000` micro-USDC	PASS
Flow-11 USDC settlement transfer	`0x6127dbec3139eadc68b5343a2ccf552fcf82595c4cdeaf81c893d51ee74754df`	Bob	Alice	`1000` micro-USDC	PASS

Runtime Evidence

QA environment:

Item	Value
OS / arch	Linux / arm64 remote QA host
Backend	k3d
Tool versions	branch-built `obol` CLI, Foundry tools available
QA agent/model	`qwen36-deep` via redacted OpenAI-compatible endpoint

Images:

Component	Image	Tag / digest	Source
serviceoffer-controller	local dev image	rebuilt during smoke	branch source
x402 facilitator	`ghcr.io/obolnetwork/x402-facilitator-prometheus-overlay`	`1.4.9`	flow-13 preflight

Kubernetes / stack:

Item	Value
Stack IDs	omitted to avoid host-local details
Namespaces	`llm`, `x402`, `erpc`, `hermes-obol-agent`
Pod readiness	all smoke-gated pods Ready during flow checks
Cleanup result	no k3d clusters remaining after final smoke

Model and routing:

Item	Value
Agent/model used	`qwen36-deep`
LiteLLM route	`paid/qwen36-deep`
Paid endpoint status	HTTP 200 with non-empty response
Auth token source	`PurchaseRequest` plus x402-buyer sidecar, no raw token logged

Artifacts and logs:

Artifact	Location / link	Notes
Release smoke report	retained on QA host, path omitted	result PASS
Flow-11 receipts	retained on QA host, path omitted	USDC settlement
Flow-14 receipts	retained on QA host, path omitted	live OBOL settlement
Flow-13 receipts	retained on QA host, path omitted	fork OBOL settlement

Demo readiness:

Item	Status	Notes
Seller visible / registered	PASS	ServiceOffer Ready; Agent ID reflected where registration was enabled
Buyer discovery works	PASS	Buyer discovered service catalog / registry data
Paid route works	PASS	`paid/qwen36-deep` returned HTTP 200
Settlement visible on-chain	PASS	Transfer receipts and exact deltas verified

Review Notes

Known gaps:

flow-11 and flow-14 recorded one SKIP each for unavailable registration receipt while ServiceOffer status already reflected the Agent ID.
PR is stacked on research/hermes-configmap-skills-delivery, so reviewers should compare against that branch for the focused delta.

Follow-ups:

None required for the validated smoke path.

Reviewer focus:

UID/GID/fsGroup 1000 ownership model versus the removed host-side chown/chmod path.
LiteLLM ConfigMap reload behavior for paid routes.
Flow-08 retry/readiness behavior around obol buy inference and LiteLLM rollout.
Post-stack Bob wallet import in dual-stack flows.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

…with #614 Applies the verified findings from the cross-review against PR #614 (every item adversarially confirmed against the refs; union merge-tree clean). PVC / upgrade path: - llm.yaml: restore container-level runAsUser/runAsGroup 1000 on x402-buyer. Clusters upgraded in place from <= rc12 keep hostPath-typed PVs where kubelet skips fsGroup; their /state dir is 1000:1000 with consumed.json written 0600 by UID 1000 — a 65532 sidecar cannot read it, Fatalf's on `load state`, and takes every paid/<model> route down. On fresh local-type PVs the explicit UID is harmless (fsGroup 65532 grants group access). embed_buyer_state_test.go updated to pin the new contract. - plans/volume-permission-hardening.md: new "Upgrading from <= v0.10.0-rc12" section — supported path is cluster recreation (wallet backup/restore), with a documented k3d chown escape hatch. troubleshooting.md gets the symptom->fix entry. The Hermes half of the legacy-PV breakage cannot be patched at runtime without reintroducing the chown machinery this PR removes, so it is a documented breaking change instead. Paid-route availability: - llm.yaml: Reloader annotation narrowed to litellm-config only. The buyer ConfigMaps (x402-buyer-config/x402-buyer-auths) are rewritten by the controller on every buy, top-up, auto-refill, and tombstone cleanup; with strategy Recreate + 1 replica the previous annotation bounced the entire inference gateway (all Hermes traffic, in-flight SSE streams) on every purchase event, inverting CLAUDE.md pitfall 7 (restart is fallback, not the default buy path). The buyer hot-reloads via /admin/reload. stack_test.go updated to pin litellm-config-only. Flow alignment with #614: - lib.sh: `stack down` -> `stack down --yes` in reset_flow_workspace. #614's flow-16 (now last in the single-stack array) intentionally leaves a live agent offer; without --yes the non-TTY ConfirmRunningServicesLoss gate refuses, graceful down is silently skipped (`|| true`), and teardown degrades to the raw k3d-delete fallback on every release-smoke run. - flow-11: post-register Ready poll 120s -> 300s to match flow-14's identical live-Base-Sepolia chain-watch path (pitfall 13 free-tier RPC throttling). Known follow-ups (not in this commit): flow-08 buy-retry top-up vs exactly-N assertions on rare partial failures; flow-11 lacks flow-14's remote-signer rolled guard; aztec PVC has no permission story (runs as root today); post-merge controller repin so released sub-agents pick up this PR's UID-1000 render (tracked in #614's pin-test note).

Records the options ladder for 'nodes joining my cluster' (far-future, not scheduled) so near-term permission decisions don't foreclose it: the host-FS-canonical coupling and its four conditions, the agent-homes-only scope, what breaks on day one of a join (WaitForFirstConsumer + no nodeSelector in any render), options 0-4 (home-node pattern -> NFS -> distributed storage -> API-mediated access -> hybrid by data class), join mechanics (native k3s server + remote agents; macOS stays k3d), and the decisions that bind today: group-1000 sharing over render-time UID matching, home-node pinning as a join prerequisite, inputs to API objects per Remaining Debt.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

bussyjd · 2026-06-09T21:06:25Z

Summary

replace host-side chown/chmod init handling with UID/GID 1000 runtime ownership for Hermes and spawned agents
keep LiteLLM/x402 buyer routing in sync when model ConfigMaps change so paid routes roll correctly
harden release smoke flows around qwen36-deep, post-stack Bob wallet seeding, PurchaseRequest readiness, and paid inference settlement checks
update the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces

Validation

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
full release smoke passed with OBOL_LLM_MODEL=qwen36-deep, including USDC, live OBOL, and fork OBOL paid inference flows

Notes

This is stacked on #610's branch so reviewers can inspect only the follow-up fixes here.

bussyjd · 2026-06-09T21:57:29Z

Summary

What changed:

Replaced host-side chown/chmod init handling with UID/GID/fsGroup 1000 runtime ownership for Hermes and spawned agent workloads.
Ensured LiteLLM rolls when model or buyer ConfigMaps change, so paid routes are refreshed when the config changes without an env var change.
Hardened paid smoke flows around qwen36-deep, inference ServiceOffers, PurchaseRequest auth counts, LiteLLM readiness, and exact settlement deltas.
Moved Bob wallet setup in dual-stack flows to post-stack obol wallet import, then asserts the signer matches deterministic Bob.
Updated the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces.

Why it matters:

Solves the PVC permission problem without making config read-only and without host-side ownership mutation.
Reduces fragile permission scaffolding while keeping runtime writes available.
Keeps release-smoke aligned with the supported obol CLI, stack, and obol kubectl surfaces.

Risk level: medium

Commit under test: 1f3088b

Base branch: research/hermes-configmap-skills-delivery

Scope

Validation

CI checks:

Check	Status	Link
CI	not checked in this comment

Unit tests:

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
Result: PASS
Commit/SHA: 1f3088b

Integration tests:

No separate integration suite was run outside the release smoke.

Flow tests:

Flow	Network	QA machine label	Worktree	Result	Artifacts
`flow-01-prerequisites`	local	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-02-stack-init-up`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-03-inference`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-04-agent`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-05-network`	Base Sepolia RPC	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-06-sell-setup`	Base Sepolia x402 pricing	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-07-sell-verify`	x402 tunnel route	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-10-anvil-facilitator`	Base Sepolia Anvil fork	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-08-buy`	Base Sepolia Anvil fork / USDC	remote ARM64 QA host	redacted	PASS	receipt retained on QA host
`flow-09-lifecycle`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-11-dual-stack`	Base Sepolia / USDC	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-14-live-obol-base-sepolia`	live Base Sepolia / OBOL	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-13-dual-stack-obol`	Base Sepolia Anvil fork / OBOL	remote ARM64 QA host	redacted	PASS	receipt summary retained on QA host

Release smoke:

Command: flows/release-smoke.sh
Flags/env: OBOL_DEVELOPMENT=true, OBOL_NONINTERACTIVE=true, OBOL_FORCE_REBUILD_LOCAL_DEV_IMAGES=serviceoffer-controller, OBOL_LLM_MODEL=qwen36-deep, RELEASE_SMOKE_INCLUDE_OBOL=true, RELEASE_SMOKE_INCLUDE_OBOL_FORK=true
Endpoint/RPC/worktree details: redacted
Result: PASS

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

Base Sepolia
Base Sepolia Anvil fork

RPC/provider:

Paid Base Sepolia RPC, redacted
Local Anvil fork for forked flows

Facilitator:

Public Obol facilitator for live Base Sepolia flows, hostname omitted
Local x402-rs facilitator for fork flow

Contracts and tokens:

Name	Address	Version / notes
Base Sepolia OBOL	`0x0a09371a8b011d5110656ceBCc70603e53FD2c78`	live token
Base Sepolia USDC	`0x036CbD53842c5426634e7929541eC2318f3dCF7e`	live/fork token
ERC-8004 identity registry	`0x8004A818BFB912233c491871b3d84c89A494BD9e`	Base Sepolia
Fork OBOL	`0x210BBd033630e5e611B7922D70b0Caabe64636d9`	flow-13 deployed token

Wallet roles:

Role	Address	Source
Alice / seller / register	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	runtime-derived from configured signer
Bob / buyer / payer	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	deterministic second-derived buyer
Single-stack flow-08 seller	`0x70997970C51812dc3A010C7d01b50e0d17dc79C8`	local flow seller

Balances:

Token	Address	Before	After	Expected delta	Actual delta
Live OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`30000000000000000`	`31000000000000000`	`+1000000000000000`	`+1000000000000000`
Live OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`4947000000000000000`	`4946000000000000000`	`-1000000000000000`	`-1000000000000000`
Fork OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`10000000000000000000`	`10001000000000000000`	`+1000000000000000`	`+1000000000000000`
Fork OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`10000000000000000000`	`9999000000000000000`	`-1000000000000000`	`-1000000000000000`

Transaction receipts:

Purpose	Tx hash	From	To	Amount / event	Status
ERC-8004 registration	`0x8bc7bf9c5daaaa3c3df889ce7ee44f483593e0b79fd9126a34838314280e1310`	Alice	registry	Agent ID reflected in ServiceOffer	PASS
Metadata / service offer				Metadata tx unavailable, ServiceOffer status reflected Agent ID	SKIP
Live OBOL approval / permit	`0x51841c0cfcfd1b5e0b23d2976a755f8ed7d11489015e885a962bf3b1c7cd878c`	Bob	Permit2	Max allowance confirmed	PASS
Live OBOL settlement transfer	`0xb78fa7b9f1097fb88fd6e5fb6d04ffbfc03337f478691b0905eb4d70f0dc2981`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Fork OBOL buyer funding	`0x733a277dd44da427bd3749b955ab4ab77fe9dadbe94358c130afab2ccd4600de`	deployer	Bob	`10` OBOL minted	PASS
Fork OBOL approval / permit	`0x40dc8bc61cb2f4dbd31d532141ef1885f3efb29144ee2fc50f712dd265ffa3c8`	Bob	Permit2	Max allowance confirmed	PASS
Fork OBOL settlement transfer	`0xb0d8974636102f1be97383c4fb1ba8aa860ef7850084ce5d425e37c1016b114d`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Flow-08 USDC settlement transfer	`0x1e7fccd95dfd47e2720896406108f94d391d4f8d5824c37f9d3cdfdc1ca7f9c2`	buyer	seller	`1000` micro-USDC	PASS
Flow-11 USDC settlement transfer	`0x6127dbec3139eadc68b5343a2ccf552fcf82595c4cdeaf81c893d51ee74754df`	Bob	Alice	`1000` micro-USDC	PASS

Runtime Evidence

QA environment:

Item	Value
OS / arch	Linux / arm64 remote QA host
Backend	k3d
Tool versions	branch-built `obol` CLI, Foundry tools available
QA agent/model	`qwen36-deep` via redacted OpenAI-compatible endpoint

Images:

Component	Image	Tag / digest	Source
serviceoffer-controller	local dev image	rebuilt during smoke	branch source
x402 facilitator	`ghcr.io/obolnetwork/x402-facilitator-prometheus-overlay`	`1.4.9`	flow-13 preflight

Kubernetes / stack:

Item	Value
Stack IDs	omitted to avoid host-local details
Namespaces	`llm`, `x402`, `erpc`, `hermes-obol-agent`
Pod readiness	all smoke-gated pods Ready during flow checks
Cleanup result	no k3d clusters remaining after final smoke

Model and routing:

Item	Value
Agent/model used	`qwen36-deep`
LiteLLM route	`paid/qwen36-deep`
Paid endpoint status	HTTP 200 with non-empty response
Auth token source	`PurchaseRequest` plus x402-buyer sidecar, no raw token logged

Artifacts and logs:

Artifact	Location / link	Notes
Release smoke report	retained on QA host, path omitted	result PASS
Flow-11 receipts	retained on QA host, path omitted	USDC settlement
Flow-14 receipts	retained on QA host, path omitted	live OBOL settlement
Flow-13 receipts	retained on QA host, path omitted	fork OBOL settlement

Demo readiness:

Item	Status	Notes
Seller visible / registered	PASS	ServiceOffer Ready; Agent ID reflected where registration was enabled
Buyer discovery works	PASS	Buyer discovered service catalog / registry data
Paid route works	PASS	`paid/qwen36-deep` returned HTTP 200
Settlement visible on-chain	PASS	Transfer receipts and exact deltas verified

Review Notes

Known gaps:

flow-11 and flow-14 recorded one SKIP each for unavailable registration receipt while ServiceOffer status already reflected the Agent ID.
PR is stacked on research/hermes-configmap-skills-delivery, so reviewers should compare against that branch for the focused delta.

Follow-ups:

None required for the validated smoke path.

Reviewer focus:

UID/GID/fsGroup 1000 ownership model versus the removed host-side chown/chmod path.
LiteLLM ConfigMap reload behavior for paid routes.
Flow-08 retry/readiness behavior around obol buy inference and LiteLLM rollout.
Post-stack Bob wallet import in dual-stack flows.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

…with #614 Applies the verified findings from the cross-review against PR #614 (every item adversarially confirmed against the refs; union merge-tree clean). PVC / upgrade path: - llm.yaml: restore container-level runAsUser/runAsGroup 1000 on x402-buyer. Clusters upgraded in place from <= rc12 keep hostPath-typed PVs where kubelet skips fsGroup; their /state dir is 1000:1000 with consumed.json written 0600 by UID 1000 — a 65532 sidecar cannot read it, Fatalf's on `load state`, and takes every paid/<model> route down. On fresh local-type PVs the explicit UID is harmless (fsGroup 65532 grants group access). embed_buyer_state_test.go updated to pin the new contract. - plans/volume-permission-hardening.md: new "Upgrading from <= v0.10.0-rc12" section — supported path is cluster recreation (wallet backup/restore), with a documented k3d chown escape hatch. troubleshooting.md gets the symptom->fix entry. The Hermes half of the legacy-PV breakage cannot be patched at runtime without reintroducing the chown machinery this PR removes, so it is a documented breaking change instead. Paid-route availability: - llm.yaml: Reloader annotation narrowed to litellm-config only. The buyer ConfigMaps (x402-buyer-config/x402-buyer-auths) are rewritten by the controller on every buy, top-up, auto-refill, and tombstone cleanup; with strategy Recreate + 1 replica the previous annotation bounced the entire inference gateway (all Hermes traffic, in-flight SSE streams) on every purchase event, inverting CLAUDE.md pitfall 7 (restart is fallback, not the default buy path). The buyer hot-reloads via /admin/reload. stack_test.go updated to pin litellm-config-only. Flow alignment with #614: - lib.sh: `stack down` -> `stack down --yes` in reset_flow_workspace. #614's flow-16 (now last in the single-stack array) intentionally leaves a live agent offer; without --yes the non-TTY ConfirmRunningServicesLoss gate refuses, graceful down is silently skipped (`|| true`), and teardown degrades to the raw k3d-delete fallback on every release-smoke run. - flow-11: post-register Ready poll 120s -> 300s to match flow-14's identical live-Base-Sepolia chain-watch path (pitfall 13 free-tier RPC throttling). Known follow-ups (not in this commit): flow-08 buy-retry top-up vs exactly-N assertions on rare partial failures; flow-11 lacks flow-14's remote-signer rolled guard; aztec PVC has no permission story (runs as root today); post-merge controller repin so released sub-agents pick up this PR's UID-1000 render (tracked in #614's pin-test note).

Records the options ladder for 'nodes joining my cluster' (far-future, not scheduled) so near-term permission decisions don't foreclose it: the host-FS-canonical coupling and its four conditions, the agent-homes-only scope, what breaks on day one of a join (WaitForFirstConsumer + no nodeSelector in any render), options 0-4 (home-node pattern -> NFS -> distributed storage -> API-mediated access -> hybrid by data class), join mechanics (native k3s server + remote agents; macOS stays k3d), and the decisions that bind today: group-1000 sharing over render-time UID matching, home-node pinning as a join prerequisite, inputs to API objects per Remaining Debt.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

bussyjd · 2026-06-09T21:06:25Z

Summary

replace host-side chown/chmod init handling with UID/GID 1000 runtime ownership for Hermes and spawned agents
keep LiteLLM/x402 buyer routing in sync when model ConfigMaps change so paid routes roll correctly
harden release smoke flows around qwen36-deep, post-stack Bob wallet seeding, PurchaseRequest readiness, and paid inference settlement checks
update the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces

Validation

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
full release smoke passed with OBOL_LLM_MODEL=qwen36-deep, including USDC, live OBOL, and fork OBOL paid inference flows

Notes

This is stacked on #610's branch so reviewers can inspect only the follow-up fixes here.

bussyjd · 2026-06-09T21:57:29Z

Summary

What changed:

Replaced host-side chown/chmod init handling with UID/GID/fsGroup 1000 runtime ownership for Hermes and spawned agent workloads.
Ensured LiteLLM rolls when model or buyer ConfigMaps change, so paid routes are refreshed when the config changes without an env var change.
Hardened paid smoke flows around qwen36-deep, inference ServiceOffers, PurchaseRequest auth counts, LiteLLM readiness, and exact settlement deltas.
Moved Bob wallet setup in dual-stack flows to post-stack obol wallet import, then asserts the signer matches deterministic Bob.
Updated the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces.

Why it matters:

Solves the PVC permission problem without making config read-only and without host-side ownership mutation.
Reduces fragile permission scaffolding while keeping runtime writes available.
Keeps release-smoke aligned with the supported obol CLI, stack, and obol kubectl surfaces.

Risk level: medium

Commit under test: 1f3088b

Base branch: research/hermes-configmap-skills-delivery

Scope

Validation

CI checks:

Check	Status	Link
CI	not checked in this comment

Unit tests:

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
Result: PASS
Commit/SHA: 1f3088b

Integration tests:

No separate integration suite was run outside the release smoke.

Flow tests:

Flow	Network	QA machine label	Worktree	Result	Artifacts
`flow-01-prerequisites`	local	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-02-stack-init-up`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-03-inference`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-04-agent`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-05-network`	Base Sepolia RPC	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-06-sell-setup`	Base Sepolia x402 pricing	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-07-sell-verify`	x402 tunnel route	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-10-anvil-facilitator`	Base Sepolia Anvil fork	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-08-buy`	Base Sepolia Anvil fork / USDC	remote ARM64 QA host	redacted	PASS	receipt retained on QA host
`flow-09-lifecycle`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-11-dual-stack`	Base Sepolia / USDC	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-14-live-obol-base-sepolia`	live Base Sepolia / OBOL	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-13-dual-stack-obol`	Base Sepolia Anvil fork / OBOL	remote ARM64 QA host	redacted	PASS	receipt summary retained on QA host

Release smoke:

Command: flows/release-smoke.sh
Flags/env: OBOL_DEVELOPMENT=true, OBOL_NONINTERACTIVE=true, OBOL_FORCE_REBUILD_LOCAL_DEV_IMAGES=serviceoffer-controller, OBOL_LLM_MODEL=qwen36-deep, RELEASE_SMOKE_INCLUDE_OBOL=true, RELEASE_SMOKE_INCLUDE_OBOL_FORK=true
Endpoint/RPC/worktree details: redacted
Result: PASS

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

Base Sepolia
Base Sepolia Anvil fork

RPC/provider:

Paid Base Sepolia RPC, redacted
Local Anvil fork for forked flows

Facilitator:

Public Obol facilitator for live Base Sepolia flows, hostname omitted
Local x402-rs facilitator for fork flow

Contracts and tokens:

Name	Address	Version / notes
Base Sepolia OBOL	`0x0a09371a8b011d5110656ceBCc70603e53FD2c78`	live token
Base Sepolia USDC	`0x036CbD53842c5426634e7929541eC2318f3dCF7e`	live/fork token
ERC-8004 identity registry	`0x8004A818BFB912233c491871b3d84c89A494BD9e`	Base Sepolia
Fork OBOL	`0x210BBd033630e5e611B7922D70b0Caabe64636d9`	flow-13 deployed token

Wallet roles:

Role	Address	Source
Alice / seller / register	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	runtime-derived from configured signer
Bob / buyer / payer	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	deterministic second-derived buyer
Single-stack flow-08 seller	`0x70997970C51812dc3A010C7d01b50e0d17dc79C8`	local flow seller

Balances:

Token	Address	Before	After	Expected delta	Actual delta
Live OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`30000000000000000`	`31000000000000000`	`+1000000000000000`	`+1000000000000000`
Live OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`4947000000000000000`	`4946000000000000000`	`-1000000000000000`	`-1000000000000000`
Fork OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`10000000000000000000`	`10001000000000000000`	`+1000000000000000`	`+1000000000000000`
Fork OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`10000000000000000000`	`9999000000000000000`	`-1000000000000000`	`-1000000000000000`

Transaction receipts:

Purpose	Tx hash	From	To	Amount / event	Status
ERC-8004 registration	`0x8bc7bf9c5daaaa3c3df889ce7ee44f483593e0b79fd9126a34838314280e1310`	Alice	registry	Agent ID reflected in ServiceOffer	PASS
Metadata / service offer				Metadata tx unavailable, ServiceOffer status reflected Agent ID	SKIP
Live OBOL approval / permit	`0x51841c0cfcfd1b5e0b23d2976a755f8ed7d11489015e885a962bf3b1c7cd878c`	Bob	Permit2	Max allowance confirmed	PASS
Live OBOL settlement transfer	`0xb78fa7b9f1097fb88fd6e5fb6d04ffbfc03337f478691b0905eb4d70f0dc2981`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Fork OBOL buyer funding	`0x733a277dd44da427bd3749b955ab4ab77fe9dadbe94358c130afab2ccd4600de`	deployer	Bob	`10` OBOL minted	PASS
Fork OBOL approval / permit	`0x40dc8bc61cb2f4dbd31d532141ef1885f3efb29144ee2fc50f712dd265ffa3c8`	Bob	Permit2	Max allowance confirmed	PASS
Fork OBOL settlement transfer	`0xb0d8974636102f1be97383c4fb1ba8aa860ef7850084ce5d425e37c1016b114d`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Flow-08 USDC settlement transfer	`0x1e7fccd95dfd47e2720896406108f94d391d4f8d5824c37f9d3cdfdc1ca7f9c2`	buyer	seller	`1000` micro-USDC	PASS
Flow-11 USDC settlement transfer	`0x6127dbec3139eadc68b5343a2ccf552fcf82595c4cdeaf81c893d51ee74754df`	Bob	Alice	`1000` micro-USDC	PASS

Runtime Evidence

QA environment:

Item	Value
OS / arch	Linux / arm64 remote QA host
Backend	k3d
Tool versions	branch-built `obol` CLI, Foundry tools available
QA agent/model	`qwen36-deep` via redacted OpenAI-compatible endpoint

Images:

Component	Image	Tag / digest	Source
serviceoffer-controller	local dev image	rebuilt during smoke	branch source
x402 facilitator	`ghcr.io/obolnetwork/x402-facilitator-prometheus-overlay`	`1.4.9`	flow-13 preflight

Kubernetes / stack:

Item	Value
Stack IDs	omitted to avoid host-local details
Namespaces	`llm`, `x402`, `erpc`, `hermes-obol-agent`
Pod readiness	all smoke-gated pods Ready during flow checks
Cleanup result	no k3d clusters remaining after final smoke

Model and routing:

Item	Value
Agent/model used	`qwen36-deep`
LiteLLM route	`paid/qwen36-deep`
Paid endpoint status	HTTP 200 with non-empty response
Auth token source	`PurchaseRequest` plus x402-buyer sidecar, no raw token logged

Artifacts and logs:

Artifact	Location / link	Notes
Release smoke report	retained on QA host, path omitted	result PASS
Flow-11 receipts	retained on QA host, path omitted	USDC settlement
Flow-14 receipts	retained on QA host, path omitted	live OBOL settlement
Flow-13 receipts	retained on QA host, path omitted	fork OBOL settlement

Demo readiness:

Item	Status	Notes
Seller visible / registered	PASS	ServiceOffer Ready; Agent ID reflected where registration was enabled
Buyer discovery works	PASS	Buyer discovered service catalog / registry data
Paid route works	PASS	`paid/qwen36-deep` returned HTTP 200
Settlement visible on-chain	PASS	Transfer receipts and exact deltas verified

Review Notes

Known gaps:

flow-11 and flow-14 recorded one SKIP each for unavailable registration receipt while ServiceOffer status already reflected the Agent ID.
PR is stacked on research/hermes-configmap-skills-delivery, so reviewers should compare against that branch for the focused delta.

Follow-ups:

None required for the validated smoke path.

Reviewer focus:

UID/GID/fsGroup 1000 ownership model versus the removed host-side chown/chmod path.
LiteLLM ConfigMap reload behavior for paid routes.
Flow-08 retry/readiness behavior around obol buy inference and LiteLLM rollout.
Post-stack Bob wallet import in dual-stack flows.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

…with #614 Applies the verified findings from the cross-review against PR #614 (every item adversarially confirmed against the refs; union merge-tree clean). PVC / upgrade path: - llm.yaml: restore container-level runAsUser/runAsGroup 1000 on x402-buyer. Clusters upgraded in place from <= rc12 keep hostPath-typed PVs where kubelet skips fsGroup; their /state dir is 1000:1000 with consumed.json written 0600 by UID 1000 — a 65532 sidecar cannot read it, Fatalf's on `load state`, and takes every paid/<model> route down. On fresh local-type PVs the explicit UID is harmless (fsGroup 65532 grants group access). embed_buyer_state_test.go updated to pin the new contract. - plans/volume-permission-hardening.md: new "Upgrading from <= v0.10.0-rc12" section — supported path is cluster recreation (wallet backup/restore), with a documented k3d chown escape hatch. troubleshooting.md gets the symptom->fix entry. The Hermes half of the legacy-PV breakage cannot be patched at runtime without reintroducing the chown machinery this PR removes, so it is a documented breaking change instead. Paid-route availability: - llm.yaml: Reloader annotation narrowed to litellm-config only. The buyer ConfigMaps (x402-buyer-config/x402-buyer-auths) are rewritten by the controller on every buy, top-up, auto-refill, and tombstone cleanup; with strategy Recreate + 1 replica the previous annotation bounced the entire inference gateway (all Hermes traffic, in-flight SSE streams) on every purchase event, inverting CLAUDE.md pitfall 7 (restart is fallback, not the default buy path). The buyer hot-reloads via /admin/reload. stack_test.go updated to pin litellm-config-only. Flow alignment with #614: - lib.sh: `stack down` -> `stack down --yes` in reset_flow_workspace. #614's flow-16 (now last in the single-stack array) intentionally leaves a live agent offer; without --yes the non-TTY ConfirmRunningServicesLoss gate refuses, graceful down is silently skipped (`|| true`), and teardown degrades to the raw k3d-delete fallback on every release-smoke run. - flow-11: post-register Ready poll 120s -> 300s to match flow-14's identical live-Base-Sepolia chain-watch path (pitfall 13 free-tier RPC throttling). Known follow-ups (not in this commit): flow-08 buy-retry top-up vs exactly-N assertions on rare partial failures; flow-11 lacks flow-14's remote-signer rolled guard; aztec PVC has no permission story (runs as root today); post-merge controller repin so released sub-agents pick up this PR's UID-1000 render (tracked in #614's pin-test note).

Records the options ladder for 'nodes joining my cluster' (far-future, not scheduled) so near-term permission decisions don't foreclose it: the host-FS-canonical coupling and its four conditions, the agent-homes-only scope, what breaks on day one of a join (WaitForFirstConsumer + no nodeSelector in any render), options 0-4 (home-node pattern -> NFS -> distributed storage -> API-mediated access -> hybrid by data class), join mechanics (native k3s server + remote agents; macOS stays k3d), and the decisions that bind today: group-1000 sharing over render-time UID matching, home-node pinning as a join prerequisite, inputs to API objects per Remaining Debt.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

bussyjd · 2026-06-09T21:06:25Z

Summary

replace host-side chown/chmod init handling with UID/GID 1000 runtime ownership for Hermes and spawned agents
keep LiteLLM/x402 buyer routing in sync when model ConfigMaps change so paid routes roll correctly
harden release smoke flows around qwen36-deep, post-stack Bob wallet seeding, PurchaseRequest readiness, and paid inference settlement checks
update the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces

Validation

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
full release smoke passed with OBOL_LLM_MODEL=qwen36-deep, including USDC, live OBOL, and fork OBOL paid inference flows

Notes

This is stacked on #610's branch so reviewers can inspect only the follow-up fixes here.

bussyjd · 2026-06-09T21:57:29Z

Summary

What changed:

Replaced host-side chown/chmod init handling with UID/GID/fsGroup 1000 runtime ownership for Hermes and spawned agent workloads.
Ensured LiteLLM rolls when model or buyer ConfigMaps change, so paid routes are refreshed when the config changes without an env var change.
Hardened paid smoke flows around qwen36-deep, inference ServiceOffers, PurchaseRequest auth counts, LiteLLM readiness, and exact settlement deltas.
Moved Bob wallet setup in dual-stack flows to post-stack obol wallet import, then asserts the signer matches deterministic Bob.
Updated the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces.

Why it matters:

Solves the PVC permission problem without making config read-only and without host-side ownership mutation.
Reduces fragile permission scaffolding while keeping runtime writes available.
Keeps release-smoke aligned with the supported obol CLI, stack, and obol kubectl surfaces.

Risk level: medium

Commit under test: 1f3088b

Base branch: research/hermes-configmap-skills-delivery

Scope

Validation

CI checks:

Check	Status	Link
CI	not checked in this comment

Unit tests:

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
Result: PASS
Commit/SHA: 1f3088b

Integration tests:

No separate integration suite was run outside the release smoke.

Flow tests:

Flow	Network	QA machine label	Worktree	Result	Artifacts
`flow-01-prerequisites`	local	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-02-stack-init-up`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-03-inference`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-04-agent`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-05-network`	Base Sepolia RPC	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-06-sell-setup`	Base Sepolia x402 pricing	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-07-sell-verify`	x402 tunnel route	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-10-anvil-facilitator`	Base Sepolia Anvil fork	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-08-buy`	Base Sepolia Anvil fork / USDC	remote ARM64 QA host	redacted	PASS	receipt retained on QA host
`flow-09-lifecycle`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-11-dual-stack`	Base Sepolia / USDC	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-14-live-obol-base-sepolia`	live Base Sepolia / OBOL	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-13-dual-stack-obol`	Base Sepolia Anvil fork / OBOL	remote ARM64 QA host	redacted	PASS	receipt summary retained on QA host

Release smoke:

Command: flows/release-smoke.sh
Flags/env: OBOL_DEVELOPMENT=true, OBOL_NONINTERACTIVE=true, OBOL_FORCE_REBUILD_LOCAL_DEV_IMAGES=serviceoffer-controller, OBOL_LLM_MODEL=qwen36-deep, RELEASE_SMOKE_INCLUDE_OBOL=true, RELEASE_SMOKE_INCLUDE_OBOL_FORK=true
Endpoint/RPC/worktree details: redacted
Result: PASS

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

Base Sepolia
Base Sepolia Anvil fork

RPC/provider:

Paid Base Sepolia RPC, redacted
Local Anvil fork for forked flows

Facilitator:

Public Obol facilitator for live Base Sepolia flows, hostname omitted
Local x402-rs facilitator for fork flow

Contracts and tokens:

Name	Address	Version / notes
Base Sepolia OBOL	`0x0a09371a8b011d5110656ceBCc70603e53FD2c78`	live token
Base Sepolia USDC	`0x036CbD53842c5426634e7929541eC2318f3dCF7e`	live/fork token
ERC-8004 identity registry	`0x8004A818BFB912233c491871b3d84c89A494BD9e`	Base Sepolia
Fork OBOL	`0x210BBd033630e5e611B7922D70b0Caabe64636d9`	flow-13 deployed token

Wallet roles:

Role	Address	Source
Alice / seller / register	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	runtime-derived from configured signer
Bob / buyer / payer	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	deterministic second-derived buyer
Single-stack flow-08 seller	`0x70997970C51812dc3A010C7d01b50e0d17dc79C8`	local flow seller

Balances:

Token	Address	Before	After	Expected delta	Actual delta
Live OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`30000000000000000`	`31000000000000000`	`+1000000000000000`	`+1000000000000000`
Live OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`4947000000000000000`	`4946000000000000000`	`-1000000000000000`	`-1000000000000000`
Fork OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`10000000000000000000`	`10001000000000000000`	`+1000000000000000`	`+1000000000000000`
Fork OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`10000000000000000000`	`9999000000000000000`	`-1000000000000000`	`-1000000000000000`

Transaction receipts:

Purpose	Tx hash	From	To	Amount / event	Status
ERC-8004 registration	`0x8bc7bf9c5daaaa3c3df889ce7ee44f483593e0b79fd9126a34838314280e1310`	Alice	registry	Agent ID reflected in ServiceOffer	PASS
Metadata / service offer				Metadata tx unavailable, ServiceOffer status reflected Agent ID	SKIP
Live OBOL approval / permit	`0x51841c0cfcfd1b5e0b23d2976a755f8ed7d11489015e885a962bf3b1c7cd878c`	Bob	Permit2	Max allowance confirmed	PASS
Live OBOL settlement transfer	`0xb78fa7b9f1097fb88fd6e5fb6d04ffbfc03337f478691b0905eb4d70f0dc2981`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Fork OBOL buyer funding	`0x733a277dd44da427bd3749b955ab4ab77fe9dadbe94358c130afab2ccd4600de`	deployer	Bob	`10` OBOL minted	PASS
Fork OBOL approval / permit	`0x40dc8bc61cb2f4dbd31d532141ef1885f3efb29144ee2fc50f712dd265ffa3c8`	Bob	Permit2	Max allowance confirmed	PASS
Fork OBOL settlement transfer	`0xb0d8974636102f1be97383c4fb1ba8aa860ef7850084ce5d425e37c1016b114d`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Flow-08 USDC settlement transfer	`0x1e7fccd95dfd47e2720896406108f94d391d4f8d5824c37f9d3cdfdc1ca7f9c2`	buyer	seller	`1000` micro-USDC	PASS
Flow-11 USDC settlement transfer	`0x6127dbec3139eadc68b5343a2ccf552fcf82595c4cdeaf81c893d51ee74754df`	Bob	Alice	`1000` micro-USDC	PASS

Runtime Evidence

QA environment:

Item	Value
OS / arch	Linux / arm64 remote QA host
Backend	k3d
Tool versions	branch-built `obol` CLI, Foundry tools available
QA agent/model	`qwen36-deep` via redacted OpenAI-compatible endpoint

Images:

Component	Image	Tag / digest	Source
serviceoffer-controller	local dev image	rebuilt during smoke	branch source
x402 facilitator	`ghcr.io/obolnetwork/x402-facilitator-prometheus-overlay`	`1.4.9`	flow-13 preflight

Kubernetes / stack:

Item	Value
Stack IDs	omitted to avoid host-local details
Namespaces	`llm`, `x402`, `erpc`, `hermes-obol-agent`
Pod readiness	all smoke-gated pods Ready during flow checks
Cleanup result	no k3d clusters remaining after final smoke

Model and routing:

Item	Value
Agent/model used	`qwen36-deep`
LiteLLM route	`paid/qwen36-deep`
Paid endpoint status	HTTP 200 with non-empty response
Auth token source	`PurchaseRequest` plus x402-buyer sidecar, no raw token logged

Artifacts and logs:

Artifact	Location / link	Notes
Release smoke report	retained on QA host, path omitted	result PASS
Flow-11 receipts	retained on QA host, path omitted	USDC settlement
Flow-14 receipts	retained on QA host, path omitted	live OBOL settlement
Flow-13 receipts	retained on QA host, path omitted	fork OBOL settlement

Demo readiness:

Item	Status	Notes
Seller visible / registered	PASS	ServiceOffer Ready; Agent ID reflected where registration was enabled
Buyer discovery works	PASS	Buyer discovered service catalog / registry data
Paid route works	PASS	`paid/qwen36-deep` returned HTTP 200
Settlement visible on-chain	PASS	Transfer receipts and exact deltas verified

Review Notes

Known gaps:

flow-11 and flow-14 recorded one SKIP each for unavailable registration receipt while ServiceOffer status already reflected the Agent ID.
PR is stacked on research/hermes-configmap-skills-delivery, so reviewers should compare against that branch for the focused delta.

Follow-ups:

None required for the validated smoke path.

Reviewer focus:

UID/GID/fsGroup 1000 ownership model versus the removed host-side chown/chmod path.
LiteLLM ConfigMap reload behavior for paid routes.
Flow-08 retry/readiness behavior around obol buy inference and LiteLLM rollout.
Post-stack Bob wallet import in dual-stack flows.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

…with #614 Applies the verified findings from the cross-review against PR #614 (every item adversarially confirmed against the refs; union merge-tree clean). PVC / upgrade path: - llm.yaml: restore container-level runAsUser/runAsGroup 1000 on x402-buyer. Clusters upgraded in place from <= rc12 keep hostPath-typed PVs where kubelet skips fsGroup; their /state dir is 1000:1000 with consumed.json written 0600 by UID 1000 — a 65532 sidecar cannot read it, Fatalf's on `load state`, and takes every paid/<model> route down. On fresh local-type PVs the explicit UID is harmless (fsGroup 65532 grants group access). embed_buyer_state_test.go updated to pin the new contract. - plans/volume-permission-hardening.md: new "Upgrading from <= v0.10.0-rc12" section — supported path is cluster recreation (wallet backup/restore), with a documented k3d chown escape hatch. troubleshooting.md gets the symptom->fix entry. The Hermes half of the legacy-PV breakage cannot be patched at runtime without reintroducing the chown machinery this PR removes, so it is a documented breaking change instead. Paid-route availability: - llm.yaml: Reloader annotation narrowed to litellm-config only. The buyer ConfigMaps (x402-buyer-config/x402-buyer-auths) are rewritten by the controller on every buy, top-up, auto-refill, and tombstone cleanup; with strategy Recreate + 1 replica the previous annotation bounced the entire inference gateway (all Hermes traffic, in-flight SSE streams) on every purchase event, inverting CLAUDE.md pitfall 7 (restart is fallback, not the default buy path). The buyer hot-reloads via /admin/reload. stack_test.go updated to pin litellm-config-only. Flow alignment with #614: - lib.sh: `stack down` -> `stack down --yes` in reset_flow_workspace. #614's flow-16 (now last in the single-stack array) intentionally leaves a live agent offer; without --yes the non-TTY ConfirmRunningServicesLoss gate refuses, graceful down is silently skipped (`|| true`), and teardown degrades to the raw k3d-delete fallback on every release-smoke run. - flow-11: post-register Ready poll 120s -> 300s to match flow-14's identical live-Base-Sepolia chain-watch path (pitfall 13 free-tier RPC throttling). Known follow-ups (not in this commit): flow-08 buy-retry top-up vs exactly-N assertions on rare partial failures; flow-11 lacks flow-14's remote-signer rolled guard; aztec PVC has no permission story (runs as root today); post-merge controller repin so released sub-agents pick up this PR's UID-1000 render (tracked in #614's pin-test note).

Records the options ladder for 'nodes joining my cluster' (far-future, not scheduled) so near-term permission decisions don't foreclose it: the host-FS-canonical coupling and its four conditions, the agent-homes-only scope, what breaks on day one of a join (WaitForFirstConsumer + no nodeSelector in any render), options 0-4 (home-node pattern -> NFS -> distributed storage -> API-mediated access -> hybrid by data class), join mechanics (native k3s server + remote agents; macOS stays k3d), and the decisions that bind today: group-1000 sharing over render-time UID matching, home-node pinning as a join prerequisite, inputs to API objects per Remaining Debt.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

bussyjd · 2026-06-09T21:06:25Z

Summary

replace host-side chown/chmod init handling with UID/GID 1000 runtime ownership for Hermes and spawned agents
keep LiteLLM/x402 buyer routing in sync when model ConfigMaps change so paid routes roll correctly
harden release smoke flows around qwen36-deep, post-stack Bob wallet seeding, PurchaseRequest readiness, and paid inference settlement checks
update the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces

Validation

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
full release smoke passed with OBOL_LLM_MODEL=qwen36-deep, including USDC, live OBOL, and fork OBOL paid inference flows

Notes

This is stacked on #610's branch so reviewers can inspect only the follow-up fixes here.

bussyjd · 2026-06-09T21:57:29Z

Summary

What changed:

Replaced host-side chown/chmod init handling with UID/GID/fsGroup 1000 runtime ownership for Hermes and spawned agent workloads.
Ensured LiteLLM rolls when model or buyer ConfigMaps change, so paid routes are refreshed when the config changes without an env var change.
Hardened paid smoke flows around qwen36-deep, inference ServiceOffers, PurchaseRequest auth counts, LiteLLM readiness, and exact settlement deltas.
Moved Bob wallet setup in dual-stack flows to post-stack obol wallet import, then asserts the signer matches deterministic Bob.
Updated the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces.

Why it matters:

Solves the PVC permission problem without making config read-only and without host-side ownership mutation.
Reduces fragile permission scaffolding while keeping runtime writes available.
Keeps release-smoke aligned with the supported obol CLI, stack, and obol kubectl surfaces.

Risk level: medium

Commit under test: 1f3088b

Base branch: research/hermes-configmap-skills-delivery

Scope

Validation

CI checks:

Check	Status	Link
CI	not checked in this comment

Unit tests:

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
Result: PASS
Commit/SHA: 1f3088b

Integration tests:

No separate integration suite was run outside the release smoke.

Flow tests:

Flow	Network	QA machine label	Worktree	Result	Artifacts
`flow-01-prerequisites`	local	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-02-stack-init-up`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-03-inference`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-04-agent`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-05-network`	Base Sepolia RPC	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-06-sell-setup`	Base Sepolia x402 pricing	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-07-sell-verify`	x402 tunnel route	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-10-anvil-facilitator`	Base Sepolia Anvil fork	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-08-buy`	Base Sepolia Anvil fork / USDC	remote ARM64 QA host	redacted	PASS	receipt retained on QA host
`flow-09-lifecycle`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-11-dual-stack`	Base Sepolia / USDC	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-14-live-obol-base-sepolia`	live Base Sepolia / OBOL	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-13-dual-stack-obol`	Base Sepolia Anvil fork / OBOL	remote ARM64 QA host	redacted	PASS	receipt summary retained on QA host

Release smoke:

Command: flows/release-smoke.sh
Flags/env: OBOL_DEVELOPMENT=true, OBOL_NONINTERACTIVE=true, OBOL_FORCE_REBUILD_LOCAL_DEV_IMAGES=serviceoffer-controller, OBOL_LLM_MODEL=qwen36-deep, RELEASE_SMOKE_INCLUDE_OBOL=true, RELEASE_SMOKE_INCLUDE_OBOL_FORK=true
Endpoint/RPC/worktree details: redacted
Result: PASS

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

Base Sepolia
Base Sepolia Anvil fork

RPC/provider:

Paid Base Sepolia RPC, redacted
Local Anvil fork for forked flows

Facilitator:

Public Obol facilitator for live Base Sepolia flows, hostname omitted
Local x402-rs facilitator for fork flow

Contracts and tokens:

Name	Address	Version / notes
Base Sepolia OBOL	`0x0a09371a8b011d5110656ceBCc70603e53FD2c78`	live token
Base Sepolia USDC	`0x036CbD53842c5426634e7929541eC2318f3dCF7e`	live/fork token
ERC-8004 identity registry	`0x8004A818BFB912233c491871b3d84c89A494BD9e`	Base Sepolia
Fork OBOL	`0x210BBd033630e5e611B7922D70b0Caabe64636d9`	flow-13 deployed token

Wallet roles:

Role	Address	Source
Alice / seller / register	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	runtime-derived from configured signer
Bob / buyer / payer	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	deterministic second-derived buyer
Single-stack flow-08 seller	`0x70997970C51812dc3A010C7d01b50e0d17dc79C8`	local flow seller

Balances:

Token	Address	Before	After	Expected delta	Actual delta
Live OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`30000000000000000`	`31000000000000000`	`+1000000000000000`	`+1000000000000000`
Live OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`4947000000000000000`	`4946000000000000000`	`-1000000000000000`	`-1000000000000000`
Fork OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`10000000000000000000`	`10001000000000000000`	`+1000000000000000`	`+1000000000000000`
Fork OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`10000000000000000000`	`9999000000000000000`	`-1000000000000000`	`-1000000000000000`

Transaction receipts:

Purpose	Tx hash	From	To	Amount / event	Status
ERC-8004 registration	`0x8bc7bf9c5daaaa3c3df889ce7ee44f483593e0b79fd9126a34838314280e1310`	Alice	registry	Agent ID reflected in ServiceOffer	PASS
Metadata / service offer				Metadata tx unavailable, ServiceOffer status reflected Agent ID	SKIP
Live OBOL approval / permit	`0x51841c0cfcfd1b5e0b23d2976a755f8ed7d11489015e885a962bf3b1c7cd878c`	Bob	Permit2	Max allowance confirmed	PASS
Live OBOL settlement transfer	`0xb78fa7b9f1097fb88fd6e5fb6d04ffbfc03337f478691b0905eb4d70f0dc2981`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Fork OBOL buyer funding	`0x733a277dd44da427bd3749b955ab4ab77fe9dadbe94358c130afab2ccd4600de`	deployer	Bob	`10` OBOL minted	PASS
Fork OBOL approval / permit	`0x40dc8bc61cb2f4dbd31d532141ef1885f3efb29144ee2fc50f712dd265ffa3c8`	Bob	Permit2	Max allowance confirmed	PASS
Fork OBOL settlement transfer	`0xb0d8974636102f1be97383c4fb1ba8aa860ef7850084ce5d425e37c1016b114d`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Flow-08 USDC settlement transfer	`0x1e7fccd95dfd47e2720896406108f94d391d4f8d5824c37f9d3cdfdc1ca7f9c2`	buyer	seller	`1000` micro-USDC	PASS
Flow-11 USDC settlement transfer	`0x6127dbec3139eadc68b5343a2ccf552fcf82595c4cdeaf81c893d51ee74754df`	Bob	Alice	`1000` micro-USDC	PASS

Runtime Evidence

QA environment:

Item	Value
OS / arch	Linux / arm64 remote QA host
Backend	k3d
Tool versions	branch-built `obol` CLI, Foundry tools available
QA agent/model	`qwen36-deep` via redacted OpenAI-compatible endpoint

Images:

Component	Image	Tag / digest	Source
serviceoffer-controller	local dev image	rebuilt during smoke	branch source
x402 facilitator	`ghcr.io/obolnetwork/x402-facilitator-prometheus-overlay`	`1.4.9`	flow-13 preflight

Kubernetes / stack:

Item	Value
Stack IDs	omitted to avoid host-local details
Namespaces	`llm`, `x402`, `erpc`, `hermes-obol-agent`
Pod readiness	all smoke-gated pods Ready during flow checks
Cleanup result	no k3d clusters remaining after final smoke

Model and routing:

Item	Value
Agent/model used	`qwen36-deep`
LiteLLM route	`paid/qwen36-deep`
Paid endpoint status	HTTP 200 with non-empty response
Auth token source	`PurchaseRequest` plus x402-buyer sidecar, no raw token logged

Artifacts and logs:

Artifact	Location / link	Notes
Release smoke report	retained on QA host, path omitted	result PASS
Flow-11 receipts	retained on QA host, path omitted	USDC settlement
Flow-14 receipts	retained on QA host, path omitted	live OBOL settlement
Flow-13 receipts	retained on QA host, path omitted	fork OBOL settlement

Demo readiness:

Item	Status	Notes
Seller visible / registered	PASS	ServiceOffer Ready; Agent ID reflected where registration was enabled
Buyer discovery works	PASS	Buyer discovered service catalog / registry data
Paid route works	PASS	`paid/qwen36-deep` returned HTTP 200
Settlement visible on-chain	PASS	Transfer receipts and exact deltas verified

Review Notes

Known gaps:

flow-11 and flow-14 recorded one SKIP each for unavailable registration receipt while ServiceOffer status already reflected the Agent ID.
PR is stacked on research/hermes-configmap-skills-delivery, so reviewers should compare against that branch for the focused delta.

Follow-ups:

None required for the validated smoke path.

Reviewer focus:

UID/GID/fsGroup 1000 ownership model versus the removed host-side chown/chmod path.
LiteLLM ConfigMap reload behavior for paid routes.
Flow-08 retry/readiness behavior around obol buy inference and LiteLLM rollout.
Post-stack Bob wallet import in dual-stack flows.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

…with #614 Applies the verified findings from the cross-review against PR #614 (every item adversarially confirmed against the refs; union merge-tree clean). PVC / upgrade path: - llm.yaml: restore container-level runAsUser/runAsGroup 1000 on x402-buyer. Clusters upgraded in place from <= rc12 keep hostPath-typed PVs where kubelet skips fsGroup; their /state dir is 1000:1000 with consumed.json written 0600 by UID 1000 — a 65532 sidecar cannot read it, Fatalf's on `load state`, and takes every paid/<model> route down. On fresh local-type PVs the explicit UID is harmless (fsGroup 65532 grants group access). embed_buyer_state_test.go updated to pin the new contract. - plans/volume-permission-hardening.md: new "Upgrading from <= v0.10.0-rc12" section — supported path is cluster recreation (wallet backup/restore), with a documented k3d chown escape hatch. troubleshooting.md gets the symptom->fix entry. The Hermes half of the legacy-PV breakage cannot be patched at runtime without reintroducing the chown machinery this PR removes, so it is a documented breaking change instead. Paid-route availability: - llm.yaml: Reloader annotation narrowed to litellm-config only. The buyer ConfigMaps (x402-buyer-config/x402-buyer-auths) are rewritten by the controller on every buy, top-up, auto-refill, and tombstone cleanup; with strategy Recreate + 1 replica the previous annotation bounced the entire inference gateway (all Hermes traffic, in-flight SSE streams) on every purchase event, inverting CLAUDE.md pitfall 7 (restart is fallback, not the default buy path). The buyer hot-reloads via /admin/reload. stack_test.go updated to pin litellm-config-only. Flow alignment with #614: - lib.sh: `stack down` -> `stack down --yes` in reset_flow_workspace. #614's flow-16 (now last in the single-stack array) intentionally leaves a live agent offer; without --yes the non-TTY ConfirmRunningServicesLoss gate refuses, graceful down is silently skipped (`|| true`), and teardown degrades to the raw k3d-delete fallback on every release-smoke run. - flow-11: post-register Ready poll 120s -> 300s to match flow-14's identical live-Base-Sepolia chain-watch path (pitfall 13 free-tier RPC throttling). Known follow-ups (not in this commit): flow-08 buy-retry top-up vs exactly-N assertions on rare partial failures; flow-11 lacks flow-14's remote-signer rolled guard; aztec PVC has no permission story (runs as root today); post-merge controller repin so released sub-agents pick up this PR's UID-1000 render (tracked in #614's pin-test note).

Records the options ladder for 'nodes joining my cluster' (far-future, not scheduled) so near-term permission decisions don't foreclose it: the host-FS-canonical coupling and its four conditions, the agent-homes-only scope, what breaks on day one of a join (WaitForFirstConsumer + no nodeSelector in any render), options 0-4 (home-node pattern -> NFS -> distributed storage -> API-mediated access -> hybrid by data class), join mechanics (native k3s server + remote agents; macOS stays k3d), and the decisions that bind today: group-1000 sharing over render-time UID matching, home-node pinning as a join prerequisite, inputs to API objects per Remaining Debt.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

bussyjd · 2026-06-09T21:06:25Z

Summary

replace host-side chown/chmod init handling with UID/GID 1000 runtime ownership for Hermes and spawned agents
keep LiteLLM/x402 buyer routing in sync when model ConfigMaps change so paid routes roll correctly
harden release smoke flows around qwen36-deep, post-stack Bob wallet seeding, PurchaseRequest readiness, and paid inference settlement checks
update the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces

Validation

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
full release smoke passed with OBOL_LLM_MODEL=qwen36-deep, including USDC, live OBOL, and fork OBOL paid inference flows

Notes

This is stacked on #610's branch so reviewers can inspect only the follow-up fixes here.

bussyjd · 2026-06-09T21:57:29Z

Summary

What changed:

Replaced host-side chown/chmod init handling with UID/GID/fsGroup 1000 runtime ownership for Hermes and spawned agent workloads.
Ensured LiteLLM rolls when model or buyer ConfigMaps change, so paid routes are refreshed when the config changes without an env var change.
Hardened paid smoke flows around qwen36-deep, inference ServiceOffers, PurchaseRequest auth counts, LiteLLM readiness, and exact settlement deltas.
Moved Bob wallet setup in dual-stack flows to post-stack obol wallet import, then asserts the signer matches deterministic Bob.
Updated the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces.

Why it matters:

Solves the PVC permission problem without making config read-only and without host-side ownership mutation.
Reduces fragile permission scaffolding while keeping runtime writes available.
Keeps release-smoke aligned with the supported obol CLI, stack, and obol kubectl surfaces.

Risk level: medium

Commit under test: 1f3088b

Base branch: research/hermes-configmap-skills-delivery

Scope

Validation

CI checks:

Check	Status	Link
CI	not checked in this comment

Unit tests:

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
Result: PASS
Commit/SHA: 1f3088b

Integration tests:

No separate integration suite was run outside the release smoke.

Flow tests:

Flow	Network	QA machine label	Worktree	Result	Artifacts
`flow-01-prerequisites`	local	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-02-stack-init-up`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-03-inference`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-04-agent`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-05-network`	Base Sepolia RPC	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-06-sell-setup`	Base Sepolia x402 pricing	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-07-sell-verify`	x402 tunnel route	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-10-anvil-facilitator`	Base Sepolia Anvil fork	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-08-buy`	Base Sepolia Anvil fork / USDC	remote ARM64 QA host	redacted	PASS	receipt retained on QA host
`flow-09-lifecycle`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-11-dual-stack`	Base Sepolia / USDC	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-14-live-obol-base-sepolia`	live Base Sepolia / OBOL	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-13-dual-stack-obol`	Base Sepolia Anvil fork / OBOL	remote ARM64 QA host	redacted	PASS	receipt summary retained on QA host

Release smoke:

Command: flows/release-smoke.sh
Flags/env: OBOL_DEVELOPMENT=true, OBOL_NONINTERACTIVE=true, OBOL_FORCE_REBUILD_LOCAL_DEV_IMAGES=serviceoffer-controller, OBOL_LLM_MODEL=qwen36-deep, RELEASE_SMOKE_INCLUDE_OBOL=true, RELEASE_SMOKE_INCLUDE_OBOL_FORK=true
Endpoint/RPC/worktree details: redacted
Result: PASS

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

Base Sepolia
Base Sepolia Anvil fork

RPC/provider:

Paid Base Sepolia RPC, redacted
Local Anvil fork for forked flows

Facilitator:

Public Obol facilitator for live Base Sepolia flows, hostname omitted
Local x402-rs facilitator for fork flow

Contracts and tokens:

Name	Address	Version / notes
Base Sepolia OBOL	`0x0a09371a8b011d5110656ceBCc70603e53FD2c78`	live token
Base Sepolia USDC	`0x036CbD53842c5426634e7929541eC2318f3dCF7e`	live/fork token
ERC-8004 identity registry	`0x8004A818BFB912233c491871b3d84c89A494BD9e`	Base Sepolia
Fork OBOL	`0x210BBd033630e5e611B7922D70b0Caabe64636d9`	flow-13 deployed token

Wallet roles:

Role	Address	Source
Alice / seller / register	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	runtime-derived from configured signer
Bob / buyer / payer	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	deterministic second-derived buyer
Single-stack flow-08 seller	`0x70997970C51812dc3A010C7d01b50e0d17dc79C8`	local flow seller

Balances:

Token	Address	Before	After	Expected delta	Actual delta
Live OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`30000000000000000`	`31000000000000000`	`+1000000000000000`	`+1000000000000000`
Live OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`4947000000000000000`	`4946000000000000000`	`-1000000000000000`	`-1000000000000000`
Fork OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`10000000000000000000`	`10001000000000000000`	`+1000000000000000`	`+1000000000000000`
Fork OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`10000000000000000000`	`9999000000000000000`	`-1000000000000000`	`-1000000000000000`

Transaction receipts:

Purpose	Tx hash	From	To	Amount / event	Status
ERC-8004 registration	`0x8bc7bf9c5daaaa3c3df889ce7ee44f483593e0b79fd9126a34838314280e1310`	Alice	registry	Agent ID reflected in ServiceOffer	PASS
Metadata / service offer				Metadata tx unavailable, ServiceOffer status reflected Agent ID	SKIP
Live OBOL approval / permit	`0x51841c0cfcfd1b5e0b23d2976a755f8ed7d11489015e885a962bf3b1c7cd878c`	Bob	Permit2	Max allowance confirmed	PASS
Live OBOL settlement transfer	`0xb78fa7b9f1097fb88fd6e5fb6d04ffbfc03337f478691b0905eb4d70f0dc2981`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Fork OBOL buyer funding	`0x733a277dd44da427bd3749b955ab4ab77fe9dadbe94358c130afab2ccd4600de`	deployer	Bob	`10` OBOL minted	PASS
Fork OBOL approval / permit	`0x40dc8bc61cb2f4dbd31d532141ef1885f3efb29144ee2fc50f712dd265ffa3c8`	Bob	Permit2	Max allowance confirmed	PASS
Fork OBOL settlement transfer	`0xb0d8974636102f1be97383c4fb1ba8aa860ef7850084ce5d425e37c1016b114d`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Flow-08 USDC settlement transfer	`0x1e7fccd95dfd47e2720896406108f94d391d4f8d5824c37f9d3cdfdc1ca7f9c2`	buyer	seller	`1000` micro-USDC	PASS
Flow-11 USDC settlement transfer	`0x6127dbec3139eadc68b5343a2ccf552fcf82595c4cdeaf81c893d51ee74754df`	Bob	Alice	`1000` micro-USDC	PASS

Runtime Evidence

QA environment:

Item	Value
OS / arch	Linux / arm64 remote QA host
Backend	k3d
Tool versions	branch-built `obol` CLI, Foundry tools available
QA agent/model	`qwen36-deep` via redacted OpenAI-compatible endpoint

Images:

Component	Image	Tag / digest	Source
serviceoffer-controller	local dev image	rebuilt during smoke	branch source
x402 facilitator	`ghcr.io/obolnetwork/x402-facilitator-prometheus-overlay`	`1.4.9`	flow-13 preflight

Kubernetes / stack:

Item	Value
Stack IDs	omitted to avoid host-local details
Namespaces	`llm`, `x402`, `erpc`, `hermes-obol-agent`
Pod readiness	all smoke-gated pods Ready during flow checks
Cleanup result	no k3d clusters remaining after final smoke

Model and routing:

Item	Value
Agent/model used	`qwen36-deep`
LiteLLM route	`paid/qwen36-deep`
Paid endpoint status	HTTP 200 with non-empty response
Auth token source	`PurchaseRequest` plus x402-buyer sidecar, no raw token logged

Artifacts and logs:

Artifact	Location / link	Notes
Release smoke report	retained on QA host, path omitted	result PASS
Flow-11 receipts	retained on QA host, path omitted	USDC settlement
Flow-14 receipts	retained on QA host, path omitted	live OBOL settlement
Flow-13 receipts	retained on QA host, path omitted	fork OBOL settlement

Demo readiness:

Item	Status	Notes
Seller visible / registered	PASS	ServiceOffer Ready; Agent ID reflected where registration was enabled
Buyer discovery works	PASS	Buyer discovered service catalog / registry data
Paid route works	PASS	`paid/qwen36-deep` returned HTTP 200
Settlement visible on-chain	PASS	Transfer receipts and exact deltas verified

Review Notes

Known gaps:

flow-11 and flow-14 recorded one SKIP each for unavailable registration receipt while ServiceOffer status already reflected the Agent ID.
PR is stacked on research/hermes-configmap-skills-delivery, so reviewers should compare against that branch for the focused delta.

Follow-ups:

None required for the validated smoke path.

Reviewer focus:

UID/GID/fsGroup 1000 ownership model versus the removed host-side chown/chmod path.
LiteLLM ConfigMap reload behavior for paid routes.
Flow-08 retry/readiness behavior around obol buy inference and LiteLLM rollout.
Post-stack Bob wallet import in dual-stack flows.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

…with #614 Applies the verified findings from the cross-review against PR #614 (every item adversarially confirmed against the refs; union merge-tree clean). PVC / upgrade path: - llm.yaml: restore container-level runAsUser/runAsGroup 1000 on x402-buyer. Clusters upgraded in place from <= rc12 keep hostPath-typed PVs where kubelet skips fsGroup; their /state dir is 1000:1000 with consumed.json written 0600 by UID 1000 — a 65532 sidecar cannot read it, Fatalf's on `load state`, and takes every paid/<model> route down. On fresh local-type PVs the explicit UID is harmless (fsGroup 65532 grants group access). embed_buyer_state_test.go updated to pin the new contract. - plans/volume-permission-hardening.md: new "Upgrading from <= v0.10.0-rc12" section — supported path is cluster recreation (wallet backup/restore), with a documented k3d chown escape hatch. troubleshooting.md gets the symptom->fix entry. The Hermes half of the legacy-PV breakage cannot be patched at runtime without reintroducing the chown machinery this PR removes, so it is a documented breaking change instead. Paid-route availability: - llm.yaml: Reloader annotation narrowed to litellm-config only. The buyer ConfigMaps (x402-buyer-config/x402-buyer-auths) are rewritten by the controller on every buy, top-up, auto-refill, and tombstone cleanup; with strategy Recreate + 1 replica the previous annotation bounced the entire inference gateway (all Hermes traffic, in-flight SSE streams) on every purchase event, inverting CLAUDE.md pitfall 7 (restart is fallback, not the default buy path). The buyer hot-reloads via /admin/reload. stack_test.go updated to pin litellm-config-only. Flow alignment with #614: - lib.sh: `stack down` -> `stack down --yes` in reset_flow_workspace. #614's flow-16 (now last in the single-stack array) intentionally leaves a live agent offer; without --yes the non-TTY ConfirmRunningServicesLoss gate refuses, graceful down is silently skipped (`|| true`), and teardown degrades to the raw k3d-delete fallback on every release-smoke run. - flow-11: post-register Ready poll 120s -> 300s to match flow-14's identical live-Base-Sepolia chain-watch path (pitfall 13 free-tier RPC throttling). Known follow-ups (not in this commit): flow-08 buy-retry top-up vs exactly-N assertions on rare partial failures; flow-11 lacks flow-14's remote-signer rolled guard; aztec PVC has no permission story (runs as root today); post-merge controller repin so released sub-agents pick up this PR's UID-1000 render (tracked in #614's pin-test note).

Records the options ladder for 'nodes joining my cluster' (far-future, not scheduled) so near-term permission decisions don't foreclose it: the host-FS-canonical coupling and its four conditions, the agent-homes-only scope, what breaks on day one of a join (WaitForFirstConsumer + no nodeSelector in any render), options 0-4 (home-node pattern -> NFS -> distributed storage -> API-mediated access -> hybrid by data class), join mechanics (native k3s server + remote agents; macOS stays k3d), and the decisions that bind today: group-1000 sharing over render-time UID matching, home-node pinning as a join prerequisite, inputs to API objects per Remaining Debt.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

bussyjd · 2026-06-09T21:06:25Z

Summary

replace host-side chown/chmod init handling with UID/GID 1000 runtime ownership for Hermes and spawned agents
keep LiteLLM/x402 buyer routing in sync when model ConfigMaps change so paid routes roll correctly
harden release smoke flows around qwen36-deep, post-stack Bob wallet seeding, PurchaseRequest readiness, and paid inference settlement checks
update the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces

Validation

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
full release smoke passed with OBOL_LLM_MODEL=qwen36-deep, including USDC, live OBOL, and fork OBOL paid inference flows

Notes

This is stacked on #610's branch so reviewers can inspect only the follow-up fixes here.

bussyjd · 2026-06-09T21:57:29Z

Summary

What changed:

Replaced host-side chown/chmod init handling with UID/GID/fsGroup 1000 runtime ownership for Hermes and spawned agent workloads.
Ensured LiteLLM rolls when model or buyer ConfigMaps change, so paid routes are refreshed when the config changes without an env var change.
Hardened paid smoke flows around qwen36-deep, inference ServiceOffers, PurchaseRequest auth counts, LiteLLM readiness, and exact settlement deltas.
Moved Bob wallet setup in dual-stack flows to post-stack obol wallet import, then asserts the signer matches deterministic Bob.
Updated the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces.

Why it matters:

Solves the PVC permission problem without making config read-only and without host-side ownership mutation.
Reduces fragile permission scaffolding while keeping runtime writes available.
Keeps release-smoke aligned with the supported obol CLI, stack, and obol kubectl surfaces.

Risk level: medium

Commit under test: 1f3088b

Base branch: research/hermes-configmap-skills-delivery

Scope

Validation

CI checks:

Check	Status	Link
CI	not checked in this comment

Unit tests:

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
Result: PASS
Commit/SHA: 1f3088b

Integration tests:

No separate integration suite was run outside the release smoke.

Flow tests:

Flow	Network	QA machine label	Worktree	Result	Artifacts
`flow-01-prerequisites`	local	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-02-stack-init-up`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-03-inference`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-04-agent`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-05-network`	Base Sepolia RPC	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-06-sell-setup`	Base Sepolia x402 pricing	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-07-sell-verify`	x402 tunnel route	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-10-anvil-facilitator`	Base Sepolia Anvil fork	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-08-buy`	Base Sepolia Anvil fork / USDC	remote ARM64 QA host	redacted	PASS	receipt retained on QA host
`flow-09-lifecycle`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-11-dual-stack`	Base Sepolia / USDC	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-14-live-obol-base-sepolia`	live Base Sepolia / OBOL	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-13-dual-stack-obol`	Base Sepolia Anvil fork / OBOL	remote ARM64 QA host	redacted	PASS	receipt summary retained on QA host

Release smoke:

Command: flows/release-smoke.sh
Flags/env: OBOL_DEVELOPMENT=true, OBOL_NONINTERACTIVE=true, OBOL_FORCE_REBUILD_LOCAL_DEV_IMAGES=serviceoffer-controller, OBOL_LLM_MODEL=qwen36-deep, RELEASE_SMOKE_INCLUDE_OBOL=true, RELEASE_SMOKE_INCLUDE_OBOL_FORK=true
Endpoint/RPC/worktree details: redacted
Result: PASS

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

Base Sepolia
Base Sepolia Anvil fork

RPC/provider:

Paid Base Sepolia RPC, redacted
Local Anvil fork for forked flows

Facilitator:

Public Obol facilitator for live Base Sepolia flows, hostname omitted
Local x402-rs facilitator for fork flow

Contracts and tokens:

Name	Address	Version / notes
Base Sepolia OBOL	`0x0a09371a8b011d5110656ceBCc70603e53FD2c78`	live token
Base Sepolia USDC	`0x036CbD53842c5426634e7929541eC2318f3dCF7e`	live/fork token
ERC-8004 identity registry	`0x8004A818BFB912233c491871b3d84c89A494BD9e`	Base Sepolia
Fork OBOL	`0x210BBd033630e5e611B7922D70b0Caabe64636d9`	flow-13 deployed token

Wallet roles:

Role	Address	Source
Alice / seller / register	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	runtime-derived from configured signer
Bob / buyer / payer	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	deterministic second-derived buyer
Single-stack flow-08 seller	`0x70997970C51812dc3A010C7d01b50e0d17dc79C8`	local flow seller

Balances:

Token	Address	Before	After	Expected delta	Actual delta
Live OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`30000000000000000`	`31000000000000000`	`+1000000000000000`	`+1000000000000000`
Live OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`4947000000000000000`	`4946000000000000000`	`-1000000000000000`	`-1000000000000000`
Fork OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`10000000000000000000`	`10001000000000000000`	`+1000000000000000`	`+1000000000000000`
Fork OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`10000000000000000000`	`9999000000000000000`	`-1000000000000000`	`-1000000000000000`

Transaction receipts:

Purpose	Tx hash	From	To	Amount / event	Status
ERC-8004 registration	`0x8bc7bf9c5daaaa3c3df889ce7ee44f483593e0b79fd9126a34838314280e1310`	Alice	registry	Agent ID reflected in ServiceOffer	PASS
Metadata / service offer				Metadata tx unavailable, ServiceOffer status reflected Agent ID	SKIP
Live OBOL approval / permit	`0x51841c0cfcfd1b5e0b23d2976a755f8ed7d11489015e885a962bf3b1c7cd878c`	Bob	Permit2	Max allowance confirmed	PASS
Live OBOL settlement transfer	`0xb78fa7b9f1097fb88fd6e5fb6d04ffbfc03337f478691b0905eb4d70f0dc2981`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Fork OBOL buyer funding	`0x733a277dd44da427bd3749b955ab4ab77fe9dadbe94358c130afab2ccd4600de`	deployer	Bob	`10` OBOL minted	PASS
Fork OBOL approval / permit	`0x40dc8bc61cb2f4dbd31d532141ef1885f3efb29144ee2fc50f712dd265ffa3c8`	Bob	Permit2	Max allowance confirmed	PASS
Fork OBOL settlement transfer	`0xb0d8974636102f1be97383c4fb1ba8aa860ef7850084ce5d425e37c1016b114d`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Flow-08 USDC settlement transfer	`0x1e7fccd95dfd47e2720896406108f94d391d4f8d5824c37f9d3cdfdc1ca7f9c2`	buyer	seller	`1000` micro-USDC	PASS
Flow-11 USDC settlement transfer	`0x6127dbec3139eadc68b5343a2ccf552fcf82595c4cdeaf81c893d51ee74754df`	Bob	Alice	`1000` micro-USDC	PASS

Runtime Evidence

QA environment:

Item	Value
OS / arch	Linux / arm64 remote QA host
Backend	k3d
Tool versions	branch-built `obol` CLI, Foundry tools available
QA agent/model	`qwen36-deep` via redacted OpenAI-compatible endpoint

Images:

Component	Image	Tag / digest	Source
serviceoffer-controller	local dev image	rebuilt during smoke	branch source
x402 facilitator	`ghcr.io/obolnetwork/x402-facilitator-prometheus-overlay`	`1.4.9`	flow-13 preflight

Kubernetes / stack:

Item	Value
Stack IDs	omitted to avoid host-local details
Namespaces	`llm`, `x402`, `erpc`, `hermes-obol-agent`
Pod readiness	all smoke-gated pods Ready during flow checks
Cleanup result	no k3d clusters remaining after final smoke

Model and routing:

Item	Value
Agent/model used	`qwen36-deep`
LiteLLM route	`paid/qwen36-deep`
Paid endpoint status	HTTP 200 with non-empty response
Auth token source	`PurchaseRequest` plus x402-buyer sidecar, no raw token logged

Artifacts and logs:

Artifact	Location / link	Notes
Release smoke report	retained on QA host, path omitted	result PASS
Flow-11 receipts	retained on QA host, path omitted	USDC settlement
Flow-14 receipts	retained on QA host, path omitted	live OBOL settlement
Flow-13 receipts	retained on QA host, path omitted	fork OBOL settlement

Demo readiness:

Item	Status	Notes
Seller visible / registered	PASS	ServiceOffer Ready; Agent ID reflected where registration was enabled
Buyer discovery works	PASS	Buyer discovered service catalog / registry data
Paid route works	PASS	`paid/qwen36-deep` returned HTTP 200
Settlement visible on-chain	PASS	Transfer receipts and exact deltas verified

Review Notes

Known gaps:

flow-11 and flow-14 recorded one SKIP each for unavailable registration receipt while ServiceOffer status already reflected the Agent ID.
PR is stacked on research/hermes-configmap-skills-delivery, so reviewers should compare against that branch for the focused delta.

Follow-ups:

None required for the validated smoke path.

Reviewer focus:

UID/GID/fsGroup 1000 ownership model versus the removed host-side chown/chmod path.
LiteLLM ConfigMap reload behavior for paid routes.
Flow-08 retry/readiness behavior around obol buy inference and LiteLLM rollout.
Post-stack Bob wallet import in dual-stack flows.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

…with #614 Applies the verified findings from the cross-review against PR #614 (every item adversarially confirmed against the refs; union merge-tree clean). PVC / upgrade path: - llm.yaml: restore container-level runAsUser/runAsGroup 1000 on x402-buyer. Clusters upgraded in place from <= rc12 keep hostPath-typed PVs where kubelet skips fsGroup; their /state dir is 1000:1000 with consumed.json written 0600 by UID 1000 — a 65532 sidecar cannot read it, Fatalf's on `load state`, and takes every paid/<model> route down. On fresh local-type PVs the explicit UID is harmless (fsGroup 65532 grants group access). embed_buyer_state_test.go updated to pin the new contract. - plans/volume-permission-hardening.md: new "Upgrading from <= v0.10.0-rc12" section — supported path is cluster recreation (wallet backup/restore), with a documented k3d chown escape hatch. troubleshooting.md gets the symptom->fix entry. The Hermes half of the legacy-PV breakage cannot be patched at runtime without reintroducing the chown machinery this PR removes, so it is a documented breaking change instead. Paid-route availability: - llm.yaml: Reloader annotation narrowed to litellm-config only. The buyer ConfigMaps (x402-buyer-config/x402-buyer-auths) are rewritten by the controller on every buy, top-up, auto-refill, and tombstone cleanup; with strategy Recreate + 1 replica the previous annotation bounced the entire inference gateway (all Hermes traffic, in-flight SSE streams) on every purchase event, inverting CLAUDE.md pitfall 7 (restart is fallback, not the default buy path). The buyer hot-reloads via /admin/reload. stack_test.go updated to pin litellm-config-only. Flow alignment with #614: - lib.sh: `stack down` -> `stack down --yes` in reset_flow_workspace. #614's flow-16 (now last in the single-stack array) intentionally leaves a live agent offer; without --yes the non-TTY ConfirmRunningServicesLoss gate refuses, graceful down is silently skipped (`|| true`), and teardown degrades to the raw k3d-delete fallback on every release-smoke run. - flow-11: post-register Ready poll 120s -> 300s to match flow-14's identical live-Base-Sepolia chain-watch path (pitfall 13 free-tier RPC throttling). Known follow-ups (not in this commit): flow-08 buy-retry top-up vs exactly-N assertions on rare partial failures; flow-11 lacks flow-14's remote-signer rolled guard; aztec PVC has no permission story (runs as root today); post-merge controller repin so released sub-agents pick up this PR's UID-1000 render (tracked in #614's pin-test note).

Records the options ladder for 'nodes joining my cluster' (far-future, not scheduled) so near-term permission decisions don't foreclose it: the host-FS-canonical coupling and its four conditions, the agent-homes-only scope, what breaks on day one of a join (WaitForFirstConsumer + no nodeSelector in any render), options 0-4 (home-node pattern -> NFS -> distributed storage -> API-mediated access -> hybrid by data class), join mechanics (native k3s server + remote agents; macOS stays k3d), and the decisions that bind today: group-1000 sharing over render-time UID matching, home-node pinning as a join prerequisite, inputs to API objects per Remaining Debt.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

bussyjd · 2026-06-09T21:06:25Z

Summary

replace host-side chown/chmod init handling with UID/GID 1000 runtime ownership for Hermes and spawned agents
keep LiteLLM/x402 buyer routing in sync when model ConfigMaps change so paid routes roll correctly
harden release smoke flows around qwen36-deep, post-stack Bob wallet seeding, PurchaseRequest readiness, and paid inference settlement checks
update the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces

Validation

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
full release smoke passed with OBOL_LLM_MODEL=qwen36-deep, including USDC, live OBOL, and fork OBOL paid inference flows

Notes

This is stacked on #610's branch so reviewers can inspect only the follow-up fixes here.

bussyjd · 2026-06-09T21:57:29Z

Summary

What changed:

Replaced host-side chown/chmod init handling with UID/GID/fsGroup 1000 runtime ownership for Hermes and spawned agent workloads.
Ensured LiteLLM rolls when model or buyer ConfigMaps change, so paid routes are refreshed when the config changes without an env var change.
Hardened paid smoke flows around qwen36-deep, inference ServiceOffers, PurchaseRequest auth counts, LiteLLM readiness, and exact settlement deltas.
Moved Bob wallet setup in dual-stack flows to post-stack obol wallet import, then asserts the signer matches deterministic Bob.
Updated the Obol stack dev runbook to keep QA CLI-first and avoid custom shell surfaces.

Why it matters:

Solves the PVC permission problem without making config read-only and without host-side ownership mutation.
Reduces fragile permission scaffolding while keeping runtime writes available.
Keeps release-smoke aligned with the supported obol CLI, stack, and obol kubectl surfaces.

Risk level: medium

Commit under test: 1f3088b

Base branch: research/hermes-configmap-skills-delivery

Scope

Validation

CI checks:

Check	Status	Link
CI	not checked in this comment

Unit tests:

bash -n flows/*.sh
git diff --check
go test ./cmd/obol ./internal/buy ./internal/hermes ./internal/serviceoffercontroller ./internal/embed ./internal/stack -count=1
Result: PASS
Commit/SHA: 1f3088b

Integration tests:

No separate integration suite was run outside the release smoke.

Flow tests:

Flow	Network	QA machine label	Worktree	Result	Artifacts
`flow-01-prerequisites`	local	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-02-stack-init-up`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-03-inference`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-04-agent`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-05-network`	Base Sepolia RPC	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-06-sell-setup`	Base Sepolia x402 pricing	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-07-sell-verify`	x402 tunnel route	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-10-anvil-facilitator`	Base Sepolia Anvil fork	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-08-buy`	Base Sepolia Anvil fork / USDC	remote ARM64 QA host	redacted	PASS	receipt retained on QA host
`flow-09-lifecycle`	local k3d	remote ARM64 QA host	redacted	PASS	release-smoke report retained on QA host
`flow-11-dual-stack`	Base Sepolia / USDC	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-14-live-obol-base-sepolia`	live Base Sepolia / OBOL	remote ARM64 QA host	redacted	SKIP only for optional registration receipt	receipt summary retained on QA host
`flow-13-dual-stack-obol`	Base Sepolia Anvil fork / OBOL	remote ARM64 QA host	redacted	PASS	receipt summary retained on QA host

Release smoke:

Command: flows/release-smoke.sh
Flags/env: OBOL_DEVELOPMENT=true, OBOL_NONINTERACTIVE=true, OBOL_FORCE_REBUILD_LOCAL_DEV_IMAGES=serviceoffer-controller, OBOL_LLM_MODEL=qwen36-deep, RELEASE_SMOKE_INCLUDE_OBOL=true, RELEASE_SMOKE_INCLUDE_OBOL_FORK=true
Endpoint/RPC/worktree details: redacted
Result: PASS

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

Base Sepolia
Base Sepolia Anvil fork

RPC/provider:

Paid Base Sepolia RPC, redacted
Local Anvil fork for forked flows

Facilitator:

Public Obol facilitator for live Base Sepolia flows, hostname omitted
Local x402-rs facilitator for fork flow

Contracts and tokens:

Name	Address	Version / notes
Base Sepolia OBOL	`0x0a09371a8b011d5110656ceBCc70603e53FD2c78`	live token
Base Sepolia USDC	`0x036CbD53842c5426634e7929541eC2318f3dCF7e`	live/fork token
ERC-8004 identity registry	`0x8004A818BFB912233c491871b3d84c89A494BD9e`	Base Sepolia
Fork OBOL	`0x210BBd033630e5e611B7922D70b0Caabe64636d9`	flow-13 deployed token

Wallet roles:

Role	Address	Source
Alice / seller / register	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	runtime-derived from configured signer
Bob / buyer / payer	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	deterministic second-derived buyer
Single-stack flow-08 seller	`0x70997970C51812dc3A010C7d01b50e0d17dc79C8`	local flow seller

Balances:

Token	Address	Before	After	Expected delta	Actual delta
Live OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`30000000000000000`	`31000000000000000`	`+1000000000000000`	`+1000000000000000`
Live OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`4947000000000000000`	`4946000000000000000`	`-1000000000000000`	`-1000000000000000`
Fork OBOL, Alice	`0xC0De030F6C37f490594F93fB99e2756703c4297E`	`10000000000000000000`	`10001000000000000000`	`+1000000000000000`	`+1000000000000000`
Fork OBOL, Bob	`0x57b0eF875DeB5A37301F1640E469a2129Da9490E`	`10000000000000000000`	`9999000000000000000`	`-1000000000000000`	`-1000000000000000`

Transaction receipts:

Purpose	Tx hash	From	To	Amount / event	Status
ERC-8004 registration	`0x8bc7bf9c5daaaa3c3df889ce7ee44f483593e0b79fd9126a34838314280e1310`	Alice	registry	Agent ID reflected in ServiceOffer	PASS
Metadata / service offer				Metadata tx unavailable, ServiceOffer status reflected Agent ID	SKIP
Live OBOL approval / permit	`0x51841c0cfcfd1b5e0b23d2976a755f8ed7d11489015e885a962bf3b1c7cd878c`	Bob	Permit2	Max allowance confirmed	PASS
Live OBOL settlement transfer	`0xb78fa7b9f1097fb88fd6e5fb6d04ffbfc03337f478691b0905eb4d70f0dc2981`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Fork OBOL buyer funding	`0x733a277dd44da427bd3749b955ab4ab77fe9dadbe94358c130afab2ccd4600de`	deployer	Bob	`10` OBOL minted	PASS
Fork OBOL approval / permit	`0x40dc8bc61cb2f4dbd31d532141ef1885f3efb29144ee2fc50f712dd265ffa3c8`	Bob	Permit2	Max allowance confirmed	PASS
Fork OBOL settlement transfer	`0xb0d8974636102f1be97383c4fb1ba8aa860ef7850084ce5d425e37c1016b114d`	Bob	Alice	`1000000000000000` wei OBOL	PASS
Flow-08 USDC settlement transfer	`0x1e7fccd95dfd47e2720896406108f94d391d4f8d5824c37f9d3cdfdc1ca7f9c2`	buyer	seller	`1000` micro-USDC	PASS
Flow-11 USDC settlement transfer	`0x6127dbec3139eadc68b5343a2ccf552fcf82595c4cdeaf81c893d51ee74754df`	Bob	Alice	`1000` micro-USDC	PASS

Runtime Evidence

QA environment:

Item	Value
OS / arch	Linux / arm64 remote QA host
Backend	k3d
Tool versions	branch-built `obol` CLI, Foundry tools available
QA agent/model	`qwen36-deep` via redacted OpenAI-compatible endpoint

Images:

Component	Image	Tag / digest	Source
serviceoffer-controller	local dev image	rebuilt during smoke	branch source
x402 facilitator	`ghcr.io/obolnetwork/x402-facilitator-prometheus-overlay`	`1.4.9`	flow-13 preflight

Kubernetes / stack:

Item	Value
Stack IDs	omitted to avoid host-local details
Namespaces	`llm`, `x402`, `erpc`, `hermes-obol-agent`
Pod readiness	all smoke-gated pods Ready during flow checks
Cleanup result	no k3d clusters remaining after final smoke

Model and routing:

Item	Value
Agent/model used	`qwen36-deep`
LiteLLM route	`paid/qwen36-deep`
Paid endpoint status	HTTP 200 with non-empty response
Auth token source	`PurchaseRequest` plus x402-buyer sidecar, no raw token logged

Artifacts and logs:

Artifact	Location / link	Notes
Release smoke report	retained on QA host, path omitted	result PASS
Flow-11 receipts	retained on QA host, path omitted	USDC settlement
Flow-14 receipts	retained on QA host, path omitted	live OBOL settlement
Flow-13 receipts	retained on QA host, path omitted	fork OBOL settlement

Demo readiness:

Item	Status	Notes
Seller visible / registered	PASS	ServiceOffer Ready; Agent ID reflected where registration was enabled
Buyer discovery works	PASS	Buyer discovered service catalog / registry data
Paid route works	PASS	`paid/qwen36-deep` returned HTTP 200
Settlement visible on-chain	PASS	Transfer receipts and exact deltas verified

Review Notes

Known gaps:

flow-11 and flow-14 recorded one SKIP each for unavailable registration receipt while ServiceOffer status already reflected the Agent ID.
PR is stacked on research/hermes-configmap-skills-delivery, so reviewers should compare against that branch for the focused delta.

Follow-ups:

None required for the validated smoke path.

Reviewer focus:

UID/GID/fsGroup 1000 ownership model versus the removed host-side chown/chmod path.
LiteLLM ConfigMap reload behavior for paid routes.
Flow-08 retry/readiness behavior around obol buy inference and LiteLLM rollout.
Post-stack Bob wallet import in dual-stack flows.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).

…with #614 Applies the verified findings from the cross-review against PR #614 (every item adversarially confirmed against the refs; union merge-tree clean). PVC / upgrade path: - llm.yaml: restore container-level runAsUser/runAsGroup 1000 on x402-buyer. Clusters upgraded in place from <= rc12 keep hostPath-typed PVs where kubelet skips fsGroup; their /state dir is 1000:1000 with consumed.json written 0600 by UID 1000 — a 65532 sidecar cannot read it, Fatalf's on `load state`, and takes every paid/<model> route down. On fresh local-type PVs the explicit UID is harmless (fsGroup 65532 grants group access). embed_buyer_state_test.go updated to pin the new contract. - plans/volume-permission-hardening.md: new "Upgrading from <= v0.10.0-rc12" section — supported path is cluster recreation (wallet backup/restore), with a documented k3d chown escape hatch. troubleshooting.md gets the symptom->fix entry. The Hermes half of the legacy-PV breakage cannot be patched at runtime without reintroducing the chown machinery this PR removes, so it is a documented breaking change instead. Paid-route availability: - llm.yaml: Reloader annotation narrowed to litellm-config only. The buyer ConfigMaps (x402-buyer-config/x402-buyer-auths) are rewritten by the controller on every buy, top-up, auto-refill, and tombstone cleanup; with strategy Recreate + 1 replica the previous annotation bounced the entire inference gateway (all Hermes traffic, in-flight SSE streams) on every purchase event, inverting CLAUDE.md pitfall 7 (restart is fallback, not the default buy path). The buyer hot-reloads via /admin/reload. stack_test.go updated to pin litellm-config-only. Flow alignment with #614: - lib.sh: `stack down` -> `stack down --yes` in reset_flow_workspace. #614's flow-16 (now last in the single-stack array) intentionally leaves a live agent offer; without --yes the non-TTY ConfirmRunningServicesLoss gate refuses, graceful down is silently skipped (`|| true`), and teardown degrades to the raw k3d-delete fallback on every release-smoke run. - flow-11: post-register Ready poll 120s -> 300s to match flow-14's identical live-Base-Sepolia chain-watch path (pitfall 13 free-tier RPC throttling). Known follow-ups (not in this commit): flow-08 buy-retry top-up vs exactly-N assertions on rare partial failures; flow-11 lacks flow-14's remote-signer rolled guard; aztec PVC has no permission story (runs as root today); post-merge controller repin so released sub-agents pick up this PR's UID-1000 render (tracked in #614's pin-test note).

Records the options ladder for 'nodes joining my cluster' (far-future, not scheduled) so near-term permission decisions don't foreclose it: the host-FS-canonical coupling and its four conditions, the agent-homes-only scope, what breaks on day one of a join (WaitForFirstConsumer + no nodeSelector in any render), options 0-4 (home-node pattern -> NFS -> distributed storage -> API-mediated access -> hybrid by data class), join mechanics (native k3s server + remote agents; macOS stays k3d), and the decisions that bind today: group-1000 sharing over render-time UID matching, home-node pinning as a join prerequisite, inputs to API objects per Remaining Debt.

Removed plans whose work landed and whose durable learnings already live in the skill references: - obol-sell-demo.md — implemented (`obol sell demo` ships in the CLI) - post-490-integration-20260513.md — integration landed in May - release-smoke-hardening-20260513.md — session retro; learnings folded into obol-stack-dev/references/release-smoke-debugging.md - inference-v1337-{buy-report,followup}-20260514.md — QA reports; the WAF/UA findings live in release-smoke-debugging.md §10 (c2dddc1) Kept: sell-agent-perf.md (in progress), openapi-402-followups.md (pending walk-through), openapi-redoc-storefront.md (phase 2 deferred), storefront-buy-inference-cta-handoff.md (active frontend handoff), volume-permission-hardening.md (PR #615 owns its rewrite).