feat: agent purchase type within hermes

[OisinKyne]

Summary

Hardens the x402 paid-flow stack after rc13 QA caught two distinct issues: stale image
pins that bypassed merged source fixes, and a silent-money-loss path where on-chain
settlement landed but the buyer reported "0 spent." Also adds the missing buyer side for
type:agent ServiceOffers (streaming SSE), plumbs spec.payment.maxTimeoutSeconds
through to the 402 response (previously dropped on the floor), and bumps the Hermes
agent image to pick up the upstream sync_skills() marker-honoring fix.

Driven by plans/rc13report.md (Kody, 2026-06-09); learnings folded into
docs/observability.md and obol-stack-dev/references/release-smoke-debugging.md so
the next debugging session doesn't relearn this from a cold start.

Changes

1. Image pins → 04bebbc (current main HEAD as of branching)

• internal/embed/infrastructure/base/templates/{x402,llm}.yaml
• Picks up ab71481 (suppress per-request verifyOnly=false log spam on the in-process
  settle path) and 86b8c9f (buyer drops expired pre-signed auths before signing). Both
  shipped in source for weeks but were not reaching the deployed cluster because the
  embedded pins lagged.
• Allowlist tests (embed_image_pin_test.go, embed_crd_test.go) updated.

2. maxTimeoutSeconds plumbed end-to-end

• ServiceOffer.spec.payment.maxTimeoutSeconds → RouteRule.MaxTimeoutSeconds →
  BuildV2RequirementWithAsset → the 402 wire response.
• Default raised from hardcoded 60s → 600s (DefaultMaxTimeoutSeconds); operators
  can set any value up to 86400s (24h) via MaxMaxTimeoutSeconds cap.
• Streaming agent flows can legitimately take minutes-to-hours; the cap is permissive
  because the auth is nonce-bound and replay-safe — the cap exists only to guard against
  operator typos (e.g. ms→s).
• Regression tests: TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds,
  TestClampMaxTimeoutSeconds, TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds.

3. New buy.py command: pay-agent (streaming SSE for type:agent offers)

• Sibling of pay, deliberately not an alias of --type inference. inference
  pushes a paid/<model> alias behind LiteLLM; agent responses are first-class data the
  calling agent wants to consume itself (memory, tool-call traces, partial results), so
  no LiteLLM wire-up.
• Forces "stream": true, sends Accept: text/event-stream, reads response
  line-by-line, flushes each SSE event to stdout so a calling Hermes/OpenClaw can re-emit
  the stream to its own user (telegram, obol hermes chat, REST clients).
• Default HTTP read timeout 1 hour; operator-overridable via --timeout.
• obol buy inference now emits a clear pointer at pay-agent instead of just refusing
  type=agent offers (internal/buy/discover.go).
• (Researched Hermes' ACP mode — JSON-RPC over stdio for editor integration, wrong layer
  for this. The seller already streams OpenAI-compatible SSE through HandleProxy; the
  buyer just needed to consume it.)

4. rc13 headline fix: surface on-chain tx hash on facilitator /settle 5xx

QA caught a 0.001 OBOL mainnet on-chain debit from a request that returned HTTP 503 "Payment settlement failed" while every signal the stack produced said "nothing was
paid." Permit2 settle tx [0xb5122d…932307](https://etherscan.io/tx/0xb5122d818a058e8bf
529380260fa2584ba3d50bfc800f1e906faca34d3932307) mined; the facilitator's post-submit
step then returned 500.

This PR doesn't fix the facilitator (that's the hosted x402.gcp.obol.tech service),
but closes the client-side robustness gap that turned a facilitator hiccup into
silent money loss:

• Verifier (internal/x402/forwardauth.go): facilitatorSettle now returns the
  parsed response alongside the error on non-200, and the settle interceptor writes
  X-PAYMENT-RESPONSE with the tx hash before committing the 503. Without this the
  on-chain hash was invisible to the buyer.
• Buyer sidecar (internal/x402/buyer/proxy.go): an upstream 5xx with
  X-PAYMENT-RESPONSE.transaction != "" now triggers ConfirmSpend on the held auth (it
  was spent on-chain), fires OnPaymentUnsettled, and logs the hash + payer + network.
  Previously the auth was released back into the pool, leaving the spent counter
  permanently off and the next reuse attempt to fail on Permit2 nonce-reuse revert.
• buy.py (internal/embed/skills/buy-x402/scripts/buy.py):
  _print_paid_request_failure decodes the settle header on 5xx and prints a loud ⚠️ SETTLEMENT MAY HAVE COMPLETED ON-CHAIN warning with the tx hash and the exact balance --chain <X> command.
• Regression tests pin the contract on both sides:
  TestForwardAuth_SettleErrorPreservesTxHashInHeader (verifier) and
  TestProxy_UpstreamErrorWithTxHash_PersistsConsume (buyer).

5. Hermes image bump: v2026.5.28 → v2026.6.5

Closes a separate sub-agent context-bloat bug uncovered while triaging an older
sub-agent during the rc13 session: v2026.5.28's sync_skills() had no
.no-bundled-skills marker check (tools/skills_sync.py:419), so every Hermes pod boot
copied ~24 stock skill categories (~1 MB of SKILL.md text, ~100k tokens of prompt-scan
bloat) from the image-baked /opt/hermes/skills into the PVC — regardless of the marker
our SeedHostFiles writes at agent-create time. Affected every CRD sub-agent, not
just stale PVCs.

Upstream fix landed in v2026.6.5 as commit 2ed96372a ("blank-slate skills"):
sync_skills() now early-returns when the marker exists (tools/skills_sync.py:467).
Verified by reading both tags' source. No containerEnv changes needed — the marker our
code already writes is now honored end-to-end.

• internal/serviceoffercontroller/agent_render.go (sub-agent default)
• internal/hermes/hermes.go (master agent default)
• internal/agentcrd/agent_contract_integration_test.go doc comment updated with the
  v2026.5.28 → v2026.6.5 root cause + upstream commit reference.

Also picks up unrelated fixes between 5.28 and 6.5: gateway max_tokens propagation
chain, multi-profile remote-dashboard sessions, several smaller items.

6. Debugging-lore additions

• docs/observability.md: new § "Verify settlement against the chain, never the sidecar
  snapshot" with the operator debugging recipe (eth_getLogs curl) and the rc13 tx hash
  as evidence.
• .agents/skills/obol-stack-dev/references/release-smoke-debugging.md § 11: same
  lesson at the runbook surface so future debugging sessions don't have to rediscover it
  from 0 spent / 2 remaining.

Follow-ups (NOT in this PR — flagged for separate work)

These are deliberately left for separate PRs because they each have non-trivial scope or
live outside this repo. Tracked here so they don't get lost:

1. Image-pin re-bump after merge. This PR's source changes don't take effect in
   deployed clusters until CI publishes new x402-verifier / x402-buyer images tagged
   with the merge-commit short SHA. A one-line follow-up to
   internal/embed/infrastructure/base/templates/{x402,llm}.yaml will pin those. Until
   then the cluster sees the 04bebbc pin, which has the ab71481 and 86b8c9f fixes but
   not the maxTimeoutSeconds plumb or the settle-tx-hash forensic fix.
   2. Wire the integration test into CI.
   TestIntegration_AgentContract_HonorsRenderedConstraints (//go:build integration, in
   internal/agentcrd/agent_contract_integration_test.go) verifies the no-bundled-skills
   contract end-to-end on a live cluster, but isn't referenced in any .github/workflows/
   or flows/release-smoke.sh. Manual-only invocation means rc13's bloat went undetected
   for weeks. Add it to release-smoke (or a Hermes-bump gate).
2. Renovate annotation on the Hermes pin. Two defaultImage / defaultHermesImage
   constants in this repo currently track the Hermes release cadence by hand. A Renovate # renovate: ... annotation would surface future bumps automatically — sized as a 2-line
   follow-up.
3. Stale /data/.hermes/skills cleanup on pre-b121f99 PVCs. Agents created before
   the streamline work (June 2) have ~/.hermes/skills/ already populated; even with the
   v2026.6.5 marker honoring, that dir is already on disk and load_skills() reads it.
   Manual fix per agent: kubectl exec -n <ns> deploy/hermes -- mv /data/.hermes/skills /data/.hermes/skills.bundled-disabled-<ts> + kubectl rollout restart deploy/hermes -n <ns>. Not worth code in this repo — strictly an upgrade-migration concern for known
   existing deployments.
4. Full on-chain receipt verification in the verifier. Today this PR exposes the tx
   hash so an operator can reconcile manually. The bigger fix is to query an in-cluster RPC
   (eRPC) for the receipt status before deciding 200 vs 5xx — when settle lands on-chain,
   the verifier could serve the upstream response and report success. Needs RPC client
   plumbed into the verifier with per-network config; sized as its own PR.
5. Settle idempotency on retry. A retried request currently relies on Permit2 nonce
   reuse reverting on-chain to prevent double-debit — that works but burns gas and surfaces
   as cascading 503s. Proper fix: an auth state machine with PENDING → SETTLED
   transitions guarded by tx receipt confirmation.
6. Facilitator-side post-submit bug. Why does mainnet OBOL /settle return 500
   after a successful on-chain submit while Base Sepolia USDC settles cleanly? Out of scope
   (hosted x402.gcp.obol.tech), but the receipt-tx for the reproducible failure is in
   plans/rc13report.md.
7. rc12 punch list still confirmed in rc13 (not touched by this PR; recorded in
   plans/rc13report.md):
   • obol wallet import re-breaks Hermes ownership (fix = rollout restart master).
   • Purchase-delete hangs on unconsumed auths; no --force / abandon path.
   • purge data-dir cleanup on headless sudo fails (cleaned manually via root
     container).
8. Permit2 allowance-drain investigation (carried over from a prior session): does
   the OBOL token's _spendAllowance special-case type(uint256).max? If it doesn't,
   every settled transfer decrements a "max" allowance until 0, and the fix is in periodic
   re-approval, not cmd_pay pre-flights.

Test plan

• go test ./... clean.
• New regression tests in this PR all pass:
  • TestForwardAuth_SettleErrorPreservesTxHashInHeader
  • TestProxy_UpstreamErrorWithTxHash_PersistsConsume
  • TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds
  • TestClampMaxTimeoutSeconds,
    TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds
• python3 buy.py pay-agent shows usage and exits 1; python3 buy.py help text
  lists the new command.
• Verified upstream Hermes source (../NousResearch/hermes-agent): v2026.5.28
sync_skills has no marker check; v2026.6.5 early-returns on the marker at
  tools/skills_sync.py:467 (commit 2ed96372a).
• Manual after merge + image republish: repeat the rc13 mainnet OBOL cycle
  (obol sell inference → curl with X-PAYMENT) and verify that a forced facilitator 500
  (if reproducible) now surfaces the tx hash to the buyer and accounts the auth as spent.
• Manual after merge + image republish: confirm maxTimeoutSeconds: 1800 in a
  ServiceOffer reaches the 402 wire response (curl the offer endpoint, decode the
  accepts[].maxTimeoutSeconds field).
• Manual after Hermes v2026.6.5 image pull: run
  TestIntegration_AgentContract_HonorsRenderedConstraints (go test -tags integration -v -run TestIntegration_AgentContract_HonorsRenderedConstraints ./internal/agentcrd/) on
  a fresh cluster and confirm /data/.hermes/skills is empty after pod ready.
• obol buy inference <type=agent-offer> returns the new actionable error pointing
  at pay-agent.

Implemented #2 + #3

#3 — Renovate annotation (3 files):

• internal/hermes/hermes.go: // renovate: datasource=docker
  depName=nousresearch/hermes-agent above defaultImage
• internal/serviceoffercontroller/agent_render.go: same annotation above
  defaultHermesImage
• renovate.json:
  • New customManager regex that captures the version after the last : from
    hermes-agent:vX.Y.Z strings (verified against both files, doesn't trip on the existing
    rawChartVersion helm annotation).
  • New packageRule grouping nousresearch/hermes-agent bumps with a labelled,
    Monday-scheduled PR template that includes a pre-merge gate explicitly reminding the
    reviewer to run release-smoke (flow-16 + flow-17 mentions).
  • Major-version-update gate that requires dashboard approval — Hermes majors have
    historically changed skill layout / marker semantics, exactly the class of regression
    that broke v2026.5.28.

#2 — CI wiring (3 files):

• flows/flow-16-sell-agent.sh: 3 new assertions mirroring the Go integration test —
  • .no-bundled-skills marker present on host PVC,
  • Pod-side /data/.hermes/obol-skills populated with the operator subset,
  • Pod-side /data/.hermes/skills absent or empty (the load-bearing check that would
    have caught the v2026.5.28 bloat).
• flows/release-smoke.sh: flow-16-sell-agent.sh added to the main flow sequence with a
  comment explaining why it's now release-gating.
• internal/agentcrd/agent_contract_integration_test.go: doc comment updated to call out
  that flow-16 now provides CI coverage of the same invariants — Go test is white-box
  developer surface, flow-16 is the black-box CI gate.

Verification:

• bash -n on both flow scripts — clean.
• python3 -m json.tool renovate.json — valid.
• Regex sanity test against both Go files matches exactly once each, captures
  currentValue=v2026.6.5, doesn't false-match the existing rawChartVersion = "2.0.2"
  annotation.
• go build ./... and go test ./... — clean.

[bussyjd]

Superseded by the v0.10.0-rc14 release train: #616 — this PR's branch is merged there verbatim (merge commit preserved, full union build + test green). Closing in favor of the consolidated release PR.

[OisinKyne]

Summary

Hardens the x402 paid-flow stack after rc13 QA caught two distinct issues: stale image
pins that bypassed merged source fixes, and a silent-money-loss path where on-chain
settlement landed but the buyer reported "0 spent." Also adds the missing buyer side for
type:agent ServiceOffers (streaming SSE), plumbs spec.payment.maxTimeoutSeconds
through to the 402 response (previously dropped on the floor), and bumps the Hermes
agent image to pick up the upstream sync_skills() marker-honoring fix.

Driven by plans/rc13report.md (Kody, 2026-06-09); learnings folded into
docs/observability.md and obol-stack-dev/references/release-smoke-debugging.md so
the next debugging session doesn't relearn this from a cold start.

Changes

1. Image pins → 04bebbc (current main HEAD as of branching)

• internal/embed/infrastructure/base/templates/{x402,llm}.yaml
• Picks up ab71481 (suppress per-request verifyOnly=false log spam on the in-process
  settle path) and 86b8c9f (buyer drops expired pre-signed auths before signing). Both
  shipped in source for weeks but were not reaching the deployed cluster because the
  embedded pins lagged.
• Allowlist tests (embed_image_pin_test.go, embed_crd_test.go) updated.

2. maxTimeoutSeconds plumbed end-to-end

• ServiceOffer.spec.payment.maxTimeoutSeconds → RouteRule.MaxTimeoutSeconds →
  BuildV2RequirementWithAsset → the 402 wire response.
• Default raised from hardcoded 60s → 600s (DefaultMaxTimeoutSeconds); operators
  can set any value up to 86400s (24h) via MaxMaxTimeoutSeconds cap.
• Streaming agent flows can legitimately take minutes-to-hours; the cap is permissive
  because the auth is nonce-bound and replay-safe — the cap exists only to guard against
  operator typos (e.g. ms→s).
• Regression tests: TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds,
  TestClampMaxTimeoutSeconds, TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds.

3. New buy.py command: pay-agent (streaming SSE for type:agent offers)

• Sibling of pay, deliberately not an alias of --type inference. inference
  pushes a paid/<model> alias behind LiteLLM; agent responses are first-class data the
  calling agent wants to consume itself (memory, tool-call traces, partial results), so
  no LiteLLM wire-up.
• Forces "stream": true, sends Accept: text/event-stream, reads response
  line-by-line, flushes each SSE event to stdout so a calling Hermes/OpenClaw can re-emit
  the stream to its own user (telegram, obol hermes chat, REST clients).
• Default HTTP read timeout 1 hour; operator-overridable via --timeout.
• obol buy inference now emits a clear pointer at pay-agent instead of just refusing
  type=agent offers (internal/buy/discover.go).
• (Researched Hermes' ACP mode — JSON-RPC over stdio for editor integration, wrong layer
  for this. The seller already streams OpenAI-compatible SSE through HandleProxy; the
  buyer just needed to consume it.)

4. rc13 headline fix: surface on-chain tx hash on facilitator /settle 5xx

QA caught a 0.001 OBOL mainnet on-chain debit from a request that returned HTTP 503 "Payment settlement failed" while every signal the stack produced said "nothing was
paid." Permit2 settle tx [0xb5122d…932307](https://etherscan.io/tx/0xb5122d818a058e8bf
529380260fa2584ba3d50bfc800f1e906faca34d3932307) mined; the facilitator's post-submit
step then returned 500.

This PR doesn't fix the facilitator (that's the hosted x402.gcp.obol.tech service),
but closes the client-side robustness gap that turned a facilitator hiccup into
silent money loss:

• Verifier (internal/x402/forwardauth.go): facilitatorSettle now returns the
  parsed response alongside the error on non-200, and the settle interceptor writes
  X-PAYMENT-RESPONSE with the tx hash before committing the 503. Without this the
  on-chain hash was invisible to the buyer.
• Buyer sidecar (internal/x402/buyer/proxy.go): an upstream 5xx with
  X-PAYMENT-RESPONSE.transaction != "" now triggers ConfirmSpend on the held auth (it
  was spent on-chain), fires OnPaymentUnsettled, and logs the hash + payer + network.
  Previously the auth was released back into the pool, leaving the spent counter
  permanently off and the next reuse attempt to fail on Permit2 nonce-reuse revert.
• buy.py (internal/embed/skills/buy-x402/scripts/buy.py):
  _print_paid_request_failure decodes the settle header on 5xx and prints a loud ⚠️ SETTLEMENT MAY HAVE COMPLETED ON-CHAIN warning with the tx hash and the exact balance --chain <X> command.
• Regression tests pin the contract on both sides:
  TestForwardAuth_SettleErrorPreservesTxHashInHeader (verifier) and
  TestProxy_UpstreamErrorWithTxHash_PersistsConsume (buyer).

5. Hermes image bump: v2026.5.28 → v2026.6.5

Closes a separate sub-agent context-bloat bug uncovered while triaging an older
sub-agent during the rc13 session: v2026.5.28's sync_skills() had no
.no-bundled-skills marker check (tools/skills_sync.py:419), so every Hermes pod boot
copied ~24 stock skill categories (~1 MB of SKILL.md text, ~100k tokens of prompt-scan
bloat) from the image-baked /opt/hermes/skills into the PVC — regardless of the marker
our SeedHostFiles writes at agent-create time. Affected every CRD sub-agent, not
just stale PVCs.

Upstream fix landed in v2026.6.5 as commit 2ed96372a ("blank-slate skills"):
sync_skills() now early-returns when the marker exists (tools/skills_sync.py:467).
Verified by reading both tags' source. No containerEnv changes needed — the marker our
code already writes is now honored end-to-end.

• internal/serviceoffercontroller/agent_render.go (sub-agent default)
• internal/hermes/hermes.go (master agent default)
• internal/agentcrd/agent_contract_integration_test.go doc comment updated with the
  v2026.5.28 → v2026.6.5 root cause + upstream commit reference.

Also picks up unrelated fixes between 5.28 and 6.5: gateway max_tokens propagation
chain, multi-profile remote-dashboard sessions, several smaller items.

6. Debugging-lore additions

• docs/observability.md: new § "Verify settlement against the chain, never the sidecar
  snapshot" with the operator debugging recipe (eth_getLogs curl) and the rc13 tx hash
  as evidence.
• .agents/skills/obol-stack-dev/references/release-smoke-debugging.md § 11: same
  lesson at the runbook surface so future debugging sessions don't have to rediscover it
  from 0 spent / 2 remaining.

Follow-ups (NOT in this PR — flagged for separate work)

These are deliberately left for separate PRs because they each have non-trivial scope or
live outside this repo. Tracked here so they don't get lost:

1. Image-pin re-bump after merge. This PR's source changes don't take effect in
   deployed clusters until CI publishes new x402-verifier / x402-buyer images tagged
   with the merge-commit short SHA. A one-line follow-up to
   internal/embed/infrastructure/base/templates/{x402,llm}.yaml will pin those. Until
   then the cluster sees the 04bebbc pin, which has the ab71481 and 86b8c9f fixes but
   not the maxTimeoutSeconds plumb or the settle-tx-hash forensic fix.
   2. Wire the integration test into CI.
   TestIntegration_AgentContract_HonorsRenderedConstraints (//go:build integration, in
   internal/agentcrd/agent_contract_integration_test.go) verifies the no-bundled-skills
   contract end-to-end on a live cluster, but isn't referenced in any .github/workflows/
   or flows/release-smoke.sh. Manual-only invocation means rc13's bloat went undetected
   for weeks. Add it to release-smoke (or a Hermes-bump gate).
2. Renovate annotation on the Hermes pin. Two defaultImage / defaultHermesImage
   constants in this repo currently track the Hermes release cadence by hand. A Renovate # renovate: ... annotation would surface future bumps automatically — sized as a 2-line
   follow-up.
3. Stale /data/.hermes/skills cleanup on pre-b121f99 PVCs. Agents created before
   the streamline work (June 2) have ~/.hermes/skills/ already populated; even with the
   v2026.6.5 marker honoring, that dir is already on disk and load_skills() reads it.
   Manual fix per agent: kubectl exec -n <ns> deploy/hermes -- mv /data/.hermes/skills /data/.hermes/skills.bundled-disabled-<ts> + kubectl rollout restart deploy/hermes -n <ns>. Not worth code in this repo — strictly an upgrade-migration concern for known
   existing deployments.
4. Full on-chain receipt verification in the verifier. Today this PR exposes the tx
   hash so an operator can reconcile manually. The bigger fix is to query an in-cluster RPC
   (eRPC) for the receipt status before deciding 200 vs 5xx — when settle lands on-chain,
   the verifier could serve the upstream response and report success. Needs RPC client
   plumbed into the verifier with per-network config; sized as its own PR.
5. Settle idempotency on retry. A retried request currently relies on Permit2 nonce
   reuse reverting on-chain to prevent double-debit — that works but burns gas and surfaces
   as cascading 503s. Proper fix: an auth state machine with PENDING → SETTLED
   transitions guarded by tx receipt confirmation.
6. Facilitator-side post-submit bug. Why does mainnet OBOL /settle return 500
   after a successful on-chain submit while Base Sepolia USDC settles cleanly? Out of scope
   (hosted x402.gcp.obol.tech), but the receipt-tx for the reproducible failure is in
   plans/rc13report.md.
7. rc12 punch list still confirmed in rc13 (not touched by this PR; recorded in
   plans/rc13report.md):
   • obol wallet import re-breaks Hermes ownership (fix = rollout restart master).
   • Purchase-delete hangs on unconsumed auths; no --force / abandon path.
   • purge data-dir cleanup on headless sudo fails (cleaned manually via root
     container).
8. Permit2 allowance-drain investigation (carried over from a prior session): does
   the OBOL token's _spendAllowance special-case type(uint256).max? If it doesn't,
   every settled transfer decrements a "max" allowance until 0, and the fix is in periodic
   re-approval, not cmd_pay pre-flights.

Test plan

• go test ./... clean.
• New regression tests in this PR all pass:
  • TestForwardAuth_SettleErrorPreservesTxHashInHeader
  • TestProxy_UpstreamErrorWithTxHash_PersistsConsume
  • TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds
  • TestClampMaxTimeoutSeconds,
    TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds
• python3 buy.py pay-agent shows usage and exits 1; python3 buy.py help text
  lists the new command.
• Verified upstream Hermes source (../NousResearch/hermes-agent): v2026.5.28
sync_skills has no marker check; v2026.6.5 early-returns on the marker at
  tools/skills_sync.py:467 (commit 2ed96372a).
• Manual after merge + image republish: repeat the rc13 mainnet OBOL cycle
  (obol sell inference → curl with X-PAYMENT) and verify that a forced facilitator 500
  (if reproducible) now surfaces the tx hash to the buyer and accounts the auth as spent.
• Manual after merge + image republish: confirm maxTimeoutSeconds: 1800 in a
  ServiceOffer reaches the 402 wire response (curl the offer endpoint, decode the
  accepts[].maxTimeoutSeconds field).
• Manual after Hermes v2026.6.5 image pull: run
  TestIntegration_AgentContract_HonorsRenderedConstraints (go test -tags integration -v -run TestIntegration_AgentContract_HonorsRenderedConstraints ./internal/agentcrd/) on
  a fresh cluster and confirm /data/.hermes/skills is empty after pod ready.
• obol buy inference <type=agent-offer> returns the new actionable error pointing
  at pay-agent.

Implemented #2 + #3

#3 — Renovate annotation (3 files):

• internal/hermes/hermes.go: // renovate: datasource=docker
  depName=nousresearch/hermes-agent above defaultImage
• internal/serviceoffercontroller/agent_render.go: same annotation above
  defaultHermesImage
• renovate.json:
  • New customManager regex that captures the version after the last : from
    hermes-agent:vX.Y.Z strings (verified against both files, doesn't trip on the existing
    rawChartVersion helm annotation).
  • New packageRule grouping nousresearch/hermes-agent bumps with a labelled,
    Monday-scheduled PR template that includes a pre-merge gate explicitly reminding the
    reviewer to run release-smoke (flow-16 + flow-17 mentions).
  • Major-version-update gate that requires dashboard approval — Hermes majors have
    historically changed skill layout / marker semantics, exactly the class of regression
    that broke v2026.5.28.

#2 — CI wiring (3 files):

• flows/flow-16-sell-agent.sh: 3 new assertions mirroring the Go integration test —
  • .no-bundled-skills marker present on host PVC,
  • Pod-side /data/.hermes/obol-skills populated with the operator subset,
  • Pod-side /data/.hermes/skills absent or empty (the load-bearing check that would
    have caught the v2026.5.28 bloat).
• flows/release-smoke.sh: flow-16-sell-agent.sh added to the main flow sequence with a
  comment explaining why it's now release-gating.
• internal/agentcrd/agent_contract_integration_test.go: doc comment updated to call out
  that flow-16 now provides CI coverage of the same invariants — Go test is white-box
  developer surface, flow-16 is the black-box CI gate.

Verification:

• bash -n on both flow scripts — clean.
• python3 -m json.tool renovate.json — valid.
• Regex sanity test against both Go files matches exactly once each, captures
  currentValue=v2026.6.5, doesn't false-match the existing rawChartVersion = "2.0.2"
  annotation.
• go build ./... and go test ./... — clean.

[bussyjd]

Superseded by the v0.10.0-rc14 release train: #616 — this PR's branch is merged there verbatim (merge commit preserved, full union build + test green). Closing in favor of the consolidated release PR.

[OisinKyne]

Summary

Hardens the x402 paid-flow stack after rc13 QA caught two distinct issues: stale image
pins that bypassed merged source fixes, and a silent-money-loss path where on-chain
settlement landed but the buyer reported "0 spent." Also adds the missing buyer side for
type:agent ServiceOffers (streaming SSE), plumbs spec.payment.maxTimeoutSeconds
through to the 402 response (previously dropped on the floor), and bumps the Hermes
agent image to pick up the upstream sync_skills() marker-honoring fix.

Driven by plans/rc13report.md (Kody, 2026-06-09); learnings folded into
docs/observability.md and obol-stack-dev/references/release-smoke-debugging.md so
the next debugging session doesn't relearn this from a cold start.

Changes

1. Image pins → 04bebbc (current main HEAD as of branching)

• internal/embed/infrastructure/base/templates/{x402,llm}.yaml
• Picks up ab71481 (suppress per-request verifyOnly=false log spam on the in-process
  settle path) and 86b8c9f (buyer drops expired pre-signed auths before signing). Both
  shipped in source for weeks but were not reaching the deployed cluster because the
  embedded pins lagged.
• Allowlist tests (embed_image_pin_test.go, embed_crd_test.go) updated.

2. maxTimeoutSeconds plumbed end-to-end

• ServiceOffer.spec.payment.maxTimeoutSeconds → RouteRule.MaxTimeoutSeconds →
  BuildV2RequirementWithAsset → the 402 wire response.
• Default raised from hardcoded 60s → 600s (DefaultMaxTimeoutSeconds); operators
  can set any value up to 86400s (24h) via MaxMaxTimeoutSeconds cap.
• Streaming agent flows can legitimately take minutes-to-hours; the cap is permissive
  because the auth is nonce-bound and replay-safe — the cap exists only to guard against
  operator typos (e.g. ms→s).
• Regression tests: TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds,
  TestClampMaxTimeoutSeconds, TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds.

3. New buy.py command: pay-agent (streaming SSE for type:agent offers)

• Sibling of pay, deliberately not an alias of --type inference. inference
  pushes a paid/<model> alias behind LiteLLM; agent responses are first-class data the
  calling agent wants to consume itself (memory, tool-call traces, partial results), so
  no LiteLLM wire-up.
• Forces "stream": true, sends Accept: text/event-stream, reads response
  line-by-line, flushes each SSE event to stdout so a calling Hermes/OpenClaw can re-emit
  the stream to its own user (telegram, obol hermes chat, REST clients).
• Default HTTP read timeout 1 hour; operator-overridable via --timeout.
• obol buy inference now emits a clear pointer at pay-agent instead of just refusing
  type=agent offers (internal/buy/discover.go).
• (Researched Hermes' ACP mode — JSON-RPC over stdio for editor integration, wrong layer
  for this. The seller already streams OpenAI-compatible SSE through HandleProxy; the
  buyer just needed to consume it.)

4. rc13 headline fix: surface on-chain tx hash on facilitator /settle 5xx

QA caught a 0.001 OBOL mainnet on-chain debit from a request that returned HTTP 503 "Payment settlement failed" while every signal the stack produced said "nothing was
paid." Permit2 settle tx [0xb5122d…932307](https://etherscan.io/tx/0xb5122d818a058e8bf
529380260fa2584ba3d50bfc800f1e906faca34d3932307) mined; the facilitator's post-submit
step then returned 500.

This PR doesn't fix the facilitator (that's the hosted x402.gcp.obol.tech service),
but closes the client-side robustness gap that turned a facilitator hiccup into
silent money loss:

• Verifier (internal/x402/forwardauth.go): facilitatorSettle now returns the
  parsed response alongside the error on non-200, and the settle interceptor writes
  X-PAYMENT-RESPONSE with the tx hash before committing the 503. Without this the
  on-chain hash was invisible to the buyer.
• Buyer sidecar (internal/x402/buyer/proxy.go): an upstream 5xx with
  X-PAYMENT-RESPONSE.transaction != "" now triggers ConfirmSpend on the held auth (it
  was spent on-chain), fires OnPaymentUnsettled, and logs the hash + payer + network.
  Previously the auth was released back into the pool, leaving the spent counter
  permanently off and the next reuse attempt to fail on Permit2 nonce-reuse revert.
• buy.py (internal/embed/skills/buy-x402/scripts/buy.py):
  _print_paid_request_failure decodes the settle header on 5xx and prints a loud ⚠️ SETTLEMENT MAY HAVE COMPLETED ON-CHAIN warning with the tx hash and the exact balance --chain <X> command.
• Regression tests pin the contract on both sides:
  TestForwardAuth_SettleErrorPreservesTxHashInHeader (verifier) and
  TestProxy_UpstreamErrorWithTxHash_PersistsConsume (buyer).

5. Hermes image bump: v2026.5.28 → v2026.6.5

Closes a separate sub-agent context-bloat bug uncovered while triaging an older
sub-agent during the rc13 session: v2026.5.28's sync_skills() had no
.no-bundled-skills marker check (tools/skills_sync.py:419), so every Hermes pod boot
copied ~24 stock skill categories (~1 MB of SKILL.md text, ~100k tokens of prompt-scan
bloat) from the image-baked /opt/hermes/skills into the PVC — regardless of the marker
our SeedHostFiles writes at agent-create time. Affected every CRD sub-agent, not
just stale PVCs.

Upstream fix landed in v2026.6.5 as commit 2ed96372a ("blank-slate skills"):
sync_skills() now early-returns when the marker exists (tools/skills_sync.py:467).
Verified by reading both tags' source. No containerEnv changes needed — the marker our
code already writes is now honored end-to-end.

• internal/serviceoffercontroller/agent_render.go (sub-agent default)
• internal/hermes/hermes.go (master agent default)
• internal/agentcrd/agent_contract_integration_test.go doc comment updated with the
  v2026.5.28 → v2026.6.5 root cause + upstream commit reference.

Also picks up unrelated fixes between 5.28 and 6.5: gateway max_tokens propagation
chain, multi-profile remote-dashboard sessions, several smaller items.

6. Debugging-lore additions

• docs/observability.md: new § "Verify settlement against the chain, never the sidecar
  snapshot" with the operator debugging recipe (eth_getLogs curl) and the rc13 tx hash
  as evidence.
• .agents/skills/obol-stack-dev/references/release-smoke-debugging.md § 11: same
  lesson at the runbook surface so future debugging sessions don't have to rediscover it
  from 0 spent / 2 remaining.

Follow-ups (NOT in this PR — flagged for separate work)

These are deliberately left for separate PRs because they each have non-trivial scope or
live outside this repo. Tracked here so they don't get lost:

1. Image-pin re-bump after merge. This PR's source changes don't take effect in
   deployed clusters until CI publishes new x402-verifier / x402-buyer images tagged
   with the merge-commit short SHA. A one-line follow-up to
   internal/embed/infrastructure/base/templates/{x402,llm}.yaml will pin those. Until
   then the cluster sees the 04bebbc pin, which has the ab71481 and 86b8c9f fixes but
   not the maxTimeoutSeconds plumb or the settle-tx-hash forensic fix.
   2. Wire the integration test into CI.
   TestIntegration_AgentContract_HonorsRenderedConstraints (//go:build integration, in
   internal/agentcrd/agent_contract_integration_test.go) verifies the no-bundled-skills
   contract end-to-end on a live cluster, but isn't referenced in any .github/workflows/
   or flows/release-smoke.sh. Manual-only invocation means rc13's bloat went undetected
   for weeks. Add it to release-smoke (or a Hermes-bump gate).
2. Renovate annotation on the Hermes pin. Two defaultImage / defaultHermesImage
   constants in this repo currently track the Hermes release cadence by hand. A Renovate # renovate: ... annotation would surface future bumps automatically — sized as a 2-line
   follow-up.
3. Stale /data/.hermes/skills cleanup on pre-b121f99 PVCs. Agents created before
   the streamline work (June 2) have ~/.hermes/skills/ already populated; even with the
   v2026.6.5 marker honoring, that dir is already on disk and load_skills() reads it.
   Manual fix per agent: kubectl exec -n <ns> deploy/hermes -- mv /data/.hermes/skills /data/.hermes/skills.bundled-disabled-<ts> + kubectl rollout restart deploy/hermes -n <ns>. Not worth code in this repo — strictly an upgrade-migration concern for known
   existing deployments.
4. Full on-chain receipt verification in the verifier. Today this PR exposes the tx
   hash so an operator can reconcile manually. The bigger fix is to query an in-cluster RPC
   (eRPC) for the receipt status before deciding 200 vs 5xx — when settle lands on-chain,
   the verifier could serve the upstream response and report success. Needs RPC client
   plumbed into the verifier with per-network config; sized as its own PR.
5. Settle idempotency on retry. A retried request currently relies on Permit2 nonce
   reuse reverting on-chain to prevent double-debit — that works but burns gas and surfaces
   as cascading 503s. Proper fix: an auth state machine with PENDING → SETTLED
   transitions guarded by tx receipt confirmation.
6. Facilitator-side post-submit bug. Why does mainnet OBOL /settle return 500
   after a successful on-chain submit while Base Sepolia USDC settles cleanly? Out of scope
   (hosted x402.gcp.obol.tech), but the receipt-tx for the reproducible failure is in
   plans/rc13report.md.
7. rc12 punch list still confirmed in rc13 (not touched by this PR; recorded in
   plans/rc13report.md):
   • obol wallet import re-breaks Hermes ownership (fix = rollout restart master).
   • Purchase-delete hangs on unconsumed auths; no --force / abandon path.
   • purge data-dir cleanup on headless sudo fails (cleaned manually via root
     container).
8. Permit2 allowance-drain investigation (carried over from a prior session): does
   the OBOL token's _spendAllowance special-case type(uint256).max? If it doesn't,
   every settled transfer decrements a "max" allowance until 0, and the fix is in periodic
   re-approval, not cmd_pay pre-flights.

Test plan

• go test ./... clean.
• New regression tests in this PR all pass:
  • TestForwardAuth_SettleErrorPreservesTxHashInHeader
  • TestProxy_UpstreamErrorWithTxHash_PersistsConsume
  • TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds
  • TestClampMaxTimeoutSeconds,
    TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds
• python3 buy.py pay-agent shows usage and exits 1; python3 buy.py help text
  lists the new command.
• Verified upstream Hermes source (../NousResearch/hermes-agent): v2026.5.28
sync_skills has no marker check; v2026.6.5 early-returns on the marker at
  tools/skills_sync.py:467 (commit 2ed96372a).
• Manual after merge + image republish: repeat the rc13 mainnet OBOL cycle
  (obol sell inference → curl with X-PAYMENT) and verify that a forced facilitator 500
  (if reproducible) now surfaces the tx hash to the buyer and accounts the auth as spent.
• Manual after merge + image republish: confirm maxTimeoutSeconds: 1800 in a
  ServiceOffer reaches the 402 wire response (curl the offer endpoint, decode the
  accepts[].maxTimeoutSeconds field).
• Manual after Hermes v2026.6.5 image pull: run
  TestIntegration_AgentContract_HonorsRenderedConstraints (go test -tags integration -v -run TestIntegration_AgentContract_HonorsRenderedConstraints ./internal/agentcrd/) on
  a fresh cluster and confirm /data/.hermes/skills is empty after pod ready.
• obol buy inference <type=agent-offer> returns the new actionable error pointing
  at pay-agent.

Implemented #2 + #3

#3 — Renovate annotation (3 files):

• internal/hermes/hermes.go: // renovate: datasource=docker
  depName=nousresearch/hermes-agent above defaultImage
• internal/serviceoffercontroller/agent_render.go: same annotation above
  defaultHermesImage
• renovate.json:
  • New customManager regex that captures the version after the last : from
    hermes-agent:vX.Y.Z strings (verified against both files, doesn't trip on the existing
    rawChartVersion helm annotation).
  • New packageRule grouping nousresearch/hermes-agent bumps with a labelled,
    Monday-scheduled PR template that includes a pre-merge gate explicitly reminding the
    reviewer to run release-smoke (flow-16 + flow-17 mentions).
  • Major-version-update gate that requires dashboard approval — Hermes majors have
    historically changed skill layout / marker semantics, exactly the class of regression
    that broke v2026.5.28.

#2 — CI wiring (3 files):

• flows/flow-16-sell-agent.sh: 3 new assertions mirroring the Go integration test —
  • .no-bundled-skills marker present on host PVC,
  • Pod-side /data/.hermes/obol-skills populated with the operator subset,
  • Pod-side /data/.hermes/skills absent or empty (the load-bearing check that would
    have caught the v2026.5.28 bloat).
• flows/release-smoke.sh: flow-16-sell-agent.sh added to the main flow sequence with a
  comment explaining why it's now release-gating.
• internal/agentcrd/agent_contract_integration_test.go: doc comment updated to call out
  that flow-16 now provides CI coverage of the same invariants — Go test is white-box
  developer surface, flow-16 is the black-box CI gate.

Verification:

• bash -n on both flow scripts — clean.
• python3 -m json.tool renovate.json — valid.
• Regex sanity test against both Go files matches exactly once each, captures
  currentValue=v2026.6.5, doesn't false-match the existing rawChartVersion = "2.0.2"
  annotation.
• go build ./... and go test ./... — clean.

[bussyjd]

Superseded by the v0.10.0-rc14 release train: #616 — this PR's branch is merged there verbatim (merge commit preserved, full union build + test green). Closing in favor of the consolidated release PR.

[OisinKyne]

Summary

Hardens the x402 paid-flow stack after rc13 QA caught two distinct issues: stale image
pins that bypassed merged source fixes, and a silent-money-loss path where on-chain
settlement landed but the buyer reported "0 spent." Also adds the missing buyer side for
type:agent ServiceOffers (streaming SSE), plumbs spec.payment.maxTimeoutSeconds
through to the 402 response (previously dropped on the floor), and bumps the Hermes
agent image to pick up the upstream sync_skills() marker-honoring fix.

Driven by plans/rc13report.md (Kody, 2026-06-09); learnings folded into
docs/observability.md and obol-stack-dev/references/release-smoke-debugging.md so
the next debugging session doesn't relearn this from a cold start.

Changes

1. Image pins → 04bebbc (current main HEAD as of branching)

• internal/embed/infrastructure/base/templates/{x402,llm}.yaml
• Picks up ab71481 (suppress per-request verifyOnly=false log spam on the in-process
  settle path) and 86b8c9f (buyer drops expired pre-signed auths before signing). Both
  shipped in source for weeks but were not reaching the deployed cluster because the
  embedded pins lagged.
• Allowlist tests (embed_image_pin_test.go, embed_crd_test.go) updated.

2. maxTimeoutSeconds plumbed end-to-end

• ServiceOffer.spec.payment.maxTimeoutSeconds → RouteRule.MaxTimeoutSeconds →
  BuildV2RequirementWithAsset → the 402 wire response.
• Default raised from hardcoded 60s → 600s (DefaultMaxTimeoutSeconds); operators
  can set any value up to 86400s (24h) via MaxMaxTimeoutSeconds cap.
• Streaming agent flows can legitimately take minutes-to-hours; the cap is permissive
  because the auth is nonce-bound and replay-safe — the cap exists only to guard against
  operator typos (e.g. ms→s).
• Regression tests: TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds,
  TestClampMaxTimeoutSeconds, TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds.

3. New buy.py command: pay-agent (streaming SSE for type:agent offers)

• Sibling of pay, deliberately not an alias of --type inference. inference
  pushes a paid/<model> alias behind LiteLLM; agent responses are first-class data the
  calling agent wants to consume itself (memory, tool-call traces, partial results), so
  no LiteLLM wire-up.
• Forces "stream": true, sends Accept: text/event-stream, reads response
  line-by-line, flushes each SSE event to stdout so a calling Hermes/OpenClaw can re-emit
  the stream to its own user (telegram, obol hermes chat, REST clients).
• Default HTTP read timeout 1 hour; operator-overridable via --timeout.
• obol buy inference now emits a clear pointer at pay-agent instead of just refusing
  type=agent offers (internal/buy/discover.go).
• (Researched Hermes' ACP mode — JSON-RPC over stdio for editor integration, wrong layer
  for this. The seller already streams OpenAI-compatible SSE through HandleProxy; the
  buyer just needed to consume it.)

4. rc13 headline fix: surface on-chain tx hash on facilitator /settle 5xx

QA caught a 0.001 OBOL mainnet on-chain debit from a request that returned HTTP 503 "Payment settlement failed" while every signal the stack produced said "nothing was
paid." Permit2 settle tx [0xb5122d…932307](https://etherscan.io/tx/0xb5122d818a058e8bf
529380260fa2584ba3d50bfc800f1e906faca34d3932307) mined; the facilitator's post-submit
step then returned 500.

This PR doesn't fix the facilitator (that's the hosted x402.gcp.obol.tech service),
but closes the client-side robustness gap that turned a facilitator hiccup into
silent money loss:

• Verifier (internal/x402/forwardauth.go): facilitatorSettle now returns the
  parsed response alongside the error on non-200, and the settle interceptor writes
  X-PAYMENT-RESPONSE with the tx hash before committing the 503. Without this the
  on-chain hash was invisible to the buyer.
• Buyer sidecar (internal/x402/buyer/proxy.go): an upstream 5xx with
  X-PAYMENT-RESPONSE.transaction != "" now triggers ConfirmSpend on the held auth (it
  was spent on-chain), fires OnPaymentUnsettled, and logs the hash + payer + network.
  Previously the auth was released back into the pool, leaving the spent counter
  permanently off and the next reuse attempt to fail on Permit2 nonce-reuse revert.
• buy.py (internal/embed/skills/buy-x402/scripts/buy.py):
  _print_paid_request_failure decodes the settle header on 5xx and prints a loud ⚠️ SETTLEMENT MAY HAVE COMPLETED ON-CHAIN warning with the tx hash and the exact balance --chain <X> command.
• Regression tests pin the contract on both sides:
  TestForwardAuth_SettleErrorPreservesTxHashInHeader (verifier) and
  TestProxy_UpstreamErrorWithTxHash_PersistsConsume (buyer).

5. Hermes image bump: v2026.5.28 → v2026.6.5

Closes a separate sub-agent context-bloat bug uncovered while triaging an older
sub-agent during the rc13 session: v2026.5.28's sync_skills() had no
.no-bundled-skills marker check (tools/skills_sync.py:419), so every Hermes pod boot
copied ~24 stock skill categories (~1 MB of SKILL.md text, ~100k tokens of prompt-scan
bloat) from the image-baked /opt/hermes/skills into the PVC — regardless of the marker
our SeedHostFiles writes at agent-create time. Affected every CRD sub-agent, not
just stale PVCs.

Upstream fix landed in v2026.6.5 as commit 2ed96372a ("blank-slate skills"):
sync_skills() now early-returns when the marker exists (tools/skills_sync.py:467).
Verified by reading both tags' source. No containerEnv changes needed — the marker our
code already writes is now honored end-to-end.

• internal/serviceoffercontroller/agent_render.go (sub-agent default)
• internal/hermes/hermes.go (master agent default)
• internal/agentcrd/agent_contract_integration_test.go doc comment updated with the
  v2026.5.28 → v2026.6.5 root cause + upstream commit reference.

Also picks up unrelated fixes between 5.28 and 6.5: gateway max_tokens propagation
chain, multi-profile remote-dashboard sessions, several smaller items.

6. Debugging-lore additions

• docs/observability.md: new § "Verify settlement against the chain, never the sidecar
  snapshot" with the operator debugging recipe (eth_getLogs curl) and the rc13 tx hash
  as evidence.
• .agents/skills/obol-stack-dev/references/release-smoke-debugging.md § 11: same
  lesson at the runbook surface so future debugging sessions don't have to rediscover it
  from 0 spent / 2 remaining.

Follow-ups (NOT in this PR — flagged for separate work)

These are deliberately left for separate PRs because they each have non-trivial scope or
live outside this repo. Tracked here so they don't get lost:

1. Image-pin re-bump after merge. This PR's source changes don't take effect in
   deployed clusters until CI publishes new x402-verifier / x402-buyer images tagged
   with the merge-commit short SHA. A one-line follow-up to
   internal/embed/infrastructure/base/templates/{x402,llm}.yaml will pin those. Until
   then the cluster sees the 04bebbc pin, which has the ab71481 and 86b8c9f fixes but
   not the maxTimeoutSeconds plumb or the settle-tx-hash forensic fix.
   2. Wire the integration test into CI.
   TestIntegration_AgentContract_HonorsRenderedConstraints (//go:build integration, in
   internal/agentcrd/agent_contract_integration_test.go) verifies the no-bundled-skills
   contract end-to-end on a live cluster, but isn't referenced in any .github/workflows/
   or flows/release-smoke.sh. Manual-only invocation means rc13's bloat went undetected
   for weeks. Add it to release-smoke (or a Hermes-bump gate).
2. Renovate annotation on the Hermes pin. Two defaultImage / defaultHermesImage
   constants in this repo currently track the Hermes release cadence by hand. A Renovate # renovate: ... annotation would surface future bumps automatically — sized as a 2-line
   follow-up.
3. Stale /data/.hermes/skills cleanup on pre-b121f99 PVCs. Agents created before
   the streamline work (June 2) have ~/.hermes/skills/ already populated; even with the
   v2026.6.5 marker honoring, that dir is already on disk and load_skills() reads it.
   Manual fix per agent: kubectl exec -n <ns> deploy/hermes -- mv /data/.hermes/skills /data/.hermes/skills.bundled-disabled-<ts> + kubectl rollout restart deploy/hermes -n <ns>. Not worth code in this repo — strictly an upgrade-migration concern for known
   existing deployments.
4. Full on-chain receipt verification in the verifier. Today this PR exposes the tx
   hash so an operator can reconcile manually. The bigger fix is to query an in-cluster RPC
   (eRPC) for the receipt status before deciding 200 vs 5xx — when settle lands on-chain,
   the verifier could serve the upstream response and report success. Needs RPC client
   plumbed into the verifier with per-network config; sized as its own PR.
5. Settle idempotency on retry. A retried request currently relies on Permit2 nonce
   reuse reverting on-chain to prevent double-debit — that works but burns gas and surfaces
   as cascading 503s. Proper fix: an auth state machine with PENDING → SETTLED
   transitions guarded by tx receipt confirmation.
6. Facilitator-side post-submit bug. Why does mainnet OBOL /settle return 500
   after a successful on-chain submit while Base Sepolia USDC settles cleanly? Out of scope
   (hosted x402.gcp.obol.tech), but the receipt-tx for the reproducible failure is in
   plans/rc13report.md.
7. rc12 punch list still confirmed in rc13 (not touched by this PR; recorded in
   plans/rc13report.md):
   • obol wallet import re-breaks Hermes ownership (fix = rollout restart master).
   • Purchase-delete hangs on unconsumed auths; no --force / abandon path.
   • purge data-dir cleanup on headless sudo fails (cleaned manually via root
     container).
8. Permit2 allowance-drain investigation (carried over from a prior session): does
   the OBOL token's _spendAllowance special-case type(uint256).max? If it doesn't,
   every settled transfer decrements a "max" allowance until 0, and the fix is in periodic
   re-approval, not cmd_pay pre-flights.

Test plan

• go test ./... clean.
• New regression tests in this PR all pass:
  • TestForwardAuth_SettleErrorPreservesTxHashInHeader
  • TestProxy_UpstreamErrorWithTxHash_PersistsConsume
  • TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds
  • TestClampMaxTimeoutSeconds,
    TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds
• python3 buy.py pay-agent shows usage and exits 1; python3 buy.py help text
  lists the new command.
• Verified upstream Hermes source (../NousResearch/hermes-agent): v2026.5.28
sync_skills has no marker check; v2026.6.5 early-returns on the marker at
  tools/skills_sync.py:467 (commit 2ed96372a).
• Manual after merge + image republish: repeat the rc13 mainnet OBOL cycle
  (obol sell inference → curl with X-PAYMENT) and verify that a forced facilitator 500
  (if reproducible) now surfaces the tx hash to the buyer and accounts the auth as spent.
• Manual after merge + image republish: confirm maxTimeoutSeconds: 1800 in a
  ServiceOffer reaches the 402 wire response (curl the offer endpoint, decode the
  accepts[].maxTimeoutSeconds field).
• Manual after Hermes v2026.6.5 image pull: run
  TestIntegration_AgentContract_HonorsRenderedConstraints (go test -tags integration -v -run TestIntegration_AgentContract_HonorsRenderedConstraints ./internal/agentcrd/) on
  a fresh cluster and confirm /data/.hermes/skills is empty after pod ready.
• obol buy inference <type=agent-offer> returns the new actionable error pointing
  at pay-agent.

Implemented #2 + #3

#3 — Renovate annotation (3 files):

• internal/hermes/hermes.go: // renovate: datasource=docker
  depName=nousresearch/hermes-agent above defaultImage
• internal/serviceoffercontroller/agent_render.go: same annotation above
  defaultHermesImage
• renovate.json:
  • New customManager regex that captures the version after the last : from
    hermes-agent:vX.Y.Z strings (verified against both files, doesn't trip on the existing
    rawChartVersion helm annotation).
  • New packageRule grouping nousresearch/hermes-agent bumps with a labelled,
    Monday-scheduled PR template that includes a pre-merge gate explicitly reminding the
    reviewer to run release-smoke (flow-16 + flow-17 mentions).
  • Major-version-update gate that requires dashboard approval — Hermes majors have
    historically changed skill layout / marker semantics, exactly the class of regression
    that broke v2026.5.28.

#2 — CI wiring (3 files):

• flows/flow-16-sell-agent.sh: 3 new assertions mirroring the Go integration test —
  • .no-bundled-skills marker present on host PVC,
  • Pod-side /data/.hermes/obol-skills populated with the operator subset,
  • Pod-side /data/.hermes/skills absent or empty (the load-bearing check that would
    have caught the v2026.5.28 bloat).
• flows/release-smoke.sh: flow-16-sell-agent.sh added to the main flow sequence with a
  comment explaining why it's now release-gating.
• internal/agentcrd/agent_contract_integration_test.go: doc comment updated to call out
  that flow-16 now provides CI coverage of the same invariants — Go test is white-box
  developer surface, flow-16 is the black-box CI gate.

Verification:

• bash -n on both flow scripts — clean.
• python3 -m json.tool renovate.json — valid.
• Regex sanity test against both Go files matches exactly once each, captures
  currentValue=v2026.6.5, doesn't false-match the existing rawChartVersion = "2.0.2"
  annotation.
• go build ./... and go test ./... — clean.

[bussyjd]

Superseded by the v0.10.0-rc14 release train: #616 — this PR's branch is merged there verbatim (merge commit preserved, full union build + test green). Closing in favor of the consolidated release PR.

[OisinKyne]

Summary

Hardens the x402 paid-flow stack after rc13 QA caught two distinct issues: stale image
pins that bypassed merged source fixes, and a silent-money-loss path where on-chain
settlement landed but the buyer reported "0 spent." Also adds the missing buyer side for
type:agent ServiceOffers (streaming SSE), plumbs spec.payment.maxTimeoutSeconds
through to the 402 response (previously dropped on the floor), and bumps the Hermes
agent image to pick up the upstream sync_skills() marker-honoring fix.

Driven by plans/rc13report.md (Kody, 2026-06-09); learnings folded into
docs/observability.md and obol-stack-dev/references/release-smoke-debugging.md so
the next debugging session doesn't relearn this from a cold start.

Changes

1. Image pins → 04bebbc (current main HEAD as of branching)

• internal/embed/infrastructure/base/templates/{x402,llm}.yaml
• Picks up ab71481 (suppress per-request verifyOnly=false log spam on the in-process
  settle path) and 86b8c9f (buyer drops expired pre-signed auths before signing). Both
  shipped in source for weeks but were not reaching the deployed cluster because the
  embedded pins lagged.
• Allowlist tests (embed_image_pin_test.go, embed_crd_test.go) updated.

2. maxTimeoutSeconds plumbed end-to-end

• ServiceOffer.spec.payment.maxTimeoutSeconds → RouteRule.MaxTimeoutSeconds →
  BuildV2RequirementWithAsset → the 402 wire response.
• Default raised from hardcoded 60s → 600s (DefaultMaxTimeoutSeconds); operators
  can set any value up to 86400s (24h) via MaxMaxTimeoutSeconds cap.
• Streaming agent flows can legitimately take minutes-to-hours; the cap is permissive
  because the auth is nonce-bound and replay-safe — the cap exists only to guard against
  operator typos (e.g. ms→s).
• Regression tests: TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds,
  TestClampMaxTimeoutSeconds, TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds.

3. New buy.py command: pay-agent (streaming SSE for type:agent offers)

• Sibling of pay, deliberately not an alias of --type inference. inference
  pushes a paid/<model> alias behind LiteLLM; agent responses are first-class data the
  calling agent wants to consume itself (memory, tool-call traces, partial results), so
  no LiteLLM wire-up.
• Forces "stream": true, sends Accept: text/event-stream, reads response
  line-by-line, flushes each SSE event to stdout so a calling Hermes/OpenClaw can re-emit
  the stream to its own user (telegram, obol hermes chat, REST clients).
• Default HTTP read timeout 1 hour; operator-overridable via --timeout.
• obol buy inference now emits a clear pointer at pay-agent instead of just refusing
  type=agent offers (internal/buy/discover.go).
• (Researched Hermes' ACP mode — JSON-RPC over stdio for editor integration, wrong layer
  for this. The seller already streams OpenAI-compatible SSE through HandleProxy; the
  buyer just needed to consume it.)

4. rc13 headline fix: surface on-chain tx hash on facilitator /settle 5xx

QA caught a 0.001 OBOL mainnet on-chain debit from a request that returned HTTP 503 "Payment settlement failed" while every signal the stack produced said "nothing was
paid." Permit2 settle tx [0xb5122d…932307](https://etherscan.io/tx/0xb5122d818a058e8bf
529380260fa2584ba3d50bfc800f1e906faca34d3932307) mined; the facilitator's post-submit
step then returned 500.

This PR doesn't fix the facilitator (that's the hosted x402.gcp.obol.tech service),
but closes the client-side robustness gap that turned a facilitator hiccup into
silent money loss:

• Verifier (internal/x402/forwardauth.go): facilitatorSettle now returns the
  parsed response alongside the error on non-200, and the settle interceptor writes
  X-PAYMENT-RESPONSE with the tx hash before committing the 503. Without this the
  on-chain hash was invisible to the buyer.
• Buyer sidecar (internal/x402/buyer/proxy.go): an upstream 5xx with
  X-PAYMENT-RESPONSE.transaction != "" now triggers ConfirmSpend on the held auth (it
  was spent on-chain), fires OnPaymentUnsettled, and logs the hash + payer + network.
  Previously the auth was released back into the pool, leaving the spent counter
  permanently off and the next reuse attempt to fail on Permit2 nonce-reuse revert.
• buy.py (internal/embed/skills/buy-x402/scripts/buy.py):
  _print_paid_request_failure decodes the settle header on 5xx and prints a loud ⚠️ SETTLEMENT MAY HAVE COMPLETED ON-CHAIN warning with the tx hash and the exact balance --chain <X> command.
• Regression tests pin the contract on both sides:
  TestForwardAuth_SettleErrorPreservesTxHashInHeader (verifier) and
  TestProxy_UpstreamErrorWithTxHash_PersistsConsume (buyer).

5. Hermes image bump: v2026.5.28 → v2026.6.5

Closes a separate sub-agent context-bloat bug uncovered while triaging an older
sub-agent during the rc13 session: v2026.5.28's sync_skills() had no
.no-bundled-skills marker check (tools/skills_sync.py:419), so every Hermes pod boot
copied ~24 stock skill categories (~1 MB of SKILL.md text, ~100k tokens of prompt-scan
bloat) from the image-baked /opt/hermes/skills into the PVC — regardless of the marker
our SeedHostFiles writes at agent-create time. Affected every CRD sub-agent, not
just stale PVCs.

Upstream fix landed in v2026.6.5 as commit 2ed96372a ("blank-slate skills"):
sync_skills() now early-returns when the marker exists (tools/skills_sync.py:467).
Verified by reading both tags' source. No containerEnv changes needed — the marker our
code already writes is now honored end-to-end.

• internal/serviceoffercontroller/agent_render.go (sub-agent default)
• internal/hermes/hermes.go (master agent default)
• internal/agentcrd/agent_contract_integration_test.go doc comment updated with the
  v2026.5.28 → v2026.6.5 root cause + upstream commit reference.

Also picks up unrelated fixes between 5.28 and 6.5: gateway max_tokens propagation
chain, multi-profile remote-dashboard sessions, several smaller items.

6. Debugging-lore additions

• docs/observability.md: new § "Verify settlement against the chain, never the sidecar
  snapshot" with the operator debugging recipe (eth_getLogs curl) and the rc13 tx hash
  as evidence.
• .agents/skills/obol-stack-dev/references/release-smoke-debugging.md § 11: same
  lesson at the runbook surface so future debugging sessions don't have to rediscover it
  from 0 spent / 2 remaining.

Follow-ups (NOT in this PR — flagged for separate work)

These are deliberately left for separate PRs because they each have non-trivial scope or
live outside this repo. Tracked here so they don't get lost:

1. Image-pin re-bump after merge. This PR's source changes don't take effect in
   deployed clusters until CI publishes new x402-verifier / x402-buyer images tagged
   with the merge-commit short SHA. A one-line follow-up to
   internal/embed/infrastructure/base/templates/{x402,llm}.yaml will pin those. Until
   then the cluster sees the 04bebbc pin, which has the ab71481 and 86b8c9f fixes but
   not the maxTimeoutSeconds plumb or the settle-tx-hash forensic fix.
   2. Wire the integration test into CI.
   TestIntegration_AgentContract_HonorsRenderedConstraints (//go:build integration, in
   internal/agentcrd/agent_contract_integration_test.go) verifies the no-bundled-skills
   contract end-to-end on a live cluster, but isn't referenced in any .github/workflows/
   or flows/release-smoke.sh. Manual-only invocation means rc13's bloat went undetected
   for weeks. Add it to release-smoke (or a Hermes-bump gate).
2. Renovate annotation on the Hermes pin. Two defaultImage / defaultHermesImage
   constants in this repo currently track the Hermes release cadence by hand. A Renovate # renovate: ... annotation would surface future bumps automatically — sized as a 2-line
   follow-up.
3. Stale /data/.hermes/skills cleanup on pre-b121f99 PVCs. Agents created before
   the streamline work (June 2) have ~/.hermes/skills/ already populated; even with the
   v2026.6.5 marker honoring, that dir is already on disk and load_skills() reads it.
   Manual fix per agent: kubectl exec -n <ns> deploy/hermes -- mv /data/.hermes/skills /data/.hermes/skills.bundled-disabled-<ts> + kubectl rollout restart deploy/hermes -n <ns>. Not worth code in this repo — strictly an upgrade-migration concern for known
   existing deployments.
4. Full on-chain receipt verification in the verifier. Today this PR exposes the tx
   hash so an operator can reconcile manually. The bigger fix is to query an in-cluster RPC
   (eRPC) for the receipt status before deciding 200 vs 5xx — when settle lands on-chain,
   the verifier could serve the upstream response and report success. Needs RPC client
   plumbed into the verifier with per-network config; sized as its own PR.
5. Settle idempotency on retry. A retried request currently relies on Permit2 nonce
   reuse reverting on-chain to prevent double-debit — that works but burns gas and surfaces
   as cascading 503s. Proper fix: an auth state machine with PENDING → SETTLED
   transitions guarded by tx receipt confirmation.
6. Facilitator-side post-submit bug. Why does mainnet OBOL /settle return 500
   after a successful on-chain submit while Base Sepolia USDC settles cleanly? Out of scope
   (hosted x402.gcp.obol.tech), but the receipt-tx for the reproducible failure is in
   plans/rc13report.md.
7. rc12 punch list still confirmed in rc13 (not touched by this PR; recorded in
   plans/rc13report.md):
   • obol wallet import re-breaks Hermes ownership (fix = rollout restart master).
   • Purchase-delete hangs on unconsumed auths; no --force / abandon path.
   • purge data-dir cleanup on headless sudo fails (cleaned manually via root
     container).
8. Permit2 allowance-drain investigation (carried over from a prior session): does
   the OBOL token's _spendAllowance special-case type(uint256).max? If it doesn't,
   every settled transfer decrements a "max" allowance until 0, and the fix is in periodic
   re-approval, not cmd_pay pre-flights.

Test plan

• go test ./... clean.
• New regression tests in this PR all pass:
  • TestForwardAuth_SettleErrorPreservesTxHashInHeader
  • TestProxy_UpstreamErrorWithTxHash_PersistsConsume
  • TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds
  • TestClampMaxTimeoutSeconds,
    TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds
• python3 buy.py pay-agent shows usage and exits 1; python3 buy.py help text
  lists the new command.
• Verified upstream Hermes source (../NousResearch/hermes-agent): v2026.5.28
sync_skills has no marker check; v2026.6.5 early-returns on the marker at
  tools/skills_sync.py:467 (commit 2ed96372a).
• Manual after merge + image republish: repeat the rc13 mainnet OBOL cycle
  (obol sell inference → curl with X-PAYMENT) and verify that a forced facilitator 500
  (if reproducible) now surfaces the tx hash to the buyer and accounts the auth as spent.
• Manual after merge + image republish: confirm maxTimeoutSeconds: 1800 in a
  ServiceOffer reaches the 402 wire response (curl the offer endpoint, decode the
  accepts[].maxTimeoutSeconds field).
• Manual after Hermes v2026.6.5 image pull: run
  TestIntegration_AgentContract_HonorsRenderedConstraints (go test -tags integration -v -run TestIntegration_AgentContract_HonorsRenderedConstraints ./internal/agentcrd/) on
  a fresh cluster and confirm /data/.hermes/skills is empty after pod ready.
• obol buy inference <type=agent-offer> returns the new actionable error pointing
  at pay-agent.

Implemented #2 + #3

#3 — Renovate annotation (3 files):

• internal/hermes/hermes.go: // renovate: datasource=docker
  depName=nousresearch/hermes-agent above defaultImage
• internal/serviceoffercontroller/agent_render.go: same annotation above
  defaultHermesImage
• renovate.json:
  • New customManager regex that captures the version after the last : from
    hermes-agent:vX.Y.Z strings (verified against both files, doesn't trip on the existing
    rawChartVersion helm annotation).
  • New packageRule grouping nousresearch/hermes-agent bumps with a labelled,
    Monday-scheduled PR template that includes a pre-merge gate explicitly reminding the
    reviewer to run release-smoke (flow-16 + flow-17 mentions).
  • Major-version-update gate that requires dashboard approval — Hermes majors have
    historically changed skill layout / marker semantics, exactly the class of regression
    that broke v2026.5.28.

#2 — CI wiring (3 files):

• flows/flow-16-sell-agent.sh: 3 new assertions mirroring the Go integration test —
  • .no-bundled-skills marker present on host PVC,
  • Pod-side /data/.hermes/obol-skills populated with the operator subset,
  • Pod-side /data/.hermes/skills absent or empty (the load-bearing check that would
    have caught the v2026.5.28 bloat).
• flows/release-smoke.sh: flow-16-sell-agent.sh added to the main flow sequence with a
  comment explaining why it's now release-gating.
• internal/agentcrd/agent_contract_integration_test.go: doc comment updated to call out
  that flow-16 now provides CI coverage of the same invariants — Go test is white-box
  developer surface, flow-16 is the black-box CI gate.

Verification:

• bash -n on both flow scripts — clean.
• python3 -m json.tool renovate.json — valid.
• Regex sanity test against both Go files matches exactly once each, captures
  currentValue=v2026.6.5, doesn't false-match the existing rawChartVersion = "2.0.2"
  annotation.
• go build ./... and go test ./... — clean.

[bussyjd]

Superseded by the v0.10.0-rc14 release train: #616 — this PR's branch is merged there verbatim (merge commit preserved, full union build + test green). Closing in favor of the consolidated release PR.

[OisinKyne]

Summary

Hardens the x402 paid-flow stack after rc13 QA caught two distinct issues: stale image
pins that bypassed merged source fixes, and a silent-money-loss path where on-chain
settlement landed but the buyer reported "0 spent." Also adds the missing buyer side for
type:agent ServiceOffers (streaming SSE), plumbs spec.payment.maxTimeoutSeconds
through to the 402 response (previously dropped on the floor), and bumps the Hermes
agent image to pick up the upstream sync_skills() marker-honoring fix.

Driven by plans/rc13report.md (Kody, 2026-06-09); learnings folded into
docs/observability.md and obol-stack-dev/references/release-smoke-debugging.md so
the next debugging session doesn't relearn this from a cold start.

Changes

1. Image pins → 04bebbc (current main HEAD as of branching)

• internal/embed/infrastructure/base/templates/{x402,llm}.yaml
• Picks up ab71481 (suppress per-request verifyOnly=false log spam on the in-process
  settle path) and 86b8c9f (buyer drops expired pre-signed auths before signing). Both
  shipped in source for weeks but were not reaching the deployed cluster because the
  embedded pins lagged.
• Allowlist tests (embed_image_pin_test.go, embed_crd_test.go) updated.

2. maxTimeoutSeconds plumbed end-to-end

• ServiceOffer.spec.payment.maxTimeoutSeconds → RouteRule.MaxTimeoutSeconds →
  BuildV2RequirementWithAsset → the 402 wire response.
• Default raised from hardcoded 60s → 600s (DefaultMaxTimeoutSeconds); operators
  can set any value up to 86400s (24h) via MaxMaxTimeoutSeconds cap.
• Streaming agent flows can legitimately take minutes-to-hours; the cap is permissive
  because the auth is nonce-bound and replay-safe — the cap exists only to guard against
  operator typos (e.g. ms→s).
• Regression tests: TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds,
  TestClampMaxTimeoutSeconds, TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds.

3. New buy.py command: pay-agent (streaming SSE for type:agent offers)

• Sibling of pay, deliberately not an alias of --type inference. inference
  pushes a paid/<model> alias behind LiteLLM; agent responses are first-class data the
  calling agent wants to consume itself (memory, tool-call traces, partial results), so
  no LiteLLM wire-up.
• Forces "stream": true, sends Accept: text/event-stream, reads response
  line-by-line, flushes each SSE event to stdout so a calling Hermes/OpenClaw can re-emit
  the stream to its own user (telegram, obol hermes chat, REST clients).
• Default HTTP read timeout 1 hour; operator-overridable via --timeout.
• obol buy inference now emits a clear pointer at pay-agent instead of just refusing
  type=agent offers (internal/buy/discover.go).
• (Researched Hermes' ACP mode — JSON-RPC over stdio for editor integration, wrong layer
  for this. The seller already streams OpenAI-compatible SSE through HandleProxy; the
  buyer just needed to consume it.)

4. rc13 headline fix: surface on-chain tx hash on facilitator /settle 5xx

QA caught a 0.001 OBOL mainnet on-chain debit from a request that returned HTTP 503 "Payment settlement failed" while every signal the stack produced said "nothing was
paid." Permit2 settle tx [0xb5122d…932307](https://etherscan.io/tx/0xb5122d818a058e8bf
529380260fa2584ba3d50bfc800f1e906faca34d3932307) mined; the facilitator's post-submit
step then returned 500.

This PR doesn't fix the facilitator (that's the hosted x402.gcp.obol.tech service),
but closes the client-side robustness gap that turned a facilitator hiccup into
silent money loss:

• Verifier (internal/x402/forwardauth.go): facilitatorSettle now returns the
  parsed response alongside the error on non-200, and the settle interceptor writes
  X-PAYMENT-RESPONSE with the tx hash before committing the 503. Without this the
  on-chain hash was invisible to the buyer.
• Buyer sidecar (internal/x402/buyer/proxy.go): an upstream 5xx with
  X-PAYMENT-RESPONSE.transaction != "" now triggers ConfirmSpend on the held auth (it
  was spent on-chain), fires OnPaymentUnsettled, and logs the hash + payer + network.
  Previously the auth was released back into the pool, leaving the spent counter
  permanently off and the next reuse attempt to fail on Permit2 nonce-reuse revert.
• buy.py (internal/embed/skills/buy-x402/scripts/buy.py):
  _print_paid_request_failure decodes the settle header on 5xx and prints a loud ⚠️ SETTLEMENT MAY HAVE COMPLETED ON-CHAIN warning with the tx hash and the exact balance --chain <X> command.
• Regression tests pin the contract on both sides:
  TestForwardAuth_SettleErrorPreservesTxHashInHeader (verifier) and
  TestProxy_UpstreamErrorWithTxHash_PersistsConsume (buyer).

5. Hermes image bump: v2026.5.28 → v2026.6.5

Closes a separate sub-agent context-bloat bug uncovered while triaging an older
sub-agent during the rc13 session: v2026.5.28's sync_skills() had no
.no-bundled-skills marker check (tools/skills_sync.py:419), so every Hermes pod boot
copied ~24 stock skill categories (~1 MB of SKILL.md text, ~100k tokens of prompt-scan
bloat) from the image-baked /opt/hermes/skills into the PVC — regardless of the marker
our SeedHostFiles writes at agent-create time. Affected every CRD sub-agent, not
just stale PVCs.

Upstream fix landed in v2026.6.5 as commit 2ed96372a ("blank-slate skills"):
sync_skills() now early-returns when the marker exists (tools/skills_sync.py:467).
Verified by reading both tags' source. No containerEnv changes needed — the marker our
code already writes is now honored end-to-end.

• internal/serviceoffercontroller/agent_render.go (sub-agent default)
• internal/hermes/hermes.go (master agent default)
• internal/agentcrd/agent_contract_integration_test.go doc comment updated with the
  v2026.5.28 → v2026.6.5 root cause + upstream commit reference.

Also picks up unrelated fixes between 5.28 and 6.5: gateway max_tokens propagation
chain, multi-profile remote-dashboard sessions, several smaller items.

6. Debugging-lore additions

• docs/observability.md: new § "Verify settlement against the chain, never the sidecar
  snapshot" with the operator debugging recipe (eth_getLogs curl) and the rc13 tx hash
  as evidence.
• .agents/skills/obol-stack-dev/references/release-smoke-debugging.md § 11: same
  lesson at the runbook surface so future debugging sessions don't have to rediscover it
  from 0 spent / 2 remaining.

Follow-ups (NOT in this PR — flagged for separate work)

These are deliberately left for separate PRs because they each have non-trivial scope or
live outside this repo. Tracked here so they don't get lost:

1. Image-pin re-bump after merge. This PR's source changes don't take effect in
   deployed clusters until CI publishes new x402-verifier / x402-buyer images tagged
   with the merge-commit short SHA. A one-line follow-up to
   internal/embed/infrastructure/base/templates/{x402,llm}.yaml will pin those. Until
   then the cluster sees the 04bebbc pin, which has the ab71481 and 86b8c9f fixes but
   not the maxTimeoutSeconds plumb or the settle-tx-hash forensic fix.
   2. Wire the integration test into CI.
   TestIntegration_AgentContract_HonorsRenderedConstraints (//go:build integration, in
   internal/agentcrd/agent_contract_integration_test.go) verifies the no-bundled-skills
   contract end-to-end on a live cluster, but isn't referenced in any .github/workflows/
   or flows/release-smoke.sh. Manual-only invocation means rc13's bloat went undetected
   for weeks. Add it to release-smoke (or a Hermes-bump gate).
2. Renovate annotation on the Hermes pin. Two defaultImage / defaultHermesImage
   constants in this repo currently track the Hermes release cadence by hand. A Renovate # renovate: ... annotation would surface future bumps automatically — sized as a 2-line
   follow-up.
3. Stale /data/.hermes/skills cleanup on pre-b121f99 PVCs. Agents created before
   the streamline work (June 2) have ~/.hermes/skills/ already populated; even with the
   v2026.6.5 marker honoring, that dir is already on disk and load_skills() reads it.
   Manual fix per agent: kubectl exec -n <ns> deploy/hermes -- mv /data/.hermes/skills /data/.hermes/skills.bundled-disabled-<ts> + kubectl rollout restart deploy/hermes -n <ns>. Not worth code in this repo — strictly an upgrade-migration concern for known
   existing deployments.
4. Full on-chain receipt verification in the verifier. Today this PR exposes the tx
   hash so an operator can reconcile manually. The bigger fix is to query an in-cluster RPC
   (eRPC) for the receipt status before deciding 200 vs 5xx — when settle lands on-chain,
   the verifier could serve the upstream response and report success. Needs RPC client
   plumbed into the verifier with per-network config; sized as its own PR.
5. Settle idempotency on retry. A retried request currently relies on Permit2 nonce
   reuse reverting on-chain to prevent double-debit — that works but burns gas and surfaces
   as cascading 503s. Proper fix: an auth state machine with PENDING → SETTLED
   transitions guarded by tx receipt confirmation.
6. Facilitator-side post-submit bug. Why does mainnet OBOL /settle return 500
   after a successful on-chain submit while Base Sepolia USDC settles cleanly? Out of scope
   (hosted x402.gcp.obol.tech), but the receipt-tx for the reproducible failure is in
   plans/rc13report.md.
7. rc12 punch list still confirmed in rc13 (not touched by this PR; recorded in
   plans/rc13report.md):
   • obol wallet import re-breaks Hermes ownership (fix = rollout restart master).
   • Purchase-delete hangs on unconsumed auths; no --force / abandon path.
   • purge data-dir cleanup on headless sudo fails (cleaned manually via root
     container).
8. Permit2 allowance-drain investigation (carried over from a prior session): does
   the OBOL token's _spendAllowance special-case type(uint256).max? If it doesn't,
   every settled transfer decrements a "max" allowance until 0, and the fix is in periodic
   re-approval, not cmd_pay pre-flights.

Test plan

• go test ./... clean.
• New regression tests in this PR all pass:
  • TestForwardAuth_SettleErrorPreservesTxHashInHeader
  • TestProxy_UpstreamErrorWithTxHash_PersistsConsume
  • TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds
  • TestClampMaxTimeoutSeconds,
    TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds
• python3 buy.py pay-agent shows usage and exits 1; python3 buy.py help text
  lists the new command.
• Verified upstream Hermes source (../NousResearch/hermes-agent): v2026.5.28
sync_skills has no marker check; v2026.6.5 early-returns on the marker at
  tools/skills_sync.py:467 (commit 2ed96372a).
• Manual after merge + image republish: repeat the rc13 mainnet OBOL cycle
  (obol sell inference → curl with X-PAYMENT) and verify that a forced facilitator 500
  (if reproducible) now surfaces the tx hash to the buyer and accounts the auth as spent.
• Manual after merge + image republish: confirm maxTimeoutSeconds: 1800 in a
  ServiceOffer reaches the 402 wire response (curl the offer endpoint, decode the
  accepts[].maxTimeoutSeconds field).
• Manual after Hermes v2026.6.5 image pull: run
  TestIntegration_AgentContract_HonorsRenderedConstraints (go test -tags integration -v -run TestIntegration_AgentContract_HonorsRenderedConstraints ./internal/agentcrd/) on
  a fresh cluster and confirm /data/.hermes/skills is empty after pod ready.
• obol buy inference <type=agent-offer> returns the new actionable error pointing
  at pay-agent.

Implemented #2 + #3

#3 — Renovate annotation (3 files):

• internal/hermes/hermes.go: // renovate: datasource=docker
  depName=nousresearch/hermes-agent above defaultImage
• internal/serviceoffercontroller/agent_render.go: same annotation above
  defaultHermesImage
• renovate.json:
  • New customManager regex that captures the version after the last : from
    hermes-agent:vX.Y.Z strings (verified against both files, doesn't trip on the existing
    rawChartVersion helm annotation).
  • New packageRule grouping nousresearch/hermes-agent bumps with a labelled,
    Monday-scheduled PR template that includes a pre-merge gate explicitly reminding the
    reviewer to run release-smoke (flow-16 + flow-17 mentions).
  • Major-version-update gate that requires dashboard approval — Hermes majors have
    historically changed skill layout / marker semantics, exactly the class of regression
    that broke v2026.5.28.

#2 — CI wiring (3 files):

• flows/flow-16-sell-agent.sh: 3 new assertions mirroring the Go integration test —
  • .no-bundled-skills marker present on host PVC,
  • Pod-side /data/.hermes/obol-skills populated with the operator subset,
  • Pod-side /data/.hermes/skills absent or empty (the load-bearing check that would
    have caught the v2026.5.28 bloat).
• flows/release-smoke.sh: flow-16-sell-agent.sh added to the main flow sequence with a
  comment explaining why it's now release-gating.
• internal/agentcrd/agent_contract_integration_test.go: doc comment updated to call out
  that flow-16 now provides CI coverage of the same invariants — Go test is white-box
  developer surface, flow-16 is the black-box CI gate.

Verification:

• bash -n on both flow scripts — clean.
• python3 -m json.tool renovate.json — valid.
• Regex sanity test against both Go files matches exactly once each, captures
  currentValue=v2026.6.5, doesn't false-match the existing rawChartVersion = "2.0.2"
  annotation.
• go build ./... and go test ./... — clean.

[bussyjd]

Superseded by the v0.10.0-rc14 release train: #616 — this PR's branch is merged there verbatim (merge commit preserved, full union build + test green). Closing in favor of the consolidated release PR.

[OisinKyne]

Summary

Hardens the x402 paid-flow stack after rc13 QA caught two distinct issues: stale image
pins that bypassed merged source fixes, and a silent-money-loss path where on-chain
settlement landed but the buyer reported "0 spent." Also adds the missing buyer side for
type:agent ServiceOffers (streaming SSE), plumbs spec.payment.maxTimeoutSeconds
through to the 402 response (previously dropped on the floor), and bumps the Hermes
agent image to pick up the upstream sync_skills() marker-honoring fix.

Driven by plans/rc13report.md (Kody, 2026-06-09); learnings folded into
docs/observability.md and obol-stack-dev/references/release-smoke-debugging.md so
the next debugging session doesn't relearn this from a cold start.

Changes

1. Image pins → 04bebbc (current main HEAD as of branching)

• internal/embed/infrastructure/base/templates/{x402,llm}.yaml
• Picks up ab71481 (suppress per-request verifyOnly=false log spam on the in-process
  settle path) and 86b8c9f (buyer drops expired pre-signed auths before signing). Both
  shipped in source for weeks but were not reaching the deployed cluster because the
  embedded pins lagged.
• Allowlist tests (embed_image_pin_test.go, embed_crd_test.go) updated.

2. maxTimeoutSeconds plumbed end-to-end

• ServiceOffer.spec.payment.maxTimeoutSeconds → RouteRule.MaxTimeoutSeconds →
  BuildV2RequirementWithAsset → the 402 wire response.
• Default raised from hardcoded 60s → 600s (DefaultMaxTimeoutSeconds); operators
  can set any value up to 86400s (24h) via MaxMaxTimeoutSeconds cap.
• Streaming agent flows can legitimately take minutes-to-hours; the cap is permissive
  because the auth is nonce-bound and replay-safe — the cap exists only to guard against
  operator typos (e.g. ms→s).
• Regression tests: TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds,
  TestClampMaxTimeoutSeconds, TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds.

3. New buy.py command: pay-agent (streaming SSE for type:agent offers)

• Sibling of pay, deliberately not an alias of --type inference. inference
  pushes a paid/<model> alias behind LiteLLM; agent responses are first-class data the
  calling agent wants to consume itself (memory, tool-call traces, partial results), so
  no LiteLLM wire-up.
• Forces "stream": true, sends Accept: text/event-stream, reads response
  line-by-line, flushes each SSE event to stdout so a calling Hermes/OpenClaw can re-emit
  the stream to its own user (telegram, obol hermes chat, REST clients).
• Default HTTP read timeout 1 hour; operator-overridable via --timeout.
• obol buy inference now emits a clear pointer at pay-agent instead of just refusing
  type=agent offers (internal/buy/discover.go).
• (Researched Hermes' ACP mode — JSON-RPC over stdio for editor integration, wrong layer
  for this. The seller already streams OpenAI-compatible SSE through HandleProxy; the
  buyer just needed to consume it.)

4. rc13 headline fix: surface on-chain tx hash on facilitator /settle 5xx

QA caught a 0.001 OBOL mainnet on-chain debit from a request that returned HTTP 503 "Payment settlement failed" while every signal the stack produced said "nothing was
paid." Permit2 settle tx [0xb5122d…932307](https://etherscan.io/tx/0xb5122d818a058e8bf
529380260fa2584ba3d50bfc800f1e906faca34d3932307) mined; the facilitator's post-submit
step then returned 500.

This PR doesn't fix the facilitator (that's the hosted x402.gcp.obol.tech service),
but closes the client-side robustness gap that turned a facilitator hiccup into
silent money loss:

• Verifier (internal/x402/forwardauth.go): facilitatorSettle now returns the
  parsed response alongside the error on non-200, and the settle interceptor writes
  X-PAYMENT-RESPONSE with the tx hash before committing the 503. Without this the
  on-chain hash was invisible to the buyer.
• Buyer sidecar (internal/x402/buyer/proxy.go): an upstream 5xx with
  X-PAYMENT-RESPONSE.transaction != "" now triggers ConfirmSpend on the held auth (it
  was spent on-chain), fires OnPaymentUnsettled, and logs the hash + payer + network.
  Previously the auth was released back into the pool, leaving the spent counter
  permanently off and the next reuse attempt to fail on Permit2 nonce-reuse revert.
• buy.py (internal/embed/skills/buy-x402/scripts/buy.py):
  _print_paid_request_failure decodes the settle header on 5xx and prints a loud ⚠️ SETTLEMENT MAY HAVE COMPLETED ON-CHAIN warning with the tx hash and the exact balance --chain <X> command.
• Regression tests pin the contract on both sides:
  TestForwardAuth_SettleErrorPreservesTxHashInHeader (verifier) and
  TestProxy_UpstreamErrorWithTxHash_PersistsConsume (buyer).

5. Hermes image bump: v2026.5.28 → v2026.6.5

Closes a separate sub-agent context-bloat bug uncovered while triaging an older
sub-agent during the rc13 session: v2026.5.28's sync_skills() had no
.no-bundled-skills marker check (tools/skills_sync.py:419), so every Hermes pod boot
copied ~24 stock skill categories (~1 MB of SKILL.md text, ~100k tokens of prompt-scan
bloat) from the image-baked /opt/hermes/skills into the PVC — regardless of the marker
our SeedHostFiles writes at agent-create time. Affected every CRD sub-agent, not
just stale PVCs.

Upstream fix landed in v2026.6.5 as commit 2ed96372a ("blank-slate skills"):
sync_skills() now early-returns when the marker exists (tools/skills_sync.py:467).
Verified by reading both tags' source. No containerEnv changes needed — the marker our
code already writes is now honored end-to-end.

• internal/serviceoffercontroller/agent_render.go (sub-agent default)
• internal/hermes/hermes.go (master agent default)
• internal/agentcrd/agent_contract_integration_test.go doc comment updated with the
  v2026.5.28 → v2026.6.5 root cause + upstream commit reference.

Also picks up unrelated fixes between 5.28 and 6.5: gateway max_tokens propagation
chain, multi-profile remote-dashboard sessions, several smaller items.

6. Debugging-lore additions

• docs/observability.md: new § "Verify settlement against the chain, never the sidecar
  snapshot" with the operator debugging recipe (eth_getLogs curl) and the rc13 tx hash
  as evidence.
• .agents/skills/obol-stack-dev/references/release-smoke-debugging.md § 11: same
  lesson at the runbook surface so future debugging sessions don't have to rediscover it
  from 0 spent / 2 remaining.

Follow-ups (NOT in this PR — flagged for separate work)

These are deliberately left for separate PRs because they each have non-trivial scope or
live outside this repo. Tracked here so they don't get lost:

1. Image-pin re-bump after merge. This PR's source changes don't take effect in
   deployed clusters until CI publishes new x402-verifier / x402-buyer images tagged
   with the merge-commit short SHA. A one-line follow-up to
   internal/embed/infrastructure/base/templates/{x402,llm}.yaml will pin those. Until
   then the cluster sees the 04bebbc pin, which has the ab71481 and 86b8c9f fixes but
   not the maxTimeoutSeconds plumb or the settle-tx-hash forensic fix.
   2. Wire the integration test into CI.
   TestIntegration_AgentContract_HonorsRenderedConstraints (//go:build integration, in
   internal/agentcrd/agent_contract_integration_test.go) verifies the no-bundled-skills
   contract end-to-end on a live cluster, but isn't referenced in any .github/workflows/
   or flows/release-smoke.sh. Manual-only invocation means rc13's bloat went undetected
   for weeks. Add it to release-smoke (or a Hermes-bump gate).
2. Renovate annotation on the Hermes pin. Two defaultImage / defaultHermesImage
   constants in this repo currently track the Hermes release cadence by hand. A Renovate # renovate: ... annotation would surface future bumps automatically — sized as a 2-line
   follow-up.
3. Stale /data/.hermes/skills cleanup on pre-b121f99 PVCs. Agents created before
   the streamline work (June 2) have ~/.hermes/skills/ already populated; even with the
   v2026.6.5 marker honoring, that dir is already on disk and load_skills() reads it.
   Manual fix per agent: kubectl exec -n <ns> deploy/hermes -- mv /data/.hermes/skills /data/.hermes/skills.bundled-disabled-<ts> + kubectl rollout restart deploy/hermes -n <ns>. Not worth code in this repo — strictly an upgrade-migration concern for known
   existing deployments.
4. Full on-chain receipt verification in the verifier. Today this PR exposes the tx
   hash so an operator can reconcile manually. The bigger fix is to query an in-cluster RPC
   (eRPC) for the receipt status before deciding 200 vs 5xx — when settle lands on-chain,
   the verifier could serve the upstream response and report success. Needs RPC client
   plumbed into the verifier with per-network config; sized as its own PR.
5. Settle idempotency on retry. A retried request currently relies on Permit2 nonce
   reuse reverting on-chain to prevent double-debit — that works but burns gas and surfaces
   as cascading 503s. Proper fix: an auth state machine with PENDING → SETTLED
   transitions guarded by tx receipt confirmation.
6. Facilitator-side post-submit bug. Why does mainnet OBOL /settle return 500
   after a successful on-chain submit while Base Sepolia USDC settles cleanly? Out of scope
   (hosted x402.gcp.obol.tech), but the receipt-tx for the reproducible failure is in
   plans/rc13report.md.
7. rc12 punch list still confirmed in rc13 (not touched by this PR; recorded in
   plans/rc13report.md):
   • obol wallet import re-breaks Hermes ownership (fix = rollout restart master).
   • Purchase-delete hangs on unconsumed auths; no --force / abandon path.
   • purge data-dir cleanup on headless sudo fails (cleaned manually via root
     container).
8. Permit2 allowance-drain investigation (carried over from a prior session): does
   the OBOL token's _spendAllowance special-case type(uint256).max? If it doesn't,
   every settled transfer decrements a "max" allowance until 0, and the fix is in periodic
   re-approval, not cmd_pay pre-flights.

Test plan

• go test ./... clean.
• New regression tests in this PR all pass:
  • TestForwardAuth_SettleErrorPreservesTxHashInHeader
  • TestProxy_UpstreamErrorWithTxHash_PersistsConsume
  • TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds
  • TestClampMaxTimeoutSeconds,
    TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds
• python3 buy.py pay-agent shows usage and exits 1; python3 buy.py help text
  lists the new command.
• Verified upstream Hermes source (../NousResearch/hermes-agent): v2026.5.28
sync_skills has no marker check; v2026.6.5 early-returns on the marker at
  tools/skills_sync.py:467 (commit 2ed96372a).
• Manual after merge + image republish: repeat the rc13 mainnet OBOL cycle
  (obol sell inference → curl with X-PAYMENT) and verify that a forced facilitator 500
  (if reproducible) now surfaces the tx hash to the buyer and accounts the auth as spent.
• Manual after merge + image republish: confirm maxTimeoutSeconds: 1800 in a
  ServiceOffer reaches the 402 wire response (curl the offer endpoint, decode the
  accepts[].maxTimeoutSeconds field).
• Manual after Hermes v2026.6.5 image pull: run
  TestIntegration_AgentContract_HonorsRenderedConstraints (go test -tags integration -v -run TestIntegration_AgentContract_HonorsRenderedConstraints ./internal/agentcrd/) on
  a fresh cluster and confirm /data/.hermes/skills is empty after pod ready.
• obol buy inference <type=agent-offer> returns the new actionable error pointing
  at pay-agent.

Implemented #2 + #3

#3 — Renovate annotation (3 files):

• internal/hermes/hermes.go: // renovate: datasource=docker
  depName=nousresearch/hermes-agent above defaultImage
• internal/serviceoffercontroller/agent_render.go: same annotation above
  defaultHermesImage
• renovate.json:
  • New customManager regex that captures the version after the last : from
    hermes-agent:vX.Y.Z strings (verified against both files, doesn't trip on the existing
    rawChartVersion helm annotation).
  • New packageRule grouping nousresearch/hermes-agent bumps with a labelled,
    Monday-scheduled PR template that includes a pre-merge gate explicitly reminding the
    reviewer to run release-smoke (flow-16 + flow-17 mentions).
  • Major-version-update gate that requires dashboard approval — Hermes majors have
    historically changed skill layout / marker semantics, exactly the class of regression
    that broke v2026.5.28.

#2 — CI wiring (3 files):

• flows/flow-16-sell-agent.sh: 3 new assertions mirroring the Go integration test —
  • .no-bundled-skills marker present on host PVC,
  • Pod-side /data/.hermes/obol-skills populated with the operator subset,
  • Pod-side /data/.hermes/skills absent or empty (the load-bearing check that would
    have caught the v2026.5.28 bloat).
• flows/release-smoke.sh: flow-16-sell-agent.sh added to the main flow sequence with a
  comment explaining why it's now release-gating.
• internal/agentcrd/agent_contract_integration_test.go: doc comment updated to call out
  that flow-16 now provides CI coverage of the same invariants — Go test is white-box
  developer surface, flow-16 is the black-box CI gate.

Verification:

• bash -n on both flow scripts — clean.
• python3 -m json.tool renovate.json — valid.
• Regex sanity test against both Go files matches exactly once each, captures
  currentValue=v2026.6.5, doesn't false-match the existing rawChartVersion = "2.0.2"
  annotation.
• go build ./... and go test ./... — clean.

[bussyjd]

Superseded by the v0.10.0-rc14 release train: #616 — this PR's branch is merged there verbatim (merge commit preserved, full union build + test green). Closing in favor of the consolidated release PR.

[OisinKyne]

Summary

Hardens the x402 paid-flow stack after rc13 QA caught two distinct issues: stale image
pins that bypassed merged source fixes, and a silent-money-loss path where on-chain
settlement landed but the buyer reported "0 spent." Also adds the missing buyer side for
type:agent ServiceOffers (streaming SSE), plumbs spec.payment.maxTimeoutSeconds
through to the 402 response (previously dropped on the floor), and bumps the Hermes
agent image to pick up the upstream sync_skills() marker-honoring fix.

Driven by plans/rc13report.md (Kody, 2026-06-09); learnings folded into
docs/observability.md and obol-stack-dev/references/release-smoke-debugging.md so
the next debugging session doesn't relearn this from a cold start.

Changes

1. Image pins → 04bebbc (current main HEAD as of branching)

• internal/embed/infrastructure/base/templates/{x402,llm}.yaml
• Picks up ab71481 (suppress per-request verifyOnly=false log spam on the in-process
  settle path) and 86b8c9f (buyer drops expired pre-signed auths before signing). Both
  shipped in source for weeks but were not reaching the deployed cluster because the
  embedded pins lagged.
• Allowlist tests (embed_image_pin_test.go, embed_crd_test.go) updated.

2. maxTimeoutSeconds plumbed end-to-end

• ServiceOffer.spec.payment.maxTimeoutSeconds → RouteRule.MaxTimeoutSeconds →
  BuildV2RequirementWithAsset → the 402 wire response.
• Default raised from hardcoded 60s → 600s (DefaultMaxTimeoutSeconds); operators
  can set any value up to 86400s (24h) via MaxMaxTimeoutSeconds cap.
• Streaming agent flows can legitimately take minutes-to-hours; the cap is permissive
  because the auth is nonce-bound and replay-safe — the cap exists only to guard against
  operator typos (e.g. ms→s).
• Regression tests: TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds,
  TestClampMaxTimeoutSeconds, TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds.

3. New buy.py command: pay-agent (streaming SSE for type:agent offers)

• Sibling of pay, deliberately not an alias of --type inference. inference
  pushes a paid/<model> alias behind LiteLLM; agent responses are first-class data the
  calling agent wants to consume itself (memory, tool-call traces, partial results), so
  no LiteLLM wire-up.
• Forces "stream": true, sends Accept: text/event-stream, reads response
  line-by-line, flushes each SSE event to stdout so a calling Hermes/OpenClaw can re-emit
  the stream to its own user (telegram, obol hermes chat, REST clients).
• Default HTTP read timeout 1 hour; operator-overridable via --timeout.
• obol buy inference now emits a clear pointer at pay-agent instead of just refusing
  type=agent offers (internal/buy/discover.go).
• (Researched Hermes' ACP mode — JSON-RPC over stdio for editor integration, wrong layer
  for this. The seller already streams OpenAI-compatible SSE through HandleProxy; the
  buyer just needed to consume it.)

4. rc13 headline fix: surface on-chain tx hash on facilitator /settle 5xx

QA caught a 0.001 OBOL mainnet on-chain debit from a request that returned HTTP 503 "Payment settlement failed" while every signal the stack produced said "nothing was
paid." Permit2 settle tx [0xb5122d…932307](https://etherscan.io/tx/0xb5122d818a058e8bf
529380260fa2584ba3d50bfc800f1e906faca34d3932307) mined; the facilitator's post-submit
step then returned 500.

This PR doesn't fix the facilitator (that's the hosted x402.gcp.obol.tech service),
but closes the client-side robustness gap that turned a facilitator hiccup into
silent money loss:

• Verifier (internal/x402/forwardauth.go): facilitatorSettle now returns the
  parsed response alongside the error on non-200, and the settle interceptor writes
  X-PAYMENT-RESPONSE with the tx hash before committing the 503. Without this the
  on-chain hash was invisible to the buyer.
• Buyer sidecar (internal/x402/buyer/proxy.go): an upstream 5xx with
  X-PAYMENT-RESPONSE.transaction != "" now triggers ConfirmSpend on the held auth (it
  was spent on-chain), fires OnPaymentUnsettled, and logs the hash + payer + network.
  Previously the auth was released back into the pool, leaving the spent counter
  permanently off and the next reuse attempt to fail on Permit2 nonce-reuse revert.
• buy.py (internal/embed/skills/buy-x402/scripts/buy.py):
  _print_paid_request_failure decodes the settle header on 5xx and prints a loud ⚠️ SETTLEMENT MAY HAVE COMPLETED ON-CHAIN warning with the tx hash and the exact balance --chain <X> command.
• Regression tests pin the contract on both sides:
  TestForwardAuth_SettleErrorPreservesTxHashInHeader (verifier) and
  TestProxy_UpstreamErrorWithTxHash_PersistsConsume (buyer).

5. Hermes image bump: v2026.5.28 → v2026.6.5

Closes a separate sub-agent context-bloat bug uncovered while triaging an older
sub-agent during the rc13 session: v2026.5.28's sync_skills() had no
.no-bundled-skills marker check (tools/skills_sync.py:419), so every Hermes pod boot
copied ~24 stock skill categories (~1 MB of SKILL.md text, ~100k tokens of prompt-scan
bloat) from the image-baked /opt/hermes/skills into the PVC — regardless of the marker
our SeedHostFiles writes at agent-create time. Affected every CRD sub-agent, not
just stale PVCs.

Upstream fix landed in v2026.6.5 as commit 2ed96372a ("blank-slate skills"):
sync_skills() now early-returns when the marker exists (tools/skills_sync.py:467).
Verified by reading both tags' source. No containerEnv changes needed — the marker our
code already writes is now honored end-to-end.

• internal/serviceoffercontroller/agent_render.go (sub-agent default)
• internal/hermes/hermes.go (master agent default)
• internal/agentcrd/agent_contract_integration_test.go doc comment updated with the
  v2026.5.28 → v2026.6.5 root cause + upstream commit reference.

Also picks up unrelated fixes between 5.28 and 6.5: gateway max_tokens propagation
chain, multi-profile remote-dashboard sessions, several smaller items.

6. Debugging-lore additions

• docs/observability.md: new § "Verify settlement against the chain, never the sidecar
  snapshot" with the operator debugging recipe (eth_getLogs curl) and the rc13 tx hash
  as evidence.
• .agents/skills/obol-stack-dev/references/release-smoke-debugging.md § 11: same
  lesson at the runbook surface so future debugging sessions don't have to rediscover it
  from 0 spent / 2 remaining.

Follow-ups (NOT in this PR — flagged for separate work)

These are deliberately left for separate PRs because they each have non-trivial scope or
live outside this repo. Tracked here so they don't get lost:

1. Image-pin re-bump after merge. This PR's source changes don't take effect in
   deployed clusters until CI publishes new x402-verifier / x402-buyer images tagged
   with the merge-commit short SHA. A one-line follow-up to
   internal/embed/infrastructure/base/templates/{x402,llm}.yaml will pin those. Until
   then the cluster sees the 04bebbc pin, which has the ab71481 and 86b8c9f fixes but
   not the maxTimeoutSeconds plumb or the settle-tx-hash forensic fix.
   2. Wire the integration test into CI.
   TestIntegration_AgentContract_HonorsRenderedConstraints (//go:build integration, in
   internal/agentcrd/agent_contract_integration_test.go) verifies the no-bundled-skills
   contract end-to-end on a live cluster, but isn't referenced in any .github/workflows/
   or flows/release-smoke.sh. Manual-only invocation means rc13's bloat went undetected
   for weeks. Add it to release-smoke (or a Hermes-bump gate).
2. Renovate annotation on the Hermes pin. Two defaultImage / defaultHermesImage
   constants in this repo currently track the Hermes release cadence by hand. A Renovate # renovate: ... annotation would surface future bumps automatically — sized as a 2-line
   follow-up.
3. Stale /data/.hermes/skills cleanup on pre-b121f99 PVCs. Agents created before
   the streamline work (June 2) have ~/.hermes/skills/ already populated; even with the
   v2026.6.5 marker honoring, that dir is already on disk and load_skills() reads it.
   Manual fix per agent: kubectl exec -n <ns> deploy/hermes -- mv /data/.hermes/skills /data/.hermes/skills.bundled-disabled-<ts> + kubectl rollout restart deploy/hermes -n <ns>. Not worth code in this repo — strictly an upgrade-migration concern for known
   existing deployments.
4. Full on-chain receipt verification in the verifier. Today this PR exposes the tx
   hash so an operator can reconcile manually. The bigger fix is to query an in-cluster RPC
   (eRPC) for the receipt status before deciding 200 vs 5xx — when settle lands on-chain,
   the verifier could serve the upstream response and report success. Needs RPC client
   plumbed into the verifier with per-network config; sized as its own PR.
5. Settle idempotency on retry. A retried request currently relies on Permit2 nonce
   reuse reverting on-chain to prevent double-debit — that works but burns gas and surfaces
   as cascading 503s. Proper fix: an auth state machine with PENDING → SETTLED
   transitions guarded by tx receipt confirmation.
6. Facilitator-side post-submit bug. Why does mainnet OBOL /settle return 500
   after a successful on-chain submit while Base Sepolia USDC settles cleanly? Out of scope
   (hosted x402.gcp.obol.tech), but the receipt-tx for the reproducible failure is in
   plans/rc13report.md.
7. rc12 punch list still confirmed in rc13 (not touched by this PR; recorded in
   plans/rc13report.md):
   • obol wallet import re-breaks Hermes ownership (fix = rollout restart master).
   • Purchase-delete hangs on unconsumed auths; no --force / abandon path.
   • purge data-dir cleanup on headless sudo fails (cleaned manually via root
     container).
8. Permit2 allowance-drain investigation (carried over from a prior session): does
   the OBOL token's _spendAllowance special-case type(uint256).max? If it doesn't,
   every settled transfer decrements a "max" allowance until 0, and the fix is in periodic
   re-approval, not cmd_pay pre-flights.

Test plan

• go test ./... clean.
• New regression tests in this PR all pass:
  • TestForwardAuth_SettleErrorPreservesTxHashInHeader
  • TestProxy_UpstreamErrorWithTxHash_PersistsConsume
  • TestRouteRuleFromOffer_PlumbsMaxTimeoutSeconds
  • TestClampMaxTimeoutSeconds,
    TestBuildV2RequirementWithAsset_HonorsMaxTimeoutSeconds
• python3 buy.py pay-agent shows usage and exits 1; python3 buy.py help text
  lists the new command.
• Verified upstream Hermes source (../NousResearch/hermes-agent): v2026.5.28
sync_skills has no marker check; v2026.6.5 early-returns on the marker at
  tools/skills_sync.py:467 (commit 2ed96372a).
• Manual after merge + image republish: repeat the rc13 mainnet OBOL cycle
  (obol sell inference → curl with X-PAYMENT) and verify that a forced facilitator 500
  (if reproducible) now surfaces the tx hash to the buyer and accounts the auth as spent.
• Manual after merge + image republish: confirm maxTimeoutSeconds: 1800 in a
  ServiceOffer reaches the 402 wire response (curl the offer endpoint, decode the
  accepts[].maxTimeoutSeconds field).
• Manual after Hermes v2026.6.5 image pull: run
  TestIntegration_AgentContract_HonorsRenderedConstraints (go test -tags integration -v -run TestIntegration_AgentContract_HonorsRenderedConstraints ./internal/agentcrd/) on
  a fresh cluster and confirm /data/.hermes/skills is empty after pod ready.
• obol buy inference <type=agent-offer> returns the new actionable error pointing
  at pay-agent.

Implemented #2 + #3

#3 — Renovate annotation (3 files):

• internal/hermes/hermes.go: // renovate: datasource=docker
  depName=nousresearch/hermes-agent above defaultImage
• internal/serviceoffercontroller/agent_render.go: same annotation above
  defaultHermesImage
• renovate.json:
  • New customManager regex that captures the version after the last : from
    hermes-agent:vX.Y.Z strings (verified against both files, doesn't trip on the existing
    rawChartVersion helm annotation).
  • New packageRule grouping nousresearch/hermes-agent bumps with a labelled,
    Monday-scheduled PR template that includes a pre-merge gate explicitly reminding the
    reviewer to run release-smoke (flow-16 + flow-17 mentions).
  • Major-version-update gate that requires dashboard approval — Hermes majors have
    historically changed skill layout / marker semantics, exactly the class of regression
    that broke v2026.5.28.

#2 — CI wiring (3 files):

• flows/flow-16-sell-agent.sh: 3 new assertions mirroring the Go integration test —
  • .no-bundled-skills marker present on host PVC,
  • Pod-side /data/.hermes/obol-skills populated with the operator subset,
  • Pod-side /data/.hermes/skills absent or empty (the load-bearing check that would
    have caught the v2026.5.28 bloat).
• flows/release-smoke.sh: flow-16-sell-agent.sh added to the main flow sequence with a
  comment explaining why it's now release-gating.
• internal/agentcrd/agent_contract_integration_test.go: doc comment updated to call out
  that flow-16 now provides CI coverage of the same invariants — Go test is white-box
  developer surface, flow-16 is the black-box CI gate.

Verification:

• bash -n on both flow scripts — clean.
• python3 -m json.tool renovate.json — valid.
• Regex sanity test against both Go files matches exactly once each, captures
  currentValue=v2026.6.5, doesn't false-match the existing rawChartVersion = "2.0.2"
  annotation.
• go build ./... and go test ./... — clean.

[bussyjd]

Superseded by the v0.10.0-rc14 release train: #616 — this PR's branch is merged there verbatim (merge commit preserved, full union build + test green). Closing in favor of the consolidated release PR.