Skip to content

rc8: reconcile stack QA fixes - #417

Merged
OisinKyne merged 21 commits into
mainfrom
integration/rc8-superset
May 5, 2026
Merged

rc8: reconcile stack QA fixes#417
OisinKyne merged 21 commits into
mainfrom
integration/rc8-superset

Conversation

@bussyjd

@bussyjdbussyjd commented May 4, 2026

Copy link
Copy Markdown
Contributor

Summary

What changed:

  • Reconciles the rc8 WIP stack fixes into one integration branch: model routing order, Hermes/frontend image/idempotency work, Helm 3 compatibility, RBAC tightening, QA runbook refresh, OBOL buyer-flow structural assertions, and paid-flow hardening.
  • Adds one intentionally WIP diagnostic commit: fix(flows): restart verifier after local pricing.

Why it matters:

  • This branch is meant to supersede the fragmented WIP PRs and give reviewers one place to inspect the rc8 candidate.
  • The verifier restart is not presented as the final production fix. It proves the local paid-flow failure is a verifier config-reload problem: before the rollout, the local facilitator received only /supported; after the rollout, the paid legacy slice hit /verify and /settle and passed.

Risk level: high while draft

Commit under test: 137a72f63f748d33ff5a1b25fda9c8f33dfd7f85

Base branch: main

Scope

  • Code
  • Charts / manifests
  • Flows / QA scripts
  • Docs / skills
  • Images / dependencies
  • Other: reconciliation of open WIP branches

Validation

CI checks:

CheckStatusLink
GitHub CIpendingdraft PR

Unit tests:

go test ./... -count=1
PASS on local dev machine before push

Integration tests:

bash -n flows/*.sh
PASS
git diff --check
PASS

Flow tests:

FlowNetworkQA machine labelWorktreeResultArtifacts
flow-13-dual-stack-obol.shBase Sepolia Anvil forkQA machine Aisolated rc8 worktreePASSreceipt summary retained on QA machine
flow-02 -> flow-06 -> flow-07 -> flow-10 -> flow-08 -> flow-09Base Sepolia Anvil forkQA machine Aisolated rc8 worktreePASS after verifier-rollout diagnosticlegacy paid log retained on QA machine
release-smoke.shmixedQA machine Aisolated rc8 worktreeNOT GREEN YETlatest full run diagnosed flow-08 stale verifier config

Release smoke:

Full release smoke is not being claimed green on this draft.
Earlier full run reached/passed live OBOL flow-14 and fork flow-13 segments, but later full runs exposed:
1. flow-13 false negative on empty agent final text despite PurchaseRequest/settlement succeeding; fixed by structural assertions.
2. flow-10 Anvil fork readiness race; fixed by polling readiness.
3. flow-08 stale x402-verifier facilitator config after local pricing change; WIP diagnostic rollout proves the cause but should be replaced before ready review.

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network:

  • Live gate: Base Sepolia, rerun pending on current HEAD.
  • Fork regression: Anvil fork of Base Sepolia.

RPC/provider:

  • Live: configured via existing QA environment, not copied here.
  • Fork: local Anvil in isolated QA worktree.

Facilitator:

  • Live flow uses the deployed public facilitator.
  • Fork/legacy paid flows use the x402-rs facilitator image; current tag still needs final review before this PR leaves draft.

Contracts and tokens:

NameAddressVersion / notes
OBOL Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4live smoke token
Fork OBOLrun-generatedForkObolToken, EIP-712 name checked in flow-13

Wallet roles:

RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical derived seller wallet
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Ecanonical derived buyer wallet
Facilitator / receiverpending final live rerundo not infer from local fork logs

Balances:

TokenAddressBeforeAfterExpected deltaActual delta
Fork OBOLAlice seller1000000000000000000010001000000000000000+1000000000000000+1000000000000000
Fork OBOLBob buyer100000000000000000009999000000000000000-1000000000000000-1000000000000000

Transaction receipts:

PurposeTx hashFromToAmount / eventStatus
flow-13 settlement transfer0x8f6c93fa25ca98b5e41ca3888eb2179288e8f3d5ddf4771a12982d2c62931fb8BobAlice1000000000000000 atomic OBOL unitsPASS
legacy paid settlement transfer0x6ca4b44c24c5d03c56d78fb5487e011cd06d3a65ef72471ccb9b6866980e3dcbbuyerseller1000 USDC atomic units in fork flowPASS

Runtime Evidence

QA environment:

ItemValue
OS / archLinux QA machine, exact host omitted
Backendk3d stack worktrees
Tool versionslocal toolchain from stack workspace; exact versions to capture in final RC run
QA agent/modelOpenAI-compatible vLLM/llama.cpp endpoint, qwen36-fast

Images:

ComponentImageTag / digestSource
x402 local facilitatorghcr.io/x402-rs/x402-facilitatorcurrent flow tag under reviewGHCR
cloudflaredcloudflare/cloudflaredchart tag plus Renovate coverage to confirmDocker Hub

Kubernetes / stack:

ItemValue
Stack IDsisolated per-flow QA worktrees
Namespacesgenerated by each stack flow
Pod readinesschecked by flows before payment assertions
Cleanup resultlegacy paid slice cleanup passed; full release-smoke cleanup/rerun still pending

Model and routing:

ItemValue
Agent/model usedqwen36-fast
LiteLLM routeobol model setup custom --no-sync, obol model prefer, single obol model sync
Paid endpoint statusHTTP 200 paid inference observed in flow-13 and legacy paid slice
Auth token sourceobol agent auth --runtime ... obol-agent

Artifacts and logs:

ArtifactLocation / linkNotes
flow-13 receiptsretained on QA machineincludes receipt summary and settlement evidence
legacy paid logretained on QA machineincludes flow-10/08/09 pass and facilitator /verify + /settle evidence
release-smoke reportretained on QA machinelatest full run is diagnostic, not a green RC receipt

Demo readiness:

ItemStatusNotes
Seller visible / registeredpartiallive OBOL current-HEAD rerun pending
Buyer discovery worksPASS in flow-13agent buy structurally confirmed by PurchaseRequest
Paid route worksPASS in flow-13 and legacy paid sliceHTTP 200 paid inference observed
Settlement visible on-chainPASS in fork flowslive current-HEAD rerun pending

Review Notes

Known gaps:

  • The x402-verifier restart after local pricing change is a diagnostic WIP, not the desired long-term behavior. Final fix should make config reload deterministic through the chart/controller/reloader path.
  • Full release smoke must be rerun from a clean QA worktree and be 100% green before this moves out of draft.
  • x402 facilitator and cloudflared image tags need one final review before ready review.

Follow-ups:

  • Replace the verifier restart with the real reload fix.
  • Rerun the full flow fleet and update this template with final receipts, balances, image versions, and cleanup status.

Reviewer focus:

  • Is the verifier config reload diagnosis correct?
  • Should the final fix live in the x402 chart, controller reconciliation, or verifier config-watcher behavior?
  • Confirm the RBAC tightening keeps the agent functional without over-broad write permissions.

@bussyjd

bussyjd commented May 4, 2026

Copy link
Copy Markdown
ContributorAuthor

PR reconciliation pass

Fetched current remote refs and compared the open PR heads against integration/rc8-superset.

PRStatus in #417Notes
#412fix/model-config-orderIncludedorigin/fix/model-config-order is an ancestor of #417. range-diff shows the two model-order commits preserved exactly.
#415oisin/413-1Included as patch-equivalentThe branch was force-updated after #417 was opened, so it is not an ancestor by SHA, but range-diff maps its current commits to #417 (0654685 -> 9d78608, e41faed -> c3fb745) with no PR-side-only patch left over.
#411Hermes recreation issuesClosed; safe subset retainedThe Hermes recreation lifecycle fixes/tests are already on main (eec1a91, ea341c7) and therefore present in #417. The selected #411 work that #415 was meant to keep is also present in #417.

Intentional supersession:

No missing open-PR work found in this pass.

@bussyjd
bussyjd marked this pull request as ready for review May 4, 2026 21:50
@bussyjdbussyjd changed the title WIP rc8: reconcile stack QA fixesrc8: reconcile stack QA fixesMay 4, 2026
@bussyjd

bussyjd commented May 4, 2026

Copy link
Copy Markdown
ContributorAuthor

Summary

What changed: rc8 superset branch reconciles the flow fixes needed for deterministic release smoke: stale ingress URL handling, fork RPC resolution for flow 10, and cluster-side Anvil reachability probing for flow 13.

Why it matters: the QA suite now distinguishes live Base Sepolia OBOL from the Anvil fork regression and the full seller/buyer payment lifecycle passes end-to-end through Hermes, LiteLLM, x402, remote-signer, and on-chain settlement.

Risk level: medium

Commit under test: 8bf94c279ba9e870c5662963469ff2a9f559fce9

Base branch: main

Scope

  • Code
  • Charts / manifests
  • Flows / QA scripts
  • Docs / skills
  • Images / dependencies
  • Other:

Validation

CI checks:

CheckStatusLink
GitHub CIPending after ready-for-review pushPR checks

Unit tests:

bash -n flows/*.sh
PASS
git diff --check
PASS
go test ./cmd/obol ./internal/stack ./internal/hermes ./internal/model ./internal/embed ./internal/x402 -count=1
PASS

Integration tests:

go test -tags integration -v -run TestIntegration_Tunnel_SellDiscoverBuySidecar_QuotaAndBalance -timeout 30m ./internal/openclaw/
PASS/SKIP locally: skipped because no local kubeconfig was active.
go test -tags integration -v -run TestBDDIntegration -timeout 10m ./internal/x402/
Not used as the final gate locally: local setup timed out while preloading/building the dev stack before assertions. Covered by the full remote release-smoke run below.

Flow tests:

FlowNetworkQA machine labelWorktreeResultArtifacts
flow-01-prerequisiteslocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-02-stack-init-uplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-03-inferencelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-04-agentlocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-05-networklocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-06-sell-setuplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-07-sell-verifylocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-10-anvil-facilitatorBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-08-buyBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSsettlement receipt archived
flow-09-lifecyclelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-11-dual-stackBase Sepolia live USDCQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-14-live-obol-base-sepoliaBase Sepolia live OBOLQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-13-dual-stack-obolBase Sepolia fork OBOLQA machine Aqa-fullsmoke-20260505-035957PASSmint, funding, settlement receipts archived

Release smoke:

RELEASE_SMOKE_INCLUDE_OBOL=true RELEASE_SMOKE_INCLUDE_OBOL_FORK=true bash flows/release-smoke.sh
Result: PASS
FAIL lines: 0
SKIP lines: 0
Exit code: 0

Live Chain Evidence

Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.

Network: Base Sepolia live and Base Sepolia fork regression

RPC/provider: public Base Sepolia RPC selected by flow resolver

Facilitator: public Obol facilitator for live flows; local x402-rs facilitator container for fork regression

Contracts and tokens:

NameAddressVersion / notes
OBOL token, live Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4name=Obol Network, symbol=OBOL, decimals=18
OBOL token, fork regression0xeA083Bd4fAfF8aec92123DC7c246aB0E7886A429deployed by flow-13 on Anvil fork
USDC, Base Sepoliacanonical Base Sepolia USDCdecimals=6

Wallet roles:

RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical funded QA seller key
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Edeterministic derived buyer key, preseeded into remote-signer
Bob remote-signer0x57b0eF875DeB5A37301F1640E469a2129Da9490Everified by flows before purchase

Balances:

TokenAddressBeforeAfterExpected deltaActual delta
USDC live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E89330008934000+1000 micro-USDC+1000 micro-USDC
USDC live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E49850004984000-1000 micro-USDC-1000 micro-USDC
OBOL live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1200000000000000013000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E999998994000000000000000999998993000000000000000-1000000000000000 wei-1000000000000000 wei
OBOL fork, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1000000000000000000010001000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL fork, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E100000000000000000009999000000000000000-1000000000000000 wei-1000000000000000 wei
USDC fork buy, buyerflow-08 buyer1000000000999999000-1000 micro-USDC-1000 micro-USDC
USDC fork buy, sellerflow-08 seller465417851465418851+1000 micro-USDC+1000 micro-USDC

Transaction receipts:

PurposeTx hashFromToAmount / eventStatus
USDC ERC-8004 registration0x82be088a8f9d4b71e900ef907f4665e449746bac85d9039b9538fa3a6231d1f3AliceERC-8004 registryAgent ID 5486PASS
USDC metadata / service offer0x6133ee020141363e0c014df3fec3b0d7e2da03a6f1102bb226523089d4d6e919AliceERC-8004 registrymetadata publishedPASS
USDC settlement transfer0x9ec3a1993499d1fb4158fcd20afa310279e700a651eed194c79383068978673eBob signerAlice1000 micro-USDCPASS
Live OBOL ERC-8004 registration0x345102a13cd6637b4459abdd12d32e51628945a95396105da1592cf05bd03145AliceERC-8004 registryAgent ID 5487PASS
Live OBOL metadata / service offer0x8b61289c6014c205d854d594f9f2c364cf1689bb9f45d2da42d7daceb4bffd2aAliceERC-8004 registrymetadata publishedPASS
Live OBOL settlement transfer0x7378375d8dda7406c04752594371e9bfb2248dbb4e5a22074fd1799ad3e479e6Bob signerAlice1000000000000000 wei OBOLPASS
Fork OBOL buyer funding0x06c9a4af53a448a2adda95da2b7d7fe743f7c653b5f2e9b046d9109a5d7a304afork minterBob signer10 OBOLPASS
Fork OBOL settlement transfer0x26253a6f994c7be25a3bdbd3609561a6ca354e182a0ec2eed601546fcce0b960Bob signerAlice1000000000000000 wei OBOLPASS
Fork USDC settlement transfer0x71c148bc426d7e0ada0de60752855097551ca4353cc09db02fbe312846edb08abuyerseller1000 micro-USDCPASS

Runtime Evidence

QA environment:

ItemValue
OS / archLinux aarch64
Backendk3d v5.8.3, k3s v1.31.5-k3s1
Tool versionsDocker 29.2.1, kubectl client v1.35.3, Helm v3.20.1, helmfile 1.4.3, Go 1.25.3, Foundry 1.5.1-stable
QA agent/modelOpenAI-compatible QA endpoint, qwen36-fast

Images:

ComponentImageTag / digestSource
cloudflaredcloudflare/cloudflared2026.3.0embedded cloudflared values
x402 facilitatorghcr.io/x402-rs/x402-facilitator1.4.7flow-10 / flow-13 fork tests
x402 verifierghcr.io/obolnetwork/x402-verifierlatestembedded base manifest
x402 buyerghcr.io/obolnetwork/x402-buyerlatestembedded LiteLLM sidecar manifest
LiteLLMghcr.io/obolnetwork/litellmsha-c16b156embedded base manifest
OpenClawghcr.io/obolnetwork/openclawv2026.4.21pinned OPENCLAW_VERSION
OpenClaw chartobol/openclaw0.4.0CLI-generated agent helmfile
remote-signer chartobol/remote-signer0.3.2CLI-generated agent helmfile
eRPCghcr.io/erpc/erpc0.0.62embedded eRPC values

Kubernetes / stack:

ItemValue
Stack IDsunique per flow worktree; removed by cleanup
Namespacesx402, llm, erpc, traefik, hermes-obol-agent exercised
Pod readinessx402 verifier/buyer, LiteLLM, Hermes API, remote-signer, eRPC, and cloudflared rolled out in flows
Cleanup resultno QA tmux sessions, no k3d clusters, no flow containers, no Anvil fork processes remaining

Model and routing:

ItemValue
Agent/model usedqwen36-fast
LiteLLM routepaid/qwen36-fast through x402-buyer sidecar
Paid endpoint statusHTTP 200 in USDC live, OBOL live, and OBOL fork paid inference
Auth token sourceobol agent auth --runtime hermes obol-agent; raw bearer tokens omitted

Artifacts and logs:

ArtifactLocation / linkNotes
Release reportretained on QA machine as release-smoke-20260505-035957-artifacts/RELEASE_REPORT.mdsanitized summary copied above
Receipt bundlesretained on QA machine under flow-11-receipts, flow-14-receipts, flow-13-receiptsincludes registration, metadata, mint/funding, settlement receipts

Demo readiness:

ItemStatusNotes
Seller visible / registeredPASSERC-8004 registration reflected for live USDC and live OBOL flows
Buyer discovery worksPASSHermes agent discovery issued and purchase confirmed by PurchaseRequest Ready=True
Paid route worksPASSpaid/qwen36-fast returned HTTP 200 with coherent response
Settlement visible on-chainPASSexact balance deltas and settlement receipts verified

Review Notes

Known gaps:

  • Standalone local integration BDD command was not used as the release gate because the local dev stack setup timed out before assertions. The remote release-smoke suite covered the end-to-end behavior and passed fully.
  • x402-buyer and x402-verifier still use latest; image pinning/digesting remains a follow-up unless intentionally handled in this PR.

Follow-ups:

  • Replace latest x402 sidecar images with stable tags or digests when CI publishes a suitable artifact.
  • Decide whether the verifier restart after x402 pricing changes should be replaced by a controller/runtime reload path rather than remaining a flow workaround.

Reviewer focus:

  • Confirm the flow split is clear: live Base Sepolia OBOL is the release gate, Anvil fork OBOL is an explicit regression path.
  • Review the flow-10 RPC resolver and flow-13 cluster-side Anvil probe for portability across QA machines.
  • Review stale OBOL_INGRESS_URL handling so .env defaults do not override dynamic high-port ingress during parallel QA.

This was referenced May 4, 2026
Closed
Closed
OisinKyne
OisinKyne approved these changes May 5, 2026
View reviewed changes

@OisinKyneOisinKyne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

biggest thing i worry about is the sponsorship removal and if we documented that everywhere. the mainnet sponsored facilitator tried to set a 7702 delegation to an unknown wallet.

We can do 7702 stuff in its own pr in future to handle broad gas sponsorship.

@OisinKyne
OisinKyne merged commit a6cd2c2 into mainMay 5, 2026
6 checks passed
@OisinKyne
OisinKyne deleted the integration/rc8-superset branch May 5, 2026 00:15
@bussyjdbussyjd mentioned this pull request Jun 27, 2026
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@OisinKyneOisinKyneOisinKyne approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@bussyjd@OisinKyne
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
rc8: reconcile stack QA fixes by bussyjd · Pull Request #417 · ObolNetwork/obol-stack · GitHub
Skip to content
rc8: reconcile stack QA fixes - #417
Merged
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
May 5, 2026
Merged
rc8: reconcile stack QA fixes#417
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
Conversation
@bussyjd
@bussyjdbussyjd
commented
May 4, 2026

Copy link


Copy Markdown

Contributor
Summary
What changed:
  • Reconciles the rc8 WIP stack fixes into one integration branch: model routing order, Hermes/frontend image/idempotency work, Helm 3 compatibility, RBAC tightening, QA runbook refresh, OBOL buyer-flow structural assertions, and paid-flow hardening.
  • Adds one intentionally WIP diagnostic commit: fix(flows): restart verifier after local pricing.
Why it matters:
  • This branch is meant to supersede the fragmented WIP PRs and give reviewers one place to inspect the rc8 candidate.
  • The verifier restart is not presented as the final production fix. It proves the local paid-flow failure is a verifier config-reload problem: before the rollout, the local facilitator received only /supported; after the rollout, the paid legacy slice hit /verify and /settle and passed.
Risk level: high while draft
Commit under test: 137a72f63f748d33ff5a1b25fda9c8f33dfd7f85
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other: reconciliation of open WIP branches
Validation
CI checks:
CheckStatusLink
GitHub CIpendingdraft PR
Unit tests:
go test ./... -count=1
PASS on local dev machine before push

Integration tests:
bash -n flows/*.sh
PASS
git diff --check
PASS

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-13-dual-stack-obol.shBase Sepolia Anvil forkQA machine Aisolated rc8 worktreePASSreceipt summary retained on QA machine
flow-02 -> flow-06 -> flow-07 -> flow-10 -> flow-08 -> flow-09Base Sepolia Anvil forkQA machine Aisolated rc8 worktreePASS after verifier-rollout diagnosticlegacy paid log retained on QA machine
release-smoke.shmixedQA machine Aisolated rc8 worktreeNOT GREEN YETlatest full run diagnosed flow-08 stale verifier config
Release smoke:
Full release smoke is not being claimed green on this draft.
Earlier full run reached/passed live OBOL flow-14 and fork flow-13 segments, but later full runs exposed:
1. flow-13 false negative on empty agent final text despite PurchaseRequest/settlement succeeding; fixed by structural assertions.
2. flow-10 Anvil fork readiness race; fixed by polling readiness.
3. flow-08 stale x402-verifier facilitator config after local pricing change; WIP diagnostic rollout proves the cause but should be replaced before ready review.

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network:
  • Live gate: Base Sepolia, rerun pending on current HEAD.
  • Fork regression: Anvil fork of Base Sepolia.
RPC/provider:
  • Live: configured via existing QA environment, not copied here.
  • Fork: local Anvil in isolated QA worktree.
Facilitator:
  • Live flow uses the deployed public facilitator.
  • Fork/legacy paid flows use the x402-rs facilitator image; current tag still needs final review before this PR leaves draft.
Contracts and tokens:
NameAddressVersion / notes
OBOL Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4live smoke token
Fork OBOLrun-generatedForkObolToken, EIP-712 name checked in flow-13
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical derived seller wallet
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Ecanonical derived buyer wallet
Facilitator / receiverpending final live rerundo not infer from local fork logs
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
Fork OBOLAlice seller1000000000000000000010001000000000000000+1000000000000000+1000000000000000
Fork OBOLBob buyer100000000000000000009999000000000000000-1000000000000000-1000000000000000
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
flow-13 settlement transfer0x8f6c93fa25ca98b5e41ca3888eb2179288e8f3d5ddf4771a12982d2c62931fb8BobAlice1000000000000000 atomic OBOL unitsPASS
legacy paid settlement transfer0x6ca4b44c24c5d03c56d78fb5487e011cd06d3a65ef72471ccb9b6866980e3dcbbuyerseller1000 USDC atomic units in fork flowPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux QA machine, exact host omitted
Backendk3d stack worktrees
Tool versionslocal toolchain from stack workspace; exact versions to capture in final RC run
QA agent/modelOpenAI-compatible vLLM/llama.cpp endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
x402 local facilitatorghcr.io/x402-rs/x402-facilitatorcurrent flow tag under reviewGHCR
cloudflaredcloudflare/cloudflaredchart tag plus Renovate coverage to confirmDocker Hub
Kubernetes / stack:
ItemValue
Stack IDsisolated per-flow QA worktrees
Namespacesgenerated by each stack flow
Pod readinesschecked by flows before payment assertions
Cleanup resultlegacy paid slice cleanup passed; full release-smoke cleanup/rerun still pending
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routeobol model setup custom --no-sync, obol model prefer, single obol model sync
Paid endpoint statusHTTP 200 paid inference observed in flow-13 and legacy paid slice
Auth token sourceobol agent auth --runtime ... obol-agent
Artifacts and logs:
ArtifactLocation / linkNotes
flow-13 receiptsretained on QA machineincludes receipt summary and settlement evidence
legacy paid logretained on QA machineincludes flow-10/08/09 pass and facilitator /verify + /settle evidence
release-smoke reportretained on QA machinelatest full run is diagnostic, not a green RC receipt
Demo readiness:
ItemStatusNotes
Seller visible / registeredpartiallive OBOL current-HEAD rerun pending
Buyer discovery worksPASS in flow-13agent buy structurally confirmed by PurchaseRequest
Paid route worksPASS in flow-13 and legacy paid sliceHTTP 200 paid inference observed
Settlement visible on-chainPASS in fork flowslive current-HEAD rerun pending
Review Notes
Known gaps:
  • The x402-verifier restart after local pricing change is a diagnostic WIP, not the desired long-term behavior. Final fix should make config reload deterministic through the chart/controller/reloader path.
  • Full release smoke must be rerun from a clean QA worktree and be 100% green before this moves out of draft.
  • x402 facilitator and cloudflared image tags need one final review before ready review.
Follow-ups:
  • Replace the verifier restart with the real reload fix.
  • Rerun the full flow fleet and update this template with final receipts, balances, image versions, and cleanup status.
Reviewer focus:
  • Is the verifier config reload diagnosis correct?
  • Should the final fix live in the x402 chart, controller reconciliation, or verifier config-watcher behavior?
  • Confirm the RBAC tightening keeps the agent functional without over-broad write permissions.
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
PR reconciliation pass
Fetched current remote refs and compared the open PR heads against integration/rc8-superset.
PRStatus in #417Notes
#412fix/model-config-orderIncludedorigin/fix/model-config-order is an ancestor of #417. range-diff shows the two model-order commits preserved exactly.
#415oisin/413-1Included as patch-equivalentThe branch was force-updated after #417 was opened, so it is not an ancestor by SHA, but range-diff maps its current commits to #417 (0654685 -> 9d78608, e41faed -> c3fb745) with no PR-side-only patch left over.
#411Hermes recreation issuesClosed; safe subset retainedThe Hermes recreation lifecycle fixes/tests are already on main (eec1a91, ea341c7) and therefore present in #417. The selected #411 work that #415 was meant to keep is also present in #417.
Intentional supersession:
No missing open-PR work found in this pass.
@bussyjd
bussyjd
marked this pull request as ready for review
May 4, 2026 21:50
@bussyjdbussyjd
changed the title
WIP rc8: reconcile stack QA fixesrc8: reconcile stack QA fixesMay 4, 2026
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
Summary
What changed: rc8 superset branch reconciles the flow fixes needed for deterministic release smoke: stale ingress URL handling, fork RPC resolution for flow 10, and cluster-side Anvil reachability probing for flow 13.
Why it matters: the QA suite now distinguishes live Base Sepolia OBOL from the Anvil fork regression and the full seller/buyer payment lifecycle passes end-to-end through Hermes, LiteLLM, x402, remote-signer, and on-chain settlement.
Risk level: medium
Commit under test: 8bf94c279ba9e870c5662963469ff2a9f559fce9
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other:
Validation
CI checks:
CheckStatusLink
GitHub CIPending after ready-for-review pushPR checks
Unit tests:
bash -n flows/*.sh
PASS
git diff --check
PASS
go test ./cmd/obol ./internal/stack ./internal/hermes ./internal/model ./internal/embed ./internal/x402 -count=1
PASS

Integration tests:
go test -tags integration -v -run TestIntegration_Tunnel_SellDiscoverBuySidecar_QuotaAndBalance -timeout 30m ./internal/openclaw/
PASS/SKIP locally: skipped because no local kubeconfig was active.
go test -tags integration -v -run TestBDDIntegration -timeout 10m ./internal/x402/
Not used as the final gate locally: local setup timed out while preloading/building the dev stack before assertions. Covered by the full remote release-smoke run below.

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-01-prerequisiteslocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-02-stack-init-uplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-03-inferencelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-04-agentlocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-05-networklocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-06-sell-setuplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-07-sell-verifylocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-10-anvil-facilitatorBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-08-buyBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSsettlement receipt archived
flow-09-lifecyclelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-11-dual-stackBase Sepolia live USDCQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-14-live-obol-base-sepoliaBase Sepolia live OBOLQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-13-dual-stack-obolBase Sepolia fork OBOLQA machine Aqa-fullsmoke-20260505-035957PASSmint, funding, settlement receipts archived
Release smoke:
RELEASE_SMOKE_INCLUDE_OBOL=true RELEASE_SMOKE_INCLUDE_OBOL_FORK=true bash flows/release-smoke.sh
Result: PASS
FAIL lines: 0
SKIP lines: 0
Exit code: 0

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network: Base Sepolia live and Base Sepolia fork regression
RPC/provider: public Base Sepolia RPC selected by flow resolver
Facilitator: public Obol facilitator for live flows; local x402-rs facilitator container for fork regression
Contracts and tokens:
NameAddressVersion / notes
OBOL token, live Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4name=Obol Network, symbol=OBOL, decimals=18
OBOL token, fork regression0xeA083Bd4fAfF8aec92123DC7c246aB0E7886A429deployed by flow-13 on Anvil fork
USDC, Base Sepoliacanonical Base Sepolia USDCdecimals=6
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical funded QA seller key
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Edeterministic derived buyer key, preseeded into remote-signer
Bob remote-signer0x57b0eF875DeB5A37301F1640E469a2129Da9490Everified by flows before purchase
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
USDC live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E89330008934000+1000 micro-USDC+1000 micro-USDC
USDC live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E49850004984000-1000 micro-USDC-1000 micro-USDC
OBOL live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1200000000000000013000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E999998994000000000000000999998993000000000000000-1000000000000000 wei-1000000000000000 wei
OBOL fork, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1000000000000000000010001000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL fork, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E100000000000000000009999000000000000000-1000000000000000 wei-1000000000000000 wei
USDC fork buy, buyerflow-08 buyer1000000000999999000-1000 micro-USDC-1000 micro-USDC
USDC fork buy, sellerflow-08 seller465417851465418851+1000 micro-USDC+1000 micro-USDC
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
USDC ERC-8004 registration0x82be088a8f9d4b71e900ef907f4665e449746bac85d9039b9538fa3a6231d1f3AliceERC-8004 registryAgent ID 5486PASS
USDC metadata / service offer0x6133ee020141363e0c014df3fec3b0d7e2da03a6f1102bb226523089d4d6e919AliceERC-8004 registrymetadata publishedPASS
USDC settlement transfer0x9ec3a1993499d1fb4158fcd20afa310279e700a651eed194c79383068978673eBob signerAlice1000 micro-USDCPASS
Live OBOL ERC-8004 registration0x345102a13cd6637b4459abdd12d32e51628945a95396105da1592cf05bd03145AliceERC-8004 registryAgent ID 5487PASS
Live OBOL metadata / service offer0x8b61289c6014c205d854d594f9f2c364cf1689bb9f45d2da42d7daceb4bffd2aAliceERC-8004 registrymetadata publishedPASS
Live OBOL settlement transfer0x7378375d8dda7406c04752594371e9bfb2248dbb4e5a22074fd1799ad3e479e6Bob signerAlice1000000000000000 wei OBOLPASS
Fork OBOL buyer funding0x06c9a4af53a448a2adda95da2b7d7fe743f7c653b5f2e9b046d9109a5d7a304afork minterBob signer10 OBOLPASS
Fork OBOL settlement transfer0x26253a6f994c7be25a3bdbd3609561a6ca354e182a0ec2eed601546fcce0b960Bob signerAlice1000000000000000 wei OBOLPASS
Fork USDC settlement transfer0x71c148bc426d7e0ada0de60752855097551ca4353cc09db02fbe312846edb08abuyerseller1000 micro-USDCPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux aarch64
Backendk3d v5.8.3, k3s v1.31.5-k3s1
Tool versionsDocker 29.2.1, kubectl client v1.35.3, Helm v3.20.1, helmfile 1.4.3, Go 1.25.3, Foundry 1.5.1-stable
QA agent/modelOpenAI-compatible QA endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
cloudflaredcloudflare/cloudflared2026.3.0embedded cloudflared values
x402 facilitatorghcr.io/x402-rs/x402-facilitator1.4.7flow-10 / flow-13 fork tests
x402 verifierghcr.io/obolnetwork/x402-verifierlatestembedded base manifest
x402 buyerghcr.io/obolnetwork/x402-buyerlatestembedded LiteLLM sidecar manifest
LiteLLMghcr.io/obolnetwork/litellmsha-c16b156embedded base manifest
OpenClawghcr.io/obolnetwork/openclawv2026.4.21pinned OPENCLAW_VERSION
OpenClaw chartobol/openclaw0.4.0CLI-generated agent helmfile
remote-signer chartobol/remote-signer0.3.2CLI-generated agent helmfile
eRPCghcr.io/erpc/erpc0.0.62embedded eRPC values
Kubernetes / stack:
ItemValue
Stack IDsunique per flow worktree; removed by cleanup
Namespacesx402, llm, erpc, traefik, hermes-obol-agent exercised
Pod readinessx402 verifier/buyer, LiteLLM, Hermes API, remote-signer, eRPC, and cloudflared rolled out in flows
Cleanup resultno QA tmux sessions, no k3d clusters, no flow containers, no Anvil fork processes remaining
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routepaid/qwen36-fast through x402-buyer sidecar
Paid endpoint statusHTTP 200 in USDC live, OBOL live, and OBOL fork paid inference
Auth token sourceobol agent auth --runtime hermes obol-agent; raw bearer tokens omitted
Artifacts and logs:
ArtifactLocation / linkNotes
Release reportretained on QA machine as release-smoke-20260505-035957-artifacts/RELEASE_REPORT.mdsanitized summary copied above
Receipt bundlesretained on QA machine under flow-11-receipts, flow-14-receipts, flow-13-receiptsincludes registration, metadata, mint/funding, settlement receipts
Demo readiness:
ItemStatusNotes
Seller visible / registeredPASSERC-8004 registration reflected for live USDC and live OBOL flows
Buyer discovery worksPASSHermes agent discovery issued and purchase confirmed by PurchaseRequest Ready=True
Paid route worksPASSpaid/qwen36-fast returned HTTP 200 with coherent response
Settlement visible on-chainPASSexact balance deltas and settlement receipts verified
Review Notes
Known gaps:
  • Standalone local integration BDD command was not used as the release gate because the local dev stack setup timed out before assertions. The remote release-smoke suite covered the end-to-end behavior and passed fully.
  • x402-buyer and x402-verifier still use latest; image pinning/digesting remains a follow-up unless intentionally handled in this PR.
Follow-ups:
  • Replace latest x402 sidecar images with stable tags or digests when CI publishes a suitable artifact.
  • Decide whether the verifier restart after x402 pricing changes should be replaced by a controller/runtime reload path rather than remaining a flow workaround.
Reviewer focus:
  • Confirm the flow split is clear: live Base Sepolia OBOL is the release gate, Anvil fork OBOL is an explicit regression path.
  • Review the flow-10 RPC resolver and flow-13 cluster-side Anvil probe for portability across QA machines.
  • Review stale OBOL_INGRESS_URL handling so .env defaults do not override dynamic high-port ingress during parallel QA.

This was referenced May 4, 2026
 Closed

 Closed

OisinKyne
OisinKyne
approved these changes
May 5, 2026
View reviewed changes
@OisinKyneOisinKyne
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

biggest thing i worry about is the sponsorship removal and if we documented that everywhere. the mainnet sponsored facilitator tried to set a 7702 delegation to an unknown wallet.
We can do 7702 stuff in its own pr in future to handle broad gas sponsorship.
@OisinKyne
OisinKyne
merged commit a6cd2c2
into
mainMay 5, 2026

6 checks passed

@OisinKyne
OisinKyne
deleted the

integration/rc8-superset

branch
May 5, 2026 00:15
@bussyjdbussyjd
mentioned this pull request
Jun 27, 2026
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@OisinKyneOisinKyneOisinKyne approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@bussyjd@OisinKyne

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
rc8: reconcile stack QA fixes by bussyjd · Pull Request #417 · ObolNetwork/obol-stack · GitHub
Skip to content
rc8: reconcile stack QA fixes - #417
Merged
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
May 5, 2026
Merged
rc8: reconcile stack QA fixes#417
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
Conversation
@bussyjd
@bussyjdbussyjd
commented
May 4, 2026

Copy link


Copy Markdown

Contributor
Summary
What changed:
  • Reconciles the rc8 WIP stack fixes into one integration branch: model routing order, Hermes/frontend image/idempotency work, Helm 3 compatibility, RBAC tightening, QA runbook refresh, OBOL buyer-flow structural assertions, and paid-flow hardening.
  • Adds one intentionally WIP diagnostic commit: fix(flows): restart verifier after local pricing.
Why it matters:
  • This branch is meant to supersede the fragmented WIP PRs and give reviewers one place to inspect the rc8 candidate.
  • The verifier restart is not presented as the final production fix. It proves the local paid-flow failure is a verifier config-reload problem: before the rollout, the local facilitator received only /supported; after the rollout, the paid legacy slice hit /verify and /settle and passed.
Risk level: high while draft
Commit under test: 137a72f63f748d33ff5a1b25fda9c8f33dfd7f85
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other: reconciliation of open WIP branches
Validation
CI checks:
CheckStatusLink
GitHub CIpendingdraft PR
Unit tests:
go test ./... -count=1
PASS on local dev machine before push

Integration tests:
bash -n flows/*.sh
PASS
git diff --check
PASS

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-13-dual-stack-obol.shBase Sepolia Anvil forkQA machine Aisolated rc8 worktreePASSreceipt summary retained on QA machine
flow-02 -> flow-06 -> flow-07 -> flow-10 -> flow-08 -> flow-09Base Sepolia Anvil forkQA machine Aisolated rc8 worktreePASS after verifier-rollout diagnosticlegacy paid log retained on QA machine
release-smoke.shmixedQA machine Aisolated rc8 worktreeNOT GREEN YETlatest full run diagnosed flow-08 stale verifier config
Release smoke:
Full release smoke is not being claimed green on this draft.
Earlier full run reached/passed live OBOL flow-14 and fork flow-13 segments, but later full runs exposed:
1. flow-13 false negative on empty agent final text despite PurchaseRequest/settlement succeeding; fixed by structural assertions.
2. flow-10 Anvil fork readiness race; fixed by polling readiness.
3. flow-08 stale x402-verifier facilitator config after local pricing change; WIP diagnostic rollout proves the cause but should be replaced before ready review.

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network:
  • Live gate: Base Sepolia, rerun pending on current HEAD.
  • Fork regression: Anvil fork of Base Sepolia.
RPC/provider:
  • Live: configured via existing QA environment, not copied here.
  • Fork: local Anvil in isolated QA worktree.
Facilitator:
  • Live flow uses the deployed public facilitator.
  • Fork/legacy paid flows use the x402-rs facilitator image; current tag still needs final review before this PR leaves draft.
Contracts and tokens:
NameAddressVersion / notes
OBOL Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4live smoke token
Fork OBOLrun-generatedForkObolToken, EIP-712 name checked in flow-13
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical derived seller wallet
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Ecanonical derived buyer wallet
Facilitator / receiverpending final live rerundo not infer from local fork logs
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
Fork OBOLAlice seller1000000000000000000010001000000000000000+1000000000000000+1000000000000000
Fork OBOLBob buyer100000000000000000009999000000000000000-1000000000000000-1000000000000000
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
flow-13 settlement transfer0x8f6c93fa25ca98b5e41ca3888eb2179288e8f3d5ddf4771a12982d2c62931fb8BobAlice1000000000000000 atomic OBOL unitsPASS
legacy paid settlement transfer0x6ca4b44c24c5d03c56d78fb5487e011cd06d3a65ef72471ccb9b6866980e3dcbbuyerseller1000 USDC atomic units in fork flowPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux QA machine, exact host omitted
Backendk3d stack worktrees
Tool versionslocal toolchain from stack workspace; exact versions to capture in final RC run
QA agent/modelOpenAI-compatible vLLM/llama.cpp endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
x402 local facilitatorghcr.io/x402-rs/x402-facilitatorcurrent flow tag under reviewGHCR
cloudflaredcloudflare/cloudflaredchart tag plus Renovate coverage to confirmDocker Hub
Kubernetes / stack:
ItemValue
Stack IDsisolated per-flow QA worktrees
Namespacesgenerated by each stack flow
Pod readinesschecked by flows before payment assertions
Cleanup resultlegacy paid slice cleanup passed; full release-smoke cleanup/rerun still pending
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routeobol model setup custom --no-sync, obol model prefer, single obol model sync
Paid endpoint statusHTTP 200 paid inference observed in flow-13 and legacy paid slice
Auth token sourceobol agent auth --runtime ... obol-agent
Artifacts and logs:
ArtifactLocation / linkNotes
flow-13 receiptsretained on QA machineincludes receipt summary and settlement evidence
legacy paid logretained on QA machineincludes flow-10/08/09 pass and facilitator /verify + /settle evidence
release-smoke reportretained on QA machinelatest full run is diagnostic, not a green RC receipt
Demo readiness:
ItemStatusNotes
Seller visible / registeredpartiallive OBOL current-HEAD rerun pending
Buyer discovery worksPASS in flow-13agent buy structurally confirmed by PurchaseRequest
Paid route worksPASS in flow-13 and legacy paid sliceHTTP 200 paid inference observed
Settlement visible on-chainPASS in fork flowslive current-HEAD rerun pending
Review Notes
Known gaps:
  • The x402-verifier restart after local pricing change is a diagnostic WIP, not the desired long-term behavior. Final fix should make config reload deterministic through the chart/controller/reloader path.
  • Full release smoke must be rerun from a clean QA worktree and be 100% green before this moves out of draft.
  • x402 facilitator and cloudflared image tags need one final review before ready review.
Follow-ups:
  • Replace the verifier restart with the real reload fix.
  • Rerun the full flow fleet and update this template with final receipts, balances, image versions, and cleanup status.
Reviewer focus:
  • Is the verifier config reload diagnosis correct?
  • Should the final fix live in the x402 chart, controller reconciliation, or verifier config-watcher behavior?
  • Confirm the RBAC tightening keeps the agent functional without over-broad write permissions.
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
PR reconciliation pass
Fetched current remote refs and compared the open PR heads against integration/rc8-superset.
PRStatus in #417Notes
#412fix/model-config-orderIncludedorigin/fix/model-config-order is an ancestor of #417. range-diff shows the two model-order commits preserved exactly.
#415oisin/413-1Included as patch-equivalentThe branch was force-updated after #417 was opened, so it is not an ancestor by SHA, but range-diff maps its current commits to #417 (0654685 -> 9d78608, e41faed -> c3fb745) with no PR-side-only patch left over.
#411Hermes recreation issuesClosed; safe subset retainedThe Hermes recreation lifecycle fixes/tests are already on main (eec1a91, ea341c7) and therefore present in #417. The selected #411 work that #415 was meant to keep is also present in #417.
Intentional supersession:
No missing open-PR work found in this pass.
@bussyjd
bussyjd
marked this pull request as ready for review
May 4, 2026 21:50
@bussyjdbussyjd
changed the title
WIP rc8: reconcile stack QA fixesrc8: reconcile stack QA fixesMay 4, 2026
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
Summary
What changed: rc8 superset branch reconciles the flow fixes needed for deterministic release smoke: stale ingress URL handling, fork RPC resolution for flow 10, and cluster-side Anvil reachability probing for flow 13.
Why it matters: the QA suite now distinguishes live Base Sepolia OBOL from the Anvil fork regression and the full seller/buyer payment lifecycle passes end-to-end through Hermes, LiteLLM, x402, remote-signer, and on-chain settlement.
Risk level: medium
Commit under test: 8bf94c279ba9e870c5662963469ff2a9f559fce9
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other:
Validation
CI checks:
CheckStatusLink
GitHub CIPending after ready-for-review pushPR checks
Unit tests:
bash -n flows/*.sh
PASS
git diff --check
PASS
go test ./cmd/obol ./internal/stack ./internal/hermes ./internal/model ./internal/embed ./internal/x402 -count=1
PASS

Integration tests:
go test -tags integration -v -run TestIntegration_Tunnel_SellDiscoverBuySidecar_QuotaAndBalance -timeout 30m ./internal/openclaw/
PASS/SKIP locally: skipped because no local kubeconfig was active.
go test -tags integration -v -run TestBDDIntegration -timeout 10m ./internal/x402/
Not used as the final gate locally: local setup timed out while preloading/building the dev stack before assertions. Covered by the full remote release-smoke run below.

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-01-prerequisiteslocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-02-stack-init-uplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-03-inferencelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-04-agentlocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-05-networklocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-06-sell-setuplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-07-sell-verifylocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-10-anvil-facilitatorBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-08-buyBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSsettlement receipt archived
flow-09-lifecyclelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-11-dual-stackBase Sepolia live USDCQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-14-live-obol-base-sepoliaBase Sepolia live OBOLQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-13-dual-stack-obolBase Sepolia fork OBOLQA machine Aqa-fullsmoke-20260505-035957PASSmint, funding, settlement receipts archived
Release smoke:
RELEASE_SMOKE_INCLUDE_OBOL=true RELEASE_SMOKE_INCLUDE_OBOL_FORK=true bash flows/release-smoke.sh
Result: PASS
FAIL lines: 0
SKIP lines: 0
Exit code: 0

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network: Base Sepolia live and Base Sepolia fork regression
RPC/provider: public Base Sepolia RPC selected by flow resolver
Facilitator: public Obol facilitator for live flows; local x402-rs facilitator container for fork regression
Contracts and tokens:
NameAddressVersion / notes
OBOL token, live Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4name=Obol Network, symbol=OBOL, decimals=18
OBOL token, fork regression0xeA083Bd4fAfF8aec92123DC7c246aB0E7886A429deployed by flow-13 on Anvil fork
USDC, Base Sepoliacanonical Base Sepolia USDCdecimals=6
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical funded QA seller key
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Edeterministic derived buyer key, preseeded into remote-signer
Bob remote-signer0x57b0eF875DeB5A37301F1640E469a2129Da9490Everified by flows before purchase
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
USDC live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E89330008934000+1000 micro-USDC+1000 micro-USDC
USDC live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E49850004984000-1000 micro-USDC-1000 micro-USDC
OBOL live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1200000000000000013000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E999998994000000000000000999998993000000000000000-1000000000000000 wei-1000000000000000 wei
OBOL fork, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1000000000000000000010001000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL fork, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E100000000000000000009999000000000000000-1000000000000000 wei-1000000000000000 wei
USDC fork buy, buyerflow-08 buyer1000000000999999000-1000 micro-USDC-1000 micro-USDC
USDC fork buy, sellerflow-08 seller465417851465418851+1000 micro-USDC+1000 micro-USDC
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
USDC ERC-8004 registration0x82be088a8f9d4b71e900ef907f4665e449746bac85d9039b9538fa3a6231d1f3AliceERC-8004 registryAgent ID 5486PASS
USDC metadata / service offer0x6133ee020141363e0c014df3fec3b0d7e2da03a6f1102bb226523089d4d6e919AliceERC-8004 registrymetadata publishedPASS
USDC settlement transfer0x9ec3a1993499d1fb4158fcd20afa310279e700a651eed194c79383068978673eBob signerAlice1000 micro-USDCPASS
Live OBOL ERC-8004 registration0x345102a13cd6637b4459abdd12d32e51628945a95396105da1592cf05bd03145AliceERC-8004 registryAgent ID 5487PASS
Live OBOL metadata / service offer0x8b61289c6014c205d854d594f9f2c364cf1689bb9f45d2da42d7daceb4bffd2aAliceERC-8004 registrymetadata publishedPASS
Live OBOL settlement transfer0x7378375d8dda7406c04752594371e9bfb2248dbb4e5a22074fd1799ad3e479e6Bob signerAlice1000000000000000 wei OBOLPASS
Fork OBOL buyer funding0x06c9a4af53a448a2adda95da2b7d7fe743f7c653b5f2e9b046d9109a5d7a304afork minterBob signer10 OBOLPASS
Fork OBOL settlement transfer0x26253a6f994c7be25a3bdbd3609561a6ca354e182a0ec2eed601546fcce0b960Bob signerAlice1000000000000000 wei OBOLPASS
Fork USDC settlement transfer0x71c148bc426d7e0ada0de60752855097551ca4353cc09db02fbe312846edb08abuyerseller1000 micro-USDCPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux aarch64
Backendk3d v5.8.3, k3s v1.31.5-k3s1
Tool versionsDocker 29.2.1, kubectl client v1.35.3, Helm v3.20.1, helmfile 1.4.3, Go 1.25.3, Foundry 1.5.1-stable
QA agent/modelOpenAI-compatible QA endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
cloudflaredcloudflare/cloudflared2026.3.0embedded cloudflared values
x402 facilitatorghcr.io/x402-rs/x402-facilitator1.4.7flow-10 / flow-13 fork tests
x402 verifierghcr.io/obolnetwork/x402-verifierlatestembedded base manifest
x402 buyerghcr.io/obolnetwork/x402-buyerlatestembedded LiteLLM sidecar manifest
LiteLLMghcr.io/obolnetwork/litellmsha-c16b156embedded base manifest
OpenClawghcr.io/obolnetwork/openclawv2026.4.21pinned OPENCLAW_VERSION
OpenClaw chartobol/openclaw0.4.0CLI-generated agent helmfile
remote-signer chartobol/remote-signer0.3.2CLI-generated agent helmfile
eRPCghcr.io/erpc/erpc0.0.62embedded eRPC values
Kubernetes / stack:
ItemValue
Stack IDsunique per flow worktree; removed by cleanup
Namespacesx402, llm, erpc, traefik, hermes-obol-agent exercised
Pod readinessx402 verifier/buyer, LiteLLM, Hermes API, remote-signer, eRPC, and cloudflared rolled out in flows
Cleanup resultno QA tmux sessions, no k3d clusters, no flow containers, no Anvil fork processes remaining
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routepaid/qwen36-fast through x402-buyer sidecar
Paid endpoint statusHTTP 200 in USDC live, OBOL live, and OBOL fork paid inference
Auth token sourceobol agent auth --runtime hermes obol-agent; raw bearer tokens omitted
Artifacts and logs:
ArtifactLocation / linkNotes
Release reportretained on QA machine as release-smoke-20260505-035957-artifacts/RELEASE_REPORT.mdsanitized summary copied above
Receipt bundlesretained on QA machine under flow-11-receipts, flow-14-receipts, flow-13-receiptsincludes registration, metadata, mint/funding, settlement receipts
Demo readiness:
ItemStatusNotes
Seller visible / registeredPASSERC-8004 registration reflected for live USDC and live OBOL flows
Buyer discovery worksPASSHermes agent discovery issued and purchase confirmed by PurchaseRequest Ready=True
Paid route worksPASSpaid/qwen36-fast returned HTTP 200 with coherent response
Settlement visible on-chainPASSexact balance deltas and settlement receipts verified
Review Notes
Known gaps:
  • Standalone local integration BDD command was not used as the release gate because the local dev stack setup timed out before assertions. The remote release-smoke suite covered the end-to-end behavior and passed fully.
  • x402-buyer and x402-verifier still use latest; image pinning/digesting remains a follow-up unless intentionally handled in this PR.
Follow-ups:
  • Replace latest x402 sidecar images with stable tags or digests when CI publishes a suitable artifact.
  • Decide whether the verifier restart after x402 pricing changes should be replaced by a controller/runtime reload path rather than remaining a flow workaround.
Reviewer focus:
  • Confirm the flow split is clear: live Base Sepolia OBOL is the release gate, Anvil fork OBOL is an explicit regression path.
  • Review the flow-10 RPC resolver and flow-13 cluster-side Anvil probe for portability across QA machines.
  • Review stale OBOL_INGRESS_URL handling so .env defaults do not override dynamic high-port ingress during parallel QA.

This was referenced May 4, 2026
 Closed

 Closed

OisinKyne
OisinKyne
approved these changes
May 5, 2026
View reviewed changes
@OisinKyneOisinKyne
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

biggest thing i worry about is the sponsorship removal and if we documented that everywhere. the mainnet sponsored facilitator tried to set a 7702 delegation to an unknown wallet.
We can do 7702 stuff in its own pr in future to handle broad gas sponsorship.
@OisinKyne
OisinKyne
merged commit a6cd2c2
into
mainMay 5, 2026

6 checks passed

@OisinKyne
OisinKyne
deleted the

integration/rc8-superset

branch
May 5, 2026 00:15
@bussyjdbussyjd
mentioned this pull request
Jun 27, 2026
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@OisinKyneOisinKyneOisinKyne approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@bussyjd@OisinKyne

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
rc8: reconcile stack QA fixes by bussyjd · Pull Request #417 · ObolNetwork/obol-stack · GitHub
Skip to content
rc8: reconcile stack QA fixes - #417
Merged
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
May 5, 2026
Merged
rc8: reconcile stack QA fixes#417
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
Conversation
@bussyjd
@bussyjdbussyjd
commented
May 4, 2026

Copy link


Copy Markdown

Contributor
Summary
What changed:
  • Reconciles the rc8 WIP stack fixes into one integration branch: model routing order, Hermes/frontend image/idempotency work, Helm 3 compatibility, RBAC tightening, QA runbook refresh, OBOL buyer-flow structural assertions, and paid-flow hardening.
  • Adds one intentionally WIP diagnostic commit: fix(flows): restart verifier after local pricing.
Why it matters:
  • This branch is meant to supersede the fragmented WIP PRs and give reviewers one place to inspect the rc8 candidate.
  • The verifier restart is not presented as the final production fix. It proves the local paid-flow failure is a verifier config-reload problem: before the rollout, the local facilitator received only /supported; after the rollout, the paid legacy slice hit /verify and /settle and passed.
Risk level: high while draft
Commit under test: 137a72f63f748d33ff5a1b25fda9c8f33dfd7f85
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other: reconciliation of open WIP branches
Validation
CI checks:
CheckStatusLink
GitHub CIpendingdraft PR
Unit tests:
go test ./... -count=1
PASS on local dev machine before push

Integration tests:
bash -n flows/*.sh
PASS
git diff --check
PASS

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-13-dual-stack-obol.shBase Sepolia Anvil forkQA machine Aisolated rc8 worktreePASSreceipt summary retained on QA machine
flow-02 -> flow-06 -> flow-07 -> flow-10 -> flow-08 -> flow-09Base Sepolia Anvil forkQA machine Aisolated rc8 worktreePASS after verifier-rollout diagnosticlegacy paid log retained on QA machine
release-smoke.shmixedQA machine Aisolated rc8 worktreeNOT GREEN YETlatest full run diagnosed flow-08 stale verifier config
Release smoke:
Full release smoke is not being claimed green on this draft.
Earlier full run reached/passed live OBOL flow-14 and fork flow-13 segments, but later full runs exposed:
1. flow-13 false negative on empty agent final text despite PurchaseRequest/settlement succeeding; fixed by structural assertions.
2. flow-10 Anvil fork readiness race; fixed by polling readiness.
3. flow-08 stale x402-verifier facilitator config after local pricing change; WIP diagnostic rollout proves the cause but should be replaced before ready review.

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network:
  • Live gate: Base Sepolia, rerun pending on current HEAD.
  • Fork regression: Anvil fork of Base Sepolia.
RPC/provider:
  • Live: configured via existing QA environment, not copied here.
  • Fork: local Anvil in isolated QA worktree.
Facilitator:
  • Live flow uses the deployed public facilitator.
  • Fork/legacy paid flows use the x402-rs facilitator image; current tag still needs final review before this PR leaves draft.
Contracts and tokens:
NameAddressVersion / notes
OBOL Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4live smoke token
Fork OBOLrun-generatedForkObolToken, EIP-712 name checked in flow-13
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical derived seller wallet
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Ecanonical derived buyer wallet
Facilitator / receiverpending final live rerundo not infer from local fork logs
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
Fork OBOLAlice seller1000000000000000000010001000000000000000+1000000000000000+1000000000000000
Fork OBOLBob buyer100000000000000000009999000000000000000-1000000000000000-1000000000000000
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
flow-13 settlement transfer0x8f6c93fa25ca98b5e41ca3888eb2179288e8f3d5ddf4771a12982d2c62931fb8BobAlice1000000000000000 atomic OBOL unitsPASS
legacy paid settlement transfer0x6ca4b44c24c5d03c56d78fb5487e011cd06d3a65ef72471ccb9b6866980e3dcbbuyerseller1000 USDC atomic units in fork flowPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux QA machine, exact host omitted
Backendk3d stack worktrees
Tool versionslocal toolchain from stack workspace; exact versions to capture in final RC run
QA agent/modelOpenAI-compatible vLLM/llama.cpp endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
x402 local facilitatorghcr.io/x402-rs/x402-facilitatorcurrent flow tag under reviewGHCR
cloudflaredcloudflare/cloudflaredchart tag plus Renovate coverage to confirmDocker Hub
Kubernetes / stack:
ItemValue
Stack IDsisolated per-flow QA worktrees
Namespacesgenerated by each stack flow
Pod readinesschecked by flows before payment assertions
Cleanup resultlegacy paid slice cleanup passed; full release-smoke cleanup/rerun still pending
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routeobol model setup custom --no-sync, obol model prefer, single obol model sync
Paid endpoint statusHTTP 200 paid inference observed in flow-13 and legacy paid slice
Auth token sourceobol agent auth --runtime ... obol-agent
Artifacts and logs:
ArtifactLocation / linkNotes
flow-13 receiptsretained on QA machineincludes receipt summary and settlement evidence
legacy paid logretained on QA machineincludes flow-10/08/09 pass and facilitator /verify + /settle evidence
release-smoke reportretained on QA machinelatest full run is diagnostic, not a green RC receipt
Demo readiness:
ItemStatusNotes
Seller visible / registeredpartiallive OBOL current-HEAD rerun pending
Buyer discovery worksPASS in flow-13agent buy structurally confirmed by PurchaseRequest
Paid route worksPASS in flow-13 and legacy paid sliceHTTP 200 paid inference observed
Settlement visible on-chainPASS in fork flowslive current-HEAD rerun pending
Review Notes
Known gaps:
  • The x402-verifier restart after local pricing change is a diagnostic WIP, not the desired long-term behavior. Final fix should make config reload deterministic through the chart/controller/reloader path.
  • Full release smoke must be rerun from a clean QA worktree and be 100% green before this moves out of draft.
  • x402 facilitator and cloudflared image tags need one final review before ready review.
Follow-ups:
  • Replace the verifier restart with the real reload fix.
  • Rerun the full flow fleet and update this template with final receipts, balances, image versions, and cleanup status.
Reviewer focus:
  • Is the verifier config reload diagnosis correct?
  • Should the final fix live in the x402 chart, controller reconciliation, or verifier config-watcher behavior?
  • Confirm the RBAC tightening keeps the agent functional without over-broad write permissions.
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
PR reconciliation pass
Fetched current remote refs and compared the open PR heads against integration/rc8-superset.
PRStatus in #417Notes
#412fix/model-config-orderIncludedorigin/fix/model-config-order is an ancestor of #417. range-diff shows the two model-order commits preserved exactly.
#415oisin/413-1Included as patch-equivalentThe branch was force-updated after #417 was opened, so it is not an ancestor by SHA, but range-diff maps its current commits to #417 (0654685 -> 9d78608, e41faed -> c3fb745) with no PR-side-only patch left over.
#411Hermes recreation issuesClosed; safe subset retainedThe Hermes recreation lifecycle fixes/tests are already on main (eec1a91, ea341c7) and therefore present in #417. The selected #411 work that #415 was meant to keep is also present in #417.
Intentional supersession:
No missing open-PR work found in this pass.
@bussyjd
bussyjd
marked this pull request as ready for review
May 4, 2026 21:50
@bussyjdbussyjd
changed the title
WIP rc8: reconcile stack QA fixesrc8: reconcile stack QA fixesMay 4, 2026
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
Summary
What changed: rc8 superset branch reconciles the flow fixes needed for deterministic release smoke: stale ingress URL handling, fork RPC resolution for flow 10, and cluster-side Anvil reachability probing for flow 13.
Why it matters: the QA suite now distinguishes live Base Sepolia OBOL from the Anvil fork regression and the full seller/buyer payment lifecycle passes end-to-end through Hermes, LiteLLM, x402, remote-signer, and on-chain settlement.
Risk level: medium
Commit under test: 8bf94c279ba9e870c5662963469ff2a9f559fce9
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other:
Validation
CI checks:
CheckStatusLink
GitHub CIPending after ready-for-review pushPR checks
Unit tests:
bash -n flows/*.sh
PASS
git diff --check
PASS
go test ./cmd/obol ./internal/stack ./internal/hermes ./internal/model ./internal/embed ./internal/x402 -count=1
PASS

Integration tests:
go test -tags integration -v -run TestIntegration_Tunnel_SellDiscoverBuySidecar_QuotaAndBalance -timeout 30m ./internal/openclaw/
PASS/SKIP locally: skipped because no local kubeconfig was active.
go test -tags integration -v -run TestBDDIntegration -timeout 10m ./internal/x402/
Not used as the final gate locally: local setup timed out while preloading/building the dev stack before assertions. Covered by the full remote release-smoke run below.

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-01-prerequisiteslocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-02-stack-init-uplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-03-inferencelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-04-agentlocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-05-networklocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-06-sell-setuplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-07-sell-verifylocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-10-anvil-facilitatorBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-08-buyBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSsettlement receipt archived
flow-09-lifecyclelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-11-dual-stackBase Sepolia live USDCQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-14-live-obol-base-sepoliaBase Sepolia live OBOLQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-13-dual-stack-obolBase Sepolia fork OBOLQA machine Aqa-fullsmoke-20260505-035957PASSmint, funding, settlement receipts archived
Release smoke:
RELEASE_SMOKE_INCLUDE_OBOL=true RELEASE_SMOKE_INCLUDE_OBOL_FORK=true bash flows/release-smoke.sh
Result: PASS
FAIL lines: 0
SKIP lines: 0
Exit code: 0

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network: Base Sepolia live and Base Sepolia fork regression
RPC/provider: public Base Sepolia RPC selected by flow resolver
Facilitator: public Obol facilitator for live flows; local x402-rs facilitator container for fork regression
Contracts and tokens:
NameAddressVersion / notes
OBOL token, live Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4name=Obol Network, symbol=OBOL, decimals=18
OBOL token, fork regression0xeA083Bd4fAfF8aec92123DC7c246aB0E7886A429deployed by flow-13 on Anvil fork
USDC, Base Sepoliacanonical Base Sepolia USDCdecimals=6
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical funded QA seller key
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Edeterministic derived buyer key, preseeded into remote-signer
Bob remote-signer0x57b0eF875DeB5A37301F1640E469a2129Da9490Everified by flows before purchase
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
USDC live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E89330008934000+1000 micro-USDC+1000 micro-USDC
USDC live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E49850004984000-1000 micro-USDC-1000 micro-USDC
OBOL live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1200000000000000013000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E999998994000000000000000999998993000000000000000-1000000000000000 wei-1000000000000000 wei
OBOL fork, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1000000000000000000010001000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL fork, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E100000000000000000009999000000000000000-1000000000000000 wei-1000000000000000 wei
USDC fork buy, buyerflow-08 buyer1000000000999999000-1000 micro-USDC-1000 micro-USDC
USDC fork buy, sellerflow-08 seller465417851465418851+1000 micro-USDC+1000 micro-USDC
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
USDC ERC-8004 registration0x82be088a8f9d4b71e900ef907f4665e449746bac85d9039b9538fa3a6231d1f3AliceERC-8004 registryAgent ID 5486PASS
USDC metadata / service offer0x6133ee020141363e0c014df3fec3b0d7e2da03a6f1102bb226523089d4d6e919AliceERC-8004 registrymetadata publishedPASS
USDC settlement transfer0x9ec3a1993499d1fb4158fcd20afa310279e700a651eed194c79383068978673eBob signerAlice1000 micro-USDCPASS
Live OBOL ERC-8004 registration0x345102a13cd6637b4459abdd12d32e51628945a95396105da1592cf05bd03145AliceERC-8004 registryAgent ID 5487PASS
Live OBOL metadata / service offer0x8b61289c6014c205d854d594f9f2c364cf1689bb9f45d2da42d7daceb4bffd2aAliceERC-8004 registrymetadata publishedPASS
Live OBOL settlement transfer0x7378375d8dda7406c04752594371e9bfb2248dbb4e5a22074fd1799ad3e479e6Bob signerAlice1000000000000000 wei OBOLPASS
Fork OBOL buyer funding0x06c9a4af53a448a2adda95da2b7d7fe743f7c653b5f2e9b046d9109a5d7a304afork minterBob signer10 OBOLPASS
Fork OBOL settlement transfer0x26253a6f994c7be25a3bdbd3609561a6ca354e182a0ec2eed601546fcce0b960Bob signerAlice1000000000000000 wei OBOLPASS
Fork USDC settlement transfer0x71c148bc426d7e0ada0de60752855097551ca4353cc09db02fbe312846edb08abuyerseller1000 micro-USDCPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux aarch64
Backendk3d v5.8.3, k3s v1.31.5-k3s1
Tool versionsDocker 29.2.1, kubectl client v1.35.3, Helm v3.20.1, helmfile 1.4.3, Go 1.25.3, Foundry 1.5.1-stable
QA agent/modelOpenAI-compatible QA endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
cloudflaredcloudflare/cloudflared2026.3.0embedded cloudflared values
x402 facilitatorghcr.io/x402-rs/x402-facilitator1.4.7flow-10 / flow-13 fork tests
x402 verifierghcr.io/obolnetwork/x402-verifierlatestembedded base manifest
x402 buyerghcr.io/obolnetwork/x402-buyerlatestembedded LiteLLM sidecar manifest
LiteLLMghcr.io/obolnetwork/litellmsha-c16b156embedded base manifest
OpenClawghcr.io/obolnetwork/openclawv2026.4.21pinned OPENCLAW_VERSION
OpenClaw chartobol/openclaw0.4.0CLI-generated agent helmfile
remote-signer chartobol/remote-signer0.3.2CLI-generated agent helmfile
eRPCghcr.io/erpc/erpc0.0.62embedded eRPC values
Kubernetes / stack:
ItemValue
Stack IDsunique per flow worktree; removed by cleanup
Namespacesx402, llm, erpc, traefik, hermes-obol-agent exercised
Pod readinessx402 verifier/buyer, LiteLLM, Hermes API, remote-signer, eRPC, and cloudflared rolled out in flows
Cleanup resultno QA tmux sessions, no k3d clusters, no flow containers, no Anvil fork processes remaining
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routepaid/qwen36-fast through x402-buyer sidecar
Paid endpoint statusHTTP 200 in USDC live, OBOL live, and OBOL fork paid inference
Auth token sourceobol agent auth --runtime hermes obol-agent; raw bearer tokens omitted
Artifacts and logs:
ArtifactLocation / linkNotes
Release reportretained on QA machine as release-smoke-20260505-035957-artifacts/RELEASE_REPORT.mdsanitized summary copied above
Receipt bundlesretained on QA machine under flow-11-receipts, flow-14-receipts, flow-13-receiptsincludes registration, metadata, mint/funding, settlement receipts
Demo readiness:
ItemStatusNotes
Seller visible / registeredPASSERC-8004 registration reflected for live USDC and live OBOL flows
Buyer discovery worksPASSHermes agent discovery issued and purchase confirmed by PurchaseRequest Ready=True
Paid route worksPASSpaid/qwen36-fast returned HTTP 200 with coherent response
Settlement visible on-chainPASSexact balance deltas and settlement receipts verified
Review Notes
Known gaps:
  • Standalone local integration BDD command was not used as the release gate because the local dev stack setup timed out before assertions. The remote release-smoke suite covered the end-to-end behavior and passed fully.
  • x402-buyer and x402-verifier still use latest; image pinning/digesting remains a follow-up unless intentionally handled in this PR.
Follow-ups:
  • Replace latest x402 sidecar images with stable tags or digests when CI publishes a suitable artifact.
  • Decide whether the verifier restart after x402 pricing changes should be replaced by a controller/runtime reload path rather than remaining a flow workaround.
Reviewer focus:
  • Confirm the flow split is clear: live Base Sepolia OBOL is the release gate, Anvil fork OBOL is an explicit regression path.
  • Review the flow-10 RPC resolver and flow-13 cluster-side Anvil probe for portability across QA machines.
  • Review stale OBOL_INGRESS_URL handling so .env defaults do not override dynamic high-port ingress during parallel QA.

This was referenced May 4, 2026
 Closed

 Closed

OisinKyne
OisinKyne
approved these changes
May 5, 2026
View reviewed changes
@OisinKyneOisinKyne
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

biggest thing i worry about is the sponsorship removal and if we documented that everywhere. the mainnet sponsored facilitator tried to set a 7702 delegation to an unknown wallet.
We can do 7702 stuff in its own pr in future to handle broad gas sponsorship.
@OisinKyne
OisinKyne
merged commit a6cd2c2
into
mainMay 5, 2026

6 checks passed

@OisinKyne
OisinKyne
deleted the

integration/rc8-superset

branch
May 5, 2026 00:15
@bussyjdbussyjd
mentioned this pull request
Jun 27, 2026
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@OisinKyneOisinKyneOisinKyne approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@bussyjd@OisinKyne

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
rc8: reconcile stack QA fixes by bussyjd · Pull Request #417 · ObolNetwork/obol-stack · GitHub
Skip to content
rc8: reconcile stack QA fixes - #417
Merged
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
May 5, 2026
Merged
rc8: reconcile stack QA fixes#417
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
Conversation
@bussyjd
@bussyjdbussyjd
commented
May 4, 2026

Copy link


Copy Markdown

Contributor
Summary
What changed:
  • Reconciles the rc8 WIP stack fixes into one integration branch: model routing order, Hermes/frontend image/idempotency work, Helm 3 compatibility, RBAC tightening, QA runbook refresh, OBOL buyer-flow structural assertions, and paid-flow hardening.
  • Adds one intentionally WIP diagnostic commit: fix(flows): restart verifier after local pricing.
Why it matters:
  • This branch is meant to supersede the fragmented WIP PRs and give reviewers one place to inspect the rc8 candidate.
  • The verifier restart is not presented as the final production fix. It proves the local paid-flow failure is a verifier config-reload problem: before the rollout, the local facilitator received only /supported; after the rollout, the paid legacy slice hit /verify and /settle and passed.
Risk level: high while draft
Commit under test: 137a72f63f748d33ff5a1b25fda9c8f33dfd7f85
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other: reconciliation of open WIP branches
Validation
CI checks:
CheckStatusLink
GitHub CIpendingdraft PR
Unit tests:
go test ./... -count=1
PASS on local dev machine before push

Integration tests:
bash -n flows/*.sh
PASS
git diff --check
PASS

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-13-dual-stack-obol.shBase Sepolia Anvil forkQA machine Aisolated rc8 worktreePASSreceipt summary retained on QA machine
flow-02 -> flow-06 -> flow-07 -> flow-10 -> flow-08 -> flow-09Base Sepolia Anvil forkQA machine Aisolated rc8 worktreePASS after verifier-rollout diagnosticlegacy paid log retained on QA machine
release-smoke.shmixedQA machine Aisolated rc8 worktreeNOT GREEN YETlatest full run diagnosed flow-08 stale verifier config
Release smoke:
Full release smoke is not being claimed green on this draft.
Earlier full run reached/passed live OBOL flow-14 and fork flow-13 segments, but later full runs exposed:
1. flow-13 false negative on empty agent final text despite PurchaseRequest/settlement succeeding; fixed by structural assertions.
2. flow-10 Anvil fork readiness race; fixed by polling readiness.
3. flow-08 stale x402-verifier facilitator config after local pricing change; WIP diagnostic rollout proves the cause but should be replaced before ready review.

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network:
  • Live gate: Base Sepolia, rerun pending on current HEAD.
  • Fork regression: Anvil fork of Base Sepolia.
RPC/provider:
  • Live: configured via existing QA environment, not copied here.
  • Fork: local Anvil in isolated QA worktree.
Facilitator:
  • Live flow uses the deployed public facilitator.
  • Fork/legacy paid flows use the x402-rs facilitator image; current tag still needs final review before this PR leaves draft.
Contracts and tokens:
NameAddressVersion / notes
OBOL Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4live smoke token
Fork OBOLrun-generatedForkObolToken, EIP-712 name checked in flow-13
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical derived seller wallet
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Ecanonical derived buyer wallet
Facilitator / receiverpending final live rerundo not infer from local fork logs
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
Fork OBOLAlice seller1000000000000000000010001000000000000000+1000000000000000+1000000000000000
Fork OBOLBob buyer100000000000000000009999000000000000000-1000000000000000-1000000000000000
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
flow-13 settlement transfer0x8f6c93fa25ca98b5e41ca3888eb2179288e8f3d5ddf4771a12982d2c62931fb8BobAlice1000000000000000 atomic OBOL unitsPASS
legacy paid settlement transfer0x6ca4b44c24c5d03c56d78fb5487e011cd06d3a65ef72471ccb9b6866980e3dcbbuyerseller1000 USDC atomic units in fork flowPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux QA machine, exact host omitted
Backendk3d stack worktrees
Tool versionslocal toolchain from stack workspace; exact versions to capture in final RC run
QA agent/modelOpenAI-compatible vLLM/llama.cpp endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
x402 local facilitatorghcr.io/x402-rs/x402-facilitatorcurrent flow tag under reviewGHCR
cloudflaredcloudflare/cloudflaredchart tag plus Renovate coverage to confirmDocker Hub
Kubernetes / stack:
ItemValue
Stack IDsisolated per-flow QA worktrees
Namespacesgenerated by each stack flow
Pod readinesschecked by flows before payment assertions
Cleanup resultlegacy paid slice cleanup passed; full release-smoke cleanup/rerun still pending
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routeobol model setup custom --no-sync, obol model prefer, single obol model sync
Paid endpoint statusHTTP 200 paid inference observed in flow-13 and legacy paid slice
Auth token sourceobol agent auth --runtime ... obol-agent
Artifacts and logs:
ArtifactLocation / linkNotes
flow-13 receiptsretained on QA machineincludes receipt summary and settlement evidence
legacy paid logretained on QA machineincludes flow-10/08/09 pass and facilitator /verify + /settle evidence
release-smoke reportretained on QA machinelatest full run is diagnostic, not a green RC receipt
Demo readiness:
ItemStatusNotes
Seller visible / registeredpartiallive OBOL current-HEAD rerun pending
Buyer discovery worksPASS in flow-13agent buy structurally confirmed by PurchaseRequest
Paid route worksPASS in flow-13 and legacy paid sliceHTTP 200 paid inference observed
Settlement visible on-chainPASS in fork flowslive current-HEAD rerun pending
Review Notes
Known gaps:
  • The x402-verifier restart after local pricing change is a diagnostic WIP, not the desired long-term behavior. Final fix should make config reload deterministic through the chart/controller/reloader path.
  • Full release smoke must be rerun from a clean QA worktree and be 100% green before this moves out of draft.
  • x402 facilitator and cloudflared image tags need one final review before ready review.
Follow-ups:
  • Replace the verifier restart with the real reload fix.
  • Rerun the full flow fleet and update this template with final receipts, balances, image versions, and cleanup status.
Reviewer focus:
  • Is the verifier config reload diagnosis correct?
  • Should the final fix live in the x402 chart, controller reconciliation, or verifier config-watcher behavior?
  • Confirm the RBAC tightening keeps the agent functional without over-broad write permissions.
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
PR reconciliation pass
Fetched current remote refs and compared the open PR heads against integration/rc8-superset.
PRStatus in #417Notes
#412fix/model-config-orderIncludedorigin/fix/model-config-order is an ancestor of #417. range-diff shows the two model-order commits preserved exactly.
#415oisin/413-1Included as patch-equivalentThe branch was force-updated after #417 was opened, so it is not an ancestor by SHA, but range-diff maps its current commits to #417 (0654685 -> 9d78608, e41faed -> c3fb745) with no PR-side-only patch left over.
#411Hermes recreation issuesClosed; safe subset retainedThe Hermes recreation lifecycle fixes/tests are already on main (eec1a91, ea341c7) and therefore present in #417. The selected #411 work that #415 was meant to keep is also present in #417.
Intentional supersession:
No missing open-PR work found in this pass.
@bussyjd
bussyjd
marked this pull request as ready for review
May 4, 2026 21:50
@bussyjdbussyjd
changed the title
WIP rc8: reconcile stack QA fixesrc8: reconcile stack QA fixesMay 4, 2026
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
Summary
What changed: rc8 superset branch reconciles the flow fixes needed for deterministic release smoke: stale ingress URL handling, fork RPC resolution for flow 10, and cluster-side Anvil reachability probing for flow 13.
Why it matters: the QA suite now distinguishes live Base Sepolia OBOL from the Anvil fork regression and the full seller/buyer payment lifecycle passes end-to-end through Hermes, LiteLLM, x402, remote-signer, and on-chain settlement.
Risk level: medium
Commit under test: 8bf94c279ba9e870c5662963469ff2a9f559fce9
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other:
Validation
CI checks:
CheckStatusLink
GitHub CIPending after ready-for-review pushPR checks
Unit tests:
bash -n flows/*.sh
PASS
git diff --check
PASS
go test ./cmd/obol ./internal/stack ./internal/hermes ./internal/model ./internal/embed ./internal/x402 -count=1
PASS

Integration tests:
go test -tags integration -v -run TestIntegration_Tunnel_SellDiscoverBuySidecar_QuotaAndBalance -timeout 30m ./internal/openclaw/
PASS/SKIP locally: skipped because no local kubeconfig was active.
go test -tags integration -v -run TestBDDIntegration -timeout 10m ./internal/x402/
Not used as the final gate locally: local setup timed out while preloading/building the dev stack before assertions. Covered by the full remote release-smoke run below.

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-01-prerequisiteslocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-02-stack-init-uplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-03-inferencelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-04-agentlocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-05-networklocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-06-sell-setuplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-07-sell-verifylocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-10-anvil-facilitatorBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-08-buyBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSsettlement receipt archived
flow-09-lifecyclelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-11-dual-stackBase Sepolia live USDCQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-14-live-obol-base-sepoliaBase Sepolia live OBOLQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-13-dual-stack-obolBase Sepolia fork OBOLQA machine Aqa-fullsmoke-20260505-035957PASSmint, funding, settlement receipts archived
Release smoke:
RELEASE_SMOKE_INCLUDE_OBOL=true RELEASE_SMOKE_INCLUDE_OBOL_FORK=true bash flows/release-smoke.sh
Result: PASS
FAIL lines: 0
SKIP lines: 0
Exit code: 0

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network: Base Sepolia live and Base Sepolia fork regression
RPC/provider: public Base Sepolia RPC selected by flow resolver
Facilitator: public Obol facilitator for live flows; local x402-rs facilitator container for fork regression
Contracts and tokens:
NameAddressVersion / notes
OBOL token, live Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4name=Obol Network, symbol=OBOL, decimals=18
OBOL token, fork regression0xeA083Bd4fAfF8aec92123DC7c246aB0E7886A429deployed by flow-13 on Anvil fork
USDC, Base Sepoliacanonical Base Sepolia USDCdecimals=6
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical funded QA seller key
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Edeterministic derived buyer key, preseeded into remote-signer
Bob remote-signer0x57b0eF875DeB5A37301F1640E469a2129Da9490Everified by flows before purchase
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
USDC live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E89330008934000+1000 micro-USDC+1000 micro-USDC
USDC live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E49850004984000-1000 micro-USDC-1000 micro-USDC
OBOL live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1200000000000000013000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E999998994000000000000000999998993000000000000000-1000000000000000 wei-1000000000000000 wei
OBOL fork, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1000000000000000000010001000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL fork, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E100000000000000000009999000000000000000-1000000000000000 wei-1000000000000000 wei
USDC fork buy, buyerflow-08 buyer1000000000999999000-1000 micro-USDC-1000 micro-USDC
USDC fork buy, sellerflow-08 seller465417851465418851+1000 micro-USDC+1000 micro-USDC
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
USDC ERC-8004 registration0x82be088a8f9d4b71e900ef907f4665e449746bac85d9039b9538fa3a6231d1f3AliceERC-8004 registryAgent ID 5486PASS
USDC metadata / service offer0x6133ee020141363e0c014df3fec3b0d7e2da03a6f1102bb226523089d4d6e919AliceERC-8004 registrymetadata publishedPASS
USDC settlement transfer0x9ec3a1993499d1fb4158fcd20afa310279e700a651eed194c79383068978673eBob signerAlice1000 micro-USDCPASS
Live OBOL ERC-8004 registration0x345102a13cd6637b4459abdd12d32e51628945a95396105da1592cf05bd03145AliceERC-8004 registryAgent ID 5487PASS
Live OBOL metadata / service offer0x8b61289c6014c205d854d594f9f2c364cf1689bb9f45d2da42d7daceb4bffd2aAliceERC-8004 registrymetadata publishedPASS
Live OBOL settlement transfer0x7378375d8dda7406c04752594371e9bfb2248dbb4e5a22074fd1799ad3e479e6Bob signerAlice1000000000000000 wei OBOLPASS
Fork OBOL buyer funding0x06c9a4af53a448a2adda95da2b7d7fe743f7c653b5f2e9b046d9109a5d7a304afork minterBob signer10 OBOLPASS
Fork OBOL settlement transfer0x26253a6f994c7be25a3bdbd3609561a6ca354e182a0ec2eed601546fcce0b960Bob signerAlice1000000000000000 wei OBOLPASS
Fork USDC settlement transfer0x71c148bc426d7e0ada0de60752855097551ca4353cc09db02fbe312846edb08abuyerseller1000 micro-USDCPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux aarch64
Backendk3d v5.8.3, k3s v1.31.5-k3s1
Tool versionsDocker 29.2.1, kubectl client v1.35.3, Helm v3.20.1, helmfile 1.4.3, Go 1.25.3, Foundry 1.5.1-stable
QA agent/modelOpenAI-compatible QA endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
cloudflaredcloudflare/cloudflared2026.3.0embedded cloudflared values
x402 facilitatorghcr.io/x402-rs/x402-facilitator1.4.7flow-10 / flow-13 fork tests
x402 verifierghcr.io/obolnetwork/x402-verifierlatestembedded base manifest
x402 buyerghcr.io/obolnetwork/x402-buyerlatestembedded LiteLLM sidecar manifest
LiteLLMghcr.io/obolnetwork/litellmsha-c16b156embedded base manifest
OpenClawghcr.io/obolnetwork/openclawv2026.4.21pinned OPENCLAW_VERSION
OpenClaw chartobol/openclaw0.4.0CLI-generated agent helmfile
remote-signer chartobol/remote-signer0.3.2CLI-generated agent helmfile
eRPCghcr.io/erpc/erpc0.0.62embedded eRPC values
Kubernetes / stack:
ItemValue
Stack IDsunique per flow worktree; removed by cleanup
Namespacesx402, llm, erpc, traefik, hermes-obol-agent exercised
Pod readinessx402 verifier/buyer, LiteLLM, Hermes API, remote-signer, eRPC, and cloudflared rolled out in flows
Cleanup resultno QA tmux sessions, no k3d clusters, no flow containers, no Anvil fork processes remaining
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routepaid/qwen36-fast through x402-buyer sidecar
Paid endpoint statusHTTP 200 in USDC live, OBOL live, and OBOL fork paid inference
Auth token sourceobol agent auth --runtime hermes obol-agent; raw bearer tokens omitted
Artifacts and logs:
ArtifactLocation / linkNotes
Release reportretained on QA machine as release-smoke-20260505-035957-artifacts/RELEASE_REPORT.mdsanitized summary copied above
Receipt bundlesretained on QA machine under flow-11-receipts, flow-14-receipts, flow-13-receiptsincludes registration, metadata, mint/funding, settlement receipts
Demo readiness:
ItemStatusNotes
Seller visible / registeredPASSERC-8004 registration reflected for live USDC and live OBOL flows
Buyer discovery worksPASSHermes agent discovery issued and purchase confirmed by PurchaseRequest Ready=True
Paid route worksPASSpaid/qwen36-fast returned HTTP 200 with coherent response
Settlement visible on-chainPASSexact balance deltas and settlement receipts verified
Review Notes
Known gaps:
  • Standalone local integration BDD command was not used as the release gate because the local dev stack setup timed out before assertions. The remote release-smoke suite covered the end-to-end behavior and passed fully.
  • x402-buyer and x402-verifier still use latest; image pinning/digesting remains a follow-up unless intentionally handled in this PR.
Follow-ups:
  • Replace latest x402 sidecar images with stable tags or digests when CI publishes a suitable artifact.
  • Decide whether the verifier restart after x402 pricing changes should be replaced by a controller/runtime reload path rather than remaining a flow workaround.
Reviewer focus:
  • Confirm the flow split is clear: live Base Sepolia OBOL is the release gate, Anvil fork OBOL is an explicit regression path.
  • Review the flow-10 RPC resolver and flow-13 cluster-side Anvil probe for portability across QA machines.
  • Review stale OBOL_INGRESS_URL handling so .env defaults do not override dynamic high-port ingress during parallel QA.

This was referenced May 4, 2026
 Closed

 Closed

OisinKyne
OisinKyne
approved these changes
May 5, 2026
View reviewed changes
@OisinKyneOisinKyne
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

biggest thing i worry about is the sponsorship removal and if we documented that everywhere. the mainnet sponsored facilitator tried to set a 7702 delegation to an unknown wallet.
We can do 7702 stuff in its own pr in future to handle broad gas sponsorship.
@OisinKyne
OisinKyne
merged commit a6cd2c2
into
mainMay 5, 2026

6 checks passed

@OisinKyne
OisinKyne
deleted the

integration/rc8-superset

branch
May 5, 2026 00:15
@bussyjdbussyjd
mentioned this pull request
Jun 27, 2026
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@OisinKyneOisinKyneOisinKyne approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@bussyjd@OisinKyne

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
rc8: reconcile stack QA fixes by bussyjd · Pull Request #417 · ObolNetwork/obol-stack · GitHub
Skip to content
rc8: reconcile stack QA fixes - #417
Merged
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
May 5, 2026
Merged
rc8: reconcile stack QA fixes#417
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
Conversation
@bussyjd
@bussyjdbussyjd
commented
May 4, 2026

Copy link


Copy Markdown

Contributor
Summary
What changed:
  • Reconciles the rc8 WIP stack fixes into one integration branch: model routing order, Hermes/frontend image/idempotency work, Helm 3 compatibility, RBAC tightening, QA runbook refresh, OBOL buyer-flow structural assertions, and paid-flow hardening.
  • Adds one intentionally WIP diagnostic commit: fix(flows): restart verifier after local pricing.
Why it matters:
  • This branch is meant to supersede the fragmented WIP PRs and give reviewers one place to inspect the rc8 candidate.
  • The verifier restart is not presented as the final production fix. It proves the local paid-flow failure is a verifier config-reload problem: before the rollout, the local facilitator received only /supported; after the rollout, the paid legacy slice hit /verify and /settle and passed.
Risk level: high while draft
Commit under test: 137a72f63f748d33ff5a1b25fda9c8f33dfd7f85
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other: reconciliation of open WIP branches
Validation
CI checks:
CheckStatusLink
GitHub CIpendingdraft PR
Unit tests:
go test ./... -count=1
PASS on local dev machine before push

Integration tests:
bash -n flows/*.sh
PASS
git diff --check
PASS

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-13-dual-stack-obol.shBase Sepolia Anvil forkQA machine Aisolated rc8 worktreePASSreceipt summary retained on QA machine
flow-02 -> flow-06 -> flow-07 -> flow-10 -> flow-08 -> flow-09Base Sepolia Anvil forkQA machine Aisolated rc8 worktreePASS after verifier-rollout diagnosticlegacy paid log retained on QA machine
release-smoke.shmixedQA machine Aisolated rc8 worktreeNOT GREEN YETlatest full run diagnosed flow-08 stale verifier config
Release smoke:
Full release smoke is not being claimed green on this draft.
Earlier full run reached/passed live OBOL flow-14 and fork flow-13 segments, but later full runs exposed:
1. flow-13 false negative on empty agent final text despite PurchaseRequest/settlement succeeding; fixed by structural assertions.
2. flow-10 Anvil fork readiness race; fixed by polling readiness.
3. flow-08 stale x402-verifier facilitator config after local pricing change; WIP diagnostic rollout proves the cause but should be replaced before ready review.

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network:
  • Live gate: Base Sepolia, rerun pending on current HEAD.
  • Fork regression: Anvil fork of Base Sepolia.
RPC/provider:
  • Live: configured via existing QA environment, not copied here.
  • Fork: local Anvil in isolated QA worktree.
Facilitator:
  • Live flow uses the deployed public facilitator.
  • Fork/legacy paid flows use the x402-rs facilitator image; current tag still needs final review before this PR leaves draft.
Contracts and tokens:
NameAddressVersion / notes
OBOL Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4live smoke token
Fork OBOLrun-generatedForkObolToken, EIP-712 name checked in flow-13
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical derived seller wallet
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Ecanonical derived buyer wallet
Facilitator / receiverpending final live rerundo not infer from local fork logs
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
Fork OBOLAlice seller1000000000000000000010001000000000000000+1000000000000000+1000000000000000
Fork OBOLBob buyer100000000000000000009999000000000000000-1000000000000000-1000000000000000
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
flow-13 settlement transfer0x8f6c93fa25ca98b5e41ca3888eb2179288e8f3d5ddf4771a12982d2c62931fb8BobAlice1000000000000000 atomic OBOL unitsPASS
legacy paid settlement transfer0x6ca4b44c24c5d03c56d78fb5487e011cd06d3a65ef72471ccb9b6866980e3dcbbuyerseller1000 USDC atomic units in fork flowPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux QA machine, exact host omitted
Backendk3d stack worktrees
Tool versionslocal toolchain from stack workspace; exact versions to capture in final RC run
QA agent/modelOpenAI-compatible vLLM/llama.cpp endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
x402 local facilitatorghcr.io/x402-rs/x402-facilitatorcurrent flow tag under reviewGHCR
cloudflaredcloudflare/cloudflaredchart tag plus Renovate coverage to confirmDocker Hub
Kubernetes / stack:
ItemValue
Stack IDsisolated per-flow QA worktrees
Namespacesgenerated by each stack flow
Pod readinesschecked by flows before payment assertions
Cleanup resultlegacy paid slice cleanup passed; full release-smoke cleanup/rerun still pending
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routeobol model setup custom --no-sync, obol model prefer, single obol model sync
Paid endpoint statusHTTP 200 paid inference observed in flow-13 and legacy paid slice
Auth token sourceobol agent auth --runtime ... obol-agent
Artifacts and logs:
ArtifactLocation / linkNotes
flow-13 receiptsretained on QA machineincludes receipt summary and settlement evidence
legacy paid logretained on QA machineincludes flow-10/08/09 pass and facilitator /verify + /settle evidence
release-smoke reportretained on QA machinelatest full run is diagnostic, not a green RC receipt
Demo readiness:
ItemStatusNotes
Seller visible / registeredpartiallive OBOL current-HEAD rerun pending
Buyer discovery worksPASS in flow-13agent buy structurally confirmed by PurchaseRequest
Paid route worksPASS in flow-13 and legacy paid sliceHTTP 200 paid inference observed
Settlement visible on-chainPASS in fork flowslive current-HEAD rerun pending
Review Notes
Known gaps:
  • The x402-verifier restart after local pricing change is a diagnostic WIP, not the desired long-term behavior. Final fix should make config reload deterministic through the chart/controller/reloader path.
  • Full release smoke must be rerun from a clean QA worktree and be 100% green before this moves out of draft.
  • x402 facilitator and cloudflared image tags need one final review before ready review.
Follow-ups:
  • Replace the verifier restart with the real reload fix.
  • Rerun the full flow fleet and update this template with final receipts, balances, image versions, and cleanup status.
Reviewer focus:
  • Is the verifier config reload diagnosis correct?
  • Should the final fix live in the x402 chart, controller reconciliation, or verifier config-watcher behavior?
  • Confirm the RBAC tightening keeps the agent functional without over-broad write permissions.
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
PR reconciliation pass
Fetched current remote refs and compared the open PR heads against integration/rc8-superset.
PRStatus in #417Notes
#412fix/model-config-orderIncludedorigin/fix/model-config-order is an ancestor of #417. range-diff shows the two model-order commits preserved exactly.
#415oisin/413-1Included as patch-equivalentThe branch was force-updated after #417 was opened, so it is not an ancestor by SHA, but range-diff maps its current commits to #417 (0654685 -> 9d78608, e41faed -> c3fb745) with no PR-side-only patch left over.
#411Hermes recreation issuesClosed; safe subset retainedThe Hermes recreation lifecycle fixes/tests are already on main (eec1a91, ea341c7) and therefore present in #417. The selected #411 work that #415 was meant to keep is also present in #417.
Intentional supersession:
No missing open-PR work found in this pass.
@bussyjd
bussyjd
marked this pull request as ready for review
May 4, 2026 21:50
@bussyjdbussyjd
changed the title
WIP rc8: reconcile stack QA fixesrc8: reconcile stack QA fixesMay 4, 2026
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
Summary
What changed: rc8 superset branch reconciles the flow fixes needed for deterministic release smoke: stale ingress URL handling, fork RPC resolution for flow 10, and cluster-side Anvil reachability probing for flow 13.
Why it matters: the QA suite now distinguishes live Base Sepolia OBOL from the Anvil fork regression and the full seller/buyer payment lifecycle passes end-to-end through Hermes, LiteLLM, x402, remote-signer, and on-chain settlement.
Risk level: medium
Commit under test: 8bf94c279ba9e870c5662963469ff2a9f559fce9
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other:
Validation
CI checks:
CheckStatusLink
GitHub CIPending after ready-for-review pushPR checks
Unit tests:
bash -n flows/*.sh
PASS
git diff --check
PASS
go test ./cmd/obol ./internal/stack ./internal/hermes ./internal/model ./internal/embed ./internal/x402 -count=1
PASS

Integration tests:
go test -tags integration -v -run TestIntegration_Tunnel_SellDiscoverBuySidecar_QuotaAndBalance -timeout 30m ./internal/openclaw/
PASS/SKIP locally: skipped because no local kubeconfig was active.
go test -tags integration -v -run TestBDDIntegration -timeout 10m ./internal/x402/
Not used as the final gate locally: local setup timed out while preloading/building the dev stack before assertions. Covered by the full remote release-smoke run below.

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-01-prerequisiteslocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-02-stack-init-uplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-03-inferencelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-04-agentlocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-05-networklocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-06-sell-setuplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-07-sell-verifylocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-10-anvil-facilitatorBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-08-buyBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSsettlement receipt archived
flow-09-lifecyclelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-11-dual-stackBase Sepolia live USDCQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-14-live-obol-base-sepoliaBase Sepolia live OBOLQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-13-dual-stack-obolBase Sepolia fork OBOLQA machine Aqa-fullsmoke-20260505-035957PASSmint, funding, settlement receipts archived
Release smoke:
RELEASE_SMOKE_INCLUDE_OBOL=true RELEASE_SMOKE_INCLUDE_OBOL_FORK=true bash flows/release-smoke.sh
Result: PASS
FAIL lines: 0
SKIP lines: 0
Exit code: 0

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network: Base Sepolia live and Base Sepolia fork regression
RPC/provider: public Base Sepolia RPC selected by flow resolver
Facilitator: public Obol facilitator for live flows; local x402-rs facilitator container for fork regression
Contracts and tokens:
NameAddressVersion / notes
OBOL token, live Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4name=Obol Network, symbol=OBOL, decimals=18
OBOL token, fork regression0xeA083Bd4fAfF8aec92123DC7c246aB0E7886A429deployed by flow-13 on Anvil fork
USDC, Base Sepoliacanonical Base Sepolia USDCdecimals=6
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical funded QA seller key
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Edeterministic derived buyer key, preseeded into remote-signer
Bob remote-signer0x57b0eF875DeB5A37301F1640E469a2129Da9490Everified by flows before purchase
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
USDC live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E89330008934000+1000 micro-USDC+1000 micro-USDC
USDC live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E49850004984000-1000 micro-USDC-1000 micro-USDC
OBOL live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1200000000000000013000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E999998994000000000000000999998993000000000000000-1000000000000000 wei-1000000000000000 wei
OBOL fork, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1000000000000000000010001000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL fork, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E100000000000000000009999000000000000000-1000000000000000 wei-1000000000000000 wei
USDC fork buy, buyerflow-08 buyer1000000000999999000-1000 micro-USDC-1000 micro-USDC
USDC fork buy, sellerflow-08 seller465417851465418851+1000 micro-USDC+1000 micro-USDC
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
USDC ERC-8004 registration0x82be088a8f9d4b71e900ef907f4665e449746bac85d9039b9538fa3a6231d1f3AliceERC-8004 registryAgent ID 5486PASS
USDC metadata / service offer0x6133ee020141363e0c014df3fec3b0d7e2da03a6f1102bb226523089d4d6e919AliceERC-8004 registrymetadata publishedPASS
USDC settlement transfer0x9ec3a1993499d1fb4158fcd20afa310279e700a651eed194c79383068978673eBob signerAlice1000 micro-USDCPASS
Live OBOL ERC-8004 registration0x345102a13cd6637b4459abdd12d32e51628945a95396105da1592cf05bd03145AliceERC-8004 registryAgent ID 5487PASS
Live OBOL metadata / service offer0x8b61289c6014c205d854d594f9f2c364cf1689bb9f45d2da42d7daceb4bffd2aAliceERC-8004 registrymetadata publishedPASS
Live OBOL settlement transfer0x7378375d8dda7406c04752594371e9bfb2248dbb4e5a22074fd1799ad3e479e6Bob signerAlice1000000000000000 wei OBOLPASS
Fork OBOL buyer funding0x06c9a4af53a448a2adda95da2b7d7fe743f7c653b5f2e9b046d9109a5d7a304afork minterBob signer10 OBOLPASS
Fork OBOL settlement transfer0x26253a6f994c7be25a3bdbd3609561a6ca354e182a0ec2eed601546fcce0b960Bob signerAlice1000000000000000 wei OBOLPASS
Fork USDC settlement transfer0x71c148bc426d7e0ada0de60752855097551ca4353cc09db02fbe312846edb08abuyerseller1000 micro-USDCPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux aarch64
Backendk3d v5.8.3, k3s v1.31.5-k3s1
Tool versionsDocker 29.2.1, kubectl client v1.35.3, Helm v3.20.1, helmfile 1.4.3, Go 1.25.3, Foundry 1.5.1-stable
QA agent/modelOpenAI-compatible QA endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
cloudflaredcloudflare/cloudflared2026.3.0embedded cloudflared values
x402 facilitatorghcr.io/x402-rs/x402-facilitator1.4.7flow-10 / flow-13 fork tests
x402 verifierghcr.io/obolnetwork/x402-verifierlatestembedded base manifest
x402 buyerghcr.io/obolnetwork/x402-buyerlatestembedded LiteLLM sidecar manifest
LiteLLMghcr.io/obolnetwork/litellmsha-c16b156embedded base manifest
OpenClawghcr.io/obolnetwork/openclawv2026.4.21pinned OPENCLAW_VERSION
OpenClaw chartobol/openclaw0.4.0CLI-generated agent helmfile
remote-signer chartobol/remote-signer0.3.2CLI-generated agent helmfile
eRPCghcr.io/erpc/erpc0.0.62embedded eRPC values
Kubernetes / stack:
ItemValue
Stack IDsunique per flow worktree; removed by cleanup
Namespacesx402, llm, erpc, traefik, hermes-obol-agent exercised
Pod readinessx402 verifier/buyer, LiteLLM, Hermes API, remote-signer, eRPC, and cloudflared rolled out in flows
Cleanup resultno QA tmux sessions, no k3d clusters, no flow containers, no Anvil fork processes remaining
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routepaid/qwen36-fast through x402-buyer sidecar
Paid endpoint statusHTTP 200 in USDC live, OBOL live, and OBOL fork paid inference
Auth token sourceobol agent auth --runtime hermes obol-agent; raw bearer tokens omitted
Artifacts and logs:
ArtifactLocation / linkNotes
Release reportretained on QA machine as release-smoke-20260505-035957-artifacts/RELEASE_REPORT.mdsanitized summary copied above
Receipt bundlesretained on QA machine under flow-11-receipts, flow-14-receipts, flow-13-receiptsincludes registration, metadata, mint/funding, settlement receipts
Demo readiness:
ItemStatusNotes
Seller visible / registeredPASSERC-8004 registration reflected for live USDC and live OBOL flows
Buyer discovery worksPASSHermes agent discovery issued and purchase confirmed by PurchaseRequest Ready=True
Paid route worksPASSpaid/qwen36-fast returned HTTP 200 with coherent response
Settlement visible on-chainPASSexact balance deltas and settlement receipts verified
Review Notes
Known gaps:
  • Standalone local integration BDD command was not used as the release gate because the local dev stack setup timed out before assertions. The remote release-smoke suite covered the end-to-end behavior and passed fully.
  • x402-buyer and x402-verifier still use latest; image pinning/digesting remains a follow-up unless intentionally handled in this PR.
Follow-ups:
  • Replace latest x402 sidecar images with stable tags or digests when CI publishes a suitable artifact.
  • Decide whether the verifier restart after x402 pricing changes should be replaced by a controller/runtime reload path rather than remaining a flow workaround.
Reviewer focus:
  • Confirm the flow split is clear: live Base Sepolia OBOL is the release gate, Anvil fork OBOL is an explicit regression path.
  • Review the flow-10 RPC resolver and flow-13 cluster-side Anvil probe for portability across QA machines.
  • Review stale OBOL_INGRESS_URL handling so .env defaults do not override dynamic high-port ingress during parallel QA.

This was referenced May 4, 2026
 Closed

 Closed

OisinKyne
OisinKyne
approved these changes
May 5, 2026
View reviewed changes
@OisinKyneOisinKyne
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

biggest thing i worry about is the sponsorship removal and if we documented that everywhere. the mainnet sponsored facilitator tried to set a 7702 delegation to an unknown wallet.
We can do 7702 stuff in its own pr in future to handle broad gas sponsorship.
@OisinKyne
OisinKyne
merged commit a6cd2c2
into
mainMay 5, 2026

6 checks passed

@OisinKyne
OisinKyne
deleted the

integration/rc8-superset

branch
May 5, 2026 00:15
@bussyjdbussyjd
mentioned this pull request
Jun 27, 2026
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@OisinKyneOisinKyneOisinKyne approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@bussyjd@OisinKyne

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
rc8: reconcile stack QA fixes by bussyjd · Pull Request #417 · ObolNetwork/obol-stack · GitHub
Skip to content
rc8: reconcile stack QA fixes - #417
Merged
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
May 5, 2026
Merged
rc8: reconcile stack QA fixes#417
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
Conversation
@bussyjd
@bussyjdbussyjd
commented
May 4, 2026

Copy link


Copy Markdown

Contributor
Summary
What changed:
  • Reconciles the rc8 WIP stack fixes into one integration branch: model routing order, Hermes/frontend image/idempotency work, Helm 3 compatibility, RBAC tightening, QA runbook refresh, OBOL buyer-flow structural assertions, and paid-flow hardening.
  • Adds one intentionally WIP diagnostic commit: fix(flows): restart verifier after local pricing.
Why it matters:
  • This branch is meant to supersede the fragmented WIP PRs and give reviewers one place to inspect the rc8 candidate.
  • The verifier restart is not presented as the final production fix. It proves the local paid-flow failure is a verifier config-reload problem: before the rollout, the local facilitator received only /supported; after the rollout, the paid legacy slice hit /verify and /settle and passed.
Risk level: high while draft
Commit under test: 137a72f63f748d33ff5a1b25fda9c8f33dfd7f85
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other: reconciliation of open WIP branches
Validation
CI checks:
CheckStatusLink
GitHub CIpendingdraft PR
Unit tests:
go test ./... -count=1
PASS on local dev machine before push

Integration tests:
bash -n flows/*.sh
PASS
git diff --check
PASS

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-13-dual-stack-obol.shBase Sepolia Anvil forkQA machine Aisolated rc8 worktreePASSreceipt summary retained on QA machine
flow-02 -> flow-06 -> flow-07 -> flow-10 -> flow-08 -> flow-09Base Sepolia Anvil forkQA machine Aisolated rc8 worktreePASS after verifier-rollout diagnosticlegacy paid log retained on QA machine
release-smoke.shmixedQA machine Aisolated rc8 worktreeNOT GREEN YETlatest full run diagnosed flow-08 stale verifier config
Release smoke:
Full release smoke is not being claimed green on this draft.
Earlier full run reached/passed live OBOL flow-14 and fork flow-13 segments, but later full runs exposed:
1. flow-13 false negative on empty agent final text despite PurchaseRequest/settlement succeeding; fixed by structural assertions.
2. flow-10 Anvil fork readiness race; fixed by polling readiness.
3. flow-08 stale x402-verifier facilitator config after local pricing change; WIP diagnostic rollout proves the cause but should be replaced before ready review.

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network:
  • Live gate: Base Sepolia, rerun pending on current HEAD.
  • Fork regression: Anvil fork of Base Sepolia.
RPC/provider:
  • Live: configured via existing QA environment, not copied here.
  • Fork: local Anvil in isolated QA worktree.
Facilitator:
  • Live flow uses the deployed public facilitator.
  • Fork/legacy paid flows use the x402-rs facilitator image; current tag still needs final review before this PR leaves draft.
Contracts and tokens:
NameAddressVersion / notes
OBOL Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4live smoke token
Fork OBOLrun-generatedForkObolToken, EIP-712 name checked in flow-13
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical derived seller wallet
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Ecanonical derived buyer wallet
Facilitator / receiverpending final live rerundo not infer from local fork logs
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
Fork OBOLAlice seller1000000000000000000010001000000000000000+1000000000000000+1000000000000000
Fork OBOLBob buyer100000000000000000009999000000000000000-1000000000000000-1000000000000000
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
flow-13 settlement transfer0x8f6c93fa25ca98b5e41ca3888eb2179288e8f3d5ddf4771a12982d2c62931fb8BobAlice1000000000000000 atomic OBOL unitsPASS
legacy paid settlement transfer0x6ca4b44c24c5d03c56d78fb5487e011cd06d3a65ef72471ccb9b6866980e3dcbbuyerseller1000 USDC atomic units in fork flowPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux QA machine, exact host omitted
Backendk3d stack worktrees
Tool versionslocal toolchain from stack workspace; exact versions to capture in final RC run
QA agent/modelOpenAI-compatible vLLM/llama.cpp endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
x402 local facilitatorghcr.io/x402-rs/x402-facilitatorcurrent flow tag under reviewGHCR
cloudflaredcloudflare/cloudflaredchart tag plus Renovate coverage to confirmDocker Hub
Kubernetes / stack:
ItemValue
Stack IDsisolated per-flow QA worktrees
Namespacesgenerated by each stack flow
Pod readinesschecked by flows before payment assertions
Cleanup resultlegacy paid slice cleanup passed; full release-smoke cleanup/rerun still pending
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routeobol model setup custom --no-sync, obol model prefer, single obol model sync
Paid endpoint statusHTTP 200 paid inference observed in flow-13 and legacy paid slice
Auth token sourceobol agent auth --runtime ... obol-agent
Artifacts and logs:
ArtifactLocation / linkNotes
flow-13 receiptsretained on QA machineincludes receipt summary and settlement evidence
legacy paid logretained on QA machineincludes flow-10/08/09 pass and facilitator /verify + /settle evidence
release-smoke reportretained on QA machinelatest full run is diagnostic, not a green RC receipt
Demo readiness:
ItemStatusNotes
Seller visible / registeredpartiallive OBOL current-HEAD rerun pending
Buyer discovery worksPASS in flow-13agent buy structurally confirmed by PurchaseRequest
Paid route worksPASS in flow-13 and legacy paid sliceHTTP 200 paid inference observed
Settlement visible on-chainPASS in fork flowslive current-HEAD rerun pending
Review Notes
Known gaps:
  • The x402-verifier restart after local pricing change is a diagnostic WIP, not the desired long-term behavior. Final fix should make config reload deterministic through the chart/controller/reloader path.
  • Full release smoke must be rerun from a clean QA worktree and be 100% green before this moves out of draft.
  • x402 facilitator and cloudflared image tags need one final review before ready review.
Follow-ups:
  • Replace the verifier restart with the real reload fix.
  • Rerun the full flow fleet and update this template with final receipts, balances, image versions, and cleanup status.
Reviewer focus:
  • Is the verifier config reload diagnosis correct?
  • Should the final fix live in the x402 chart, controller reconciliation, or verifier config-watcher behavior?
  • Confirm the RBAC tightening keeps the agent functional without over-broad write permissions.
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
PR reconciliation pass
Fetched current remote refs and compared the open PR heads against integration/rc8-superset.
PRStatus in #417Notes
#412fix/model-config-orderIncludedorigin/fix/model-config-order is an ancestor of #417. range-diff shows the two model-order commits preserved exactly.
#415oisin/413-1Included as patch-equivalentThe branch was force-updated after #417 was opened, so it is not an ancestor by SHA, but range-diff maps its current commits to #417 (0654685 -> 9d78608, e41faed -> c3fb745) with no PR-side-only patch left over.
#411Hermes recreation issuesClosed; safe subset retainedThe Hermes recreation lifecycle fixes/tests are already on main (eec1a91, ea341c7) and therefore present in #417. The selected #411 work that #415 was meant to keep is also present in #417.
Intentional supersession:
No missing open-PR work found in this pass.
@bussyjd
bussyjd
marked this pull request as ready for review
May 4, 2026 21:50
@bussyjdbussyjd
changed the title
WIP rc8: reconcile stack QA fixesrc8: reconcile stack QA fixesMay 4, 2026
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
Summary
What changed: rc8 superset branch reconciles the flow fixes needed for deterministic release smoke: stale ingress URL handling, fork RPC resolution for flow 10, and cluster-side Anvil reachability probing for flow 13.
Why it matters: the QA suite now distinguishes live Base Sepolia OBOL from the Anvil fork regression and the full seller/buyer payment lifecycle passes end-to-end through Hermes, LiteLLM, x402, remote-signer, and on-chain settlement.
Risk level: medium
Commit under test: 8bf94c279ba9e870c5662963469ff2a9f559fce9
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other:
Validation
CI checks:
CheckStatusLink
GitHub CIPending after ready-for-review pushPR checks
Unit tests:
bash -n flows/*.sh
PASS
git diff --check
PASS
go test ./cmd/obol ./internal/stack ./internal/hermes ./internal/model ./internal/embed ./internal/x402 -count=1
PASS

Integration tests:
go test -tags integration -v -run TestIntegration_Tunnel_SellDiscoverBuySidecar_QuotaAndBalance -timeout 30m ./internal/openclaw/
PASS/SKIP locally: skipped because no local kubeconfig was active.
go test -tags integration -v -run TestBDDIntegration -timeout 10m ./internal/x402/
Not used as the final gate locally: local setup timed out while preloading/building the dev stack before assertions. Covered by the full remote release-smoke run below.

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-01-prerequisiteslocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-02-stack-init-uplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-03-inferencelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-04-agentlocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-05-networklocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-06-sell-setuplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-07-sell-verifylocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-10-anvil-facilitatorBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-08-buyBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSsettlement receipt archived
flow-09-lifecyclelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-11-dual-stackBase Sepolia live USDCQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-14-live-obol-base-sepoliaBase Sepolia live OBOLQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-13-dual-stack-obolBase Sepolia fork OBOLQA machine Aqa-fullsmoke-20260505-035957PASSmint, funding, settlement receipts archived
Release smoke:
RELEASE_SMOKE_INCLUDE_OBOL=true RELEASE_SMOKE_INCLUDE_OBOL_FORK=true bash flows/release-smoke.sh
Result: PASS
FAIL lines: 0
SKIP lines: 0
Exit code: 0

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network: Base Sepolia live and Base Sepolia fork regression
RPC/provider: public Base Sepolia RPC selected by flow resolver
Facilitator: public Obol facilitator for live flows; local x402-rs facilitator container for fork regression
Contracts and tokens:
NameAddressVersion / notes
OBOL token, live Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4name=Obol Network, symbol=OBOL, decimals=18
OBOL token, fork regression0xeA083Bd4fAfF8aec92123DC7c246aB0E7886A429deployed by flow-13 on Anvil fork
USDC, Base Sepoliacanonical Base Sepolia USDCdecimals=6
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical funded QA seller key
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Edeterministic derived buyer key, preseeded into remote-signer
Bob remote-signer0x57b0eF875DeB5A37301F1640E469a2129Da9490Everified by flows before purchase
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
USDC live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E89330008934000+1000 micro-USDC+1000 micro-USDC
USDC live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E49850004984000-1000 micro-USDC-1000 micro-USDC
OBOL live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1200000000000000013000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E999998994000000000000000999998993000000000000000-1000000000000000 wei-1000000000000000 wei
OBOL fork, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1000000000000000000010001000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL fork, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E100000000000000000009999000000000000000-1000000000000000 wei-1000000000000000 wei
USDC fork buy, buyerflow-08 buyer1000000000999999000-1000 micro-USDC-1000 micro-USDC
USDC fork buy, sellerflow-08 seller465417851465418851+1000 micro-USDC+1000 micro-USDC
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
USDC ERC-8004 registration0x82be088a8f9d4b71e900ef907f4665e449746bac85d9039b9538fa3a6231d1f3AliceERC-8004 registryAgent ID 5486PASS
USDC metadata / service offer0x6133ee020141363e0c014df3fec3b0d7e2da03a6f1102bb226523089d4d6e919AliceERC-8004 registrymetadata publishedPASS
USDC settlement transfer0x9ec3a1993499d1fb4158fcd20afa310279e700a651eed194c79383068978673eBob signerAlice1000 micro-USDCPASS
Live OBOL ERC-8004 registration0x345102a13cd6637b4459abdd12d32e51628945a95396105da1592cf05bd03145AliceERC-8004 registryAgent ID 5487PASS
Live OBOL metadata / service offer0x8b61289c6014c205d854d594f9f2c364cf1689bb9f45d2da42d7daceb4bffd2aAliceERC-8004 registrymetadata publishedPASS
Live OBOL settlement transfer0x7378375d8dda7406c04752594371e9bfb2248dbb4e5a22074fd1799ad3e479e6Bob signerAlice1000000000000000 wei OBOLPASS
Fork OBOL buyer funding0x06c9a4af53a448a2adda95da2b7d7fe743f7c653b5f2e9b046d9109a5d7a304afork minterBob signer10 OBOLPASS
Fork OBOL settlement transfer0x26253a6f994c7be25a3bdbd3609561a6ca354e182a0ec2eed601546fcce0b960Bob signerAlice1000000000000000 wei OBOLPASS
Fork USDC settlement transfer0x71c148bc426d7e0ada0de60752855097551ca4353cc09db02fbe312846edb08abuyerseller1000 micro-USDCPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux aarch64
Backendk3d v5.8.3, k3s v1.31.5-k3s1
Tool versionsDocker 29.2.1, kubectl client v1.35.3, Helm v3.20.1, helmfile 1.4.3, Go 1.25.3, Foundry 1.5.1-stable
QA agent/modelOpenAI-compatible QA endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
cloudflaredcloudflare/cloudflared2026.3.0embedded cloudflared values
x402 facilitatorghcr.io/x402-rs/x402-facilitator1.4.7flow-10 / flow-13 fork tests
x402 verifierghcr.io/obolnetwork/x402-verifierlatestembedded base manifest
x402 buyerghcr.io/obolnetwork/x402-buyerlatestembedded LiteLLM sidecar manifest
LiteLLMghcr.io/obolnetwork/litellmsha-c16b156embedded base manifest
OpenClawghcr.io/obolnetwork/openclawv2026.4.21pinned OPENCLAW_VERSION
OpenClaw chartobol/openclaw0.4.0CLI-generated agent helmfile
remote-signer chartobol/remote-signer0.3.2CLI-generated agent helmfile
eRPCghcr.io/erpc/erpc0.0.62embedded eRPC values
Kubernetes / stack:
ItemValue
Stack IDsunique per flow worktree; removed by cleanup
Namespacesx402, llm, erpc, traefik, hermes-obol-agent exercised
Pod readinessx402 verifier/buyer, LiteLLM, Hermes API, remote-signer, eRPC, and cloudflared rolled out in flows
Cleanup resultno QA tmux sessions, no k3d clusters, no flow containers, no Anvil fork processes remaining
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routepaid/qwen36-fast through x402-buyer sidecar
Paid endpoint statusHTTP 200 in USDC live, OBOL live, and OBOL fork paid inference
Auth token sourceobol agent auth --runtime hermes obol-agent; raw bearer tokens omitted
Artifacts and logs:
ArtifactLocation / linkNotes
Release reportretained on QA machine as release-smoke-20260505-035957-artifacts/RELEASE_REPORT.mdsanitized summary copied above
Receipt bundlesretained on QA machine under flow-11-receipts, flow-14-receipts, flow-13-receiptsincludes registration, metadata, mint/funding, settlement receipts
Demo readiness:
ItemStatusNotes
Seller visible / registeredPASSERC-8004 registration reflected for live USDC and live OBOL flows
Buyer discovery worksPASSHermes agent discovery issued and purchase confirmed by PurchaseRequest Ready=True
Paid route worksPASSpaid/qwen36-fast returned HTTP 200 with coherent response
Settlement visible on-chainPASSexact balance deltas and settlement receipts verified
Review Notes
Known gaps:
  • Standalone local integration BDD command was not used as the release gate because the local dev stack setup timed out before assertions. The remote release-smoke suite covered the end-to-end behavior and passed fully.
  • x402-buyer and x402-verifier still use latest; image pinning/digesting remains a follow-up unless intentionally handled in this PR.
Follow-ups:
  • Replace latest x402 sidecar images with stable tags or digests when CI publishes a suitable artifact.
  • Decide whether the verifier restart after x402 pricing changes should be replaced by a controller/runtime reload path rather than remaining a flow workaround.
Reviewer focus:
  • Confirm the flow split is clear: live Base Sepolia OBOL is the release gate, Anvil fork OBOL is an explicit regression path.
  • Review the flow-10 RPC resolver and flow-13 cluster-side Anvil probe for portability across QA machines.
  • Review stale OBOL_INGRESS_URL handling so .env defaults do not override dynamic high-port ingress during parallel QA.

This was referenced May 4, 2026
 Closed

 Closed

OisinKyne
OisinKyne
approved these changes
May 5, 2026
View reviewed changes
@OisinKyneOisinKyne
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

biggest thing i worry about is the sponsorship removal and if we documented that everywhere. the mainnet sponsored facilitator tried to set a 7702 delegation to an unknown wallet.
We can do 7702 stuff in its own pr in future to handle broad gas sponsorship.
@OisinKyne
OisinKyne
merged commit a6cd2c2
into
mainMay 5, 2026

6 checks passed

@OisinKyne
OisinKyne
deleted the

integration/rc8-superset

branch
May 5, 2026 00:15
@bussyjdbussyjd
mentioned this pull request
Jun 27, 2026
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@OisinKyneOisinKyneOisinKyne approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@bussyjd@OisinKyne

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
rc8: reconcile stack QA fixes by bussyjd · Pull Request #417 · ObolNetwork/obol-stack · GitHub
Skip to content
rc8: reconcile stack QA fixes - #417
Merged
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
May 5, 2026
Merged
rc8: reconcile stack QA fixes#417
OisinKyne merged 21 commits into
mainfrom 
integration/rc8-superset
Conversation
@bussyjd
@bussyjdbussyjd
commented
May 4, 2026

Copy link


Copy Markdown

Contributor
Summary
What changed:
  • Reconciles the rc8 WIP stack fixes into one integration branch: model routing order, Hermes/frontend image/idempotency work, Helm 3 compatibility, RBAC tightening, QA runbook refresh, OBOL buyer-flow structural assertions, and paid-flow hardening.
  • Adds one intentionally WIP diagnostic commit: fix(flows): restart verifier after local pricing.
Why it matters:
  • This branch is meant to supersede the fragmented WIP PRs and give reviewers one place to inspect the rc8 candidate.
  • The verifier restart is not presented as the final production fix. It proves the local paid-flow failure is a verifier config-reload problem: before the rollout, the local facilitator received only /supported; after the rollout, the paid legacy slice hit /verify and /settle and passed.
Risk level: high while draft
Commit under test: 137a72f63f748d33ff5a1b25fda9c8f33dfd7f85
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other: reconciliation of open WIP branches
Validation
CI checks:
CheckStatusLink
GitHub CIpendingdraft PR
Unit tests:
go test ./... -count=1
PASS on local dev machine before push

Integration tests:
bash -n flows/*.sh
PASS
git diff --check
PASS

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-13-dual-stack-obol.shBase Sepolia Anvil forkQA machine Aisolated rc8 worktreePASSreceipt summary retained on QA machine
flow-02 -> flow-06 -> flow-07 -> flow-10 -> flow-08 -> flow-09Base Sepolia Anvil forkQA machine Aisolated rc8 worktreePASS after verifier-rollout diagnosticlegacy paid log retained on QA machine
release-smoke.shmixedQA machine Aisolated rc8 worktreeNOT GREEN YETlatest full run diagnosed flow-08 stale verifier config
Release smoke:
Full release smoke is not being claimed green on this draft.
Earlier full run reached/passed live OBOL flow-14 and fork flow-13 segments, but later full runs exposed:
1. flow-13 false negative on empty agent final text despite PurchaseRequest/settlement succeeding; fixed by structural assertions.
2. flow-10 Anvil fork readiness race; fixed by polling readiness.
3. flow-08 stale x402-verifier facilitator config after local pricing change; WIP diagnostic rollout proves the cause but should be replaced before ready review.

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network:
  • Live gate: Base Sepolia, rerun pending on current HEAD.
  • Fork regression: Anvil fork of Base Sepolia.
RPC/provider:
  • Live: configured via existing QA environment, not copied here.
  • Fork: local Anvil in isolated QA worktree.
Facilitator:
  • Live flow uses the deployed public facilitator.
  • Fork/legacy paid flows use the x402-rs facilitator image; current tag still needs final review before this PR leaves draft.
Contracts and tokens:
NameAddressVersion / notes
OBOL Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4live smoke token
Fork OBOLrun-generatedForkObolToken, EIP-712 name checked in flow-13
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical derived seller wallet
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Ecanonical derived buyer wallet
Facilitator / receiverpending final live rerundo not infer from local fork logs
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
Fork OBOLAlice seller1000000000000000000010001000000000000000+1000000000000000+1000000000000000
Fork OBOLBob buyer100000000000000000009999000000000000000-1000000000000000-1000000000000000
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
flow-13 settlement transfer0x8f6c93fa25ca98b5e41ca3888eb2179288e8f3d5ddf4771a12982d2c62931fb8BobAlice1000000000000000 atomic OBOL unitsPASS
legacy paid settlement transfer0x6ca4b44c24c5d03c56d78fb5487e011cd06d3a65ef72471ccb9b6866980e3dcbbuyerseller1000 USDC atomic units in fork flowPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux QA machine, exact host omitted
Backendk3d stack worktrees
Tool versionslocal toolchain from stack workspace; exact versions to capture in final RC run
QA agent/modelOpenAI-compatible vLLM/llama.cpp endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
x402 local facilitatorghcr.io/x402-rs/x402-facilitatorcurrent flow tag under reviewGHCR
cloudflaredcloudflare/cloudflaredchart tag plus Renovate coverage to confirmDocker Hub
Kubernetes / stack:
ItemValue
Stack IDsisolated per-flow QA worktrees
Namespacesgenerated by each stack flow
Pod readinesschecked by flows before payment assertions
Cleanup resultlegacy paid slice cleanup passed; full release-smoke cleanup/rerun still pending
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routeobol model setup custom --no-sync, obol model prefer, single obol model sync
Paid endpoint statusHTTP 200 paid inference observed in flow-13 and legacy paid slice
Auth token sourceobol agent auth --runtime ... obol-agent
Artifacts and logs:
ArtifactLocation / linkNotes
flow-13 receiptsretained on QA machineincludes receipt summary and settlement evidence
legacy paid logretained on QA machineincludes flow-10/08/09 pass and facilitator /verify + /settle evidence
release-smoke reportretained on QA machinelatest full run is diagnostic, not a green RC receipt
Demo readiness:
ItemStatusNotes
Seller visible / registeredpartiallive OBOL current-HEAD rerun pending
Buyer discovery worksPASS in flow-13agent buy structurally confirmed by PurchaseRequest
Paid route worksPASS in flow-13 and legacy paid sliceHTTP 200 paid inference observed
Settlement visible on-chainPASS in fork flowslive current-HEAD rerun pending
Review Notes
Known gaps:
  • The x402-verifier restart after local pricing change is a diagnostic WIP, not the desired long-term behavior. Final fix should make config reload deterministic through the chart/controller/reloader path.
  • Full release smoke must be rerun from a clean QA worktree and be 100% green before this moves out of draft.
  • x402 facilitator and cloudflared image tags need one final review before ready review.
Follow-ups:
  • Replace the verifier restart with the real reload fix.
  • Rerun the full flow fleet and update this template with final receipts, balances, image versions, and cleanup status.
Reviewer focus:
  • Is the verifier config reload diagnosis correct?
  • Should the final fix live in the x402 chart, controller reconciliation, or verifier config-watcher behavior?
  • Confirm the RBAC tightening keeps the agent functional without over-broad write permissions.
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
PR reconciliation pass
Fetched current remote refs and compared the open PR heads against integration/rc8-superset.
PRStatus in #417Notes
#412fix/model-config-orderIncludedorigin/fix/model-config-order is an ancestor of #417. range-diff shows the two model-order commits preserved exactly.
#415oisin/413-1Included as patch-equivalentThe branch was force-updated after #417 was opened, so it is not an ancestor by SHA, but range-diff maps its current commits to #417 (0654685 -> 9d78608, e41faed -> c3fb745) with no PR-side-only patch left over.
#411Hermes recreation issuesClosed; safe subset retainedThe Hermes recreation lifecycle fixes/tests are already on main (eec1a91, ea341c7) and therefore present in #417. The selected #411 work that #415 was meant to keep is also present in #417.
Intentional supersession:
No missing open-PR work found in this pass.
@bussyjd
bussyjd
marked this pull request as ready for review
May 4, 2026 21:50
@bussyjdbussyjd
changed the title
WIP rc8: reconcile stack QA fixesrc8: reconcile stack QA fixesMay 4, 2026
@bussyjd
bussyjd
commented
May 4, 2026

Copy link


Copy Markdown

ContributorAuthor
Summary
What changed: rc8 superset branch reconciles the flow fixes needed for deterministic release smoke: stale ingress URL handling, fork RPC resolution for flow 10, and cluster-side Anvil reachability probing for flow 13.
Why it matters: the QA suite now distinguishes live Base Sepolia OBOL from the Anvil fork regression and the full seller/buyer payment lifecycle passes end-to-end through Hermes, LiteLLM, x402, remote-signer, and on-chain settlement.
Risk level: medium
Commit under test: 8bf94c279ba9e870c5662963469ff2a9f559fce9
Base branch: main
Scope
  •  Code
  •  Charts / manifests
  •  Flows / QA scripts
  •  Docs / skills
  •  Images / dependencies
  •  Other:
Validation
CI checks:
CheckStatusLink
GitHub CIPending after ready-for-review pushPR checks
Unit tests:
bash -n flows/*.sh
PASS
git diff --check
PASS
go test ./cmd/obol ./internal/stack ./internal/hermes ./internal/model ./internal/embed ./internal/x402 -count=1
PASS

Integration tests:
go test -tags integration -v -run TestIntegration_Tunnel_SellDiscoverBuySidecar_QuotaAndBalance -timeout 30m ./internal/openclaw/
PASS/SKIP locally: skipped because no local kubeconfig was active.
go test -tags integration -v -run TestBDDIntegration -timeout 10m ./internal/x402/
Not used as the final gate locally: local setup timed out while preloading/building the dev stack before assertions. Covered by the full remote release-smoke run below.

Flow tests:
FlowNetworkQA machine labelWorktreeResultArtifacts
flow-01-prerequisiteslocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-02-stack-init-uplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-03-inferencelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-04-agentlocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-05-networklocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-06-sell-setuplocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-07-sell-verifylocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-10-anvil-facilitatorBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-08-buyBase Sepolia forkQA machine Aqa-fullsmoke-20260505-035957PASSsettlement receipt archived
flow-09-lifecyclelocal QAQA machine Aqa-fullsmoke-20260505-035957PASSrelease-smoke artifacts retained on QA machine
flow-11-dual-stackBase Sepolia live USDCQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-14-live-obol-base-sepoliaBase Sepolia live OBOLQA machine Aqa-fullsmoke-20260505-035957PASSregistration, metadata, settlement receipts archived
flow-13-dual-stack-obolBase Sepolia fork OBOLQA machine Aqa-fullsmoke-20260505-035957PASSmint, funding, settlement receipts archived
Release smoke:
RELEASE_SMOKE_INCLUDE_OBOL=true RELEASE_SMOKE_INCLUDE_OBOL_FORK=true bash flows/release-smoke.sh
Result: PASS
FAIL lines: 0
SKIP lines: 0
Exit code: 0

Live Chain Evidence
Do not include private keys, seed phrases, passwords, hostnames, personal paths, or raw bearer tokens.
Network: Base Sepolia live and Base Sepolia fork regression
RPC/provider: public Base Sepolia RPC selected by flow resolver
Facilitator: public Obol facilitator for live flows; local x402-rs facilitator container for fork regression
Contracts and tokens:
NameAddressVersion / notes
OBOL token, live Base Sepolia0x54AE82bc871a4E3E8E2FE1173Cb864B8563D44D4name=Obol Network, symbol=OBOL, decimals=18
OBOL token, fork regression0xeA083Bd4fAfF8aec92123DC7c246aB0E7886A429deployed by flow-13 on Anvil fork
USDC, Base Sepoliacanonical Base Sepolia USDCdecimals=6
Wallet roles:
RoleAddressSource
Alice / seller / register0xC0De030F6C37f490594F93fB99e2756703c4297Ecanonical funded QA seller key
Bob / buyer / payer0x57b0eF875DeB5A37301F1640E469a2129Da9490Edeterministic derived buyer key, preseeded into remote-signer
Bob remote-signer0x57b0eF875DeB5A37301F1640E469a2129Da9490Everified by flows before purchase
Balances:
TokenAddressBeforeAfterExpected deltaActual delta
USDC live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E89330008934000+1000 micro-USDC+1000 micro-USDC
USDC live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E49850004984000-1000 micro-USDC-1000 micro-USDC
OBOL live, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1200000000000000013000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL live, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E999998994000000000000000999998993000000000000000-1000000000000000 wei-1000000000000000 wei
OBOL fork, Alice0xC0De030F6C37f490594F93fB99e2756703c4297E1000000000000000000010001000000000000000+1000000000000000 wei+1000000000000000 wei
OBOL fork, Bob signer0x57b0eF875DeB5A37301F1640E469a2129Da9490E100000000000000000009999000000000000000-1000000000000000 wei-1000000000000000 wei
USDC fork buy, buyerflow-08 buyer1000000000999999000-1000 micro-USDC-1000 micro-USDC
USDC fork buy, sellerflow-08 seller465417851465418851+1000 micro-USDC+1000 micro-USDC
Transaction receipts:
PurposeTx hashFromToAmount / eventStatus
USDC ERC-8004 registration0x82be088a8f9d4b71e900ef907f4665e449746bac85d9039b9538fa3a6231d1f3AliceERC-8004 registryAgent ID 5486PASS
USDC metadata / service offer0x6133ee020141363e0c014df3fec3b0d7e2da03a6f1102bb226523089d4d6e919AliceERC-8004 registrymetadata publishedPASS
USDC settlement transfer0x9ec3a1993499d1fb4158fcd20afa310279e700a651eed194c79383068978673eBob signerAlice1000 micro-USDCPASS
Live OBOL ERC-8004 registration0x345102a13cd6637b4459abdd12d32e51628945a95396105da1592cf05bd03145AliceERC-8004 registryAgent ID 5487PASS
Live OBOL metadata / service offer0x8b61289c6014c205d854d594f9f2c364cf1689bb9f45d2da42d7daceb4bffd2aAliceERC-8004 registrymetadata publishedPASS
Live OBOL settlement transfer0x7378375d8dda7406c04752594371e9bfb2248dbb4e5a22074fd1799ad3e479e6Bob signerAlice1000000000000000 wei OBOLPASS
Fork OBOL buyer funding0x06c9a4af53a448a2adda95da2b7d7fe743f7c653b5f2e9b046d9109a5d7a304afork minterBob signer10 OBOLPASS
Fork OBOL settlement transfer0x26253a6f994c7be25a3bdbd3609561a6ca354e182a0ec2eed601546fcce0b960Bob signerAlice1000000000000000 wei OBOLPASS
Fork USDC settlement transfer0x71c148bc426d7e0ada0de60752855097551ca4353cc09db02fbe312846edb08abuyerseller1000 micro-USDCPASS
Runtime Evidence
QA environment:
ItemValue
OS / archLinux aarch64
Backendk3d v5.8.3, k3s v1.31.5-k3s1
Tool versionsDocker 29.2.1, kubectl client v1.35.3, Helm v3.20.1, helmfile 1.4.3, Go 1.25.3, Foundry 1.5.1-stable
QA agent/modelOpenAI-compatible QA endpoint, qwen36-fast
Images:
ComponentImageTag / digestSource
cloudflaredcloudflare/cloudflared2026.3.0embedded cloudflared values
x402 facilitatorghcr.io/x402-rs/x402-facilitator1.4.7flow-10 / flow-13 fork tests
x402 verifierghcr.io/obolnetwork/x402-verifierlatestembedded base manifest
x402 buyerghcr.io/obolnetwork/x402-buyerlatestembedded LiteLLM sidecar manifest
LiteLLMghcr.io/obolnetwork/litellmsha-c16b156embedded base manifest
OpenClawghcr.io/obolnetwork/openclawv2026.4.21pinned OPENCLAW_VERSION
OpenClaw chartobol/openclaw0.4.0CLI-generated agent helmfile
remote-signer chartobol/remote-signer0.3.2CLI-generated agent helmfile
eRPCghcr.io/erpc/erpc0.0.62embedded eRPC values
Kubernetes / stack:
ItemValue
Stack IDsunique per flow worktree; removed by cleanup
Namespacesx402, llm, erpc, traefik, hermes-obol-agent exercised
Pod readinessx402 verifier/buyer, LiteLLM, Hermes API, remote-signer, eRPC, and cloudflared rolled out in flows
Cleanup resultno QA tmux sessions, no k3d clusters, no flow containers, no Anvil fork processes remaining
Model and routing:
ItemValue
Agent/model usedqwen36-fast
LiteLLM routepaid/qwen36-fast through x402-buyer sidecar
Paid endpoint statusHTTP 200 in USDC live, OBOL live, and OBOL fork paid inference
Auth token sourceobol agent auth --runtime hermes obol-agent; raw bearer tokens omitted
Artifacts and logs:
ArtifactLocation / linkNotes
Release reportretained on QA machine as release-smoke-20260505-035957-artifacts/RELEASE_REPORT.mdsanitized summary copied above
Receipt bundlesretained on QA machine under flow-11-receipts, flow-14-receipts, flow-13-receiptsincludes registration, metadata, mint/funding, settlement receipts
Demo readiness:
ItemStatusNotes
Seller visible / registeredPASSERC-8004 registration reflected for live USDC and live OBOL flows
Buyer discovery worksPASSHermes agent discovery issued and purchase confirmed by PurchaseRequest Ready=True
Paid route worksPASSpaid/qwen36-fast returned HTTP 200 with coherent response
Settlement visible on-chainPASSexact balance deltas and settlement receipts verified
Review Notes
Known gaps:
  • Standalone local integration BDD command was not used as the release gate because the local dev stack setup timed out before assertions. The remote release-smoke suite covered the end-to-end behavior and passed fully.
  • x402-buyer and x402-verifier still use latest; image pinning/digesting remains a follow-up unless intentionally handled in this PR.
Follow-ups:
  • Replace latest x402 sidecar images with stable tags or digests when CI publishes a suitable artifact.
  • Decide whether the verifier restart after x402 pricing changes should be replaced by a controller/runtime reload path rather than remaining a flow workaround.
Reviewer focus:
  • Confirm the flow split is clear: live Base Sepolia OBOL is the release gate, Anvil fork OBOL is an explicit regression path.
  • Review the flow-10 RPC resolver and flow-13 cluster-side Anvil probe for portability across QA machines.
  • Review stale OBOL_INGRESS_URL handling so .env defaults do not override dynamic high-port ingress during parallel QA.

This was referenced May 4, 2026
 Closed

 Closed

OisinKyne
OisinKyne
approved these changes
May 5, 2026
View reviewed changes
@OisinKyneOisinKyne
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

biggest thing i worry about is the sponsorship removal and if we documented that everywhere. the mainnet sponsored facilitator tried to set a 7702 delegation to an unknown wallet.
We can do 7702 stuff in its own pr in future to handle broad gas sponsorship.
@OisinKyne
OisinKyne
merged commit a6cd2c2
into
mainMay 5, 2026

6 checks passed

@OisinKyne
OisinKyne
deleted the

integration/rc8-superset

branch
May 5, 2026 00:15
@bussyjdbussyjd
mentioned this pull request
Jun 27, 2026
 Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@OisinKyneOisinKyneOisinKyne approved these changes

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

2 participants

@bussyjd@OisinKyne