From 8b17dbcd9cafaee1ddfcc6fb61da1288e5ca8baf Mon Sep 17 00:00:00 2001 From: bussyjd Date: Mon, 11 May 2026 20:43:13 +0800 Subject: [PATCH 1/3] test(flow-08): tighten buy-side correctness assertions per specialist review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three correctness gaps surfaced by an audit against the named payment invariants in references/live-obol-qa.md and references/paid-commerce.md: 1. Buyer-wallet invariant. flow-08 previously funded whatever wallet the default obol-agent happened to generate at stack init — the exact "do not fund a generated signer" anti-pattern named in the live-OBOL QA reference. Now derives the deterministic Bob address from .env REMOTE_SIGNER_PRIVATE_KEY (the canonical keccak-of-abi-encode pattern used by flow-11/13/14) and asserts AGENT_WALLET == BOB_WALLET before funding. The flow header documents the upstream pre-seed requirement. 2. Exact balance deltas. Replaces "seller balance increased" + missing buyer-side check with strict pre/post deltas on both sides: post_seller - pre_seller == PAID_AMOUNT AND pre_buyer - post_buyer == PAID_AMOUNT. Also removes a swallowed- failure else-branch that emitted `pass` when the seller balance had neither increased nor stayed equal (i.e. decrease was reported as pass). 3. Decouple paid-inference correctness from model wording. The pre- existing assertion required the model to return the verbatim string "USDC payment smoke test passed." Replaced with a structural check: HTTP 200 + non-empty TEXT. The verbatim match is kept as a separate informational `pass` line. Aligns with paid-commerce.md ("do not rely on agent wording"). Secondary correctness tightenings rolled in: - Fail-fast on empty PAID_AMOUNT from the 402 body (previously silent; only surfaced much later at the settlement-receipt step). - Master-key read failure now `emit_metrics; exit 1` instead of continuing with an empty bearer token. - x402-buyer auth-pool assertion now requires the exact expected count (EXPECTED_AUTHS, derived from BUY_BUDGET_USDC / per-request price) instead of the loose `remaining=[1-9]` (single-digit) pattern. - New post-call step asserts remaining decremented by exactly 1 — the spend-proof half of the sidecar contract. - Anvil funding poll regex broadened from exact `^1000000000 ` to `^[1-9][0-9]{8,} ` so a re-run with pre-existing balance doesn't fail the poll. The unused BUY_AUTH_COUNT=5 declaration is removed; the same value is now derived and asserted via EXPECTED_AUTHS. --- flows/flow-08-buy.sh | 136 +++++++++++++++++++++++++++++++------------ 1 file changed, 100 insertions(+), 36 deletions(-) diff --git a/flows/flow-08-buy.sh b/flows/flow-08-buy.sh index 1c653f037..26c3e9603 100755 --- a/flows/flow-08-buy.sh +++ b/flows/flow-08-buy.sh @@ -1,6 +1,13 @@ #!/bin/bash # Flow 08: Buy — monetize-inference.md §2.1-2.5. # Requires: flow-06 (ServiceOffer Ready) + flow-10 (Anvil + facilitator running). +# +# Buyer-wallet invariant: the default obol-agent must be pre-seeded with the +# deterministic "Bob" key derived from .env REMOTE_SIGNER_PRIVATE_KEY so +# funding here lands on a reproducible address. flow-08 asserts the match +# below and fails fast if obol-agent generated a random wallet (which would +# pass storage-slot funding but defeat the named "do not fund a generated +# signer" invariant in references/live-obol-qa.md). source "$(dirname "$0")/lib.sh" TUNNEL_OUTPUT=$("$OBOL" tunnel status 2>&1) || true @@ -29,8 +36,23 @@ AGENT_NS="hermes-obol-agent" AGENT_DEPLOY="hermes" AGENT_CONTAINER="hermes" AGENT_BUY_PY="/data/.hermes/obol-skills/buy-x402/scripts/buy.py" -BUY_AUTH_COUNT=5 BUY_BUDGET_USDC="0.005" +# Derived from BUY_BUDGET_USDC / flow-06 per-request price ("0.001" USDC). +# Used to assert the sidecar publishes the expected number of auths and that +# the post-call remaining count decreases by exactly one. +EXPECTED_AUTHS=5 + +# Derive deterministic Bob from REMOTE_SIGNER_PRIVATE_KEY (canonical pattern +# from flow-11). flow-08 asserts that the default obol-agent's wallet equals +# this address; if not, the agent was generated rather than pre-seeded. +SIGNER_KEY=$({ grep -E '^[[:space:]]*REMOTE_SIGNER_PRIVATE_KEY=' "$OBOL_ROOT/.env" 2>/dev/null || true; } | head -1 | cut -d= -f2-) +[ -n "$SIGNER_KEY" ] || SIGNER_KEY="${REMOTE_SIGNER_PRIVATE_KEY:-}" +if [ -z "$SIGNER_KEY" ]; then + fail "REMOTE_SIGNER_PRIVATE_KEY not found in .env or environment" + emit_metrics; exit 1 +fi +BOB_PRIVATE_KEY=$(env -u CHAIN cast keccak "$(env -u CHAIN cast abi-encode 'f(bytes32,uint256)' "$SIGNER_KEY" 2)") +BOB_WALLET=$(env -u CHAIN cast wallet address --private-key "$BOB_PRIVATE_KEY" 2>/dev/null) purchase_request_ready() { "$OBOL" kubectl get purchaserequests.obol.org "$PURCHASE_NAME" -n "$AGENT_NS" \ @@ -192,6 +214,10 @@ d = json.load(sys.stdin) a = d['accepts'][0] print(a.get('amount') or a.get('maxAmountRequired') or '') " 2>/dev/null | tr -d '[:space:]') +if [ -z "$PAID_AMOUNT" ]; then + fail "Could not parse PAID_AMOUNT from 402 body — ${body_402:0:200}" + emit_metrics; exit 1 +fi step "Supported paid flow uses public tunnel URL" if [ -n "$TUNNEL_URL" ]; then @@ -208,12 +234,14 @@ else fail "Could not pin eRPC base-sepolia to local Anvil — ${network_out:0:200}" fi -step "Agent wallet discovered" +step "Agent wallet matches deterministic Bob" AGENT_WALLET=$("$OBOL" agent wallet list obol-agent 2>/dev/null | grep -oE '0x[a-fA-F0-9]{40}' | head -1 || true) -if [ -n "$AGENT_WALLET" ]; then - pass "Agent wallet: $AGENT_WALLET" -else +if [ -z "$AGENT_WALLET" ]; then fail "Could not resolve obol-agent wallet address" +elif [ "$(printf '%s' "$AGENT_WALLET" | tr '[:upper:]' '[:lower:]')" != "$(printf '%s' "$BOB_WALLET" | tr '[:upper:]' '[:lower:]')" ]; then + fail "Agent wallet $AGENT_WALLET != deterministic Bob $BOB_WALLET (preseed missing; obol-agent must be created with REMOTE_SIGNER_PRIVATE_KEY-derived Bob — see references/live-obol-qa.md)" +else + pass "Agent wallet matches deterministic Bob: $AGENT_WALLET" fi step "Fund agent wallet with USDC on local Anvil" @@ -227,7 +255,7 @@ else fail "Could not fund agent wallet on Anvil — ${AGENT_SLOT:0:120}" fi -poll_step_grep "Agent wallet funded on local Anvil" "^1000000000 " 24 5 agent_wallet_anvil_balance +poll_step_grep "Agent wallet funded on local Anvil" "^[1-9][0-9]{8,} " 24 5 agent_wallet_anvil_balance step "Ensure PurchaseRequest auth pool via obol buy inference" buy_out=$("$OBOL" buy inference "$PURCHASE_NAME" \ @@ -243,7 +271,7 @@ else fi poll_step_grep "PurchaseRequest Ready" "True" 36 5 purchase_request_ready -poll_step_grep "x402-buyer has a live auth pool" "$PURCHASE_NAME: remaining=[1-9]" 36 5 buyer_sidecar_status +poll_step_grep "x402-buyer has exactly $EXPECTED_AUTHS auths" "$PURCHASE_NAME: remaining=$EXPECTED_AUTHS " 36 5 buyer_sidecar_status buyer_status=$(buyer_sidecar_status) PAID_MODEL=$(echo "$buyer_status" | grep "^$PURCHASE_NAME:" | grep -oE 'model=[^ ]+' | head -1 | cut -d= -f2) @@ -254,14 +282,20 @@ fi LITELLM_MASTER_KEY=$("$OBOL" kubectl get secret litellm-secrets -n llm -o jsonpath='{.data.LITELLM_MASTER_KEY}' 2>/dev/null | base64 -d 2>/dev/null || true) if [ -z "$LITELLM_MASTER_KEY" ]; then fail "Could not read LiteLLM master key" + emit_metrics; exit 1 fi -# §2.4 pre-capture: Record seller balance BEFORE paid inference to verify settlement. +# §2.4 pre-capture: Record buyer + seller balances BEFORE paid inference so we +# can assert that settlement moved EXACTLY PAID_AMOUNT in each direction. PRE_SELLER_BAL="" +PRE_BUYER_BAL="" if command -v cast &>/dev/null; then PRE_SELLER_BAL=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$SELLER_WALLET" \ --rpc-url "$ANVIL_RPC" 2>&1) || true [[ "$PRE_SELLER_BAL" =~ ^[0-9] ]] || PRE_SELLER_BAL="" + PRE_BUYER_BAL=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$AGENT_WALLET" \ + --rpc-url "$ANVIL_RPC" 2>&1) || true + [[ "$PRE_BUYER_BAL" =~ ^[0-9] ]] || PRE_BUYER_BAL="" fi # Capture start block immediately before the paid request. @@ -273,11 +307,21 @@ fi step "Paid inference via LiteLLM paid/* route" paid_out=$(litellm_paid_inference) -if echo "$paid_out" | grep -q "STATUS=200" && \ - echo "$paid_out" | grep -q "TEXT=.*USDC payment smoke test passed\."; then - pass "Paid inference succeeded via $PAID_MODEL" +# Structural assertion only: HTTP 200 + non-empty TEXT. Payment correctness +# must not depend on the model returning a verbatim sentence (references/ +# paid-commerce.md: "Do not rely on agent wording"). +paid_status_ok=0; paid_text="" +if echo "$paid_out" | grep -q "STATUS=200"; then + paid_status_ok=1 + paid_text=$(echo "$paid_out" | grep -E '^TEXT=' | head -1 | sed -E 's/^TEXT=//') +fi +if [ "$paid_status_ok" = "1" ] && [ -n "${paid_text// /}" ]; then + pass "Paid inference: HTTP 200 + non-empty content via $PAID_MODEL" + if echo "$paid_text" | grep -qF "USDC payment smoke test passed."; then + pass "(informational) model also returned the verbatim instruction sentence" + fi else - fail "Paid inference failed — ${paid_out:0:500}" + fail "Paid inference failed (status_ok=$paid_status_ok text_empty=$([ -z "${paid_text// /}" ] && echo yes || echo no)) — ${paid_out:0:500}" fi step "On-chain: settlement receipt" @@ -296,40 +340,60 @@ else fi fi -# §2.4: Balance checks (requires cast/Foundry) -# Use exit-code check + numeric pattern to avoid false positives from cast error messages +# §2.4: Balance + sidecar checks. Each side must move by EXACTLY PAID_AMOUNT +# — "increased" is not a settlement proof. See references/paid-commerce.md. if command -v cast &>/dev/null; then - step "Buyer USDC balance check" - # env -u CHAIN: CHAIN=base-sepolia conflicts with foundry (expects uint64) - if buyer_bal=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$AGENT_WALLET" \ - --rpc-url "$ANVIL_RPC" 2>&1) && [[ "$buyer_bal" =~ ^[0-9] ]]; then - pass "Buyer USDC balance: $buyer_bal" + step "Seller USDC balance increased by exactly PAID_AMOUNT" + seller_bal=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$SELLER_WALLET" \ + --rpc-url "$ANVIL_RPC" 2>&1) || true + if ! [[ "$seller_bal" =~ ^[0-9] ]]; then + fail "Seller balance read failed — ${seller_bal:0:100}" else - fail "Buyer balance check failed — ${buyer_bal:0:100}" + post_seller=$(echo "$seller_bal" | grep -oE '^[0-9]+' | head -1) + pre_seller=$(echo "${PRE_SELLER_BAL:-}" | grep -oE '^[0-9]+' | head -1) + if [ -z "$pre_seller" ] || [ -z "$post_seller" ] || [ -z "$PAID_AMOUNT" ]; then + fail "Seller delta unverifiable (pre=${pre_seller:-?} post=${post_seller:-?} amount=${PAID_AMOUNT:-?})" + else + delta=$(( post_seller - pre_seller )) + if [ "$delta" = "$PAID_AMOUNT" ]; then + pass "Seller delta exactly PAID_AMOUNT: $pre_seller → $post_seller (Δ=$delta)" + else + fail "Seller delta $delta != PAID_AMOUNT $PAID_AMOUNT ($pre_seller → $post_seller)" + fi + fi fi - step "Seller USDC balance increased after payment (§2.4 settlement)" - if seller_bal=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$SELLER_WALLET" \ - --rpc-url "$ANVIL_RPC" 2>&1) && [[ "$seller_bal" =~ ^[0-9] ]]; then - # If we captured a pre-balance, verify it increased (actual settlement check) - if [ -n "$PRE_SELLER_BAL" ] && echo "$paid_out" | grep -q "STATUS=200"; then - pre_num=$(echo "$PRE_SELLER_BAL" | grep -oE '^[0-9]+' | head -1) - post_num=$(echo "$seller_bal" | grep -oE '^[0-9]+' | head -1) - if [ -n "$pre_num" ] && [ -n "$post_num" ] && [ "$post_num" -gt "$pre_num" ] 2>/dev/null; then - pass "Seller USDC balance increased: $pre_num → $post_num (payment settled)" - elif [ "$post_num" = "$pre_num" ]; then - fail "Seller balance unchanged after payment: $pre_num (settlement may have failed)" + step "Buyer USDC balance decreased by exactly PAID_AMOUNT" + buyer_bal=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$AGENT_WALLET" \ + --rpc-url "$ANVIL_RPC" 2>&1) || true + if ! [[ "$buyer_bal" =~ ^[0-9] ]]; then + fail "Buyer balance read failed — ${buyer_bal:0:100}" + else + post_buyer=$(echo "$buyer_bal" | grep -oE '^[0-9]+' | head -1) + pre_buyer=$(echo "${PRE_BUYER_BAL:-}" | grep -oE '^[0-9]+' | head -1) + if [ -z "$pre_buyer" ] || [ -z "$post_buyer" ] || [ -z "$PAID_AMOUNT" ]; then + fail "Buyer delta unverifiable (pre=${pre_buyer:-?} post=${post_buyer:-?} amount=${PAID_AMOUNT:-?})" + else + delta=$(( pre_buyer - post_buyer )) + if [ "$delta" = "$PAID_AMOUNT" ]; then + pass "Buyer delta exactly PAID_AMOUNT: $pre_buyer → $post_buyer (Δ=$delta)" else - pass "Seller USDC balance: $seller_bal (pre-balance: ${PRE_SELLER_BAL:-unknown})" + fail "Buyer delta $delta != PAID_AMOUNT $PAID_AMOUNT ($pre_buyer → $post_buyer)" fi - else - pass "Seller USDC balance: $seller_bal" fi - else - fail "Seller balance check failed — ${seller_bal:0:100}" fi else fail "cast (Foundry) not installed — skipping balance checks" fi +step "x402-buyer auth pool decremented by 1" +remaining_after=$(buyer_sidecar_status | grep "^$PURCHASE_NAME:" | grep -oE 'remaining=[0-9]+' | head -1 | cut -d= -f2) +if [ -z "$remaining_after" ]; then + fail "Could not read x402-buyer remaining after paid call" +elif [ "$remaining_after" = "$(( EXPECTED_AUTHS - 1 ))" ]; then + pass "x402-buyer remaining decremented: $EXPECTED_AUTHS → $remaining_after" +else + fail "x402-buyer remaining=$remaining_after, expected $(( EXPECTED_AUTHS - 1 )) (one auth should have been spent)" +fi + emit_metrics From 2aecd5df9cafc7fabfe2dca096f03ea1bf5860a5 Mon Sep 17 00:00:00 2001 From: bussyjd Date: Tue, 12 May 2026 09:50:52 +0800 Subject: [PATCH 2/3] fix(flow-08): make full buy-side smoke green (RPC, grep flag, sidecar race) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three independent bugs were keeping flow-08 red against the new buyer-wallet invariants — fixing all three takes the buy-side smoke from 10/16 to 16/16: 1. Drop publicnode.com from the Base Sepolia fork-RPC candidates and lead with archive-capable endpoints (drpc, sepolia.base.org, tenderly, onfinality, sentio, pocket). publicnode is non-archive, so once the Anvil fork drifted past its retention window the facilitator's `eth_getStorageAt` for USDC balances returned `state at block #N is pruned` and every paid call failed with `Payment verification failed`. List source: chainlist.org/rpcs.json, archive-tested against USDC. 2. Switch run_step_grep / poll_step_grep from `grep -q` (BRE) to `grep -qE` (ERE). Step [8]'s pattern `^[1-9][0-9]{8,} ` uses an ERE quantifier; under BRE the braces are literal and the pattern can never match, so the step was silently timing out for 120s on every run even though the underlying `cast call balanceOf` was returning the expected 1e9 USDC value. Plain-substring callers (step [11], etc.) are not affected. 3. Wrap step [16] (`x402-buyer auth pool decremented by 1`) in poll_step_grep. The buyer sidecar persists the spent-auth state asynchronously after the upstream returns, so a one-shot read of /status could still report `remaining=EXPECTED_AUTHS` for a few seconds even when settlement, the on-chain Transfer, and the buyer/seller balance deltas had already cleared. Also silence Foundry's nightly-build stderr warning globally for flow runs (FOUNDRY_DISABLE_NIGHTLY_WARNING=1). Nightly is what we want for Base Sepolia archive-lookup support, but its per-invocation warning contaminated cast output and triggered exactly the kind of pattern-match false-FAIL that (2) above was already vulnerable to. Skill update: add a step-6 rule to the obol-stack-dev skill that dev-branch work must use OBOL_DEVELOPMENT=true on obolup.sh and obol stack up — without it, the installer pulls the latest release binary and local branch changes are never exercised. Smoke evidence: clean flow-08 run on this branch posts 16/16 PASS with on-chain settlement tx 0x8da4bc3990853fce60b942fd6bc435ed0c373cdb44b228c8e5dea92a83da75b8, exact ±1000 micro-USDC deltas on buyer and seller, and the sidecar correctly decremented to remaining=4. --- .agents/skills/obol-stack-dev/SKILL.md | 1 + flows/flow-08-buy.sh | 16 +++++++-------- flows/lib.sh | 28 ++++++++++++++++++++++---- 3 files changed, 32 insertions(+), 13 deletions(-) diff --git a/.agents/skills/obol-stack-dev/SKILL.md b/.agents/skills/obol-stack-dev/SKILL.md index a3600ff56..54c01789e 100644 --- a/.agents/skills/obol-stack-dev/SKILL.md +++ b/.agents/skills/obol-stack-dev/SKILL.md @@ -19,6 +19,7 @@ Treat this skill as an operational router. Load only the reference needed for th 3. Use separate QA worktrees on remote machines. 4. Never leak hostnames, personal paths, passwords, or private keys into skill files or PR text. 5. Validate with the narrowest command set that covers the change. +6. On a dev branch (anything other than `main` with the latest release tag), use `OBOL_DEVELOPMENT=true` for `./obolup.sh` and `obol stack up`. The plain `./obolup.sh` downloads the latest tagged release binary and will not exercise local branch changes. If you started a fresh install without it, kill obolup and rerun with the env var before continuing. ## Reference Router diff --git a/flows/flow-08-buy.sh b/flows/flow-08-buy.sh index 26c3e9603..9490eff48 100755 --- a/flows/flow-08-buy.sh +++ b/flows/flow-08-buy.sh @@ -386,14 +386,12 @@ else fail "cast (Foundry) not installed — skipping balance checks" fi -step "x402-buyer auth pool decremented by 1" -remaining_after=$(buyer_sidecar_status | grep "^$PURCHASE_NAME:" | grep -oE 'remaining=[0-9]+' | head -1 | cut -d= -f2) -if [ -z "$remaining_after" ]; then - fail "Could not read x402-buyer remaining after paid call" -elif [ "$remaining_after" = "$(( EXPECTED_AUTHS - 1 ))" ]; then - pass "x402-buyer remaining decremented: $EXPECTED_AUTHS → $remaining_after" -else - fail "x402-buyer remaining=$remaining_after, expected $(( EXPECTED_AUTHS - 1 )) (one auth should have been spent)" -fi +# The buyer sidecar persists the spent-auth state asynchronously after the +# upstream returns, so /status can still report the pre-call count for a few +# seconds even when the on-chain settlement and the buyer/seller balance +# deltas have already cleared. Poll instead of one-shot to remove the race. +poll_step_grep "x402-buyer auth pool decremented by 1" \ + "$PURCHASE_NAME: remaining=$(( EXPECTED_AUTHS - 1 )) " \ + 12 5 buyer_sidecar_status emit_metrics diff --git a/flows/lib.sh b/flows/lib.sh index 70fbf1faa..47ff77848 100755 --- a/flows/lib.sh +++ b/flows/lib.sh @@ -15,6 +15,13 @@ else export PATH="$HOME/.foundry/bin:$HOME/.local/bin:$PATH:/usr/local/go/bin" fi +# Foundry nightly prints a stderr warning on every cast/anvil invocation; the +# flow scripts pattern-match cast output, so the noise causes false FAILs at +# steps that grep for hex/decimal values. Silence it globally for flow runs — +# nightly is the only build that publishes new chain support promptly enough +# for Base Sepolia archive lookups not to drift. +export FOUNDRY_DISABLE_NIGHTLY_WARNING="${FOUNDRY_DISABLE_NIGHTLY_WARNING:-1}" + OBOL_ROOT="${OBOL_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}" # Only an explicit override variable should pin the ingress URL. Using the # computed/exported OBOL_INGRESS_URL itself here can leak a stale port (for @@ -286,7 +293,8 @@ run_step_grep() { local desc="$1"; local pattern="$2"; shift 2 step "$desc" local out - if out=$("$@" 2>&1) && echo "$out" | grep -q "$pattern"; then + # grep -E for parity with poll_step_grep — callers can use ERE quantifiers. + if out=$("$@" 2>&1) && echo "$out" | grep -qE "$pattern"; then pass "$desc" else fail "$desc — pattern '$pattern' not found — ${out:0:200}" @@ -310,11 +318,15 @@ poll_step() { # Poll a command until its output matches a grep pattern poll_step_grep() { local desc="$1"; local pattern="$2"; local max="$3"; local delay="$4"; shift 4 + # grep -E so callers can use ERE quantifiers like {N,} — without -E, grep + # treats the braces literally and the pattern never matches even when the + # output is what the caller intended. Callers that pass plain substrings + # (no special regex chars) are unaffected. step "$desc (polling, max ${max}x${delay}s)" for i in $(seq 1 "$max"); do local out out=$("$@" 2>&1) || true - if echo "$out" | grep -q "$pattern"; then + if echo "$out" | grep -qE "$pattern"; then pass "$desc (attempt $i)" return 0 fi @@ -667,11 +679,19 @@ base_sepolia_rpc_candidates() { printf '%s\n' "$BASE_SEPOLIA_RPC" fi + # Archive-capable endpoints first. publicnode.com is intentionally omitted — + # confirmed non-archive against eth_getStorageAt at historical blocks, which + # causes Anvil-fork-based facilitator verifies to fail with "state at block + # #N is pruned" once the fork drifts past the upstream's retention window. + # Source: chainlist.org/rpcs.json, filtered to chainId 84532, archive-tested + # via historical eth_getStorageAt against USDC (0x036C…CF7e). printf '%s\n' \ - "https://base-sepolia-rpc.publicnode.com" \ "https://base-sepolia.drpc.org" \ "https://sepolia.base.org" \ - "https://base-sepolia.gateway.tenderly.co" + "https://base-sepolia.gateway.tenderly.co" \ + "https://base-sepolia.api.onfinality.io/public" \ + "https://base-sepolia.rpc.sentio.xyz" \ + "https://base-testnet.api.pocket.network" } resolve_base_sepolia_rpc() { From b011fd2c831acbfc213dabecdee631866d51e679 Mon Sep 17 00:00:00 2001 From: bussyjd Date: Tue, 12 May 2026 09:52:28 +0800 Subject: [PATCH 3/3] =?UTF-8?q?docs(skill):=20capture=20flow-08=20takeaway?= =?UTF-8?q?s=20=E2=80=94=20nightly=20Anvil,=20archive=20RPC,=20grep=20-E?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Pull the lessons from the flow-08 green-up into the obol-stack-dev skill so future sessions don't rediscover them: - paid-commerce.md: Anvil must be nightly (stable lags ~5mo behind on Base Sepolia archive lookups); fork-RPC must be archive (publicnode is out, drpc/base/tenderly/onfinality/sentio/pocket are in); long-lived Anvil drifts past upstream retention; FOUNDRY_DISABLE_NIGHTLY_WARNING=1 is load-bearing; poll_step_grep / run_step_grep use grep -E so ERE quantifiers work; sidecar /status is asynchronously consistent with the spent-auth count. - dev-environment.md: the OBOL_DEVELOPMENT=true obolup wrapper is `go run` and its per-invocation rebuild trips short port-forward polls — build a real binary into .workspace/bin/obol before running flows. Foundry isn't managed by obolup; install nightly via foundryup. - troubleshooting.md: three new entries with concrete diagnoses and fix commands — facilitator "state pruned" 503, the silent ERE-quantifier pattern timeout, and the PurchaseRequest tombstone-cleanup ritual when the controller's finalizer doesn't fire. --- .../references/dev-environment.md | 18 ++++++++++ .../references/paid-commerce.md | 7 ++++ .../references/troubleshooting.md | 36 +++++++++++++++++++ 3 files changed, 61 insertions(+) diff --git a/.agents/skills/obol-stack-dev/references/dev-environment.md b/.agents/skills/obol-stack-dev/references/dev-environment.md index 32674c37a..c7198a4c4 100644 --- a/.agents/skills/obol-stack-dev/references/dev-environment.md +++ b/.agents/skills/obol-stack-dev/references/dev-environment.md @@ -37,6 +37,24 @@ go build -o .workspace/bin/obol ./cmd/obol **Important**: Always rebuild after code changes. The `.workspace/bin/obol` is a compiled binary, not a `go run` wrapper. +`obolup.sh` with `OBOL_DEVELOPMENT=true` installs a `go run -a` *wrapper* at `.workspace/bin/obol` for rapid iteration. That wrapper recompiles on every invocation, and any flow step that backgrounds a port-forward and polls within ~5–8 seconds (e.g. `flow-06` step 15) will false-FAIL because the port-forward isn't listening yet. Replace the wrapper with a real binary before running flows: + +```bash +mv .workspace/bin/obol .workspace/bin/obol.wrapper +go build -o .workspace/bin/obol ./cmd/obol +``` + +## Foundry + +`obolup.sh` does not manage Foundry. Install separately and **use nightly, not stable** — stable lags far enough behind that Base Sepolia archive-lookup support drifts, and `flow-08` / `flow-11` payment verification then dies with `state at block #N is pruned` from the facilitator. Install: + +```bash +curl -L https://foundry.paradigm.xyz | bash +foundryup --install nightly +``` + +`flows/lib.sh` sets `FOUNDRY_DISABLE_NIGHTLY_WARNING=1` so nightly's per-invocation stderr warning doesn't bleed into `cast` output and break pattern-matching assertions. Keep that export. + ## Environment Variables When running tests or the binary outside the normal `obol` CLI flow, set these explicitly: diff --git a/.agents/skills/obol-stack-dev/references/paid-commerce.md b/.agents/skills/obol-stack-dev/references/paid-commerce.md index 2ab98a82e..c2b4c5814 100644 --- a/.agents/skills/obol-stack-dev/references/paid-commerce.md +++ b/.agents/skills/obol-stack-dev/references/paid-commerce.md @@ -90,5 +90,12 @@ Valid paths: - `obol stack up` leaves cloudflared at zero replicas. Flows that apply ServiceOffer YAML directly must explicitly scale/restart cloudflared before reading tunnel status. - `--namespace` on `obol sell http` sets both ServiceOffer namespace and upstream service namespace. Use the same namespace for follow-up `sell status|stop|delete`. - For Anvil fork flows, bind Anvil to `0.0.0.0` and point each cluster eRPC at `host.k3d.internal:$ANVIL_PORT`. +- **Anvil must be nightly, not stable.** Stable lags ~5 months behind on Base Sepolia archive-lookup support. With stable, the facilitator's EIP-3009 `eth_getStorageAt` against USDC fails with `state at block #N is pruned` once the fork drifts past the upstream RPC's retention window. Install with `foundryup --install nightly`. +- **Anvil fork-RPC must be archive.** `publicnode.com` is non-archive and was the historic default in `flows/lib.sh::base_sepolia_rpc_candidates` — it is now removed. The archive-capable candidates currently in use: `drpc.org`, `sepolia.base.org`, `tenderly`, `onfinality`, `sentio`, `pocket`. Validate any new addition with a historical `eth_getStorageAt` before trusting it. Source list: chainlist.org/rpcs.json filtered to chainId 84532. +- **A long-lived Anvil drifts.** Fork base block stays fixed at startup; after a few hours of real-network advancement the facilitator's historical state lookups can land before the fork base and miss locally. For release smoke / repeat runs, recreate Anvil between sessions or accept that the fork window is bounded. +- Foundry nightly prints a stderr warning per invocation that contaminates `cast` stdout when stderr is merged (`2>&1`). `flows/lib.sh` exports `FOUNDRY_DISABLE_NIGHTLY_WARNING=1`; preserve this. Any per-flow `cast … 2>&1` pipeline that hits a `grep`/regex assertion will false-FAIL without it. +- `flows/lib.sh::poll_step_grep` and `run_step_grep` use `grep -E`. Patterns are ERE — `{N,}` etc. work as quantifiers. Without `-E` (the pre-fix behaviour), `^[1-9][0-9]{8,} ` and similar patterns silently never match and the step times out even when the output is correct. +- **Sidecar clean-up after PurchaseRequest deletion.** The buyer sidecar state is split across two ConfigMaps in the `llm` namespace (`x402-buyer-config`, `x402-buyer-auths`) plus the in-memory sidecar process. The controller's tombstone cleanup is the supported path; if you bypass it by stripping the finalizer, also remove the per-PR keys from both ConfigMaps and `kubectl rollout restart deployment/litellm -n llm` — otherwise `/status` will still report stale `remaining=` for the deleted purchase. +- `flow-08` step 16 (`x402-buyer auth pool decremented by 1`) is polled, not one-shot. The sidecar persists the spent-auth count asynchronously after the upstream returns; a one-shot read can still report the pre-call count for several seconds even when settlement, the on-chain Transfer, and the buyer/seller balance deltas have cleared. - x402-rs Permit2 support is configured by `eip2612_gas_sponsoring=true` on `v2-eip155-exact`; there is no standalone `v2-eip155-permit2` scheme. - On aarch64 GPU hosts, if cloudflared image pulls stall through the k3d mirror, use `flows/lib.sh::ensure_image_in_k3d cloudflare/cloudflared:2026.3.0 obol-stack-`. diff --git a/.agents/skills/obol-stack-dev/references/troubleshooting.md b/.agents/skills/obol-stack-dev/references/troubleshooting.md index 154afc54f..49c88fb75 100644 --- a/.agents/skills/obol-stack-dev/references/troubleshooting.md +++ b/.agents/skills/obol-stack-dev/references/troubleshooting.md @@ -204,3 +204,39 @@ curl -s http://localhost:18789/v1/chat/completions \ # Check Ollama models curl -s http://localhost:11434/api/tags | jq '.models[].name' ``` + +### `flow-08` payment verification 503 / `state at block #N is pruned` + +**Cause**: facilitator (`x402-rs/x402-facilitator`) does an `eth_getStorageAt` against the Anvil fork's USDC balances slot at a historical block. Anvil forwards to its `--fork-url`, and if the upstream is non-archive (`publicnode.com`) or the fork has drifted past the upstream's retention window, the upstream returns `state at block #N is pruned`. Facilitator surfaces that as `verify_eip3009_payment` → 500, x402-verifier returns 402-retry-failed, LiteLLM returns `503 Payment verification failed`. + +**Fix**: restart Anvil + facilitator with a fresh fork against an archive RPC. `flows/lib.sh::base_sepolia_rpc_candidates` now lists archive endpoints first (`drpc.org`, `sepolia.base.org`, `tenderly`, `onfinality`, `sentio`, `pocket`); `publicnode.com` is excluded. + +```bash +docker rm -f obol-flow10-x402-facilitator +pkill -f '^anvil ' +bash flows/flow-10-anvil-facilitator.sh +``` + +### `flow-08` step 8 `pattern '^[1-9][0-9]{8,} ' not found after 120s` + +**Cause**: pre-fix `poll_step_grep` / `run_step_grep` used `grep -q` (BRE), so ERE quantifier `{8,}` was treated as literal text and never matched the `cast call balanceOf` output even when the value was correct. Side-effect: a Foundry nightly stderr warning leaking through `2>&1` would also fail any cast pattern match. + +**Fix**: both helpers now use `grep -qE`; `FOUNDRY_DISABLE_NIGHTLY_WARNING=1` is exported. Nothing to do operationally — confirm the helpers haven't been reverted. + +### PurchaseRequest stuck in `Terminating` after `kubectl delete` + +**Cause**: the serviceoffer-controller's `obol.org/purchase-finalizer` is responsible for tombstone cleanup (deleting per-PR keys from `x402-buyer-config` / `x402-buyer-auths`, signalling the sidecar). If the controller is unhealthy or paused, deletion hangs on the finalizer. + +**Fix (manual cleanup ritual)**: + +```bash +kubectl patch purchaserequest -n hermes-obol-agent --type=merge \ + -p '{"metadata":{"finalizers":[]}}' +kubectl patch cm x402-buyer-config -n llm --type=json \ + -p='[{"op":"remove","path":"/data/.json"}]' +kubectl patch cm x402-buyer-auths -n llm --type=json \ + -p='[{"op":"remove","path":"/data/.json"}]' +kubectl rollout restart deployment/litellm -n llm +``` + +Without the ConfigMap+restart steps, the sidecar continues to report the deleted PR in `/status` and the next `flow-08` run sees a polluted starting auth pool.