test(flow-08): tighten buy-side correctness assertions

.agents/skills/obol-stack-dev/SKILL.md

@@ -19,6 +19,7 @@ Treat this skill as an operational router. Load only the reference needed for th
 3. Use separate QA worktrees on remote machines.
 4. Never leak hostnames, personal paths, passwords, or private keys into skill files or PR text.
 5. Validate with the narrowest command set that covers the change.
+6. On a dev branch (anything other than `main` with the latest release tag), use `OBOL_DEVELOPMENT=true` for `./obolup.sh` and `obol stack up`. The plain `./obolup.sh` downloads the latest tagged release binary and will not exercise local branch changes. If you started a fresh install without it, kill obolup and rerun with the env var before continuing.
 
 ## Reference Router
 

.agents/skills/obol-stack-dev/references/dev-environment.md

@@ -37,6 +37,24 @@ go build -o .workspace/bin/obol ./cmd/obol
 
 **Important**: Always rebuild after code changes. The `.workspace/bin/obol` is a compiled binary, not a `go run` wrapper.
 
+`obolup.sh` with `OBOL_DEVELOPMENT=true` installs a `go run -a` *wrapper* at `.workspace/bin/obol` for rapid iteration. That wrapper recompiles on every invocation, and any flow step that backgrounds a port-forward and polls within ~5–8 seconds (e.g. `flow-06` step 15) will false-FAIL because the port-forward isn't listening yet. Replace the wrapper with a real binary before running flows:
+
+```bash
+mv .workspace/bin/obol .workspace/bin/obol.wrapper
+go build -o .workspace/bin/obol ./cmd/obol
+```
+
+## Foundry
+
+`obolup.sh` does not manage Foundry. Install separately and **use nightly, not stable** — stable lags far enough behind that Base Sepolia archive-lookup support drifts, and `flow-08` / `flow-11` payment verification then dies with `state at block #N is pruned` from the facilitator. Install:
+
+```bash
+curl -L https://foundry.paradigm.xyz | bash
+foundryup --install nightly
+```
+
+`flows/lib.sh` sets `FOUNDRY_DISABLE_NIGHTLY_WARNING=1` so nightly's per-invocation stderr warning doesn't bleed into `cast` output and break pattern-matching assertions. Keep that export.
+
 ## Environment Variables
 
 When running tests or the binary outside the normal `obol` CLI flow, set these explicitly:

.agents/skills/obol-stack-dev/references/paid-commerce.md

@@ -90,5 +90,12 @@ Valid paths:
 - `obol stack up` leaves cloudflared at zero replicas. Flows that apply ServiceOffer YAML directly must explicitly scale/restart cloudflared before reading tunnel status.
 - `--namespace` on `obol sell http` sets both ServiceOffer namespace and upstream service namespace. Use the same namespace for follow-up `sell status|stop|delete`.
 - For Anvil fork flows, bind Anvil to `0.0.0.0` and point each cluster eRPC at `host.k3d.internal:$ANVIL_PORT`.
+- **Anvil must be nightly, not stable.** Stable lags ~5 months behind on Base Sepolia archive-lookup support. With stable, the facilitator's EIP-3009 `eth_getStorageAt` against USDC fails with `state at block #N is pruned` once the fork drifts past the upstream RPC's retention window. Install with `foundryup --install nightly`.
+- **Anvil fork-RPC must be archive.** `publicnode.com` is non-archive and was the historic default in `flows/lib.sh::base_sepolia_rpc_candidates` — it is now removed. The archive-capable candidates currently in use: `drpc.org`, `sepolia.base.org`, `tenderly`, `onfinality`, `sentio`, `pocket`. Validate any new addition with a historical `eth_getStorageAt` before trusting it. Source list: chainlist.org/rpcs.json filtered to chainId 84532.
+- **A long-lived Anvil drifts.** Fork base block stays fixed at startup; after a few hours of real-network advancement the facilitator's historical state lookups can land before the fork base and miss locally. For release smoke / repeat runs, recreate Anvil between sessions or accept that the fork window is bounded.
+- Foundry nightly prints a stderr warning per invocation that contaminates `cast` stdout when stderr is merged (`2>&1`). `flows/lib.sh` exports `FOUNDRY_DISABLE_NIGHTLY_WARNING=1`; preserve this. Any per-flow `cast … 2>&1` pipeline that hits a `grep`/regex assertion will false-FAIL without it.
+- `flows/lib.sh::poll_step_grep` and `run_step_grep` use `grep -E`. Patterns are ERE — `{N,}` etc. work as quantifiers. Without `-E` (the pre-fix behaviour), `^[1-9][0-9]{8,} ` and similar patterns silently never match and the step times out even when the output is correct.
+- **Sidecar clean-up after PurchaseRequest deletion.** The buyer sidecar state is split across two ConfigMaps in the `llm` namespace (`x402-buyer-config`, `x402-buyer-auths`) plus the in-memory sidecar process. The controller's tombstone cleanup is the supported path; if you bypass it by stripping the finalizer, also remove the per-PR keys from both ConfigMaps and `kubectl rollout restart deployment/litellm -n llm` — otherwise `/status` will still report stale `remaining=` for the deleted purchase.
+- `flow-08` step 16 (`x402-buyer auth pool decremented by 1`) is polled, not one-shot. The sidecar persists the spent-auth count asynchronously after the upstream returns; a one-shot read can still report the pre-call count for several seconds even when settlement, the on-chain Transfer, and the buyer/seller balance deltas have cleared.
 - x402-rs Permit2 support is configured by `eip2612_gas_sponsoring=true` on `v2-eip155-exact`; there is no standalone `v2-eip155-permit2` scheme.
 - On aarch64 GPU hosts, if cloudflared image pulls stall through the k3d mirror, use `flows/lib.sh::ensure_image_in_k3d cloudflare/cloudflared:2026.3.0 obol-stack-<stack-id>`.

.agents/skills/obol-stack-dev/references/troubleshooting.md

@@ -204,3 +204,39 @@ curl -s http://localhost:18789/v1/chat/completions \
 # Check Ollama models
 curl -s http://localhost:11434/api/tags | jq '.models[].name'
 ```
+
+### `flow-08` payment verification 503 / `state at block #N is pruned`
+
+**Cause**: facilitator (`x402-rs/x402-facilitator`) does an `eth_getStorageAt` against the Anvil fork's USDC balances slot at a historical block. Anvil forwards to its `--fork-url`, and if the upstream is non-archive (`publicnode.com`) or the fork has drifted past the upstream's retention window, the upstream returns `state at block #N is pruned`. Facilitator surfaces that as `verify_eip3009_payment` → 500, x402-verifier returns 402-retry-failed, LiteLLM returns `503 Payment verification failed`.
+
+**Fix**: restart Anvil + facilitator with a fresh fork against an archive RPC. `flows/lib.sh::base_sepolia_rpc_candidates` now lists archive endpoints first (`drpc.org`, `sepolia.base.org`, `tenderly`, `onfinality`, `sentio`, `pocket`); `publicnode.com` is excluded.
+
+```bash
+docker rm -f obol-flow10-x402-facilitator
+pkill -f '^anvil '
+bash flows/flow-10-anvil-facilitator.sh
+```
+
+### `flow-08` step 8 `pattern '^[1-9][0-9]{8,} ' not found after 120s`
+
+**Cause**: pre-fix `poll_step_grep` / `run_step_grep` used `grep -q` (BRE), so ERE quantifier `{8,}` was treated as literal text and never matched the `cast call balanceOf` output even when the value was correct. Side-effect: a Foundry nightly stderr warning leaking through `2>&1` would also fail any cast pattern match.
+
+**Fix**: both helpers now use `grep -qE`; `FOUNDRY_DISABLE_NIGHTLY_WARNING=1` is exported. Nothing to do operationally — confirm the helpers haven't been reverted.
+
+### PurchaseRequest stuck in `Terminating` after `kubectl delete`
+
+**Cause**: the serviceoffer-controller's `obol.org/purchase-finalizer` is responsible for tombstone cleanup (deleting per-PR keys from `x402-buyer-config` / `x402-buyer-auths`, signalling the sidecar). If the controller is unhealthy or paused, deletion hangs on the finalizer.
+
+**Fix (manual cleanup ritual)**:
+
+```bash
+kubectl patch purchaserequest <name> -n hermes-obol-agent --type=merge \
+ -p '{"metadata":{"finalizers":[]}}'
+kubectl patch cm x402-buyer-config -n llm --type=json \
+ -p='[{"op":"remove","path":"/data/<name>.json"}]'
+kubectl patch cm x402-buyer-auths -n llm --type=json \
+ -p='[{"op":"remove","path":"/data/<name>.json"}]'
+kubectl rollout restart deployment/litellm -n llm
+```
+
+Without the ConfigMap+restart steps, the sidecar continues to report the deleted PR in `/status` and the next `flow-08` run sees a polluted starting auth pool.

flows/flow-08-buy.sh

@@ -1,6 +1,13 @@
 #!/bin/bash
 # Flow 08: Buy — monetize-inference.md §2.1-2.5.
 # Requires: flow-06 (ServiceOffer Ready) + flow-10 (Anvil + facilitator running).
+#
+# Buyer-wallet invariant: the default obol-agent must be pre-seeded with the
+# deterministic "Bob" key derived from .env REMOTE_SIGNER_PRIVATE_KEY so
+# funding here lands on a reproducible address. flow-08 asserts the match
+# below and fails fast if obol-agent generated a random wallet (which would
+# pass storage-slot funding but defeat the named "do not fund a generated
+# signer" invariant in references/live-obol-qa.md).
 source "$(dirname "$0")/lib.sh"
 
 TUNNEL_OUTPUT=$("$OBOL" tunnel status 2>&1) || true
@@ -29,8 +36,23 @@ AGENT_NS="hermes-obol-agent"
 AGENT_DEPLOY="hermes"
 AGENT_CONTAINER="hermes"
 AGENT_BUY_PY="/data/.hermes/obol-skills/buy-x402/scripts/buy.py"
-BUY_AUTH_COUNT=5
 BUY_BUDGET_USDC="0.005"
+# Derived from BUY_BUDGET_USDC / flow-06 per-request price ("0.001" USDC).
+# Used to assert the sidecar publishes the expected number of auths and that
+# the post-call remaining count decreases by exactly one.
+EXPECTED_AUTHS=5
+
+# Derive deterministic Bob from REMOTE_SIGNER_PRIVATE_KEY (canonical pattern
+# from flow-11). flow-08 asserts that the default obol-agent's wallet equals
+# this address; if not, the agent was generated rather than pre-seeded.
+SIGNER_KEY=$({ grep -E '^[[:space:]]*REMOTE_SIGNER_PRIVATE_KEY=' "$OBOL_ROOT/.env" 2>/dev/null || true; } | head -1 | cut -d= -f2-)
+[ -n "$SIGNER_KEY" ] || SIGNER_KEY="${REMOTE_SIGNER_PRIVATE_KEY:-}"
+if [ -z "$SIGNER_KEY" ]; then
+ fail "REMOTE_SIGNER_PRIVATE_KEY not found in .env or environment"
+ emit_metrics; exit 1
+fi
+BOB_PRIVATE_KEY=$(env -u CHAIN cast keccak "$(env -u CHAIN cast abi-encode 'f(bytes32,uint256)' "$SIGNER_KEY" 2)")
+BOB_WALLET=$(env -u CHAIN cast wallet address --private-key "$BOB_PRIVATE_KEY" 2>/dev/null)
 
 purchase_request_ready() {
  "$OBOL" kubectl get purchaserequests.obol.org "$PURCHASE_NAME" -n "$AGENT_NS" \
@@ -192,6 +214,10 @@ d = json.load(sys.stdin)
 a = d['accepts'][0]
 print(a.get('amount') or a.get('maxAmountRequired') or '')
 " 2>/dev/null | tr -d '[:space:]')
+if [ -z "$PAID_AMOUNT" ]; then
+ fail "Could not parse PAID_AMOUNT from 402 body — ${body_402:0:200}"
+ emit_metrics; exit 1
+fi
 
 step "Supported paid flow uses public tunnel URL"
 if [ -n "$TUNNEL_URL" ]; then
@@ -208,12 +234,14 @@ else
  fail "Could not pin eRPC base-sepolia to local Anvil — ${network_out:0:200}"
 fi
 
-step "Agent wallet discovered"
+step "Agent wallet matches deterministic Bob"
 AGENT_WALLET=$("$OBOL" agent wallet list obol-agent 2>/dev/null | grep -oE '0x[a-fA-F0-9]{40}' | head -1 || true)
-if [ -n "$AGENT_WALLET" ]; then
- pass "Agent wallet: $AGENT_WALLET"
-else
+if [ -z "$AGENT_WALLET" ]; then
  fail "Could not resolve obol-agent wallet address"
+elif [ "$(printf '%s' "$AGENT_WALLET" | tr '[:upper:]' '[:lower:]')" != "$(printf '%s' "$BOB_WALLET" | tr '[:upper:]' '[:lower:]')" ]; then
+ fail "Agent wallet $AGENT_WALLET != deterministic Bob $BOB_WALLET (preseed missing; obol-agent must be created with REMOTE_SIGNER_PRIVATE_KEY-derived Bob — see references/live-obol-qa.md)"
+else
+ pass "Agent wallet matches deterministic Bob: $AGENT_WALLET"
 fi
 
 step "Fund agent wallet with USDC on local Anvil"
@@ -227,7 +255,7 @@ else
  fail "Could not fund agent wallet on Anvil — ${AGENT_SLOT:0:120}"
 fi
 
-poll_step_grep "Agent wallet funded on local Anvil" "^1000000000 " 24 5 agent_wallet_anvil_balance
+poll_step_grep "Agent wallet funded on local Anvil" "^[1-9][0-9]{8,} " 24 5 agent_wallet_anvil_balance
 
 step "Ensure PurchaseRequest auth pool via obol buy inference"
 buy_out=$("$OBOL" buy inference "$PURCHASE_NAME" \
@@ -243,7 +271,7 @@ else
 fi
 
 poll_step_grep "PurchaseRequest Ready" "True" 36 5 purchase_request_ready
-poll_step_grep "x402-buyer has a live auth pool" "$PURCHASE_NAME: remaining=[1-9]" 36 5 buyer_sidecar_status
+poll_step_grep "x402-buyer has exactly $EXPECTED_AUTHS auths" "$PURCHASE_NAME: remaining=$EXPECTED_AUTHS " 36 5 buyer_sidecar_status
 
 buyer_status=$(buyer_sidecar_status)
 PAID_MODEL=$(echo "$buyer_status" | grep "^$PURCHASE_NAME:" | grep -oE 'model=[^ ]+' | head -1 | cut -d= -f2)
@@ -254,14 +282,20 @@ fi
 LITELLM_MASTER_KEY=$("$OBOL" kubectl get secret litellm-secrets -n llm -o jsonpath='{.data.LITELLM_MASTER_KEY}' 2>/dev/null | base64 -d 2>/dev/null || true)
 if [ -z "$LITELLM_MASTER_KEY" ]; then
  fail "Could not read LiteLLM master key"
+ emit_metrics; exit 1
 fi
 
-# §2.4 pre-capture: Record seller balance BEFORE paid inference to verify settlement.
+# §2.4 pre-capture: Record buyer + seller balances BEFORE paid inference so we
+# can assert that settlement moved EXACTLY PAID_AMOUNT in each direction.
 PRE_SELLER_BAL=""
+PRE_BUYER_BAL=""
 if command -v cast &>/dev/null; then
  PRE_SELLER_BAL=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$SELLER_WALLET" \
  --rpc-url "$ANVIL_RPC" 2>&1) || true
  [[ "$PRE_SELLER_BAL" =~ ^[0-9] ]] || PRE_SELLER_BAL=""
+ PRE_BUYER_BAL=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$AGENT_WALLET" \
+ --rpc-url "$ANVIL_RPC" 2>&1) || true
+ [[ "$PRE_BUYER_BAL" =~ ^[0-9] ]] || PRE_BUYER_BAL=""
 fi
 
 # Capture start block immediately before the paid request.
@@ -273,11 +307,21 @@ fi
 
 step "Paid inference via LiteLLM paid/* route"
 paid_out=$(litellm_paid_inference)
-if echo "$paid_out" | grep -q "STATUS=200" && \
- echo "$paid_out" | grep -q "TEXT=.*USDC payment smoke test passed\."; then
- pass "Paid inference succeeded via $PAID_MODEL"
+# Structural assertion only: HTTP 200 + non-empty TEXT. Payment correctness
+# must not depend on the model returning a verbatim sentence (references/
+# paid-commerce.md: "Do not rely on agent wording").
+paid_status_ok=0; paid_text=""
+if echo "$paid_out" | grep -q "STATUS=200"; then
+ paid_status_ok=1
+ paid_text=$(echo "$paid_out" | grep -E '^TEXT=' | head -1 | sed -E 's/^TEXT=//')
+fi
+if [ "$paid_status_ok" = "1" ] && [ -n "${paid_text// /}" ]; then
+ pass "Paid inference: HTTP 200 + non-empty content via $PAID_MODEL"
+ if echo "$paid_text" | grep -qF "USDC payment smoke test passed."; then
+ pass "(informational) model also returned the verbatim instruction sentence"
+ fi
 else
- fail "Paid inference failed — ${paid_out:0:500}"
+ fail "Paid inference failed (status_ok=$paid_status_ok text_empty=$([ -z "${paid_text// /}" ] && echo yes || echo no)) — ${paid_out:0:500}"
 fi
 
 step "On-chain: settlement receipt"
@@ -296,40 +340,58 @@ else
  fi
 fi
 
-# §2.4: Balance checks (requires cast/Foundry)
-# Use exit-code check + numeric pattern to avoid false positives from cast error messages
+# §2.4: Balance + sidecar checks. Each side must move by EXACTLY PAID_AMOUNT
+# — "increased" is not a settlement proof. See references/paid-commerce.md.
 if command -v cast &>/dev/null; then
- step "Buyer USDC balance check"
-# env -u CHAIN: CHAIN=base-sepolia conflicts with foundry (expects uint64)
-if buyer_bal=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$AGENT_WALLET" \
- --rpc-url "$ANVIL_RPC" 2>&1) && [[ "$buyer_bal" =~ ^[0-9] ]]; then
-pass "Buyer USDC balance: $buyer_bal"
+ step "Seller USDC balance increased by exactly PAID_AMOUNT"
+seller_bal=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$SELLER_WALLET" \
+ --rpc-url "$ANVIL_RPC" 2>&1) || true
+if ! [[ "$seller_bal" =~ ^[0-9] ]]; then
+fail "Seller balance read failed — ${seller_bal:0:100}"
  else
- fail "Buyer balance check failed — ${buyer_bal:0:100}"
+ post_seller=$(echo "$seller_bal" | grep -oE '^[0-9]+' | head -1)
+ pre_seller=$(echo "${PRE_SELLER_BAL:-}" | grep -oE '^[0-9]+' | head -1)
+ if [ -z "$pre_seller" ] || [ -z "$post_seller" ] || [ -z "$PAID_AMOUNT" ]; then
+ fail "Seller delta unverifiable (pre=${pre_seller:-?} post=${post_seller:-?} amount=${PAID_AMOUNT:-?})"
+ else
+ delta=$(( post_seller - pre_seller ))
+ if [ "$delta" = "$PAID_AMOUNT" ]; then
+ pass "Seller delta exactly PAID_AMOUNT: $pre_seller → $post_seller (Δ=$delta)"
+ else
+ fail "Seller delta $delta != PAID_AMOUNT $PAID_AMOUNT ($pre_seller → $post_seller)"
+ fi
+ fi
  fi
 
- step "Seller USDC balance increased after payment (§2.4 settlement)"
- if seller_bal=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$SELLER_WALLET" \
- --rpc-url "$ANVIL_RPC" 2>&1) && [[ "$seller_bal" =~ ^[0-9] ]]; then
- # If we captured a pre-balance, verify it increased (actual settlement check)
- if [ -n "$PRE_SELLER_BAL" ] && echo "$paid_out" | grep -q "STATUS=200"; then
- pre_num=$(echo "$PRE_SELLER_BAL" | grep -oE '^[0-9]+' | head -1)
- post_num=$(echo "$seller_bal" | grep -oE '^[0-9]+' | head -1)
- if [ -n "$pre_num" ] && [ -n "$post_num" ] && [ "$post_num" -gt "$pre_num" ] 2>/dev/null; then
- pass "Seller USDC balance increased: $pre_num → $post_num (payment settled)"
- elif [ "$post_num" = "$pre_num" ]; then
- fail "Seller balance unchanged after payment: $pre_num (settlement may have failed)"
+ step "Buyer USDC balance decreased by exactly PAID_AMOUNT"
+ buyer_bal=$(env -u CHAIN cast call "$USDC_ADDRESS" "balanceOf(address)(uint256)" "$AGENT_WALLET" \
+ --rpc-url "$ANVIL_RPC" 2>&1) || true
+ if ! [[ "$buyer_bal" =~ ^[0-9] ]]; then
+ fail "Buyer balance read failed — ${buyer_bal:0:100}"
+ else
+ post_buyer=$(echo "$buyer_bal" | grep -oE '^[0-9]+' | head -1)
+ pre_buyer=$(echo "${PRE_BUYER_BAL:-}" | grep -oE '^[0-9]+' | head -1)
+ if [ -z "$pre_buyer" ] || [ -z "$post_buyer" ] || [ -z "$PAID_AMOUNT" ]; then
+ fail "Buyer delta unverifiable (pre=${pre_buyer:-?} post=${post_buyer:-?} amount=${PAID_AMOUNT:-?})"
+ else
+ delta=$(( pre_buyer - post_buyer ))
+ if [ "$delta" = "$PAID_AMOUNT" ]; then
+ pass "Buyer delta exactly PAID_AMOUNT: $pre_buyer → $post_buyer (Δ=$delta)"
  else
-pass "Seller USDC balance: $seller_bal (pre-balance: ${PRE_SELLER_BAL:-unknown})"
+fail "Buyer delta $delta != PAID_AMOUNT $PAID_AMOUNT ($pre_buyer → $post_buyer)"
  fi
- else
- pass "Seller USDC balance: $seller_bal"
  fi
- else
- fail "Seller balance check failed — ${seller_bal:0:100}"
  fi
 else
  fail "cast (Foundry) not installed — skipping balance checks"
 fi
 
+# The buyer sidecar persists the spent-auth state asynchronously after the
+# upstream returns, so /status can still report the pre-call count for a few
+# seconds even when the on-chain settlement and the buyer/seller balance
+# deltas have already cleared. Poll instead of one-shot to remove the race.
+poll_step_grep "x402-buyer auth pool decremented by 1" \
+ "$PURCHASE_NAME: remaining=$(( EXPECTED_AUTHS - 1 )) " \
+ 12 5 buyer_sidecar_status
+
 emit_metrics