fix: x402 verifier TLS and facilitator compatibility by bussyjd · Pull Request #320 · ObolNetwork/obol-stack

bussyjd · 2026-04-08T05:20:35Z

Summary

CA certificate bundle: The x402-verifier distroless image has no CA store. obol sell pricing now populates the ca-certificates ConfigMap from the host's cert bundle so TLS verification of facilitator.x402.rs works out of the box.
Missing Description field: The facilitator rejects verify requests without description in PaymentRequirement. Added it from the route pattern.

Validated on Base Sepolia testnet

Real x402 payment flow between two ARM64 nodes running Nemotron 120B (120B parameter model, tensor-parallel across two GPUs):

Alice (seller): obolup.sh → obol stack up → obol sell pricing → obol sell http nemotron → tunnel
Bob (buyer): discover .well-known → probe 402 → sign EIP-712 → pay → HTTP 200 + real inference

On-chain receipt

Field	Value
Settlement tx	`0xd769953bab675ab97a5140dbf7b5583367c788ecb445251c19fea7194c231ec0`
Network	Base Sepolia (84532)
Facilitator	`facilitator.x402.rs` (real public settlement)
Amount	1000 micro-USDC (0.001 USDC)
Seller balance delta	+0.001 USDC
Buyer balance delta	-0.001 USDC

Test plan

go build ./... compiles
go test ./internal/x402/... passes
Real Base Sepolia payment: 402 → sign → 200 + on-chain settlement
Inference from Nemotron 120B served through the paid route

Two fixes validated with real Base Sepolia x402 payments between two DGX Spark nodes running Nemotron 120B inference. 1. **CA certificate bundle**: The x402-verifier runs in a distroless container with no CA store. TLS verification of the public facilitator (facilitator.x402.rs) fails with "x509: certificate signed by unknown authority". Fix: `obol sell pricing` now reads the host CA bundle and patches it into the `ca-certificates` ConfigMap mounted by the verifier. 2. **Missing Description field**: The facilitator rejects verify requests that lack a `description` field in PaymentRequirement with "invalid_format". Fix: populate Description from the route pattern when building the payment requirement. ## Validated testnet flow ### Alice (seller) ``` obolup.sh # bootstrap dependencies obol stack init && obol stack up obol model setup custom --name nemotron-120b \ --endpoint http://host.k3d.internal:8000/v1 \ --model "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" obol sell pricing --wallet 0xC0De...97E --chain base-sepolia obol sell http nemotron \ --wallet 0xC0De...97E --chain base-sepolia \ --per-request 0.001 --namespace llm \ --upstream litellm --port 4000 \ --health-path /health/readiness \ --register --register-name "Nemotron 120B on DGX Spark" obol tunnel restart ``` ### Bob (buyer) ``` # 1. Discover curl $TUNNEL/.well-known/agent-registration.json # → name: "Nemotron 120B on DGX Spark", x402Support: true # 2. Probe curl -X POST $TUNNEL/services/nemotron/v1/chat/completions # → 402: payTo=0xC0De...97E, amount=1000, network=base-sepolia # 3. Sign EIP-712 TransferWithAuthorization + pay python3 bob_buy.py # → 200: "The meaning of life is to discover and pursue purpose" ``` ### On-chain receipts (Base Sepolia) | Tx | Description | |----|-------------| | 0xd769953b...c231ec0 | x402 settlement: Bob→Alice 0.001 USDC via ERC-3009 | Balance change: Alice +0.001 USDC, Bob -0.001 USDC. Facilitator: https://facilitator.x402.rs (real public settlement).

Replace the third-party facilitator.x402.rs with the Obol-operated facilitator at x402.gcp.obol.tech. This gives us control over uptime, chain support, and monitoring (Grafana dashboards already deployed in obol-infrastructure). Introduces DefaultFacilitatorURL constant in internal/x402 and updates all references: CLI flag default, config loader, standalone inference gateway, and deployment store. Companion PR in obol-infrastructure adds Base Sepolia (84532) to the facilitator's chain config alongside Base Mainnet (8453).

Address #321 — LiteLLM reliability improvements: 1. Hot-add models via /model/new API instead of restarting the deployment. ConfigMap still patched for persistence. Restart only triggered when API keys change (Secret mount requires it). 2. Scale to 2 replicas with RollingUpdate (maxUnavailable: 0, maxSurge: 1) so a new pod is ready before any old pod terminates. 3. PodDisruptionBudget (minAvailable: 1) prevents both replicas from being down simultaneously during voluntary disruptions. 4. preStop hook (sleep 10) gives EndpointSlice time to deregister the terminating pod before SIGTERM — prevents in-flight request drops during rolling updates. 5. Reloader annotation on litellm-secrets — Stakater Reloader triggers rolling restart on API key rotation, no manual restart. 6. terminationGracePeriodSeconds: 60 — long inference requests (e.g. Nemotron 120B at 30s+) have time to complete.

…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd · 2026-04-08T08:33:54Z

Split into separate PRs: #322 (TLS fix), #323 (facilitator default), #324 (LiteLLM reliability), #325 (obolup.sh)

…est-pinned) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: ObolNetwork/obol-stack-front-end#322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321)

…est-pinned) (#505) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: https://github.com/ObolNetwork/obol-stack-front-end/pull/322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321) Co-authored-by: bussyjd <bussyjd@users.noreply.github.com>

bussyjd · 2026-04-08T05:20:35Z

Summary

CA certificate bundle: The x402-verifier distroless image has no CA store. obol sell pricing now populates the ca-certificates ConfigMap from the host's cert bundle so TLS verification of facilitator.x402.rs works out of the box.
Missing Description field: The facilitator rejects verify requests without description in PaymentRequirement. Added it from the route pattern.

Validated on Base Sepolia testnet

Real x402 payment flow between two ARM64 nodes running Nemotron 120B (120B parameter model, tensor-parallel across two GPUs):

Alice (seller): obolup.sh → obol stack up → obol sell pricing → obol sell http nemotron → tunnel
Bob (buyer): discover .well-known → probe 402 → sign EIP-712 → pay → HTTP 200 + real inference

On-chain receipt

Field	Value
Settlement tx	`0xd769953bab675ab97a5140dbf7b5583367c788ecb445251c19fea7194c231ec0`
Network	Base Sepolia (84532)
Facilitator	`facilitator.x402.rs` (real public settlement)
Amount	1000 micro-USDC (0.001 USDC)
Seller balance delta	+0.001 USDC
Buyer balance delta	-0.001 USDC

Test plan

go build ./... compiles
go test ./internal/x402/... passes
Real Base Sepolia payment: 402 → sign → 200 + on-chain settlement
Inference from Nemotron 120B served through the paid route

Two fixes validated with real Base Sepolia x402 payments between two DGX Spark nodes running Nemotron 120B inference. 1. **CA certificate bundle**: The x402-verifier runs in a distroless container with no CA store. TLS verification of the public facilitator (facilitator.x402.rs) fails with "x509: certificate signed by unknown authority". Fix: `obol sell pricing` now reads the host CA bundle and patches it into the `ca-certificates` ConfigMap mounted by the verifier. 2. **Missing Description field**: The facilitator rejects verify requests that lack a `description` field in PaymentRequirement with "invalid_format". Fix: populate Description from the route pattern when building the payment requirement. ## Validated testnet flow ### Alice (seller) ``` obolup.sh # bootstrap dependencies obol stack init && obol stack up obol model setup custom --name nemotron-120b \ --endpoint http://host.k3d.internal:8000/v1 \ --model "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" obol sell pricing --wallet 0xC0De...97E --chain base-sepolia obol sell http nemotron \ --wallet 0xC0De...97E --chain base-sepolia \ --per-request 0.001 --namespace llm \ --upstream litellm --port 4000 \ --health-path /health/readiness \ --register --register-name "Nemotron 120B on DGX Spark" obol tunnel restart ``` ### Bob (buyer) ``` # 1. Discover curl $TUNNEL/.well-known/agent-registration.json # → name: "Nemotron 120B on DGX Spark", x402Support: true # 2. Probe curl -X POST $TUNNEL/services/nemotron/v1/chat/completions # → 402: payTo=0xC0De...97E, amount=1000, network=base-sepolia # 3. Sign EIP-712 TransferWithAuthorization + pay python3 bob_buy.py # → 200: "The meaning of life is to discover and pursue purpose" ``` ### On-chain receipts (Base Sepolia) | Tx | Description | |----|-------------| | 0xd769953b...c231ec0 | x402 settlement: Bob→Alice 0.001 USDC via ERC-3009 | Balance change: Alice +0.001 USDC, Bob -0.001 USDC. Facilitator: https://facilitator.x402.rs (real public settlement).

Replace the third-party facilitator.x402.rs with the Obol-operated facilitator at x402.gcp.obol.tech. This gives us control over uptime, chain support, and monitoring (Grafana dashboards already deployed in obol-infrastructure). Introduces DefaultFacilitatorURL constant in internal/x402 and updates all references: CLI flag default, config loader, standalone inference gateway, and deployment store. Companion PR in obol-infrastructure adds Base Sepolia (84532) to the facilitator's chain config alongside Base Mainnet (8453).

Address #321 — LiteLLM reliability improvements: 1. Hot-add models via /model/new API instead of restarting the deployment. ConfigMap still patched for persistence. Restart only triggered when API keys change (Secret mount requires it). 2. Scale to 2 replicas with RollingUpdate (maxUnavailable: 0, maxSurge: 1) so a new pod is ready before any old pod terminates. 3. PodDisruptionBudget (minAvailable: 1) prevents both replicas from being down simultaneously during voluntary disruptions. 4. preStop hook (sleep 10) gives EndpointSlice time to deregister the terminating pod before SIGTERM — prevents in-flight request drops during rolling updates. 5. Reloader annotation on litellm-secrets — Stakater Reloader triggers rolling restart on API key rotation, no manual restart. 6. terminationGracePeriodSeconds: 60 — long inference requests (e.g. Nemotron 120B at 30s+) have time to complete.

…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd · 2026-04-08T08:33:54Z

Split into separate PRs: #322 (TLS fix), #323 (facilitator default), #324 (LiteLLM reliability), #325 (obolup.sh)

…est-pinned) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: ObolNetwork/obol-stack-front-end#322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321)

…est-pinned) (#505) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: https://github.com/ObolNetwork/obol-stack-front-end/pull/322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321) Co-authored-by: bussyjd <bussyjd@users.noreply.github.com>

bussyjd · 2026-04-08T05:20:35Z

Summary

CA certificate bundle: The x402-verifier distroless image has no CA store. obol sell pricing now populates the ca-certificates ConfigMap from the host's cert bundle so TLS verification of facilitator.x402.rs works out of the box.
Missing Description field: The facilitator rejects verify requests without description in PaymentRequirement. Added it from the route pattern.

Validated on Base Sepolia testnet

Real x402 payment flow between two ARM64 nodes running Nemotron 120B (120B parameter model, tensor-parallel across two GPUs):

Alice (seller): obolup.sh → obol stack up → obol sell pricing → obol sell http nemotron → tunnel
Bob (buyer): discover .well-known → probe 402 → sign EIP-712 → pay → HTTP 200 + real inference

On-chain receipt

Field	Value
Settlement tx	`0xd769953bab675ab97a5140dbf7b5583367c788ecb445251c19fea7194c231ec0`
Network	Base Sepolia (84532)
Facilitator	`facilitator.x402.rs` (real public settlement)
Amount	1000 micro-USDC (0.001 USDC)
Seller balance delta	+0.001 USDC
Buyer balance delta	-0.001 USDC

Test plan

go build ./... compiles
go test ./internal/x402/... passes
Real Base Sepolia payment: 402 → sign → 200 + on-chain settlement
Inference from Nemotron 120B served through the paid route

Two fixes validated with real Base Sepolia x402 payments between two DGX Spark nodes running Nemotron 120B inference. 1. **CA certificate bundle**: The x402-verifier runs in a distroless container with no CA store. TLS verification of the public facilitator (facilitator.x402.rs) fails with "x509: certificate signed by unknown authority". Fix: `obol sell pricing` now reads the host CA bundle and patches it into the `ca-certificates` ConfigMap mounted by the verifier. 2. **Missing Description field**: The facilitator rejects verify requests that lack a `description` field in PaymentRequirement with "invalid_format". Fix: populate Description from the route pattern when building the payment requirement. ## Validated testnet flow ### Alice (seller) ``` obolup.sh # bootstrap dependencies obol stack init && obol stack up obol model setup custom --name nemotron-120b \ --endpoint http://host.k3d.internal:8000/v1 \ --model "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" obol sell pricing --wallet 0xC0De...97E --chain base-sepolia obol sell http nemotron \ --wallet 0xC0De...97E --chain base-sepolia \ --per-request 0.001 --namespace llm \ --upstream litellm --port 4000 \ --health-path /health/readiness \ --register --register-name "Nemotron 120B on DGX Spark" obol tunnel restart ``` ### Bob (buyer) ``` # 1. Discover curl $TUNNEL/.well-known/agent-registration.json # → name: "Nemotron 120B on DGX Spark", x402Support: true # 2. Probe curl -X POST $TUNNEL/services/nemotron/v1/chat/completions # → 402: payTo=0xC0De...97E, amount=1000, network=base-sepolia # 3. Sign EIP-712 TransferWithAuthorization + pay python3 bob_buy.py # → 200: "The meaning of life is to discover and pursue purpose" ``` ### On-chain receipts (Base Sepolia) | Tx | Description | |----|-------------| | 0xd769953b...c231ec0 | x402 settlement: Bob→Alice 0.001 USDC via ERC-3009 | Balance change: Alice +0.001 USDC, Bob -0.001 USDC. Facilitator: https://facilitator.x402.rs (real public settlement).

Replace the third-party facilitator.x402.rs with the Obol-operated facilitator at x402.gcp.obol.tech. This gives us control over uptime, chain support, and monitoring (Grafana dashboards already deployed in obol-infrastructure). Introduces DefaultFacilitatorURL constant in internal/x402 and updates all references: CLI flag default, config loader, standalone inference gateway, and deployment store. Companion PR in obol-infrastructure adds Base Sepolia (84532) to the facilitator's chain config alongside Base Mainnet (8453).

Address #321 — LiteLLM reliability improvements: 1. Hot-add models via /model/new API instead of restarting the deployment. ConfigMap still patched for persistence. Restart only triggered when API keys change (Secret mount requires it). 2. Scale to 2 replicas with RollingUpdate (maxUnavailable: 0, maxSurge: 1) so a new pod is ready before any old pod terminates. 3. PodDisruptionBudget (minAvailable: 1) prevents both replicas from being down simultaneously during voluntary disruptions. 4. preStop hook (sleep 10) gives EndpointSlice time to deregister the terminating pod before SIGTERM — prevents in-flight request drops during rolling updates. 5. Reloader annotation on litellm-secrets — Stakater Reloader triggers rolling restart on API key rotation, no manual restart. 6. terminationGracePeriodSeconds: 60 — long inference requests (e.g. Nemotron 120B at 30s+) have time to complete.

…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd · 2026-04-08T08:33:54Z

Split into separate PRs: #322 (TLS fix), #323 (facilitator default), #324 (LiteLLM reliability), #325 (obolup.sh)

…est-pinned) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: ObolNetwork/obol-stack-front-end#322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321)

…est-pinned) (#505) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: https://github.com/ObolNetwork/obol-stack-front-end/pull/322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321) Co-authored-by: bussyjd <bussyjd@users.noreply.github.com>

bussyjd · 2026-04-08T05:20:35Z

Summary

CA certificate bundle: The x402-verifier distroless image has no CA store. obol sell pricing now populates the ca-certificates ConfigMap from the host's cert bundle so TLS verification of facilitator.x402.rs works out of the box.
Missing Description field: The facilitator rejects verify requests without description in PaymentRequirement. Added it from the route pattern.

Validated on Base Sepolia testnet

Real x402 payment flow between two ARM64 nodes running Nemotron 120B (120B parameter model, tensor-parallel across two GPUs):

Alice (seller): obolup.sh → obol stack up → obol sell pricing → obol sell http nemotron → tunnel
Bob (buyer): discover .well-known → probe 402 → sign EIP-712 → pay → HTTP 200 + real inference

On-chain receipt

Field	Value
Settlement tx	`0xd769953bab675ab97a5140dbf7b5583367c788ecb445251c19fea7194c231ec0`
Network	Base Sepolia (84532)
Facilitator	`facilitator.x402.rs` (real public settlement)
Amount	1000 micro-USDC (0.001 USDC)
Seller balance delta	+0.001 USDC
Buyer balance delta	-0.001 USDC

Test plan

go build ./... compiles
go test ./internal/x402/... passes
Real Base Sepolia payment: 402 → sign → 200 + on-chain settlement
Inference from Nemotron 120B served through the paid route

Two fixes validated with real Base Sepolia x402 payments between two DGX Spark nodes running Nemotron 120B inference. 1. **CA certificate bundle**: The x402-verifier runs in a distroless container with no CA store. TLS verification of the public facilitator (facilitator.x402.rs) fails with "x509: certificate signed by unknown authority". Fix: `obol sell pricing` now reads the host CA bundle and patches it into the `ca-certificates` ConfigMap mounted by the verifier. 2. **Missing Description field**: The facilitator rejects verify requests that lack a `description` field in PaymentRequirement with "invalid_format". Fix: populate Description from the route pattern when building the payment requirement. ## Validated testnet flow ### Alice (seller) ``` obolup.sh # bootstrap dependencies obol stack init && obol stack up obol model setup custom --name nemotron-120b \ --endpoint http://host.k3d.internal:8000/v1 \ --model "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" obol sell pricing --wallet 0xC0De...97E --chain base-sepolia obol sell http nemotron \ --wallet 0xC0De...97E --chain base-sepolia \ --per-request 0.001 --namespace llm \ --upstream litellm --port 4000 \ --health-path /health/readiness \ --register --register-name "Nemotron 120B on DGX Spark" obol tunnel restart ``` ### Bob (buyer) ``` # 1. Discover curl $TUNNEL/.well-known/agent-registration.json # → name: "Nemotron 120B on DGX Spark", x402Support: true # 2. Probe curl -X POST $TUNNEL/services/nemotron/v1/chat/completions # → 402: payTo=0xC0De...97E, amount=1000, network=base-sepolia # 3. Sign EIP-712 TransferWithAuthorization + pay python3 bob_buy.py # → 200: "The meaning of life is to discover and pursue purpose" ``` ### On-chain receipts (Base Sepolia) | Tx | Description | |----|-------------| | 0xd769953b...c231ec0 | x402 settlement: Bob→Alice 0.001 USDC via ERC-3009 | Balance change: Alice +0.001 USDC, Bob -0.001 USDC. Facilitator: https://facilitator.x402.rs (real public settlement).

Replace the third-party facilitator.x402.rs with the Obol-operated facilitator at x402.gcp.obol.tech. This gives us control over uptime, chain support, and monitoring (Grafana dashboards already deployed in obol-infrastructure). Introduces DefaultFacilitatorURL constant in internal/x402 and updates all references: CLI flag default, config loader, standalone inference gateway, and deployment store. Companion PR in obol-infrastructure adds Base Sepolia (84532) to the facilitator's chain config alongside Base Mainnet (8453).

Address #321 — LiteLLM reliability improvements: 1. Hot-add models via /model/new API instead of restarting the deployment. ConfigMap still patched for persistence. Restart only triggered when API keys change (Secret mount requires it). 2. Scale to 2 replicas with RollingUpdate (maxUnavailable: 0, maxSurge: 1) so a new pod is ready before any old pod terminates. 3. PodDisruptionBudget (minAvailable: 1) prevents both replicas from being down simultaneously during voluntary disruptions. 4. preStop hook (sleep 10) gives EndpointSlice time to deregister the terminating pod before SIGTERM — prevents in-flight request drops during rolling updates. 5. Reloader annotation on litellm-secrets — Stakater Reloader triggers rolling restart on API key rotation, no manual restart. 6. terminationGracePeriodSeconds: 60 — long inference requests (e.g. Nemotron 120B at 30s+) have time to complete.

…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd · 2026-04-08T08:33:54Z

Split into separate PRs: #322 (TLS fix), #323 (facilitator default), #324 (LiteLLM reliability), #325 (obolup.sh)

…est-pinned) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: ObolNetwork/obol-stack-front-end#322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321)

…est-pinned) (#505) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: https://github.com/ObolNetwork/obol-stack-front-end/pull/322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321) Co-authored-by: bussyjd <bussyjd@users.noreply.github.com>

bussyjd · 2026-04-08T05:20:35Z

Summary

CA certificate bundle: The x402-verifier distroless image has no CA store. obol sell pricing now populates the ca-certificates ConfigMap from the host's cert bundle so TLS verification of facilitator.x402.rs works out of the box.
Missing Description field: The facilitator rejects verify requests without description in PaymentRequirement. Added it from the route pattern.

Validated on Base Sepolia testnet

Real x402 payment flow between two ARM64 nodes running Nemotron 120B (120B parameter model, tensor-parallel across two GPUs):

Alice (seller): obolup.sh → obol stack up → obol sell pricing → obol sell http nemotron → tunnel
Bob (buyer): discover .well-known → probe 402 → sign EIP-712 → pay → HTTP 200 + real inference

On-chain receipt

Field	Value
Settlement tx	`0xd769953bab675ab97a5140dbf7b5583367c788ecb445251c19fea7194c231ec0`
Network	Base Sepolia (84532)
Facilitator	`facilitator.x402.rs` (real public settlement)
Amount	1000 micro-USDC (0.001 USDC)
Seller balance delta	+0.001 USDC
Buyer balance delta	-0.001 USDC

Test plan

go build ./... compiles
go test ./internal/x402/... passes
Real Base Sepolia payment: 402 → sign → 200 + on-chain settlement
Inference from Nemotron 120B served through the paid route

Two fixes validated with real Base Sepolia x402 payments between two DGX Spark nodes running Nemotron 120B inference. 1. **CA certificate bundle**: The x402-verifier runs in a distroless container with no CA store. TLS verification of the public facilitator (facilitator.x402.rs) fails with "x509: certificate signed by unknown authority". Fix: `obol sell pricing` now reads the host CA bundle and patches it into the `ca-certificates` ConfigMap mounted by the verifier. 2. **Missing Description field**: The facilitator rejects verify requests that lack a `description` field in PaymentRequirement with "invalid_format". Fix: populate Description from the route pattern when building the payment requirement. ## Validated testnet flow ### Alice (seller) ``` obolup.sh # bootstrap dependencies obol stack init && obol stack up obol model setup custom --name nemotron-120b \ --endpoint http://host.k3d.internal:8000/v1 \ --model "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" obol sell pricing --wallet 0xC0De...97E --chain base-sepolia obol sell http nemotron \ --wallet 0xC0De...97E --chain base-sepolia \ --per-request 0.001 --namespace llm \ --upstream litellm --port 4000 \ --health-path /health/readiness \ --register --register-name "Nemotron 120B on DGX Spark" obol tunnel restart ``` ### Bob (buyer) ``` # 1. Discover curl $TUNNEL/.well-known/agent-registration.json # → name: "Nemotron 120B on DGX Spark", x402Support: true # 2. Probe curl -X POST $TUNNEL/services/nemotron/v1/chat/completions # → 402: payTo=0xC0De...97E, amount=1000, network=base-sepolia # 3. Sign EIP-712 TransferWithAuthorization + pay python3 bob_buy.py # → 200: "The meaning of life is to discover and pursue purpose" ``` ### On-chain receipts (Base Sepolia) | Tx | Description | |----|-------------| | 0xd769953b...c231ec0 | x402 settlement: Bob→Alice 0.001 USDC via ERC-3009 | Balance change: Alice +0.001 USDC, Bob -0.001 USDC. Facilitator: https://facilitator.x402.rs (real public settlement).

Replace the third-party facilitator.x402.rs with the Obol-operated facilitator at x402.gcp.obol.tech. This gives us control over uptime, chain support, and monitoring (Grafana dashboards already deployed in obol-infrastructure). Introduces DefaultFacilitatorURL constant in internal/x402 and updates all references: CLI flag default, config loader, standalone inference gateway, and deployment store. Companion PR in obol-infrastructure adds Base Sepolia (84532) to the facilitator's chain config alongside Base Mainnet (8453).

Address #321 — LiteLLM reliability improvements: 1. Hot-add models via /model/new API instead of restarting the deployment. ConfigMap still patched for persistence. Restart only triggered when API keys change (Secret mount requires it). 2. Scale to 2 replicas with RollingUpdate (maxUnavailable: 0, maxSurge: 1) so a new pod is ready before any old pod terminates. 3. PodDisruptionBudget (minAvailable: 1) prevents both replicas from being down simultaneously during voluntary disruptions. 4. preStop hook (sleep 10) gives EndpointSlice time to deregister the terminating pod before SIGTERM — prevents in-flight request drops during rolling updates. 5. Reloader annotation on litellm-secrets — Stakater Reloader triggers rolling restart on API key rotation, no manual restart. 6. terminationGracePeriodSeconds: 60 — long inference requests (e.g. Nemotron 120B at 30s+) have time to complete.

…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd · 2026-04-08T08:33:54Z

Split into separate PRs: #322 (TLS fix), #323 (facilitator default), #324 (LiteLLM reliability), #325 (obolup.sh)

…est-pinned) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: ObolNetwork/obol-stack-front-end#322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321)

…est-pinned) (#505) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: https://github.com/ObolNetwork/obol-stack-front-end/pull/322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321) Co-authored-by: bussyjd <bussyjd@users.noreply.github.com>

bussyjd · 2026-04-08T05:20:35Z

Summary

CA certificate bundle: The x402-verifier distroless image has no CA store. obol sell pricing now populates the ca-certificates ConfigMap from the host's cert bundle so TLS verification of facilitator.x402.rs works out of the box.
Missing Description field: The facilitator rejects verify requests without description in PaymentRequirement. Added it from the route pattern.

Validated on Base Sepolia testnet

Real x402 payment flow between two ARM64 nodes running Nemotron 120B (120B parameter model, tensor-parallel across two GPUs):

Alice (seller): obolup.sh → obol stack up → obol sell pricing → obol sell http nemotron → tunnel
Bob (buyer): discover .well-known → probe 402 → sign EIP-712 → pay → HTTP 200 + real inference

On-chain receipt

Field	Value
Settlement tx	`0xd769953bab675ab97a5140dbf7b5583367c788ecb445251c19fea7194c231ec0`
Network	Base Sepolia (84532)
Facilitator	`facilitator.x402.rs` (real public settlement)
Amount	1000 micro-USDC (0.001 USDC)
Seller balance delta	+0.001 USDC
Buyer balance delta	-0.001 USDC

Test plan

go build ./... compiles
go test ./internal/x402/... passes
Real Base Sepolia payment: 402 → sign → 200 + on-chain settlement
Inference from Nemotron 120B served through the paid route

Two fixes validated with real Base Sepolia x402 payments between two DGX Spark nodes running Nemotron 120B inference. 1. **CA certificate bundle**: The x402-verifier runs in a distroless container with no CA store. TLS verification of the public facilitator (facilitator.x402.rs) fails with "x509: certificate signed by unknown authority". Fix: `obol sell pricing` now reads the host CA bundle and patches it into the `ca-certificates` ConfigMap mounted by the verifier. 2. **Missing Description field**: The facilitator rejects verify requests that lack a `description` field in PaymentRequirement with "invalid_format". Fix: populate Description from the route pattern when building the payment requirement. ## Validated testnet flow ### Alice (seller) ``` obolup.sh # bootstrap dependencies obol stack init && obol stack up obol model setup custom --name nemotron-120b \ --endpoint http://host.k3d.internal:8000/v1 \ --model "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" obol sell pricing --wallet 0xC0De...97E --chain base-sepolia obol sell http nemotron \ --wallet 0xC0De...97E --chain base-sepolia \ --per-request 0.001 --namespace llm \ --upstream litellm --port 4000 \ --health-path /health/readiness \ --register --register-name "Nemotron 120B on DGX Spark" obol tunnel restart ``` ### Bob (buyer) ``` # 1. Discover curl $TUNNEL/.well-known/agent-registration.json # → name: "Nemotron 120B on DGX Spark", x402Support: true # 2. Probe curl -X POST $TUNNEL/services/nemotron/v1/chat/completions # → 402: payTo=0xC0De...97E, amount=1000, network=base-sepolia # 3. Sign EIP-712 TransferWithAuthorization + pay python3 bob_buy.py # → 200: "The meaning of life is to discover and pursue purpose" ``` ### On-chain receipts (Base Sepolia) | Tx | Description | |----|-------------| | 0xd769953b...c231ec0 | x402 settlement: Bob→Alice 0.001 USDC via ERC-3009 | Balance change: Alice +0.001 USDC, Bob -0.001 USDC. Facilitator: https://facilitator.x402.rs (real public settlement).

Replace the third-party facilitator.x402.rs with the Obol-operated facilitator at x402.gcp.obol.tech. This gives us control over uptime, chain support, and monitoring (Grafana dashboards already deployed in obol-infrastructure). Introduces DefaultFacilitatorURL constant in internal/x402 and updates all references: CLI flag default, config loader, standalone inference gateway, and deployment store. Companion PR in obol-infrastructure adds Base Sepolia (84532) to the facilitator's chain config alongside Base Mainnet (8453).

Address #321 — LiteLLM reliability improvements: 1. Hot-add models via /model/new API instead of restarting the deployment. ConfigMap still patched for persistence. Restart only triggered when API keys change (Secret mount requires it). 2. Scale to 2 replicas with RollingUpdate (maxUnavailable: 0, maxSurge: 1) so a new pod is ready before any old pod terminates. 3. PodDisruptionBudget (minAvailable: 1) prevents both replicas from being down simultaneously during voluntary disruptions. 4. preStop hook (sleep 10) gives EndpointSlice time to deregister the terminating pod before SIGTERM — prevents in-flight request drops during rolling updates. 5. Reloader annotation on litellm-secrets — Stakater Reloader triggers rolling restart on API key rotation, no manual restart. 6. terminationGracePeriodSeconds: 60 — long inference requests (e.g. Nemotron 120B at 30s+) have time to complete.

…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd · 2026-04-08T08:33:54Z

Split into separate PRs: #322 (TLS fix), #323 (facilitator default), #324 (LiteLLM reliability), #325 (obolup.sh)

…est-pinned) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: ObolNetwork/obol-stack-front-end#322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321)

…est-pinned) (#505) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: https://github.com/ObolNetwork/obol-stack-front-end/pull/322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321) Co-authored-by: bussyjd <bussyjd@users.noreply.github.com>

bussyjd · 2026-04-08T05:20:35Z

Summary

CA certificate bundle: The x402-verifier distroless image has no CA store. obol sell pricing now populates the ca-certificates ConfigMap from the host's cert bundle so TLS verification of facilitator.x402.rs works out of the box.
Missing Description field: The facilitator rejects verify requests without description in PaymentRequirement. Added it from the route pattern.

Validated on Base Sepolia testnet

Real x402 payment flow between two ARM64 nodes running Nemotron 120B (120B parameter model, tensor-parallel across two GPUs):

Alice (seller): obolup.sh → obol stack up → obol sell pricing → obol sell http nemotron → tunnel
Bob (buyer): discover .well-known → probe 402 → sign EIP-712 → pay → HTTP 200 + real inference

On-chain receipt

Field	Value
Settlement tx	`0xd769953bab675ab97a5140dbf7b5583367c788ecb445251c19fea7194c231ec0`
Network	Base Sepolia (84532)
Facilitator	`facilitator.x402.rs` (real public settlement)
Amount	1000 micro-USDC (0.001 USDC)
Seller balance delta	+0.001 USDC
Buyer balance delta	-0.001 USDC

Test plan

go build ./... compiles
go test ./internal/x402/... passes
Real Base Sepolia payment: 402 → sign → 200 + on-chain settlement
Inference from Nemotron 120B served through the paid route

Two fixes validated with real Base Sepolia x402 payments between two DGX Spark nodes running Nemotron 120B inference. 1. **CA certificate bundle**: The x402-verifier runs in a distroless container with no CA store. TLS verification of the public facilitator (facilitator.x402.rs) fails with "x509: certificate signed by unknown authority". Fix: `obol sell pricing` now reads the host CA bundle and patches it into the `ca-certificates` ConfigMap mounted by the verifier. 2. **Missing Description field**: The facilitator rejects verify requests that lack a `description` field in PaymentRequirement with "invalid_format". Fix: populate Description from the route pattern when building the payment requirement. ## Validated testnet flow ### Alice (seller) ``` obolup.sh # bootstrap dependencies obol stack init && obol stack up obol model setup custom --name nemotron-120b \ --endpoint http://host.k3d.internal:8000/v1 \ --model "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" obol sell pricing --wallet 0xC0De...97E --chain base-sepolia obol sell http nemotron \ --wallet 0xC0De...97E --chain base-sepolia \ --per-request 0.001 --namespace llm \ --upstream litellm --port 4000 \ --health-path /health/readiness \ --register --register-name "Nemotron 120B on DGX Spark" obol tunnel restart ``` ### Bob (buyer) ``` # 1. Discover curl $TUNNEL/.well-known/agent-registration.json # → name: "Nemotron 120B on DGX Spark", x402Support: true # 2. Probe curl -X POST $TUNNEL/services/nemotron/v1/chat/completions # → 402: payTo=0xC0De...97E, amount=1000, network=base-sepolia # 3. Sign EIP-712 TransferWithAuthorization + pay python3 bob_buy.py # → 200: "The meaning of life is to discover and pursue purpose" ``` ### On-chain receipts (Base Sepolia) | Tx | Description | |----|-------------| | 0xd769953b...c231ec0 | x402 settlement: Bob→Alice 0.001 USDC via ERC-3009 | Balance change: Alice +0.001 USDC, Bob -0.001 USDC. Facilitator: https://facilitator.x402.rs (real public settlement).

Replace the third-party facilitator.x402.rs with the Obol-operated facilitator at x402.gcp.obol.tech. This gives us control over uptime, chain support, and monitoring (Grafana dashboards already deployed in obol-infrastructure). Introduces DefaultFacilitatorURL constant in internal/x402 and updates all references: CLI flag default, config loader, standalone inference gateway, and deployment store. Companion PR in obol-infrastructure adds Base Sepolia (84532) to the facilitator's chain config alongside Base Mainnet (8453).

Address #321 — LiteLLM reliability improvements: 1. Hot-add models via /model/new API instead of restarting the deployment. ConfigMap still patched for persistence. Restart only triggered when API keys change (Secret mount requires it). 2. Scale to 2 replicas with RollingUpdate (maxUnavailable: 0, maxSurge: 1) so a new pod is ready before any old pod terminates. 3. PodDisruptionBudget (minAvailable: 1) prevents both replicas from being down simultaneously during voluntary disruptions. 4. preStop hook (sleep 10) gives EndpointSlice time to deregister the terminating pod before SIGTERM — prevents in-flight request drops during rolling updates. 5. Reloader annotation on litellm-secrets — Stakater Reloader triggers rolling restart on API key rotation, no manual restart. 6. terminationGracePeriodSeconds: 60 — long inference requests (e.g. Nemotron 120B at 30s+) have time to complete.

…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd · 2026-04-08T08:33:54Z

Split into separate PRs: #322 (TLS fix), #323 (facilitator default), #324 (LiteLLM reliability), #325 (obolup.sh)

…est-pinned) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: ObolNetwork/obol-stack-front-end#322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321)

…est-pinned) (#505) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: https://github.com/ObolNetwork/obol-stack-front-end/pull/322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321) Co-authored-by: bussyjd <bussyjd@users.noreply.github.com>

bussyjd · 2026-04-08T05:20:35Z

Summary

CA certificate bundle: The x402-verifier distroless image has no CA store. obol sell pricing now populates the ca-certificates ConfigMap from the host's cert bundle so TLS verification of facilitator.x402.rs works out of the box.
Missing Description field: The facilitator rejects verify requests without description in PaymentRequirement. Added it from the route pattern.

Validated on Base Sepolia testnet

Real x402 payment flow between two ARM64 nodes running Nemotron 120B (120B parameter model, tensor-parallel across two GPUs):

Alice (seller): obolup.sh → obol stack up → obol sell pricing → obol sell http nemotron → tunnel
Bob (buyer): discover .well-known → probe 402 → sign EIP-712 → pay → HTTP 200 + real inference

On-chain receipt

Field	Value
Settlement tx	`0xd769953bab675ab97a5140dbf7b5583367c788ecb445251c19fea7194c231ec0`
Network	Base Sepolia (84532)
Facilitator	`facilitator.x402.rs` (real public settlement)
Amount	1000 micro-USDC (0.001 USDC)
Seller balance delta	+0.001 USDC
Buyer balance delta	-0.001 USDC

Test plan

go build ./... compiles
go test ./internal/x402/... passes
Real Base Sepolia payment: 402 → sign → 200 + on-chain settlement
Inference from Nemotron 120B served through the paid route

Two fixes validated with real Base Sepolia x402 payments between two DGX Spark nodes running Nemotron 120B inference. 1. **CA certificate bundle**: The x402-verifier runs in a distroless container with no CA store. TLS verification of the public facilitator (facilitator.x402.rs) fails with "x509: certificate signed by unknown authority". Fix: `obol sell pricing` now reads the host CA bundle and patches it into the `ca-certificates` ConfigMap mounted by the verifier. 2. **Missing Description field**: The facilitator rejects verify requests that lack a `description` field in PaymentRequirement with "invalid_format". Fix: populate Description from the route pattern when building the payment requirement. ## Validated testnet flow ### Alice (seller) ``` obolup.sh # bootstrap dependencies obol stack init && obol stack up obol model setup custom --name nemotron-120b \ --endpoint http://host.k3d.internal:8000/v1 \ --model "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" obol sell pricing --wallet 0xC0De...97E --chain base-sepolia obol sell http nemotron \ --wallet 0xC0De...97E --chain base-sepolia \ --per-request 0.001 --namespace llm \ --upstream litellm --port 4000 \ --health-path /health/readiness \ --register --register-name "Nemotron 120B on DGX Spark" obol tunnel restart ``` ### Bob (buyer) ``` # 1. Discover curl $TUNNEL/.well-known/agent-registration.json # → name: "Nemotron 120B on DGX Spark", x402Support: true # 2. Probe curl -X POST $TUNNEL/services/nemotron/v1/chat/completions # → 402: payTo=0xC0De...97E, amount=1000, network=base-sepolia # 3. Sign EIP-712 TransferWithAuthorization + pay python3 bob_buy.py # → 200: "The meaning of life is to discover and pursue purpose" ``` ### On-chain receipts (Base Sepolia) | Tx | Description | |----|-------------| | 0xd769953b...c231ec0 | x402 settlement: Bob→Alice 0.001 USDC via ERC-3009 | Balance change: Alice +0.001 USDC, Bob -0.001 USDC. Facilitator: https://facilitator.x402.rs (real public settlement).

Replace the third-party facilitator.x402.rs with the Obol-operated facilitator at x402.gcp.obol.tech. This gives us control over uptime, chain support, and monitoring (Grafana dashboards already deployed in obol-infrastructure). Introduces DefaultFacilitatorURL constant in internal/x402 and updates all references: CLI flag default, config loader, standalone inference gateway, and deployment store. Companion PR in obol-infrastructure adds Base Sepolia (84532) to the facilitator's chain config alongside Base Mainnet (8453).

Address #321 — LiteLLM reliability improvements: 1. Hot-add models via /model/new API instead of restarting the deployment. ConfigMap still patched for persistence. Restart only triggered when API keys change (Secret mount requires it). 2. Scale to 2 replicas with RollingUpdate (maxUnavailable: 0, maxSurge: 1) so a new pod is ready before any old pod terminates. 3. PodDisruptionBudget (minAvailable: 1) prevents both replicas from being down simultaneously during voluntary disruptions. 4. preStop hook (sleep 10) gives EndpointSlice time to deregister the terminating pod before SIGTERM — prevents in-flight request drops during rolling updates. 5. Reloader annotation on litellm-secrets — Stakater Reloader triggers rolling restart on API key rotation, no manual restart. 6. terminationGracePeriodSeconds: 60 — long inference requests (e.g. Nemotron 120B at 30s+) have time to complete.

…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd · 2026-04-08T08:33:54Z

Split into separate PRs: #322 (TLS fix), #323 (facilitator default), #324 (LiteLLM reliability), #325 (obolup.sh)

…est-pinned) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: ObolNetwork/obol-stack-front-end#322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321)

…est-pinned) (#505) Promotes the frontend pin to the v0.1.25-rc1 release candidate, which combines 10 Dependabot bumps integrated via PR #322 on the frontend repo (all 8 CI checks green on the integration branch). Multi-arch index digest: sha256:e7b38ca43771c29475d6831dbee53adb5d2685137ecb7d5878c82e4ecebee92a Per-platform digests: linux/amd64 sha256:2d39e666dd7f807e4a6b56e7f7f509cd96b8427fcabe413bbd9000da55f2cf55 linux/arm64 sha256:6afb996dd6c21e68d714f4aa1ef0c51b2a4553e24d72629529e053e611d889b3 Frontend release: https://github.com/ObolNetwork/obol-stack-front-end/releases/tag/v0.1.25-rc1 Frontend integration PR: https://github.com/ObolNetwork/obol-stack-front-end/pull/322 Dep bumps included: @typescript-eslint/eslint-plugin 8.59.1 → 8.59.2 (#312) @tanstack/react-query 5.100.9 → 5.100.10 (#313) @playwright/test 1.59.1 → 1.60.0 (#314) thread-stream 4.0.0 → 4.2.0 (#315) @copilotkit/runtime 1.56.3 → 1.57.1 (#316) @types/node 25.6.2 → 25.9.0 (#317) viem 2.48.11 → 2.49.3 (#318) lint-staged 16.4.0 → 17.0.5 (#319) react-dom 19.2.5 → 19.2.6 (#320) dompurify 3.4.2 → 3.4.5 (#321) Co-authored-by: bussyjd <bussyjd@users.noreply.github.com>

bussyjd added 2 commits April 8, 2026 14:20

bussyjd mentioned this pull request Apr 8, 2026
LiteLLM reliability: hot-reload, zero-downtime restarts, and single-replica fragility #321
Closed

github-advanced-securityBot found potential problems Apr 8, 2026
View reviewed changes

Comment threadinternal/model/model.go Fixed

Potential fix for pull request finding 'CodeQL / Potentially unsafe q…
9706aea
…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd closed this Apr 8, 2026

OisinKyne deleted the fix/x402-facilitator-compat branch April 23, 2026 19:26

bussyjd mentioned this pull request May 19, 2026
chore(frontend): bump obol-stack-front-end v0.1.24 → v0.1.25-rc1 (digest-pinned) #505
Merged
5 tasks

bussyjd mentioned this pull request Jul 6, 2026
fix(llm): zero-downtime LiteLLM config operations — hot-path Reloader stance + x402-buyer split (#321) #707
Closed

bussyjd added 2 commits April 8, 2026 14:20

bussyjd mentioned this pull request Apr 8, 2026
LiteLLM reliability: hot-reload, zero-downtime restarts, and single-replica fragility #321
Closed

github-advanced-securityBot found potential problems Apr 8, 2026
View reviewed changes

Comment threadinternal/model/model.go Fixed

Potential fix for pull request finding 'CodeQL / Potentially unsafe q…
9706aea
…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd closed this Apr 8, 2026

OisinKyne deleted the fix/x402-facilitator-compat branch April 23, 2026 19:26

bussyjd mentioned this pull request May 19, 2026
chore(frontend): bump obol-stack-front-end v0.1.24 → v0.1.25-rc1 (digest-pinned) #505
Merged
5 tasks

bussyjd mentioned this pull request Jul 6, 2026
fix(llm): zero-downtime LiteLLM config operations — hot-path Reloader stance + x402-buyer split (#321) #707
Closed

bussyjd added 2 commits April 8, 2026 14:20

bussyjd mentioned this pull request Apr 8, 2026
LiteLLM reliability: hot-reload, zero-downtime restarts, and single-replica fragility #321
Closed

github-advanced-securityBot found potential problems Apr 8, 2026
View reviewed changes

Comment threadinternal/model/model.go Fixed

Potential fix for pull request finding 'CodeQL / Potentially unsafe q…
9706aea
…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd closed this Apr 8, 2026

OisinKyne deleted the fix/x402-facilitator-compat branch April 23, 2026 19:26

bussyjd mentioned this pull request May 19, 2026
chore(frontend): bump obol-stack-front-end v0.1.24 → v0.1.25-rc1 (digest-pinned) #505
Merged
5 tasks

bussyjd mentioned this pull request Jul 6, 2026
fix(llm): zero-downtime LiteLLM config operations — hot-path Reloader stance + x402-buyer split (#321) #707
Closed

bussyjd added 2 commits April 8, 2026 14:20

bussyjd mentioned this pull request Apr 8, 2026
LiteLLM reliability: hot-reload, zero-downtime restarts, and single-replica fragility #321
Closed

github-advanced-securityBot found potential problems Apr 8, 2026
View reviewed changes

Comment threadinternal/model/model.go Fixed

Potential fix for pull request finding 'CodeQL / Potentially unsafe q…
9706aea
…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd closed this Apr 8, 2026

OisinKyne deleted the fix/x402-facilitator-compat branch April 23, 2026 19:26

bussyjd mentioned this pull request May 19, 2026
chore(frontend): bump obol-stack-front-end v0.1.24 → v0.1.25-rc1 (digest-pinned) #505
Merged
5 tasks

bussyjd mentioned this pull request Jul 6, 2026
fix(llm): zero-downtime LiteLLM config operations — hot-path Reloader stance + x402-buyer split (#321) #707
Closed

bussyjd added 2 commits April 8, 2026 14:20

bussyjd mentioned this pull request Apr 8, 2026
LiteLLM reliability: hot-reload, zero-downtime restarts, and single-replica fragility #321
Closed

github-advanced-securityBot found potential problems Apr 8, 2026
View reviewed changes

Comment threadinternal/model/model.go Fixed

Potential fix for pull request finding 'CodeQL / Potentially unsafe q…
9706aea
…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd closed this Apr 8, 2026

OisinKyne deleted the fix/x402-facilitator-compat branch April 23, 2026 19:26

bussyjd mentioned this pull request May 19, 2026
chore(frontend): bump obol-stack-front-end v0.1.24 → v0.1.25-rc1 (digest-pinned) #505
Merged
5 tasks

bussyjd mentioned this pull request Jul 6, 2026
fix(llm): zero-downtime LiteLLM config operations — hot-path Reloader stance + x402-buyer split (#321) #707
Closed

bussyjd added 2 commits April 8, 2026 14:20

bussyjd mentioned this pull request Apr 8, 2026
LiteLLM reliability: hot-reload, zero-downtime restarts, and single-replica fragility #321
Closed

github-advanced-securityBot found potential problems Apr 8, 2026
View reviewed changes

Comment threadinternal/model/model.go Fixed

Potential fix for pull request finding 'CodeQL / Potentially unsafe q…
9706aea
…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd closed this Apr 8, 2026

OisinKyne deleted the fix/x402-facilitator-compat branch April 23, 2026 19:26

bussyjd mentioned this pull request May 19, 2026
chore(frontend): bump obol-stack-front-end v0.1.24 → v0.1.25-rc1 (digest-pinned) #505
Merged
5 tasks

bussyjd mentioned this pull request Jul 6, 2026
fix(llm): zero-downtime LiteLLM config operations — hot-path Reloader stance + x402-buyer split (#321) #707
Closed

bussyjd added 2 commits April 8, 2026 14:20

bussyjd mentioned this pull request Apr 8, 2026
LiteLLM reliability: hot-reload, zero-downtime restarts, and single-replica fragility #321
Closed

github-advanced-securityBot found potential problems Apr 8, 2026
View reviewed changes

Comment threadinternal/model/model.go Fixed

Potential fix for pull request finding 'CodeQL / Potentially unsafe q…
9706aea
…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd closed this Apr 8, 2026

OisinKyne deleted the fix/x402-facilitator-compat branch April 23, 2026 19:26

bussyjd mentioned this pull request May 19, 2026
chore(frontend): bump obol-stack-front-end v0.1.24 → v0.1.25-rc1 (digest-pinned) #505
Merged
5 tasks

bussyjd mentioned this pull request Jul 6, 2026
fix(llm): zero-downtime LiteLLM config operations — hot-path Reloader stance + x402-buyer split (#321) #707
Closed

bussyjd added 2 commits April 8, 2026 14:20

bussyjd mentioned this pull request Apr 8, 2026
LiteLLM reliability: hot-reload, zero-downtime restarts, and single-replica fragility #321
Closed

github-advanced-securityBot found potential problems Apr 8, 2026
View reviewed changes

Comment threadinternal/model/model.go Fixed

Potential fix for pull request finding 'CodeQL / Potentially unsafe q…
9706aea
…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>

bussyjd closed this Apr 8, 2026

OisinKyne deleted the fix/x402-facilitator-compat branch April 23, 2026 19:26

bussyjd mentioned this pull request May 19, 2026
chore(frontend): bump obol-stack-front-end v0.1.24 → v0.1.25-rc1 (digest-pinned) #505
Merged
5 tasks

bussyjd mentioned this pull request Jul 6, 2026
fix(llm): zero-downtime LiteLLM config operations — hot-path Reloader stance + x402-buyer split (#321) #707
Closed

Uh oh!

Conversation

bussyjd commented Apr 8, 2026

Summary

Validated on Base Sepolia testnet

On-chain receipt

Test plan

Uh oh!

Uh oh!

bussyjd commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

bussyjd commented Apr 8, 2026

Summary

Validated on Base Sepolia testnet

On-chain receipt

Test plan

Uh oh!

Uh oh!

bussyjd commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

bussyjd commented Apr 8, 2026

Summary

Validated on Base Sepolia testnet

On-chain receipt

Test plan

Uh oh!

Uh oh!

bussyjd commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

bussyjd commented Apr 8, 2026

Summary

Validated on Base Sepolia testnet

On-chain receipt

Test plan

Uh oh!

Uh oh!

bussyjd commented Apr 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

bussyjd commented Apr 8, 2026

Summary