Uh oh!
There was an error while loading. Please reload this page.
feat: LiteLLM zero-downtime, Obol facilitator default, x402 TLS fix - #324
Closed
bussyjd wants to merge 5 commits into
Closed
feat: LiteLLM zero-downtime, Obol facilitator default, x402 TLS fix#324bussyjd wants to merge 5 commits into
bussyjd wants to merge 5 commits into
Conversation
added 3 commits
April 8, 2026 17:31
Two fixes validated with real Base Sepolia x402 payments between two DGX Spark nodes running Nemotron 120B inference. 1. **CA certificate bundle**: The x402-verifier runs in a distroless container with no CA store. TLS verification of the public facilitator (facilitator.x402.rs) fails with "x509: certificate signed by unknown authority". Fix: `obol sell pricing` now reads the host CA bundle and patches it into the `ca-certificates` ConfigMap mounted by the verifier. 2. **Missing Description field**: The facilitator rejects verify requests that lack a `description` field in PaymentRequirement with "invalid_format". Fix: populate Description from the route pattern when building the payment requirement. ## Validated testnet flow ### Alice (seller) ``` obolup.sh # bootstrap dependencies obol stack init && obol stack up obol model setup custom --name nemotron-120b \ --endpoint http://host.k3d.internal:8000/v1 \ --model "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" obol sell pricing --wallet 0xC0De...97E --chain base-sepolia obol sell http nemotron \ --wallet 0xC0De...97E --chain base-sepolia \ --per-request 0.001 --namespace llm \ --upstream litellm --port 4000 \ --health-path /health/readiness \ --register --register-name "Nemotron 120B on DGX Spark" obol tunnel restart ``` ### Bob (buyer) ``` # 1. Discover curl $TUNNEL/.well-known/agent-registration.json # → name: "Nemotron 120B on DGX Spark", x402Support: true # 2. Probe curl -X POST $TUNNEL/services/nemotron/v1/chat/completions # → 402: payTo=0xC0De...97E, amount=1000, network=base-sepolia # 3. Sign EIP-712 TransferWithAuthorization + pay python3 bob_buy.py # → 200: "The meaning of life is to discover and pursue purpose" ``` ### On-chain receipts (Base Sepolia) | Tx | Description | |----|-------------| | 0xd769953b...c231ec0 | x402 settlement: Bob→Alice 0.001 USDC via ERC-3009 | Balance change: Alice +0.001 USDC, Bob -0.001 USDC. Facilitator: https://facilitator.x402.rs (real public settlement).
Replace the third-party facilitator.x402.rs with the Obol-operated facilitator at x402.gcp.obol.tech. This gives us control over uptime, chain support, and monitoring (Grafana dashboards already deployed in obol-infrastructure). Introduces DefaultFacilitatorURL constant in internal/x402 and updates all references: CLI flag default, config loader, standalone inference gateway, and deployment store. Companion PR in obol-infrastructure adds Base Sepolia (84532) to the facilitator's chain config alongside Base Mainnet (8453).
Address #321 — LiteLLM reliability improvements: 1. Hot-add models via /model/new API instead of restarting the deployment. ConfigMap still patched for persistence. Restart only triggered when API keys change (Secret mount requires it). 2. Scale to 2 replicas with RollingUpdate (maxUnavailable: 0, maxSurge: 1) so a new pod is ready before any old pod terminates. 3. PodDisruptionBudget (minAvailable: 1) prevents both replicas from being down simultaneously during voluntary disruptions. 4. preStop hook (sleep 10) gives EndpointSlice time to deregister the terminating pod before SIGTERM — prevents in-flight request drops during rolling updates. 5. Reloader annotation on litellm-secrets — Stakater Reloader triggers rolling restart on API key rotation, no manual restart. 6. terminationGracePeriodSeconds: 60 — long inference requests (e.g. Nemotron 120B at 30s+) have time to complete.
4 tasks
Uh oh!
There was an error while loading. Please reload this page.
…uoting' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Signed-off-by: JeanDaniel Bussy <SilverSurfer972@gmail.com>
This was referenced Apr 8, 2026
- model_hotadd_test.go: test ConfigureLiteLLM branching (API key →
restart vs empty → hot-add), payload construction, multiple entries,
and mock server acceptance of /model/new requests
- setup_ca_test.go: test CA bundle patch JSON structure, no-bundle skip,
and empty-file skip
- verifier_test.go: assert Description field ("Payment required for ...")
is present in 402 response bodybussyjd
commented
Apr 9, 2026
ContributorAuthor
Superseded by the validated integration branch \ and the \ prerelease cut from it. The release-candidate branch now carries the tested sell → discover → buy → settle path, updated docs/skills, and the final x402/buy-side fixes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Consolidates #322 (x402 verifier TLS) and #323 (Obol facilitator default) into a single PR with LiteLLM zero-downtime deployment. Three improvements:
1. x402 verifier TLS fix
The distroless x402-verifier image ships without a CA store, causing TLS verification of external HTTPS facilitators to fail.
populateCABundle()reads the host's CA bundle and patches it into theca-certificatesConfigMap in the x402 namespace.2. Obol-operated facilitator default
Switches the default facilitator URL from
facilitator.x402.rstox402.gcp.obol.techvia a singleDefaultFacilitatorURLconstant. All references consolidated.3. LiteLLM zero-downtime config
hotAddModels()calls/model/newAPI via kubectl exec — no restart needed when only models change (no API key update)maxUnavailable: 0, PodDisruptionBudget (minAvailable: 1), preStop hook with 10s drainTest coverage
model_hotadd_test.go(NEW): ConfigureLiteLLM branching (API key -> restart, empty -> hot-add), payload construction, multiple entries, mock server /model/new acceptancesetup_ca_test.go(NEW): CA bundle patch JSON structure, no-bundle skip, empty-file skipverifier_test.go(UPDATED): Description field assertion in 402 response bodySupersedes
Test plan
go build ./...passesgo test ./internal/model/passes (including new hotAddModels tests)go test ./internal/x402/passes (including CA bundle + Description tests)