fix(model): register discovered endpoints with /v1 api_base - #745

Closed
bussyjd wants to merge 2 commits into
mainfrom
fix/discovery-api-base-v1
Closed

fix(model): register discovered endpoints with /v1 api_base#745
bussyjd wants to merge 2 commits into
mainfrom
fix/discovery-api-base-v1

Conversation

@bussyjd

@bussyjdbussyjd commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Local-server auto-discovery (buildDiscoveredProvider, since v0.10.0 69f25bf) registers OpenAI-compatible endpoints with a bare api_base. LiteLLM's OpenAI provider does not append /v1 (CLAUDE.md pitfall 6), so requests through the discovered entry hit POST /chat/completions → vLLM 404 {'detail':'Not Found'}. When an explicit obol model setup custom --endpoint .../v1 entry shares the model group, LiteLLM shuffles ~50/50 between the good and poisoned deployment.

Evidence (v0.13.0-rc4 release-smoke on spark1): vLLM access log 06:12:15 POST /chat/completions 404 → retry 06:12:17 POST /v1/chat/completions 200 (flow-03 step[2] flake), and 06:25:38 POST /chat/completions 404 matching Alice's verifier handler returned 404, skipping settlement (flow-11 step[43] hard fail — the only red step in an otherwise green run). Reproduced by curl: /v1/chat/completions → 200, /chat/completions → 404.

Not an rc4 regression — latent since v0.10.0, exposed on QA hosts where vLLM sits on the discovery-probed port with served-model-name == OBOL_LLM_MODEL. Blocks a green release-smoke, so it should ride in v0.13.0.

Companion: #744 (flow-03 retry + surface HTTP status).

https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk

@bussyjd
fix(model): register discovered endpoints with /v1 api_base
171731b
Local-server auto-discovery registered OpenAI-compatible endpoints
(vLLM on :8000) with a bare api_base. LiteLLM's OpenAI provider does
not append /v1 (CLAUDE.md pitfall 6), so every request through the
discovered entry hit POST /chat/completions and 404'd. When an explicit
'obol model setup custom --endpoint .../v1' entry shared the same model
group, requests coin-flipped between the good and bad deployment —
root cause of the intermittent release-smoke flow-11 step[43] 404 and
the flow-03 step[2] flake (latent since v0.10.0, 69f25bf).
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
bussyjdforce-pushed the fix/discovery-api-base-v1 branch from 78c2880 to 171731bCompareJuly 14, 2026 07:03
@bussyjd

bussyjd commented Jul 14, 2026

Copy link
Copy Markdown
ContributorAuthor

Full /v1 pipeline audit (requested before merge)

Every producer and consumer of OpenAI-compatible endpoint URLs in the repo, verified against source:

#SiteConventionVerdict
1obol model setup custom → AddCustomEndpointWithOptions (model.go:995)endpoint must include /v1; enforced by validation step 2, which probes <endpoint>/chat/completions — a /v1-less vLLM endpoint fails the inference probe and is rejected at setup time✅ correct, self-protecting
2auto-discovery → buildDiscoveredProvider (discover.go)was the only violator: wrote bare api_base. Discovery's own gate (fetchModels, detect.go:282) requires GET <base>/v1/models → 200, so every endpoint it can emit provably serves under /v1 — the +"/v1" append is safe by construction for all discoverable server types (vLLM, llama.cpp, LM Studio)✅ fixed by this PR
3shared chokepoint buildCustomEndpointEntryWithOptions (model.go)writes api_base verbatim; both callers now uphold the invariant✅ invariant now documented on the builder
4WarnAndStripV1Suffix (model.go:1699)dead code (zero callers) whose doc claimed the opposite ("LiteLLM auto-appends /v1") — the fossil that kept re-seeding this confusion✅ deleted in this PR
5llm.yaml paid/* default routeapi_base: http://x402-buyer...:8402/v1with/v1, pitfall documented inline (ffa7cf4 fixed this same class)✅ correct
6per-purchase entries → buyerAPIBase() (purchase_helpers.go:281)http://x402-buyer.<ns>...:8402/v1 — with /v1, plus legacy-base migration✅ correct
7buyer-side URL building (buyprompts.ChatCompletionsURL, buy/discover.go:567)opposite convention — base without/v1, appends /v1/chat/completions itself; internally consistent and documented as lockstep with the verifier✅ correct (different but coherent convention)
8seller-side verifier normalizeChatCompletionsPath (verifier.go:857)tolerantly rewrites "", /v1, /chat/completions → /v1/chat/completions for inference/agent offers✅ correct
9hermes agent renderer + openclawhardcoded http://litellm...:4000/v1✅ correct
10Ollama pathseparate ollama/ provider (native API, no /v1 semantics) — explicitly excluded from discovery (shouldRegisterEndpoint)✅ not applicable

Root contract, now written on the chokepoint: LiteLLM's openai/ provider sends to <api_base>/chat/completions verbatim and never appends /v1 — so every LiteLLM api_base must carry /v1, while buyer-tooling bases must not (they append the full path).

Known residual gap (out of scope, tracked): the config-drift checker compares model names only, so a same-name entry with a divergent api_base (exactly this incident's poisoned group) is invisible to it.

@bussyjd
bussyjd enabled auto-merge (squash) July 14, 2026 07:15
This was referenced Jul 14, 2026
Open
Closed
@bussyjdbussyjd mentioned this pull request Jul 14, 2026
Open
bussyjd added a commit that referenced this pull request Jul 14, 2026
@bussyjd
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
49be12f
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjdbussyjd mentioned this pull request Jul 14, 2026
Merged
4 tasks
@bussyjd

bussyjd commented Jul 14, 2026

Copy link
Copy Markdown
ContributorAuthor

Superseded by #749 (integration/v0.13.0-rc4) — this branch is merged into the integration branch verbatim (merge commits a02b62f/7d963125) and ships there. Validated by the fully green release smoke rc4-run7-20260714-171005 (18/18 flows incl. flow-13/14, on-chain receipts in #749).

@bussyjdbussyjd closed this Jul 14, 2026
auto-merge was automatically disabled July 14, 2026 10:50

Pull request was closed

OisinKyne pushed a commit that referenced this pull request Jul 14, 2026
@bussyjd@OisinKyne
chore(model): drop dead HasProviderConfigured
cb960cb
Zero callers anywhere (including tests) — flagged by the /v1 pipeline
audit's dead-code sweep alongside WarnAndStripV1Suffix (removed in
#745). Other deadcode-tool hits in internal/{buy,inference} are
test-covered exported API and were left alone.
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
OisinKyne pushed a commit that referenced this pull request Jul 14, 2026
@bussyjd@OisinKyne
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
1c5bb04
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bussyjd
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
fix(model): register discovered endpoints with /v1 api_base - #745
Closed
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Closed
fix(model): register discovered endpoints with /v1 api_base#745
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Conversation
@bussyjd
@bussyjdbussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

Contributor
Local-server auto-discovery (buildDiscoveredProvider, since v0.10.0 69f25bf) registers OpenAI-compatible endpoints with a bare api_base. LiteLLM's OpenAI provider does not append /v1 (CLAUDE.md pitfall 6), so requests through the discovered entry hit POST /chat/completions → vLLM 404 {'detail':'Not Found'}. When an explicit obol model setup custom --endpoint .../v1 entry shares the model group, LiteLLM shuffles ~50/50 between the good and poisoned deployment.
Evidence (v0.13.0-rc4 release-smoke on spark1): vLLM access log 06:12:15 POST /chat/completions 404 → retry 06:12:17 POST /v1/chat/completions 200 (flow-03 step[2] flake), and 06:25:38 POST /chat/completions 404 matching Alice's verifier handler returned 404, skipping settlement (flow-11 step[43] hard fail — the only red step in an otherwise green run). Reproduced by curl: /v1/chat/completions → 200, /chat/completions → 404.
Not an rc4 regression — latent since v0.10.0, exposed on QA hosts where vLLM sits on the discovery-probed port with served-model-name == OBOL_LLM_MODEL. Blocks a green release-smoke, so it should ride in v0.13.0.
Companion: #744 (flow-03 retry + surface HTTP status).
https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
fix(model): register discovered endpoints with /v1 api_base
171731b
Local-server auto-discovery registered OpenAI-compatible endpoints
(vLLM on :8000) with a bare api_base. LiteLLM's OpenAI provider does
not append /v1 (CLAUDE.md pitfall 6), so every request through the
discovered entry hit POST /chat/completions and 404'd. When an explicit
'obol model setup custom --endpoint .../v1' entry shared the same model
group, requests coin-flipped between the good and bad deployment —
root cause of the intermittent release-smoke flow-11 step[43] 404 and
the flow-03 step[2] flake (latent since v0.10.0, 69f25bf).
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
bussyjdforce-pushed
the
fix/discovery-api-base-v1
branch
from
78c2880 to
171731bCompareJuly 14, 2026 07:03
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Full /v1 pipeline audit (requested before merge)
Every producer and consumer of OpenAI-compatible endpoint URLs in the repo, verified against source:
#SiteConventionVerdict
1obol model setup custom → AddCustomEndpointWithOptions (model.go:995)endpoint must include /v1; enforced by validation step 2, which probes <endpoint>/chat/completions — a /v1-less vLLM endpoint fails the inference probe and is rejected at setup time✅ correct, self-protecting
2auto-discovery → buildDiscoveredProvider (discover.go)was the only violator: wrote bare api_base. Discovery's own gate (fetchModels, detect.go:282) requires GET <base>/v1/models → 200, so every endpoint it can emit provably serves under /v1 — the +"/v1" append is safe by construction for all discoverable server types (vLLM, llama.cpp, LM Studio)✅ fixed by this PR
3shared chokepoint buildCustomEndpointEntryWithOptions (model.go)writes api_base verbatim; both callers now uphold the invariant✅ invariant now documented on the builder
4WarnAndStripV1Suffix (model.go:1699)dead code (zero callers) whose doc claimed the opposite ("LiteLLM auto-appends /v1") — the fossil that kept re-seeding this confusion✅ deleted in this PR
5llm.yaml paid/* default routeapi_base: http://x402-buyer...:8402/v1with/v1, pitfall documented inline (ffa7cf4 fixed this same class)✅ correct
6per-purchase entries → buyerAPIBase() (purchase_helpers.go:281)http://x402-buyer.<ns>...:8402/v1 — with /v1, plus legacy-base migration✅ correct
7buyer-side URL building (buyprompts.ChatCompletionsURL, buy/discover.go:567)opposite convention — base without/v1, appends /v1/chat/completions itself; internally consistent and documented as lockstep with the verifier✅ correct (different but coherent convention)
8seller-side verifier normalizeChatCompletionsPath (verifier.go:857)tolerantly rewrites "", /v1, /chat/completions → /v1/chat/completions for inference/agent offers✅ correct
9hermes agent renderer + openclawhardcoded http://litellm...:4000/v1✅ correct
10Ollama pathseparate ollama/ provider (native API, no /v1 semantics) — explicitly excluded from discovery (shouldRegisterEndpoint)✅ not applicable
Root contract, now written on the chokepoint: LiteLLM's openai/ provider sends to <api_base>/chat/completions verbatim and never appends /v1 — so every LiteLLM api_base must carry /v1, while buyer-tooling bases must not (they append the full path).
Known residual gap (out of scope, tracked): the config-drift checker compares model names only, so a same-name entry with a divergent api_base (exactly this incident's poisoned group) is invisible to it.
@bussyjd
bussyjd
enabled auto-merge (squash)
July 14, 2026 07:15

This was referenced Jul 14, 2026

Open

 Closed

@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026

Open

bussyjd added a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
49be12f
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026
 Merged

4 tasks
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #749 (integration/v0.13.0-rc4) — this branch is merged into the integration branch verbatim (merge commits a02b62f/7d963125) and ships there. Validated by the fully green release smoke rc4-run7-20260714-171005 (18/18 flows incl. flow-13/14, on-chain receipts in #749).
@bussyjdbussyjd
closed this
Jul 14, 2026

auto-merge was automatically disabled
July 14, 2026 10:50
Pull request was closed
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
chore(model): drop dead HasProviderConfigured
cb960cb
Zero callers anywhere (including tests) — flagged by the /v1 pipeline
audit's dead-code sweep alongside WarnAndStripV1Suffix (removed in
#745). Other deadcode-tool hits in internal/{buy,inference} are
test-covered exported API and were left alone.
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
1c5bb04
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix(model): register discovered endpoints with /v1 api_base - #745
Closed
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Closed
fix(model): register discovered endpoints with /v1 api_base#745
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Conversation
@bussyjd
@bussyjdbussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

Contributor
Local-server auto-discovery (buildDiscoveredProvider, since v0.10.0 69f25bf) registers OpenAI-compatible endpoints with a bare api_base. LiteLLM's OpenAI provider does not append /v1 (CLAUDE.md pitfall 6), so requests through the discovered entry hit POST /chat/completions → vLLM 404 {'detail':'Not Found'}. When an explicit obol model setup custom --endpoint .../v1 entry shares the model group, LiteLLM shuffles ~50/50 between the good and poisoned deployment.
Evidence (v0.13.0-rc4 release-smoke on spark1): vLLM access log 06:12:15 POST /chat/completions 404 → retry 06:12:17 POST /v1/chat/completions 200 (flow-03 step[2] flake), and 06:25:38 POST /chat/completions 404 matching Alice's verifier handler returned 404, skipping settlement (flow-11 step[43] hard fail — the only red step in an otherwise green run). Reproduced by curl: /v1/chat/completions → 200, /chat/completions → 404.
Not an rc4 regression — latent since v0.10.0, exposed on QA hosts where vLLM sits on the discovery-probed port with served-model-name == OBOL_LLM_MODEL. Blocks a green release-smoke, so it should ride in v0.13.0.
Companion: #744 (flow-03 retry + surface HTTP status).
https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
fix(model): register discovered endpoints with /v1 api_base
171731b
Local-server auto-discovery registered OpenAI-compatible endpoints
(vLLM on :8000) with a bare api_base. LiteLLM's OpenAI provider does
not append /v1 (CLAUDE.md pitfall 6), so every request through the
discovered entry hit POST /chat/completions and 404'd. When an explicit
'obol model setup custom --endpoint .../v1' entry shared the same model
group, requests coin-flipped between the good and bad deployment —
root cause of the intermittent release-smoke flow-11 step[43] 404 and
the flow-03 step[2] flake (latent since v0.10.0, 69f25bf).
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
bussyjdforce-pushed
the
fix/discovery-api-base-v1
branch
from
78c2880 to
171731bCompareJuly 14, 2026 07:03
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Full /v1 pipeline audit (requested before merge)
Every producer and consumer of OpenAI-compatible endpoint URLs in the repo, verified against source:
#SiteConventionVerdict
1obol model setup custom → AddCustomEndpointWithOptions (model.go:995)endpoint must include /v1; enforced by validation step 2, which probes <endpoint>/chat/completions — a /v1-less vLLM endpoint fails the inference probe and is rejected at setup time✅ correct, self-protecting
2auto-discovery → buildDiscoveredProvider (discover.go)was the only violator: wrote bare api_base. Discovery's own gate (fetchModels, detect.go:282) requires GET <base>/v1/models → 200, so every endpoint it can emit provably serves under /v1 — the +"/v1" append is safe by construction for all discoverable server types (vLLM, llama.cpp, LM Studio)✅ fixed by this PR
3shared chokepoint buildCustomEndpointEntryWithOptions (model.go)writes api_base verbatim; both callers now uphold the invariant✅ invariant now documented on the builder
4WarnAndStripV1Suffix (model.go:1699)dead code (zero callers) whose doc claimed the opposite ("LiteLLM auto-appends /v1") — the fossil that kept re-seeding this confusion✅ deleted in this PR
5llm.yaml paid/* default routeapi_base: http://x402-buyer...:8402/v1with/v1, pitfall documented inline (ffa7cf4 fixed this same class)✅ correct
6per-purchase entries → buyerAPIBase() (purchase_helpers.go:281)http://x402-buyer.<ns>...:8402/v1 — with /v1, plus legacy-base migration✅ correct
7buyer-side URL building (buyprompts.ChatCompletionsURL, buy/discover.go:567)opposite convention — base without/v1, appends /v1/chat/completions itself; internally consistent and documented as lockstep with the verifier✅ correct (different but coherent convention)
8seller-side verifier normalizeChatCompletionsPath (verifier.go:857)tolerantly rewrites "", /v1, /chat/completions → /v1/chat/completions for inference/agent offers✅ correct
9hermes agent renderer + openclawhardcoded http://litellm...:4000/v1✅ correct
10Ollama pathseparate ollama/ provider (native API, no /v1 semantics) — explicitly excluded from discovery (shouldRegisterEndpoint)✅ not applicable
Root contract, now written on the chokepoint: LiteLLM's openai/ provider sends to <api_base>/chat/completions verbatim and never appends /v1 — so every LiteLLM api_base must carry /v1, while buyer-tooling bases must not (they append the full path).
Known residual gap (out of scope, tracked): the config-drift checker compares model names only, so a same-name entry with a divergent api_base (exactly this incident's poisoned group) is invisible to it.
@bussyjd
bussyjd
enabled auto-merge (squash)
July 14, 2026 07:15

This was referenced Jul 14, 2026

Open

 Closed

@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026

Open

bussyjd added a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
49be12f
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026
 Merged

4 tasks
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #749 (integration/v0.13.0-rc4) — this branch is merged into the integration branch verbatim (merge commits a02b62f/7d963125) and ships there. Validated by the fully green release smoke rc4-run7-20260714-171005 (18/18 flows incl. flow-13/14, on-chain receipts in #749).
@bussyjdbussyjd
closed this
Jul 14, 2026

auto-merge was automatically disabled
July 14, 2026 10:50
Pull request was closed
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
chore(model): drop dead HasProviderConfigured
cb960cb
Zero callers anywhere (including tests) — flagged by the /v1 pipeline
audit's dead-code sweep alongside WarnAndStripV1Suffix (removed in
#745). Other deadcode-tool hits in internal/{buy,inference} are
test-covered exported API and were left alone.
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
1c5bb04
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix(model): register discovered endpoints with /v1 api_base - #745
Closed
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Closed
fix(model): register discovered endpoints with /v1 api_base#745
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Conversation
@bussyjd
@bussyjdbussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

Contributor
Local-server auto-discovery (buildDiscoveredProvider, since v0.10.0 69f25bf) registers OpenAI-compatible endpoints with a bare api_base. LiteLLM's OpenAI provider does not append /v1 (CLAUDE.md pitfall 6), so requests through the discovered entry hit POST /chat/completions → vLLM 404 {'detail':'Not Found'}. When an explicit obol model setup custom --endpoint .../v1 entry shares the model group, LiteLLM shuffles ~50/50 between the good and poisoned deployment.
Evidence (v0.13.0-rc4 release-smoke on spark1): vLLM access log 06:12:15 POST /chat/completions 404 → retry 06:12:17 POST /v1/chat/completions 200 (flow-03 step[2] flake), and 06:25:38 POST /chat/completions 404 matching Alice's verifier handler returned 404, skipping settlement (flow-11 step[43] hard fail — the only red step in an otherwise green run). Reproduced by curl: /v1/chat/completions → 200, /chat/completions → 404.
Not an rc4 regression — latent since v0.10.0, exposed on QA hosts where vLLM sits on the discovery-probed port with served-model-name == OBOL_LLM_MODEL. Blocks a green release-smoke, so it should ride in v0.13.0.
Companion: #744 (flow-03 retry + surface HTTP status).
https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
fix(model): register discovered endpoints with /v1 api_base
171731b
Local-server auto-discovery registered OpenAI-compatible endpoints
(vLLM on :8000) with a bare api_base. LiteLLM's OpenAI provider does
not append /v1 (CLAUDE.md pitfall 6), so every request through the
discovered entry hit POST /chat/completions and 404'd. When an explicit
'obol model setup custom --endpoint .../v1' entry shared the same model
group, requests coin-flipped between the good and bad deployment —
root cause of the intermittent release-smoke flow-11 step[43] 404 and
the flow-03 step[2] flake (latent since v0.10.0, 69f25bf).
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
bussyjdforce-pushed
the
fix/discovery-api-base-v1
branch
from
78c2880 to
171731bCompareJuly 14, 2026 07:03
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Full /v1 pipeline audit (requested before merge)
Every producer and consumer of OpenAI-compatible endpoint URLs in the repo, verified against source:
#SiteConventionVerdict
1obol model setup custom → AddCustomEndpointWithOptions (model.go:995)endpoint must include /v1; enforced by validation step 2, which probes <endpoint>/chat/completions — a /v1-less vLLM endpoint fails the inference probe and is rejected at setup time✅ correct, self-protecting
2auto-discovery → buildDiscoveredProvider (discover.go)was the only violator: wrote bare api_base. Discovery's own gate (fetchModels, detect.go:282) requires GET <base>/v1/models → 200, so every endpoint it can emit provably serves under /v1 — the +"/v1" append is safe by construction for all discoverable server types (vLLM, llama.cpp, LM Studio)✅ fixed by this PR
3shared chokepoint buildCustomEndpointEntryWithOptions (model.go)writes api_base verbatim; both callers now uphold the invariant✅ invariant now documented on the builder
4WarnAndStripV1Suffix (model.go:1699)dead code (zero callers) whose doc claimed the opposite ("LiteLLM auto-appends /v1") — the fossil that kept re-seeding this confusion✅ deleted in this PR
5llm.yaml paid/* default routeapi_base: http://x402-buyer...:8402/v1with/v1, pitfall documented inline (ffa7cf4 fixed this same class)✅ correct
6per-purchase entries → buyerAPIBase() (purchase_helpers.go:281)http://x402-buyer.<ns>...:8402/v1 — with /v1, plus legacy-base migration✅ correct
7buyer-side URL building (buyprompts.ChatCompletionsURL, buy/discover.go:567)opposite convention — base without/v1, appends /v1/chat/completions itself; internally consistent and documented as lockstep with the verifier✅ correct (different but coherent convention)
8seller-side verifier normalizeChatCompletionsPath (verifier.go:857)tolerantly rewrites "", /v1, /chat/completions → /v1/chat/completions for inference/agent offers✅ correct
9hermes agent renderer + openclawhardcoded http://litellm...:4000/v1✅ correct
10Ollama pathseparate ollama/ provider (native API, no /v1 semantics) — explicitly excluded from discovery (shouldRegisterEndpoint)✅ not applicable
Root contract, now written on the chokepoint: LiteLLM's openai/ provider sends to <api_base>/chat/completions verbatim and never appends /v1 — so every LiteLLM api_base must carry /v1, while buyer-tooling bases must not (they append the full path).
Known residual gap (out of scope, tracked): the config-drift checker compares model names only, so a same-name entry with a divergent api_base (exactly this incident's poisoned group) is invisible to it.
@bussyjd
bussyjd
enabled auto-merge (squash)
July 14, 2026 07:15

This was referenced Jul 14, 2026

Open

 Closed

@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026

Open

bussyjd added a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
49be12f
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026
 Merged

4 tasks
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #749 (integration/v0.13.0-rc4) — this branch is merged into the integration branch verbatim (merge commits a02b62f/7d963125) and ships there. Validated by the fully green release smoke rc4-run7-20260714-171005 (18/18 flows incl. flow-13/14, on-chain receipts in #749).
@bussyjdbussyjd
closed this
Jul 14, 2026

auto-merge was automatically disabled
July 14, 2026 10:50
Pull request was closed
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
chore(model): drop dead HasProviderConfigured
cb960cb
Zero callers anywhere (including tests) — flagged by the /v1 pipeline
audit's dead-code sweep alongside WarnAndStripV1Suffix (removed in
#745). Other deadcode-tool hits in internal/{buy,inference} are
test-covered exported API and were left alone.
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
1c5bb04
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
fix(model): register discovered endpoints with /v1 api_base - #745
Closed
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Closed
fix(model): register discovered endpoints with /v1 api_base#745
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Conversation
@bussyjd
@bussyjdbussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

Contributor
Local-server auto-discovery (buildDiscoveredProvider, since v0.10.0 69f25bf) registers OpenAI-compatible endpoints with a bare api_base. LiteLLM's OpenAI provider does not append /v1 (CLAUDE.md pitfall 6), so requests through the discovered entry hit POST /chat/completions → vLLM 404 {'detail':'Not Found'}. When an explicit obol model setup custom --endpoint .../v1 entry shares the model group, LiteLLM shuffles ~50/50 between the good and poisoned deployment.
Evidence (v0.13.0-rc4 release-smoke on spark1): vLLM access log 06:12:15 POST /chat/completions 404 → retry 06:12:17 POST /v1/chat/completions 200 (flow-03 step[2] flake), and 06:25:38 POST /chat/completions 404 matching Alice's verifier handler returned 404, skipping settlement (flow-11 step[43] hard fail — the only red step in an otherwise green run). Reproduced by curl: /v1/chat/completions → 200, /chat/completions → 404.
Not an rc4 regression — latent since v0.10.0, exposed on QA hosts where vLLM sits on the discovery-probed port with served-model-name == OBOL_LLM_MODEL. Blocks a green release-smoke, so it should ride in v0.13.0.
Companion: #744 (flow-03 retry + surface HTTP status).
https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
fix(model): register discovered endpoints with /v1 api_base
171731b
Local-server auto-discovery registered OpenAI-compatible endpoints
(vLLM on :8000) with a bare api_base. LiteLLM's OpenAI provider does
not append /v1 (CLAUDE.md pitfall 6), so every request through the
discovered entry hit POST /chat/completions and 404'd. When an explicit
'obol model setup custom --endpoint .../v1' entry shared the same model
group, requests coin-flipped between the good and bad deployment —
root cause of the intermittent release-smoke flow-11 step[43] 404 and
the flow-03 step[2] flake (latent since v0.10.0, 69f25bf).
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
bussyjdforce-pushed
the
fix/discovery-api-base-v1
branch
from
78c2880 to
171731bCompareJuly 14, 2026 07:03
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Full /v1 pipeline audit (requested before merge)
Every producer and consumer of OpenAI-compatible endpoint URLs in the repo, verified against source:
#SiteConventionVerdict
1obol model setup custom → AddCustomEndpointWithOptions (model.go:995)endpoint must include /v1; enforced by validation step 2, which probes <endpoint>/chat/completions — a /v1-less vLLM endpoint fails the inference probe and is rejected at setup time✅ correct, self-protecting
2auto-discovery → buildDiscoveredProvider (discover.go)was the only violator: wrote bare api_base. Discovery's own gate (fetchModels, detect.go:282) requires GET <base>/v1/models → 200, so every endpoint it can emit provably serves under /v1 — the +"/v1" append is safe by construction for all discoverable server types (vLLM, llama.cpp, LM Studio)✅ fixed by this PR
3shared chokepoint buildCustomEndpointEntryWithOptions (model.go)writes api_base verbatim; both callers now uphold the invariant✅ invariant now documented on the builder
4WarnAndStripV1Suffix (model.go:1699)dead code (zero callers) whose doc claimed the opposite ("LiteLLM auto-appends /v1") — the fossil that kept re-seeding this confusion✅ deleted in this PR
5llm.yaml paid/* default routeapi_base: http://x402-buyer...:8402/v1with/v1, pitfall documented inline (ffa7cf4 fixed this same class)✅ correct
6per-purchase entries → buyerAPIBase() (purchase_helpers.go:281)http://x402-buyer.<ns>...:8402/v1 — with /v1, plus legacy-base migration✅ correct
7buyer-side URL building (buyprompts.ChatCompletionsURL, buy/discover.go:567)opposite convention — base without/v1, appends /v1/chat/completions itself; internally consistent and documented as lockstep with the verifier✅ correct (different but coherent convention)
8seller-side verifier normalizeChatCompletionsPath (verifier.go:857)tolerantly rewrites "", /v1, /chat/completions → /v1/chat/completions for inference/agent offers✅ correct
9hermes agent renderer + openclawhardcoded http://litellm...:4000/v1✅ correct
10Ollama pathseparate ollama/ provider (native API, no /v1 semantics) — explicitly excluded from discovery (shouldRegisterEndpoint)✅ not applicable
Root contract, now written on the chokepoint: LiteLLM's openai/ provider sends to <api_base>/chat/completions verbatim and never appends /v1 — so every LiteLLM api_base must carry /v1, while buyer-tooling bases must not (they append the full path).
Known residual gap (out of scope, tracked): the config-drift checker compares model names only, so a same-name entry with a divergent api_base (exactly this incident's poisoned group) is invisible to it.
@bussyjd
bussyjd
enabled auto-merge (squash)
July 14, 2026 07:15

This was referenced Jul 14, 2026

Open

 Closed

@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026

Open

bussyjd added a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
49be12f
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026
 Merged

4 tasks
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #749 (integration/v0.13.0-rc4) — this branch is merged into the integration branch verbatim (merge commits a02b62f/7d963125) and ships there. Validated by the fully green release smoke rc4-run7-20260714-171005 (18/18 flows incl. flow-13/14, on-chain receipts in #749).
@bussyjdbussyjd
closed this
Jul 14, 2026

auto-merge was automatically disabled
July 14, 2026 10:50
Pull request was closed
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
chore(model): drop dead HasProviderConfigured
cb960cb
Zero callers anywhere (including tests) — flagged by the /v1 pipeline
audit's dead-code sweep alongside WarnAndStripV1Suffix (removed in
#745). Other deadcode-tool hits in internal/{buy,inference} are
test-covered exported API and were left alone.
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
1c5bb04
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix(model): register discovered endpoints with /v1 api_base - #745
Closed
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Closed
fix(model): register discovered endpoints with /v1 api_base#745
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Conversation
@bussyjd
@bussyjdbussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

Contributor
Local-server auto-discovery (buildDiscoveredProvider, since v0.10.0 69f25bf) registers OpenAI-compatible endpoints with a bare api_base. LiteLLM's OpenAI provider does not append /v1 (CLAUDE.md pitfall 6), so requests through the discovered entry hit POST /chat/completions → vLLM 404 {'detail':'Not Found'}. When an explicit obol model setup custom --endpoint .../v1 entry shares the model group, LiteLLM shuffles ~50/50 between the good and poisoned deployment.
Evidence (v0.13.0-rc4 release-smoke on spark1): vLLM access log 06:12:15 POST /chat/completions 404 → retry 06:12:17 POST /v1/chat/completions 200 (flow-03 step[2] flake), and 06:25:38 POST /chat/completions 404 matching Alice's verifier handler returned 404, skipping settlement (flow-11 step[43] hard fail — the only red step in an otherwise green run). Reproduced by curl: /v1/chat/completions → 200, /chat/completions → 404.
Not an rc4 regression — latent since v0.10.0, exposed on QA hosts where vLLM sits on the discovery-probed port with served-model-name == OBOL_LLM_MODEL. Blocks a green release-smoke, so it should ride in v0.13.0.
Companion: #744 (flow-03 retry + surface HTTP status).
https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
fix(model): register discovered endpoints with /v1 api_base
171731b
Local-server auto-discovery registered OpenAI-compatible endpoints
(vLLM on :8000) with a bare api_base. LiteLLM's OpenAI provider does
not append /v1 (CLAUDE.md pitfall 6), so every request through the
discovered entry hit POST /chat/completions and 404'd. When an explicit
'obol model setup custom --endpoint .../v1' entry shared the same model
group, requests coin-flipped between the good and bad deployment —
root cause of the intermittent release-smoke flow-11 step[43] 404 and
the flow-03 step[2] flake (latent since v0.10.0, 69f25bf).
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
bussyjdforce-pushed
the
fix/discovery-api-base-v1
branch
from
78c2880 to
171731bCompareJuly 14, 2026 07:03
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Full /v1 pipeline audit (requested before merge)
Every producer and consumer of OpenAI-compatible endpoint URLs in the repo, verified against source:
#SiteConventionVerdict
1obol model setup custom → AddCustomEndpointWithOptions (model.go:995)endpoint must include /v1; enforced by validation step 2, which probes <endpoint>/chat/completions — a /v1-less vLLM endpoint fails the inference probe and is rejected at setup time✅ correct, self-protecting
2auto-discovery → buildDiscoveredProvider (discover.go)was the only violator: wrote bare api_base. Discovery's own gate (fetchModels, detect.go:282) requires GET <base>/v1/models → 200, so every endpoint it can emit provably serves under /v1 — the +"/v1" append is safe by construction for all discoverable server types (vLLM, llama.cpp, LM Studio)✅ fixed by this PR
3shared chokepoint buildCustomEndpointEntryWithOptions (model.go)writes api_base verbatim; both callers now uphold the invariant✅ invariant now documented on the builder
4WarnAndStripV1Suffix (model.go:1699)dead code (zero callers) whose doc claimed the opposite ("LiteLLM auto-appends /v1") — the fossil that kept re-seeding this confusion✅ deleted in this PR
5llm.yaml paid/* default routeapi_base: http://x402-buyer...:8402/v1with/v1, pitfall documented inline (ffa7cf4 fixed this same class)✅ correct
6per-purchase entries → buyerAPIBase() (purchase_helpers.go:281)http://x402-buyer.<ns>...:8402/v1 — with /v1, plus legacy-base migration✅ correct
7buyer-side URL building (buyprompts.ChatCompletionsURL, buy/discover.go:567)opposite convention — base without/v1, appends /v1/chat/completions itself; internally consistent and documented as lockstep with the verifier✅ correct (different but coherent convention)
8seller-side verifier normalizeChatCompletionsPath (verifier.go:857)tolerantly rewrites "", /v1, /chat/completions → /v1/chat/completions for inference/agent offers✅ correct
9hermes agent renderer + openclawhardcoded http://litellm...:4000/v1✅ correct
10Ollama pathseparate ollama/ provider (native API, no /v1 semantics) — explicitly excluded from discovery (shouldRegisterEndpoint)✅ not applicable
Root contract, now written on the chokepoint: LiteLLM's openai/ provider sends to <api_base>/chat/completions verbatim and never appends /v1 — so every LiteLLM api_base must carry /v1, while buyer-tooling bases must not (they append the full path).
Known residual gap (out of scope, tracked): the config-drift checker compares model names only, so a same-name entry with a divergent api_base (exactly this incident's poisoned group) is invisible to it.
@bussyjd
bussyjd
enabled auto-merge (squash)
July 14, 2026 07:15

This was referenced Jul 14, 2026

Open

 Closed

@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026

Open

bussyjd added a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
49be12f
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026
 Merged

4 tasks
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #749 (integration/v0.13.0-rc4) — this branch is merged into the integration branch verbatim (merge commits a02b62f/7d963125) and ships there. Validated by the fully green release smoke rc4-run7-20260714-171005 (18/18 flows incl. flow-13/14, on-chain receipts in #749).
@bussyjdbussyjd
closed this
Jul 14, 2026

auto-merge was automatically disabled
July 14, 2026 10:50
Pull request was closed
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
chore(model): drop dead HasProviderConfigured
cb960cb
Zero callers anywhere (including tests) — flagged by the /v1 pipeline
audit's dead-code sweep alongside WarnAndStripV1Suffix (removed in
#745). Other deadcode-tool hits in internal/{buy,inference} are
test-covered exported API and were left alone.
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
1c5bb04
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
fix(model): register discovered endpoints with /v1 api_base - #745
Closed
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Closed
fix(model): register discovered endpoints with /v1 api_base#745
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Conversation
@bussyjd
@bussyjdbussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

Contributor
Local-server auto-discovery (buildDiscoveredProvider, since v0.10.0 69f25bf) registers OpenAI-compatible endpoints with a bare api_base. LiteLLM's OpenAI provider does not append /v1 (CLAUDE.md pitfall 6), so requests through the discovered entry hit POST /chat/completions → vLLM 404 {'detail':'Not Found'}. When an explicit obol model setup custom --endpoint .../v1 entry shares the model group, LiteLLM shuffles ~50/50 between the good and poisoned deployment.
Evidence (v0.13.0-rc4 release-smoke on spark1): vLLM access log 06:12:15 POST /chat/completions 404 → retry 06:12:17 POST /v1/chat/completions 200 (flow-03 step[2] flake), and 06:25:38 POST /chat/completions 404 matching Alice's verifier handler returned 404, skipping settlement (flow-11 step[43] hard fail — the only red step in an otherwise green run). Reproduced by curl: /v1/chat/completions → 200, /chat/completions → 404.
Not an rc4 regression — latent since v0.10.0, exposed on QA hosts where vLLM sits on the discovery-probed port with served-model-name == OBOL_LLM_MODEL. Blocks a green release-smoke, so it should ride in v0.13.0.
Companion: #744 (flow-03 retry + surface HTTP status).
https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
fix(model): register discovered endpoints with /v1 api_base
171731b
Local-server auto-discovery registered OpenAI-compatible endpoints
(vLLM on :8000) with a bare api_base. LiteLLM's OpenAI provider does
not append /v1 (CLAUDE.md pitfall 6), so every request through the
discovered entry hit POST /chat/completions and 404'd. When an explicit
'obol model setup custom --endpoint .../v1' entry shared the same model
group, requests coin-flipped between the good and bad deployment —
root cause of the intermittent release-smoke flow-11 step[43] 404 and
the flow-03 step[2] flake (latent since v0.10.0, 69f25bf).
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
bussyjdforce-pushed
the
fix/discovery-api-base-v1
branch
from
78c2880 to
171731bCompareJuly 14, 2026 07:03
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Full /v1 pipeline audit (requested before merge)
Every producer and consumer of OpenAI-compatible endpoint URLs in the repo, verified against source:
#SiteConventionVerdict
1obol model setup custom → AddCustomEndpointWithOptions (model.go:995)endpoint must include /v1; enforced by validation step 2, which probes <endpoint>/chat/completions — a /v1-less vLLM endpoint fails the inference probe and is rejected at setup time✅ correct, self-protecting
2auto-discovery → buildDiscoveredProvider (discover.go)was the only violator: wrote bare api_base. Discovery's own gate (fetchModels, detect.go:282) requires GET <base>/v1/models → 200, so every endpoint it can emit provably serves under /v1 — the +"/v1" append is safe by construction for all discoverable server types (vLLM, llama.cpp, LM Studio)✅ fixed by this PR
3shared chokepoint buildCustomEndpointEntryWithOptions (model.go)writes api_base verbatim; both callers now uphold the invariant✅ invariant now documented on the builder
4WarnAndStripV1Suffix (model.go:1699)dead code (zero callers) whose doc claimed the opposite ("LiteLLM auto-appends /v1") — the fossil that kept re-seeding this confusion✅ deleted in this PR
5llm.yaml paid/* default routeapi_base: http://x402-buyer...:8402/v1with/v1, pitfall documented inline (ffa7cf4 fixed this same class)✅ correct
6per-purchase entries → buyerAPIBase() (purchase_helpers.go:281)http://x402-buyer.<ns>...:8402/v1 — with /v1, plus legacy-base migration✅ correct
7buyer-side URL building (buyprompts.ChatCompletionsURL, buy/discover.go:567)opposite convention — base without/v1, appends /v1/chat/completions itself; internally consistent and documented as lockstep with the verifier✅ correct (different but coherent convention)
8seller-side verifier normalizeChatCompletionsPath (verifier.go:857)tolerantly rewrites "", /v1, /chat/completions → /v1/chat/completions for inference/agent offers✅ correct
9hermes agent renderer + openclawhardcoded http://litellm...:4000/v1✅ correct
10Ollama pathseparate ollama/ provider (native API, no /v1 semantics) — explicitly excluded from discovery (shouldRegisterEndpoint)✅ not applicable
Root contract, now written on the chokepoint: LiteLLM's openai/ provider sends to <api_base>/chat/completions verbatim and never appends /v1 — so every LiteLLM api_base must carry /v1, while buyer-tooling bases must not (they append the full path).
Known residual gap (out of scope, tracked): the config-drift checker compares model names only, so a same-name entry with a divergent api_base (exactly this incident's poisoned group) is invisible to it.
@bussyjd
bussyjd
enabled auto-merge (squash)
July 14, 2026 07:15

This was referenced Jul 14, 2026

Open

 Closed

@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026

Open

bussyjd added a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
49be12f
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026
 Merged

4 tasks
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #749 (integration/v0.13.0-rc4) — this branch is merged into the integration branch verbatim (merge commits a02b62f/7d963125) and ships there. Validated by the fully green release smoke rc4-run7-20260714-171005 (18/18 flows incl. flow-13/14, on-chain receipts in #749).
@bussyjdbussyjd
closed this
Jul 14, 2026

auto-merge was automatically disabled
July 14, 2026 10:50
Pull request was closed
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
chore(model): drop dead HasProviderConfigured
cb960cb
Zero callers anywhere (including tests) — flagged by the /v1 pipeline
audit's dead-code sweep alongside WarnAndStripV1Suffix (removed in
#745). Other deadcode-tool hits in internal/{buy,inference} are
test-covered exported API and were left alone.
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
1c5bb04
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@bussyjd

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
fix(model): register discovered endpoints with /v1 api_base - #745
Closed
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Closed
fix(model): register discovered endpoints with /v1 api_base#745
bussyjd wants to merge 2 commits into
mainfrom 
fix/discovery-api-base-v1
Conversation
@bussyjd
@bussyjdbussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

Contributor
Local-server auto-discovery (buildDiscoveredProvider, since v0.10.0 69f25bf) registers OpenAI-compatible endpoints with a bare api_base. LiteLLM's OpenAI provider does not append /v1 (CLAUDE.md pitfall 6), so requests through the discovered entry hit POST /chat/completions → vLLM 404 {'detail':'Not Found'}. When an explicit obol model setup custom --endpoint .../v1 entry shares the model group, LiteLLM shuffles ~50/50 between the good and poisoned deployment.
Evidence (v0.13.0-rc4 release-smoke on spark1): vLLM access log 06:12:15 POST /chat/completions 404 → retry 06:12:17 POST /v1/chat/completions 200 (flow-03 step[2] flake), and 06:25:38 POST /chat/completions 404 matching Alice's verifier handler returned 404, skipping settlement (flow-11 step[43] hard fail — the only red step in an otherwise green run). Reproduced by curl: /v1/chat/completions → 200, /chat/completions → 404.
Not an rc4 regression — latent since v0.10.0, exposed on QA hosts where vLLM sits on the discovery-probed port with served-model-name == OBOL_LLM_MODEL. Blocks a green release-smoke, so it should ride in v0.13.0.
Companion: #744 (flow-03 retry + surface HTTP status).
https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
fix(model): register discovered endpoints with /v1 api_base
171731b
Local-server auto-discovery registered OpenAI-compatible endpoints
(vLLM on :8000) with a bare api_base. LiteLLM's OpenAI provider does
not append /v1 (CLAUDE.md pitfall 6), so every request through the
discovered entry hit POST /chat/completions and 404'd. When an explicit
'obol model setup custom --endpoint .../v1' entry shared the same model
group, requests coin-flipped between the good and bad deployment —
root cause of the intermittent release-smoke flow-11 step[43] 404 and
the flow-03 step[2] flake (latent since v0.10.0, 69f25bf).
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjd
bussyjdforce-pushed
the
fix/discovery-api-base-v1
branch
from
78c2880 to
171731bCompareJuly 14, 2026 07:03
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Full /v1 pipeline audit (requested before merge)
Every producer and consumer of OpenAI-compatible endpoint URLs in the repo, verified against source:
#SiteConventionVerdict
1obol model setup custom → AddCustomEndpointWithOptions (model.go:995)endpoint must include /v1; enforced by validation step 2, which probes <endpoint>/chat/completions — a /v1-less vLLM endpoint fails the inference probe and is rejected at setup time✅ correct, self-protecting
2auto-discovery → buildDiscoveredProvider (discover.go)was the only violator: wrote bare api_base. Discovery's own gate (fetchModels, detect.go:282) requires GET <base>/v1/models → 200, so every endpoint it can emit provably serves under /v1 — the +"/v1" append is safe by construction for all discoverable server types (vLLM, llama.cpp, LM Studio)✅ fixed by this PR
3shared chokepoint buildCustomEndpointEntryWithOptions (model.go)writes api_base verbatim; both callers now uphold the invariant✅ invariant now documented on the builder
4WarnAndStripV1Suffix (model.go:1699)dead code (zero callers) whose doc claimed the opposite ("LiteLLM auto-appends /v1") — the fossil that kept re-seeding this confusion✅ deleted in this PR
5llm.yaml paid/* default routeapi_base: http://x402-buyer...:8402/v1with/v1, pitfall documented inline (ffa7cf4 fixed this same class)✅ correct
6per-purchase entries → buyerAPIBase() (purchase_helpers.go:281)http://x402-buyer.<ns>...:8402/v1 — with /v1, plus legacy-base migration✅ correct
7buyer-side URL building (buyprompts.ChatCompletionsURL, buy/discover.go:567)opposite convention — base without/v1, appends /v1/chat/completions itself; internally consistent and documented as lockstep with the verifier✅ correct (different but coherent convention)
8seller-side verifier normalizeChatCompletionsPath (verifier.go:857)tolerantly rewrites "", /v1, /chat/completions → /v1/chat/completions for inference/agent offers✅ correct
9hermes agent renderer + openclawhardcoded http://litellm...:4000/v1✅ correct
10Ollama pathseparate ollama/ provider (native API, no /v1 semantics) — explicitly excluded from discovery (shouldRegisterEndpoint)✅ not applicable
Root contract, now written on the chokepoint: LiteLLM's openai/ provider sends to <api_base>/chat/completions verbatim and never appends /v1 — so every LiteLLM api_base must carry /v1, while buyer-tooling bases must not (they append the full path).
Known residual gap (out of scope, tracked): the config-drift checker compares model names only, so a same-name entry with a divergent api_base (exactly this incident's poisoned group) is invisible to it.
@bussyjd
bussyjd
enabled auto-merge (squash)
July 14, 2026 07:15

This was referenced Jul 14, 2026

Open

 Closed

@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026

Open

bussyjd added a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
49be12f
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
@bussyjdbussyjd
mentioned this pull request
Jul 14, 2026
 Merged

4 tasks
@bussyjd
bussyjd
commented
Jul 14, 2026

Copy link


Copy Markdown

ContributorAuthor
Superseded by #749 (integration/v0.13.0-rc4) — this branch is merged into the integration branch verbatim (merge commits a02b62f/7d963125) and ships there. Validated by the fully green release smoke rc4-run7-20260714-171005 (18/18 flows incl. flow-13/14, on-chain receipts in #749).
@bussyjdbussyjd
closed this
Jul 14, 2026

auto-merge was automatically disabled
July 14, 2026 10:50
Pull request was closed
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
chore(model): drop dead HasProviderConfigured
cb960cb
Zero callers anywhere (including tests) — flagged by the /v1 pipeline
audit's dead-code sweep alongside WarnAndStripV1Suffix (removed in
#745). Other deadcode-tool hits in internal/{buy,inference} are
test-covered exported API and were left alone.
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
OisinKyne pushed a commit
that referenced
this pull request
Jul 14, 2026
@bussyjd@OisinKyne
docs: refresh CLAUDE.md pitfalls + obol-stack-dev skill (v3.2.0)
1c5bb04
- pitfall 6 generalized: EVERY LiteLLM openai/ api_base must include
/v1 (LiteLLM posts <api_base>/chat/completions verbatim); buyer-side
tooling uses the opposite convention; discovery fixed in #745
- new pitfall 22: poisoned model group (two deployments, one bare
api_base -> intermittent 404, no retry; GET /model/info vs CM is
the decisive check)
- new pitfall 23: buyer disconnect != no charge (hermes non-streaming
path completes + settles after abort; keep runs short via
spec.maxTurns; maxConcurrentRuns semantics; 4xx never settled)
- skill: lessons table updated (poisoned model group, parserless QA
endpoint -> relaunch with publisher-exact launcher), version 3.2.0
Claude-Session: https://claude.ai/code/session_014YjPMViNrZ7zBVgUQzwEKk
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers


No reviews


Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@bussyjd