Summary
When vision.json is configured with the openai-codex provider (e.g. openai-codex / gpt-5.6-terra), describe_image always fails with HTTP 403:
Vision tool error: Vision model returned 403: <html>...<meta name="viewport" .../></html>
Root cause
callVisionModel in lib/delegate.ts unconditionally POSTs to ${baseUrl}/chat/completions. For the openai-codex provider, baseUrl is https://chatgpt.com/backend-api, and the Codex backend:
- does not expose
/chat/completions (hence the 403), - instead requires the Responses API over SSE at
https://chatgpt.com/backend-api/codex/responses (same URL resolution pi's own openai-codex-responses provider uses in resolveCodexUrl).
The extension already receives the resolved Model (which carries api: "openai-codex-responses") and the OAuth access token via getApiKeyAndHeaders, but the request shape is hard-coded for OpenAI-compatible chat/completions.
Backend constraints (verified empirically against the live backend)
Iterating on the raw endpoint, the Codex backend rejects requests that don't match:
| Requirement | Detail |
|---|
| Endpoint | {baseUrl}/codex/responses (not /chat/completions, not /responses) |
store | must be false → 400 {"detail":"Store must be set to false"} otherwise |
stream | must be true → 400 {"detail":"Stream must be set to true"} otherwise |
temperature | unsupported → 400 {"detail":"Unsupported parameter: temperature"} |
max_output_tokens / max_tokens | unsupported → 400 {"detail":"Unsupported parameter: max_output_tokens"} |
| Image part | {"type":"input_image","image_url":"data:image/jpeg;base64,..."} |
| Text part | {"type":"input_text","text":"..."} |
| Headers | Authorization: Bearer <oauth access token>, chatgpt-account-id: <from JWT claim "https://api.openai.com/auth".chatgpt_account_id>, originator, OpenAI-Beta: responses=experimental, Accept: text/event-stream |
| Output | SSE events; text is assembled from response.output_text.delta deltas (response.completed ends the stream) |
Proposed fix
Branch inside callVisionModel on visionModel.api === "openai-codex-responses" and use the Responses-API-over-SSE path for that provider, keeping the existing chat/completions path for all other providers. The OAuth access token from getApiKeyAndHeaders(model).apiKey is a JWT; chatgpt-account-id is extracted from its payload (mirroring extractAccountId in pi's openai-codex-responses.js).
Full patch (against main, 0.5.2):
@@ -68,6 +68,150 @@
return { reasoning_effort: level };
}
+// ── OpenAI Codex backend (openai-codex-responses) support ─────────────+// The Codex backend (chatgpt.com/backend-api) does NOT expose /chat/completions:+// it requires the Responses API over SSE at /codex/responses, with specific+// auth headers (chatgpt-account-id from the OAuth JWT, originator) and a body+// that forbids temperature / max_output_tokens. This mirrors how pi's own+// openai-codex-responses provider builds its requests.+function isCodexApi(visionModel: Model<Api>): boolean {+ return visionModel.api === "openai-codex-responses";+}++/** Extract the `chatgpt-account-id` claim from the Codex OAuth access token. */+function extractCodexAccountId(apiKey: string | undefined): string | undefined {+ if (!apiKey) return undefined;+ try {+ const parts = apiKey.split(".");+ const payloadPart = parts[1];+ if (parts.length !== 3 || !payloadPart) return undefined;+ const payload = JSON.parse(atob(payloadPart)) as Record<string, unknown>;+ const auth = payload?.["https://api.openai.com/auth"] as+ | { chatgpt_account_id?: unknown }+ | undefined;+ const accountId = auth?.chatgpt_account_id;+ return typeof accountId === "string" ? accountId : undefined;+ } catch {+ return undefined;+ }+}++/** Resolve the Codex Responses endpoint from the provider baseUrl. */+function resolveCodexUrl(baseUrl: string): string {+ const normalized = baseUrl.replace(/\/+$/, "");+ if (normalized.endsWith("/codex/responses")) return normalized;+ if (normalized.endsWith("/codex")) return `${normalized}/responses`;+ return `${normalized}/codex/responses`;+}++/**+ * Call the OpenAI Codex backend (Responses API over SSE). The backend+ * requires `store: false`, `stream: true`, no temperature / max tokens,+ * plus the chatgpt-account-id + originator headers. The final text is+ * assembled from `response.output_text.delta` SSE events.+ */+async function callCodexVisionModel(+ visionModel: Model<Api>,+ apiKey: string | undefined,+ providerHeaders: Record<string, string> | undefined,+ image: LoadedImage,+ prompt: string,+ signal: AbortSignal | undefined,+ systemPrompt?: string,+): Promise<string> {+ const accountId = extractCodexAccountId(apiKey);+ const headers: Record<string, string> = {+ "Content-Type": "application/json",+ Accept: "text/event-stream",+ "OpenAI-Beta": "responses=experimental",+ originator: "pi",+ };+ if (apiKey) headers.Authorization = `Bearer ${apiKey}`;+ if (accountId) headers["chatgpt-account-id"] = accountId;+ if (providerHeaders) Object.assign(headers, providerHeaders);++ const body: Record<string, unknown> = {+ model: visionModel.id,+ store: false,+ stream: true,+ instructions:+ systemPrompt && systemPrompt.length > 0+ ? systemPrompt+ : "You are a helpful assistant.",+ input: [+ {+ role: "user",+ content: [+ {+ type: "input_image",+ image_url: `data:${image.mimeType};base64,${image.data}`,+ },+ { type: "input_text", text: prompt },+ ],+ },+ ],+ text: { verbosity: "low" },+ };++ const response = await fetch(resolveCodexUrl(visionModel.baseUrl), {+ method: "POST",+ headers,+ body: JSON.stringify(body),+ signal,+ });++ if (!response.ok) {+ const errBody = await response.text().catch(() => "");+ throw new Error(+ `Vision model returned ${response.status}: ${errBody.slice(0, 500)}`,+ );+ }++ if (!response.body) {+ throw new Error("Vision model returned no body");+ }+ const reader = response.body.getReader();+ const decoder = new TextDecoder();+ let buffer = "";+ const deltas: string[] = [];+ try {+ while (true) {+ const { done, value } = await reader.read();+ if (done) break;+ buffer += decoder.decode(value, { stream: true });+ let boundary: number;+ while ((boundary = buffer.indexOf("\n\n")) !== -1) {+ const event = buffer.slice(0, boundary);+ buffer = buffer.slice(boundary + 2);+ for (const line of event.split("\n")) {+ if (!line.startsWith("data: ")) continue;+ let data: unknown;+ try {+ data = JSON.parse(line.slice(6));+ } catch {+ continue;+ }+ if (+ data &&+ typeof data === "object" &&+ (data as { type?: unknown }).type === "response.output_text.delta"+ ) {+ const delta = (data as { delta?: unknown }).delta;+ if (typeof delta === "string") deltas.push(delta);+ }+ }+ }+ }+ } finally {+ reader.releaseLock();+ }+ const text = deltas.join("");+ if (!text) {+ throw new Error("Vision model returned no content in the response");+ }+ return text;+}+
/**
* Call the vision model's OpenAI-compat chat/completions endpoint with the
* image as a data URL + the user's prompt (and an optional system prompt).
@@ -132,6 +276,10 @@
reasoning: ReasoningLevel,
systemPrompt?: string,
): Promise<string> {
+ // Codex backend: Responses API over SSE, not chat/completions.+ if (isCodexApi(visionModel)) {+ return callCodexVisionModel(visionModel, apiKey, providerHeaders, image, prompt, signal, systemPrompt);+ }
const baseUrl = visionModel.baseUrl.replace(/\/+$/, "");Verification
pnpm typecheck (tsc --noEmit): passespnpm test:run (353 tests): passes with the patch (T44/T48 concurrency tests are timing-flaky on a slow CPU-only machine — they flake equally on clean main)- End-to-end against the live Codex backend with a real image:
describe_image on an openai-codex / gpt-5.6-terra config returns a correct description (HTTP 200, SSE parsed, ~5s)
Notes / open questions
- The patch relies on
apiKey being the OAuth access token (JWT), which is what getApiKeyAndHeaders returns for the openai-codex provider — same contract pi's own codex provider uses (options.apiKey). reasoning effort is currently ignored on the Codex path (the backend also rejects temperature); if reasoning should be forwarded, the mapping would be reasoning: { effort } per pi's buildRequestBody.- If the maintainers prefer, I can open a PR with this change (plus a unit test that asserts the codex branch builds the
/codex/responses URL and body shape).
Summary
When
vision.jsonis configured with theopenai-codexprovider (e.g.openai-codex/gpt-5.6-terra),describe_imagealways fails with HTTP 403:Root cause
callVisionModelinlib/delegate.tsunconditionally POSTs to${baseUrl}/chat/completions. For theopenai-codexprovider,baseUrlishttps://chatgpt.com/backend-api, and the Codex backend:/chat/completions(hence the 403),https://chatgpt.com/backend-api/codex/responses(same URL resolution pi's ownopenai-codex-responsesprovider uses inresolveCodexUrl).The extension already receives the resolved
Model(which carriesapi: "openai-codex-responses") and the OAuth access token viagetApiKeyAndHeaders, but the request shape is hard-coded for OpenAI-compatible chat/completions.Backend constraints (verified empirically against the live backend)
Iterating on the raw endpoint, the Codex backend rejects requests that don't match:
{baseUrl}/codex/responses(not/chat/completions, not/responses)storefalse→400 {"detail":"Store must be set to false"}otherwisestreamtrue→400 {"detail":"Stream must be set to true"}otherwisetemperature400 {"detail":"Unsupported parameter: temperature"}max_output_tokens/max_tokens400 {"detail":"Unsupported parameter: max_output_tokens"}{"type":"input_image","image_url":"data:image/jpeg;base64,..."}{"type":"input_text","text":"..."}Authorization: Bearer <oauth access token>,chatgpt-account-id: <from JWT claim "https://api.openai.com/auth".chatgpt_account_id>,originator,OpenAI-Beta: responses=experimental,Accept: text/event-streamresponse.output_text.deltadeltas (response.completedends the stream)Proposed fix
Branch inside
callVisionModelonvisionModel.api === "openai-codex-responses"and use the Responses-API-over-SSE path for that provider, keeping the existing chat/completions path for all other providers. The OAuth access token fromgetApiKeyAndHeaders(model).apiKeyis a JWT;chatgpt-account-idis extracted from its payload (mirroringextractAccountIdin pi'sopenai-codex-responses.js).Full patch (against
main, 0.5.2):Verification
pnpm typecheck(tsc --noEmit): passespnpm test:run(353 tests): passes with the patch (T44/T48 concurrency tests are timing-flaky on a slow CPU-only machine — they flake equally on cleanmain)describe_imageon anopenai-codex/gpt-5.6-terraconfig returns a correct description (HTTP 200, SSE parsed, ~5s)Notes / open questions
apiKeybeing the OAuth access token (JWT), which is whatgetApiKeyAndHeadersreturns for theopenai-codexprovider — same contract pi's own codex provider uses (options.apiKey).reasoningeffort is currently ignored on the Codex path (the backend also rejectstemperature); if reasoning should be forwarded, the mapping would bereasoning: { effort }per pi'sbuildRequestBody./codex/responsesURL and body shape).