Problem
openkb's wiki compiler (openkb/agent/compiler.py) calls litellm.completion() /
litellm.acompletion() in buffered (non-streaming) mode. On a long-running compile
(large documents, many concepts, verbose models), some corporate LLM gateways sit
in front of the actual provider and enforce an idle timeout on buffered requests
— if no bytes are sent back to the client for N seconds while the gateway waits for
the upstream provider to finish generating the full response, the gateway kills the
connection with a Gateway Timeout (502/504), even though the underlying provider
call would have eventually succeeded.
This has been observed against a corporate AI.proxy gateway deployed on AWS: any
step whose completion takes longer than the gateway's idle window (large concept
pages, long summaries, etc.) fails with a timeout, forcing a retry (or, without a
retry mechanism, aborting the whole compile step) purely because of buffering, not
because the model itself was slow to start responding.
Reproduction
- Command:
openkb add <large-document> (or any compile step that produces a long
LLM response) - Provider: corporate
AI.proxy gateway (OpenAI-compatible, deployed at AWS) in
front of the real LLM provider - Observed: the LLM call hangs until the gateway's idle timeout fires, then raises a
Gateway Timeout style error (502/504) even though the model would have completed
the response given enough time — the request is only ever a single buffered
round-trip, so no bytes flow until the entire response is ready. - Expected: as long as the provider is actively generating tokens, the call should
not be killed by an idle-connection timeout.
Context
openkb/agent/compiler.py: _llm_call() (sync) and _llm_call_async() (async)
are the two call sites that invoke litellm.completion() / litellm.acompletion()
for every wiki-compilation LLM call (summaries, concept plans, concept/entity
pages, overviews).- LiteLLM pinned at
1.87.2 (pyproject.toml).
Proposed solution
Switch both call sites to litellm.completion(..., stream=True) /
litellm.acompletion(..., stream=True). Streaming keeps the HTTP connection's bytes
flowing as tokens arrive, so an idle-timeout gateway never observes a silent
connection and doesn't kill the request. The streamed chunks are then merged back
into the same response shape the rest of the compiler already expects, via
LiteLLM's own litellm.stream_chunk_builder() — so downstream code (response.choices[0].message.content,
response.usage, response.choices[0].finish_reason) doesn't need to change.
This is a straight swap to streaming-only (no stream=True/False config toggle);
an exception raised mid-stream must be treated as a complete failure — no partial
buffer is used — matching today's all-or-nothing behavior for a failed
litellm.completion() call.
Problem
openkb's wiki compiler (openkb/agent/compiler.py) callslitellm.completion()/litellm.acompletion()in buffered (non-streaming) mode. On a long-running compile(large documents, many concepts, verbose models), some corporate LLM gateways sit
in front of the actual provider and enforce an idle timeout on buffered requests
— if no bytes are sent back to the client for N seconds while the gateway waits for
the upstream provider to finish generating the full response, the gateway kills the
connection with a Gateway Timeout (502/504), even though the underlying provider
call would have eventually succeeded.
This has been observed against a corporate
AI.proxygateway deployed on AWS: anystep whose completion takes longer than the gateway's idle window (large concept
pages, long summaries, etc.) fails with a timeout, forcing a retry (or, without a
retry mechanism, aborting the whole compile step) purely because of buffering, not
because the model itself was slow to start responding.
Reproduction
openkb add <large-document>(or any compile step that produces a longLLM response)
AI.proxygateway (OpenAI-compatible, deployed at AWS) infront of the real LLM provider
Gateway Timeout style error (502/504) even though the model would have completed
the response given enough time — the request is only ever a single buffered
round-trip, so no bytes flow until the entire response is ready.
not be killed by an idle-connection timeout.
Context
openkb/agent/compiler.py:_llm_call()(sync) and_llm_call_async()(async)are the two call sites that invoke
litellm.completion()/litellm.acompletion()for every wiki-compilation LLM call (summaries, concept plans, concept/entity
pages, overviews).
1.87.2(pyproject.toml).Proposed solution
Switch both call sites to
litellm.completion(..., stream=True)/litellm.acompletion(..., stream=True). Streaming keeps the HTTP connection's bytesflowing as tokens arrive, so an idle-timeout gateway never observes a silent
connection and doesn't kill the request. The streamed chunks are then merged back
into the same response shape the rest of the compiler already expects, via
LiteLLM's own
litellm.stream_chunk_builder()— so downstream code (response.choices[0].message.content,response.usage,response.choices[0].finish_reason) doesn't need to change.This is a straight swap to streaming-only (no
stream=True/Falseconfig toggle);an exception raised mid-stream must be treated as a complete failure — no partial
buffer is used — matching today's all-or-nothing behavior for a failed
litellm.completion()call.