Has anyone actually succeeded in using OpenKB on a mac/M3 pro (36GB) with LM Studio runtime?

[SonnyRajagopalan]

Hello,

I have tried now multiple times to get to use my LM Studio runtime for openKB on my M3 Pro/36GB macbook pro. LM Studio running on localhost, and then these examples from the README.md on their github page:

openkb add paper.pdf
openkb add ~/papers/ # Add a whole directory
openkb add https://arxiv.org/pdf/2509.11420 # Or fetch from a URL

With the models qwen/qwen2.5-coder-14b, qwen/qwen3-14b and google/gemma-4-31b. I have had absolutely no luck in ingesting any document(s) with any of these models (atleast no sign of progress in multiple hours).

Let's take LM studio and gemma-4-31b. 2 hours after the start of the session, I see these errors:

$ openkb add https://arxiv.org/pdf/2509.11420
Downloading: https://arxiv.org/pdf/2509.11420
 Saved: raw/2509.11420v1_6.pdf (2.1 MB PDF)
Adding: 2509.11420v1_6.pdf
 Long document detected — indexing with PageIndex...
start find_toc_pages
no toc found
process_no_toc
start_index: 1
divide page_list to groups 3
start generate_toc_init
************* Retrying *************
root ERROR: Error: litellm.Timeout: APITimeoutError - Request timed out. Error_str: Request timed out.
************* Retrying *************
root ERROR: Error: litellm.Timeout: APITimeoutError - Request timed out. Error_str: Request timed out.
************* Retrying *************
root ERROR: Error: litellm.Timeout: APITimeoutError - Request timed out. Error_str: Request timed out.
************* Retrying *************
root ERROR: Error: litellm.Timeout: APITimeoutError - Request timed out. Error_str: Request timed out.
************* Retrying *************
root ERROR: Error: litellm.Timeout: APITimeoutError - Request timed out. Error_str: Request timed out.
start generate_toc_continue
openkb.indexer WARNING: PageIndex attempt 1/3 failed for 2509.11420v1_6.pdf: Failed to index /Users/sonny/Code/open-kb/raw/2509.11420v1_6.pdf: 'dict' object has no attribute 'extend'
start find_toc_pages

(repeat the steps two more PageIndex attempts and then)
 [ERROR] Indexing failed: Failed to index 2509.11420v1_6.pdf after 3 attempts:
...
$ 

My .openkb/config.yaml:

# .openkb/config.yaml

llm:
 provider: "openai"
 model: "openai/gemma-4-31b"

pipeline:
 long_document_threshold: 60
 chunk_size: 2000

and my .env:

# .env
# Force LiteLLM to send all OpenAI-provider traffic to LM Studio
OPENAI_API_BASE=http://localhost:1234/v1

# Use any placeholder string (LM Studio doesn't validate this, but LiteLLM requires its presence)
LLM_API_KEY=local-lm-studio-key

Does anyone have any idea what I might me doing wrong? Has anyone gotten openKB working on local AI?

[SonnyRajagopalan]

I figured out atleast one problem. It turns out that I need to change the value of _JSON_RESPONSE_FORMAT in $OPENKB_DIR/.venv/lib/python3.12/site-packages/openkb/agent/compiler.py to text. It's a quick fix and a rather clunky one to do this, but I changed the line after the import statements in compiler.py from

_JSON_RESPONSE_FORMAT = {"type": "json_object"}

to

_JSON_RESPONSE_FORMAT = {"type": "text"}

and that seemed to make openkb work.

However, a simple 15-page PDF (https://arxiv.org/pdf/1706.03762v5) took over 2 hours to complete, and towards the end, I did end up getting a lot of these errors anyway:

openkb.agent.compiler WARNING: Entity generation failed: litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': 'Context size has been exceeded.'}

[KylinMountain]

Hi @SonnyRajagopalan — this isn't a bug, it's two local-runtime limits. The APITimeoutError is LiteLLM's default timeout aborting your slow local calls (the 'dict' has no attribute 'extend' is just the downstream symptom). The Context size exceeded is LM Studio's default context window (often 4096) being too small.

Two fixes, no need to edit site-packages:

1. In LM Studio, raise the loaded model's Context Length (8k–32k) and reload.
2. Update config.yaml — your llm./pipeline. keys are an old schema and are ignored. Current format is flat with a litellm: block:

model: openai/gemma-4-31b
language: en
litellm:
 timeout: 3600 # local models are slow — be generous
 drop_params: true # drops the json_object param LM Studio rejects

drop_params: true replaces your _JSON_RESPONSE_FORMAT edit. Note 14B–31B models are slow at PageIndex's long-context work — try a short PDF first, or a cloud key to confirm the pipeline. Full reference: examples/configuration/.

[SonnyRajagopalan]

Thank you, @KylinMountain .

I removed the $OPENKB_DIR/ and then reinstalled openkb. I made the config.yaml exactly what you show above (kept .env the same) and still get this error:

$ openkb add https://arxiv.org/pdf/2310.06825 
Downloading: https://arxiv.org/pdf/2310.06825
 Saved: raw/2310.06825v1.pdf (3.6 MB PDF)
Adding: 2310.06825v1.pdf
 Compiling short doc...
 summary Retrying compilation in 2s...
.. summary [ERROR] Compilation failed: litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': "'response_format.type' must be 'json_schema' or 'text'"}

I wonder if the drop_params settings is not taking for some reason.

[jeffm]

I had the same problem on Windows. Based on my futzing, it does not appear that drop_params has any effect. In the end, I had CODEX modify the codebase to create an exception for lm_studio, and, combined with a vastly increased timeout and context window, I got it to work on a relatively small document (~21kb of pure text) in a reasonable period of time.

[arunov]

My workaround was to use my LM Studio model as anthropic/gemma-4-31b-qat.