Skip to content

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex - #4

Merged
rejojer merged 102 commits into
mainfrom
dev
Apr 8, 2026
Merged

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex#4
rejojer merged 102 commits into
mainfrom
dev

Conversation

@KylinMountain

Copy link
Copy Markdown
Collaborator

Summary

OpenKB — Karpathy's LLM Knowledge Base workflow as a CLI, powered by PageIndex.

Drop documents in. Get an auto-maintained, cross-linked wiki out.

Features

  • okb init — Interactive setup
  • okb add — Short docs (pymupdf) + long PDFs (PageIndex local/cloud)
  • okb query — Streaming Q&A with PageIndex cloud streaming
  • okb watch — Auto-compile on file changes
  • okb lint — Structural + knowledge health checks
  • okb list / status — Knowledge base overview
  • Obsidian compatible wiki output

Tech Stack

PageIndex, markitdown, OpenAI Agents SDK, LiteLLM, Click, watchdog

KylinMountainand others added 30 commits April 6, 2026 23:13
Sets up pyproject.toml (hatchling, direct-refs allowed, Python >=3.11),
.gitignore, openkb/__init__.py, a Click CLI stub with all 7 commands
(init, add, query, watch, lint, list, status), and tests/conftest.py
with kb_dir and sample_tree fixtures. Package installs cleanly in a
Python 3.12 venv; okb --help shows all commands; pytest collects 0
tests without error.
Add openkb/config.py (DEFAULT_CONFIG, load_config, save_config),
openkb/state.py (HashRegistry with SHA-256 file hashing and JSON
persistence), and openkb/schema.py (SCHEMA_MD constant). All 17 tests
written first (red) then implemented (green).
Creates full KB directory structure (raw/, wiki/sources/images/,
wiki/summaries/, wiki/concepts/, wiki/reports/), writes SCHEMA.md,
index.md, config.yaml and hashes.json; guards against re-initialisation.
Three tests in tests/test_cli.py cover structure, schema content, and
the already-initialized guard, all via CliRunner.isolated_filesystem.
Implements extract_base64_images and copy_relative_images with full test
coverage for single/multiple images, invalid base64, missing files, and
URL filtering.
Implements ConvertResult dataclass, get_pdf_page_count, and
convert_document with hash-dedup, markdown passthrough, PDF long-doc
detection, MarkItDown conversion, and image extraction integration.
Implements render_source_md and render_summary_md with YAML frontmatter,
recursive heading hierarchy (h1–h6 capped), page ranges, and separate
text/summary views for source and summary wiki pages.
Implements IndexResult dataclass and index_long_document which creates
a LocalClient with full node text/summary/description flags, adds the
PDF via PageIndex, fetches structure, and writes source and summary
wiki pages via the tree renderer.
Implements list_wiki_files, read_wiki_file, and write_wiki_file as plain
functions in openkb/agent/tools.py without @function_tool decoration,
ready to be wrapped when building the agent. Full test coverage including
edge cases for missing files/dirs, filtering to .md only, and parent dir
creation.
Implements build_compiler_agent, compile_short_doc, compile_long_doc in
openkb/agent/compiler.py with function_tool-wrapped wiki tools and
SCHEMA_MD-enriched instructions. Long-doc variant includes get_page_content.
Tests mock Runner.run to avoid real LLM calls.
Replaces the add stub with full orchestration: convert_document,
index_long_document for long PDFs, and compiler agent calls.
Adds SUPPORTED_EXTENSIONS set, _find_kb_dir, _add_single_file helpers.
Adds python-dotenv dependency and load_dotenv() at startup.
Implements pageindex_retrieve (structure -> LLM relevance -> page fetch),
build_query_agent with list/read/retrieve tools, and run_query coroutine.
Wires up `okb query` in cli.py.
Implements DebouncedHandler (collects events, ignores dirs/dotfiles, resets
timer on burst) and watch_directory (Observer loop, Ctrl+C safe).
Wires up `okb watch` in cli.py.
Implements find_broken_links, find_orphans, find_missing_entries,
check_index_sync, and run_structural_lint with full Markdown report.
Covers wikilink resolution, orphan detection, raw/wiki entry matching,
and index.md sync checking.
Implements build_lint_agent with list/read tools and instructions for
semantic quality checks (contradictions, gaps, staleness, redundancy).
run_knowledge_lint runs the agent and returns the report string.
okb lint combines structural + knowledge lint and writes timestamped report.
Tests verify list shows documents table and concepts, status shows
per-directory file counts and total indexed. Both check missing-init guard.
Previously the converter registered the file hash immediately, so if
LLM compilation failed the file was marked as "done" and retries
would skip it. Now the hash is only registered by the CLI after
successful compilation.
Also: install markitdown[all] for PDF support, add python-dotenv.
…pport
- Switch from col._backend.get_document_structure() to col.get_document_structure()
- Add 3x retry for PageIndex indexing (stochastic TOC accuracy)
- Fix storage path to use .db extension
- Remove .doc from supported extensions (markitdown only supports .docx)
- Note: col.get_page_content() still missing from PageIndex public API,
using col._backend.get_page_content() as workaround
Replace col._backend.get_page_content(col._name, doc_id, spec) with
col.get_page_content(doc_id, spec). Now all PageIndex access uses
public API only.
rejojer added 15 commits April 8, 2026 05:01
Rename CLI command and state dir from okb to openkb
- Hardcode reading LLM_API_KEY env var instead of indirecting through config
- Remove llm_api_key_env from DEFAULT_CONFIG, okb init prompts, and config.yaml
- Provider-specific env vars (OPENAI_API_KEY, etc.) still work via LiteLLM auto-detection
- One less config field, one less okb init step
The OpenAI Agents SDK requires a litellm/ prefix to route non-OpenAI
models through LiteLLM. Without it, models like anthropic/claude-sonnet-4-6
fail with "Unknown prefix". This adds the prefix at all Agent() call sites
while keeping litellm.completion() calls unchanged.
Also updates README quick start comments and model format docs.
Fix: add litellm/ prefix for Agents SDK model routing
@KylinMountain

Copy link
Copy Markdown
CollaboratorAuthor

Code review

Found 1 issue:

  1. extract_pdf_images and convert_pdf_with_images in images.py open pymupdf documents with explicit .close() instead of context managers. If an exception is raised during page iteration (e.g. corrupt image block, pixmap allocation failure), the PDF file handle leaks. This is the same bug pattern that was already fixed in converter.py:get_pdf_page_count (commit c525455), but images.py was missed. Fix: replace doc = pymupdf.open(...) / doc.close() with with pymupdf.open(...) as doc:.

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] !=1: # not an image block
continue
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
save_path=images_dir/filename
pix.save(str(save_path))
pix=None
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
continue
rel_path=f"images/{doc_name}/{filename}"
page_images.setdefault(page_num, []).append(rel_path)
doc.close()
returnpage_images

OpenKB/openkb/images.py

Lines 89 to 125 in 1637697

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
parts.append(f"\n\n<!-- Page {page_num} -->\n")
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] ==0: # text block
lines= []
forlineinblock["lines"]:
spans_text="".join(span["text"] forspaninline["spans"])
lines.append(spans_text)
parts.append("\n".join(lines))
elifblock["type"] ==1: # image block
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
(images_dir/filename).write_bytes(pix.tobytes("png"))
pix=None
parts.append(f"\n![image](images/{doc_name}/{filename})\n")
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
doc.close()
return"\n".join(parts)

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer

Copy link
Copy Markdown
Member

Code review

Found 2 issues:

  1. README claims LiteLLM is "pinned to a safe version" but pyproject.toml has no version pin. Line 67 of README.md states LiteLLM is (pinned to a safe version), but pyproject.toml line 17 lists the dependency as bare "litellm" with no version constraint (==, >=, ~=, etc.). Any version -- including potentially insecure ones -- can be installed.

OpenKB/README.md

Lines 66 to 68 in 854294c

OpenKB comes with [multi-LLM support](https://docs.litellm.ai/docs/providers) (e.g., OpenAI, Claude, Gemini) via [LiteLLM](https://github.com/BerriAI/litellm) (pinned to a [safe version](https://docs.litellm.ai/blog/security-update-march-2026)).

OpenKB/pyproject.toml

Lines 16 to 18 in 854294c

"watchdog>=3.0",
"litellm",
"openai-agents",

  1. test_short_pdf_converted_via_markitdown mocks the wrong code path. The test patches openkb.converter.MarkItDown and openkb.converter.pymupdf.open, but converter.py line 99-101 routes short PDFs through convert_pdf_with_images() (from openkb.images), not MarkItDown. The MarkItDown mock is never exercised, and convert_pdf_with_images is not mocked, so the test either fails at runtime or passes for the wrong reasons.

classTestConvertDocumentPdfShort:
deftest_short_pdf_converted_via_markitdown(self, kb_dir, tmp_path):
"""PDF under threshold is converted with markitdown."""
src=tmp_path/"short.pdf"
src.write_bytes(b"%PDF-1.4 fake content")
fake_result=MagicMock()
fake_result.text_content="# Short PDF\n\nConverted content."
with (
patch("openkb.converter.pymupdf.open") asmock_mu,
patch("openkb.converter.MarkItDown") asmock_mid_cls,
):
fake_doc=MagicMock()
fake_doc.page_count=5# below default threshold of 20
fake_doc.__enter__=MagicMock(return_value=fake_doc)
fake_doc.__exit__=MagicMock(return_value=False)
mock_mu.return_value=fake_doc
mock_mid_cls.return_value.convert.return_value=fake_result
result=convert_document(src, kb_dir)
assertresult.skippedisFalse
assertresult.is_long_docisFalse
assertresult.source_pathisnotNone
assertresult.source_path.exists()

markdown=copy_relative_images(markdown, src.parent, doc_name, images_dir)
elifsrc.suffix.lower() ==".pdf":
# Use pymupdf dict-mode for PDFs: text + images inline at correct positions
markdown=convert_pdf_with_images(src, doc_name, images_dir)
else:

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer
rejojer merged commit f0963f6 into mainApr 8, 2026
KylinMountain added a commit that referenced this pull request May 24, 2026
Architectural review (4 parallel Opus auditors) found that the skill_runner
core was already generic, but the deck SURFACE was still fused to
Editorial Monocle. Fixed:
* validator: now takes optional `grammar` param (DeckGrammar TypedDict);
skill-agnostic by default (only checks file present, parses, ≥5
slides, self-contained). Third-party deck skills (guizang, swiss)
now pass validation cleanly. Editorial-specific rules opt-in via
`EDITORIAL_MONOCLE_GRAMMAR`. (finding #2)
* skills/openkb-deck-editorial/SKILL.md: declares its grammar +
output_path_template under `od:` frontmatter — `run_skill` reads
these and applies them post-run.
* run_skill: now honors frontmatter `od.mode`, `od.output_path_template`,
`od.deck_grammar`. When mode=="deck" and template is set, the runner
injects the path into intent, verifies the file exists post-run, and
runs validate_deck with the skill's grammar. Validation result is
returned via new SkillRunResult dataclass. (findings #4, #5)
* `openkb deck new --skill <name>`: CLI flag accepts any installed deck
skill (default openkb-deck-editorial). guizang and swiss now usable
from the scripted CLI, not only freeform chat. (finding #1)
* `/deck new --skill <name>` chat slash: same flag, parsed positionally
alongside --critique. (finding #1)
* tests/test_read_kb_file.py: 13 new tests mirroring test_write_kb_file
for the read-side allow-list. Pins refusal of `.openkb/config.yaml`,
`.env`, `raw/`, `..` traversal, absolute paths. (finding #6)
* Generator deck branch: no longer calls validate_deck directly; just
propagates run_deck_create's SkillRunResult.validation up. Validation
is now a property of "this skill declared mode=deck", not of "this
CLI path was taken".
Existing tests updated:
* tests/test_deck_validator.py: explicit grammar arg on Editorial-
specific tests; added test_guizang_shape_passes_generic_mode +
test_missing_cover_ignored_in_generic_mode to pin both modes.
* tests/test_deck_creator.py: mocks return SkillRunResult; new
test_run_deck_create_honors_skill_name_override for --skill flag.
* tests/test_generator.py: deck dispatch test mocks SkillRunResult.
Below-threshold findings deferred:
* Generator if/else → registry (score 70) — works, just not extensible
via plugin; future.
* Iteration backup in chat freeform path (score 75) — needs write_kb_file
hook; separate change.
* run_skill / scan_local_skills / _handle_slash_critique direct tests
(scores 60-70) — covered indirectly by integration; can add later.
Regression: 538 tests pass (was 523 pre-fix; net +15 = 13 new
read_kb_file tests + 2 new validator-mode tests).
KylinMountain added a commit that referenced this pull request May 31, 2026
…lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
KylinMountain added a commit that referenced this pull request Jun 1, 2026
…mpile backfill (#78)
* feat(compiler): _read_entity_briefs for entity plan context
* test(compiler): parity tests for _read_entity_briefs
* feat(compiler): _write_entity with type/aliases frontmatter
* test(compiler): assert source ordering in _write_entity; count=1 in _set_fm_line
Add explicit ordering assertion in test_update_prepends_source_keeps_type
verifying the deterministic json.dumps form ("summaries/b.md", "summaries/a.md").
Pass count=1 to re.sub in _set_fm_line to make first-occurrence intent explicit.
* feat(lint): include entities/ in wikilink whitelist
* feat(compiler): summary<->entity backlinks
* test(compiler): restore assertion erroneously deleted in 3c8aa93
* feat(compiler): index.md Entities section
* feat(compiler): remove_doc_from_entity_pages + index cleanup
* feat(compiler): plan prompt + parser for entities group
Also wires the entity track into _compile_concepts (Tasks 7 + 8 combined,
since the {entity_briefs} placeholder and the _CONCEPTS_PLAN_USER.format call
are co-dependent — splitting would leave an intermediate red state).
- add _ENTITY_TYPES, _filter_entity_items, _parse_entities_plan
- rewrite _CONCEPTS_PLAN_USER to request nested concepts+entities groups
- add _ENTITY_PAGE_USER / _ENTITY_UPDATE_USER prompts
- read entity briefs and pass both briefs to the plan prompt
- parse nested 'concepts' group with legacy flat-list/flat-dict fallbacks
- generate entities in their own asyncio.gather (4-arity tuples)
- strip ghost links + _write_entity each; handle entity related cross-links
- backlink summary<->entities; pass entity_names/entity_meta to _update_index
* fix(compiler): related entities must not downgrade index labels
Mirror the concept track: collect related-entity slugs into a separate
local list used only for backlinks; pass only created/updated entity_names
(+entity_meta) to _update_index. Defense-in-depth in _update_index: only
_replace_section_entry when name is in entity_meta, otherwise only insert
if the link is absent, so a related-only entity can never clobber a
pre-existing correct (type + brief) index line with "(other)".
Adds regression test test_related_entity_does_not_downgrade_index_label.
* feat(schema): declare entities/ page type and taxonomy
* feat(query): point who/what questions at entities/
* docs(readme): document entities/ page type
* feat(cli): scaffold entities/ in init and count it in status
- `openkb init` now creates wiki/entities/ alongside wiki/concepts/
- init seed index.md gains ## Entities between ## Concepts and ## Explorations,
matching the _update_index template in compiler.py
- print_status subdirs list gains "entities" after "concepts"
- Tests updated: assert wiki/entities/ exists and index.md contains ## Entities;
status test asserts "entities" appears in output
* fix(compiler): resolve entity-page review findings (dangling links + dedup)
Addresses code-review findings on the entity-pages feature:
- Fix dangling wikilink after `openkb remove`: entity removal now strips
standalone `See also: [[summaries/{doc}]]` lines (the related-entity
backlink form), matching the concept path, and cli.py adds modified
entity pages to the lint sweep scope so surviving pages are cleaned.
- Unify the parallel concept/entity helpers into shared cores
(_backlink_summary_pages, _backlink_pages, _remove_doc_from_pages) with
thin per-type wrappers, so cleanup logic can no longer drift between the
two page types (this is what caused the dangling-link bug).
- Route related-entity cross-refs through _add_related_link (now page-type
aware) instead of an inline reimplementation — removes a duplicate file
read/write and keeps backlink creation symmetric with teardown.
- Centralize the entity-type enum: prompts derive their type list from a
single _ENTITY_TYPE_LIST source via import-time substitution.
- Count entity items in the "all dropped as malformed" plan warning.
- Drop the unreachable else branch in _update_index's entity loop.
- Add regression test for the See-also strip on a surviving entity page.
All 542 tests pass.
* fix(compiler): add [[entities/X]] whitelist rule + restore concept-topic guard
Remaining review findings after a7a06ed:
- _KNOWN_TARGETS_USER now states the [[entities/Z]] rule, so entity links
the LLM is told to write aren't silently stripped as ghosts.
- Restore the dropped 'Do NOT create concepts that are just the document
topic itself' plan rule to prevent redundant title-mirror concepts.
* feat(entities): shared page-dir constants + surface entities in list/lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
* feat(entities): remove preview lists entity-page actions (#1)
The dry-run/confirmation block now scans wiki/entities/ with the same
frontmatter sources: logic as concepts, emits DELETE/MODIFY action lines
per entity page, and prints an 'N entity(s) will be DELETED' summary.
Execution path (remove_doc_from_entity_pages) unchanged.
* docs(entities): document entity pages in shipped openkb skill (#8)
Note wiki/entities/ holds named-thing pages (people/orgs/places/
products/works/events) with a type: frontmatter field, that index.md
has a ## Entities section, and that 'who/what is X' questions should
read the matching entities/ page first.
* fix(compiler): don't write raw JSON body on empty LLM content
In the parse-succeeded branch of _gen_create/_gen_update/_gen_entity_create/
_gen_entity_update, fall back to "" instead of the raw JSON string when the
content field is empty/null. _require_nonempty_content then raises and the
page is dropped, rather than writing the JSON envelope as the markdown body.
The parse-FAILED (except) branch keeps content=raw as the legitimate
non-JSON fallback.
* fix(compiler): graceful scalar plan + rebuild malformed entity frontmatter
- _compile_concepts: guard a non-dict/non-list parsed plan (JSON scalar)
before calling .get(), taking the empty-plan path (write v1 summary if
applicable + update index + return) instead of risking AttributeError.
- _write_entity: when an existing page has an opening --- but no closing
delimiter (or no frontmatter), rebuild valid sources/type/brief frontmatter
rather than writing a body-only page that drops the metadata.
* fix(compiler): keep ## Entities before ## Explorations; drop dead param + overlap gathers
- _update_index: insert ## Entities before ## Explorations on older index.md
files that predate the section (new _ensure_h2_section_before helper),
preserving canonical order instead of appending at EOF.
- _filter_entity_items: drop the unused 'label' parameter and update call
sites in _parse_entities_plan.
- _compile_concepts: overlap concept and entity generation in one outer
asyncio.gather (they share cached context and the same concurrency
semaphore); result/error handling per list is unchanged.
* test(compiler): cover empty-content skip, scalar plan, malformed entity FM, Entities order
Add regression tests for the four compiler fixes:
- empty {"content":""} response skips the page (no raw JSON body)
- JSON scalar plan handled gracefully (no AttributeError)
- _write_entity rebuilds frontmatter when closing --- is missing
- _update_index inserts ## Entities before ## Explorations
* fix(compiler): silence spurious 'hand-edited' warning on backlink section creation
_backlink_summary_pages / _backlink_pages create ## Entities / ## Related
Documents sections as a normal first-time operation; pass quiet=True so
_ensure_h2_section no longer logs the index-drift warning in that case.
Index-repair callers keep the warning.
* feat(cli): add `recompile` command to re-run compile on indexed docs
Re-runs the current compile_short_doc/compile_long_doc pipeline on
already-indexed docs so pre-feature KBs gain the entities/ layer and
refresh to the current format. Reuses on-disk sources/summaries and the
registry's PageIndex doc_id — does not re-index or re-convert.
Supports a positional <doc_name> (resolved via _resolve_doc_identifier)
or --all (with a regeneration-warning confirmation, bypassed by --yes),
--dry-run (enumerate only, no LLM calls/writes), and --refresh-schema
(back up + overwrite wiki/AGENTS.md when it differs from AGENTS_MD).
Processes docs sequentially with per-doc progress, skips+warns on
missing sources / summaries / doc_id, prints a recompiled/skipped
summary, and appends a recompile entry to log.md.
* test(cli): recompile dispatch/dry-run/skip/refresh-schema
* docs(readme): document openkb recompile
* fix(cli): recompile --refresh-schema no-ops when AGENTS.md absent; tighten guard tests
Match the spec (and the helper's own docstring): _refresh_schema returns
early when wiki/AGENTS.md is missing rather than materializing the default
(get_agents_md already falls back to it at runtime). Tighten the doc/--all
guard tests to assert the exact message + that no compile runs, and add the
missing-AGENTS.md no-op test.
* fix(compiler): drop non-existent 'related' slugs so they don't create dangling links
The plan's 'related' list is meant to reference existing pages, but the LLM
sometimes lists slugs for pages that don't exist. Those were added to the
wikilink whitelist (so body references survived ghost-stripping) and
back-linked into the summary's Related section, yet no page was ever created
(related items are linked, never generated) — producing a flood of broken
[[concepts/...]] / [[entities/...]] links (esp. on feature-dense docs).
Filter related_items / entity_related to slugs that exist on disk.
* fix: remove-preview detects JSON-quoted sources; _write_entity preserves sources on malformed FM
- remove --dry-run preview parsed the sources list with a hand-rolled comma
split that kept JSON quotes (["summaries/x.md"]), so the marker never
matched and the preview always reported 0 affected concept/entity pages
(executor was correct). Extract _scan_affected_pages using the real
_parse_yaml_list_value; dedups the two copied scan loops too.
- _write_entity's malformed-frontmatter rebuild seeded sources with only the
new doc, dropping prior sources for multi-source entities. Recover existing
sources from the broken block and merge.
Both bugs were masked by tests using unquoted / single-source fixtures.
* feat(cli): rename remove --keep-empty-concepts → --keep-empty (covers entities too)
This PR wired entity pages into 'openkb remove', so the flag now governs
concept AND entity retention — but the name still said 'concepts'. Make
--keep-empty the canonical name (clear that it covers both), keep
--keep-empty-concepts as a backward-compatible alias, and update the
preview/summary messages, docstring, and README accordingly.
* feat(compiler): config-driven entity types (entity_types overrides the default enum)
Add an optional 'entity_types:' key in .openkb/config.yaml. When present it
overrides the default person/organization/place/product/work/event/other
vocabulary everywhere — the plan prompt, the entity-page prompts, and
create/update validation/coercion; when absent, behavior is byte-identical.
Prompt templates keep an __ENTITY_TYPES__ token now substituted at call time
(per-KB) inside _compile_concepts, and the resolved valid-type set is threaded
into _parse_entities_plan / _filter_entity_items and the _gen_entity_* coercion.
'other' is always ensured as the coercion fallback; malformed config falls back
to the default with a warning. Documented in config.yaml.example + README.
* fix(compiler): harden config-driven entity types (crash-proof + complete the override)
Review of the config-entity-types feature surfaced two real issues:
- A config 'entity_types' value containing '{' or '}' was substituted into the
prompt template BEFORE .format() ran → KeyError/ValueError crashing every
compile. Swap to format-then-replace at all 3 call sites (types_str is now an
inert literal), and sanitize resolved types to a safe label charset (also
skips YAML nulls/ints so str(None) can't become the type 'none').
- The AGENTS_MD system schema hardcoded 'type: is one of: <7 defaults>',
contradicting a custom entity_types in the higher-weight system message.
Reword it to frame those as the configurable default and defer the
authoritative set to the compilation prompt (which is config-driven).
Also drop the now-dead _ENTITY_TYPES_STR + its stale import-time-substitution
comment. +2 regression tests (sanitization; brace-in-type doesn't crash).
* refactor: move entity-type resolution to config layer + co-locate remove-preview scan
Altitude cleanups from the review:
- Move resolve_entity_types + DEFAULT_ENTITY_TYPES into openkb/config.py (the
config layer owns config validation/normalization; any command can reuse it
without importing the heavy compiler module). compiler.py imports them;
_ENTITY_TYPE_LIST/_ENTITY_TYPES remain as the default alias/validation set.
- Move the remove dry-run preview scan from cli.py into compiler.py as
scan_affected_pages, beside remove_doc_from_*_pages and sharing
_parse_yaml_list_value — so preview and executor can't drift on how the
sources list is parsed (root cause of the earlier JSON-quote preview bug).
---------
Co-authored-by: Claude <noreply@anthropic.com>
calebfavor added a commit to railroadmedia/MusoraOpenKB that referenced this pull request Jul 2, 2026
Implements docs/smart-hierarchy-distillation-plan.md — a RAPTOR-style bottom-up
distillation that builds a multi-layer, LLM-navigable pathway hierarchy over the
flat concept leaves, the intended pivot from the top-down bootstrap() cold-start.
Engine (openkb/topic_tree.py):
- distill(): reads leaves recursively, clusters each layer into LLM-named sized
categories, summarizes each into a parent pathway node, links same-layer peers
sideways, repeats to a single root. Invariants enforced: exactly one root,
always >= 2 layers, bounded depth, no concept loss. Builds into a staging dir
and atomically swaps in (mid-build LLM failure never loses concepts).
- write_pathway_md(): pathway node format — layer/children/related frontmatter +
distilled summary + linked child index + Related pathways section.
- Sideways links are bidirectional, same-layer, top-K, no self-links.
Config (openkb/config.py): HierarchyConfig + resolve_hierarchy() for the
`hierarchy:` block (target/min/max fanout, max_depth, summary token caps,
sideways settings) with validation + back-compat.
LLM callables (openkb/topic_tree_llm.py): make_distill_cluster (AGENTS.md-guided,
sized category naming), make_distill_summarize, make_relate.
Integration: `openkb distill` CLI command; AGENTS.md `## Hierarchy` guidance
section injected into distill prompts; query tree-descent prompt now follows
`related` sideways links; lint registers topic-dir names so pathway/sideways
wikilinks resolve.
Tests (+31): config parsing, distill invariants/edges/sideways/data-loss,
fake-LLM CLI integration + idempotent re-distill, and tier-4 regression pins
(VectifyAI#4 sideways links resolve — red without the lint change; VectifyAI#5 single-root/
min-2-layer edge sizes). Full suite: 937 passed, 10 llm deselected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
KylinMountain added a commit that referenced this pull request Jul 21, 2026
…es (#198)
* feat(api): delete a knowledge base (POST /api/v1/kb/delete + openkb delete-kb)
Physical, irreversible KB deletion with a type-the-name confirmation.
- config.delete_kb: rmtree the KB directory + unregister it from the global
registry (known_kbs / kb_aliases / default_kb, via the new unregister_kb).
Guards against deleting a non-KB path; tolerates a ghost registry entry
(directory already gone) by just unregistering.
- POST /api/v1/kb/delete: confirm_name must equal kb, re-checked server-side.
Lives in a new api_kbs_router (sibling of the config/graph/output routers) so
api.py stays under the 800-line gate; delete_kb self-locks (config lock), no
create_app closure needed.
- openkb delete-kb NAME: type-the-name prompt (or --yes) then delete.
Tests: endpoint (success + unregister, confirm mismatch, non-KB target) in
test_api.py; config (physical delete + unregister, refuse non-KB, ghost
tolerance) in test_config.py.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): delete a wiki page (POST /api/v1/page/delete) with backlink impact
Delete a concept/entity page and degrade references cleanly instead of leaving
them dangling:
- page_ops.delete_wiki_page: dry_run reports the backlink pages (whose inbound
[[wikilinks]] would be demoted) without touching anything; execute removes the
page under the KB ingest lock, strips its index.md entry outright
(compiler.remove_doc_from_index — the entry is a [[link]] line), and demotes
the now-dangling inbound [[links]] to plain text in ONLY those backlink pages
(lint.fix_broken_links restrict_to). The page ref is "section/stem", validated
to concepts/ or entities/ only (path-traversal-safe).
- New api_pages_router hosts POST /api/v1/page (read — relocated here from
api.py to keep it under the 800-line gate) + POST /api/v1/page/delete.
Tests: dry-run impact + execute (page gone, inbound link demoted to text, index
entry removed, sibling kept), 404 on missing page, and rejection of unsafe /
non-editable refs.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): edit a wiki page (PUT /api/v1/page) + link context (POST /api/v1/page/links)
- page_ops.edit_wiki_page: replace a concept/entity page's BODY while preserving
its code-managed OKF frontmatter (type/description/sources) verbatim; any
frontmatter in the submitted content is dropped. Dead [[wikilinks]] the user
typed are demoted to plain text on save (OpenKB keeps the wiki broken-link-free)
and returned as ghosts_stripped so the UI can warn. Atomic write under the KB
ingest lock.
- page_ops.page_link_context (POST /api/v1/page/links): outbound + inbound links
for the edit-impact panel. Editing the body does not break either (links are
path-based), so this is context, not a blocker — the honest impact model.
- PUT /api/v1/page added to api_pages_router.
Tests: edit preserves frontmatter + keeps a resolvable link + demotes a dead one
(reports ghosts_stripped) + replaces the body; 404 on a missing page; links
reports out/backlinks.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): delete a knowledge base from the settings sheet (type-name confirm)
Adds a Danger Zone section to KbSettingsSheet: a type-the-name confirmation
gates deleteKb() (POST /api/v1/kb/delete); on success KbDetail navigates back
to the KB list. New deleteKb client + kbSettings danger* keys + common cancel
(zh + en, identical key sets).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): in-reader page edit + delete with impact preview (F2/F3)
For an open concept/entity page, the reader header now shows Edit and Delete:
- Delete: a dry-run first (deletePage dryRun) surfaces the backlink pages whose
[[links]] will demote to plain text; a red confirm card lists them, then the
real delete closes the page and refreshes the inventory.
- Edit: a body textarea (frontmatter is preserved server-side), Save via
PUT /api/v1/page, a links panel (getPageLinks: out/backlinks), a "recompile
may overwrite" note, and a toast listing any dead links demoted to text.
Adds the deletePage/editPage/getPageLinks wiki clients and kb:pageOps.* locale
keys (zh + en, identical key sets). Build green (i18n guard + tsc + vite).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(workbench): address code-review findings (content-mgmt correctness + concurrency)
Backend (page_ops / kb_admin / lint):
- delete/edit now do their read-modify-write INSIDE the KB ingest lock and
re-check the page exists under it: no stale backlink snapshot, no resurrecting
a concurrently-deleted page, no 500 on a racing unlink (missing_ok=True). [#4,#5,#6]
- delete_kb serializes rmtree under the ingest lock (was fully unlocked). Moved
delete_kb/unregister_kb into a new openkb/kb_admin.py so config.py drops back
under the per-file line gate (751 lines). [#1,#15]
- page deletion no longer scans or rewrites lint's _EXCLUDED_FILES
(AGENTS.md/SCHEMA.md/log.md); fix_broken_links(restrict_to) also refuses them,
which protects `openkb remove` too. [#2]
- edit_wiki_page treats `content` as the body verbatim — no naive frontmatter
split that silently dropped a body opening with a '---' block. [#3]
- delete uses the real on-disk stem for the exact-match index-entry removal
(case-insensitive FS), and adds index.md to the demotion set so a [[target]]
embedded in another entry's brief no longer dangles. [#7,#9]
API / CLI:
- delete-KB endpoint & CLI can now clean a ghost registry entry (registered name
whose directory was removed by hand), catch delete_kb's ValueError -> 400, and
the endpoint is typed `-> KbDeleteResponse`. [#8,#13,E5]
Frontend (KbDetail):
- Deleting a KB dispatches `openkb:reload-kbs` so the sidebar refreshes
immediately (was stale until a manual reload) — live-test fix.
- Removed the dead `pageReloadSeq` path (editPage always returns content). [#14]
Tests: excluded-doc-untouched, '---'-body preservation, ghost-KB unregister;
non-KB-target test updated to the new resolution model. Backend 1234 passed,
mypy/ruff clean, frontend build green.
Not changed (intentional): pages_linking_to is NOT folded into build_graph (that
excludes explorations/ backlinks) [#10]; the preview+confirm delete inherently
scans per request [#11]; EDITABLE_SECTIONS stays duplicated in the frontend,
which can't import the Python constant [#12].
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(workbench): allow editing summary pages (edit ⊇ delete section sets)
Summaries are compiled markdown like concepts/entities (a recompile regenerates
them), so they are now editable via PUT /api/v1/page. They remain NOT
independently deletable — a summary is removed by deleting its source document.
Split page_ops into EDITABLE_SECTIONS (concepts/entities/summaries) vs
DELETABLE_SECTIONS (concepts/entities); validate_page_ref takes the allowlist.
Frontend ReaderBody gates Edit on canEdit (incl. summaries), Delete on canDelete.
Test: summary editable (frontmatter preserved) + summary delete rejected (400).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(api): cross-platform delete_kb + endpoint OSError handling (xhigh review #1,#2)
- delete_kb no longer holds the ingest lock DURING rmtree (the lock file lives
inside the KB dir; Windows cannot delete an open file — the prior review-fix
regressed this). It now takes the lock as a BARRIER (drain + wait out any
in-flight mutation), releases it, re-checks existence, then rmtrees. [#1]
- delete-KB endpoint maps FileNotFoundError to an idempotent success (a
concurrent delete already removed the tree) and other OSError to a clean 500
with a message, instead of an uncaught 500 stack trace. [#2]
Tests: endpoint OSError -> clean 500, FileNotFoundError -> 200 deleted.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…y-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…ane (#197) (#199)
* feat(web): read a document's converted source text in the Documents pane (#197)
The Documents pane listed each ingested doc (name/type/hash) with no way to
read the converted full text ingestion produced under wiki/sources/.
Backend: new POST /api/v1/document/source resolves a doc hash to its source
text — short docs read <doc_name>.md; long docs concatenate the per-page
<doc_name>.json (page text joined by a thematic break). Hash is the identifier
(unique, avoids doc_name/stem collisions); resolution prefers the registry's
stored source_path then falls back to the wiki/sources/<doc_name>.{md,json}
convention, with a path-traversal guard. Read-only (sources are do-not-edit).
Frontend: document rows are now clickable and open a wide read-only slide-out
reader (MarkdownView) — ESC/overlay/close to dismiss, independent scroll,
content cached + memoized per hash, native find-in-page preserved (no
virtualization). Delete stays inline (stopPropagation). Closed drawer is inert.
Known limitation: images embedded in long-doc pages are not rendered inline yet.
* fix(web): address xhigh code-review findings for the document reader (#197)
Correctness / a11y:
- Rebuild the reader drawer on Radix Dialog (like KbSettingsSheet) instead of
a hand-rolled overlay: proper modal focus trap, initial + return focus,
Escape, and background inert (was: aria-modal with none of it) [#4]. This
also removes the hand-rolled window keydown listener that re-subscribed every
render [#7].
- Restructure each document row so the open-reader target is a real <button>
and the delete control is a SIBLING, not nested. Keyboard-activating delete
no longer bubbles into opening the reader, and the invalid nested-interactive
markup is gone [#1, #5].
- Resolve a source file by the doc's own type (long → .json first, else .md),
so two docs sharing a doc_name each resolve to their own file rather than
whichever extension is tried first [#2].
- Guard source reads: skip non-dict page entries, reject non-list JSON, and
return a controlled 500 on corrupt/unreadable sources instead of an
unhandled exception [#3].
- Invalidate the per-hash content cache when the inventory changes, so a
reopen after recompile refetches instead of serving stale text [#6].
Fetch/cache/memoized body moved to DocumentsPane so they survive the drawer's
unmount-on-close. #8 (frontmatter stripping) intentionally not applied: source
docs render verbatim (a user's own frontmatter is content, unlike wiki-page OKF
metadata). Adds tests for the collision and malformed-JSON paths.
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…verride) (#200)
* feat(api): configurable entity_types (global + per-KB) via the config API
entity_types (the entity-extraction vocabulary) is now surfaced through the
config read/write API, layering global.yaml -> KB config.yaml like the other
scalars: a KB list overrides the global list wholesale, an explicit null
inherits, and unset falls back to DEFAULT_ENTITY_TYPES. The compiler already
consumed config["entity_types"] via resolve_entity_types; this just exposes it.
- GLOBAL_SCALAR_KEYS gains "entity_types" (layering + per-key `sources` tracking;
the value-not-None-wins rule is type-agnostic, so it works for a list).
- _KbConfigWritable / GlobalConfigValues / KbConfigResponse / GlobalConfigResponse
carry entity_types; read_kb_config/read_global_config report the cleaned
EFFECTIVE list (resolve_entity_types) plus the raw global value for the badge.
- PATCH /api/v1/kb/config and PATCH /api/v1/config accept entity_types.
Tests: KB override (cleaned + source 'kb') + null revert, global patch, global
inheritance; updated the global-defaults shape assertion. Frontend UI follows.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): entity-types config UI — chips editor (global default + per-KB override)
- EntityTypesEditor: shared controlled chips editor (Enter/comma to add, x to
remove; "other" is a fixed always-included chip; IME-safe composition).
- KbSettingsSheet: an EntityTypesRow with the same inherit/override Switch as the
scalar rows — turning override on seeds+persists the KB's own list, off reverts
via null; inherited state shows the global/default list as a badge. Each chip
change persists and adopts the server-cleaned response.
- Settings (general tab): a global entity-types chips editor, order-sensitive
diff into the save patch (joins the existing dirty/SaveBar flow).
- "changes affect future recompiles only" note on both surfaces.
New keys in common/kbSettings/settings (zh + en, identical sets). Build green
(i18n guard OK). Backend was committed in 92f8f41.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(entity-types): address xhigh review — DEFAULT_CONFIG parity, empty-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@KylinMountain@rejojer@zmtomorrow
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex by KylinMountain · Pull Request #4 · VectifyAI/OpenKB · GitHub
Skip to content

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex - #4

Merged
rejojer merged 102 commits into
mainfrom
dev
Apr 8, 2026
Merged

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex#4
rejojer merged 102 commits into
mainfrom
dev

Conversation

@KylinMountain

Copy link
Copy Markdown
Collaborator

Summary

OpenKB — Karpathy's LLM Knowledge Base workflow as a CLI, powered by PageIndex.

Drop documents in. Get an auto-maintained, cross-linked wiki out.

Features

  • okb init — Interactive setup
  • okb add — Short docs (pymupdf) + long PDFs (PageIndex local/cloud)
  • okb query — Streaming Q&A with PageIndex cloud streaming
  • okb watch — Auto-compile on file changes
  • okb lint — Structural + knowledge health checks
  • okb list / status — Knowledge base overview
  • Obsidian compatible wiki output

Tech Stack

PageIndex, markitdown, OpenAI Agents SDK, LiteLLM, Click, watchdog

KylinMountainand others added 30 commits April 6, 2026 23:13
Sets up pyproject.toml (hatchling, direct-refs allowed, Python >=3.11),
.gitignore, openkb/__init__.py, a Click CLI stub with all 7 commands
(init, add, query, watch, lint, list, status), and tests/conftest.py
with kb_dir and sample_tree fixtures. Package installs cleanly in a
Python 3.12 venv; okb --help shows all commands; pytest collects 0
tests without error.
Add openkb/config.py (DEFAULT_CONFIG, load_config, save_config),
openkb/state.py (HashRegistry with SHA-256 file hashing and JSON
persistence), and openkb/schema.py (SCHEMA_MD constant). All 17 tests
written first (red) then implemented (green).
Creates full KB directory structure (raw/, wiki/sources/images/,
wiki/summaries/, wiki/concepts/, wiki/reports/), writes SCHEMA.md,
index.md, config.yaml and hashes.json; guards against re-initialisation.
Three tests in tests/test_cli.py cover structure, schema content, and
the already-initialized guard, all via CliRunner.isolated_filesystem.
Implements extract_base64_images and copy_relative_images with full test
coverage for single/multiple images, invalid base64, missing files, and
URL filtering.
Implements ConvertResult dataclass, get_pdf_page_count, and
convert_document with hash-dedup, markdown passthrough, PDF long-doc
detection, MarkItDown conversion, and image extraction integration.
Implements render_source_md and render_summary_md with YAML frontmatter,
recursive heading hierarchy (h1–h6 capped), page ranges, and separate
text/summary views for source and summary wiki pages.
Implements IndexResult dataclass and index_long_document which creates
a LocalClient with full node text/summary/description flags, adds the
PDF via PageIndex, fetches structure, and writes source and summary
wiki pages via the tree renderer.
Implements list_wiki_files, read_wiki_file, and write_wiki_file as plain
functions in openkb/agent/tools.py without @function_tool decoration,
ready to be wrapped when building the agent. Full test coverage including
edge cases for missing files/dirs, filtering to .md only, and parent dir
creation.
Implements build_compiler_agent, compile_short_doc, compile_long_doc in
openkb/agent/compiler.py with function_tool-wrapped wiki tools and
SCHEMA_MD-enriched instructions. Long-doc variant includes get_page_content.
Tests mock Runner.run to avoid real LLM calls.
Replaces the add stub with full orchestration: convert_document,
index_long_document for long PDFs, and compiler agent calls.
Adds SUPPORTED_EXTENSIONS set, _find_kb_dir, _add_single_file helpers.
Adds python-dotenv dependency and load_dotenv() at startup.
Implements pageindex_retrieve (structure -> LLM relevance -> page fetch),
build_query_agent with list/read/retrieve tools, and run_query coroutine.
Wires up `okb query` in cli.py.
Implements DebouncedHandler (collects events, ignores dirs/dotfiles, resets
timer on burst) and watch_directory (Observer loop, Ctrl+C safe).
Wires up `okb watch` in cli.py.
Implements find_broken_links, find_orphans, find_missing_entries,
check_index_sync, and run_structural_lint with full Markdown report.
Covers wikilink resolution, orphan detection, raw/wiki entry matching,
and index.md sync checking.
Implements build_lint_agent with list/read tools and instructions for
semantic quality checks (contradictions, gaps, staleness, redundancy).
run_knowledge_lint runs the agent and returns the report string.
okb lint combines structural + knowledge lint and writes timestamped report.
Tests verify list shows documents table and concepts, status shows
per-directory file counts and total indexed. Both check missing-init guard.
Previously the converter registered the file hash immediately, so if
LLM compilation failed the file was marked as "done" and retries
would skip it. Now the hash is only registered by the CLI after
successful compilation.
Also: install markitdown[all] for PDF support, add python-dotenv.
…pport
- Switch from col._backend.get_document_structure() to col.get_document_structure()
- Add 3x retry for PageIndex indexing (stochastic TOC accuracy)
- Fix storage path to use .db extension
- Remove .doc from supported extensions (markitdown only supports .docx)
- Note: col.get_page_content() still missing from PageIndex public API,
using col._backend.get_page_content() as workaround
Replace col._backend.get_page_content(col._name, doc_id, spec) with
col.get_page_content(doc_id, spec). Now all PageIndex access uses
public API only.
rejojer added 15 commits April 8, 2026 05:01
Rename CLI command and state dir from okb to openkb
- Hardcode reading LLM_API_KEY env var instead of indirecting through config
- Remove llm_api_key_env from DEFAULT_CONFIG, okb init prompts, and config.yaml
- Provider-specific env vars (OPENAI_API_KEY, etc.) still work via LiteLLM auto-detection
- One less config field, one less okb init step
The OpenAI Agents SDK requires a litellm/ prefix to route non-OpenAI
models through LiteLLM. Without it, models like anthropic/claude-sonnet-4-6
fail with "Unknown prefix". This adds the prefix at all Agent() call sites
while keeping litellm.completion() calls unchanged.
Also updates README quick start comments and model format docs.
Fix: add litellm/ prefix for Agents SDK model routing
@KylinMountain

Copy link
Copy Markdown
CollaboratorAuthor

Code review

Found 1 issue:

  1. extract_pdf_images and convert_pdf_with_images in images.py open pymupdf documents with explicit .close() instead of context managers. If an exception is raised during page iteration (e.g. corrupt image block, pixmap allocation failure), the PDF file handle leaks. This is the same bug pattern that was already fixed in converter.py:get_pdf_page_count (commit c525455), but images.py was missed. Fix: replace doc = pymupdf.open(...) / doc.close() with with pymupdf.open(...) as doc:.

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] !=1: # not an image block
continue
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
save_path=images_dir/filename
pix.save(str(save_path))
pix=None
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
continue
rel_path=f"images/{doc_name}/{filename}"
page_images.setdefault(page_num, []).append(rel_path)
doc.close()
returnpage_images

OpenKB/openkb/images.py

Lines 89 to 125 in 1637697

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
parts.append(f"\n\n<!-- Page {page_num} -->\n")
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] ==0: # text block
lines= []
forlineinblock["lines"]:
spans_text="".join(span["text"] forspaninline["spans"])
lines.append(spans_text)
parts.append("\n".join(lines))
elifblock["type"] ==1: # image block
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
(images_dir/filename).write_bytes(pix.tobytes("png"))
pix=None
parts.append(f"\n![image](images/{doc_name}/{filename})\n")
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
doc.close()
return"\n".join(parts)

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer

Copy link
Copy Markdown
Member

Code review

Found 2 issues:

  1. README claims LiteLLM is "pinned to a safe version" but pyproject.toml has no version pin. Line 67 of README.md states LiteLLM is (pinned to a safe version), but pyproject.toml line 17 lists the dependency as bare "litellm" with no version constraint (==, >=, ~=, etc.). Any version -- including potentially insecure ones -- can be installed.

OpenKB/README.md

Lines 66 to 68 in 854294c

OpenKB comes with [multi-LLM support](https://docs.litellm.ai/docs/providers) (e.g., OpenAI, Claude, Gemini) via [LiteLLM](https://github.com/BerriAI/litellm) (pinned to a [safe version](https://docs.litellm.ai/blog/security-update-march-2026)).

OpenKB/pyproject.toml

Lines 16 to 18 in 854294c

"watchdog>=3.0",
"litellm",
"openai-agents",

  1. test_short_pdf_converted_via_markitdown mocks the wrong code path. The test patches openkb.converter.MarkItDown and openkb.converter.pymupdf.open, but converter.py line 99-101 routes short PDFs through convert_pdf_with_images() (from openkb.images), not MarkItDown. The MarkItDown mock is never exercised, and convert_pdf_with_images is not mocked, so the test either fails at runtime or passes for the wrong reasons.

classTestConvertDocumentPdfShort:
deftest_short_pdf_converted_via_markitdown(self, kb_dir, tmp_path):
"""PDF under threshold is converted with markitdown."""
src=tmp_path/"short.pdf"
src.write_bytes(b"%PDF-1.4 fake content")
fake_result=MagicMock()
fake_result.text_content="# Short PDF\n\nConverted content."
with (
patch("openkb.converter.pymupdf.open") asmock_mu,
patch("openkb.converter.MarkItDown") asmock_mid_cls,
):
fake_doc=MagicMock()
fake_doc.page_count=5# below default threshold of 20
fake_doc.__enter__=MagicMock(return_value=fake_doc)
fake_doc.__exit__=MagicMock(return_value=False)
mock_mu.return_value=fake_doc
mock_mid_cls.return_value.convert.return_value=fake_result
result=convert_document(src, kb_dir)
assertresult.skippedisFalse
assertresult.is_long_docisFalse
assertresult.source_pathisnotNone
assertresult.source_path.exists()

markdown=copy_relative_images(markdown, src.parent, doc_name, images_dir)
elifsrc.suffix.lower() ==".pdf":
# Use pymupdf dict-mode for PDFs: text + images inline at correct positions
markdown=convert_pdf_with_images(src, doc_name, images_dir)
else:

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer
rejojer merged commit f0963f6 into mainApr 8, 2026
KylinMountain added a commit that referenced this pull request May 24, 2026
Architectural review (4 parallel Opus auditors) found that the skill_runner
core was already generic, but the deck SURFACE was still fused to
Editorial Monocle. Fixed:
* validator: now takes optional `grammar` param (DeckGrammar TypedDict);
skill-agnostic by default (only checks file present, parses, ≥5
slides, self-contained). Third-party deck skills (guizang, swiss)
now pass validation cleanly. Editorial-specific rules opt-in via
`EDITORIAL_MONOCLE_GRAMMAR`. (finding #2)
* skills/openkb-deck-editorial/SKILL.md: declares its grammar +
output_path_template under `od:` frontmatter — `run_skill` reads
these and applies them post-run.
* run_skill: now honors frontmatter `od.mode`, `od.output_path_template`,
`od.deck_grammar`. When mode=="deck" and template is set, the runner
injects the path into intent, verifies the file exists post-run, and
runs validate_deck with the skill's grammar. Validation result is
returned via new SkillRunResult dataclass. (findings #4, #5)
* `openkb deck new --skill <name>`: CLI flag accepts any installed deck
skill (default openkb-deck-editorial). guizang and swiss now usable
from the scripted CLI, not only freeform chat. (finding #1)
* `/deck new --skill <name>` chat slash: same flag, parsed positionally
alongside --critique. (finding #1)
* tests/test_read_kb_file.py: 13 new tests mirroring test_write_kb_file
for the read-side allow-list. Pins refusal of `.openkb/config.yaml`,
`.env`, `raw/`, `..` traversal, absolute paths. (finding #6)
* Generator deck branch: no longer calls validate_deck directly; just
propagates run_deck_create's SkillRunResult.validation up. Validation
is now a property of "this skill declared mode=deck", not of "this
CLI path was taken".
Existing tests updated:
* tests/test_deck_validator.py: explicit grammar arg on Editorial-
specific tests; added test_guizang_shape_passes_generic_mode +
test_missing_cover_ignored_in_generic_mode to pin both modes.
* tests/test_deck_creator.py: mocks return SkillRunResult; new
test_run_deck_create_honors_skill_name_override for --skill flag.
* tests/test_generator.py: deck dispatch test mocks SkillRunResult.
Below-threshold findings deferred:
* Generator if/else → registry (score 70) — works, just not extensible
via plugin; future.
* Iteration backup in chat freeform path (score 75) — needs write_kb_file
hook; separate change.
* run_skill / scan_local_skills / _handle_slash_critique direct tests
(scores 60-70) — covered indirectly by integration; can add later.
Regression: 538 tests pass (was 523 pre-fix; net +15 = 13 new
read_kb_file tests + 2 new validator-mode tests).
KylinMountain added a commit that referenced this pull request May 31, 2026
…lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
KylinMountain added a commit that referenced this pull request Jun 1, 2026
…mpile backfill (#78)
* feat(compiler): _read_entity_briefs for entity plan context
* test(compiler): parity tests for _read_entity_briefs
* feat(compiler): _write_entity with type/aliases frontmatter
* test(compiler): assert source ordering in _write_entity; count=1 in _set_fm_line
Add explicit ordering assertion in test_update_prepends_source_keeps_type
verifying the deterministic json.dumps form ("summaries/b.md", "summaries/a.md").
Pass count=1 to re.sub in _set_fm_line to make first-occurrence intent explicit.
* feat(lint): include entities/ in wikilink whitelist
* feat(compiler): summary<->entity backlinks
* test(compiler): restore assertion erroneously deleted in 3c8aa93
* feat(compiler): index.md Entities section
* feat(compiler): remove_doc_from_entity_pages + index cleanup
* feat(compiler): plan prompt + parser for entities group
Also wires the entity track into _compile_concepts (Tasks 7 + 8 combined,
since the {entity_briefs} placeholder and the _CONCEPTS_PLAN_USER.format call
are co-dependent — splitting would leave an intermediate red state).
- add _ENTITY_TYPES, _filter_entity_items, _parse_entities_plan
- rewrite _CONCEPTS_PLAN_USER to request nested concepts+entities groups
- add _ENTITY_PAGE_USER / _ENTITY_UPDATE_USER prompts
- read entity briefs and pass both briefs to the plan prompt
- parse nested 'concepts' group with legacy flat-list/flat-dict fallbacks
- generate entities in their own asyncio.gather (4-arity tuples)
- strip ghost links + _write_entity each; handle entity related cross-links
- backlink summary<->entities; pass entity_names/entity_meta to _update_index
* fix(compiler): related entities must not downgrade index labels
Mirror the concept track: collect related-entity slugs into a separate
local list used only for backlinks; pass only created/updated entity_names
(+entity_meta) to _update_index. Defense-in-depth in _update_index: only
_replace_section_entry when name is in entity_meta, otherwise only insert
if the link is absent, so a related-only entity can never clobber a
pre-existing correct (type + brief) index line with "(other)".
Adds regression test test_related_entity_does_not_downgrade_index_label.
* feat(schema): declare entities/ page type and taxonomy
* feat(query): point who/what questions at entities/
* docs(readme): document entities/ page type
* feat(cli): scaffold entities/ in init and count it in status
- `openkb init` now creates wiki/entities/ alongside wiki/concepts/
- init seed index.md gains ## Entities between ## Concepts and ## Explorations,
matching the _update_index template in compiler.py
- print_status subdirs list gains "entities" after "concepts"
- Tests updated: assert wiki/entities/ exists and index.md contains ## Entities;
status test asserts "entities" appears in output
* fix(compiler): resolve entity-page review findings (dangling links + dedup)
Addresses code-review findings on the entity-pages feature:
- Fix dangling wikilink after `openkb remove`: entity removal now strips
standalone `See also: [[summaries/{doc}]]` lines (the related-entity
backlink form), matching the concept path, and cli.py adds modified
entity pages to the lint sweep scope so surviving pages are cleaned.
- Unify the parallel concept/entity helpers into shared cores
(_backlink_summary_pages, _backlink_pages, _remove_doc_from_pages) with
thin per-type wrappers, so cleanup logic can no longer drift between the
two page types (this is what caused the dangling-link bug).
- Route related-entity cross-refs through _add_related_link (now page-type
aware) instead of an inline reimplementation — removes a duplicate file
read/write and keeps backlink creation symmetric with teardown.
- Centralize the entity-type enum: prompts derive their type list from a
single _ENTITY_TYPE_LIST source via import-time substitution.
- Count entity items in the "all dropped as malformed" plan warning.
- Drop the unreachable else branch in _update_index's entity loop.
- Add regression test for the See-also strip on a surviving entity page.
All 542 tests pass.
* fix(compiler): add [[entities/X]] whitelist rule + restore concept-topic guard
Remaining review findings after a7a06ed:
- _KNOWN_TARGETS_USER now states the [[entities/Z]] rule, so entity links
the LLM is told to write aren't silently stripped as ghosts.
- Restore the dropped 'Do NOT create concepts that are just the document
topic itself' plan rule to prevent redundant title-mirror concepts.
* feat(entities): shared page-dir constants + surface entities in list/lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
* feat(entities): remove preview lists entity-page actions (#1)
The dry-run/confirmation block now scans wiki/entities/ with the same
frontmatter sources: logic as concepts, emits DELETE/MODIFY action lines
per entity page, and prints an 'N entity(s) will be DELETED' summary.
Execution path (remove_doc_from_entity_pages) unchanged.
* docs(entities): document entity pages in shipped openkb skill (#8)
Note wiki/entities/ holds named-thing pages (people/orgs/places/
products/works/events) with a type: frontmatter field, that index.md
has a ## Entities section, and that 'who/what is X' questions should
read the matching entities/ page first.
* fix(compiler): don't write raw JSON body on empty LLM content
In the parse-succeeded branch of _gen_create/_gen_update/_gen_entity_create/
_gen_entity_update, fall back to "" instead of the raw JSON string when the
content field is empty/null. _require_nonempty_content then raises and the
page is dropped, rather than writing the JSON envelope as the markdown body.
The parse-FAILED (except) branch keeps content=raw as the legitimate
non-JSON fallback.
* fix(compiler): graceful scalar plan + rebuild malformed entity frontmatter
- _compile_concepts: guard a non-dict/non-list parsed plan (JSON scalar)
before calling .get(), taking the empty-plan path (write v1 summary if
applicable + update index + return) instead of risking AttributeError.
- _write_entity: when an existing page has an opening --- but no closing
delimiter (or no frontmatter), rebuild valid sources/type/brief frontmatter
rather than writing a body-only page that drops the metadata.
* fix(compiler): keep ## Entities before ## Explorations; drop dead param + overlap gathers
- _update_index: insert ## Entities before ## Explorations on older index.md
files that predate the section (new _ensure_h2_section_before helper),
preserving canonical order instead of appending at EOF.
- _filter_entity_items: drop the unused 'label' parameter and update call
sites in _parse_entities_plan.
- _compile_concepts: overlap concept and entity generation in one outer
asyncio.gather (they share cached context and the same concurrency
semaphore); result/error handling per list is unchanged.
* test(compiler): cover empty-content skip, scalar plan, malformed entity FM, Entities order
Add regression tests for the four compiler fixes:
- empty {"content":""} response skips the page (no raw JSON body)
- JSON scalar plan handled gracefully (no AttributeError)
- _write_entity rebuilds frontmatter when closing --- is missing
- _update_index inserts ## Entities before ## Explorations
* fix(compiler): silence spurious 'hand-edited' warning on backlink section creation
_backlink_summary_pages / _backlink_pages create ## Entities / ## Related
Documents sections as a normal first-time operation; pass quiet=True so
_ensure_h2_section no longer logs the index-drift warning in that case.
Index-repair callers keep the warning.
* feat(cli): add `recompile` command to re-run compile on indexed docs
Re-runs the current compile_short_doc/compile_long_doc pipeline on
already-indexed docs so pre-feature KBs gain the entities/ layer and
refresh to the current format. Reuses on-disk sources/summaries and the
registry's PageIndex doc_id — does not re-index or re-convert.
Supports a positional <doc_name> (resolved via _resolve_doc_identifier)
or --all (with a regeneration-warning confirmation, bypassed by --yes),
--dry-run (enumerate only, no LLM calls/writes), and --refresh-schema
(back up + overwrite wiki/AGENTS.md when it differs from AGENTS_MD).
Processes docs sequentially with per-doc progress, skips+warns on
missing sources / summaries / doc_id, prints a recompiled/skipped
summary, and appends a recompile entry to log.md.
* test(cli): recompile dispatch/dry-run/skip/refresh-schema
* docs(readme): document openkb recompile
* fix(cli): recompile --refresh-schema no-ops when AGENTS.md absent; tighten guard tests
Match the spec (and the helper's own docstring): _refresh_schema returns
early when wiki/AGENTS.md is missing rather than materializing the default
(get_agents_md already falls back to it at runtime). Tighten the doc/--all
guard tests to assert the exact message + that no compile runs, and add the
missing-AGENTS.md no-op test.
* fix(compiler): drop non-existent 'related' slugs so they don't create dangling links
The plan's 'related' list is meant to reference existing pages, but the LLM
sometimes lists slugs for pages that don't exist. Those were added to the
wikilink whitelist (so body references survived ghost-stripping) and
back-linked into the summary's Related section, yet no page was ever created
(related items are linked, never generated) — producing a flood of broken
[[concepts/...]] / [[entities/...]] links (esp. on feature-dense docs).
Filter related_items / entity_related to slugs that exist on disk.
* fix: remove-preview detects JSON-quoted sources; _write_entity preserves sources on malformed FM
- remove --dry-run preview parsed the sources list with a hand-rolled comma
split that kept JSON quotes (["summaries/x.md"]), so the marker never
matched and the preview always reported 0 affected concept/entity pages
(executor was correct). Extract _scan_affected_pages using the real
_parse_yaml_list_value; dedups the two copied scan loops too.
- _write_entity's malformed-frontmatter rebuild seeded sources with only the
new doc, dropping prior sources for multi-source entities. Recover existing
sources from the broken block and merge.
Both bugs were masked by tests using unquoted / single-source fixtures.
* feat(cli): rename remove --keep-empty-concepts → --keep-empty (covers entities too)
This PR wired entity pages into 'openkb remove', so the flag now governs
concept AND entity retention — but the name still said 'concepts'. Make
--keep-empty the canonical name (clear that it covers both), keep
--keep-empty-concepts as a backward-compatible alias, and update the
preview/summary messages, docstring, and README accordingly.
* feat(compiler): config-driven entity types (entity_types overrides the default enum)
Add an optional 'entity_types:' key in .openkb/config.yaml. When present it
overrides the default person/organization/place/product/work/event/other
vocabulary everywhere — the plan prompt, the entity-page prompts, and
create/update validation/coercion; when absent, behavior is byte-identical.
Prompt templates keep an __ENTITY_TYPES__ token now substituted at call time
(per-KB) inside _compile_concepts, and the resolved valid-type set is threaded
into _parse_entities_plan / _filter_entity_items and the _gen_entity_* coercion.
'other' is always ensured as the coercion fallback; malformed config falls back
to the default with a warning. Documented in config.yaml.example + README.
* fix(compiler): harden config-driven entity types (crash-proof + complete the override)
Review of the config-entity-types feature surfaced two real issues:
- A config 'entity_types' value containing '{' or '}' was substituted into the
prompt template BEFORE .format() ran → KeyError/ValueError crashing every
compile. Swap to format-then-replace at all 3 call sites (types_str is now an
inert literal), and sanitize resolved types to a safe label charset (also
skips YAML nulls/ints so str(None) can't become the type 'none').
- The AGENTS_MD system schema hardcoded 'type: is one of: <7 defaults>',
contradicting a custom entity_types in the higher-weight system message.
Reword it to frame those as the configurable default and defer the
authoritative set to the compilation prompt (which is config-driven).
Also drop the now-dead _ENTITY_TYPES_STR + its stale import-time-substitution
comment. +2 regression tests (sanitization; brace-in-type doesn't crash).
* refactor: move entity-type resolution to config layer + co-locate remove-preview scan
Altitude cleanups from the review:
- Move resolve_entity_types + DEFAULT_ENTITY_TYPES into openkb/config.py (the
config layer owns config validation/normalization; any command can reuse it
without importing the heavy compiler module). compiler.py imports them;
_ENTITY_TYPE_LIST/_ENTITY_TYPES remain as the default alias/validation set.
- Move the remove dry-run preview scan from cli.py into compiler.py as
scan_affected_pages, beside remove_doc_from_*_pages and sharing
_parse_yaml_list_value — so preview and executor can't drift on how the
sources list is parsed (root cause of the earlier JSON-quote preview bug).
---------
Co-authored-by: Claude <noreply@anthropic.com>
calebfavor added a commit to railroadmedia/MusoraOpenKB that referenced this pull request Jul 2, 2026
Implements docs/smart-hierarchy-distillation-plan.md — a RAPTOR-style bottom-up
distillation that builds a multi-layer, LLM-navigable pathway hierarchy over the
flat concept leaves, the intended pivot from the top-down bootstrap() cold-start.
Engine (openkb/topic_tree.py):
- distill(): reads leaves recursively, clusters each layer into LLM-named sized
categories, summarizes each into a parent pathway node, links same-layer peers
sideways, repeats to a single root. Invariants enforced: exactly one root,
always >= 2 layers, bounded depth, no concept loss. Builds into a staging dir
and atomically swaps in (mid-build LLM failure never loses concepts).
- write_pathway_md(): pathway node format — layer/children/related frontmatter +
distilled summary + linked child index + Related pathways section.
- Sideways links are bidirectional, same-layer, top-K, no self-links.
Config (openkb/config.py): HierarchyConfig + resolve_hierarchy() for the
`hierarchy:` block (target/min/max fanout, max_depth, summary token caps,
sideways settings) with validation + back-compat.
LLM callables (openkb/topic_tree_llm.py): make_distill_cluster (AGENTS.md-guided,
sized category naming), make_distill_summarize, make_relate.
Integration: `openkb distill` CLI command; AGENTS.md `## Hierarchy` guidance
section injected into distill prompts; query tree-descent prompt now follows
`related` sideways links; lint registers topic-dir names so pathway/sideways
wikilinks resolve.
Tests (+31): config parsing, distill invariants/edges/sideways/data-loss,
fake-LLM CLI integration + idempotent re-distill, and tier-4 regression pins
(VectifyAI#4 sideways links resolve — red without the lint change; VectifyAI#5 single-root/
min-2-layer edge sizes). Full suite: 937 passed, 10 llm deselected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
KylinMountain added a commit that referenced this pull request Jul 21, 2026
…es (#198)
* feat(api): delete a knowledge base (POST /api/v1/kb/delete + openkb delete-kb)
Physical, irreversible KB deletion with a type-the-name confirmation.
- config.delete_kb: rmtree the KB directory + unregister it from the global
registry (known_kbs / kb_aliases / default_kb, via the new unregister_kb).
Guards against deleting a non-KB path; tolerates a ghost registry entry
(directory already gone) by just unregistering.
- POST /api/v1/kb/delete: confirm_name must equal kb, re-checked server-side.
Lives in a new api_kbs_router (sibling of the config/graph/output routers) so
api.py stays under the 800-line gate; delete_kb self-locks (config lock), no
create_app closure needed.
- openkb delete-kb NAME: type-the-name prompt (or --yes) then delete.
Tests: endpoint (success + unregister, confirm mismatch, non-KB target) in
test_api.py; config (physical delete + unregister, refuse non-KB, ghost
tolerance) in test_config.py.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): delete a wiki page (POST /api/v1/page/delete) with backlink impact
Delete a concept/entity page and degrade references cleanly instead of leaving
them dangling:
- page_ops.delete_wiki_page: dry_run reports the backlink pages (whose inbound
[[wikilinks]] would be demoted) without touching anything; execute removes the
page under the KB ingest lock, strips its index.md entry outright
(compiler.remove_doc_from_index — the entry is a [[link]] line), and demotes
the now-dangling inbound [[links]] to plain text in ONLY those backlink pages
(lint.fix_broken_links restrict_to). The page ref is "section/stem", validated
to concepts/ or entities/ only (path-traversal-safe).
- New api_pages_router hosts POST /api/v1/page (read — relocated here from
api.py to keep it under the 800-line gate) + POST /api/v1/page/delete.
Tests: dry-run impact + execute (page gone, inbound link demoted to text, index
entry removed, sibling kept), 404 on missing page, and rejection of unsafe /
non-editable refs.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): edit a wiki page (PUT /api/v1/page) + link context (POST /api/v1/page/links)
- page_ops.edit_wiki_page: replace a concept/entity page's BODY while preserving
its code-managed OKF frontmatter (type/description/sources) verbatim; any
frontmatter in the submitted content is dropped. Dead [[wikilinks]] the user
typed are demoted to plain text on save (OpenKB keeps the wiki broken-link-free)
and returned as ghosts_stripped so the UI can warn. Atomic write under the KB
ingest lock.
- page_ops.page_link_context (POST /api/v1/page/links): outbound + inbound links
for the edit-impact panel. Editing the body does not break either (links are
path-based), so this is context, not a blocker — the honest impact model.
- PUT /api/v1/page added to api_pages_router.
Tests: edit preserves frontmatter + keeps a resolvable link + demotes a dead one
(reports ghosts_stripped) + replaces the body; 404 on a missing page; links
reports out/backlinks.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): delete a knowledge base from the settings sheet (type-name confirm)
Adds a Danger Zone section to KbSettingsSheet: a type-the-name confirmation
gates deleteKb() (POST /api/v1/kb/delete); on success KbDetail navigates back
to the KB list. New deleteKb client + kbSettings danger* keys + common cancel
(zh + en, identical key sets).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): in-reader page edit + delete with impact preview (F2/F3)
For an open concept/entity page, the reader header now shows Edit and Delete:
- Delete: a dry-run first (deletePage dryRun) surfaces the backlink pages whose
[[links]] will demote to plain text; a red confirm card lists them, then the
real delete closes the page and refreshes the inventory.
- Edit: a body textarea (frontmatter is preserved server-side), Save via
PUT /api/v1/page, a links panel (getPageLinks: out/backlinks), a "recompile
may overwrite" note, and a toast listing any dead links demoted to text.
Adds the deletePage/editPage/getPageLinks wiki clients and kb:pageOps.* locale
keys (zh + en, identical key sets). Build green (i18n guard + tsc + vite).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(workbench): address code-review findings (content-mgmt correctness + concurrency)
Backend (page_ops / kb_admin / lint):
- delete/edit now do their read-modify-write INSIDE the KB ingest lock and
re-check the page exists under it: no stale backlink snapshot, no resurrecting
a concurrently-deleted page, no 500 on a racing unlink (missing_ok=True). [#4,#5,#6]
- delete_kb serializes rmtree under the ingest lock (was fully unlocked). Moved
delete_kb/unregister_kb into a new openkb/kb_admin.py so config.py drops back
under the per-file line gate (751 lines). [#1,#15]
- page deletion no longer scans or rewrites lint's _EXCLUDED_FILES
(AGENTS.md/SCHEMA.md/log.md); fix_broken_links(restrict_to) also refuses them,
which protects `openkb remove` too. [#2]
- edit_wiki_page treats `content` as the body verbatim — no naive frontmatter
split that silently dropped a body opening with a '---' block. [#3]
- delete uses the real on-disk stem for the exact-match index-entry removal
(case-insensitive FS), and adds index.md to the demotion set so a [[target]]
embedded in another entry's brief no longer dangles. [#7,#9]
API / CLI:
- delete-KB endpoint & CLI can now clean a ghost registry entry (registered name
whose directory was removed by hand), catch delete_kb's ValueError -> 400, and
the endpoint is typed `-> KbDeleteResponse`. [#8,#13,E5]
Frontend (KbDetail):
- Deleting a KB dispatches `openkb:reload-kbs` so the sidebar refreshes
immediately (was stale until a manual reload) — live-test fix.
- Removed the dead `pageReloadSeq` path (editPage always returns content). [#14]
Tests: excluded-doc-untouched, '---'-body preservation, ghost-KB unregister;
non-KB-target test updated to the new resolution model. Backend 1234 passed,
mypy/ruff clean, frontend build green.
Not changed (intentional): pages_linking_to is NOT folded into build_graph (that
excludes explorations/ backlinks) [#10]; the preview+confirm delete inherently
scans per request [#11]; EDITABLE_SECTIONS stays duplicated in the frontend,
which can't import the Python constant [#12].
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(workbench): allow editing summary pages (edit ⊇ delete section sets)
Summaries are compiled markdown like concepts/entities (a recompile regenerates
them), so they are now editable via PUT /api/v1/page. They remain NOT
independently deletable — a summary is removed by deleting its source document.
Split page_ops into EDITABLE_SECTIONS (concepts/entities/summaries) vs
DELETABLE_SECTIONS (concepts/entities); validate_page_ref takes the allowlist.
Frontend ReaderBody gates Edit on canEdit (incl. summaries), Delete on canDelete.
Test: summary editable (frontmatter preserved) + summary delete rejected (400).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(api): cross-platform delete_kb + endpoint OSError handling (xhigh review #1,#2)
- delete_kb no longer holds the ingest lock DURING rmtree (the lock file lives
inside the KB dir; Windows cannot delete an open file — the prior review-fix
regressed this). It now takes the lock as a BARRIER (drain + wait out any
in-flight mutation), releases it, re-checks existence, then rmtrees. [#1]
- delete-KB endpoint maps FileNotFoundError to an idempotent success (a
concurrent delete already removed the tree) and other OSError to a clean 500
with a message, instead of an uncaught 500 stack trace. [#2]
Tests: endpoint OSError -> clean 500, FileNotFoundError -> 200 deleted.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…y-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…ane (#197) (#199)
* feat(web): read a document's converted source text in the Documents pane (#197)
The Documents pane listed each ingested doc (name/type/hash) with no way to
read the converted full text ingestion produced under wiki/sources/.
Backend: new POST /api/v1/document/source resolves a doc hash to its source
text — short docs read <doc_name>.md; long docs concatenate the per-page
<doc_name>.json (page text joined by a thematic break). Hash is the identifier
(unique, avoids doc_name/stem collisions); resolution prefers the registry's
stored source_path then falls back to the wiki/sources/<doc_name>.{md,json}
convention, with a path-traversal guard. Read-only (sources are do-not-edit).
Frontend: document rows are now clickable and open a wide read-only slide-out
reader (MarkdownView) — ESC/overlay/close to dismiss, independent scroll,
content cached + memoized per hash, native find-in-page preserved (no
virtualization). Delete stays inline (stopPropagation). Closed drawer is inert.
Known limitation: images embedded in long-doc pages are not rendered inline yet.
* fix(web): address xhigh code-review findings for the document reader (#197)
Correctness / a11y:
- Rebuild the reader drawer on Radix Dialog (like KbSettingsSheet) instead of
a hand-rolled overlay: proper modal focus trap, initial + return focus,
Escape, and background inert (was: aria-modal with none of it) [#4]. This
also removes the hand-rolled window keydown listener that re-subscribed every
render [#7].
- Restructure each document row so the open-reader target is a real <button>
and the delete control is a SIBLING, not nested. Keyboard-activating delete
no longer bubbles into opening the reader, and the invalid nested-interactive
markup is gone [#1, #5].
- Resolve a source file by the doc's own type (long → .json first, else .md),
so two docs sharing a doc_name each resolve to their own file rather than
whichever extension is tried first [#2].
- Guard source reads: skip non-dict page entries, reject non-list JSON, and
return a controlled 500 on corrupt/unreadable sources instead of an
unhandled exception [#3].
- Invalidate the per-hash content cache when the inventory changes, so a
reopen after recompile refetches instead of serving stale text [#6].
Fetch/cache/memoized body moved to DocumentsPane so they survive the drawer's
unmount-on-close. #8 (frontmatter stripping) intentionally not applied: source
docs render verbatim (a user's own frontmatter is content, unlike wiki-page OKF
metadata). Adds tests for the collision and malformed-JSON paths.
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…verride) (#200)
* feat(api): configurable entity_types (global + per-KB) via the config API
entity_types (the entity-extraction vocabulary) is now surfaced through the
config read/write API, layering global.yaml -> KB config.yaml like the other
scalars: a KB list overrides the global list wholesale, an explicit null
inherits, and unset falls back to DEFAULT_ENTITY_TYPES. The compiler already
consumed config["entity_types"] via resolve_entity_types; this just exposes it.
- GLOBAL_SCALAR_KEYS gains "entity_types" (layering + per-key `sources` tracking;
the value-not-None-wins rule is type-agnostic, so it works for a list).
- _KbConfigWritable / GlobalConfigValues / KbConfigResponse / GlobalConfigResponse
carry entity_types; read_kb_config/read_global_config report the cleaned
EFFECTIVE list (resolve_entity_types) plus the raw global value for the badge.
- PATCH /api/v1/kb/config and PATCH /api/v1/config accept entity_types.
Tests: KB override (cleaned + source 'kb') + null revert, global patch, global
inheritance; updated the global-defaults shape assertion. Frontend UI follows.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): entity-types config UI — chips editor (global default + per-KB override)
- EntityTypesEditor: shared controlled chips editor (Enter/comma to add, x to
remove; "other" is a fixed always-included chip; IME-safe composition).
- KbSettingsSheet: an EntityTypesRow with the same inherit/override Switch as the
scalar rows — turning override on seeds+persists the KB's own list, off reverts
via null; inherited state shows the global/default list as a badge. Each chip
change persists and adopts the server-cleaned response.
- Settings (general tab): a global entity-types chips editor, order-sensitive
diff into the save patch (joins the existing dirty/SaveBar flow).
- "changes affect future recompiles only" note on both surfaces.
New keys in common/kbSettings/settings (zh + en, identical sets). Build green
(i18n guard OK). Backend was committed in 92f8f41.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(entity-types): address xhigh review — DEFAULT_CONFIG parity, empty-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@KylinMountain@rejojer@zmtomorrow
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex by KylinMountain · Pull Request #4 · VectifyAI/OpenKB · GitHub
Skip to content

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex - #4

Merged
rejojer merged 102 commits into
mainfrom
dev
Apr 8, 2026
Merged

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex#4
rejojer merged 102 commits into
mainfrom
dev

Conversation

@KylinMountain

Copy link
Copy Markdown
Collaborator

Summary

OpenKB — Karpathy's LLM Knowledge Base workflow as a CLI, powered by PageIndex.

Drop documents in. Get an auto-maintained, cross-linked wiki out.

Features

  • okb init — Interactive setup
  • okb add — Short docs (pymupdf) + long PDFs (PageIndex local/cloud)
  • okb query — Streaming Q&A with PageIndex cloud streaming
  • okb watch — Auto-compile on file changes
  • okb lint — Structural + knowledge health checks
  • okb list / status — Knowledge base overview
  • Obsidian compatible wiki output

Tech Stack

PageIndex, markitdown, OpenAI Agents SDK, LiteLLM, Click, watchdog

KylinMountainand others added 30 commits April 6, 2026 23:13
Sets up pyproject.toml (hatchling, direct-refs allowed, Python >=3.11),
.gitignore, openkb/__init__.py, a Click CLI stub with all 7 commands
(init, add, query, watch, lint, list, status), and tests/conftest.py
with kb_dir and sample_tree fixtures. Package installs cleanly in a
Python 3.12 venv; okb --help shows all commands; pytest collects 0
tests without error.
Add openkb/config.py (DEFAULT_CONFIG, load_config, save_config),
openkb/state.py (HashRegistry with SHA-256 file hashing and JSON
persistence), and openkb/schema.py (SCHEMA_MD constant). All 17 tests
written first (red) then implemented (green).
Creates full KB directory structure (raw/, wiki/sources/images/,
wiki/summaries/, wiki/concepts/, wiki/reports/), writes SCHEMA.md,
index.md, config.yaml and hashes.json; guards against re-initialisation.
Three tests in tests/test_cli.py cover structure, schema content, and
the already-initialized guard, all via CliRunner.isolated_filesystem.
Implements extract_base64_images and copy_relative_images with full test
coverage for single/multiple images, invalid base64, missing files, and
URL filtering.
Implements ConvertResult dataclass, get_pdf_page_count, and
convert_document with hash-dedup, markdown passthrough, PDF long-doc
detection, MarkItDown conversion, and image extraction integration.
Implements render_source_md and render_summary_md with YAML frontmatter,
recursive heading hierarchy (h1–h6 capped), page ranges, and separate
text/summary views for source and summary wiki pages.
Implements IndexResult dataclass and index_long_document which creates
a LocalClient with full node text/summary/description flags, adds the
PDF via PageIndex, fetches structure, and writes source and summary
wiki pages via the tree renderer.
Implements list_wiki_files, read_wiki_file, and write_wiki_file as plain
functions in openkb/agent/tools.py without @function_tool decoration,
ready to be wrapped when building the agent. Full test coverage including
edge cases for missing files/dirs, filtering to .md only, and parent dir
creation.
Implements build_compiler_agent, compile_short_doc, compile_long_doc in
openkb/agent/compiler.py with function_tool-wrapped wiki tools and
SCHEMA_MD-enriched instructions. Long-doc variant includes get_page_content.
Tests mock Runner.run to avoid real LLM calls.
Replaces the add stub with full orchestration: convert_document,
index_long_document for long PDFs, and compiler agent calls.
Adds SUPPORTED_EXTENSIONS set, _find_kb_dir, _add_single_file helpers.
Adds python-dotenv dependency and load_dotenv() at startup.
Implements pageindex_retrieve (structure -> LLM relevance -> page fetch),
build_query_agent with list/read/retrieve tools, and run_query coroutine.
Wires up `okb query` in cli.py.
Implements DebouncedHandler (collects events, ignores dirs/dotfiles, resets
timer on burst) and watch_directory (Observer loop, Ctrl+C safe).
Wires up `okb watch` in cli.py.
Implements find_broken_links, find_orphans, find_missing_entries,
check_index_sync, and run_structural_lint with full Markdown report.
Covers wikilink resolution, orphan detection, raw/wiki entry matching,
and index.md sync checking.
Implements build_lint_agent with list/read tools and instructions for
semantic quality checks (contradictions, gaps, staleness, redundancy).
run_knowledge_lint runs the agent and returns the report string.
okb lint combines structural + knowledge lint and writes timestamped report.
Tests verify list shows documents table and concepts, status shows
per-directory file counts and total indexed. Both check missing-init guard.
Previously the converter registered the file hash immediately, so if
LLM compilation failed the file was marked as "done" and retries
would skip it. Now the hash is only registered by the CLI after
successful compilation.
Also: install markitdown[all] for PDF support, add python-dotenv.
…pport
- Switch from col._backend.get_document_structure() to col.get_document_structure()
- Add 3x retry for PageIndex indexing (stochastic TOC accuracy)
- Fix storage path to use .db extension
- Remove .doc from supported extensions (markitdown only supports .docx)
- Note: col.get_page_content() still missing from PageIndex public API,
using col._backend.get_page_content() as workaround
Replace col._backend.get_page_content(col._name, doc_id, spec) with
col.get_page_content(doc_id, spec). Now all PageIndex access uses
public API only.
rejojer added 15 commits April 8, 2026 05:01
Rename CLI command and state dir from okb to openkb
- Hardcode reading LLM_API_KEY env var instead of indirecting through config
- Remove llm_api_key_env from DEFAULT_CONFIG, okb init prompts, and config.yaml
- Provider-specific env vars (OPENAI_API_KEY, etc.) still work via LiteLLM auto-detection
- One less config field, one less okb init step
The OpenAI Agents SDK requires a litellm/ prefix to route non-OpenAI
models through LiteLLM. Without it, models like anthropic/claude-sonnet-4-6
fail with "Unknown prefix". This adds the prefix at all Agent() call sites
while keeping litellm.completion() calls unchanged.
Also updates README quick start comments and model format docs.
Fix: add litellm/ prefix for Agents SDK model routing
@KylinMountain

Copy link
Copy Markdown
CollaboratorAuthor

Code review

Found 1 issue:

  1. extract_pdf_images and convert_pdf_with_images in images.py open pymupdf documents with explicit .close() instead of context managers. If an exception is raised during page iteration (e.g. corrupt image block, pixmap allocation failure), the PDF file handle leaks. This is the same bug pattern that was already fixed in converter.py:get_pdf_page_count (commit c525455), but images.py was missed. Fix: replace doc = pymupdf.open(...) / doc.close() with with pymupdf.open(...) as doc:.

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] !=1: # not an image block
continue
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
save_path=images_dir/filename
pix.save(str(save_path))
pix=None
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
continue
rel_path=f"images/{doc_name}/{filename}"
page_images.setdefault(page_num, []).append(rel_path)
doc.close()
returnpage_images

OpenKB/openkb/images.py

Lines 89 to 125 in 1637697

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
parts.append(f"\n\n<!-- Page {page_num} -->\n")
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] ==0: # text block
lines= []
forlineinblock["lines"]:
spans_text="".join(span["text"] forspaninline["spans"])
lines.append(spans_text)
parts.append("\n".join(lines))
elifblock["type"] ==1: # image block
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
(images_dir/filename).write_bytes(pix.tobytes("png"))
pix=None
parts.append(f"\n![image](images/{doc_name}/{filename})\n")
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
doc.close()
return"\n".join(parts)

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer

Copy link
Copy Markdown
Member

Code review

Found 2 issues:

  1. README claims LiteLLM is "pinned to a safe version" but pyproject.toml has no version pin. Line 67 of README.md states LiteLLM is (pinned to a safe version), but pyproject.toml line 17 lists the dependency as bare "litellm" with no version constraint (==, >=, ~=, etc.). Any version -- including potentially insecure ones -- can be installed.

OpenKB/README.md

Lines 66 to 68 in 854294c

OpenKB comes with [multi-LLM support](https://docs.litellm.ai/docs/providers) (e.g., OpenAI, Claude, Gemini) via [LiteLLM](https://github.com/BerriAI/litellm) (pinned to a [safe version](https://docs.litellm.ai/blog/security-update-march-2026)).

OpenKB/pyproject.toml

Lines 16 to 18 in 854294c

"watchdog>=3.0",
"litellm",
"openai-agents",

  1. test_short_pdf_converted_via_markitdown mocks the wrong code path. The test patches openkb.converter.MarkItDown and openkb.converter.pymupdf.open, but converter.py line 99-101 routes short PDFs through convert_pdf_with_images() (from openkb.images), not MarkItDown. The MarkItDown mock is never exercised, and convert_pdf_with_images is not mocked, so the test either fails at runtime or passes for the wrong reasons.

classTestConvertDocumentPdfShort:
deftest_short_pdf_converted_via_markitdown(self, kb_dir, tmp_path):
"""PDF under threshold is converted with markitdown."""
src=tmp_path/"short.pdf"
src.write_bytes(b"%PDF-1.4 fake content")
fake_result=MagicMock()
fake_result.text_content="# Short PDF\n\nConverted content."
with (
patch("openkb.converter.pymupdf.open") asmock_mu,
patch("openkb.converter.MarkItDown") asmock_mid_cls,
):
fake_doc=MagicMock()
fake_doc.page_count=5# below default threshold of 20
fake_doc.__enter__=MagicMock(return_value=fake_doc)
fake_doc.__exit__=MagicMock(return_value=False)
mock_mu.return_value=fake_doc
mock_mid_cls.return_value.convert.return_value=fake_result
result=convert_document(src, kb_dir)
assertresult.skippedisFalse
assertresult.is_long_docisFalse
assertresult.source_pathisnotNone
assertresult.source_path.exists()

markdown=copy_relative_images(markdown, src.parent, doc_name, images_dir)
elifsrc.suffix.lower() ==".pdf":
# Use pymupdf dict-mode for PDFs: text + images inline at correct positions
markdown=convert_pdf_with_images(src, doc_name, images_dir)
else:

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer
rejojer merged commit f0963f6 into mainApr 8, 2026
KylinMountain added a commit that referenced this pull request May 24, 2026
Architectural review (4 parallel Opus auditors) found that the skill_runner
core was already generic, but the deck SURFACE was still fused to
Editorial Monocle. Fixed:
* validator: now takes optional `grammar` param (DeckGrammar TypedDict);
skill-agnostic by default (only checks file present, parses, ≥5
slides, self-contained). Third-party deck skills (guizang, swiss)
now pass validation cleanly. Editorial-specific rules opt-in via
`EDITORIAL_MONOCLE_GRAMMAR`. (finding #2)
* skills/openkb-deck-editorial/SKILL.md: declares its grammar +
output_path_template under `od:` frontmatter — `run_skill` reads
these and applies them post-run.
* run_skill: now honors frontmatter `od.mode`, `od.output_path_template`,
`od.deck_grammar`. When mode=="deck" and template is set, the runner
injects the path into intent, verifies the file exists post-run, and
runs validate_deck with the skill's grammar. Validation result is
returned via new SkillRunResult dataclass. (findings #4, #5)
* `openkb deck new --skill <name>`: CLI flag accepts any installed deck
skill (default openkb-deck-editorial). guizang and swiss now usable
from the scripted CLI, not only freeform chat. (finding #1)
* `/deck new --skill <name>` chat slash: same flag, parsed positionally
alongside --critique. (finding #1)
* tests/test_read_kb_file.py: 13 new tests mirroring test_write_kb_file
for the read-side allow-list. Pins refusal of `.openkb/config.yaml`,
`.env`, `raw/`, `..` traversal, absolute paths. (finding #6)
* Generator deck branch: no longer calls validate_deck directly; just
propagates run_deck_create's SkillRunResult.validation up. Validation
is now a property of "this skill declared mode=deck", not of "this
CLI path was taken".
Existing tests updated:
* tests/test_deck_validator.py: explicit grammar arg on Editorial-
specific tests; added test_guizang_shape_passes_generic_mode +
test_missing_cover_ignored_in_generic_mode to pin both modes.
* tests/test_deck_creator.py: mocks return SkillRunResult; new
test_run_deck_create_honors_skill_name_override for --skill flag.
* tests/test_generator.py: deck dispatch test mocks SkillRunResult.
Below-threshold findings deferred:
* Generator if/else → registry (score 70) — works, just not extensible
via plugin; future.
* Iteration backup in chat freeform path (score 75) — needs write_kb_file
hook; separate change.
* run_skill / scan_local_skills / _handle_slash_critique direct tests
(scores 60-70) — covered indirectly by integration; can add later.
Regression: 538 tests pass (was 523 pre-fix; net +15 = 13 new
read_kb_file tests + 2 new validator-mode tests).
KylinMountain added a commit that referenced this pull request May 31, 2026
…lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
KylinMountain added a commit that referenced this pull request Jun 1, 2026
…mpile backfill (#78)
* feat(compiler): _read_entity_briefs for entity plan context
* test(compiler): parity tests for _read_entity_briefs
* feat(compiler): _write_entity with type/aliases frontmatter
* test(compiler): assert source ordering in _write_entity; count=1 in _set_fm_line
Add explicit ordering assertion in test_update_prepends_source_keeps_type
verifying the deterministic json.dumps form ("summaries/b.md", "summaries/a.md").
Pass count=1 to re.sub in _set_fm_line to make first-occurrence intent explicit.
* feat(lint): include entities/ in wikilink whitelist
* feat(compiler): summary<->entity backlinks
* test(compiler): restore assertion erroneously deleted in 3c8aa93
* feat(compiler): index.md Entities section
* feat(compiler): remove_doc_from_entity_pages + index cleanup
* feat(compiler): plan prompt + parser for entities group
Also wires the entity track into _compile_concepts (Tasks 7 + 8 combined,
since the {entity_briefs} placeholder and the _CONCEPTS_PLAN_USER.format call
are co-dependent — splitting would leave an intermediate red state).
- add _ENTITY_TYPES, _filter_entity_items, _parse_entities_plan
- rewrite _CONCEPTS_PLAN_USER to request nested concepts+entities groups
- add _ENTITY_PAGE_USER / _ENTITY_UPDATE_USER prompts
- read entity briefs and pass both briefs to the plan prompt
- parse nested 'concepts' group with legacy flat-list/flat-dict fallbacks
- generate entities in their own asyncio.gather (4-arity tuples)
- strip ghost links + _write_entity each; handle entity related cross-links
- backlink summary<->entities; pass entity_names/entity_meta to _update_index
* fix(compiler): related entities must not downgrade index labels
Mirror the concept track: collect related-entity slugs into a separate
local list used only for backlinks; pass only created/updated entity_names
(+entity_meta) to _update_index. Defense-in-depth in _update_index: only
_replace_section_entry when name is in entity_meta, otherwise only insert
if the link is absent, so a related-only entity can never clobber a
pre-existing correct (type + brief) index line with "(other)".
Adds regression test test_related_entity_does_not_downgrade_index_label.
* feat(schema): declare entities/ page type and taxonomy
* feat(query): point who/what questions at entities/
* docs(readme): document entities/ page type
* feat(cli): scaffold entities/ in init and count it in status
- `openkb init` now creates wiki/entities/ alongside wiki/concepts/
- init seed index.md gains ## Entities between ## Concepts and ## Explorations,
matching the _update_index template in compiler.py
- print_status subdirs list gains "entities" after "concepts"
- Tests updated: assert wiki/entities/ exists and index.md contains ## Entities;
status test asserts "entities" appears in output
* fix(compiler): resolve entity-page review findings (dangling links + dedup)
Addresses code-review findings on the entity-pages feature:
- Fix dangling wikilink after `openkb remove`: entity removal now strips
standalone `See also: [[summaries/{doc}]]` lines (the related-entity
backlink form), matching the concept path, and cli.py adds modified
entity pages to the lint sweep scope so surviving pages are cleaned.
- Unify the parallel concept/entity helpers into shared cores
(_backlink_summary_pages, _backlink_pages, _remove_doc_from_pages) with
thin per-type wrappers, so cleanup logic can no longer drift between the
two page types (this is what caused the dangling-link bug).
- Route related-entity cross-refs through _add_related_link (now page-type
aware) instead of an inline reimplementation — removes a duplicate file
read/write and keeps backlink creation symmetric with teardown.
- Centralize the entity-type enum: prompts derive their type list from a
single _ENTITY_TYPE_LIST source via import-time substitution.
- Count entity items in the "all dropped as malformed" plan warning.
- Drop the unreachable else branch in _update_index's entity loop.
- Add regression test for the See-also strip on a surviving entity page.
All 542 tests pass.
* fix(compiler): add [[entities/X]] whitelist rule + restore concept-topic guard
Remaining review findings after a7a06ed:
- _KNOWN_TARGETS_USER now states the [[entities/Z]] rule, so entity links
the LLM is told to write aren't silently stripped as ghosts.
- Restore the dropped 'Do NOT create concepts that are just the document
topic itself' plan rule to prevent redundant title-mirror concepts.
* feat(entities): shared page-dir constants + surface entities in list/lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
* feat(entities): remove preview lists entity-page actions (#1)
The dry-run/confirmation block now scans wiki/entities/ with the same
frontmatter sources: logic as concepts, emits DELETE/MODIFY action lines
per entity page, and prints an 'N entity(s) will be DELETED' summary.
Execution path (remove_doc_from_entity_pages) unchanged.
* docs(entities): document entity pages in shipped openkb skill (#8)
Note wiki/entities/ holds named-thing pages (people/orgs/places/
products/works/events) with a type: frontmatter field, that index.md
has a ## Entities section, and that 'who/what is X' questions should
read the matching entities/ page first.
* fix(compiler): don't write raw JSON body on empty LLM content
In the parse-succeeded branch of _gen_create/_gen_update/_gen_entity_create/
_gen_entity_update, fall back to "" instead of the raw JSON string when the
content field is empty/null. _require_nonempty_content then raises and the
page is dropped, rather than writing the JSON envelope as the markdown body.
The parse-FAILED (except) branch keeps content=raw as the legitimate
non-JSON fallback.
* fix(compiler): graceful scalar plan + rebuild malformed entity frontmatter
- _compile_concepts: guard a non-dict/non-list parsed plan (JSON scalar)
before calling .get(), taking the empty-plan path (write v1 summary if
applicable + update index + return) instead of risking AttributeError.
- _write_entity: when an existing page has an opening --- but no closing
delimiter (or no frontmatter), rebuild valid sources/type/brief frontmatter
rather than writing a body-only page that drops the metadata.
* fix(compiler): keep ## Entities before ## Explorations; drop dead param + overlap gathers
- _update_index: insert ## Entities before ## Explorations on older index.md
files that predate the section (new _ensure_h2_section_before helper),
preserving canonical order instead of appending at EOF.
- _filter_entity_items: drop the unused 'label' parameter and update call
sites in _parse_entities_plan.
- _compile_concepts: overlap concept and entity generation in one outer
asyncio.gather (they share cached context and the same concurrency
semaphore); result/error handling per list is unchanged.
* test(compiler): cover empty-content skip, scalar plan, malformed entity FM, Entities order
Add regression tests for the four compiler fixes:
- empty {"content":""} response skips the page (no raw JSON body)
- JSON scalar plan handled gracefully (no AttributeError)
- _write_entity rebuilds frontmatter when closing --- is missing
- _update_index inserts ## Entities before ## Explorations
* fix(compiler): silence spurious 'hand-edited' warning on backlink section creation
_backlink_summary_pages / _backlink_pages create ## Entities / ## Related
Documents sections as a normal first-time operation; pass quiet=True so
_ensure_h2_section no longer logs the index-drift warning in that case.
Index-repair callers keep the warning.
* feat(cli): add `recompile` command to re-run compile on indexed docs
Re-runs the current compile_short_doc/compile_long_doc pipeline on
already-indexed docs so pre-feature KBs gain the entities/ layer and
refresh to the current format. Reuses on-disk sources/summaries and the
registry's PageIndex doc_id — does not re-index or re-convert.
Supports a positional <doc_name> (resolved via _resolve_doc_identifier)
or --all (with a regeneration-warning confirmation, bypassed by --yes),
--dry-run (enumerate only, no LLM calls/writes), and --refresh-schema
(back up + overwrite wiki/AGENTS.md when it differs from AGENTS_MD).
Processes docs sequentially with per-doc progress, skips+warns on
missing sources / summaries / doc_id, prints a recompiled/skipped
summary, and appends a recompile entry to log.md.
* test(cli): recompile dispatch/dry-run/skip/refresh-schema
* docs(readme): document openkb recompile
* fix(cli): recompile --refresh-schema no-ops when AGENTS.md absent; tighten guard tests
Match the spec (and the helper's own docstring): _refresh_schema returns
early when wiki/AGENTS.md is missing rather than materializing the default
(get_agents_md already falls back to it at runtime). Tighten the doc/--all
guard tests to assert the exact message + that no compile runs, and add the
missing-AGENTS.md no-op test.
* fix(compiler): drop non-existent 'related' slugs so they don't create dangling links
The plan's 'related' list is meant to reference existing pages, but the LLM
sometimes lists slugs for pages that don't exist. Those were added to the
wikilink whitelist (so body references survived ghost-stripping) and
back-linked into the summary's Related section, yet no page was ever created
(related items are linked, never generated) — producing a flood of broken
[[concepts/...]] / [[entities/...]] links (esp. on feature-dense docs).
Filter related_items / entity_related to slugs that exist on disk.
* fix: remove-preview detects JSON-quoted sources; _write_entity preserves sources on malformed FM
- remove --dry-run preview parsed the sources list with a hand-rolled comma
split that kept JSON quotes (["summaries/x.md"]), so the marker never
matched and the preview always reported 0 affected concept/entity pages
(executor was correct). Extract _scan_affected_pages using the real
_parse_yaml_list_value; dedups the two copied scan loops too.
- _write_entity's malformed-frontmatter rebuild seeded sources with only the
new doc, dropping prior sources for multi-source entities. Recover existing
sources from the broken block and merge.
Both bugs were masked by tests using unquoted / single-source fixtures.
* feat(cli): rename remove --keep-empty-concepts → --keep-empty (covers entities too)
This PR wired entity pages into 'openkb remove', so the flag now governs
concept AND entity retention — but the name still said 'concepts'. Make
--keep-empty the canonical name (clear that it covers both), keep
--keep-empty-concepts as a backward-compatible alias, and update the
preview/summary messages, docstring, and README accordingly.
* feat(compiler): config-driven entity types (entity_types overrides the default enum)
Add an optional 'entity_types:' key in .openkb/config.yaml. When present it
overrides the default person/organization/place/product/work/event/other
vocabulary everywhere — the plan prompt, the entity-page prompts, and
create/update validation/coercion; when absent, behavior is byte-identical.
Prompt templates keep an __ENTITY_TYPES__ token now substituted at call time
(per-KB) inside _compile_concepts, and the resolved valid-type set is threaded
into _parse_entities_plan / _filter_entity_items and the _gen_entity_* coercion.
'other' is always ensured as the coercion fallback; malformed config falls back
to the default with a warning. Documented in config.yaml.example + README.
* fix(compiler): harden config-driven entity types (crash-proof + complete the override)
Review of the config-entity-types feature surfaced two real issues:
- A config 'entity_types' value containing '{' or '}' was substituted into the
prompt template BEFORE .format() ran → KeyError/ValueError crashing every
compile. Swap to format-then-replace at all 3 call sites (types_str is now an
inert literal), and sanitize resolved types to a safe label charset (also
skips YAML nulls/ints so str(None) can't become the type 'none').
- The AGENTS_MD system schema hardcoded 'type: is one of: <7 defaults>',
contradicting a custom entity_types in the higher-weight system message.
Reword it to frame those as the configurable default and defer the
authoritative set to the compilation prompt (which is config-driven).
Also drop the now-dead _ENTITY_TYPES_STR + its stale import-time-substitution
comment. +2 regression tests (sanitization; brace-in-type doesn't crash).
* refactor: move entity-type resolution to config layer + co-locate remove-preview scan
Altitude cleanups from the review:
- Move resolve_entity_types + DEFAULT_ENTITY_TYPES into openkb/config.py (the
config layer owns config validation/normalization; any command can reuse it
without importing the heavy compiler module). compiler.py imports them;
_ENTITY_TYPE_LIST/_ENTITY_TYPES remain as the default alias/validation set.
- Move the remove dry-run preview scan from cli.py into compiler.py as
scan_affected_pages, beside remove_doc_from_*_pages and sharing
_parse_yaml_list_value — so preview and executor can't drift on how the
sources list is parsed (root cause of the earlier JSON-quote preview bug).
---------
Co-authored-by: Claude <noreply@anthropic.com>
calebfavor added a commit to railroadmedia/MusoraOpenKB that referenced this pull request Jul 2, 2026
Implements docs/smart-hierarchy-distillation-plan.md — a RAPTOR-style bottom-up
distillation that builds a multi-layer, LLM-navigable pathway hierarchy over the
flat concept leaves, the intended pivot from the top-down bootstrap() cold-start.
Engine (openkb/topic_tree.py):
- distill(): reads leaves recursively, clusters each layer into LLM-named sized
categories, summarizes each into a parent pathway node, links same-layer peers
sideways, repeats to a single root. Invariants enforced: exactly one root,
always >= 2 layers, bounded depth, no concept loss. Builds into a staging dir
and atomically swaps in (mid-build LLM failure never loses concepts).
- write_pathway_md(): pathway node format — layer/children/related frontmatter +
distilled summary + linked child index + Related pathways section.
- Sideways links are bidirectional, same-layer, top-K, no self-links.
Config (openkb/config.py): HierarchyConfig + resolve_hierarchy() for the
`hierarchy:` block (target/min/max fanout, max_depth, summary token caps,
sideways settings) with validation + back-compat.
LLM callables (openkb/topic_tree_llm.py): make_distill_cluster (AGENTS.md-guided,
sized category naming), make_distill_summarize, make_relate.
Integration: `openkb distill` CLI command; AGENTS.md `## Hierarchy` guidance
section injected into distill prompts; query tree-descent prompt now follows
`related` sideways links; lint registers topic-dir names so pathway/sideways
wikilinks resolve.
Tests (+31): config parsing, distill invariants/edges/sideways/data-loss,
fake-LLM CLI integration + idempotent re-distill, and tier-4 regression pins
(VectifyAI#4 sideways links resolve — red without the lint change; VectifyAI#5 single-root/
min-2-layer edge sizes). Full suite: 937 passed, 10 llm deselected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
KylinMountain added a commit that referenced this pull request Jul 21, 2026
…es (#198)
* feat(api): delete a knowledge base (POST /api/v1/kb/delete + openkb delete-kb)
Physical, irreversible KB deletion with a type-the-name confirmation.
- config.delete_kb: rmtree the KB directory + unregister it from the global
registry (known_kbs / kb_aliases / default_kb, via the new unregister_kb).
Guards against deleting a non-KB path; tolerates a ghost registry entry
(directory already gone) by just unregistering.
- POST /api/v1/kb/delete: confirm_name must equal kb, re-checked server-side.
Lives in a new api_kbs_router (sibling of the config/graph/output routers) so
api.py stays under the 800-line gate; delete_kb self-locks (config lock), no
create_app closure needed.
- openkb delete-kb NAME: type-the-name prompt (or --yes) then delete.
Tests: endpoint (success + unregister, confirm mismatch, non-KB target) in
test_api.py; config (physical delete + unregister, refuse non-KB, ghost
tolerance) in test_config.py.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): delete a wiki page (POST /api/v1/page/delete) with backlink impact
Delete a concept/entity page and degrade references cleanly instead of leaving
them dangling:
- page_ops.delete_wiki_page: dry_run reports the backlink pages (whose inbound
[[wikilinks]] would be demoted) without touching anything; execute removes the
page under the KB ingest lock, strips its index.md entry outright
(compiler.remove_doc_from_index — the entry is a [[link]] line), and demotes
the now-dangling inbound [[links]] to plain text in ONLY those backlink pages
(lint.fix_broken_links restrict_to). The page ref is "section/stem", validated
to concepts/ or entities/ only (path-traversal-safe).
- New api_pages_router hosts POST /api/v1/page (read — relocated here from
api.py to keep it under the 800-line gate) + POST /api/v1/page/delete.
Tests: dry-run impact + execute (page gone, inbound link demoted to text, index
entry removed, sibling kept), 404 on missing page, and rejection of unsafe /
non-editable refs.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): edit a wiki page (PUT /api/v1/page) + link context (POST /api/v1/page/links)
- page_ops.edit_wiki_page: replace a concept/entity page's BODY while preserving
its code-managed OKF frontmatter (type/description/sources) verbatim; any
frontmatter in the submitted content is dropped. Dead [[wikilinks]] the user
typed are demoted to plain text on save (OpenKB keeps the wiki broken-link-free)
and returned as ghosts_stripped so the UI can warn. Atomic write under the KB
ingest lock.
- page_ops.page_link_context (POST /api/v1/page/links): outbound + inbound links
for the edit-impact panel. Editing the body does not break either (links are
path-based), so this is context, not a blocker — the honest impact model.
- PUT /api/v1/page added to api_pages_router.
Tests: edit preserves frontmatter + keeps a resolvable link + demotes a dead one
(reports ghosts_stripped) + replaces the body; 404 on a missing page; links
reports out/backlinks.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): delete a knowledge base from the settings sheet (type-name confirm)
Adds a Danger Zone section to KbSettingsSheet: a type-the-name confirmation
gates deleteKb() (POST /api/v1/kb/delete); on success KbDetail navigates back
to the KB list. New deleteKb client + kbSettings danger* keys + common cancel
(zh + en, identical key sets).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): in-reader page edit + delete with impact preview (F2/F3)
For an open concept/entity page, the reader header now shows Edit and Delete:
- Delete: a dry-run first (deletePage dryRun) surfaces the backlink pages whose
[[links]] will demote to plain text; a red confirm card lists them, then the
real delete closes the page and refreshes the inventory.
- Edit: a body textarea (frontmatter is preserved server-side), Save via
PUT /api/v1/page, a links panel (getPageLinks: out/backlinks), a "recompile
may overwrite" note, and a toast listing any dead links demoted to text.
Adds the deletePage/editPage/getPageLinks wiki clients and kb:pageOps.* locale
keys (zh + en, identical key sets). Build green (i18n guard + tsc + vite).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(workbench): address code-review findings (content-mgmt correctness + concurrency)
Backend (page_ops / kb_admin / lint):
- delete/edit now do their read-modify-write INSIDE the KB ingest lock and
re-check the page exists under it: no stale backlink snapshot, no resurrecting
a concurrently-deleted page, no 500 on a racing unlink (missing_ok=True). [#4,#5,#6]
- delete_kb serializes rmtree under the ingest lock (was fully unlocked). Moved
delete_kb/unregister_kb into a new openkb/kb_admin.py so config.py drops back
under the per-file line gate (751 lines). [#1,#15]
- page deletion no longer scans or rewrites lint's _EXCLUDED_FILES
(AGENTS.md/SCHEMA.md/log.md); fix_broken_links(restrict_to) also refuses them,
which protects `openkb remove` too. [#2]
- edit_wiki_page treats `content` as the body verbatim — no naive frontmatter
split that silently dropped a body opening with a '---' block. [#3]
- delete uses the real on-disk stem for the exact-match index-entry removal
(case-insensitive FS), and adds index.md to the demotion set so a [[target]]
embedded in another entry's brief no longer dangles. [#7,#9]
API / CLI:
- delete-KB endpoint & CLI can now clean a ghost registry entry (registered name
whose directory was removed by hand), catch delete_kb's ValueError -> 400, and
the endpoint is typed `-> KbDeleteResponse`. [#8,#13,E5]
Frontend (KbDetail):
- Deleting a KB dispatches `openkb:reload-kbs` so the sidebar refreshes
immediately (was stale until a manual reload) — live-test fix.
- Removed the dead `pageReloadSeq` path (editPage always returns content). [#14]
Tests: excluded-doc-untouched, '---'-body preservation, ghost-KB unregister;
non-KB-target test updated to the new resolution model. Backend 1234 passed,
mypy/ruff clean, frontend build green.
Not changed (intentional): pages_linking_to is NOT folded into build_graph (that
excludes explorations/ backlinks) [#10]; the preview+confirm delete inherently
scans per request [#11]; EDITABLE_SECTIONS stays duplicated in the frontend,
which can't import the Python constant [#12].
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(workbench): allow editing summary pages (edit ⊇ delete section sets)
Summaries are compiled markdown like concepts/entities (a recompile regenerates
them), so they are now editable via PUT /api/v1/page. They remain NOT
independently deletable — a summary is removed by deleting its source document.
Split page_ops into EDITABLE_SECTIONS (concepts/entities/summaries) vs
DELETABLE_SECTIONS (concepts/entities); validate_page_ref takes the allowlist.
Frontend ReaderBody gates Edit on canEdit (incl. summaries), Delete on canDelete.
Test: summary editable (frontmatter preserved) + summary delete rejected (400).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(api): cross-platform delete_kb + endpoint OSError handling (xhigh review #1,#2)
- delete_kb no longer holds the ingest lock DURING rmtree (the lock file lives
inside the KB dir; Windows cannot delete an open file — the prior review-fix
regressed this). It now takes the lock as a BARRIER (drain + wait out any
in-flight mutation), releases it, re-checks existence, then rmtrees. [#1]
- delete-KB endpoint maps FileNotFoundError to an idempotent success (a
concurrent delete already removed the tree) and other OSError to a clean 500
with a message, instead of an uncaught 500 stack trace. [#2]
Tests: endpoint OSError -> clean 500, FileNotFoundError -> 200 deleted.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…y-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…ane (#197) (#199)
* feat(web): read a document's converted source text in the Documents pane (#197)
The Documents pane listed each ingested doc (name/type/hash) with no way to
read the converted full text ingestion produced under wiki/sources/.
Backend: new POST /api/v1/document/source resolves a doc hash to its source
text — short docs read <doc_name>.md; long docs concatenate the per-page
<doc_name>.json (page text joined by a thematic break). Hash is the identifier
(unique, avoids doc_name/stem collisions); resolution prefers the registry's
stored source_path then falls back to the wiki/sources/<doc_name>.{md,json}
convention, with a path-traversal guard. Read-only (sources are do-not-edit).
Frontend: document rows are now clickable and open a wide read-only slide-out
reader (MarkdownView) — ESC/overlay/close to dismiss, independent scroll,
content cached + memoized per hash, native find-in-page preserved (no
virtualization). Delete stays inline (stopPropagation). Closed drawer is inert.
Known limitation: images embedded in long-doc pages are not rendered inline yet.
* fix(web): address xhigh code-review findings for the document reader (#197)
Correctness / a11y:
- Rebuild the reader drawer on Radix Dialog (like KbSettingsSheet) instead of
a hand-rolled overlay: proper modal focus trap, initial + return focus,
Escape, and background inert (was: aria-modal with none of it) [#4]. This
also removes the hand-rolled window keydown listener that re-subscribed every
render [#7].
- Restructure each document row so the open-reader target is a real <button>
and the delete control is a SIBLING, not nested. Keyboard-activating delete
no longer bubbles into opening the reader, and the invalid nested-interactive
markup is gone [#1, #5].
- Resolve a source file by the doc's own type (long → .json first, else .md),
so two docs sharing a doc_name each resolve to their own file rather than
whichever extension is tried first [#2].
- Guard source reads: skip non-dict page entries, reject non-list JSON, and
return a controlled 500 on corrupt/unreadable sources instead of an
unhandled exception [#3].
- Invalidate the per-hash content cache when the inventory changes, so a
reopen after recompile refetches instead of serving stale text [#6].
Fetch/cache/memoized body moved to DocumentsPane so they survive the drawer's
unmount-on-close. #8 (frontmatter stripping) intentionally not applied: source
docs render verbatim (a user's own frontmatter is content, unlike wiki-page OKF
metadata). Adds tests for the collision and malformed-JSON paths.
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…verride) (#200)
* feat(api): configurable entity_types (global + per-KB) via the config API
entity_types (the entity-extraction vocabulary) is now surfaced through the
config read/write API, layering global.yaml -> KB config.yaml like the other
scalars: a KB list overrides the global list wholesale, an explicit null
inherits, and unset falls back to DEFAULT_ENTITY_TYPES. The compiler already
consumed config["entity_types"] via resolve_entity_types; this just exposes it.
- GLOBAL_SCALAR_KEYS gains "entity_types" (layering + per-key `sources` tracking;
the value-not-None-wins rule is type-agnostic, so it works for a list).
- _KbConfigWritable / GlobalConfigValues / KbConfigResponse / GlobalConfigResponse
carry entity_types; read_kb_config/read_global_config report the cleaned
EFFECTIVE list (resolve_entity_types) plus the raw global value for the badge.
- PATCH /api/v1/kb/config and PATCH /api/v1/config accept entity_types.
Tests: KB override (cleaned + source 'kb') + null revert, global patch, global
inheritance; updated the global-defaults shape assertion. Frontend UI follows.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): entity-types config UI — chips editor (global default + per-KB override)
- EntityTypesEditor: shared controlled chips editor (Enter/comma to add, x to
remove; "other" is a fixed always-included chip; IME-safe composition).
- KbSettingsSheet: an EntityTypesRow with the same inherit/override Switch as the
scalar rows — turning override on seeds+persists the KB's own list, off reverts
via null; inherited state shows the global/default list as a badge. Each chip
change persists and adopts the server-cleaned response.
- Settings (general tab): a global entity-types chips editor, order-sensitive
diff into the save patch (joins the existing dirty/SaveBar flow).
- "changes affect future recompiles only" note on both surfaces.
New keys in common/kbSettings/settings (zh + en, identical sets). Build green
(i18n guard OK). Backend was committed in 92f8f41.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(entity-types): address xhigh review — DEFAULT_CONFIG parity, empty-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@KylinMountain@rejojer@zmtomorrow
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex by KylinMountain · Pull Request #4 · VectifyAI/OpenKB · GitHub
Skip to content

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex - #4

Merged
rejojer merged 102 commits into
mainfrom
dev
Apr 8, 2026
Merged

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex#4
rejojer merged 102 commits into
mainfrom
dev

Conversation

@KylinMountain

Copy link
Copy Markdown
Collaborator

Summary

OpenKB — Karpathy's LLM Knowledge Base workflow as a CLI, powered by PageIndex.

Drop documents in. Get an auto-maintained, cross-linked wiki out.

Features

  • okb init — Interactive setup
  • okb add — Short docs (pymupdf) + long PDFs (PageIndex local/cloud)
  • okb query — Streaming Q&A with PageIndex cloud streaming
  • okb watch — Auto-compile on file changes
  • okb lint — Structural + knowledge health checks
  • okb list / status — Knowledge base overview
  • Obsidian compatible wiki output

Tech Stack

PageIndex, markitdown, OpenAI Agents SDK, LiteLLM, Click, watchdog

KylinMountainand others added 30 commits April 6, 2026 23:13
Sets up pyproject.toml (hatchling, direct-refs allowed, Python >=3.11),
.gitignore, openkb/__init__.py, a Click CLI stub with all 7 commands
(init, add, query, watch, lint, list, status), and tests/conftest.py
with kb_dir and sample_tree fixtures. Package installs cleanly in a
Python 3.12 venv; okb --help shows all commands; pytest collects 0
tests without error.
Add openkb/config.py (DEFAULT_CONFIG, load_config, save_config),
openkb/state.py (HashRegistry with SHA-256 file hashing and JSON
persistence), and openkb/schema.py (SCHEMA_MD constant). All 17 tests
written first (red) then implemented (green).
Creates full KB directory structure (raw/, wiki/sources/images/,
wiki/summaries/, wiki/concepts/, wiki/reports/), writes SCHEMA.md,
index.md, config.yaml and hashes.json; guards against re-initialisation.
Three tests in tests/test_cli.py cover structure, schema content, and
the already-initialized guard, all via CliRunner.isolated_filesystem.
Implements extract_base64_images and copy_relative_images with full test
coverage for single/multiple images, invalid base64, missing files, and
URL filtering.
Implements ConvertResult dataclass, get_pdf_page_count, and
convert_document with hash-dedup, markdown passthrough, PDF long-doc
detection, MarkItDown conversion, and image extraction integration.
Implements render_source_md and render_summary_md with YAML frontmatter,
recursive heading hierarchy (h1–h6 capped), page ranges, and separate
text/summary views for source and summary wiki pages.
Implements IndexResult dataclass and index_long_document which creates
a LocalClient with full node text/summary/description flags, adds the
PDF via PageIndex, fetches structure, and writes source and summary
wiki pages via the tree renderer.
Implements list_wiki_files, read_wiki_file, and write_wiki_file as plain
functions in openkb/agent/tools.py without @function_tool decoration,
ready to be wrapped when building the agent. Full test coverage including
edge cases for missing files/dirs, filtering to .md only, and parent dir
creation.
Implements build_compiler_agent, compile_short_doc, compile_long_doc in
openkb/agent/compiler.py with function_tool-wrapped wiki tools and
SCHEMA_MD-enriched instructions. Long-doc variant includes get_page_content.
Tests mock Runner.run to avoid real LLM calls.
Replaces the add stub with full orchestration: convert_document,
index_long_document for long PDFs, and compiler agent calls.
Adds SUPPORTED_EXTENSIONS set, _find_kb_dir, _add_single_file helpers.
Adds python-dotenv dependency and load_dotenv() at startup.
Implements pageindex_retrieve (structure -> LLM relevance -> page fetch),
build_query_agent with list/read/retrieve tools, and run_query coroutine.
Wires up `okb query` in cli.py.
Implements DebouncedHandler (collects events, ignores dirs/dotfiles, resets
timer on burst) and watch_directory (Observer loop, Ctrl+C safe).
Wires up `okb watch` in cli.py.
Implements find_broken_links, find_orphans, find_missing_entries,
check_index_sync, and run_structural_lint with full Markdown report.
Covers wikilink resolution, orphan detection, raw/wiki entry matching,
and index.md sync checking.
Implements build_lint_agent with list/read tools and instructions for
semantic quality checks (contradictions, gaps, staleness, redundancy).
run_knowledge_lint runs the agent and returns the report string.
okb lint combines structural + knowledge lint and writes timestamped report.
Tests verify list shows documents table and concepts, status shows
per-directory file counts and total indexed. Both check missing-init guard.
Previously the converter registered the file hash immediately, so if
LLM compilation failed the file was marked as "done" and retries
would skip it. Now the hash is only registered by the CLI after
successful compilation.
Also: install markitdown[all] for PDF support, add python-dotenv.
…pport
- Switch from col._backend.get_document_structure() to col.get_document_structure()
- Add 3x retry for PageIndex indexing (stochastic TOC accuracy)
- Fix storage path to use .db extension
- Remove .doc from supported extensions (markitdown only supports .docx)
- Note: col.get_page_content() still missing from PageIndex public API,
using col._backend.get_page_content() as workaround
Replace col._backend.get_page_content(col._name, doc_id, spec) with
col.get_page_content(doc_id, spec). Now all PageIndex access uses
public API only.
rejojer added 15 commits April 8, 2026 05:01
Rename CLI command and state dir from okb to openkb
- Hardcode reading LLM_API_KEY env var instead of indirecting through config
- Remove llm_api_key_env from DEFAULT_CONFIG, okb init prompts, and config.yaml
- Provider-specific env vars (OPENAI_API_KEY, etc.) still work via LiteLLM auto-detection
- One less config field, one less okb init step
The OpenAI Agents SDK requires a litellm/ prefix to route non-OpenAI
models through LiteLLM. Without it, models like anthropic/claude-sonnet-4-6
fail with "Unknown prefix". This adds the prefix at all Agent() call sites
while keeping litellm.completion() calls unchanged.
Also updates README quick start comments and model format docs.
Fix: add litellm/ prefix for Agents SDK model routing
@KylinMountain

Copy link
Copy Markdown
CollaboratorAuthor

Code review

Found 1 issue:

  1. extract_pdf_images and convert_pdf_with_images in images.py open pymupdf documents with explicit .close() instead of context managers. If an exception is raised during page iteration (e.g. corrupt image block, pixmap allocation failure), the PDF file handle leaks. This is the same bug pattern that was already fixed in converter.py:get_pdf_page_count (commit c525455), but images.py was missed. Fix: replace doc = pymupdf.open(...) / doc.close() with with pymupdf.open(...) as doc:.

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] !=1: # not an image block
continue
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
save_path=images_dir/filename
pix.save(str(save_path))
pix=None
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
continue
rel_path=f"images/{doc_name}/{filename}"
page_images.setdefault(page_num, []).append(rel_path)
doc.close()
returnpage_images

OpenKB/openkb/images.py

Lines 89 to 125 in 1637697

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
parts.append(f"\n\n<!-- Page {page_num} -->\n")
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] ==0: # text block
lines= []
forlineinblock["lines"]:
spans_text="".join(span["text"] forspaninline["spans"])
lines.append(spans_text)
parts.append("\n".join(lines))
elifblock["type"] ==1: # image block
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
(images_dir/filename).write_bytes(pix.tobytes("png"))
pix=None
parts.append(f"\n![image](images/{doc_name}/{filename})\n")
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
doc.close()
return"\n".join(parts)

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer

Copy link
Copy Markdown
Member

Code review

Found 2 issues:

  1. README claims LiteLLM is "pinned to a safe version" but pyproject.toml has no version pin. Line 67 of README.md states LiteLLM is (pinned to a safe version), but pyproject.toml line 17 lists the dependency as bare "litellm" with no version constraint (==, >=, ~=, etc.). Any version -- including potentially insecure ones -- can be installed.

OpenKB/README.md

Lines 66 to 68 in 854294c

OpenKB comes with [multi-LLM support](https://docs.litellm.ai/docs/providers) (e.g., OpenAI, Claude, Gemini) via [LiteLLM](https://github.com/BerriAI/litellm) (pinned to a [safe version](https://docs.litellm.ai/blog/security-update-march-2026)).

OpenKB/pyproject.toml

Lines 16 to 18 in 854294c

"watchdog>=3.0",
"litellm",
"openai-agents",

  1. test_short_pdf_converted_via_markitdown mocks the wrong code path. The test patches openkb.converter.MarkItDown and openkb.converter.pymupdf.open, but converter.py line 99-101 routes short PDFs through convert_pdf_with_images() (from openkb.images), not MarkItDown. The MarkItDown mock is never exercised, and convert_pdf_with_images is not mocked, so the test either fails at runtime or passes for the wrong reasons.

classTestConvertDocumentPdfShort:
deftest_short_pdf_converted_via_markitdown(self, kb_dir, tmp_path):
"""PDF under threshold is converted with markitdown."""
src=tmp_path/"short.pdf"
src.write_bytes(b"%PDF-1.4 fake content")
fake_result=MagicMock()
fake_result.text_content="# Short PDF\n\nConverted content."
with (
patch("openkb.converter.pymupdf.open") asmock_mu,
patch("openkb.converter.MarkItDown") asmock_mid_cls,
):
fake_doc=MagicMock()
fake_doc.page_count=5# below default threshold of 20
fake_doc.__enter__=MagicMock(return_value=fake_doc)
fake_doc.__exit__=MagicMock(return_value=False)
mock_mu.return_value=fake_doc
mock_mid_cls.return_value.convert.return_value=fake_result
result=convert_document(src, kb_dir)
assertresult.skippedisFalse
assertresult.is_long_docisFalse
assertresult.source_pathisnotNone
assertresult.source_path.exists()

markdown=copy_relative_images(markdown, src.parent, doc_name, images_dir)
elifsrc.suffix.lower() ==".pdf":
# Use pymupdf dict-mode for PDFs: text + images inline at correct positions
markdown=convert_pdf_with_images(src, doc_name, images_dir)
else:

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer
rejojer merged commit f0963f6 into mainApr 8, 2026
KylinMountain added a commit that referenced this pull request May 24, 2026
Architectural review (4 parallel Opus auditors) found that the skill_runner
core was already generic, but the deck SURFACE was still fused to
Editorial Monocle. Fixed:
* validator: now takes optional `grammar` param (DeckGrammar TypedDict);
skill-agnostic by default (only checks file present, parses, ≥5
slides, self-contained). Third-party deck skills (guizang, swiss)
now pass validation cleanly. Editorial-specific rules opt-in via
`EDITORIAL_MONOCLE_GRAMMAR`. (finding #2)
* skills/openkb-deck-editorial/SKILL.md: declares its grammar +
output_path_template under `od:` frontmatter — `run_skill` reads
these and applies them post-run.
* run_skill: now honors frontmatter `od.mode`, `od.output_path_template`,
`od.deck_grammar`. When mode=="deck" and template is set, the runner
injects the path into intent, verifies the file exists post-run, and
runs validate_deck with the skill's grammar. Validation result is
returned via new SkillRunResult dataclass. (findings #4, #5)
* `openkb deck new --skill <name>`: CLI flag accepts any installed deck
skill (default openkb-deck-editorial). guizang and swiss now usable
from the scripted CLI, not only freeform chat. (finding #1)
* `/deck new --skill <name>` chat slash: same flag, parsed positionally
alongside --critique. (finding #1)
* tests/test_read_kb_file.py: 13 new tests mirroring test_write_kb_file
for the read-side allow-list. Pins refusal of `.openkb/config.yaml`,
`.env`, `raw/`, `..` traversal, absolute paths. (finding #6)
* Generator deck branch: no longer calls validate_deck directly; just
propagates run_deck_create's SkillRunResult.validation up. Validation
is now a property of "this skill declared mode=deck", not of "this
CLI path was taken".
Existing tests updated:
* tests/test_deck_validator.py: explicit grammar arg on Editorial-
specific tests; added test_guizang_shape_passes_generic_mode +
test_missing_cover_ignored_in_generic_mode to pin both modes.
* tests/test_deck_creator.py: mocks return SkillRunResult; new
test_run_deck_create_honors_skill_name_override for --skill flag.
* tests/test_generator.py: deck dispatch test mocks SkillRunResult.
Below-threshold findings deferred:
* Generator if/else → registry (score 70) — works, just not extensible
via plugin; future.
* Iteration backup in chat freeform path (score 75) — needs write_kb_file
hook; separate change.
* run_skill / scan_local_skills / _handle_slash_critique direct tests
(scores 60-70) — covered indirectly by integration; can add later.
Regression: 538 tests pass (was 523 pre-fix; net +15 = 13 new
read_kb_file tests + 2 new validator-mode tests).
KylinMountain added a commit that referenced this pull request May 31, 2026
…lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
KylinMountain added a commit that referenced this pull request Jun 1, 2026
…mpile backfill (#78)
* feat(compiler): _read_entity_briefs for entity plan context
* test(compiler): parity tests for _read_entity_briefs
* feat(compiler): _write_entity with type/aliases frontmatter
* test(compiler): assert source ordering in _write_entity; count=1 in _set_fm_line
Add explicit ordering assertion in test_update_prepends_source_keeps_type
verifying the deterministic json.dumps form ("summaries/b.md", "summaries/a.md").
Pass count=1 to re.sub in _set_fm_line to make first-occurrence intent explicit.
* feat(lint): include entities/ in wikilink whitelist
* feat(compiler): summary<->entity backlinks
* test(compiler): restore assertion erroneously deleted in 3c8aa93
* feat(compiler): index.md Entities section
* feat(compiler): remove_doc_from_entity_pages + index cleanup
* feat(compiler): plan prompt + parser for entities group
Also wires the entity track into _compile_concepts (Tasks 7 + 8 combined,
since the {entity_briefs} placeholder and the _CONCEPTS_PLAN_USER.format call
are co-dependent — splitting would leave an intermediate red state).
- add _ENTITY_TYPES, _filter_entity_items, _parse_entities_plan
- rewrite _CONCEPTS_PLAN_USER to request nested concepts+entities groups
- add _ENTITY_PAGE_USER / _ENTITY_UPDATE_USER prompts
- read entity briefs and pass both briefs to the plan prompt
- parse nested 'concepts' group with legacy flat-list/flat-dict fallbacks
- generate entities in their own asyncio.gather (4-arity tuples)
- strip ghost links + _write_entity each; handle entity related cross-links
- backlink summary<->entities; pass entity_names/entity_meta to _update_index
* fix(compiler): related entities must not downgrade index labels
Mirror the concept track: collect related-entity slugs into a separate
local list used only for backlinks; pass only created/updated entity_names
(+entity_meta) to _update_index. Defense-in-depth in _update_index: only
_replace_section_entry when name is in entity_meta, otherwise only insert
if the link is absent, so a related-only entity can never clobber a
pre-existing correct (type + brief) index line with "(other)".
Adds regression test test_related_entity_does_not_downgrade_index_label.
* feat(schema): declare entities/ page type and taxonomy
* feat(query): point who/what questions at entities/
* docs(readme): document entities/ page type
* feat(cli): scaffold entities/ in init and count it in status
- `openkb init` now creates wiki/entities/ alongside wiki/concepts/
- init seed index.md gains ## Entities between ## Concepts and ## Explorations,
matching the _update_index template in compiler.py
- print_status subdirs list gains "entities" after "concepts"
- Tests updated: assert wiki/entities/ exists and index.md contains ## Entities;
status test asserts "entities" appears in output
* fix(compiler): resolve entity-page review findings (dangling links + dedup)
Addresses code-review findings on the entity-pages feature:
- Fix dangling wikilink after `openkb remove`: entity removal now strips
standalone `See also: [[summaries/{doc}]]` lines (the related-entity
backlink form), matching the concept path, and cli.py adds modified
entity pages to the lint sweep scope so surviving pages are cleaned.
- Unify the parallel concept/entity helpers into shared cores
(_backlink_summary_pages, _backlink_pages, _remove_doc_from_pages) with
thin per-type wrappers, so cleanup logic can no longer drift between the
two page types (this is what caused the dangling-link bug).
- Route related-entity cross-refs through _add_related_link (now page-type
aware) instead of an inline reimplementation — removes a duplicate file
read/write and keeps backlink creation symmetric with teardown.
- Centralize the entity-type enum: prompts derive their type list from a
single _ENTITY_TYPE_LIST source via import-time substitution.
- Count entity items in the "all dropped as malformed" plan warning.
- Drop the unreachable else branch in _update_index's entity loop.
- Add regression test for the See-also strip on a surviving entity page.
All 542 tests pass.
* fix(compiler): add [[entities/X]] whitelist rule + restore concept-topic guard
Remaining review findings after a7a06ed:
- _KNOWN_TARGETS_USER now states the [[entities/Z]] rule, so entity links
the LLM is told to write aren't silently stripped as ghosts.
- Restore the dropped 'Do NOT create concepts that are just the document
topic itself' plan rule to prevent redundant title-mirror concepts.
* feat(entities): shared page-dir constants + surface entities in list/lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
* feat(entities): remove preview lists entity-page actions (#1)
The dry-run/confirmation block now scans wiki/entities/ with the same
frontmatter sources: logic as concepts, emits DELETE/MODIFY action lines
per entity page, and prints an 'N entity(s) will be DELETED' summary.
Execution path (remove_doc_from_entity_pages) unchanged.
* docs(entities): document entity pages in shipped openkb skill (#8)
Note wiki/entities/ holds named-thing pages (people/orgs/places/
products/works/events) with a type: frontmatter field, that index.md
has a ## Entities section, and that 'who/what is X' questions should
read the matching entities/ page first.
* fix(compiler): don't write raw JSON body on empty LLM content
In the parse-succeeded branch of _gen_create/_gen_update/_gen_entity_create/
_gen_entity_update, fall back to "" instead of the raw JSON string when the
content field is empty/null. _require_nonempty_content then raises and the
page is dropped, rather than writing the JSON envelope as the markdown body.
The parse-FAILED (except) branch keeps content=raw as the legitimate
non-JSON fallback.
* fix(compiler): graceful scalar plan + rebuild malformed entity frontmatter
- _compile_concepts: guard a non-dict/non-list parsed plan (JSON scalar)
before calling .get(), taking the empty-plan path (write v1 summary if
applicable + update index + return) instead of risking AttributeError.
- _write_entity: when an existing page has an opening --- but no closing
delimiter (or no frontmatter), rebuild valid sources/type/brief frontmatter
rather than writing a body-only page that drops the metadata.
* fix(compiler): keep ## Entities before ## Explorations; drop dead param + overlap gathers
- _update_index: insert ## Entities before ## Explorations on older index.md
files that predate the section (new _ensure_h2_section_before helper),
preserving canonical order instead of appending at EOF.
- _filter_entity_items: drop the unused 'label' parameter and update call
sites in _parse_entities_plan.
- _compile_concepts: overlap concept and entity generation in one outer
asyncio.gather (they share cached context and the same concurrency
semaphore); result/error handling per list is unchanged.
* test(compiler): cover empty-content skip, scalar plan, malformed entity FM, Entities order
Add regression tests for the four compiler fixes:
- empty {"content":""} response skips the page (no raw JSON body)
- JSON scalar plan handled gracefully (no AttributeError)
- _write_entity rebuilds frontmatter when closing --- is missing
- _update_index inserts ## Entities before ## Explorations
* fix(compiler): silence spurious 'hand-edited' warning on backlink section creation
_backlink_summary_pages / _backlink_pages create ## Entities / ## Related
Documents sections as a normal first-time operation; pass quiet=True so
_ensure_h2_section no longer logs the index-drift warning in that case.
Index-repair callers keep the warning.
* feat(cli): add `recompile` command to re-run compile on indexed docs
Re-runs the current compile_short_doc/compile_long_doc pipeline on
already-indexed docs so pre-feature KBs gain the entities/ layer and
refresh to the current format. Reuses on-disk sources/summaries and the
registry's PageIndex doc_id — does not re-index or re-convert.
Supports a positional <doc_name> (resolved via _resolve_doc_identifier)
or --all (with a regeneration-warning confirmation, bypassed by --yes),
--dry-run (enumerate only, no LLM calls/writes), and --refresh-schema
(back up + overwrite wiki/AGENTS.md when it differs from AGENTS_MD).
Processes docs sequentially with per-doc progress, skips+warns on
missing sources / summaries / doc_id, prints a recompiled/skipped
summary, and appends a recompile entry to log.md.
* test(cli): recompile dispatch/dry-run/skip/refresh-schema
* docs(readme): document openkb recompile
* fix(cli): recompile --refresh-schema no-ops when AGENTS.md absent; tighten guard tests
Match the spec (and the helper's own docstring): _refresh_schema returns
early when wiki/AGENTS.md is missing rather than materializing the default
(get_agents_md already falls back to it at runtime). Tighten the doc/--all
guard tests to assert the exact message + that no compile runs, and add the
missing-AGENTS.md no-op test.
* fix(compiler): drop non-existent 'related' slugs so they don't create dangling links
The plan's 'related' list is meant to reference existing pages, but the LLM
sometimes lists slugs for pages that don't exist. Those were added to the
wikilink whitelist (so body references survived ghost-stripping) and
back-linked into the summary's Related section, yet no page was ever created
(related items are linked, never generated) — producing a flood of broken
[[concepts/...]] / [[entities/...]] links (esp. on feature-dense docs).
Filter related_items / entity_related to slugs that exist on disk.
* fix: remove-preview detects JSON-quoted sources; _write_entity preserves sources on malformed FM
- remove --dry-run preview parsed the sources list with a hand-rolled comma
split that kept JSON quotes (["summaries/x.md"]), so the marker never
matched and the preview always reported 0 affected concept/entity pages
(executor was correct). Extract _scan_affected_pages using the real
_parse_yaml_list_value; dedups the two copied scan loops too.
- _write_entity's malformed-frontmatter rebuild seeded sources with only the
new doc, dropping prior sources for multi-source entities. Recover existing
sources from the broken block and merge.
Both bugs were masked by tests using unquoted / single-source fixtures.
* feat(cli): rename remove --keep-empty-concepts → --keep-empty (covers entities too)
This PR wired entity pages into 'openkb remove', so the flag now governs
concept AND entity retention — but the name still said 'concepts'. Make
--keep-empty the canonical name (clear that it covers both), keep
--keep-empty-concepts as a backward-compatible alias, and update the
preview/summary messages, docstring, and README accordingly.
* feat(compiler): config-driven entity types (entity_types overrides the default enum)
Add an optional 'entity_types:' key in .openkb/config.yaml. When present it
overrides the default person/organization/place/product/work/event/other
vocabulary everywhere — the plan prompt, the entity-page prompts, and
create/update validation/coercion; when absent, behavior is byte-identical.
Prompt templates keep an __ENTITY_TYPES__ token now substituted at call time
(per-KB) inside _compile_concepts, and the resolved valid-type set is threaded
into _parse_entities_plan / _filter_entity_items and the _gen_entity_* coercion.
'other' is always ensured as the coercion fallback; malformed config falls back
to the default with a warning. Documented in config.yaml.example + README.
* fix(compiler): harden config-driven entity types (crash-proof + complete the override)
Review of the config-entity-types feature surfaced two real issues:
- A config 'entity_types' value containing '{' or '}' was substituted into the
prompt template BEFORE .format() ran → KeyError/ValueError crashing every
compile. Swap to format-then-replace at all 3 call sites (types_str is now an
inert literal), and sanitize resolved types to a safe label charset (also
skips YAML nulls/ints so str(None) can't become the type 'none').
- The AGENTS_MD system schema hardcoded 'type: is one of: <7 defaults>',
contradicting a custom entity_types in the higher-weight system message.
Reword it to frame those as the configurable default and defer the
authoritative set to the compilation prompt (which is config-driven).
Also drop the now-dead _ENTITY_TYPES_STR + its stale import-time-substitution
comment. +2 regression tests (sanitization; brace-in-type doesn't crash).
* refactor: move entity-type resolution to config layer + co-locate remove-preview scan
Altitude cleanups from the review:
- Move resolve_entity_types + DEFAULT_ENTITY_TYPES into openkb/config.py (the
config layer owns config validation/normalization; any command can reuse it
without importing the heavy compiler module). compiler.py imports them;
_ENTITY_TYPE_LIST/_ENTITY_TYPES remain as the default alias/validation set.
- Move the remove dry-run preview scan from cli.py into compiler.py as
scan_affected_pages, beside remove_doc_from_*_pages and sharing
_parse_yaml_list_value — so preview and executor can't drift on how the
sources list is parsed (root cause of the earlier JSON-quote preview bug).
---------
Co-authored-by: Claude <noreply@anthropic.com>
calebfavor added a commit to railroadmedia/MusoraOpenKB that referenced this pull request Jul 2, 2026
Implements docs/smart-hierarchy-distillation-plan.md — a RAPTOR-style bottom-up
distillation that builds a multi-layer, LLM-navigable pathway hierarchy over the
flat concept leaves, the intended pivot from the top-down bootstrap() cold-start.
Engine (openkb/topic_tree.py):
- distill(): reads leaves recursively, clusters each layer into LLM-named sized
categories, summarizes each into a parent pathway node, links same-layer peers
sideways, repeats to a single root. Invariants enforced: exactly one root,
always >= 2 layers, bounded depth, no concept loss. Builds into a staging dir
and atomically swaps in (mid-build LLM failure never loses concepts).
- write_pathway_md(): pathway node format — layer/children/related frontmatter +
distilled summary + linked child index + Related pathways section.
- Sideways links are bidirectional, same-layer, top-K, no self-links.
Config (openkb/config.py): HierarchyConfig + resolve_hierarchy() for the
`hierarchy:` block (target/min/max fanout, max_depth, summary token caps,
sideways settings) with validation + back-compat.
LLM callables (openkb/topic_tree_llm.py): make_distill_cluster (AGENTS.md-guided,
sized category naming), make_distill_summarize, make_relate.
Integration: `openkb distill` CLI command; AGENTS.md `## Hierarchy` guidance
section injected into distill prompts; query tree-descent prompt now follows
`related` sideways links; lint registers topic-dir names so pathway/sideways
wikilinks resolve.
Tests (+31): config parsing, distill invariants/edges/sideways/data-loss,
fake-LLM CLI integration + idempotent re-distill, and tier-4 regression pins
(VectifyAI#4 sideways links resolve — red without the lint change; VectifyAI#5 single-root/
min-2-layer edge sizes). Full suite: 937 passed, 10 llm deselected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
KylinMountain added a commit that referenced this pull request Jul 21, 2026
…es (#198)
* feat(api): delete a knowledge base (POST /api/v1/kb/delete + openkb delete-kb)
Physical, irreversible KB deletion with a type-the-name confirmation.
- config.delete_kb: rmtree the KB directory + unregister it from the global
registry (known_kbs / kb_aliases / default_kb, via the new unregister_kb).
Guards against deleting a non-KB path; tolerates a ghost registry entry
(directory already gone) by just unregistering.
- POST /api/v1/kb/delete: confirm_name must equal kb, re-checked server-side.
Lives in a new api_kbs_router (sibling of the config/graph/output routers) so
api.py stays under the 800-line gate; delete_kb self-locks (config lock), no
create_app closure needed.
- openkb delete-kb NAME: type-the-name prompt (or --yes) then delete.
Tests: endpoint (success + unregister, confirm mismatch, non-KB target) in
test_api.py; config (physical delete + unregister, refuse non-KB, ghost
tolerance) in test_config.py.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): delete a wiki page (POST /api/v1/page/delete) with backlink impact
Delete a concept/entity page and degrade references cleanly instead of leaving
them dangling:
- page_ops.delete_wiki_page: dry_run reports the backlink pages (whose inbound
[[wikilinks]] would be demoted) without touching anything; execute removes the
page under the KB ingest lock, strips its index.md entry outright
(compiler.remove_doc_from_index — the entry is a [[link]] line), and demotes
the now-dangling inbound [[links]] to plain text in ONLY those backlink pages
(lint.fix_broken_links restrict_to). The page ref is "section/stem", validated
to concepts/ or entities/ only (path-traversal-safe).
- New api_pages_router hosts POST /api/v1/page (read — relocated here from
api.py to keep it under the 800-line gate) + POST /api/v1/page/delete.
Tests: dry-run impact + execute (page gone, inbound link demoted to text, index
entry removed, sibling kept), 404 on missing page, and rejection of unsafe /
non-editable refs.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): edit a wiki page (PUT /api/v1/page) + link context (POST /api/v1/page/links)
- page_ops.edit_wiki_page: replace a concept/entity page's BODY while preserving
its code-managed OKF frontmatter (type/description/sources) verbatim; any
frontmatter in the submitted content is dropped. Dead [[wikilinks]] the user
typed are demoted to plain text on save (OpenKB keeps the wiki broken-link-free)
and returned as ghosts_stripped so the UI can warn. Atomic write under the KB
ingest lock.
- page_ops.page_link_context (POST /api/v1/page/links): outbound + inbound links
for the edit-impact panel. Editing the body does not break either (links are
path-based), so this is context, not a blocker — the honest impact model.
- PUT /api/v1/page added to api_pages_router.
Tests: edit preserves frontmatter + keeps a resolvable link + demotes a dead one
(reports ghosts_stripped) + replaces the body; 404 on a missing page; links
reports out/backlinks.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): delete a knowledge base from the settings sheet (type-name confirm)
Adds a Danger Zone section to KbSettingsSheet: a type-the-name confirmation
gates deleteKb() (POST /api/v1/kb/delete); on success KbDetail navigates back
to the KB list. New deleteKb client + kbSettings danger* keys + common cancel
(zh + en, identical key sets).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): in-reader page edit + delete with impact preview (F2/F3)
For an open concept/entity page, the reader header now shows Edit and Delete:
- Delete: a dry-run first (deletePage dryRun) surfaces the backlink pages whose
[[links]] will demote to plain text; a red confirm card lists them, then the
real delete closes the page and refreshes the inventory.
- Edit: a body textarea (frontmatter is preserved server-side), Save via
PUT /api/v1/page, a links panel (getPageLinks: out/backlinks), a "recompile
may overwrite" note, and a toast listing any dead links demoted to text.
Adds the deletePage/editPage/getPageLinks wiki clients and kb:pageOps.* locale
keys (zh + en, identical key sets). Build green (i18n guard + tsc + vite).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(workbench): address code-review findings (content-mgmt correctness + concurrency)
Backend (page_ops / kb_admin / lint):
- delete/edit now do their read-modify-write INSIDE the KB ingest lock and
re-check the page exists under it: no stale backlink snapshot, no resurrecting
a concurrently-deleted page, no 500 on a racing unlink (missing_ok=True). [#4,#5,#6]
- delete_kb serializes rmtree under the ingest lock (was fully unlocked). Moved
delete_kb/unregister_kb into a new openkb/kb_admin.py so config.py drops back
under the per-file line gate (751 lines). [#1,#15]
- page deletion no longer scans or rewrites lint's _EXCLUDED_FILES
(AGENTS.md/SCHEMA.md/log.md); fix_broken_links(restrict_to) also refuses them,
which protects `openkb remove` too. [#2]
- edit_wiki_page treats `content` as the body verbatim — no naive frontmatter
split that silently dropped a body opening with a '---' block. [#3]
- delete uses the real on-disk stem for the exact-match index-entry removal
(case-insensitive FS), and adds index.md to the demotion set so a [[target]]
embedded in another entry's brief no longer dangles. [#7,#9]
API / CLI:
- delete-KB endpoint & CLI can now clean a ghost registry entry (registered name
whose directory was removed by hand), catch delete_kb's ValueError -> 400, and
the endpoint is typed `-> KbDeleteResponse`. [#8,#13,E5]
Frontend (KbDetail):
- Deleting a KB dispatches `openkb:reload-kbs` so the sidebar refreshes
immediately (was stale until a manual reload) — live-test fix.
- Removed the dead `pageReloadSeq` path (editPage always returns content). [#14]
Tests: excluded-doc-untouched, '---'-body preservation, ghost-KB unregister;
non-KB-target test updated to the new resolution model. Backend 1234 passed,
mypy/ruff clean, frontend build green.
Not changed (intentional): pages_linking_to is NOT folded into build_graph (that
excludes explorations/ backlinks) [#10]; the preview+confirm delete inherently
scans per request [#11]; EDITABLE_SECTIONS stays duplicated in the frontend,
which can't import the Python constant [#12].
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(workbench): allow editing summary pages (edit ⊇ delete section sets)
Summaries are compiled markdown like concepts/entities (a recompile regenerates
them), so they are now editable via PUT /api/v1/page. They remain NOT
independently deletable — a summary is removed by deleting its source document.
Split page_ops into EDITABLE_SECTIONS (concepts/entities/summaries) vs
DELETABLE_SECTIONS (concepts/entities); validate_page_ref takes the allowlist.
Frontend ReaderBody gates Edit on canEdit (incl. summaries), Delete on canDelete.
Test: summary editable (frontmatter preserved) + summary delete rejected (400).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(api): cross-platform delete_kb + endpoint OSError handling (xhigh review #1,#2)
- delete_kb no longer holds the ingest lock DURING rmtree (the lock file lives
inside the KB dir; Windows cannot delete an open file — the prior review-fix
regressed this). It now takes the lock as a BARRIER (drain + wait out any
in-flight mutation), releases it, re-checks existence, then rmtrees. [#1]
- delete-KB endpoint maps FileNotFoundError to an idempotent success (a
concurrent delete already removed the tree) and other OSError to a clean 500
with a message, instead of an uncaught 500 stack trace. [#2]
Tests: endpoint OSError -> clean 500, FileNotFoundError -> 200 deleted.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…y-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…ane (#197) (#199)
* feat(web): read a document's converted source text in the Documents pane (#197)
The Documents pane listed each ingested doc (name/type/hash) with no way to
read the converted full text ingestion produced under wiki/sources/.
Backend: new POST /api/v1/document/source resolves a doc hash to its source
text — short docs read <doc_name>.md; long docs concatenate the per-page
<doc_name>.json (page text joined by a thematic break). Hash is the identifier
(unique, avoids doc_name/stem collisions); resolution prefers the registry's
stored source_path then falls back to the wiki/sources/<doc_name>.{md,json}
convention, with a path-traversal guard. Read-only (sources are do-not-edit).
Frontend: document rows are now clickable and open a wide read-only slide-out
reader (MarkdownView) — ESC/overlay/close to dismiss, independent scroll,
content cached + memoized per hash, native find-in-page preserved (no
virtualization). Delete stays inline (stopPropagation). Closed drawer is inert.
Known limitation: images embedded in long-doc pages are not rendered inline yet.
* fix(web): address xhigh code-review findings for the document reader (#197)
Correctness / a11y:
- Rebuild the reader drawer on Radix Dialog (like KbSettingsSheet) instead of
a hand-rolled overlay: proper modal focus trap, initial + return focus,
Escape, and background inert (was: aria-modal with none of it) [#4]. This
also removes the hand-rolled window keydown listener that re-subscribed every
render [#7].
- Restructure each document row so the open-reader target is a real <button>
and the delete control is a SIBLING, not nested. Keyboard-activating delete
no longer bubbles into opening the reader, and the invalid nested-interactive
markup is gone [#1, #5].
- Resolve a source file by the doc's own type (long → .json first, else .md),
so two docs sharing a doc_name each resolve to their own file rather than
whichever extension is tried first [#2].
- Guard source reads: skip non-dict page entries, reject non-list JSON, and
return a controlled 500 on corrupt/unreadable sources instead of an
unhandled exception [#3].
- Invalidate the per-hash content cache when the inventory changes, so a
reopen after recompile refetches instead of serving stale text [#6].
Fetch/cache/memoized body moved to DocumentsPane so they survive the drawer's
unmount-on-close. #8 (frontmatter stripping) intentionally not applied: source
docs render verbatim (a user's own frontmatter is content, unlike wiki-page OKF
metadata). Adds tests for the collision and malformed-JSON paths.
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…verride) (#200)
* feat(api): configurable entity_types (global + per-KB) via the config API
entity_types (the entity-extraction vocabulary) is now surfaced through the
config read/write API, layering global.yaml -> KB config.yaml like the other
scalars: a KB list overrides the global list wholesale, an explicit null
inherits, and unset falls back to DEFAULT_ENTITY_TYPES. The compiler already
consumed config["entity_types"] via resolve_entity_types; this just exposes it.
- GLOBAL_SCALAR_KEYS gains "entity_types" (layering + per-key `sources` tracking;
the value-not-None-wins rule is type-agnostic, so it works for a list).
- _KbConfigWritable / GlobalConfigValues / KbConfigResponse / GlobalConfigResponse
carry entity_types; read_kb_config/read_global_config report the cleaned
EFFECTIVE list (resolve_entity_types) plus the raw global value for the badge.
- PATCH /api/v1/kb/config and PATCH /api/v1/config accept entity_types.
Tests: KB override (cleaned + source 'kb') + null revert, global patch, global
inheritance; updated the global-defaults shape assertion. Frontend UI follows.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): entity-types config UI — chips editor (global default + per-KB override)
- EntityTypesEditor: shared controlled chips editor (Enter/comma to add, x to
remove; "other" is a fixed always-included chip; IME-safe composition).
- KbSettingsSheet: an EntityTypesRow with the same inherit/override Switch as the
scalar rows — turning override on seeds+persists the KB's own list, off reverts
via null; inherited state shows the global/default list as a badge. Each chip
change persists and adopts the server-cleaned response.
- Settings (general tab): a global entity-types chips editor, order-sensitive
diff into the save patch (joins the existing dirty/SaveBar flow).
- "changes affect future recompiles only" note on both surfaces.
New keys in common/kbSettings/settings (zh + en, identical sets). Build green
(i18n guard OK). Backend was committed in 92f8f41.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(entity-types): address xhigh review — DEFAULT_CONFIG parity, empty-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@KylinMountain@rejojer@zmtomorrow
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex by KylinMountain · Pull Request #4 · VectifyAI/OpenKB · GitHub
Skip to content

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex - #4

Merged
rejojer merged 102 commits into
mainfrom
dev
Apr 8, 2026
Merged

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex#4
rejojer merged 102 commits into
mainfrom
dev

Conversation

@KylinMountain

Copy link
Copy Markdown
Collaborator

Summary

OpenKB — Karpathy's LLM Knowledge Base workflow as a CLI, powered by PageIndex.

Drop documents in. Get an auto-maintained, cross-linked wiki out.

Features

  • okb init — Interactive setup
  • okb add — Short docs (pymupdf) + long PDFs (PageIndex local/cloud)
  • okb query — Streaming Q&A with PageIndex cloud streaming
  • okb watch — Auto-compile on file changes
  • okb lint — Structural + knowledge health checks
  • okb list / status — Knowledge base overview
  • Obsidian compatible wiki output

Tech Stack

PageIndex, markitdown, OpenAI Agents SDK, LiteLLM, Click, watchdog

KylinMountainand others added 30 commits April 6, 2026 23:13
Sets up pyproject.toml (hatchling, direct-refs allowed, Python >=3.11),
.gitignore, openkb/__init__.py, a Click CLI stub with all 7 commands
(init, add, query, watch, lint, list, status), and tests/conftest.py
with kb_dir and sample_tree fixtures. Package installs cleanly in a
Python 3.12 venv; okb --help shows all commands; pytest collects 0
tests without error.
Add openkb/config.py (DEFAULT_CONFIG, load_config, save_config),
openkb/state.py (HashRegistry with SHA-256 file hashing and JSON
persistence), and openkb/schema.py (SCHEMA_MD constant). All 17 tests
written first (red) then implemented (green).
Creates full KB directory structure (raw/, wiki/sources/images/,
wiki/summaries/, wiki/concepts/, wiki/reports/), writes SCHEMA.md,
index.md, config.yaml and hashes.json; guards against re-initialisation.
Three tests in tests/test_cli.py cover structure, schema content, and
the already-initialized guard, all via CliRunner.isolated_filesystem.
Implements extract_base64_images and copy_relative_images with full test
coverage for single/multiple images, invalid base64, missing files, and
URL filtering.
Implements ConvertResult dataclass, get_pdf_page_count, and
convert_document with hash-dedup, markdown passthrough, PDF long-doc
detection, MarkItDown conversion, and image extraction integration.
Implements render_source_md and render_summary_md with YAML frontmatter,
recursive heading hierarchy (h1–h6 capped), page ranges, and separate
text/summary views for source and summary wiki pages.
Implements IndexResult dataclass and index_long_document which creates
a LocalClient with full node text/summary/description flags, adds the
PDF via PageIndex, fetches structure, and writes source and summary
wiki pages via the tree renderer.
Implements list_wiki_files, read_wiki_file, and write_wiki_file as plain
functions in openkb/agent/tools.py without @function_tool decoration,
ready to be wrapped when building the agent. Full test coverage including
edge cases for missing files/dirs, filtering to .md only, and parent dir
creation.
Implements build_compiler_agent, compile_short_doc, compile_long_doc in
openkb/agent/compiler.py with function_tool-wrapped wiki tools and
SCHEMA_MD-enriched instructions. Long-doc variant includes get_page_content.
Tests mock Runner.run to avoid real LLM calls.
Replaces the add stub with full orchestration: convert_document,
index_long_document for long PDFs, and compiler agent calls.
Adds SUPPORTED_EXTENSIONS set, _find_kb_dir, _add_single_file helpers.
Adds python-dotenv dependency and load_dotenv() at startup.
Implements pageindex_retrieve (structure -> LLM relevance -> page fetch),
build_query_agent with list/read/retrieve tools, and run_query coroutine.
Wires up `okb query` in cli.py.
Implements DebouncedHandler (collects events, ignores dirs/dotfiles, resets
timer on burst) and watch_directory (Observer loop, Ctrl+C safe).
Wires up `okb watch` in cli.py.
Implements find_broken_links, find_orphans, find_missing_entries,
check_index_sync, and run_structural_lint with full Markdown report.
Covers wikilink resolution, orphan detection, raw/wiki entry matching,
and index.md sync checking.
Implements build_lint_agent with list/read tools and instructions for
semantic quality checks (contradictions, gaps, staleness, redundancy).
run_knowledge_lint runs the agent and returns the report string.
okb lint combines structural + knowledge lint and writes timestamped report.
Tests verify list shows documents table and concepts, status shows
per-directory file counts and total indexed. Both check missing-init guard.
Previously the converter registered the file hash immediately, so if
LLM compilation failed the file was marked as "done" and retries
would skip it. Now the hash is only registered by the CLI after
successful compilation.
Also: install markitdown[all] for PDF support, add python-dotenv.
…pport
- Switch from col._backend.get_document_structure() to col.get_document_structure()
- Add 3x retry for PageIndex indexing (stochastic TOC accuracy)
- Fix storage path to use .db extension
- Remove .doc from supported extensions (markitdown only supports .docx)
- Note: col.get_page_content() still missing from PageIndex public API,
using col._backend.get_page_content() as workaround
Replace col._backend.get_page_content(col._name, doc_id, spec) with
col.get_page_content(doc_id, spec). Now all PageIndex access uses
public API only.
rejojer added 15 commits April 8, 2026 05:01
Rename CLI command and state dir from okb to openkb
- Hardcode reading LLM_API_KEY env var instead of indirecting through config
- Remove llm_api_key_env from DEFAULT_CONFIG, okb init prompts, and config.yaml
- Provider-specific env vars (OPENAI_API_KEY, etc.) still work via LiteLLM auto-detection
- One less config field, one less okb init step
The OpenAI Agents SDK requires a litellm/ prefix to route non-OpenAI
models through LiteLLM. Without it, models like anthropic/claude-sonnet-4-6
fail with "Unknown prefix". This adds the prefix at all Agent() call sites
while keeping litellm.completion() calls unchanged.
Also updates README quick start comments and model format docs.
Fix: add litellm/ prefix for Agents SDK model routing
@KylinMountain

Copy link
Copy Markdown
CollaboratorAuthor

Code review

Found 1 issue:

  1. extract_pdf_images and convert_pdf_with_images in images.py open pymupdf documents with explicit .close() instead of context managers. If an exception is raised during page iteration (e.g. corrupt image block, pixmap allocation failure), the PDF file handle leaks. This is the same bug pattern that was already fixed in converter.py:get_pdf_page_count (commit c525455), but images.py was missed. Fix: replace doc = pymupdf.open(...) / doc.close() with with pymupdf.open(...) as doc:.

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] !=1: # not an image block
continue
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
save_path=images_dir/filename
pix.save(str(save_path))
pix=None
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
continue
rel_path=f"images/{doc_name}/{filename}"
page_images.setdefault(page_num, []).append(rel_path)
doc.close()
returnpage_images

OpenKB/openkb/images.py

Lines 89 to 125 in 1637697

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
parts.append(f"\n\n<!-- Page {page_num} -->\n")
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] ==0: # text block
lines= []
forlineinblock["lines"]:
spans_text="".join(span["text"] forspaninline["spans"])
lines.append(spans_text)
parts.append("\n".join(lines))
elifblock["type"] ==1: # image block
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
(images_dir/filename).write_bytes(pix.tobytes("png"))
pix=None
parts.append(f"\n![image](images/{doc_name}/{filename})\n")
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
doc.close()
return"\n".join(parts)

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer

Copy link
Copy Markdown
Member

Code review

Found 2 issues:

  1. README claims LiteLLM is "pinned to a safe version" but pyproject.toml has no version pin. Line 67 of README.md states LiteLLM is (pinned to a safe version), but pyproject.toml line 17 lists the dependency as bare "litellm" with no version constraint (==, >=, ~=, etc.). Any version -- including potentially insecure ones -- can be installed.

OpenKB/README.md

Lines 66 to 68 in 854294c

OpenKB comes with [multi-LLM support](https://docs.litellm.ai/docs/providers) (e.g., OpenAI, Claude, Gemini) via [LiteLLM](https://github.com/BerriAI/litellm) (pinned to a [safe version](https://docs.litellm.ai/blog/security-update-march-2026)).

OpenKB/pyproject.toml

Lines 16 to 18 in 854294c

"watchdog>=3.0",
"litellm",
"openai-agents",

  1. test_short_pdf_converted_via_markitdown mocks the wrong code path. The test patches openkb.converter.MarkItDown and openkb.converter.pymupdf.open, but converter.py line 99-101 routes short PDFs through convert_pdf_with_images() (from openkb.images), not MarkItDown. The MarkItDown mock is never exercised, and convert_pdf_with_images is not mocked, so the test either fails at runtime or passes for the wrong reasons.

classTestConvertDocumentPdfShort:
deftest_short_pdf_converted_via_markitdown(self, kb_dir, tmp_path):
"""PDF under threshold is converted with markitdown."""
src=tmp_path/"short.pdf"
src.write_bytes(b"%PDF-1.4 fake content")
fake_result=MagicMock()
fake_result.text_content="# Short PDF\n\nConverted content."
with (
patch("openkb.converter.pymupdf.open") asmock_mu,
patch("openkb.converter.MarkItDown") asmock_mid_cls,
):
fake_doc=MagicMock()
fake_doc.page_count=5# below default threshold of 20
fake_doc.__enter__=MagicMock(return_value=fake_doc)
fake_doc.__exit__=MagicMock(return_value=False)
mock_mu.return_value=fake_doc
mock_mid_cls.return_value.convert.return_value=fake_result
result=convert_document(src, kb_dir)
assertresult.skippedisFalse
assertresult.is_long_docisFalse
assertresult.source_pathisnotNone
assertresult.source_path.exists()

markdown=copy_relative_images(markdown, src.parent, doc_name, images_dir)
elifsrc.suffix.lower() ==".pdf":
# Use pymupdf dict-mode for PDFs: text + images inline at correct positions
markdown=convert_pdf_with_images(src, doc_name, images_dir)
else:

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer
rejojer merged commit f0963f6 into mainApr 8, 2026
KylinMountain added a commit that referenced this pull request May 24, 2026
Architectural review (4 parallel Opus auditors) found that the skill_runner
core was already generic, but the deck SURFACE was still fused to
Editorial Monocle. Fixed:
* validator: now takes optional `grammar` param (DeckGrammar TypedDict);
skill-agnostic by default (only checks file present, parses, ≥5
slides, self-contained). Third-party deck skills (guizang, swiss)
now pass validation cleanly. Editorial-specific rules opt-in via
`EDITORIAL_MONOCLE_GRAMMAR`. (finding #2)
* skills/openkb-deck-editorial/SKILL.md: declares its grammar +
output_path_template under `od:` frontmatter — `run_skill` reads
these and applies them post-run.
* run_skill: now honors frontmatter `od.mode`, `od.output_path_template`,
`od.deck_grammar`. When mode=="deck" and template is set, the runner
injects the path into intent, verifies the file exists post-run, and
runs validate_deck with the skill's grammar. Validation result is
returned via new SkillRunResult dataclass. (findings #4, #5)
* `openkb deck new --skill <name>`: CLI flag accepts any installed deck
skill (default openkb-deck-editorial). guizang and swiss now usable
from the scripted CLI, not only freeform chat. (finding #1)
* `/deck new --skill <name>` chat slash: same flag, parsed positionally
alongside --critique. (finding #1)
* tests/test_read_kb_file.py: 13 new tests mirroring test_write_kb_file
for the read-side allow-list. Pins refusal of `.openkb/config.yaml`,
`.env`, `raw/`, `..` traversal, absolute paths. (finding #6)
* Generator deck branch: no longer calls validate_deck directly; just
propagates run_deck_create's SkillRunResult.validation up. Validation
is now a property of "this skill declared mode=deck", not of "this
CLI path was taken".
Existing tests updated:
* tests/test_deck_validator.py: explicit grammar arg on Editorial-
specific tests; added test_guizang_shape_passes_generic_mode +
test_missing_cover_ignored_in_generic_mode to pin both modes.
* tests/test_deck_creator.py: mocks return SkillRunResult; new
test_run_deck_create_honors_skill_name_override for --skill flag.
* tests/test_generator.py: deck dispatch test mocks SkillRunResult.
Below-threshold findings deferred:
* Generator if/else → registry (score 70) — works, just not extensible
via plugin; future.
* Iteration backup in chat freeform path (score 75) — needs write_kb_file
hook; separate change.
* run_skill / scan_local_skills / _handle_slash_critique direct tests
(scores 60-70) — covered indirectly by integration; can add later.
Regression: 538 tests pass (was 523 pre-fix; net +15 = 13 new
read_kb_file tests + 2 new validator-mode tests).
KylinMountain added a commit that referenced this pull request May 31, 2026
…lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
KylinMountain added a commit that referenced this pull request Jun 1, 2026
…mpile backfill (#78)
* feat(compiler): _read_entity_briefs for entity plan context
* test(compiler): parity tests for _read_entity_briefs
* feat(compiler): _write_entity with type/aliases frontmatter
* test(compiler): assert source ordering in _write_entity; count=1 in _set_fm_line
Add explicit ordering assertion in test_update_prepends_source_keeps_type
verifying the deterministic json.dumps form ("summaries/b.md", "summaries/a.md").
Pass count=1 to re.sub in _set_fm_line to make first-occurrence intent explicit.
* feat(lint): include entities/ in wikilink whitelist
* feat(compiler): summary<->entity backlinks
* test(compiler): restore assertion erroneously deleted in 3c8aa93
* feat(compiler): index.md Entities section
* feat(compiler): remove_doc_from_entity_pages + index cleanup
* feat(compiler): plan prompt + parser for entities group
Also wires the entity track into _compile_concepts (Tasks 7 + 8 combined,
since the {entity_briefs} placeholder and the _CONCEPTS_PLAN_USER.format call
are co-dependent — splitting would leave an intermediate red state).
- add _ENTITY_TYPES, _filter_entity_items, _parse_entities_plan
- rewrite _CONCEPTS_PLAN_USER to request nested concepts+entities groups
- add _ENTITY_PAGE_USER / _ENTITY_UPDATE_USER prompts
- read entity briefs and pass both briefs to the plan prompt
- parse nested 'concepts' group with legacy flat-list/flat-dict fallbacks
- generate entities in their own asyncio.gather (4-arity tuples)
- strip ghost links + _write_entity each; handle entity related cross-links
- backlink summary<->entities; pass entity_names/entity_meta to _update_index
* fix(compiler): related entities must not downgrade index labels
Mirror the concept track: collect related-entity slugs into a separate
local list used only for backlinks; pass only created/updated entity_names
(+entity_meta) to _update_index. Defense-in-depth in _update_index: only
_replace_section_entry when name is in entity_meta, otherwise only insert
if the link is absent, so a related-only entity can never clobber a
pre-existing correct (type + brief) index line with "(other)".
Adds regression test test_related_entity_does_not_downgrade_index_label.
* feat(schema): declare entities/ page type and taxonomy
* feat(query): point who/what questions at entities/
* docs(readme): document entities/ page type
* feat(cli): scaffold entities/ in init and count it in status
- `openkb init` now creates wiki/entities/ alongside wiki/concepts/
- init seed index.md gains ## Entities between ## Concepts and ## Explorations,
matching the _update_index template in compiler.py
- print_status subdirs list gains "entities" after "concepts"
- Tests updated: assert wiki/entities/ exists and index.md contains ## Entities;
status test asserts "entities" appears in output
* fix(compiler): resolve entity-page review findings (dangling links + dedup)
Addresses code-review findings on the entity-pages feature:
- Fix dangling wikilink after `openkb remove`: entity removal now strips
standalone `See also: [[summaries/{doc}]]` lines (the related-entity
backlink form), matching the concept path, and cli.py adds modified
entity pages to the lint sweep scope so surviving pages are cleaned.
- Unify the parallel concept/entity helpers into shared cores
(_backlink_summary_pages, _backlink_pages, _remove_doc_from_pages) with
thin per-type wrappers, so cleanup logic can no longer drift between the
two page types (this is what caused the dangling-link bug).
- Route related-entity cross-refs through _add_related_link (now page-type
aware) instead of an inline reimplementation — removes a duplicate file
read/write and keeps backlink creation symmetric with teardown.
- Centralize the entity-type enum: prompts derive their type list from a
single _ENTITY_TYPE_LIST source via import-time substitution.
- Count entity items in the "all dropped as malformed" plan warning.
- Drop the unreachable else branch in _update_index's entity loop.
- Add regression test for the See-also strip on a surviving entity page.
All 542 tests pass.
* fix(compiler): add [[entities/X]] whitelist rule + restore concept-topic guard
Remaining review findings after a7a06ed:
- _KNOWN_TARGETS_USER now states the [[entities/Z]] rule, so entity links
the LLM is told to write aren't silently stripped as ghosts.
- Restore the dropped 'Do NOT create concepts that are just the document
topic itself' plan rule to prevent redundant title-mirror concepts.
* feat(entities): shared page-dir constants + surface entities in list/lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
* feat(entities): remove preview lists entity-page actions (#1)
The dry-run/confirmation block now scans wiki/entities/ with the same
frontmatter sources: logic as concepts, emits DELETE/MODIFY action lines
per entity page, and prints an 'N entity(s) will be DELETED' summary.
Execution path (remove_doc_from_entity_pages) unchanged.
* docs(entities): document entity pages in shipped openkb skill (#8)
Note wiki/entities/ holds named-thing pages (people/orgs/places/
products/works/events) with a type: frontmatter field, that index.md
has a ## Entities section, and that 'who/what is X' questions should
read the matching entities/ page first.
* fix(compiler): don't write raw JSON body on empty LLM content
In the parse-succeeded branch of _gen_create/_gen_update/_gen_entity_create/
_gen_entity_update, fall back to "" instead of the raw JSON string when the
content field is empty/null. _require_nonempty_content then raises and the
page is dropped, rather than writing the JSON envelope as the markdown body.
The parse-FAILED (except) branch keeps content=raw as the legitimate
non-JSON fallback.
* fix(compiler): graceful scalar plan + rebuild malformed entity frontmatter
- _compile_concepts: guard a non-dict/non-list parsed plan (JSON scalar)
before calling .get(), taking the empty-plan path (write v1 summary if
applicable + update index + return) instead of risking AttributeError.
- _write_entity: when an existing page has an opening --- but no closing
delimiter (or no frontmatter), rebuild valid sources/type/brief frontmatter
rather than writing a body-only page that drops the metadata.
* fix(compiler): keep ## Entities before ## Explorations; drop dead param + overlap gathers
- _update_index: insert ## Entities before ## Explorations on older index.md
files that predate the section (new _ensure_h2_section_before helper),
preserving canonical order instead of appending at EOF.
- _filter_entity_items: drop the unused 'label' parameter and update call
sites in _parse_entities_plan.
- _compile_concepts: overlap concept and entity generation in one outer
asyncio.gather (they share cached context and the same concurrency
semaphore); result/error handling per list is unchanged.
* test(compiler): cover empty-content skip, scalar plan, malformed entity FM, Entities order
Add regression tests for the four compiler fixes:
- empty {"content":""} response skips the page (no raw JSON body)
- JSON scalar plan handled gracefully (no AttributeError)
- _write_entity rebuilds frontmatter when closing --- is missing
- _update_index inserts ## Entities before ## Explorations
* fix(compiler): silence spurious 'hand-edited' warning on backlink section creation
_backlink_summary_pages / _backlink_pages create ## Entities / ## Related
Documents sections as a normal first-time operation; pass quiet=True so
_ensure_h2_section no longer logs the index-drift warning in that case.
Index-repair callers keep the warning.
* feat(cli): add `recompile` command to re-run compile on indexed docs
Re-runs the current compile_short_doc/compile_long_doc pipeline on
already-indexed docs so pre-feature KBs gain the entities/ layer and
refresh to the current format. Reuses on-disk sources/summaries and the
registry's PageIndex doc_id — does not re-index or re-convert.
Supports a positional <doc_name> (resolved via _resolve_doc_identifier)
or --all (with a regeneration-warning confirmation, bypassed by --yes),
--dry-run (enumerate only, no LLM calls/writes), and --refresh-schema
(back up + overwrite wiki/AGENTS.md when it differs from AGENTS_MD).
Processes docs sequentially with per-doc progress, skips+warns on
missing sources / summaries / doc_id, prints a recompiled/skipped
summary, and appends a recompile entry to log.md.
* test(cli): recompile dispatch/dry-run/skip/refresh-schema
* docs(readme): document openkb recompile
* fix(cli): recompile --refresh-schema no-ops when AGENTS.md absent; tighten guard tests
Match the spec (and the helper's own docstring): _refresh_schema returns
early when wiki/AGENTS.md is missing rather than materializing the default
(get_agents_md already falls back to it at runtime). Tighten the doc/--all
guard tests to assert the exact message + that no compile runs, and add the
missing-AGENTS.md no-op test.
* fix(compiler): drop non-existent 'related' slugs so they don't create dangling links
The plan's 'related' list is meant to reference existing pages, but the LLM
sometimes lists slugs for pages that don't exist. Those were added to the
wikilink whitelist (so body references survived ghost-stripping) and
back-linked into the summary's Related section, yet no page was ever created
(related items are linked, never generated) — producing a flood of broken
[[concepts/...]] / [[entities/...]] links (esp. on feature-dense docs).
Filter related_items / entity_related to slugs that exist on disk.
* fix: remove-preview detects JSON-quoted sources; _write_entity preserves sources on malformed FM
- remove --dry-run preview parsed the sources list with a hand-rolled comma
split that kept JSON quotes (["summaries/x.md"]), so the marker never
matched and the preview always reported 0 affected concept/entity pages
(executor was correct). Extract _scan_affected_pages using the real
_parse_yaml_list_value; dedups the two copied scan loops too.
- _write_entity's malformed-frontmatter rebuild seeded sources with only the
new doc, dropping prior sources for multi-source entities. Recover existing
sources from the broken block and merge.
Both bugs were masked by tests using unquoted / single-source fixtures.
* feat(cli): rename remove --keep-empty-concepts → --keep-empty (covers entities too)
This PR wired entity pages into 'openkb remove', so the flag now governs
concept AND entity retention — but the name still said 'concepts'. Make
--keep-empty the canonical name (clear that it covers both), keep
--keep-empty-concepts as a backward-compatible alias, and update the
preview/summary messages, docstring, and README accordingly.
* feat(compiler): config-driven entity types (entity_types overrides the default enum)
Add an optional 'entity_types:' key in .openkb/config.yaml. When present it
overrides the default person/organization/place/product/work/event/other
vocabulary everywhere — the plan prompt, the entity-page prompts, and
create/update validation/coercion; when absent, behavior is byte-identical.
Prompt templates keep an __ENTITY_TYPES__ token now substituted at call time
(per-KB) inside _compile_concepts, and the resolved valid-type set is threaded
into _parse_entities_plan / _filter_entity_items and the _gen_entity_* coercion.
'other' is always ensured as the coercion fallback; malformed config falls back
to the default with a warning. Documented in config.yaml.example + README.
* fix(compiler): harden config-driven entity types (crash-proof + complete the override)
Review of the config-entity-types feature surfaced two real issues:
- A config 'entity_types' value containing '{' or '}' was substituted into the
prompt template BEFORE .format() ran → KeyError/ValueError crashing every
compile. Swap to format-then-replace at all 3 call sites (types_str is now an
inert literal), and sanitize resolved types to a safe label charset (also
skips YAML nulls/ints so str(None) can't become the type 'none').
- The AGENTS_MD system schema hardcoded 'type: is one of: <7 defaults>',
contradicting a custom entity_types in the higher-weight system message.
Reword it to frame those as the configurable default and defer the
authoritative set to the compilation prompt (which is config-driven).
Also drop the now-dead _ENTITY_TYPES_STR + its stale import-time-substitution
comment. +2 regression tests (sanitization; brace-in-type doesn't crash).
* refactor: move entity-type resolution to config layer + co-locate remove-preview scan
Altitude cleanups from the review:
- Move resolve_entity_types + DEFAULT_ENTITY_TYPES into openkb/config.py (the
config layer owns config validation/normalization; any command can reuse it
without importing the heavy compiler module). compiler.py imports them;
_ENTITY_TYPE_LIST/_ENTITY_TYPES remain as the default alias/validation set.
- Move the remove dry-run preview scan from cli.py into compiler.py as
scan_affected_pages, beside remove_doc_from_*_pages and sharing
_parse_yaml_list_value — so preview and executor can't drift on how the
sources list is parsed (root cause of the earlier JSON-quote preview bug).
---------
Co-authored-by: Claude <noreply@anthropic.com>
calebfavor added a commit to railroadmedia/MusoraOpenKB that referenced this pull request Jul 2, 2026
Implements docs/smart-hierarchy-distillation-plan.md — a RAPTOR-style bottom-up
distillation that builds a multi-layer, LLM-navigable pathway hierarchy over the
flat concept leaves, the intended pivot from the top-down bootstrap() cold-start.
Engine (openkb/topic_tree.py):
- distill(): reads leaves recursively, clusters each layer into LLM-named sized
categories, summarizes each into a parent pathway node, links same-layer peers
sideways, repeats to a single root. Invariants enforced: exactly one root,
always >= 2 layers, bounded depth, no concept loss. Builds into a staging dir
and atomically swaps in (mid-build LLM failure never loses concepts).
- write_pathway_md(): pathway node format — layer/children/related frontmatter +
distilled summary + linked child index + Related pathways section.
- Sideways links are bidirectional, same-layer, top-K, no self-links.
Config (openkb/config.py): HierarchyConfig + resolve_hierarchy() for the
`hierarchy:` block (target/min/max fanout, max_depth, summary token caps,
sideways settings) with validation + back-compat.
LLM callables (openkb/topic_tree_llm.py): make_distill_cluster (AGENTS.md-guided,
sized category naming), make_distill_summarize, make_relate.
Integration: `openkb distill` CLI command; AGENTS.md `## Hierarchy` guidance
section injected into distill prompts; query tree-descent prompt now follows
`related` sideways links; lint registers topic-dir names so pathway/sideways
wikilinks resolve.
Tests (+31): config parsing, distill invariants/edges/sideways/data-loss,
fake-LLM CLI integration + idempotent re-distill, and tier-4 regression pins
(VectifyAI#4 sideways links resolve — red without the lint change; VectifyAI#5 single-root/
min-2-layer edge sizes). Full suite: 937 passed, 10 llm deselected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
KylinMountain added a commit that referenced this pull request Jul 21, 2026
…es (#198)
* feat(api): delete a knowledge base (POST /api/v1/kb/delete + openkb delete-kb)
Physical, irreversible KB deletion with a type-the-name confirmation.
- config.delete_kb: rmtree the KB directory + unregister it from the global
registry (known_kbs / kb_aliases / default_kb, via the new unregister_kb).
Guards against deleting a non-KB path; tolerates a ghost registry entry
(directory already gone) by just unregistering.
- POST /api/v1/kb/delete: confirm_name must equal kb, re-checked server-side.
Lives in a new api_kbs_router (sibling of the config/graph/output routers) so
api.py stays under the 800-line gate; delete_kb self-locks (config lock), no
create_app closure needed.
- openkb delete-kb NAME: type-the-name prompt (or --yes) then delete.
Tests: endpoint (success + unregister, confirm mismatch, non-KB target) in
test_api.py; config (physical delete + unregister, refuse non-KB, ghost
tolerance) in test_config.py.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): delete a wiki page (POST /api/v1/page/delete) with backlink impact
Delete a concept/entity page and degrade references cleanly instead of leaving
them dangling:
- page_ops.delete_wiki_page: dry_run reports the backlink pages (whose inbound
[[wikilinks]] would be demoted) without touching anything; execute removes the
page under the KB ingest lock, strips its index.md entry outright
(compiler.remove_doc_from_index — the entry is a [[link]] line), and demotes
the now-dangling inbound [[links]] to plain text in ONLY those backlink pages
(lint.fix_broken_links restrict_to). The page ref is "section/stem", validated
to concepts/ or entities/ only (path-traversal-safe).
- New api_pages_router hosts POST /api/v1/page (read — relocated here from
api.py to keep it under the 800-line gate) + POST /api/v1/page/delete.
Tests: dry-run impact + execute (page gone, inbound link demoted to text, index
entry removed, sibling kept), 404 on missing page, and rejection of unsafe /
non-editable refs.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): edit a wiki page (PUT /api/v1/page) + link context (POST /api/v1/page/links)
- page_ops.edit_wiki_page: replace a concept/entity page's BODY while preserving
its code-managed OKF frontmatter (type/description/sources) verbatim; any
frontmatter in the submitted content is dropped. Dead [[wikilinks]] the user
typed are demoted to plain text on save (OpenKB keeps the wiki broken-link-free)
and returned as ghosts_stripped so the UI can warn. Atomic write under the KB
ingest lock.
- page_ops.page_link_context (POST /api/v1/page/links): outbound + inbound links
for the edit-impact panel. Editing the body does not break either (links are
path-based), so this is context, not a blocker — the honest impact model.
- PUT /api/v1/page added to api_pages_router.
Tests: edit preserves frontmatter + keeps a resolvable link + demotes a dead one
(reports ghosts_stripped) + replaces the body; 404 on a missing page; links
reports out/backlinks.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): delete a knowledge base from the settings sheet (type-name confirm)
Adds a Danger Zone section to KbSettingsSheet: a type-the-name confirmation
gates deleteKb() (POST /api/v1/kb/delete); on success KbDetail navigates back
to the KB list. New deleteKb client + kbSettings danger* keys + common cancel
(zh + en, identical key sets).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): in-reader page edit + delete with impact preview (F2/F3)
For an open concept/entity page, the reader header now shows Edit and Delete:
- Delete: a dry-run first (deletePage dryRun) surfaces the backlink pages whose
[[links]] will demote to plain text; a red confirm card lists them, then the
real delete closes the page and refreshes the inventory.
- Edit: a body textarea (frontmatter is preserved server-side), Save via
PUT /api/v1/page, a links panel (getPageLinks: out/backlinks), a "recompile
may overwrite" note, and a toast listing any dead links demoted to text.
Adds the deletePage/editPage/getPageLinks wiki clients and kb:pageOps.* locale
keys (zh + en, identical key sets). Build green (i18n guard + tsc + vite).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(workbench): address code-review findings (content-mgmt correctness + concurrency)
Backend (page_ops / kb_admin / lint):
- delete/edit now do their read-modify-write INSIDE the KB ingest lock and
re-check the page exists under it: no stale backlink snapshot, no resurrecting
a concurrently-deleted page, no 500 on a racing unlink (missing_ok=True). [#4,#5,#6]
- delete_kb serializes rmtree under the ingest lock (was fully unlocked). Moved
delete_kb/unregister_kb into a new openkb/kb_admin.py so config.py drops back
under the per-file line gate (751 lines). [#1,#15]
- page deletion no longer scans or rewrites lint's _EXCLUDED_FILES
(AGENTS.md/SCHEMA.md/log.md); fix_broken_links(restrict_to) also refuses them,
which protects `openkb remove` too. [#2]
- edit_wiki_page treats `content` as the body verbatim — no naive frontmatter
split that silently dropped a body opening with a '---' block. [#3]
- delete uses the real on-disk stem for the exact-match index-entry removal
(case-insensitive FS), and adds index.md to the demotion set so a [[target]]
embedded in another entry's brief no longer dangles. [#7,#9]
API / CLI:
- delete-KB endpoint & CLI can now clean a ghost registry entry (registered name
whose directory was removed by hand), catch delete_kb's ValueError -> 400, and
the endpoint is typed `-> KbDeleteResponse`. [#8,#13,E5]
Frontend (KbDetail):
- Deleting a KB dispatches `openkb:reload-kbs` so the sidebar refreshes
immediately (was stale until a manual reload) — live-test fix.
- Removed the dead `pageReloadSeq` path (editPage always returns content). [#14]
Tests: excluded-doc-untouched, '---'-body preservation, ghost-KB unregister;
non-KB-target test updated to the new resolution model. Backend 1234 passed,
mypy/ruff clean, frontend build green.
Not changed (intentional): pages_linking_to is NOT folded into build_graph (that
excludes explorations/ backlinks) [#10]; the preview+confirm delete inherently
scans per request [#11]; EDITABLE_SECTIONS stays duplicated in the frontend,
which can't import the Python constant [#12].
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(workbench): allow editing summary pages (edit ⊇ delete section sets)
Summaries are compiled markdown like concepts/entities (a recompile regenerates
them), so they are now editable via PUT /api/v1/page. They remain NOT
independently deletable — a summary is removed by deleting its source document.
Split page_ops into EDITABLE_SECTIONS (concepts/entities/summaries) vs
DELETABLE_SECTIONS (concepts/entities); validate_page_ref takes the allowlist.
Frontend ReaderBody gates Edit on canEdit (incl. summaries), Delete on canDelete.
Test: summary editable (frontmatter preserved) + summary delete rejected (400).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(api): cross-platform delete_kb + endpoint OSError handling (xhigh review #1,#2)
- delete_kb no longer holds the ingest lock DURING rmtree (the lock file lives
inside the KB dir; Windows cannot delete an open file — the prior review-fix
regressed this). It now takes the lock as a BARRIER (drain + wait out any
in-flight mutation), releases it, re-checks existence, then rmtrees. [#1]
- delete-KB endpoint maps FileNotFoundError to an idempotent success (a
concurrent delete already removed the tree) and other OSError to a clean 500
with a message, instead of an uncaught 500 stack trace. [#2]
Tests: endpoint OSError -> clean 500, FileNotFoundError -> 200 deleted.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…y-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…ane (#197) (#199)
* feat(web): read a document's converted source text in the Documents pane (#197)
The Documents pane listed each ingested doc (name/type/hash) with no way to
read the converted full text ingestion produced under wiki/sources/.
Backend: new POST /api/v1/document/source resolves a doc hash to its source
text — short docs read <doc_name>.md; long docs concatenate the per-page
<doc_name>.json (page text joined by a thematic break). Hash is the identifier
(unique, avoids doc_name/stem collisions); resolution prefers the registry's
stored source_path then falls back to the wiki/sources/<doc_name>.{md,json}
convention, with a path-traversal guard. Read-only (sources are do-not-edit).
Frontend: document rows are now clickable and open a wide read-only slide-out
reader (MarkdownView) — ESC/overlay/close to dismiss, independent scroll,
content cached + memoized per hash, native find-in-page preserved (no
virtualization). Delete stays inline (stopPropagation). Closed drawer is inert.
Known limitation: images embedded in long-doc pages are not rendered inline yet.
* fix(web): address xhigh code-review findings for the document reader (#197)
Correctness / a11y:
- Rebuild the reader drawer on Radix Dialog (like KbSettingsSheet) instead of
a hand-rolled overlay: proper modal focus trap, initial + return focus,
Escape, and background inert (was: aria-modal with none of it) [#4]. This
also removes the hand-rolled window keydown listener that re-subscribed every
render [#7].
- Restructure each document row so the open-reader target is a real <button>
and the delete control is a SIBLING, not nested. Keyboard-activating delete
no longer bubbles into opening the reader, and the invalid nested-interactive
markup is gone [#1, #5].
- Resolve a source file by the doc's own type (long → .json first, else .md),
so two docs sharing a doc_name each resolve to their own file rather than
whichever extension is tried first [#2].
- Guard source reads: skip non-dict page entries, reject non-list JSON, and
return a controlled 500 on corrupt/unreadable sources instead of an
unhandled exception [#3].
- Invalidate the per-hash content cache when the inventory changes, so a
reopen after recompile refetches instead of serving stale text [#6].
Fetch/cache/memoized body moved to DocumentsPane so they survive the drawer's
unmount-on-close. #8 (frontmatter stripping) intentionally not applied: source
docs render verbatim (a user's own frontmatter is content, unlike wiki-page OKF
metadata). Adds tests for the collision and malformed-JSON paths.
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…verride) (#200)
* feat(api): configurable entity_types (global + per-KB) via the config API
entity_types (the entity-extraction vocabulary) is now surfaced through the
config read/write API, layering global.yaml -> KB config.yaml like the other
scalars: a KB list overrides the global list wholesale, an explicit null
inherits, and unset falls back to DEFAULT_ENTITY_TYPES. The compiler already
consumed config["entity_types"] via resolve_entity_types; this just exposes it.
- GLOBAL_SCALAR_KEYS gains "entity_types" (layering + per-key `sources` tracking;
the value-not-None-wins rule is type-agnostic, so it works for a list).
- _KbConfigWritable / GlobalConfigValues / KbConfigResponse / GlobalConfigResponse
carry entity_types; read_kb_config/read_global_config report the cleaned
EFFECTIVE list (resolve_entity_types) plus the raw global value for the badge.
- PATCH /api/v1/kb/config and PATCH /api/v1/config accept entity_types.
Tests: KB override (cleaned + source 'kb') + null revert, global patch, global
inheritance; updated the global-defaults shape assertion. Frontend UI follows.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): entity-types config UI — chips editor (global default + per-KB override)
- EntityTypesEditor: shared controlled chips editor (Enter/comma to add, x to
remove; "other" is a fixed always-included chip; IME-safe composition).
- KbSettingsSheet: an EntityTypesRow with the same inherit/override Switch as the
scalar rows — turning override on seeds+persists the KB's own list, off reverts
via null; inherited state shows the global/default list as a badge. Each chip
change persists and adopts the server-cleaned response.
- Settings (general tab): a global entity-types chips editor, order-sensitive
diff into the save patch (joins the existing dirty/SaveBar flow).
- "changes affect future recompiles only" note on both surfaces.
New keys in common/kbSettings/settings (zh + en, identical sets). Build green
(i18n guard OK). Backend was committed in 92f8f41.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(entity-types): address xhigh review — DEFAULT_CONFIG parity, empty-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@KylinMountain@rejojer@zmtomorrow
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex by KylinMountain · Pull Request #4 · VectifyAI/OpenKB · GitHub
Skip to content

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex - #4

Merged
rejojer merged 102 commits into
mainfrom
dev
Apr 8, 2026
Merged

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex#4
rejojer merged 102 commits into
mainfrom
dev

Conversation

@KylinMountain

Copy link
Copy Markdown
Collaborator

Summary

OpenKB — Karpathy's LLM Knowledge Base workflow as a CLI, powered by PageIndex.

Drop documents in. Get an auto-maintained, cross-linked wiki out.

Features

  • okb init — Interactive setup
  • okb add — Short docs (pymupdf) + long PDFs (PageIndex local/cloud)
  • okb query — Streaming Q&A with PageIndex cloud streaming
  • okb watch — Auto-compile on file changes
  • okb lint — Structural + knowledge health checks
  • okb list / status — Knowledge base overview
  • Obsidian compatible wiki output

Tech Stack

PageIndex, markitdown, OpenAI Agents SDK, LiteLLM, Click, watchdog

KylinMountainand others added 30 commits April 6, 2026 23:13
Sets up pyproject.toml (hatchling, direct-refs allowed, Python >=3.11),
.gitignore, openkb/__init__.py, a Click CLI stub with all 7 commands
(init, add, query, watch, lint, list, status), and tests/conftest.py
with kb_dir and sample_tree fixtures. Package installs cleanly in a
Python 3.12 venv; okb --help shows all commands; pytest collects 0
tests without error.
Add openkb/config.py (DEFAULT_CONFIG, load_config, save_config),
openkb/state.py (HashRegistry with SHA-256 file hashing and JSON
persistence), and openkb/schema.py (SCHEMA_MD constant). All 17 tests
written first (red) then implemented (green).
Creates full KB directory structure (raw/, wiki/sources/images/,
wiki/summaries/, wiki/concepts/, wiki/reports/), writes SCHEMA.md,
index.md, config.yaml and hashes.json; guards against re-initialisation.
Three tests in tests/test_cli.py cover structure, schema content, and
the already-initialized guard, all via CliRunner.isolated_filesystem.
Implements extract_base64_images and copy_relative_images with full test
coverage for single/multiple images, invalid base64, missing files, and
URL filtering.
Implements ConvertResult dataclass, get_pdf_page_count, and
convert_document with hash-dedup, markdown passthrough, PDF long-doc
detection, MarkItDown conversion, and image extraction integration.
Implements render_source_md and render_summary_md with YAML frontmatter,
recursive heading hierarchy (h1–h6 capped), page ranges, and separate
text/summary views for source and summary wiki pages.
Implements IndexResult dataclass and index_long_document which creates
a LocalClient with full node text/summary/description flags, adds the
PDF via PageIndex, fetches structure, and writes source and summary
wiki pages via the tree renderer.
Implements list_wiki_files, read_wiki_file, and write_wiki_file as plain
functions in openkb/agent/tools.py without @function_tool decoration,
ready to be wrapped when building the agent. Full test coverage including
edge cases for missing files/dirs, filtering to .md only, and parent dir
creation.
Implements build_compiler_agent, compile_short_doc, compile_long_doc in
openkb/agent/compiler.py with function_tool-wrapped wiki tools and
SCHEMA_MD-enriched instructions. Long-doc variant includes get_page_content.
Tests mock Runner.run to avoid real LLM calls.
Replaces the add stub with full orchestration: convert_document,
index_long_document for long PDFs, and compiler agent calls.
Adds SUPPORTED_EXTENSIONS set, _find_kb_dir, _add_single_file helpers.
Adds python-dotenv dependency and load_dotenv() at startup.
Implements pageindex_retrieve (structure -> LLM relevance -> page fetch),
build_query_agent with list/read/retrieve tools, and run_query coroutine.
Wires up `okb query` in cli.py.
Implements DebouncedHandler (collects events, ignores dirs/dotfiles, resets
timer on burst) and watch_directory (Observer loop, Ctrl+C safe).
Wires up `okb watch` in cli.py.
Implements find_broken_links, find_orphans, find_missing_entries,
check_index_sync, and run_structural_lint with full Markdown report.
Covers wikilink resolution, orphan detection, raw/wiki entry matching,
and index.md sync checking.
Implements build_lint_agent with list/read tools and instructions for
semantic quality checks (contradictions, gaps, staleness, redundancy).
run_knowledge_lint runs the agent and returns the report string.
okb lint combines structural + knowledge lint and writes timestamped report.
Tests verify list shows documents table and concepts, status shows
per-directory file counts and total indexed. Both check missing-init guard.
Previously the converter registered the file hash immediately, so if
LLM compilation failed the file was marked as "done" and retries
would skip it. Now the hash is only registered by the CLI after
successful compilation.
Also: install markitdown[all] for PDF support, add python-dotenv.
…pport
- Switch from col._backend.get_document_structure() to col.get_document_structure()
- Add 3x retry for PageIndex indexing (stochastic TOC accuracy)
- Fix storage path to use .db extension
- Remove .doc from supported extensions (markitdown only supports .docx)
- Note: col.get_page_content() still missing from PageIndex public API,
using col._backend.get_page_content() as workaround
Replace col._backend.get_page_content(col._name, doc_id, spec) with
col.get_page_content(doc_id, spec). Now all PageIndex access uses
public API only.
rejojer added 15 commits April 8, 2026 05:01
Rename CLI command and state dir from okb to openkb
- Hardcode reading LLM_API_KEY env var instead of indirecting through config
- Remove llm_api_key_env from DEFAULT_CONFIG, okb init prompts, and config.yaml
- Provider-specific env vars (OPENAI_API_KEY, etc.) still work via LiteLLM auto-detection
- One less config field, one less okb init step
The OpenAI Agents SDK requires a litellm/ prefix to route non-OpenAI
models through LiteLLM. Without it, models like anthropic/claude-sonnet-4-6
fail with "Unknown prefix". This adds the prefix at all Agent() call sites
while keeping litellm.completion() calls unchanged.
Also updates README quick start comments and model format docs.
Fix: add litellm/ prefix for Agents SDK model routing
@KylinMountain

Copy link
Copy Markdown
CollaboratorAuthor

Code review

Found 1 issue:

  1. extract_pdf_images and convert_pdf_with_images in images.py open pymupdf documents with explicit .close() instead of context managers. If an exception is raised during page iteration (e.g. corrupt image block, pixmap allocation failure), the PDF file handle leaks. This is the same bug pattern that was already fixed in converter.py:get_pdf_page_count (commit c525455), but images.py was missed. Fix: replace doc = pymupdf.open(...) / doc.close() with with pymupdf.open(...) as doc:.

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] !=1: # not an image block
continue
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
save_path=images_dir/filename
pix.save(str(save_path))
pix=None
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
continue
rel_path=f"images/{doc_name}/{filename}"
page_images.setdefault(page_num, []).append(rel_path)
doc.close()
returnpage_images

OpenKB/openkb/images.py

Lines 89 to 125 in 1637697

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
parts.append(f"\n\n<!-- Page {page_num} -->\n")
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] ==0: # text block
lines= []
forlineinblock["lines"]:
spans_text="".join(span["text"] forspaninline["spans"])
lines.append(spans_text)
parts.append("\n".join(lines))
elifblock["type"] ==1: # image block
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
(images_dir/filename).write_bytes(pix.tobytes("png"))
pix=None
parts.append(f"\n![image](images/{doc_name}/{filename})\n")
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
doc.close()
return"\n".join(parts)

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer

Copy link
Copy Markdown
Member

Code review

Found 2 issues:

  1. README claims LiteLLM is "pinned to a safe version" but pyproject.toml has no version pin. Line 67 of README.md states LiteLLM is (pinned to a safe version), but pyproject.toml line 17 lists the dependency as bare "litellm" with no version constraint (==, >=, ~=, etc.). Any version -- including potentially insecure ones -- can be installed.

OpenKB/README.md

Lines 66 to 68 in 854294c

OpenKB comes with [multi-LLM support](https://docs.litellm.ai/docs/providers) (e.g., OpenAI, Claude, Gemini) via [LiteLLM](https://github.com/BerriAI/litellm) (pinned to a [safe version](https://docs.litellm.ai/blog/security-update-march-2026)).

OpenKB/pyproject.toml

Lines 16 to 18 in 854294c

"watchdog>=3.0",
"litellm",
"openai-agents",

  1. test_short_pdf_converted_via_markitdown mocks the wrong code path. The test patches openkb.converter.MarkItDown and openkb.converter.pymupdf.open, but converter.py line 99-101 routes short PDFs through convert_pdf_with_images() (from openkb.images), not MarkItDown. The MarkItDown mock is never exercised, and convert_pdf_with_images is not mocked, so the test either fails at runtime or passes for the wrong reasons.

classTestConvertDocumentPdfShort:
deftest_short_pdf_converted_via_markitdown(self, kb_dir, tmp_path):
"""PDF under threshold is converted with markitdown."""
src=tmp_path/"short.pdf"
src.write_bytes(b"%PDF-1.4 fake content")
fake_result=MagicMock()
fake_result.text_content="# Short PDF\n\nConverted content."
with (
patch("openkb.converter.pymupdf.open") asmock_mu,
patch("openkb.converter.MarkItDown") asmock_mid_cls,
):
fake_doc=MagicMock()
fake_doc.page_count=5# below default threshold of 20
fake_doc.__enter__=MagicMock(return_value=fake_doc)
fake_doc.__exit__=MagicMock(return_value=False)
mock_mu.return_value=fake_doc
mock_mid_cls.return_value.convert.return_value=fake_result
result=convert_document(src, kb_dir)
assertresult.skippedisFalse
assertresult.is_long_docisFalse
assertresult.source_pathisnotNone
assertresult.source_path.exists()

markdown=copy_relative_images(markdown, src.parent, doc_name, images_dir)
elifsrc.suffix.lower() ==".pdf":
# Use pymupdf dict-mode for PDFs: text + images inline at correct positions
markdown=convert_pdf_with_images(src, doc_name, images_dir)
else:

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer
rejojer merged commit f0963f6 into mainApr 8, 2026
KylinMountain added a commit that referenced this pull request May 24, 2026
Architectural review (4 parallel Opus auditors) found that the skill_runner
core was already generic, but the deck SURFACE was still fused to
Editorial Monocle. Fixed:
* validator: now takes optional `grammar` param (DeckGrammar TypedDict);
skill-agnostic by default (only checks file present, parses, ≥5
slides, self-contained). Third-party deck skills (guizang, swiss)
now pass validation cleanly. Editorial-specific rules opt-in via
`EDITORIAL_MONOCLE_GRAMMAR`. (finding #2)
* skills/openkb-deck-editorial/SKILL.md: declares its grammar +
output_path_template under `od:` frontmatter — `run_skill` reads
these and applies them post-run.
* run_skill: now honors frontmatter `od.mode`, `od.output_path_template`,
`od.deck_grammar`. When mode=="deck" and template is set, the runner
injects the path into intent, verifies the file exists post-run, and
runs validate_deck with the skill's grammar. Validation result is
returned via new SkillRunResult dataclass. (findings #4, #5)
* `openkb deck new --skill <name>`: CLI flag accepts any installed deck
skill (default openkb-deck-editorial). guizang and swiss now usable
from the scripted CLI, not only freeform chat. (finding #1)
* `/deck new --skill <name>` chat slash: same flag, parsed positionally
alongside --critique. (finding #1)
* tests/test_read_kb_file.py: 13 new tests mirroring test_write_kb_file
for the read-side allow-list. Pins refusal of `.openkb/config.yaml`,
`.env`, `raw/`, `..` traversal, absolute paths. (finding #6)
* Generator deck branch: no longer calls validate_deck directly; just
propagates run_deck_create's SkillRunResult.validation up. Validation
is now a property of "this skill declared mode=deck", not of "this
CLI path was taken".
Existing tests updated:
* tests/test_deck_validator.py: explicit grammar arg on Editorial-
specific tests; added test_guizang_shape_passes_generic_mode +
test_missing_cover_ignored_in_generic_mode to pin both modes.
* tests/test_deck_creator.py: mocks return SkillRunResult; new
test_run_deck_create_honors_skill_name_override for --skill flag.
* tests/test_generator.py: deck dispatch test mocks SkillRunResult.
Below-threshold findings deferred:
* Generator if/else → registry (score 70) — works, just not extensible
via plugin; future.
* Iteration backup in chat freeform path (score 75) — needs write_kb_file
hook; separate change.
* run_skill / scan_local_skills / _handle_slash_critique direct tests
(scores 60-70) — covered indirectly by integration; can add later.
Regression: 538 tests pass (was 523 pre-fix; net +15 = 13 new
read_kb_file tests + 2 new validator-mode tests).
KylinMountain added a commit that referenced this pull request May 31, 2026
…lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
KylinMountain added a commit that referenced this pull request Jun 1, 2026
…mpile backfill (#78)
* feat(compiler): _read_entity_briefs for entity plan context
* test(compiler): parity tests for _read_entity_briefs
* feat(compiler): _write_entity with type/aliases frontmatter
* test(compiler): assert source ordering in _write_entity; count=1 in _set_fm_line
Add explicit ordering assertion in test_update_prepends_source_keeps_type
verifying the deterministic json.dumps form ("summaries/b.md", "summaries/a.md").
Pass count=1 to re.sub in _set_fm_line to make first-occurrence intent explicit.
* feat(lint): include entities/ in wikilink whitelist
* feat(compiler): summary<->entity backlinks
* test(compiler): restore assertion erroneously deleted in 3c8aa93
* feat(compiler): index.md Entities section
* feat(compiler): remove_doc_from_entity_pages + index cleanup
* feat(compiler): plan prompt + parser for entities group
Also wires the entity track into _compile_concepts (Tasks 7 + 8 combined,
since the {entity_briefs} placeholder and the _CONCEPTS_PLAN_USER.format call
are co-dependent — splitting would leave an intermediate red state).
- add _ENTITY_TYPES, _filter_entity_items, _parse_entities_plan
- rewrite _CONCEPTS_PLAN_USER to request nested concepts+entities groups
- add _ENTITY_PAGE_USER / _ENTITY_UPDATE_USER prompts
- read entity briefs and pass both briefs to the plan prompt
- parse nested 'concepts' group with legacy flat-list/flat-dict fallbacks
- generate entities in their own asyncio.gather (4-arity tuples)
- strip ghost links + _write_entity each; handle entity related cross-links
- backlink summary<->entities; pass entity_names/entity_meta to _update_index
* fix(compiler): related entities must not downgrade index labels
Mirror the concept track: collect related-entity slugs into a separate
local list used only for backlinks; pass only created/updated entity_names
(+entity_meta) to _update_index. Defense-in-depth in _update_index: only
_replace_section_entry when name is in entity_meta, otherwise only insert
if the link is absent, so a related-only entity can never clobber a
pre-existing correct (type + brief) index line with "(other)".
Adds regression test test_related_entity_does_not_downgrade_index_label.
* feat(schema): declare entities/ page type and taxonomy
* feat(query): point who/what questions at entities/
* docs(readme): document entities/ page type
* feat(cli): scaffold entities/ in init and count it in status
- `openkb init` now creates wiki/entities/ alongside wiki/concepts/
- init seed index.md gains ## Entities between ## Concepts and ## Explorations,
matching the _update_index template in compiler.py
- print_status subdirs list gains "entities" after "concepts"
- Tests updated: assert wiki/entities/ exists and index.md contains ## Entities;
status test asserts "entities" appears in output
* fix(compiler): resolve entity-page review findings (dangling links + dedup)
Addresses code-review findings on the entity-pages feature:
- Fix dangling wikilink after `openkb remove`: entity removal now strips
standalone `See also: [[summaries/{doc}]]` lines (the related-entity
backlink form), matching the concept path, and cli.py adds modified
entity pages to the lint sweep scope so surviving pages are cleaned.
- Unify the parallel concept/entity helpers into shared cores
(_backlink_summary_pages, _backlink_pages, _remove_doc_from_pages) with
thin per-type wrappers, so cleanup logic can no longer drift between the
two page types (this is what caused the dangling-link bug).
- Route related-entity cross-refs through _add_related_link (now page-type
aware) instead of an inline reimplementation — removes a duplicate file
read/write and keeps backlink creation symmetric with teardown.
- Centralize the entity-type enum: prompts derive their type list from a
single _ENTITY_TYPE_LIST source via import-time substitution.
- Count entity items in the "all dropped as malformed" plan warning.
- Drop the unreachable else branch in _update_index's entity loop.
- Add regression test for the See-also strip on a surviving entity page.
All 542 tests pass.
* fix(compiler): add [[entities/X]] whitelist rule + restore concept-topic guard
Remaining review findings after a7a06ed:
- _KNOWN_TARGETS_USER now states the [[entities/Z]] rule, so entity links
the LLM is told to write aren't silently stripped as ghosts.
- Restore the dropped 'Do NOT create concepts that are just the document
topic itself' plan rule to prevent redundant title-mirror concepts.
* feat(entities): shared page-dir constants + surface entities in list/lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
* feat(entities): remove preview lists entity-page actions (#1)
The dry-run/confirmation block now scans wiki/entities/ with the same
frontmatter sources: logic as concepts, emits DELETE/MODIFY action lines
per entity page, and prints an 'N entity(s) will be DELETED' summary.
Execution path (remove_doc_from_entity_pages) unchanged.
* docs(entities): document entity pages in shipped openkb skill (#8)
Note wiki/entities/ holds named-thing pages (people/orgs/places/
products/works/events) with a type: frontmatter field, that index.md
has a ## Entities section, and that 'who/what is X' questions should
read the matching entities/ page first.
* fix(compiler): don't write raw JSON body on empty LLM content
In the parse-succeeded branch of _gen_create/_gen_update/_gen_entity_create/
_gen_entity_update, fall back to "" instead of the raw JSON string when the
content field is empty/null. _require_nonempty_content then raises and the
page is dropped, rather than writing the JSON envelope as the markdown body.
The parse-FAILED (except) branch keeps content=raw as the legitimate
non-JSON fallback.
* fix(compiler): graceful scalar plan + rebuild malformed entity frontmatter
- _compile_concepts: guard a non-dict/non-list parsed plan (JSON scalar)
before calling .get(), taking the empty-plan path (write v1 summary if
applicable + update index + return) instead of risking AttributeError.
- _write_entity: when an existing page has an opening --- but no closing
delimiter (or no frontmatter), rebuild valid sources/type/brief frontmatter
rather than writing a body-only page that drops the metadata.
* fix(compiler): keep ## Entities before ## Explorations; drop dead param + overlap gathers
- _update_index: insert ## Entities before ## Explorations on older index.md
files that predate the section (new _ensure_h2_section_before helper),
preserving canonical order instead of appending at EOF.
- _filter_entity_items: drop the unused 'label' parameter and update call
sites in _parse_entities_plan.
- _compile_concepts: overlap concept and entity generation in one outer
asyncio.gather (they share cached context and the same concurrency
semaphore); result/error handling per list is unchanged.
* test(compiler): cover empty-content skip, scalar plan, malformed entity FM, Entities order
Add regression tests for the four compiler fixes:
- empty {"content":""} response skips the page (no raw JSON body)
- JSON scalar plan handled gracefully (no AttributeError)
- _write_entity rebuilds frontmatter when closing --- is missing
- _update_index inserts ## Entities before ## Explorations
* fix(compiler): silence spurious 'hand-edited' warning on backlink section creation
_backlink_summary_pages / _backlink_pages create ## Entities / ## Related
Documents sections as a normal first-time operation; pass quiet=True so
_ensure_h2_section no longer logs the index-drift warning in that case.
Index-repair callers keep the warning.
* feat(cli): add `recompile` command to re-run compile on indexed docs
Re-runs the current compile_short_doc/compile_long_doc pipeline on
already-indexed docs so pre-feature KBs gain the entities/ layer and
refresh to the current format. Reuses on-disk sources/summaries and the
registry's PageIndex doc_id — does not re-index or re-convert.
Supports a positional <doc_name> (resolved via _resolve_doc_identifier)
or --all (with a regeneration-warning confirmation, bypassed by --yes),
--dry-run (enumerate only, no LLM calls/writes), and --refresh-schema
(back up + overwrite wiki/AGENTS.md when it differs from AGENTS_MD).
Processes docs sequentially with per-doc progress, skips+warns on
missing sources / summaries / doc_id, prints a recompiled/skipped
summary, and appends a recompile entry to log.md.
* test(cli): recompile dispatch/dry-run/skip/refresh-schema
* docs(readme): document openkb recompile
* fix(cli): recompile --refresh-schema no-ops when AGENTS.md absent; tighten guard tests
Match the spec (and the helper's own docstring): _refresh_schema returns
early when wiki/AGENTS.md is missing rather than materializing the default
(get_agents_md already falls back to it at runtime). Tighten the doc/--all
guard tests to assert the exact message + that no compile runs, and add the
missing-AGENTS.md no-op test.
* fix(compiler): drop non-existent 'related' slugs so they don't create dangling links
The plan's 'related' list is meant to reference existing pages, but the LLM
sometimes lists slugs for pages that don't exist. Those were added to the
wikilink whitelist (so body references survived ghost-stripping) and
back-linked into the summary's Related section, yet no page was ever created
(related items are linked, never generated) — producing a flood of broken
[[concepts/...]] / [[entities/...]] links (esp. on feature-dense docs).
Filter related_items / entity_related to slugs that exist on disk.
* fix: remove-preview detects JSON-quoted sources; _write_entity preserves sources on malformed FM
- remove --dry-run preview parsed the sources list with a hand-rolled comma
split that kept JSON quotes (["summaries/x.md"]), so the marker never
matched and the preview always reported 0 affected concept/entity pages
(executor was correct). Extract _scan_affected_pages using the real
_parse_yaml_list_value; dedups the two copied scan loops too.
- _write_entity's malformed-frontmatter rebuild seeded sources with only the
new doc, dropping prior sources for multi-source entities. Recover existing
sources from the broken block and merge.
Both bugs were masked by tests using unquoted / single-source fixtures.
* feat(cli): rename remove --keep-empty-concepts → --keep-empty (covers entities too)
This PR wired entity pages into 'openkb remove', so the flag now governs
concept AND entity retention — but the name still said 'concepts'. Make
--keep-empty the canonical name (clear that it covers both), keep
--keep-empty-concepts as a backward-compatible alias, and update the
preview/summary messages, docstring, and README accordingly.
* feat(compiler): config-driven entity types (entity_types overrides the default enum)
Add an optional 'entity_types:' key in .openkb/config.yaml. When present it
overrides the default person/organization/place/product/work/event/other
vocabulary everywhere — the plan prompt, the entity-page prompts, and
create/update validation/coercion; when absent, behavior is byte-identical.
Prompt templates keep an __ENTITY_TYPES__ token now substituted at call time
(per-KB) inside _compile_concepts, and the resolved valid-type set is threaded
into _parse_entities_plan / _filter_entity_items and the _gen_entity_* coercion.
'other' is always ensured as the coercion fallback; malformed config falls back
to the default with a warning. Documented in config.yaml.example + README.
* fix(compiler): harden config-driven entity types (crash-proof + complete the override)
Review of the config-entity-types feature surfaced two real issues:
- A config 'entity_types' value containing '{' or '}' was substituted into the
prompt template BEFORE .format() ran → KeyError/ValueError crashing every
compile. Swap to format-then-replace at all 3 call sites (types_str is now an
inert literal), and sanitize resolved types to a safe label charset (also
skips YAML nulls/ints so str(None) can't become the type 'none').
- The AGENTS_MD system schema hardcoded 'type: is one of: <7 defaults>',
contradicting a custom entity_types in the higher-weight system message.
Reword it to frame those as the configurable default and defer the
authoritative set to the compilation prompt (which is config-driven).
Also drop the now-dead _ENTITY_TYPES_STR + its stale import-time-substitution
comment. +2 regression tests (sanitization; brace-in-type doesn't crash).
* refactor: move entity-type resolution to config layer + co-locate remove-preview scan
Altitude cleanups from the review:
- Move resolve_entity_types + DEFAULT_ENTITY_TYPES into openkb/config.py (the
config layer owns config validation/normalization; any command can reuse it
without importing the heavy compiler module). compiler.py imports them;
_ENTITY_TYPE_LIST/_ENTITY_TYPES remain as the default alias/validation set.
- Move the remove dry-run preview scan from cli.py into compiler.py as
scan_affected_pages, beside remove_doc_from_*_pages and sharing
_parse_yaml_list_value — so preview and executor can't drift on how the
sources list is parsed (root cause of the earlier JSON-quote preview bug).
---------
Co-authored-by: Claude <noreply@anthropic.com>
calebfavor added a commit to railroadmedia/MusoraOpenKB that referenced this pull request Jul 2, 2026
Implements docs/smart-hierarchy-distillation-plan.md — a RAPTOR-style bottom-up
distillation that builds a multi-layer, LLM-navigable pathway hierarchy over the
flat concept leaves, the intended pivot from the top-down bootstrap() cold-start.
Engine (openkb/topic_tree.py):
- distill(): reads leaves recursively, clusters each layer into LLM-named sized
categories, summarizes each into a parent pathway node, links same-layer peers
sideways, repeats to a single root. Invariants enforced: exactly one root,
always >= 2 layers, bounded depth, no concept loss. Builds into a staging dir
and atomically swaps in (mid-build LLM failure never loses concepts).
- write_pathway_md(): pathway node format — layer/children/related frontmatter +
distilled summary + linked child index + Related pathways section.
- Sideways links are bidirectional, same-layer, top-K, no self-links.
Config (openkb/config.py): HierarchyConfig + resolve_hierarchy() for the
`hierarchy:` block (target/min/max fanout, max_depth, summary token caps,
sideways settings) with validation + back-compat.
LLM callables (openkb/topic_tree_llm.py): make_distill_cluster (AGENTS.md-guided,
sized category naming), make_distill_summarize, make_relate.
Integration: `openkb distill` CLI command; AGENTS.md `## Hierarchy` guidance
section injected into distill prompts; query tree-descent prompt now follows
`related` sideways links; lint registers topic-dir names so pathway/sideways
wikilinks resolve.
Tests (+31): config parsing, distill invariants/edges/sideways/data-loss,
fake-LLM CLI integration + idempotent re-distill, and tier-4 regression pins
(VectifyAI#4 sideways links resolve — red without the lint change; VectifyAI#5 single-root/
min-2-layer edge sizes). Full suite: 937 passed, 10 llm deselected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
KylinMountain added a commit that referenced this pull request Jul 21, 2026
…es (#198)
* feat(api): delete a knowledge base (POST /api/v1/kb/delete + openkb delete-kb)
Physical, irreversible KB deletion with a type-the-name confirmation.
- config.delete_kb: rmtree the KB directory + unregister it from the global
registry (known_kbs / kb_aliases / default_kb, via the new unregister_kb).
Guards against deleting a non-KB path; tolerates a ghost registry entry
(directory already gone) by just unregistering.
- POST /api/v1/kb/delete: confirm_name must equal kb, re-checked server-side.
Lives in a new api_kbs_router (sibling of the config/graph/output routers) so
api.py stays under the 800-line gate; delete_kb self-locks (config lock), no
create_app closure needed.
- openkb delete-kb NAME: type-the-name prompt (or --yes) then delete.
Tests: endpoint (success + unregister, confirm mismatch, non-KB target) in
test_api.py; config (physical delete + unregister, refuse non-KB, ghost
tolerance) in test_config.py.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): delete a wiki page (POST /api/v1/page/delete) with backlink impact
Delete a concept/entity page and degrade references cleanly instead of leaving
them dangling:
- page_ops.delete_wiki_page: dry_run reports the backlink pages (whose inbound
[[wikilinks]] would be demoted) without touching anything; execute removes the
page under the KB ingest lock, strips its index.md entry outright
(compiler.remove_doc_from_index — the entry is a [[link]] line), and demotes
the now-dangling inbound [[links]] to plain text in ONLY those backlink pages
(lint.fix_broken_links restrict_to). The page ref is "section/stem", validated
to concepts/ or entities/ only (path-traversal-safe).
- New api_pages_router hosts POST /api/v1/page (read — relocated here from
api.py to keep it under the 800-line gate) + POST /api/v1/page/delete.
Tests: dry-run impact + execute (page gone, inbound link demoted to text, index
entry removed, sibling kept), 404 on missing page, and rejection of unsafe /
non-editable refs.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): edit a wiki page (PUT /api/v1/page) + link context (POST /api/v1/page/links)
- page_ops.edit_wiki_page: replace a concept/entity page's BODY while preserving
its code-managed OKF frontmatter (type/description/sources) verbatim; any
frontmatter in the submitted content is dropped. Dead [[wikilinks]] the user
typed are demoted to plain text on save (OpenKB keeps the wiki broken-link-free)
and returned as ghosts_stripped so the UI can warn. Atomic write under the KB
ingest lock.
- page_ops.page_link_context (POST /api/v1/page/links): outbound + inbound links
for the edit-impact panel. Editing the body does not break either (links are
path-based), so this is context, not a blocker — the honest impact model.
- PUT /api/v1/page added to api_pages_router.
Tests: edit preserves frontmatter + keeps a resolvable link + demotes a dead one
(reports ghosts_stripped) + replaces the body; 404 on a missing page; links
reports out/backlinks.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): delete a knowledge base from the settings sheet (type-name confirm)
Adds a Danger Zone section to KbSettingsSheet: a type-the-name confirmation
gates deleteKb() (POST /api/v1/kb/delete); on success KbDetail navigates back
to the KB list. New deleteKb client + kbSettings danger* keys + common cancel
(zh + en, identical key sets).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): in-reader page edit + delete with impact preview (F2/F3)
For an open concept/entity page, the reader header now shows Edit and Delete:
- Delete: a dry-run first (deletePage dryRun) surfaces the backlink pages whose
[[links]] will demote to plain text; a red confirm card lists them, then the
real delete closes the page and refreshes the inventory.
- Edit: a body textarea (frontmatter is preserved server-side), Save via
PUT /api/v1/page, a links panel (getPageLinks: out/backlinks), a "recompile
may overwrite" note, and a toast listing any dead links demoted to text.
Adds the deletePage/editPage/getPageLinks wiki clients and kb:pageOps.* locale
keys (zh + en, identical key sets). Build green (i18n guard + tsc + vite).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(workbench): address code-review findings (content-mgmt correctness + concurrency)
Backend (page_ops / kb_admin / lint):
- delete/edit now do their read-modify-write INSIDE the KB ingest lock and
re-check the page exists under it: no stale backlink snapshot, no resurrecting
a concurrently-deleted page, no 500 on a racing unlink (missing_ok=True). [#4,#5,#6]
- delete_kb serializes rmtree under the ingest lock (was fully unlocked). Moved
delete_kb/unregister_kb into a new openkb/kb_admin.py so config.py drops back
under the per-file line gate (751 lines). [#1,#15]
- page deletion no longer scans or rewrites lint's _EXCLUDED_FILES
(AGENTS.md/SCHEMA.md/log.md); fix_broken_links(restrict_to) also refuses them,
which protects `openkb remove` too. [#2]
- edit_wiki_page treats `content` as the body verbatim — no naive frontmatter
split that silently dropped a body opening with a '---' block. [#3]
- delete uses the real on-disk stem for the exact-match index-entry removal
(case-insensitive FS), and adds index.md to the demotion set so a [[target]]
embedded in another entry's brief no longer dangles. [#7,#9]
API / CLI:
- delete-KB endpoint & CLI can now clean a ghost registry entry (registered name
whose directory was removed by hand), catch delete_kb's ValueError -> 400, and
the endpoint is typed `-> KbDeleteResponse`. [#8,#13,E5]
Frontend (KbDetail):
- Deleting a KB dispatches `openkb:reload-kbs` so the sidebar refreshes
immediately (was stale until a manual reload) — live-test fix.
- Removed the dead `pageReloadSeq` path (editPage always returns content). [#14]
Tests: excluded-doc-untouched, '---'-body preservation, ghost-KB unregister;
non-KB-target test updated to the new resolution model. Backend 1234 passed,
mypy/ruff clean, frontend build green.
Not changed (intentional): pages_linking_to is NOT folded into build_graph (that
excludes explorations/ backlinks) [#10]; the preview+confirm delete inherently
scans per request [#11]; EDITABLE_SECTIONS stays duplicated in the frontend,
which can't import the Python constant [#12].
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(workbench): allow editing summary pages (edit ⊇ delete section sets)
Summaries are compiled markdown like concepts/entities (a recompile regenerates
them), so they are now editable via PUT /api/v1/page. They remain NOT
independently deletable — a summary is removed by deleting its source document.
Split page_ops into EDITABLE_SECTIONS (concepts/entities/summaries) vs
DELETABLE_SECTIONS (concepts/entities); validate_page_ref takes the allowlist.
Frontend ReaderBody gates Edit on canEdit (incl. summaries), Delete on canDelete.
Test: summary editable (frontmatter preserved) + summary delete rejected (400).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(api): cross-platform delete_kb + endpoint OSError handling (xhigh review #1,#2)
- delete_kb no longer holds the ingest lock DURING rmtree (the lock file lives
inside the KB dir; Windows cannot delete an open file — the prior review-fix
regressed this). It now takes the lock as a BARRIER (drain + wait out any
in-flight mutation), releases it, re-checks existence, then rmtrees. [#1]
- delete-KB endpoint maps FileNotFoundError to an idempotent success (a
concurrent delete already removed the tree) and other OSError to a clean 500
with a message, instead of an uncaught 500 stack trace. [#2]
Tests: endpoint OSError -> clean 500, FileNotFoundError -> 200 deleted.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…y-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…ane (#197) (#199)
* feat(web): read a document's converted source text in the Documents pane (#197)
The Documents pane listed each ingested doc (name/type/hash) with no way to
read the converted full text ingestion produced under wiki/sources/.
Backend: new POST /api/v1/document/source resolves a doc hash to its source
text — short docs read <doc_name>.md; long docs concatenate the per-page
<doc_name>.json (page text joined by a thematic break). Hash is the identifier
(unique, avoids doc_name/stem collisions); resolution prefers the registry's
stored source_path then falls back to the wiki/sources/<doc_name>.{md,json}
convention, with a path-traversal guard. Read-only (sources are do-not-edit).
Frontend: document rows are now clickable and open a wide read-only slide-out
reader (MarkdownView) — ESC/overlay/close to dismiss, independent scroll,
content cached + memoized per hash, native find-in-page preserved (no
virtualization). Delete stays inline (stopPropagation). Closed drawer is inert.
Known limitation: images embedded in long-doc pages are not rendered inline yet.
* fix(web): address xhigh code-review findings for the document reader (#197)
Correctness / a11y:
- Rebuild the reader drawer on Radix Dialog (like KbSettingsSheet) instead of
a hand-rolled overlay: proper modal focus trap, initial + return focus,
Escape, and background inert (was: aria-modal with none of it) [#4]. This
also removes the hand-rolled window keydown listener that re-subscribed every
render [#7].
- Restructure each document row so the open-reader target is a real <button>
and the delete control is a SIBLING, not nested. Keyboard-activating delete
no longer bubbles into opening the reader, and the invalid nested-interactive
markup is gone [#1, #5].
- Resolve a source file by the doc's own type (long → .json first, else .md),
so two docs sharing a doc_name each resolve to their own file rather than
whichever extension is tried first [#2].
- Guard source reads: skip non-dict page entries, reject non-list JSON, and
return a controlled 500 on corrupt/unreadable sources instead of an
unhandled exception [#3].
- Invalidate the per-hash content cache when the inventory changes, so a
reopen after recompile refetches instead of serving stale text [#6].
Fetch/cache/memoized body moved to DocumentsPane so they survive the drawer's
unmount-on-close. #8 (frontmatter stripping) intentionally not applied: source
docs render verbatim (a user's own frontmatter is content, unlike wiki-page OKF
metadata). Adds tests for the collision and malformed-JSON paths.
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…verride) (#200)
* feat(api): configurable entity_types (global + per-KB) via the config API
entity_types (the entity-extraction vocabulary) is now surfaced through the
config read/write API, layering global.yaml -> KB config.yaml like the other
scalars: a KB list overrides the global list wholesale, an explicit null
inherits, and unset falls back to DEFAULT_ENTITY_TYPES. The compiler already
consumed config["entity_types"] via resolve_entity_types; this just exposes it.
- GLOBAL_SCALAR_KEYS gains "entity_types" (layering + per-key `sources` tracking;
the value-not-None-wins rule is type-agnostic, so it works for a list).
- _KbConfigWritable / GlobalConfigValues / KbConfigResponse / GlobalConfigResponse
carry entity_types; read_kb_config/read_global_config report the cleaned
EFFECTIVE list (resolve_entity_types) plus the raw global value for the badge.
- PATCH /api/v1/kb/config and PATCH /api/v1/config accept entity_types.
Tests: KB override (cleaned + source 'kb') + null revert, global patch, global
inheritance; updated the global-defaults shape assertion. Frontend UI follows.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): entity-types config UI — chips editor (global default + per-KB override)
- EntityTypesEditor: shared controlled chips editor (Enter/comma to add, x to
remove; "other" is a fixed always-included chip; IME-safe composition).
- KbSettingsSheet: an EntityTypesRow with the same inherit/override Switch as the
scalar rows — turning override on seeds+persists the KB's own list, off reverts
via null; inherited state shows the global/default list as a badge. Each chip
change persists and adopts the server-cleaned response.
- Settings (general tab): a global entity-types chips editor, order-sensitive
diff into the save patch (joins the existing dirty/SaveBar flow).
- "changes affect future recompiles only" note on both surfaces.
New keys in common/kbSettings/settings (zh + en, identical sets). Build green
(i18n guard OK). Backend was committed in 92f8f41.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(entity-types): address xhigh review — DEFAULT_CONFIG parity, empty-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@KylinMountain@rejojer@zmtomorrow
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); })(); feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex by KylinMountain · Pull Request #4 · VectifyAI/OpenKB · GitHub
Skip to content

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex - #4

Merged
rejojer merged 102 commits into
mainfrom
dev
Apr 8, 2026
Merged

feat: OpenKB MVP — Karpathy's LLM Knowledge Base, powered by PageIndex#4
rejojer merged 102 commits into
mainfrom
dev

Conversation

@KylinMountain

Copy link
Copy Markdown
Collaborator

Summary

OpenKB — Karpathy's LLM Knowledge Base workflow as a CLI, powered by PageIndex.

Drop documents in. Get an auto-maintained, cross-linked wiki out.

Features

  • okb init — Interactive setup
  • okb add — Short docs (pymupdf) + long PDFs (PageIndex local/cloud)
  • okb query — Streaming Q&A with PageIndex cloud streaming
  • okb watch — Auto-compile on file changes
  • okb lint — Structural + knowledge health checks
  • okb list / status — Knowledge base overview
  • Obsidian compatible wiki output

Tech Stack

PageIndex, markitdown, OpenAI Agents SDK, LiteLLM, Click, watchdog

KylinMountainand others added 30 commits April 6, 2026 23:13
Sets up pyproject.toml (hatchling, direct-refs allowed, Python >=3.11),
.gitignore, openkb/__init__.py, a Click CLI stub with all 7 commands
(init, add, query, watch, lint, list, status), and tests/conftest.py
with kb_dir and sample_tree fixtures. Package installs cleanly in a
Python 3.12 venv; okb --help shows all commands; pytest collects 0
tests without error.
Add openkb/config.py (DEFAULT_CONFIG, load_config, save_config),
openkb/state.py (HashRegistry with SHA-256 file hashing and JSON
persistence), and openkb/schema.py (SCHEMA_MD constant). All 17 tests
written first (red) then implemented (green).
Creates full KB directory structure (raw/, wiki/sources/images/,
wiki/summaries/, wiki/concepts/, wiki/reports/), writes SCHEMA.md,
index.md, config.yaml and hashes.json; guards against re-initialisation.
Three tests in tests/test_cli.py cover structure, schema content, and
the already-initialized guard, all via CliRunner.isolated_filesystem.
Implements extract_base64_images and copy_relative_images with full test
coverage for single/multiple images, invalid base64, missing files, and
URL filtering.
Implements ConvertResult dataclass, get_pdf_page_count, and
convert_document with hash-dedup, markdown passthrough, PDF long-doc
detection, MarkItDown conversion, and image extraction integration.
Implements render_source_md and render_summary_md with YAML frontmatter,
recursive heading hierarchy (h1–h6 capped), page ranges, and separate
text/summary views for source and summary wiki pages.
Implements IndexResult dataclass and index_long_document which creates
a LocalClient with full node text/summary/description flags, adds the
PDF via PageIndex, fetches structure, and writes source and summary
wiki pages via the tree renderer.
Implements list_wiki_files, read_wiki_file, and write_wiki_file as plain
functions in openkb/agent/tools.py without @function_tool decoration,
ready to be wrapped when building the agent. Full test coverage including
edge cases for missing files/dirs, filtering to .md only, and parent dir
creation.
Implements build_compiler_agent, compile_short_doc, compile_long_doc in
openkb/agent/compiler.py with function_tool-wrapped wiki tools and
SCHEMA_MD-enriched instructions. Long-doc variant includes get_page_content.
Tests mock Runner.run to avoid real LLM calls.
Replaces the add stub with full orchestration: convert_document,
index_long_document for long PDFs, and compiler agent calls.
Adds SUPPORTED_EXTENSIONS set, _find_kb_dir, _add_single_file helpers.
Adds python-dotenv dependency and load_dotenv() at startup.
Implements pageindex_retrieve (structure -> LLM relevance -> page fetch),
build_query_agent with list/read/retrieve tools, and run_query coroutine.
Wires up `okb query` in cli.py.
Implements DebouncedHandler (collects events, ignores dirs/dotfiles, resets
timer on burst) and watch_directory (Observer loop, Ctrl+C safe).
Wires up `okb watch` in cli.py.
Implements find_broken_links, find_orphans, find_missing_entries,
check_index_sync, and run_structural_lint with full Markdown report.
Covers wikilink resolution, orphan detection, raw/wiki entry matching,
and index.md sync checking.
Implements build_lint_agent with list/read tools and instructions for
semantic quality checks (contradictions, gaps, staleness, redundancy).
run_knowledge_lint runs the agent and returns the report string.
okb lint combines structural + knowledge lint and writes timestamped report.
Tests verify list shows documents table and concepts, status shows
per-directory file counts and total indexed. Both check missing-init guard.
Previously the converter registered the file hash immediately, so if
LLM compilation failed the file was marked as "done" and retries
would skip it. Now the hash is only registered by the CLI after
successful compilation.
Also: install markitdown[all] for PDF support, add python-dotenv.
…pport
- Switch from col._backend.get_document_structure() to col.get_document_structure()
- Add 3x retry for PageIndex indexing (stochastic TOC accuracy)
- Fix storage path to use .db extension
- Remove .doc from supported extensions (markitdown only supports .docx)
- Note: col.get_page_content() still missing from PageIndex public API,
using col._backend.get_page_content() as workaround
Replace col._backend.get_page_content(col._name, doc_id, spec) with
col.get_page_content(doc_id, spec). Now all PageIndex access uses
public API only.
rejojer added 15 commits April 8, 2026 05:01
Rename CLI command and state dir from okb to openkb
- Hardcode reading LLM_API_KEY env var instead of indirecting through config
- Remove llm_api_key_env from DEFAULT_CONFIG, okb init prompts, and config.yaml
- Provider-specific env vars (OPENAI_API_KEY, etc.) still work via LiteLLM auto-detection
- One less config field, one less okb init step
The OpenAI Agents SDK requires a litellm/ prefix to route non-OpenAI
models through LiteLLM. Without it, models like anthropic/claude-sonnet-4-6
fail with "Unknown prefix". This adds the prefix at all Agent() call sites
while keeping litellm.completion() calls unchanged.
Also updates README quick start comments and model format docs.
Fix: add litellm/ prefix for Agents SDK model routing
@KylinMountain

Copy link
Copy Markdown
CollaboratorAuthor

Code review

Found 1 issue:

  1. extract_pdf_images and convert_pdf_with_images in images.py open pymupdf documents with explicit .close() instead of context managers. If an exception is raised during page iteration (e.g. corrupt image block, pixmap allocation failure), the PDF file handle leaks. This is the same bug pattern that was already fixed in converter.py:get_pdf_page_count (commit c525455), but images.py was missed. Fix: replace doc = pymupdf.open(...) / doc.close() with with pymupdf.open(...) as doc:.

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] !=1: # not an image block
continue
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
save_path=images_dir/filename
pix.save(str(save_path))
pix=None
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
continue
rel_path=f"images/{doc_name}/{filename}"
page_images.setdefault(page_num, []).append(rel_path)
doc.close()
returnpage_images

OpenKB/openkb/images.py

Lines 89 to 125 in 1637697

doc=pymupdf.open(str(pdf_path))
forpage_idxinrange(len(doc)):
page=doc[page_idx]
page_num=page_idx+1
parts.append(f"\n\n<!-- Page {page_num} -->\n")
forblockinpage.get_text("dict")["blocks"]:
ifblock["type"] ==0: # text block
lines= []
forlineinblock["lines"]:
spans_text="".join(span["text"] forspaninline["spans"])
lines.append(spans_text)
parts.append("\n".join(lines))
elifblock["type"] ==1: # image block
width=block.get("width", 0)
height=block.get("height", 0)
ifwidth<_MIN_IMAGE_DIMorheight<_MIN_IMAGE_DIM:
continue
image_bytes=block.get("image")
ifnotimage_bytes:
continue
try:
pix=pymupdf.Pixmap(image_bytes)
ifpix.n>4:
pix=pymupdf.Pixmap(pymupdf.csRGB, pix)
img_counter+=1
filename=f"p{page_num}_img{img_counter}.png"
(images_dir/filename).write_bytes(pix.tobytes("png"))
pix=None
parts.append(f"\n![image](images/{doc_name}/{filename})\n")
exceptException:
logger.warning("Failed to save image block on page %d", page_num)
doc.close()
return"\n".join(parts)

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer

Copy link
Copy Markdown
Member

Code review

Found 2 issues:

  1. README claims LiteLLM is "pinned to a safe version" but pyproject.toml has no version pin. Line 67 of README.md states LiteLLM is (pinned to a safe version), but pyproject.toml line 17 lists the dependency as bare "litellm" with no version constraint (==, >=, ~=, etc.). Any version -- including potentially insecure ones -- can be installed.

OpenKB/README.md

Lines 66 to 68 in 854294c

OpenKB comes with [multi-LLM support](https://docs.litellm.ai/docs/providers) (e.g., OpenAI, Claude, Gemini) via [LiteLLM](https://github.com/BerriAI/litellm) (pinned to a [safe version](https://docs.litellm.ai/blog/security-update-march-2026)).

OpenKB/pyproject.toml

Lines 16 to 18 in 854294c

"watchdog>=3.0",
"litellm",
"openai-agents",

  1. test_short_pdf_converted_via_markitdown mocks the wrong code path. The test patches openkb.converter.MarkItDown and openkb.converter.pymupdf.open, but converter.py line 99-101 routes short PDFs through convert_pdf_with_images() (from openkb.images), not MarkItDown. The MarkItDown mock is never exercised, and convert_pdf_with_images is not mocked, so the test either fails at runtime or passes for the wrong reasons.

classTestConvertDocumentPdfShort:
deftest_short_pdf_converted_via_markitdown(self, kb_dir, tmp_path):
"""PDF under threshold is converted with markitdown."""
src=tmp_path/"short.pdf"
src.write_bytes(b"%PDF-1.4 fake content")
fake_result=MagicMock()
fake_result.text_content="# Short PDF\n\nConverted content."
with (
patch("openkb.converter.pymupdf.open") asmock_mu,
patch("openkb.converter.MarkItDown") asmock_mid_cls,
):
fake_doc=MagicMock()
fake_doc.page_count=5# below default threshold of 20
fake_doc.__enter__=MagicMock(return_value=fake_doc)
fake_doc.__exit__=MagicMock(return_value=False)
mock_mu.return_value=fake_doc
mock_mid_cls.return_value.convert.return_value=fake_result
result=convert_document(src, kb_dir)
assertresult.skippedisFalse
assertresult.is_long_docisFalse
assertresult.source_pathisnotNone
assertresult.source_path.exists()

markdown=copy_relative_images(markdown, src.parent, doc_name, images_dir)
elifsrc.suffix.lower() ==".pdf":
# Use pymupdf dict-mode for PDFs: text + images inline at correct positions
markdown=convert_pdf_with_images(src, doc_name, images_dir)
else:

🤖 Generated with Claude Code

- If this code review was useful, please react with 👍. Otherwise, react with 👎.

@rejojer
rejojer merged commit f0963f6 into mainApr 8, 2026
KylinMountain added a commit that referenced this pull request May 24, 2026
Architectural review (4 parallel Opus auditors) found that the skill_runner
core was already generic, but the deck SURFACE was still fused to
Editorial Monocle. Fixed:
* validator: now takes optional `grammar` param (DeckGrammar TypedDict);
skill-agnostic by default (only checks file present, parses, ≥5
slides, self-contained). Third-party deck skills (guizang, swiss)
now pass validation cleanly. Editorial-specific rules opt-in via
`EDITORIAL_MONOCLE_GRAMMAR`. (finding #2)
* skills/openkb-deck-editorial/SKILL.md: declares its grammar +
output_path_template under `od:` frontmatter — `run_skill` reads
these and applies them post-run.
* run_skill: now honors frontmatter `od.mode`, `od.output_path_template`,
`od.deck_grammar`. When mode=="deck" and template is set, the runner
injects the path into intent, verifies the file exists post-run, and
runs validate_deck with the skill's grammar. Validation result is
returned via new SkillRunResult dataclass. (findings #4, #5)
* `openkb deck new --skill <name>`: CLI flag accepts any installed deck
skill (default openkb-deck-editorial). guizang and swiss now usable
from the scripted CLI, not only freeform chat. (finding #1)
* `/deck new --skill <name>` chat slash: same flag, parsed positionally
alongside --critique. (finding #1)
* tests/test_read_kb_file.py: 13 new tests mirroring test_write_kb_file
for the read-side allow-list. Pins refusal of `.openkb/config.yaml`,
`.env`, `raw/`, `..` traversal, absolute paths. (finding #6)
* Generator deck branch: no longer calls validate_deck directly; just
propagates run_deck_create's SkillRunResult.validation up. Validation
is now a property of "this skill declared mode=deck", not of "this
CLI path was taken".
Existing tests updated:
* tests/test_deck_validator.py: explicit grammar arg on Editorial-
specific tests; added test_guizang_shape_passes_generic_mode +
test_missing_cover_ignored_in_generic_mode to pin both modes.
* tests/test_deck_creator.py: mocks return SkillRunResult; new
test_run_deck_create_honors_skill_name_override for --skill flag.
* tests/test_generator.py: deck dispatch test mocks SkillRunResult.
Below-threshold findings deferred:
* Generator if/else → registry (score 70) — works, just not extensible
via plugin; future.
* Iteration backup in chat freeform path (score 75) — needs write_kb_file
hook; separate change.
* run_skill / scan_local_skills / _handle_slash_critique direct tests
(scores 60-70) — covered indirectly by integration; can add later.
Regression: 538 tests pass (was 523 pre-fix; net +15 = 13 new
read_kb_file tests + 2 new validator-mode tests).
KylinMountain added a commit that referenced this pull request May 31, 2026
…lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
KylinMountain added a commit that referenced this pull request Jun 1, 2026
…mpile backfill (#78)
* feat(compiler): _read_entity_briefs for entity plan context
* test(compiler): parity tests for _read_entity_briefs
* feat(compiler): _write_entity with type/aliases frontmatter
* test(compiler): assert source ordering in _write_entity; count=1 in _set_fm_line
Add explicit ordering assertion in test_update_prepends_source_keeps_type
verifying the deterministic json.dumps form ("summaries/b.md", "summaries/a.md").
Pass count=1 to re.sub in _set_fm_line to make first-occurrence intent explicit.
* feat(lint): include entities/ in wikilink whitelist
* feat(compiler): summary<->entity backlinks
* test(compiler): restore assertion erroneously deleted in 3c8aa93
* feat(compiler): index.md Entities section
* feat(compiler): remove_doc_from_entity_pages + index cleanup
* feat(compiler): plan prompt + parser for entities group
Also wires the entity track into _compile_concepts (Tasks 7 + 8 combined,
since the {entity_briefs} placeholder and the _CONCEPTS_PLAN_USER.format call
are co-dependent — splitting would leave an intermediate red state).
- add _ENTITY_TYPES, _filter_entity_items, _parse_entities_plan
- rewrite _CONCEPTS_PLAN_USER to request nested concepts+entities groups
- add _ENTITY_PAGE_USER / _ENTITY_UPDATE_USER prompts
- read entity briefs and pass both briefs to the plan prompt
- parse nested 'concepts' group with legacy flat-list/flat-dict fallbacks
- generate entities in their own asyncio.gather (4-arity tuples)
- strip ghost links + _write_entity each; handle entity related cross-links
- backlink summary<->entities; pass entity_names/entity_meta to _update_index
* fix(compiler): related entities must not downgrade index labels
Mirror the concept track: collect related-entity slugs into a separate
local list used only for backlinks; pass only created/updated entity_names
(+entity_meta) to _update_index. Defense-in-depth in _update_index: only
_replace_section_entry when name is in entity_meta, otherwise only insert
if the link is absent, so a related-only entity can never clobber a
pre-existing correct (type + brief) index line with "(other)".
Adds regression test test_related_entity_does_not_downgrade_index_label.
* feat(schema): declare entities/ page type and taxonomy
* feat(query): point who/what questions at entities/
* docs(readme): document entities/ page type
* feat(cli): scaffold entities/ in init and count it in status
- `openkb init` now creates wiki/entities/ alongside wiki/concepts/
- init seed index.md gains ## Entities between ## Concepts and ## Explorations,
matching the _update_index template in compiler.py
- print_status subdirs list gains "entities" after "concepts"
- Tests updated: assert wiki/entities/ exists and index.md contains ## Entities;
status test asserts "entities" appears in output
* fix(compiler): resolve entity-page review findings (dangling links + dedup)
Addresses code-review findings on the entity-pages feature:
- Fix dangling wikilink after `openkb remove`: entity removal now strips
standalone `See also: [[summaries/{doc}]]` lines (the related-entity
backlink form), matching the concept path, and cli.py adds modified
entity pages to the lint sweep scope so surviving pages are cleaned.
- Unify the parallel concept/entity helpers into shared cores
(_backlink_summary_pages, _backlink_pages, _remove_doc_from_pages) with
thin per-type wrappers, so cleanup logic can no longer drift between the
two page types (this is what caused the dangling-link bug).
- Route related-entity cross-refs through _add_related_link (now page-type
aware) instead of an inline reimplementation — removes a duplicate file
read/write and keeps backlink creation symmetric with teardown.
- Centralize the entity-type enum: prompts derive their type list from a
single _ENTITY_TYPE_LIST source via import-time substitution.
- Count entity items in the "all dropped as malformed" plan warning.
- Drop the unreachable else branch in _update_index's entity loop.
- Add regression test for the See-also strip on a surviving entity page.
All 542 tests pass.
* fix(compiler): add [[entities/X]] whitelist rule + restore concept-topic guard
Remaining review findings after a7a06ed:
- _KNOWN_TARGETS_USER now states the [[entities/Z]] rule, so entity links
the LLM is told to write aren't silently stripped as ghosts.
- Restore the dropped 'Do NOT create concepts that are just the document
topic itself' plan rule to prevent redundant title-mirror concepts.
* feat(entities): shared page-dir constants + surface entities in list/lint/status/skill-gate/linter
Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single
source of truth; replace duplicated index-seed literals in cli init and
compiler._update_index with INDEX_SEED.
- openkb list / chat /list: add an Entities section (#2)
- lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages
missing from index.md are flagged (#4)
- skill-new gate: count entities/ as compiled content (#5)
- status last-compile: derive from summaries/concepts/entities mtimes (#12)
- semantic linter: read entities/, check contradictions/redundancy/
coverage/orphans (#3)
* feat(entities): remove preview lists entity-page actions (#1)
The dry-run/confirmation block now scans wiki/entities/ with the same
frontmatter sources: logic as concepts, emits DELETE/MODIFY action lines
per entity page, and prints an 'N entity(s) will be DELETED' summary.
Execution path (remove_doc_from_entity_pages) unchanged.
* docs(entities): document entity pages in shipped openkb skill (#8)
Note wiki/entities/ holds named-thing pages (people/orgs/places/
products/works/events) with a type: frontmatter field, that index.md
has a ## Entities section, and that 'who/what is X' questions should
read the matching entities/ page first.
* fix(compiler): don't write raw JSON body on empty LLM content
In the parse-succeeded branch of _gen_create/_gen_update/_gen_entity_create/
_gen_entity_update, fall back to "" instead of the raw JSON string when the
content field is empty/null. _require_nonempty_content then raises and the
page is dropped, rather than writing the JSON envelope as the markdown body.
The parse-FAILED (except) branch keeps content=raw as the legitimate
non-JSON fallback.
* fix(compiler): graceful scalar plan + rebuild malformed entity frontmatter
- _compile_concepts: guard a non-dict/non-list parsed plan (JSON scalar)
before calling .get(), taking the empty-plan path (write v1 summary if
applicable + update index + return) instead of risking AttributeError.
- _write_entity: when an existing page has an opening --- but no closing
delimiter (or no frontmatter), rebuild valid sources/type/brief frontmatter
rather than writing a body-only page that drops the metadata.
* fix(compiler): keep ## Entities before ## Explorations; drop dead param + overlap gathers
- _update_index: insert ## Entities before ## Explorations on older index.md
files that predate the section (new _ensure_h2_section_before helper),
preserving canonical order instead of appending at EOF.
- _filter_entity_items: drop the unused 'label' parameter and update call
sites in _parse_entities_plan.
- _compile_concepts: overlap concept and entity generation in one outer
asyncio.gather (they share cached context and the same concurrency
semaphore); result/error handling per list is unchanged.
* test(compiler): cover empty-content skip, scalar plan, malformed entity FM, Entities order
Add regression tests for the four compiler fixes:
- empty {"content":""} response skips the page (no raw JSON body)
- JSON scalar plan handled gracefully (no AttributeError)
- _write_entity rebuilds frontmatter when closing --- is missing
- _update_index inserts ## Entities before ## Explorations
* fix(compiler): silence spurious 'hand-edited' warning on backlink section creation
_backlink_summary_pages / _backlink_pages create ## Entities / ## Related
Documents sections as a normal first-time operation; pass quiet=True so
_ensure_h2_section no longer logs the index-drift warning in that case.
Index-repair callers keep the warning.
* feat(cli): add `recompile` command to re-run compile on indexed docs
Re-runs the current compile_short_doc/compile_long_doc pipeline on
already-indexed docs so pre-feature KBs gain the entities/ layer and
refresh to the current format. Reuses on-disk sources/summaries and the
registry's PageIndex doc_id — does not re-index or re-convert.
Supports a positional <doc_name> (resolved via _resolve_doc_identifier)
or --all (with a regeneration-warning confirmation, bypassed by --yes),
--dry-run (enumerate only, no LLM calls/writes), and --refresh-schema
(back up + overwrite wiki/AGENTS.md when it differs from AGENTS_MD).
Processes docs sequentially with per-doc progress, skips+warns on
missing sources / summaries / doc_id, prints a recompiled/skipped
summary, and appends a recompile entry to log.md.
* test(cli): recompile dispatch/dry-run/skip/refresh-schema
* docs(readme): document openkb recompile
* fix(cli): recompile --refresh-schema no-ops when AGENTS.md absent; tighten guard tests
Match the spec (and the helper's own docstring): _refresh_schema returns
early when wiki/AGENTS.md is missing rather than materializing the default
(get_agents_md already falls back to it at runtime). Tighten the doc/--all
guard tests to assert the exact message + that no compile runs, and add the
missing-AGENTS.md no-op test.
* fix(compiler): drop non-existent 'related' slugs so they don't create dangling links
The plan's 'related' list is meant to reference existing pages, but the LLM
sometimes lists slugs for pages that don't exist. Those were added to the
wikilink whitelist (so body references survived ghost-stripping) and
back-linked into the summary's Related section, yet no page was ever created
(related items are linked, never generated) — producing a flood of broken
[[concepts/...]] / [[entities/...]] links (esp. on feature-dense docs).
Filter related_items / entity_related to slugs that exist on disk.
* fix: remove-preview detects JSON-quoted sources; _write_entity preserves sources on malformed FM
- remove --dry-run preview parsed the sources list with a hand-rolled comma
split that kept JSON quotes (["summaries/x.md"]), so the marker never
matched and the preview always reported 0 affected concept/entity pages
(executor was correct). Extract _scan_affected_pages using the real
_parse_yaml_list_value; dedups the two copied scan loops too.
- _write_entity's malformed-frontmatter rebuild seeded sources with only the
new doc, dropping prior sources for multi-source entities. Recover existing
sources from the broken block and merge.
Both bugs were masked by tests using unquoted / single-source fixtures.
* feat(cli): rename remove --keep-empty-concepts → --keep-empty (covers entities too)
This PR wired entity pages into 'openkb remove', so the flag now governs
concept AND entity retention — but the name still said 'concepts'. Make
--keep-empty the canonical name (clear that it covers both), keep
--keep-empty-concepts as a backward-compatible alias, and update the
preview/summary messages, docstring, and README accordingly.
* feat(compiler): config-driven entity types (entity_types overrides the default enum)
Add an optional 'entity_types:' key in .openkb/config.yaml. When present it
overrides the default person/organization/place/product/work/event/other
vocabulary everywhere — the plan prompt, the entity-page prompts, and
create/update validation/coercion; when absent, behavior is byte-identical.
Prompt templates keep an __ENTITY_TYPES__ token now substituted at call time
(per-KB) inside _compile_concepts, and the resolved valid-type set is threaded
into _parse_entities_plan / _filter_entity_items and the _gen_entity_* coercion.
'other' is always ensured as the coercion fallback; malformed config falls back
to the default with a warning. Documented in config.yaml.example + README.
* fix(compiler): harden config-driven entity types (crash-proof + complete the override)
Review of the config-entity-types feature surfaced two real issues:
- A config 'entity_types' value containing '{' or '}' was substituted into the
prompt template BEFORE .format() ran → KeyError/ValueError crashing every
compile. Swap to format-then-replace at all 3 call sites (types_str is now an
inert literal), and sanitize resolved types to a safe label charset (also
skips YAML nulls/ints so str(None) can't become the type 'none').
- The AGENTS_MD system schema hardcoded 'type: is one of: <7 defaults>',
contradicting a custom entity_types in the higher-weight system message.
Reword it to frame those as the configurable default and defer the
authoritative set to the compilation prompt (which is config-driven).
Also drop the now-dead _ENTITY_TYPES_STR + its stale import-time-substitution
comment. +2 regression tests (sanitization; brace-in-type doesn't crash).
* refactor: move entity-type resolution to config layer + co-locate remove-preview scan
Altitude cleanups from the review:
- Move resolve_entity_types + DEFAULT_ENTITY_TYPES into openkb/config.py (the
config layer owns config validation/normalization; any command can reuse it
without importing the heavy compiler module). compiler.py imports them;
_ENTITY_TYPE_LIST/_ENTITY_TYPES remain as the default alias/validation set.
- Move the remove dry-run preview scan from cli.py into compiler.py as
scan_affected_pages, beside remove_doc_from_*_pages and sharing
_parse_yaml_list_value — so preview and executor can't drift on how the
sources list is parsed (root cause of the earlier JSON-quote preview bug).
---------
Co-authored-by: Claude <noreply@anthropic.com>
calebfavor added a commit to railroadmedia/MusoraOpenKB that referenced this pull request Jul 2, 2026
Implements docs/smart-hierarchy-distillation-plan.md — a RAPTOR-style bottom-up
distillation that builds a multi-layer, LLM-navigable pathway hierarchy over the
flat concept leaves, the intended pivot from the top-down bootstrap() cold-start.
Engine (openkb/topic_tree.py):
- distill(): reads leaves recursively, clusters each layer into LLM-named sized
categories, summarizes each into a parent pathway node, links same-layer peers
sideways, repeats to a single root. Invariants enforced: exactly one root,
always >= 2 layers, bounded depth, no concept loss. Builds into a staging dir
and atomically swaps in (mid-build LLM failure never loses concepts).
- write_pathway_md(): pathway node format — layer/children/related frontmatter +
distilled summary + linked child index + Related pathways section.
- Sideways links are bidirectional, same-layer, top-K, no self-links.
Config (openkb/config.py): HierarchyConfig + resolve_hierarchy() for the
`hierarchy:` block (target/min/max fanout, max_depth, summary token caps,
sideways settings) with validation + back-compat.
LLM callables (openkb/topic_tree_llm.py): make_distill_cluster (AGENTS.md-guided,
sized category naming), make_distill_summarize, make_relate.
Integration: `openkb distill` CLI command; AGENTS.md `## Hierarchy` guidance
section injected into distill prompts; query tree-descent prompt now follows
`related` sideways links; lint registers topic-dir names so pathway/sideways
wikilinks resolve.
Tests (+31): config parsing, distill invariants/edges/sideways/data-loss,
fake-LLM CLI integration + idempotent re-distill, and tier-4 regression pins
(VectifyAI#4 sideways links resolve — red without the lint change; VectifyAI#5 single-root/
min-2-layer edge sizes). Full suite: 937 passed, 10 llm deselected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
KylinMountain added a commit that referenced this pull request Jul 21, 2026
…es (#198)
* feat(api): delete a knowledge base (POST /api/v1/kb/delete + openkb delete-kb)
Physical, irreversible KB deletion with a type-the-name confirmation.
- config.delete_kb: rmtree the KB directory + unregister it from the global
registry (known_kbs / kb_aliases / default_kb, via the new unregister_kb).
Guards against deleting a non-KB path; tolerates a ghost registry entry
(directory already gone) by just unregistering.
- POST /api/v1/kb/delete: confirm_name must equal kb, re-checked server-side.
Lives in a new api_kbs_router (sibling of the config/graph/output routers) so
api.py stays under the 800-line gate; delete_kb self-locks (config lock), no
create_app closure needed.
- openkb delete-kb NAME: type-the-name prompt (or --yes) then delete.
Tests: endpoint (success + unregister, confirm mismatch, non-KB target) in
test_api.py; config (physical delete + unregister, refuse non-KB, ghost
tolerance) in test_config.py.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): delete a wiki page (POST /api/v1/page/delete) with backlink impact
Delete a concept/entity page and degrade references cleanly instead of leaving
them dangling:
- page_ops.delete_wiki_page: dry_run reports the backlink pages (whose inbound
[[wikilinks]] would be demoted) without touching anything; execute removes the
page under the KB ingest lock, strips its index.md entry outright
(compiler.remove_doc_from_index — the entry is a [[link]] line), and demotes
the now-dangling inbound [[links]] to plain text in ONLY those backlink pages
(lint.fix_broken_links restrict_to). The page ref is "section/stem", validated
to concepts/ or entities/ only (path-traversal-safe).
- New api_pages_router hosts POST /api/v1/page (read — relocated here from
api.py to keep it under the 800-line gate) + POST /api/v1/page/delete.
Tests: dry-run impact + execute (page gone, inbound link demoted to text, index
entry removed, sibling kept), 404 on missing page, and rejection of unsafe /
non-editable refs.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(api): edit a wiki page (PUT /api/v1/page) + link context (POST /api/v1/page/links)
- page_ops.edit_wiki_page: replace a concept/entity page's BODY while preserving
its code-managed OKF frontmatter (type/description/sources) verbatim; any
frontmatter in the submitted content is dropped. Dead [[wikilinks]] the user
typed are demoted to plain text on save (OpenKB keeps the wiki broken-link-free)
and returned as ghosts_stripped so the UI can warn. Atomic write under the KB
ingest lock.
- page_ops.page_link_context (POST /api/v1/page/links): outbound + inbound links
for the edit-impact panel. Editing the body does not break either (links are
path-based), so this is context, not a blocker — the honest impact model.
- PUT /api/v1/page added to api_pages_router.
Tests: edit preserves frontmatter + keeps a resolvable link + demotes a dead one
(reports ghosts_stripped) + replaces the body; 404 on a missing page; links
reports out/backlinks.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): delete a knowledge base from the settings sheet (type-name confirm)
Adds a Danger Zone section to KbSettingsSheet: a type-the-name confirmation
gates deleteKb() (POST /api/v1/kb/delete); on success KbDetail navigates back
to the KB list. New deleteKb client + kbSettings danger* keys + common cancel
(zh + en, identical key sets).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): in-reader page edit + delete with impact preview (F2/F3)
For an open concept/entity page, the reader header now shows Edit and Delete:
- Delete: a dry-run first (deletePage dryRun) surfaces the backlink pages whose
[[links]] will demote to plain text; a red confirm card lists them, then the
real delete closes the page and refreshes the inventory.
- Edit: a body textarea (frontmatter is preserved server-side), Save via
PUT /api/v1/page, a links panel (getPageLinks: out/backlinks), a "recompile
may overwrite" note, and a toast listing any dead links demoted to text.
Adds the deletePage/editPage/getPageLinks wiki clients and kb:pageOps.* locale
keys (zh + en, identical key sets). Build green (i18n guard + tsc + vite).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(workbench): address code-review findings (content-mgmt correctness + concurrency)
Backend (page_ops / kb_admin / lint):
- delete/edit now do their read-modify-write INSIDE the KB ingest lock and
re-check the page exists under it: no stale backlink snapshot, no resurrecting
a concurrently-deleted page, no 500 on a racing unlink (missing_ok=True). [#4,#5,#6]
- delete_kb serializes rmtree under the ingest lock (was fully unlocked). Moved
delete_kb/unregister_kb into a new openkb/kb_admin.py so config.py drops back
under the per-file line gate (751 lines). [#1,#15]
- page deletion no longer scans or rewrites lint's _EXCLUDED_FILES
(AGENTS.md/SCHEMA.md/log.md); fix_broken_links(restrict_to) also refuses them,
which protects `openkb remove` too. [#2]
- edit_wiki_page treats `content` as the body verbatim — no naive frontmatter
split that silently dropped a body opening with a '---' block. [#3]
- delete uses the real on-disk stem for the exact-match index-entry removal
(case-insensitive FS), and adds index.md to the demotion set so a [[target]]
embedded in another entry's brief no longer dangles. [#7,#9]
API / CLI:
- delete-KB endpoint & CLI can now clean a ghost registry entry (registered name
whose directory was removed by hand), catch delete_kb's ValueError -> 400, and
the endpoint is typed `-> KbDeleteResponse`. [#8,#13,E5]
Frontend (KbDetail):
- Deleting a KB dispatches `openkb:reload-kbs` so the sidebar refreshes
immediately (was stale until a manual reload) — live-test fix.
- Removed the dead `pageReloadSeq` path (editPage always returns content). [#14]
Tests: excluded-doc-untouched, '---'-body preservation, ghost-KB unregister;
non-KB-target test updated to the new resolution model. Backend 1234 passed,
mypy/ruff clean, frontend build green.
Not changed (intentional): pages_linking_to is NOT folded into build_graph (that
excludes explorations/ backlinks) [#10]; the preview+confirm delete inherently
scans per request [#11]; EDITABLE_SECTIONS stays duplicated in the frontend,
which can't import the Python constant [#12].
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(workbench): allow editing summary pages (edit ⊇ delete section sets)
Summaries are compiled markdown like concepts/entities (a recompile regenerates
them), so they are now editable via PUT /api/v1/page. They remain NOT
independently deletable — a summary is removed by deleting its source document.
Split page_ops into EDITABLE_SECTIONS (concepts/entities/summaries) vs
DELETABLE_SECTIONS (concepts/entities); validate_page_ref takes the allowlist.
Frontend ReaderBody gates Edit on canEdit (incl. summaries), Delete on canDelete.
Test: summary editable (frontmatter preserved) + summary delete rejected (400).
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(api): cross-platform delete_kb + endpoint OSError handling (xhigh review #1,#2)
- delete_kb no longer holds the ingest lock DURING rmtree (the lock file lives
inside the KB dir; Windows cannot delete an open file — the prior review-fix
regressed this). It now takes the lock as a BARRIER (drain + wait out any
in-flight mutation), releases it, re-checks existence, then rmtrees. [#1]
- delete-KB endpoint maps FileNotFoundError to an idempotent success (a
concurrent delete already removed the tree) and other OSError to a clean 500
with a message, instead of an uncaught 500 stack trace. [#2]
Tests: endpoint OSError -> clean 500, FileNotFoundError -> 200 deleted.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…y-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…ane (#197) (#199)
* feat(web): read a document's converted source text in the Documents pane (#197)
The Documents pane listed each ingested doc (name/type/hash) with no way to
read the converted full text ingestion produced under wiki/sources/.
Backend: new POST /api/v1/document/source resolves a doc hash to its source
text — short docs read <doc_name>.md; long docs concatenate the per-page
<doc_name>.json (page text joined by a thematic break). Hash is the identifier
(unique, avoids doc_name/stem collisions); resolution prefers the registry's
stored source_path then falls back to the wiki/sources/<doc_name>.{md,json}
convention, with a path-traversal guard. Read-only (sources are do-not-edit).
Frontend: document rows are now clickable and open a wide read-only slide-out
reader (MarkdownView) — ESC/overlay/close to dismiss, independent scroll,
content cached + memoized per hash, native find-in-page preserved (no
virtualization). Delete stays inline (stopPropagation). Closed drawer is inert.
Known limitation: images embedded in long-doc pages are not rendered inline yet.
* fix(web): address xhigh code-review findings for the document reader (#197)
Correctness / a11y:
- Rebuild the reader drawer on Radix Dialog (like KbSettingsSheet) instead of
a hand-rolled overlay: proper modal focus trap, initial + return focus,
Escape, and background inert (was: aria-modal with none of it) [#4]. This
also removes the hand-rolled window keydown listener that re-subscribed every
render [#7].
- Restructure each document row so the open-reader target is a real <button>
and the delete control is a SIBLING, not nested. Keyboard-activating delete
no longer bubbles into opening the reader, and the invalid nested-interactive
markup is gone [#1, #5].
- Resolve a source file by the doc's own type (long → .json first, else .md),
so two docs sharing a doc_name each resolve to their own file rather than
whichever extension is tried first [#2].
- Guard source reads: skip non-dict page entries, reject non-list JSON, and
return a controlled 500 on corrupt/unreadable sources instead of an
unhandled exception [#3].
- Invalidate the per-hash content cache when the inventory changes, so a
reopen after recompile refetches instead of serving stale text [#6].
Fetch/cache/memoized body moved to DocumentsPane so they survive the drawer's
unmount-on-close. #8 (frontmatter stripping) intentionally not applied: source
docs render verbatim (a user's own frontmatter is content, unlike wiki-page OKF
metadata). Adds tests for the collision and malformed-JSON paths.
KylinMountain added a commit that referenced this pull request Jul 22, 2026
…verride) (#200)
* feat(api): configurable entity_types (global + per-KB) via the config API
entity_types (the entity-extraction vocabulary) is now surfaced through the
config read/write API, layering global.yaml -> KB config.yaml like the other
scalars: a KB list overrides the global list wholesale, an explicit null
inherits, and unset falls back to DEFAULT_ENTITY_TYPES. The compiler already
consumed config["entity_types"] via resolve_entity_types; this just exposes it.
- GLOBAL_SCALAR_KEYS gains "entity_types" (layering + per-key `sources` tracking;
the value-not-None-wins rule is type-agnostic, so it works for a list).
- _KbConfigWritable / GlobalConfigValues / KbConfigResponse / GlobalConfigResponse
carry entity_types; read_kb_config/read_global_config report the cleaned
EFFECTIVE list (resolve_entity_types) plus the raw global value for the badge.
- PATCH /api/v1/kb/config and PATCH /api/v1/config accept entity_types.
Tests: KB override (cleaned + source 'kb') + null revert, global patch, global
inheritance; updated the global-defaults shape assertion. Frontend UI follows.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* feat(web): entity-types config UI — chips editor (global default + per-KB override)
- EntityTypesEditor: shared controlled chips editor (Enter/comma to add, x to
remove; "other" is a fixed always-included chip; IME-safe composition).
- KbSettingsSheet: an EntityTypesRow with the same inherit/override Switch as the
scalar rows — turning override on seeds+persists the KB's own list, off reverts
via null; inherited state shows the global/default list as a badge. Each chip
change persists and adopts the server-cleaned response.
- Settings (general tab): a global entity-types chips editor, order-sensitive
diff into the save patch (joins the existing dirty/SaveBar flow).
- "changes affect future recompiles only" note on both surfaces.
New keys in common/kbSettings/settings (zh + en, identical sets). Build green
(i18n guard OK). Backend was committed in 92f8f41.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
* fix(entity-types): address xhigh review — DEFAULT_CONFIG parity, empty-list inherit, silent reads, set-diff
- config: define DEFAULT_ENTITY_TYPES before DEFAULT_CONFIG and add
`entity_types` to DEFAULT_CONFIG so the key layers like every other
GLOBAL_SCALAR_KEY and always appears in the effective config (#1).
- config: resolve_effective_config treats an empty entity_types list the
same as null → inherit, so a KB that cleared its override doesn't pin an
empty vocabulary (#2).
- config: resolve_entity_types(config, *, warn=True); config-read paths pass
warn=False so a plain GET doesn't spam coercion warnings (#4).
- api_config: both read paths call resolve_entity_types(..., warn=False).
- KbSettingsSheet: inherited badge shows the effective list, not
`globalValue ?? effective`; drop the now-unused globalValue prop (#3).
- Settings: global entity_types diff is order-insensitive — the vocabulary
is a set, so re-adding a removed type is not a change (#6).
Backend pytest 1240 passed; ruff/format/mypy clean; frontend build green.
Claude-Session: https://claude.ai/code/session_01XMxbhmAkxxVV8CFWCZDBaG
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@KylinMountain@rejojer@zmtomorrow