Uh oh!
There was an error while loading. Please reload this page.
feat: entity pages — first-class entities/ type (configurable) + recompile backfill - #78
Merged
Conversation
…set_fm_line
Add explicit ordering assertion in test_update_prepends_source_keeps_type
verifying the deterministic json.dumps form ("summaries/b.md", "summaries/a.md").
Pass count=1 to re.sub in _set_fm_line to make first-occurrence intent explicit.Also wires the entity track into _compile_concepts (Tasks 7 + 8 combined,
since the {entity_briefs} placeholder and the _CONCEPTS_PLAN_USER.format call
are co-dependent — splitting would leave an intermediate red state).
- add _ENTITY_TYPES, _filter_entity_items, _parse_entities_plan
- rewrite _CONCEPTS_PLAN_USER to request nested concepts+entities groups
- add _ENTITY_PAGE_USER / _ENTITY_UPDATE_USER prompts
- read entity briefs and pass both briefs to the plan prompt
- parse nested 'concepts' group with legacy flat-list/flat-dict fallbacks
- generate entities in their own asyncio.gather (4-arity tuples)
- strip ghost links + _write_entity each; handle entity related cross-links
- backlink summary<->entities; pass entity_names/entity_meta to _update_indexMirror the concept track: collect related-entity slugs into a separate local list used only for backlinks; pass only created/updated entity_names (+entity_meta) to _update_index. Defense-in-depth in _update_index: only _replace_section_entry when name is in entity_meta, otherwise only insert if the link is absent, so a related-only entity can never clobber a pre-existing correct (type + brief) index line with "(other)". Adds regression test test_related_entity_does_not_downgrade_index_label.
- `openkb init` now creates wiki/entities/ alongside wiki/concepts/ - init seed index.md gains ## Entities between ## Concepts and ## Explorations, matching the _update_index template in compiler.py - print_status subdirs list gains "entities" after "concepts" - Tests updated: assert wiki/entities/ exists and index.md contains ## Entities; status test asserts "entities" appears in output
…dedup)
Addresses code-review findings on the entity-pages feature:
- Fix dangling wikilink after `openkb remove`: entity removal now strips
standalone `See also: [[summaries/{doc}]]` lines (the related-entity
backlink form), matching the concept path, and cli.py adds modified
entity pages to the lint sweep scope so surviving pages are cleaned.
- Unify the parallel concept/entity helpers into shared cores
(_backlink_summary_pages, _backlink_pages, _remove_doc_from_pages) with
thin per-type wrappers, so cleanup logic can no longer drift between the
two page types (this is what caused the dangling-link bug).
- Route related-entity cross-refs through _add_related_link (now page-type
aware) instead of an inline reimplementation — removes a duplicate file
read/write and keeps backlink creation symmetric with teardown.
- Centralize the entity-type enum: prompts derive their type list from a
single _ENTITY_TYPE_LIST source via import-time substitution.
- Count entity items in the "all dropped as malformed" plan warning.
- Drop the unreachable else branch in _update_index's entity loop.
- Add regression test for the See-also strip on a surviving entity page.
All 542 tests pass.…pic guard Remaining review findings after a7a06ed: - _KNOWN_TARGETS_USER now states the [[entities/Z]] rule, so entity links the LLM is told to write aren't silently stripped as ghosts. - Restore the dropped 'Do NOT create concepts that are just the document topic itself' plan rule to prevent redundant title-mirror concepts.
…lint/status/skill-gate/linter Add PAGE_CONTENT_DIRS and INDEX_SEED to openkb/schema.py as the single source of truth; replace duplicated index-seed literals in cli init and compiler._update_index with INDEX_SEED. - openkb list / chat /list: add an Entities section (#2) - lint.check_index_sync: iterate PAGE_CONTENT_DIRS so entities/ pages missing from index.md are flagged (#4) - skill-new gate: count entities/ as compiled content (#5) - status last-compile: derive from summaries/concepts/entities mtimes (#12) - semantic linter: read entities/, check contradictions/redundancy/ coverage/orphans (#3)
The dry-run/confirmation block now scans wiki/entities/ with the same frontmatter sources: logic as concepts, emits DELETE/MODIFY action lines per entity page, and prints an 'N entity(s) will be DELETED' summary. Execution path (remove_doc_from_entity_pages) unchanged.
Note wiki/entities/ holds named-thing pages (people/orgs/places/ products/works/events) with a type: frontmatter field, that index.md has a ## Entities section, and that 'who/what is X' questions should read the matching entities/ page first.
In the parse-succeeded branch of _gen_create/_gen_update/_gen_entity_create/ _gen_entity_update, fall back to "" instead of the raw JSON string when the content field is empty/null. _require_nonempty_content then raises and the page is dropped, rather than writing the JSON envelope as the markdown body. The parse-FAILED (except) branch keeps content=raw as the legitimate non-JSON fallback.
…atter - _compile_concepts: guard a non-dict/non-list parsed plan (JSON scalar) before calling .get(), taking the empty-plan path (write v1 summary if applicable + update index + return) instead of risking AttributeError. - _write_entity: when an existing page has an opening --- but no closing delimiter (or no frontmatter), rebuild valid sources/type/brief frontmatter rather than writing a body-only page that drops the metadata.
…am + overlap gathers - _update_index: insert ## Entities before ## Explorations on older index.md files that predate the section (new _ensure_h2_section_before helper), preserving canonical order instead of appending at EOF. - _filter_entity_items: drop the unused 'label' parameter and update call sites in _parse_entities_plan. - _compile_concepts: overlap concept and entity generation in one outer asyncio.gather (they share cached context and the same concurrency semaphore); result/error handling per list is unchanged.
…ty FM, Entities order
Add regression tests for the four compiler fixes:
- empty {"content":""} response skips the page (no raw JSON body)
- JSON scalar plan handled gracefully (no AttributeError)
- _write_entity rebuilds frontmatter when closing --- is missing
- _update_index inserts ## Entities before ## Explorations…tion creation _backlink_summary_pages / _backlink_pages create ## Entities / ## Related Documents sections as a normal first-time operation; pass quiet=True so _ensure_h2_section no longer logs the index-drift warning in that case. Index-repair callers keep the warning.
Re-runs the current compile_short_doc/compile_long_doc pipeline on already-indexed docs so pre-feature KBs gain the entities/ layer and refresh to the current format. Reuses on-disk sources/summaries and the registry's PageIndex doc_id — does not re-index or re-convert. Supports a positional <doc_name> (resolved via _resolve_doc_identifier) or --all (with a regeneration-warning confirmation, bypassed by --yes), --dry-run (enumerate only, no LLM calls/writes), and --refresh-schema (back up + overwrite wiki/AGENTS.md when it differs from AGENTS_MD). Processes docs sequentially with per-doc progress, skips+warns on missing sources / summaries / doc_id, prints a recompiled/skipped summary, and appends a recompile entry to log.md.
…ghten guard tests Match the spec (and the helper's own docstring): _refresh_schema returns early when wiki/AGENTS.md is missing rather than materializing the default (get_agents_md already falls back to it at runtime). Tighten the doc/--all guard tests to assert the exact message + that no compile runs, and add the missing-AGENTS.md no-op test.
… dangling links The plan's 'related' list is meant to reference existing pages, but the LLM sometimes lists slugs for pages that don't exist. Those were added to the wikilink whitelist (so body references survived ghost-stripping) and back-linked into the summary's Related section, yet no page was ever created (related items are linked, never generated) — producing a flood of broken [[concepts/...]] / [[entities/...]] links (esp. on feature-dense docs). Filter related_items / entity_related to slugs that exist on disk.
…ves sources on malformed FM - remove --dry-run preview parsed the sources list with a hand-rolled comma split that kept JSON quotes (["summaries/x.md"]), so the marker never matched and the preview always reported 0 affected concept/entity pages (executor was correct). Extract _scan_affected_pages using the real _parse_yaml_list_value; dedups the two copied scan loops too. - _write_entity's malformed-frontmatter rebuild seeded sources with only the new doc, dropping prior sources for multi-source entities. Recover existing sources from the broken block and merge. Both bugs were masked by tests using unquoted / single-source fixtures.
… entities too) This PR wired entity pages into 'openkb remove', so the flag now governs concept AND entity retention — but the name still said 'concepts'. Make --keep-empty the canonical name (clear that it covers both), keep --keep-empty-concepts as a backward-compatible alias, and update the preview/summary messages, docstring, and README accordingly.
…e default enum) Add an optional 'entity_types:' key in .openkb/config.yaml. When present it overrides the default person/organization/place/product/work/event/other vocabulary everywhere — the plan prompt, the entity-page prompts, and create/update validation/coercion; when absent, behavior is byte-identical. Prompt templates keep an __ENTITY_TYPES__ token now substituted at call time (per-KB) inside _compile_concepts, and the resolved valid-type set is threaded into _parse_entities_plan / _filter_entity_items and the _gen_entity_* coercion. 'other' is always ensured as the coercion fallback; malformed config falls back to the default with a warning. Documented in config.yaml.example + README.
Uh oh!
There was an error while loading. Please reload this page.
…ete the override)
Review of the config-entity-types feature surfaced two real issues:
- A config 'entity_types' value containing '{' or '}' was substituted into the
prompt template BEFORE .format() ran → KeyError/ValueError crashing every
compile. Swap to format-then-replace at all 3 call sites (types_str is now an
inert literal), and sanitize resolved types to a safe label charset (also
skips YAML nulls/ints so str(None) can't become the type 'none').
- The AGENTS_MD system schema hardcoded 'type: is one of: <7 defaults>',
contradicting a custom entity_types in the higher-weight system message.
Reword it to frame those as the configurable default and defer the
authoritative set to the compilation prompt (which is config-driven).
Also drop the now-dead _ENTITY_TYPES_STR + its stale import-time-substitution
comment. +2 regression tests (sanitization; brace-in-type doesn't crash).…ove-preview scan Altitude cleanups from the review: - Move resolve_entity_types + DEFAULT_ENTITY_TYPES into openkb/config.py (the config layer owns config validation/normalization; any command can reuse it without importing the heavy compiler module). compiler.py imports them; _ENTITY_TYPE_LIST/_ENTITY_TYPES remain as the default alias/validation set. - Move the remove dry-run preview scan from cli.py into compiler.py as scan_affected_pages, beside remove_doc_from_*_pages and sharing _parse_yaml_list_value — so preview and executor can't drift on how the sources list is parsed (root cause of the earlier JSON-quote preview bug).
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a first-class entity page type (
wiki/entities/) — specific named things (people, organizations, places, products, named works, events) — maintained incrementally alongsideconcepts/, distinct from abstract concept pages. Closes the largest gap found in the Karpathy LLM Wiki alignment audit (entity pages were named as a first-class type but absent). The type vocabulary is configurable, andopenkb recompilebackfills entities onto KBs compiled before the feature.Entity identification is folded into the existing concept-plan LLM call (one nested
{concepts, entities}response) — no extra call, rides the prompt cache, salience-filtered. Frontmatter:type,sources,brief, optionalaliases.Entity pages
_read_entity_briefs,_write_entity,_parse_entities_plan, and an entity generation track inside the shared_compile_concepts(so short- and long-doc ingest both get entities).entities/registered in the wikilink whitelist; bidirectional summary↔entity backlinks;## Entitiessection inindex.md(inserted before## Explorationson legacy indexes).openkb remove, with entity actions shown in the--dry-runpreview.Surface integration (entities handled everywhere concepts/summaries are)
openkb list/ chat/list,openkb status, structural lint (check_index_sync, orphans), the semantic linter, the skill-new content gate, theAGENTS.mdschema, and thequerystrategy ("who/what is X" →entities/).openkb initscaffoldsentities/; sharedPAGE_CONTENT_DIRS/INDEX_SEEDconstants.openkb recompile [--all | <doc>] [--dry-run] [--yes] [--refresh-schema]Re-runs the existing compile pipeline on already-indexed docs (no new compile mode) to backfill entities on pre-feature KBs. Long docs reuse the on-disk PageIndex;
--refresh-schemabacks up + refreshes a staleAGENTS.md.Configurable entity types
Optional
entity_types:in.openkb/config.yamloverrides the defaultperson/organization/place/product/work/event/othervocabulary across the plan prompt, entity-page prompts, and validation; omit it for the default. Resolution/validation lives in the config layer (openkb/config.py).Notable fixes & hardening (found via review + live e2e)
relatedslugs so they don't create dangling[[concepts/...]]/[[entities/...]]links (was a flood on feature-dense docs).remove --dry-runpreview now detects JSON-quotedsources:(previously always reported 0 affected pages); preview scan co-located with the executor to prevent drift._write_entitypreserves prior sources on malformed frontmatter; related-only entities don't downgrade index labels to(other).entity_typesis crash-proof (no.format()KeyError on{/}) and the system schema no longer contradicts a custom vocabulary.contentno longer writes the raw JSON envelope as a page body; graceful handling of scalar plan responses;[[entities/X]]whitelist rule in the plan prompt; restored the "don't create a concept that's just the doc topic" guard.remove --keep-empty-concepts→--keep-empty(covers entities too; old flag kept as alias).Non-goals / future work
No LangExtract / TF-IDF, no concept migration, no
relationshipsfrontmatter. Future: char-level source grounding; scale-time retrieval ranking; a shared preview/executor "remaining sources" computation; a concept-type config knob.Test plan
pytest -q→ 587 passedqueryreads them;list/status/lintsurface them;recompilebackfills them;remove --dry-runpreview lists affected entity/concept pages