Compiler writes an unvalidated model non-answer verbatim into wiki/index.md as a document description

[dc-senf]

Summary

While evaluating OpenKB as a knowledge-base compiler over an engineering-log corpus, a 30-document probe produced an index.md row whose description reads literally no change from previous output. That is a model non-answer persisted verbatim as a document's one-line brief. Between the LLM response and the index write there is no check that a generated brief is actually a brief — only a truthiness test, which a non-empty non-answer passes.

Environment

• openkb 0.4.5 (pip), Python 3.13, Windows
• Model glm-5p2 via LiteLLM → Fireworks
• Run date 2026-08-11; 28 documents compiled (2 of 30 were already ingested and skipped)
• Code cited against commit ff54396

Observed

wiki/index.md, line 25:

- [[summaries/task-0087-runner-safety]] (short) — no change from previous output

The same string also reached the summary page's frontmatter, wiki/summaries/task-0087-runner-safety.md:

---
type: "Summary"
description: "no change from previous output"
doc_type: short
full_text: "sources/task-0087-runner-safety.md"
---

Measured rate in this run: 1 of 28 documents (~4%).

Where it happens

The prompt asks for a specific artifact — openkb/agent/compiler.py:71:

- "description": A single sentence (under 100 chars) describing the document's main contribution

The field is read straight off the parsed response with no validation — openkb/agent/compiler.py:2262:

doc_brief = summary_parsed.get("description", "")

and written into the index behind a truthiness test only — openkb/agent/compiler.py:1541-1542:

if doc_brief:
 doc_entry += f" — {doc_brief}"

"no change from previous output" is non-empty, so it passes. The same value reaches the summary page via _write_summary(..., description=doc_brief) (openkb/agent/compiler.py:1672 and 2153).

The structural lint does not catch it afterwards either: find_missing_okf_fields tests only for non-empty (openkb/lint.py:565-567), and index.md is explicitly exempt from that check (openkb/lint.py:541).

Why it matters downstream

index.md is step 1 of the query agent's documented search strategy — openkb/agent/query.py:28:

1. Read index.md to see all documents and concepts with brief summaries.

So the brief is the retrieval agent's routing signal. A row carrying a non-answer makes that document effectively unroutable: there is nothing in the index to indicate what it contains.

Reproduction shape

1. openkb init, then compile a corpus of ~30 documents one at a time.
2. Inspect wiki/index.md under ## Documents.
3. Any row whose text is a meta-comment about the generation rather than a description of the document is this defect.

The specific trigger string is model-dependent and we measured one model only [inferred]. The absence of validation is not model-dependent — no model's description output is checked on this path [measured, from the code above].

Suggested minimal fix

Validate description before it is written, at the point it is read (compiler.py:2262) or immediately before the index/summary write:

• reject a brief that is empty after strip, that exceeds the 100-char contract the prompt states, or that matches a small set of non-answer patterns (no change, unchanged, same as, n/a, ...);
• on rejection, re-request once; if it still fails, write the row without a brief, or with an explicit marker, rather than persisting the non-answer.

Writing no brief is recoverable and visible to find_missing_okf_fields. A wrong brief is neither.

Happy to send a PR along these lines if the shape looks right to you.

Reported by Claude Code.