fix(compiler): preserve frontmatter when closing "---" has no trailing newline by jichaowang02-lang · Pull Request #121 · VectifyAI/OpenKB

jichaowang02-lang · 2026-06-20T03:06:08Z

Summary

A frontmatter-only concept/entity page that ends right at the closing ---
with no trailing newline gets its frontmatter silently destroyed when a
source is added or removed: the opening ---, the type / description
keys, and the existing sources: list are all dropped — corrupting the page
and losing source provenance.

Root cause

frontmatter.split is lossless and, for a page with no trailing newline after
the closing delimiter, returns a block ending in a bare \n--- (not
\n---\n):

# openkb/frontmatter.py — split()after=text.find("\n", nl+1) # newline ending the closing '---' lineifafter==-1:
returntext, ""# closing '---' has no trailing newline

But both _prepend_source_to_frontmatter and _remove_source_from_frontmatter
located the closing delimiter with fm_block.rpartition("\n---\n"), which
matches nothing in that case — so fm_prefix becomes "", every frontmatter
line is lost, and the block is rebuilt from nothing:

fm_prefix, _, _=fm_block.rpartition("\n---\n") # -> ("", "", fm_block)fm_lines=fm_prefix.split("\n") # -> [""]

Concrete trace (_prepend_source_to_frontmatter):

in : '---\nsources: ["summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---'
out: '\nsources: ["summaries/p2.md"]\n---\n' # type, description, p1.md, opening --- all gone

This is reached during normal use via _add_related_link (run for every
related concept/entity at compile time) and the openkb remove flow
(remove_doc_from_{concept,entity}_pages).

Fix

Strip whichever closing form is actually present and re-append the same one:

closing="\n---\n"iffm_block.endswith("\n---\n") else"\n---"fm_prefix=fm_block[: -len(closing)]

With the fix the same input yields
---\nsources: ["summaries/p2.md", "summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---
— all keys and the prior source preserved. Pages that have a body (the common
case) are byte-for-byte unaffected.

Testing

$ pytest tests/test_compiler.py::TestFrontmatterSourceMutation -q
3 passed
$ ruff check openkb/agent/compiler.py
All checks passed!

Adds TestFrontmatterSourceMutation (prepend + remove on a no-trailing-newline
page, plus a with-body regression guard). The existing _add_related_link /
remove_doc_from_entity_pages tests only used pages with a body, so this
path was uncovered.

Note: the rest of tests/test_compiler.py has some pre-existing failures
in the LLM-mock suites that are unrelated to this change (they reproduce on
main without this patch); the frontmatter tests above pass.

…g newline `frontmatter.split` returns the frontmatter block ending in a bare "\n---" (not "\n---\n") for a page that ends right at the closing delimiter with no trailing newline — e.g. a frontmatter-only concept/entity page. Both `_prepend_source_to_frontmatter` and `_remove_source_from_frontmatter` located the closing delimiter with `fm_block.rpartition("\n---\n")`, which finds nothing in that case: `fm_prefix` becomes "" and every existing frontmatter line (the opening "---", `type`, `description`, and the prior `sources:` list) is dropped — silently corrupting the page and losing source provenance. Strip whichever closing form is actually present ("\n---\n" or a bare "\n---") and re-append the same one. Pages that have a body (the common case) are byte-for-byte unaffected. This is reached in practice via `_add_related_link` (run for every related concept/entity during compile) and the `openkb remove` flow (`remove_doc_from_{concept,entity}_pages`). Adds `TestFrontmatterSourceMutation` covering prepend + remove on a no-trailing-newline page, plus a with-body regression guard.

Copilot

Pull request overview

Fixes a compiler corruption bug where mutating a YAML frontmatter sources: list could drop the entire frontmatter when the closing --- delimiter is at EOF with no trailing newline. This makes frontmatter source provenance updates safe for “frontmatter-only” pages.

Changes:

Update _prepend_source_to_frontmatter and _remove_source_from_frontmatter to correctly strip/re-append either \n---\n or bare \n--- closings.
Add regression tests covering prepend/remove behavior for no-trailing-newline pages and a with-body guard.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`openkb/agent/compiler.py`	Preserves existing frontmatter keys/sources by handling both closing delimiter forms when rewriting `sources:`.
`tests/test_compiler.py`	Adds targeted tests to prevent regressions for frontmatter-only pages without a trailing newline.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.



+class TestFrontmatterSourceMutation:
+ """``_prepend``/``_remove_source_from_frontmatter`` must preserve existing


…tmatter)

KylinMountain

Thanks @jichaowang02-lang — nice catch. The no-trailing-newline corruption is a real one (hits via _add_related_link and openkb remove), and the fix handles both closing forms cleanly. LGTM 👍

jichaowang02-lang · 2026-06-20T03:06:08Z

Summary

A frontmatter-only concept/entity page that ends right at the closing ---
with no trailing newline gets its frontmatter silently destroyed when a
source is added or removed: the opening ---, the type / description
keys, and the existing sources: list are all dropped — corrupting the page
and losing source provenance.

Root cause

frontmatter.split is lossless and, for a page with no trailing newline after
the closing delimiter, returns a block ending in a bare \n--- (not
\n---\n):

# openkb/frontmatter.py — split()after=text.find("\n", nl+1) # newline ending the closing '---' lineifafter==-1:
returntext, ""# closing '---' has no trailing newline

But both _prepend_source_to_frontmatter and _remove_source_from_frontmatter
located the closing delimiter with fm_block.rpartition("\n---\n"), which
matches nothing in that case — so fm_prefix becomes "", every frontmatter
line is lost, and the block is rebuilt from nothing:

fm_prefix, _, _=fm_block.rpartition("\n---\n") # -> ("", "", fm_block)fm_lines=fm_prefix.split("\n") # -> [""]

Concrete trace (_prepend_source_to_frontmatter):

in : '---\nsources: ["summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---'
out: '\nsources: ["summaries/p2.md"]\n---\n' # type, description, p1.md, opening --- all gone

This is reached during normal use via _add_related_link (run for every
related concept/entity at compile time) and the openkb remove flow
(remove_doc_from_{concept,entity}_pages).

Fix

Strip whichever closing form is actually present and re-append the same one:

closing="\n---\n"iffm_block.endswith("\n---\n") else"\n---"fm_prefix=fm_block[: -len(closing)]

With the fix the same input yields
---\nsources: ["summaries/p2.md", "summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---
— all keys and the prior source preserved. Pages that have a body (the common
case) are byte-for-byte unaffected.

Testing

$ pytest tests/test_compiler.py::TestFrontmatterSourceMutation -q
3 passed
$ ruff check openkb/agent/compiler.py
All checks passed!

Adds TestFrontmatterSourceMutation (prepend + remove on a no-trailing-newline
page, plus a with-body regression guard). The existing _add_related_link /
remove_doc_from_entity_pages tests only used pages with a body, so this
path was uncovered.

Note: the rest of tests/test_compiler.py has some pre-existing failures
in the LLM-mock suites that are unrelated to this change (they reproduce on
main without this patch); the frontmatter tests above pass.

…g newline `frontmatter.split` returns the frontmatter block ending in a bare "\n---" (not "\n---\n") for a page that ends right at the closing delimiter with no trailing newline — e.g. a frontmatter-only concept/entity page. Both `_prepend_source_to_frontmatter` and `_remove_source_from_frontmatter` located the closing delimiter with `fm_block.rpartition("\n---\n")`, which finds nothing in that case: `fm_prefix` becomes "" and every existing frontmatter line (the opening "---", `type`, `description`, and the prior `sources:` list) is dropped — silently corrupting the page and losing source provenance. Strip whichever closing form is actually present ("\n---\n" or a bare "\n---") and re-append the same one. Pages that have a body (the common case) are byte-for-byte unaffected. This is reached in practice via `_add_related_link` (run for every related concept/entity during compile) and the `openkb remove` flow (`remove_doc_from_{concept,entity}_pages`). Adds `TestFrontmatterSourceMutation` covering prepend + remove on a no-trailing-newline page, plus a with-body regression guard.

Copilot

Pull request overview

Fixes a compiler corruption bug where mutating a YAML frontmatter sources: list could drop the entire frontmatter when the closing --- delimiter is at EOF with no trailing newline. This makes frontmatter source provenance updates safe for “frontmatter-only” pages.

Changes:

Update _prepend_source_to_frontmatter and _remove_source_from_frontmatter to correctly strip/re-append either \n---\n or bare \n--- closings.
Add regression tests covering prepend/remove behavior for no-trailing-newline pages and a with-body guard.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`openkb/agent/compiler.py`	Preserves existing frontmatter keys/sources by handling both closing delimiter forms when rewriting `sources:`.
`tests/test_compiler.py`	Adds targeted tests to prevent regressions for frontmatter-only pages without a trailing newline.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.



+class TestFrontmatterSourceMutation:
+ """``_prepend``/``_remove_source_from_frontmatter`` must preserve existing


…tmatter)

KylinMountain

Thanks @jichaowang02-lang — nice catch. The no-trailing-newline corruption is a real one (hits via _add_related_link and openkb remove), and the fix handles both closing forms cleanly. LGTM 👍

jichaowang02-lang · 2026-06-20T03:06:08Z

Summary

A frontmatter-only concept/entity page that ends right at the closing ---
with no trailing newline gets its frontmatter silently destroyed when a
source is added or removed: the opening ---, the type / description
keys, and the existing sources: list are all dropped — corrupting the page
and losing source provenance.

Root cause

frontmatter.split is lossless and, for a page with no trailing newline after
the closing delimiter, returns a block ending in a bare \n--- (not
\n---\n):

# openkb/frontmatter.py — split()after=text.find("\n", nl+1) # newline ending the closing '---' lineifafter==-1:
returntext, ""# closing '---' has no trailing newline

But both _prepend_source_to_frontmatter and _remove_source_from_frontmatter
located the closing delimiter with fm_block.rpartition("\n---\n"), which
matches nothing in that case — so fm_prefix becomes "", every frontmatter
line is lost, and the block is rebuilt from nothing:

fm_prefix, _, _=fm_block.rpartition("\n---\n") # -> ("", "", fm_block)fm_lines=fm_prefix.split("\n") # -> [""]

Concrete trace (_prepend_source_to_frontmatter):

in : '---\nsources: ["summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---'
out: '\nsources: ["summaries/p2.md"]\n---\n' # type, description, p1.md, opening --- all gone

This is reached during normal use via _add_related_link (run for every
related concept/entity at compile time) and the openkb remove flow
(remove_doc_from_{concept,entity}_pages).

Fix

Strip whichever closing form is actually present and re-append the same one:

closing="\n---\n"iffm_block.endswith("\n---\n") else"\n---"fm_prefix=fm_block[: -len(closing)]

With the fix the same input yields
---\nsources: ["summaries/p2.md", "summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---
— all keys and the prior source preserved. Pages that have a body (the common
case) are byte-for-byte unaffected.

Testing

$ pytest tests/test_compiler.py::TestFrontmatterSourceMutation -q
3 passed
$ ruff check openkb/agent/compiler.py
All checks passed!

Adds TestFrontmatterSourceMutation (prepend + remove on a no-trailing-newline
page, plus a with-body regression guard). The existing _add_related_link /
remove_doc_from_entity_pages tests only used pages with a body, so this
path was uncovered.

Note: the rest of tests/test_compiler.py has some pre-existing failures
in the LLM-mock suites that are unrelated to this change (they reproduce on
main without this patch); the frontmatter tests above pass.

…g newline `frontmatter.split` returns the frontmatter block ending in a bare "\n---" (not "\n---\n") for a page that ends right at the closing delimiter with no trailing newline — e.g. a frontmatter-only concept/entity page. Both `_prepend_source_to_frontmatter` and `_remove_source_from_frontmatter` located the closing delimiter with `fm_block.rpartition("\n---\n")`, which finds nothing in that case: `fm_prefix` becomes "" and every existing frontmatter line (the opening "---", `type`, `description`, and the prior `sources:` list) is dropped — silently corrupting the page and losing source provenance. Strip whichever closing form is actually present ("\n---\n" or a bare "\n---") and re-append the same one. Pages that have a body (the common case) are byte-for-byte unaffected. This is reached in practice via `_add_related_link` (run for every related concept/entity during compile) and the `openkb remove` flow (`remove_doc_from_{concept,entity}_pages`). Adds `TestFrontmatterSourceMutation` covering prepend + remove on a no-trailing-newline page, plus a with-body regression guard.

Copilot

Pull request overview

Fixes a compiler corruption bug where mutating a YAML frontmatter sources: list could drop the entire frontmatter when the closing --- delimiter is at EOF with no trailing newline. This makes frontmatter source provenance updates safe for “frontmatter-only” pages.

Changes:

Update _prepend_source_to_frontmatter and _remove_source_from_frontmatter to correctly strip/re-append either \n---\n or bare \n--- closings.
Add regression tests covering prepend/remove behavior for no-trailing-newline pages and a with-body guard.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`openkb/agent/compiler.py`	Preserves existing frontmatter keys/sources by handling both closing delimiter forms when rewriting `sources:`.
`tests/test_compiler.py`	Adds targeted tests to prevent regressions for frontmatter-only pages without a trailing newline.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.



+class TestFrontmatterSourceMutation:
+ """``_prepend``/``_remove_source_from_frontmatter`` must preserve existing


…tmatter)

KylinMountain

Thanks @jichaowang02-lang — nice catch. The no-trailing-newline corruption is a real one (hits via _add_related_link and openkb remove), and the fix handles both closing forms cleanly. LGTM 👍

jichaowang02-lang · 2026-06-20T03:06:08Z

Summary

A frontmatter-only concept/entity page that ends right at the closing ---
with no trailing newline gets its frontmatter silently destroyed when a
source is added or removed: the opening ---, the type / description
keys, and the existing sources: list are all dropped — corrupting the page
and losing source provenance.

Root cause

frontmatter.split is lossless and, for a page with no trailing newline after
the closing delimiter, returns a block ending in a bare \n--- (not
\n---\n):

# openkb/frontmatter.py — split()after=text.find("\n", nl+1) # newline ending the closing '---' lineifafter==-1:
returntext, ""# closing '---' has no trailing newline

But both _prepend_source_to_frontmatter and _remove_source_from_frontmatter
located the closing delimiter with fm_block.rpartition("\n---\n"), which
matches nothing in that case — so fm_prefix becomes "", every frontmatter
line is lost, and the block is rebuilt from nothing:

fm_prefix, _, _=fm_block.rpartition("\n---\n") # -> ("", "", fm_block)fm_lines=fm_prefix.split("\n") # -> [""]

Concrete trace (_prepend_source_to_frontmatter):

in : '---\nsources: ["summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---'
out: '\nsources: ["summaries/p2.md"]\n---\n' # type, description, p1.md, opening --- all gone

This is reached during normal use via _add_related_link (run for every
related concept/entity at compile time) and the openkb remove flow
(remove_doc_from_{concept,entity}_pages).

Fix

Strip whichever closing form is actually present and re-append the same one:

closing="\n---\n"iffm_block.endswith("\n---\n") else"\n---"fm_prefix=fm_block[: -len(closing)]

With the fix the same input yields
---\nsources: ["summaries/p2.md", "summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---
— all keys and the prior source preserved. Pages that have a body (the common
case) are byte-for-byte unaffected.

Testing

$ pytest tests/test_compiler.py::TestFrontmatterSourceMutation -q
3 passed
$ ruff check openkb/agent/compiler.py
All checks passed!

Adds TestFrontmatterSourceMutation (prepend + remove on a no-trailing-newline
page, plus a with-body regression guard). The existing _add_related_link /
remove_doc_from_entity_pages tests only used pages with a body, so this
path was uncovered.

Note: the rest of tests/test_compiler.py has some pre-existing failures
in the LLM-mock suites that are unrelated to this change (they reproduce on
main without this patch); the frontmatter tests above pass.

…g newline `frontmatter.split` returns the frontmatter block ending in a bare "\n---" (not "\n---\n") for a page that ends right at the closing delimiter with no trailing newline — e.g. a frontmatter-only concept/entity page. Both `_prepend_source_to_frontmatter` and `_remove_source_from_frontmatter` located the closing delimiter with `fm_block.rpartition("\n---\n")`, which finds nothing in that case: `fm_prefix` becomes "" and every existing frontmatter line (the opening "---", `type`, `description`, and the prior `sources:` list) is dropped — silently corrupting the page and losing source provenance. Strip whichever closing form is actually present ("\n---\n" or a bare "\n---") and re-append the same one. Pages that have a body (the common case) are byte-for-byte unaffected. This is reached in practice via `_add_related_link` (run for every related concept/entity during compile) and the `openkb remove` flow (`remove_doc_from_{concept,entity}_pages`). Adds `TestFrontmatterSourceMutation` covering prepend + remove on a no-trailing-newline page, plus a with-body regression guard.

Copilot

Pull request overview

Fixes a compiler corruption bug where mutating a YAML frontmatter sources: list could drop the entire frontmatter when the closing --- delimiter is at EOF with no trailing newline. This makes frontmatter source provenance updates safe for “frontmatter-only” pages.

Changes:

Update _prepend_source_to_frontmatter and _remove_source_from_frontmatter to correctly strip/re-append either \n---\n or bare \n--- closings.
Add regression tests covering prepend/remove behavior for no-trailing-newline pages and a with-body guard.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`openkb/agent/compiler.py`	Preserves existing frontmatter keys/sources by handling both closing delimiter forms when rewriting `sources:`.
`tests/test_compiler.py`	Adds targeted tests to prevent regressions for frontmatter-only pages without a trailing newline.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.



+class TestFrontmatterSourceMutation:
+ """``_prepend``/``_remove_source_from_frontmatter`` must preserve existing


…tmatter)

KylinMountain

Thanks @jichaowang02-lang — nice catch. The no-trailing-newline corruption is a real one (hits via _add_related_link and openkb remove), and the fix handles both closing forms cleanly. LGTM 👍

jichaowang02-lang · 2026-06-20T03:06:08Z

Summary

A frontmatter-only concept/entity page that ends right at the closing ---
with no trailing newline gets its frontmatter silently destroyed when a
source is added or removed: the opening ---, the type / description
keys, and the existing sources: list are all dropped — corrupting the page
and losing source provenance.

Root cause

frontmatter.split is lossless and, for a page with no trailing newline after
the closing delimiter, returns a block ending in a bare \n--- (not
\n---\n):

# openkb/frontmatter.py — split()after=text.find("\n", nl+1) # newline ending the closing '---' lineifafter==-1:
returntext, ""# closing '---' has no trailing newline

But both _prepend_source_to_frontmatter and _remove_source_from_frontmatter
located the closing delimiter with fm_block.rpartition("\n---\n"), which
matches nothing in that case — so fm_prefix becomes "", every frontmatter
line is lost, and the block is rebuilt from nothing:

fm_prefix, _, _=fm_block.rpartition("\n---\n") # -> ("", "", fm_block)fm_lines=fm_prefix.split("\n") # -> [""]

Concrete trace (_prepend_source_to_frontmatter):

in : '---\nsources: ["summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---'
out: '\nsources: ["summaries/p2.md"]\n---\n' # type, description, p1.md, opening --- all gone

This is reached during normal use via _add_related_link (run for every
related concept/entity at compile time) and the openkb remove flow
(remove_doc_from_{concept,entity}_pages).

Fix

Strip whichever closing form is actually present and re-append the same one:

closing="\n---\n"iffm_block.endswith("\n---\n") else"\n---"fm_prefix=fm_block[: -len(closing)]

With the fix the same input yields
---\nsources: ["summaries/p2.md", "summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---
— all keys and the prior source preserved. Pages that have a body (the common
case) are byte-for-byte unaffected.

Testing

$ pytest tests/test_compiler.py::TestFrontmatterSourceMutation -q
3 passed
$ ruff check openkb/agent/compiler.py
All checks passed!

Adds TestFrontmatterSourceMutation (prepend + remove on a no-trailing-newline
page, plus a with-body regression guard). The existing _add_related_link /
remove_doc_from_entity_pages tests only used pages with a body, so this
path was uncovered.

Note: the rest of tests/test_compiler.py has some pre-existing failures
in the LLM-mock suites that are unrelated to this change (they reproduce on
main without this patch); the frontmatter tests above pass.

…g newline `frontmatter.split` returns the frontmatter block ending in a bare "\n---" (not "\n---\n") for a page that ends right at the closing delimiter with no trailing newline — e.g. a frontmatter-only concept/entity page. Both `_prepend_source_to_frontmatter` and `_remove_source_from_frontmatter` located the closing delimiter with `fm_block.rpartition("\n---\n")`, which finds nothing in that case: `fm_prefix` becomes "" and every existing frontmatter line (the opening "---", `type`, `description`, and the prior `sources:` list) is dropped — silently corrupting the page and losing source provenance. Strip whichever closing form is actually present ("\n---\n" or a bare "\n---") and re-append the same one. Pages that have a body (the common case) are byte-for-byte unaffected. This is reached in practice via `_add_related_link` (run for every related concept/entity during compile) and the `openkb remove` flow (`remove_doc_from_{concept,entity}_pages`). Adds `TestFrontmatterSourceMutation` covering prepend + remove on a no-trailing-newline page, plus a with-body regression guard.

Copilot

Pull request overview

Fixes a compiler corruption bug where mutating a YAML frontmatter sources: list could drop the entire frontmatter when the closing --- delimiter is at EOF with no trailing newline. This makes frontmatter source provenance updates safe for “frontmatter-only” pages.

Changes:

Update _prepend_source_to_frontmatter and _remove_source_from_frontmatter to correctly strip/re-append either \n---\n or bare \n--- closings.
Add regression tests covering prepend/remove behavior for no-trailing-newline pages and a with-body guard.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`openkb/agent/compiler.py`	Preserves existing frontmatter keys/sources by handling both closing delimiter forms when rewriting `sources:`.
`tests/test_compiler.py`	Adds targeted tests to prevent regressions for frontmatter-only pages without a trailing newline.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.



+class TestFrontmatterSourceMutation:
+ """``_prepend``/``_remove_source_from_frontmatter`` must preserve existing


…tmatter)

KylinMountain

Thanks @jichaowang02-lang — nice catch. The no-trailing-newline corruption is a real one (hits via _add_related_link and openkb remove), and the fix handles both closing forms cleanly. LGTM 👍

jichaowang02-lang · 2026-06-20T03:06:08Z

Summary

A frontmatter-only concept/entity page that ends right at the closing ---
with no trailing newline gets its frontmatter silently destroyed when a
source is added or removed: the opening ---, the type / description
keys, and the existing sources: list are all dropped — corrupting the page
and losing source provenance.

Root cause

frontmatter.split is lossless and, for a page with no trailing newline after
the closing delimiter, returns a block ending in a bare \n--- (not
\n---\n):

# openkb/frontmatter.py — split()after=text.find("\n", nl+1) # newline ending the closing '---' lineifafter==-1:
returntext, ""# closing '---' has no trailing newline

But both _prepend_source_to_frontmatter and _remove_source_from_frontmatter
located the closing delimiter with fm_block.rpartition("\n---\n"), which
matches nothing in that case — so fm_prefix becomes "", every frontmatter
line is lost, and the block is rebuilt from nothing:

fm_prefix, _, _=fm_block.rpartition("\n---\n") # -> ("", "", fm_block)fm_lines=fm_prefix.split("\n") # -> [""]

Concrete trace (_prepend_source_to_frontmatter):

in : '---\nsources: ["summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---'
out: '\nsources: ["summaries/p2.md"]\n---\n' # type, description, p1.md, opening --- all gone

This is reached during normal use via _add_related_link (run for every
related concept/entity at compile time) and the openkb remove flow
(remove_doc_from_{concept,entity}_pages).

Fix

Strip whichever closing form is actually present and re-append the same one:

closing="\n---\n"iffm_block.endswith("\n---\n") else"\n---"fm_prefix=fm_block[: -len(closing)]

With the fix the same input yields
---\nsources: ["summaries/p2.md", "summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---
— all keys and the prior source preserved. Pages that have a body (the common
case) are byte-for-byte unaffected.

Testing

$ pytest tests/test_compiler.py::TestFrontmatterSourceMutation -q
3 passed
$ ruff check openkb/agent/compiler.py
All checks passed!

Adds TestFrontmatterSourceMutation (prepend + remove on a no-trailing-newline
page, plus a with-body regression guard). The existing _add_related_link /
remove_doc_from_entity_pages tests only used pages with a body, so this
path was uncovered.

Note: the rest of tests/test_compiler.py has some pre-existing failures
in the LLM-mock suites that are unrelated to this change (they reproduce on
main without this patch); the frontmatter tests above pass.

…g newline `frontmatter.split` returns the frontmatter block ending in a bare "\n---" (not "\n---\n") for a page that ends right at the closing delimiter with no trailing newline — e.g. a frontmatter-only concept/entity page. Both `_prepend_source_to_frontmatter` and `_remove_source_from_frontmatter` located the closing delimiter with `fm_block.rpartition("\n---\n")`, which finds nothing in that case: `fm_prefix` becomes "" and every existing frontmatter line (the opening "---", `type`, `description`, and the prior `sources:` list) is dropped — silently corrupting the page and losing source provenance. Strip whichever closing form is actually present ("\n---\n" or a bare "\n---") and re-append the same one. Pages that have a body (the common case) are byte-for-byte unaffected. This is reached in practice via `_add_related_link` (run for every related concept/entity during compile) and the `openkb remove` flow (`remove_doc_from_{concept,entity}_pages`). Adds `TestFrontmatterSourceMutation` covering prepend + remove on a no-trailing-newline page, plus a with-body regression guard.

Copilot

Pull request overview

Fixes a compiler corruption bug where mutating a YAML frontmatter sources: list could drop the entire frontmatter when the closing --- delimiter is at EOF with no trailing newline. This makes frontmatter source provenance updates safe for “frontmatter-only” pages.

Changes:

Update _prepend_source_to_frontmatter and _remove_source_from_frontmatter to correctly strip/re-append either \n---\n or bare \n--- closings.
Add regression tests covering prepend/remove behavior for no-trailing-newline pages and a with-body guard.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`openkb/agent/compiler.py`	Preserves existing frontmatter keys/sources by handling both closing delimiter forms when rewriting `sources:`.
`tests/test_compiler.py`	Adds targeted tests to prevent regressions for frontmatter-only pages without a trailing newline.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.



+class TestFrontmatterSourceMutation:
+ """``_prepend``/``_remove_source_from_frontmatter`` must preserve existing


…tmatter)

KylinMountain

Thanks @jichaowang02-lang — nice catch. The no-trailing-newline corruption is a real one (hits via _add_related_link and openkb remove), and the fix handles both closing forms cleanly. LGTM 👍

jichaowang02-lang · 2026-06-20T03:06:08Z

Summary

A frontmatter-only concept/entity page that ends right at the closing ---
with no trailing newline gets its frontmatter silently destroyed when a
source is added or removed: the opening ---, the type / description
keys, and the existing sources: list are all dropped — corrupting the page
and losing source provenance.

Root cause

frontmatter.split is lossless and, for a page with no trailing newline after
the closing delimiter, returns a block ending in a bare \n--- (not
\n---\n):

# openkb/frontmatter.py — split()after=text.find("\n", nl+1) # newline ending the closing '---' lineifafter==-1:
returntext, ""# closing '---' has no trailing newline

But both _prepend_source_to_frontmatter and _remove_source_from_frontmatter
located the closing delimiter with fm_block.rpartition("\n---\n"), which
matches nothing in that case — so fm_prefix becomes "", every frontmatter
line is lost, and the block is rebuilt from nothing:

fm_prefix, _, _=fm_block.rpartition("\n---\n") # -> ("", "", fm_block)fm_lines=fm_prefix.split("\n") # -> [""]

Concrete trace (_prepend_source_to_frontmatter):

in : '---\nsources: ["summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---'
out: '\nsources: ["summaries/p2.md"]\n---\n' # type, description, p1.md, opening --- all gone

This is reached during normal use via _add_related_link (run for every
related concept/entity at compile time) and the openkb remove flow
(remove_doc_from_{concept,entity}_pages).

Fix

Strip whichever closing form is actually present and re-append the same one:

closing="\n---\n"iffm_block.endswith("\n---\n") else"\n---"fm_prefix=fm_block[: -len(closing)]

With the fix the same input yields
---\nsources: ["summaries/p2.md", "summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---
— all keys and the prior source preserved. Pages that have a body (the common
case) are byte-for-byte unaffected.

Testing

$ pytest tests/test_compiler.py::TestFrontmatterSourceMutation -q
3 passed
$ ruff check openkb/agent/compiler.py
All checks passed!

Adds TestFrontmatterSourceMutation (prepend + remove on a no-trailing-newline
page, plus a with-body regression guard). The existing _add_related_link /
remove_doc_from_entity_pages tests only used pages with a body, so this
path was uncovered.

Note: the rest of tests/test_compiler.py has some pre-existing failures
in the LLM-mock suites that are unrelated to this change (they reproduce on
main without this patch); the frontmatter tests above pass.

…g newline `frontmatter.split` returns the frontmatter block ending in a bare "\n---" (not "\n---\n") for a page that ends right at the closing delimiter with no trailing newline — e.g. a frontmatter-only concept/entity page. Both `_prepend_source_to_frontmatter` and `_remove_source_from_frontmatter` located the closing delimiter with `fm_block.rpartition("\n---\n")`, which finds nothing in that case: `fm_prefix` becomes "" and every existing frontmatter line (the opening "---", `type`, `description`, and the prior `sources:` list) is dropped — silently corrupting the page and losing source provenance. Strip whichever closing form is actually present ("\n---\n" or a bare "\n---") and re-append the same one. Pages that have a body (the common case) are byte-for-byte unaffected. This is reached in practice via `_add_related_link` (run for every related concept/entity during compile) and the `openkb remove` flow (`remove_doc_from_{concept,entity}_pages`). Adds `TestFrontmatterSourceMutation` covering prepend + remove on a no-trailing-newline page, plus a with-body regression guard.

Copilot

Pull request overview

Fixes a compiler corruption bug where mutating a YAML frontmatter sources: list could drop the entire frontmatter when the closing --- delimiter is at EOF with no trailing newline. This makes frontmatter source provenance updates safe for “frontmatter-only” pages.

Changes:

Update _prepend_source_to_frontmatter and _remove_source_from_frontmatter to correctly strip/re-append either \n---\n or bare \n--- closings.
Add regression tests covering prepend/remove behavior for no-trailing-newline pages and a with-body guard.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`openkb/agent/compiler.py`	Preserves existing frontmatter keys/sources by handling both closing delimiter forms when rewriting `sources:`.
`tests/test_compiler.py`	Adds targeted tests to prevent regressions for frontmatter-only pages without a trailing newline.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.



+class TestFrontmatterSourceMutation:
+ """``_prepend``/``_remove_source_from_frontmatter`` must preserve existing


…tmatter)

KylinMountain

Thanks @jichaowang02-lang — nice catch. The no-trailing-newline corruption is a real one (hits via _add_related_link and openkb remove), and the fix handles both closing forms cleanly. LGTM 👍

jichaowang02-lang · 2026-06-20T03:06:08Z

Summary

A frontmatter-only concept/entity page that ends right at the closing ---
with no trailing newline gets its frontmatter silently destroyed when a
source is added or removed: the opening ---, the type / description
keys, and the existing sources: list are all dropped — corrupting the page
and losing source provenance.

Root cause

frontmatter.split is lossless and, for a page with no trailing newline after
the closing delimiter, returns a block ending in a bare \n--- (not
\n---\n):

# openkb/frontmatter.py — split()after=text.find("\n", nl+1) # newline ending the closing '---' lineifafter==-1:
returntext, ""# closing '---' has no trailing newline

But both _prepend_source_to_frontmatter and _remove_source_from_frontmatter
located the closing delimiter with fm_block.rpartition("\n---\n"), which
matches nothing in that case — so fm_prefix becomes "", every frontmatter
line is lost, and the block is rebuilt from nothing:

fm_prefix, _, _=fm_block.rpartition("\n---\n") # -> ("", "", fm_block)fm_lines=fm_prefix.split("\n") # -> [""]

Concrete trace (_prepend_source_to_frontmatter):

in : '---\nsources: ["summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---'
out: '\nsources: ["summaries/p2.md"]\n---\n' # type, description, p1.md, opening --- all gone

This is reached during normal use via _add_related_link (run for every
related concept/entity at compile time) and the openkb remove flow
(remove_doc_from_{concept,entity}_pages).

Fix

Strip whichever closing form is actually present and re-append the same one:

closing="\n---\n"iffm_block.endswith("\n---\n") else"\n---"fm_prefix=fm_block[: -len(closing)]

With the fix the same input yields
---\nsources: ["summaries/p2.md", "summaries/p1.md"]\ntype: "Concept"\ndescription: "Focus"\n---
— all keys and the prior source preserved. Pages that have a body (the common
case) are byte-for-byte unaffected.

Testing

$ pytest tests/test_compiler.py::TestFrontmatterSourceMutation -q
3 passed
$ ruff check openkb/agent/compiler.py
All checks passed!

Adds TestFrontmatterSourceMutation (prepend + remove on a no-trailing-newline
page, plus a with-body regression guard). The existing _add_related_link /
remove_doc_from_entity_pages tests only used pages with a body, so this
path was uncovered.

Note: the rest of tests/test_compiler.py has some pre-existing failures
in the LLM-mock suites that are unrelated to this change (they reproduce on
main without this patch); the frontmatter tests above pass.

…g newline `frontmatter.split` returns the frontmatter block ending in a bare "\n---" (not "\n---\n") for a page that ends right at the closing delimiter with no trailing newline — e.g. a frontmatter-only concept/entity page. Both `_prepend_source_to_frontmatter` and `_remove_source_from_frontmatter` located the closing delimiter with `fm_block.rpartition("\n---\n")`, which finds nothing in that case: `fm_prefix` becomes "" and every existing frontmatter line (the opening "---", `type`, `description`, and the prior `sources:` list) is dropped — silently corrupting the page and losing source provenance. Strip whichever closing form is actually present ("\n---\n" or a bare "\n---") and re-append the same one. Pages that have a body (the common case) are byte-for-byte unaffected. This is reached in practice via `_add_related_link` (run for every related concept/entity during compile) and the `openkb remove` flow (`remove_doc_from_{concept,entity}_pages`). Adds `TestFrontmatterSourceMutation` covering prepend + remove on a no-trailing-newline page, plus a with-body regression guard.

Copilot

Pull request overview

Fixes a compiler corruption bug where mutating a YAML frontmatter sources: list could drop the entire frontmatter when the closing --- delimiter is at EOF with no trailing newline. This makes frontmatter source provenance updates safe for “frontmatter-only” pages.

Changes:

Update _prepend_source_to_frontmatter and _remove_source_from_frontmatter to correctly strip/re-append either \n---\n or bare \n--- closings.
Add regression tests covering prepend/remove behavior for no-trailing-newline pages and a with-body guard.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
`openkb/agent/compiler.py`	Preserves existing frontmatter keys/sources by handling both closing delimiter forms when rewriting `sources:`.
`tests/test_compiler.py`	Adds targeted tests to prevent regressions for frontmatter-only pages without a trailing newline.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.



+class TestFrontmatterSourceMutation:
+ """``_prepend``/``_remove_source_from_frontmatter`` must preserve existing


…tmatter)

KylinMountain

Thanks @jichaowang02-lang — nice catch. The no-trailing-newline corruption is a real one (hits via _add_related_link and openkb remove), and the fix handles both closing forms cleanly. LGTM 👍

CopilotAI review requested due to automatic review settings June 20, 2026 03:06

Copilotstarted reviewing on behalf of jichaowang02-lang June 20, 2026 03:06 View session

CopilotAI reviewed Jun 20, 2026
View reviewed changes

Comment threadtests/test_compiler.py Outdated

class TestFrontmatterSourceMutation:
"""``_prepend``/``_remove_source_from_frontmatter`` must preserve existing

test: fix docstring referencing _prepend (now _prepend_source_to_fron…
f2366cc
…tmatter)

KylinMountain approved these changes Jun 24, 2026
View reviewed changes

KylinMountain merged commit b2a4775 into VectifyAI:mainJun 24, 2026

CopilotAI review requested due to automatic review settings June 20, 2026 03:06

Copilotstarted reviewing on behalf of jichaowang02-lang June 20, 2026 03:06 View session

CopilotAI reviewed Jun 20, 2026
View reviewed changes

Comment threadtests/test_compiler.py Outdated

class TestFrontmatterSourceMutation:
"""``_prepend``/``_remove_source_from_frontmatter`` must preserve existing

test: fix docstring referencing _prepend (now _prepend_source_to_fron…
f2366cc
…tmatter)

KylinMountain approved these changes Jun 24, 2026
View reviewed changes

KylinMountain merged commit b2a4775 into VectifyAI:mainJun 24, 2026

CopilotAI review requested due to automatic review settings June 20, 2026 03:06

Copilotstarted reviewing on behalf of jichaowang02-lang June 20, 2026 03:06 View session

CopilotAI reviewed Jun 20, 2026
View reviewed changes

Comment threadtests/test_compiler.py Outdated

class TestFrontmatterSourceMutation:
"""``_prepend``/``_remove_source_from_frontmatter`` must preserve existing

test: fix docstring referencing _prepend (now _prepend_source_to_fron…
f2366cc
…tmatter)

KylinMountain approved these changes Jun 24, 2026
View reviewed changes

KylinMountain merged commit b2a4775 into VectifyAI:mainJun 24, 2026

CopilotAI review requested due to automatic review settings June 20, 2026 03:06

Copilotstarted reviewing on behalf of jichaowang02-lang June 20, 2026 03:06 View session

CopilotAI reviewed Jun 20, 2026
View reviewed changes

Comment threadtests/test_compiler.py Outdated

class TestFrontmatterSourceMutation:
"""``_prepend``/``_remove_source_from_frontmatter`` must preserve existing

test: fix docstring referencing _prepend (now _prepend_source_to_fron…
f2366cc
…tmatter)

KylinMountain approved these changes Jun 24, 2026
View reviewed changes

KylinMountain merged commit b2a4775 into VectifyAI:mainJun 24, 2026

CopilotAI review requested due to automatic review settings June 20, 2026 03:06

Copilotstarted reviewing on behalf of jichaowang02-lang June 20, 2026 03:06 View session

CopilotAI reviewed Jun 20, 2026
View reviewed changes

Comment threadtests/test_compiler.py Outdated

class TestFrontmatterSourceMutation:
"""``_prepend``/``_remove_source_from_frontmatter`` must preserve existing

test: fix docstring referencing _prepend (now _prepend_source_to_fron…
f2366cc
…tmatter)

KylinMountain approved these changes Jun 24, 2026
View reviewed changes

KylinMountain merged commit b2a4775 into VectifyAI:mainJun 24, 2026

CopilotAI review requested due to automatic review settings June 20, 2026 03:06

Copilotstarted reviewing on behalf of jichaowang02-lang June 20, 2026 03:06 View session

CopilotAI reviewed Jun 20, 2026
View reviewed changes

Comment threadtests/test_compiler.py Outdated

class TestFrontmatterSourceMutation:
"""``_prepend``/``_remove_source_from_frontmatter`` must preserve existing

test: fix docstring referencing _prepend (now _prepend_source_to_fron…
f2366cc
…tmatter)

KylinMountain approved these changes Jun 24, 2026
View reviewed changes

KylinMountain merged commit b2a4775 into VectifyAI:mainJun 24, 2026

CopilotAI review requested due to automatic review settings June 20, 2026 03:06

Copilotstarted reviewing on behalf of jichaowang02-lang June 20, 2026 03:06 View session

CopilotAI reviewed Jun 20, 2026
View reviewed changes

Comment threadtests/test_compiler.py Outdated

class TestFrontmatterSourceMutation:
"""``_prepend``/``_remove_source_from_frontmatter`` must preserve existing

test: fix docstring referencing _prepend (now _prepend_source_to_fron…
f2366cc
…tmatter)

KylinMountain approved these changes Jun 24, 2026
View reviewed changes

KylinMountain merged commit b2a4775 into VectifyAI:mainJun 24, 2026

CopilotAI review requested due to automatic review settings June 20, 2026 03:06

Copilotstarted reviewing on behalf of jichaowang02-lang June 20, 2026 03:06 View session

CopilotAI reviewed Jun 20, 2026
View reviewed changes

Comment threadtests/test_compiler.py Outdated

class TestFrontmatterSourceMutation:
"""``_prepend``/``_remove_source_from_frontmatter`` must preserve existing

test: fix docstring referencing _prepend (now _prepend_source_to_fron…
f2366cc
…tmatter)

KylinMountain approved these changes Jun 24, 2026
View reviewed changes

KylinMountain merged commit b2a4775 into VectifyAI:mainJun 24, 2026



		class TestFrontmatterSourceMutation:
		"""``_prepend``/``_remove_source_from_frontmatter`` must preserve existing

Uh oh!

Conversation

jichaowang02-lang commented Jun 20, 2026

Summary

Root cause

Fix

Testing

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

KylinMountain left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jichaowang02-lang commented Jun 20, 2026

Summary

Root cause

Fix

Testing

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

KylinMountain left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jichaowang02-lang commented Jun 20, 2026

Summary

Root cause

Fix

Testing

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

KylinMountain left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jichaowang02-lang commented Jun 20, 2026

Summary

Root cause

Fix

Testing

Uh oh!