fix(compiler): cap short-doc source text via configurable max_doc_chars (#73)

[hudsonwa]

What

Fixes#73 (partially, the "graceful degradation" minimum it explicitly allows).

The short-doc compile path sends the entire markdown source file as a single LLM message (compile_short_doc reads the whole file and injects it into the summary prompt). An oversized markdown doc therefore overflows the model context and either fails or has its tail silently dropped — which is exactly what breaks ingesting large markdown files with local/private models.

This adds a configurable hard cap, max_doc_chars (default 500k chars ≈ 125k tokens — a pure safety ceiling that never affects normal docs), that truncates the payload and appends an explicit marker so the model knows the tail was cut and the user can see it happened. Local-model users lower max_doc_chars to fit their context window.

Why this scope

Issue #73 proposes a full heading-aware chunking + hierarchical-synthesis feature (medium/large). This PR ships the small, safe, "at minimum" increment the issue itself lists first: graceful degradation instead of a hard failure. It's a strict safety net; structured chunking can build on top later.

Changes

• openkb/config.py — add max_doc_chars to DEFAULT_CONFIG and GLOBAL_SCALAR_KEYS (layers global → KB like pageindex_threshold).
• openkb/api_models.py — expose max_doc_chars in the writable config schema (keeps GLOBAL_SCALAR_KEYS == _KB_CONFIG_WRITABLE_KEYS).
• openkb/agent/compiler.py — new _maybe_truncate_doc(content, max_doc_chars, doc_name) helper; compile_short_doc applies it after reading the source.
• docs — config.yaml.example, README.md, examples/configuration/README.md.
• tests — test_compiler.py: 4 unit tests on the helper + 2 integration tests proving compile_short_doc truncates an oversized doc (marker present, tail gone) and passes through a doc within budget.

Verification

• RED→GREEN proven: with the source fix reverted, _maybe_truncate_doc doesn't exist and the new tests fail at import (RED); with the fix, all 6 pass (GREEN).
• pytest tests/test_compiler.py tests/test_config.py tests/test_api.py tests/test_api_documents.py tests/test_api_watch.py tests/test_indexer.py tests/test_file_size.py → 422 passed.
• ruff check clean; ruff format clean.

No new dependencies. Diff: ~37 source lines + docs + tests.

[hudsonwa]

What

Fixes#73 (partially, the "graceful degradation" minimum it explicitly allows).

The short-doc compile path sends the entire markdown source file as a single LLM message (compile_short_doc reads the whole file and injects it into the summary prompt). An oversized markdown doc therefore overflows the model context and either fails or has its tail silently dropped — which is exactly what breaks ingesting large markdown files with local/private models.

This adds a configurable hard cap, max_doc_chars (default 500k chars ≈ 125k tokens — a pure safety ceiling that never affects normal docs), that truncates the payload and appends an explicit marker so the model knows the tail was cut and the user can see it happened. Local-model users lower max_doc_chars to fit their context window.

Why this scope

Issue #73 proposes a full heading-aware chunking + hierarchical-synthesis feature (medium/large). This PR ships the small, safe, "at minimum" increment the issue itself lists first: graceful degradation instead of a hard failure. It's a strict safety net; structured chunking can build on top later.

Changes

• openkb/config.py — add max_doc_chars to DEFAULT_CONFIG and GLOBAL_SCALAR_KEYS (layers global → KB like pageindex_threshold).
• openkb/api_models.py — expose max_doc_chars in the writable config schema (keeps GLOBAL_SCALAR_KEYS == _KB_CONFIG_WRITABLE_KEYS).
• openkb/agent/compiler.py — new _maybe_truncate_doc(content, max_doc_chars, doc_name) helper; compile_short_doc applies it after reading the source.
• docs — config.yaml.example, README.md, examples/configuration/README.md.
• tests — test_compiler.py: 4 unit tests on the helper + 2 integration tests proving compile_short_doc truncates an oversized doc (marker present, tail gone) and passes through a doc within budget.

Verification

• RED→GREEN proven: with the source fix reverted, _maybe_truncate_doc doesn't exist and the new tests fail at import (RED); with the fix, all 6 pass (GREEN).
• pytest tests/test_compiler.py tests/test_config.py tests/test_api.py tests/test_api_documents.py tests/test_api_watch.py tests/test_indexer.py tests/test_file_size.py → 422 passed.
• ruff check clean; ruff format clean.

No new dependencies. Diff: ~37 source lines + docs + tests.

[hudsonwa]

What

Fixes#73 (partially, the "graceful degradation" minimum it explicitly allows).

The short-doc compile path sends the entire markdown source file as a single LLM message (compile_short_doc reads the whole file and injects it into the summary prompt). An oversized markdown doc therefore overflows the model context and either fails or has its tail silently dropped — which is exactly what breaks ingesting large markdown files with local/private models.

This adds a configurable hard cap, max_doc_chars (default 500k chars ≈ 125k tokens — a pure safety ceiling that never affects normal docs), that truncates the payload and appends an explicit marker so the model knows the tail was cut and the user can see it happened. Local-model users lower max_doc_chars to fit their context window.

Why this scope

Issue #73 proposes a full heading-aware chunking + hierarchical-synthesis feature (medium/large). This PR ships the small, safe, "at minimum" increment the issue itself lists first: graceful degradation instead of a hard failure. It's a strict safety net; structured chunking can build on top later.

Changes

• openkb/config.py — add max_doc_chars to DEFAULT_CONFIG and GLOBAL_SCALAR_KEYS (layers global → KB like pageindex_threshold).
• openkb/api_models.py — expose max_doc_chars in the writable config schema (keeps GLOBAL_SCALAR_KEYS == _KB_CONFIG_WRITABLE_KEYS).
• openkb/agent/compiler.py — new _maybe_truncate_doc(content, max_doc_chars, doc_name) helper; compile_short_doc applies it after reading the source.
• docs — config.yaml.example, README.md, examples/configuration/README.md.
• tests — test_compiler.py: 4 unit tests on the helper + 2 integration tests proving compile_short_doc truncates an oversized doc (marker present, tail gone) and passes through a doc within budget.

Verification

• RED→GREEN proven: with the source fix reverted, _maybe_truncate_doc doesn't exist and the new tests fail at import (RED); with the fix, all 6 pass (GREEN).
• pytest tests/test_compiler.py tests/test_config.py tests/test_api.py tests/test_api_documents.py tests/test_api_watch.py tests/test_indexer.py tests/test_file_size.py → 422 passed.
• ruff check clean; ruff format clean.

No new dependencies. Diff: ~37 source lines + docs + tests.

[hudsonwa]

What

Fixes#73 (partially, the "graceful degradation" minimum it explicitly allows).

The short-doc compile path sends the entire markdown source file as a single LLM message (compile_short_doc reads the whole file and injects it into the summary prompt). An oversized markdown doc therefore overflows the model context and either fails or has its tail silently dropped — which is exactly what breaks ingesting large markdown files with local/private models.

This adds a configurable hard cap, max_doc_chars (default 500k chars ≈ 125k tokens — a pure safety ceiling that never affects normal docs), that truncates the payload and appends an explicit marker so the model knows the tail was cut and the user can see it happened. Local-model users lower max_doc_chars to fit their context window.

Why this scope

Issue #73 proposes a full heading-aware chunking + hierarchical-synthesis feature (medium/large). This PR ships the small, safe, "at minimum" increment the issue itself lists first: graceful degradation instead of a hard failure. It's a strict safety net; structured chunking can build on top later.

Changes

• openkb/config.py — add max_doc_chars to DEFAULT_CONFIG and GLOBAL_SCALAR_KEYS (layers global → KB like pageindex_threshold).
• openkb/api_models.py — expose max_doc_chars in the writable config schema (keeps GLOBAL_SCALAR_KEYS == _KB_CONFIG_WRITABLE_KEYS).
• openkb/agent/compiler.py — new _maybe_truncate_doc(content, max_doc_chars, doc_name) helper; compile_short_doc applies it after reading the source.
• docs — config.yaml.example, README.md, examples/configuration/README.md.
• tests — test_compiler.py: 4 unit tests on the helper + 2 integration tests proving compile_short_doc truncates an oversized doc (marker present, tail gone) and passes through a doc within budget.

Verification

• RED→GREEN proven: with the source fix reverted, _maybe_truncate_doc doesn't exist and the new tests fail at import (RED); with the fix, all 6 pass (GREEN).
• pytest tests/test_compiler.py tests/test_config.py tests/test_api.py tests/test_api_documents.py tests/test_api_watch.py tests/test_indexer.py tests/test_file_size.py → 422 passed.
• ruff check clean; ruff format clean.

No new dependencies. Diff: ~37 source lines + docs + tests.

[hudsonwa]

What

Fixes#73 (partially, the "graceful degradation" minimum it explicitly allows).

The short-doc compile path sends the entire markdown source file as a single LLM message (compile_short_doc reads the whole file and injects it into the summary prompt). An oversized markdown doc therefore overflows the model context and either fails or has its tail silently dropped — which is exactly what breaks ingesting large markdown files with local/private models.

This adds a configurable hard cap, max_doc_chars (default 500k chars ≈ 125k tokens — a pure safety ceiling that never affects normal docs), that truncates the payload and appends an explicit marker so the model knows the tail was cut and the user can see it happened. Local-model users lower max_doc_chars to fit their context window.

Why this scope

Issue #73 proposes a full heading-aware chunking + hierarchical-synthesis feature (medium/large). This PR ships the small, safe, "at minimum" increment the issue itself lists first: graceful degradation instead of a hard failure. It's a strict safety net; structured chunking can build on top later.

Changes

• openkb/config.py — add max_doc_chars to DEFAULT_CONFIG and GLOBAL_SCALAR_KEYS (layers global → KB like pageindex_threshold).
• openkb/api_models.py — expose max_doc_chars in the writable config schema (keeps GLOBAL_SCALAR_KEYS == _KB_CONFIG_WRITABLE_KEYS).
• openkb/agent/compiler.py — new _maybe_truncate_doc(content, max_doc_chars, doc_name) helper; compile_short_doc applies it after reading the source.
• docs — config.yaml.example, README.md, examples/configuration/README.md.
• tests — test_compiler.py: 4 unit tests on the helper + 2 integration tests proving compile_short_doc truncates an oversized doc (marker present, tail gone) and passes through a doc within budget.

Verification

• RED→GREEN proven: with the source fix reverted, _maybe_truncate_doc doesn't exist and the new tests fail at import (RED); with the fix, all 6 pass (GREEN).
• pytest tests/test_compiler.py tests/test_config.py tests/test_api.py tests/test_api_documents.py tests/test_api_watch.py tests/test_indexer.py tests/test_file_size.py → 422 passed.
• ruff check clean; ruff format clean.

No new dependencies. Diff: ~37 source lines + docs + tests.

[hudsonwa]

What

Fixes#73 (partially, the "graceful degradation" minimum it explicitly allows).

The short-doc compile path sends the entire markdown source file as a single LLM message (compile_short_doc reads the whole file and injects it into the summary prompt). An oversized markdown doc therefore overflows the model context and either fails or has its tail silently dropped — which is exactly what breaks ingesting large markdown files with local/private models.

This adds a configurable hard cap, max_doc_chars (default 500k chars ≈ 125k tokens — a pure safety ceiling that never affects normal docs), that truncates the payload and appends an explicit marker so the model knows the tail was cut and the user can see it happened. Local-model users lower max_doc_chars to fit their context window.

Why this scope

Issue #73 proposes a full heading-aware chunking + hierarchical-synthesis feature (medium/large). This PR ships the small, safe, "at minimum" increment the issue itself lists first: graceful degradation instead of a hard failure. It's a strict safety net; structured chunking can build on top later.

Changes

• openkb/config.py — add max_doc_chars to DEFAULT_CONFIG and GLOBAL_SCALAR_KEYS (layers global → KB like pageindex_threshold).
• openkb/api_models.py — expose max_doc_chars in the writable config schema (keeps GLOBAL_SCALAR_KEYS == _KB_CONFIG_WRITABLE_KEYS).
• openkb/agent/compiler.py — new _maybe_truncate_doc(content, max_doc_chars, doc_name) helper; compile_short_doc applies it after reading the source.
• docs — config.yaml.example, README.md, examples/configuration/README.md.
• tests — test_compiler.py: 4 unit tests on the helper + 2 integration tests proving compile_short_doc truncates an oversized doc (marker present, tail gone) and passes through a doc within budget.

Verification

• RED→GREEN proven: with the source fix reverted, _maybe_truncate_doc doesn't exist and the new tests fail at import (RED); with the fix, all 6 pass (GREEN).
• pytest tests/test_compiler.py tests/test_config.py tests/test_api.py tests/test_api_documents.py tests/test_api_watch.py tests/test_indexer.py tests/test_file_size.py → 422 passed.
• ruff check clean; ruff format clean.

No new dependencies. Diff: ~37 source lines + docs + tests.

[hudsonwa]

What

Fixes#73 (partially, the "graceful degradation" minimum it explicitly allows).

The short-doc compile path sends the entire markdown source file as a single LLM message (compile_short_doc reads the whole file and injects it into the summary prompt). An oversized markdown doc therefore overflows the model context and either fails or has its tail silently dropped — which is exactly what breaks ingesting large markdown files with local/private models.

This adds a configurable hard cap, max_doc_chars (default 500k chars ≈ 125k tokens — a pure safety ceiling that never affects normal docs), that truncates the payload and appends an explicit marker so the model knows the tail was cut and the user can see it happened. Local-model users lower max_doc_chars to fit their context window.

Why this scope

Issue #73 proposes a full heading-aware chunking + hierarchical-synthesis feature (medium/large). This PR ships the small, safe, "at minimum" increment the issue itself lists first: graceful degradation instead of a hard failure. It's a strict safety net; structured chunking can build on top later.

Changes

• openkb/config.py — add max_doc_chars to DEFAULT_CONFIG and GLOBAL_SCALAR_KEYS (layers global → KB like pageindex_threshold).
• openkb/api_models.py — expose max_doc_chars in the writable config schema (keeps GLOBAL_SCALAR_KEYS == _KB_CONFIG_WRITABLE_KEYS).
• openkb/agent/compiler.py — new _maybe_truncate_doc(content, max_doc_chars, doc_name) helper; compile_short_doc applies it after reading the source.
• docs — config.yaml.example, README.md, examples/configuration/README.md.
• tests — test_compiler.py: 4 unit tests on the helper + 2 integration tests proving compile_short_doc truncates an oversized doc (marker present, tail gone) and passes through a doc within budget.

Verification

• RED→GREEN proven: with the source fix reverted, _maybe_truncate_doc doesn't exist and the new tests fail at import (RED); with the fix, all 6 pass (GREEN).
• pytest tests/test_compiler.py tests/test_config.py tests/test_api.py tests/test_api_documents.py tests/test_api_watch.py tests/test_indexer.py tests/test_file_size.py → 422 passed.
• ruff check clean; ruff format clean.

No new dependencies. Diff: ~37 source lines + docs + tests.

[hudsonwa]

What

Fixes#73 (partially, the "graceful degradation" minimum it explicitly allows).

The short-doc compile path sends the entire markdown source file as a single LLM message (compile_short_doc reads the whole file and injects it into the summary prompt). An oversized markdown doc therefore overflows the model context and either fails or has its tail silently dropped — which is exactly what breaks ingesting large markdown files with local/private models.

This adds a configurable hard cap, max_doc_chars (default 500k chars ≈ 125k tokens — a pure safety ceiling that never affects normal docs), that truncates the payload and appends an explicit marker so the model knows the tail was cut and the user can see it happened. Local-model users lower max_doc_chars to fit their context window.

Why this scope

Issue #73 proposes a full heading-aware chunking + hierarchical-synthesis feature (medium/large). This PR ships the small, safe, "at minimum" increment the issue itself lists first: graceful degradation instead of a hard failure. It's a strict safety net; structured chunking can build on top later.

Changes

• openkb/config.py — add max_doc_chars to DEFAULT_CONFIG and GLOBAL_SCALAR_KEYS (layers global → KB like pageindex_threshold).
• openkb/api_models.py — expose max_doc_chars in the writable config schema (keeps GLOBAL_SCALAR_KEYS == _KB_CONFIG_WRITABLE_KEYS).
• openkb/agent/compiler.py — new _maybe_truncate_doc(content, max_doc_chars, doc_name) helper; compile_short_doc applies it after reading the source.
• docs — config.yaml.example, README.md, examples/configuration/README.md.
• tests — test_compiler.py: 4 unit tests on the helper + 2 integration tests proving compile_short_doc truncates an oversized doc (marker present, tail gone) and passes through a doc within budget.

Verification

• RED→GREEN proven: with the source fix reverted, _maybe_truncate_doc doesn't exist and the new tests fail at import (RED); with the fix, all 6 pass (GREEN).
• pytest tests/test_compiler.py tests/test_config.py tests/test_api.py tests/test_api_documents.py tests/test_api_watch.py tests/test_indexer.py tests/test_file_size.py → 422 passed.
• ruff check clean; ruff format clean.

No new dependencies. Diff: ~37 source lines + docs + tests.