emrg: memory index governance — cap index size and enforce hygiene - #941
Conversation
Rant 2026-08-23T08:04:26: MEMORY.md must stay a pure index and converge to <=50 entries, preventing index bloat from overflowing request bodies (413). Evidence: a session MEMORY.md reached 280KB / 752 lines because consolidation only fired on client disconnect. Changes: - memory.py: write-time title truncation (<=512 chars) in add_entry; render-time fallback in to_markdown for legacy dirty data (filename kept reachable); SessionMemoryStore soft count/size guard (warn only, never auto-delete) - daemon.py: _maybe_reflect_memory steers reflection toward consolidation when index crosses thresholds; _consolidate_session_memories now caps the index at <=50 entries - system.j2 / evolution_prompt.md / open_source_prompt.md: Memory Hygiene rules (pure index, title cap, update-in-place, merge instead of append) - tests: +4 (truncation x2, soft guard x2); Agent.md count 1003->1007 Local remediation (outside repo): aitokenpool session MEMORY.md trimmed 280KB -> 24KB (752 -> 68 lines, 50 rows); 597 detail files untouched.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle (self-review)
Verified locally:
uv run pytest tests/→ 1006 passed, 1 skipped (+4 new memory-governance tests)uv run python -c "from emrg.client.app import run_client"→ OKuv run python -m emrg --help→ OK
Changes are scoped: write-time title truncation + render-time fallback + soft guard in memory.py; hygiene steering in daemon.py reflection/consolidation prompts; Memory Hygiene rules in system.j2 and both prompt templates. Consolidation remains LLM-driven — nothing auto-deletes.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle (independent review)
Fresh-eyes review of the full diff:
_truncate_index_titleuses aNonesentinel default (avoids the NameError), write-time truncation is primary, render-time fallback covers legacy dirty data — filename (detail file) preserved in both paths.- Soft guard (
SessionMemoryStore._warn_index_thresholds) is warn-only and non-destructive; consolidation stays LLM-driven via the updated reflection/consolidation prompts (index cap ≤50 entries, pure-index rule). - Hygiene note in
_reflectis computed on-demand and only appended when thresholds are crossed — no overhead in the common path. - +4 tests cover both truncation paths and both guard states (below-threshold silent / at-threshold warns + non-destructive check); Agent.md count 1003→1007 matches.
- CI: test + test-windows both pass (run 32607536232).
No issues found.
argszero
left a comment
There was a problem hiding this comment.
✅ LGTM — cycle (3rd consecutive approval)
Head 8de48f6 unchanged since the previous two reviews; CI test + test-windows both pass (run 32607536232). Full diff was reviewed fresh in the previous cycle: write-time + render-time title truncation with filename preservation, non-destructive soft guards, LLM-driven consolidation steering, +4 tests matching the Agent.md count bump (1003→1007). No issues — approving to merge.
Uh oh!
There was an error while loading. Please reload this page.
Summary
Addresses host rant
2026-08-23T08:04:26(projectemrg): MEMORY.md must stay a pure index and converge to ≤50 entries so the index can never bloat request bodies (413).Evidence: a long-lived evolution session's
memory/MEMORY.mdreached 280,943 bytes / 752 lines (731 per-cycle index rows, avg 384 chars each). Root cause:_consolidate_session_memoriesonly fires on client disconnect, so long-lived daemon sessions never trigger consolidation, and reflection was append-only.Changes
emrg/memory.pyMemoryIndex.add_entry(primary guard)to_markdownfor legacy dirty index data (filename kept reachable)SessionMemoryStoresoft count/size guard — logs a warning when index > 100 entries or > 50 KB (warn only, never auto-deletes; consolidation stays LLM-driven)emrg/server/daemon.py_maybe_reflect_memory: when the session index crosses thresholds, injects a Memory Hygiene section steering the reflection LLM toward consolidation (merge/supersede instead of append)_consolidate_session_memories: adds an explicit index cap — index must converge to ≤50 entries (detail.mdfiles remain the source of truth and may exceed 50)system.j2(Memory Hygiene rules),evolution_prompt.md(Step 6 index-line norm),open_source_prompt.md(Recording memory-hygiene note)Local remediation (outside this repo, host
~/scmcopy,.emrg/gitignored): the 280 KB aitokenpool sessionMEMORY.mdwas trimmed to 23,897 bytes / 50 rows; its 597cycle-*.mddetail files are untouched (source of truth).Verification
uv run pytest tests/→ 1006 passed, 1 skippeduv run python -c "from emrg.client.app import run_client"→ OKuv run python -m emrg --help→ OK