Skip to content

emrg: memory index governance — cap index size and enforce hygiene - #941

Merged
argszero merged 1 commit into
masterfrom
feature/memory-index-governance
Aug 23, 2026
Merged

emrg: memory index governance — cap index size and enforce hygiene#941
argszero merged 1 commit into
masterfrom
feature/memory-index-governance

Conversation

@argszero

Copy link
Copy Markdown
Owner

Summary

Addresses host rant 2026-08-23T08:04:26 (project emrg): MEMORY.md must stay a pure index and converge to ≤50 entries so the index can never bloat request bodies (413).

Evidence: a long-lived evolution session's memory/MEMORY.md reached 280,943 bytes / 752 lines (731 per-cycle index rows, avg 384 chars each). Root cause: _consolidate_session_memories only fires on client disconnect, so long-lived daemon sessions never trigger consolidation, and reflection was append-only.

Changes

  1. emrg/memory.py
    • Write-time title truncation (≤512 chars) in MemoryIndex.add_entry (primary guard)
    • Render-time fallback in to_markdown for legacy dirty index data (filename kept reachable)
    • SessionMemoryStore soft count/size guard — logs a warning when index > 100 entries or > 50 KB (warn only, never auto-deletes; consolidation stays LLM-driven)
  2. emrg/server/daemon.py
    • _maybe_reflect_memory: when the session index crosses thresholds, injects a Memory Hygiene section steering the reflection LLM toward consolidation (merge/supersede instead of append)
    • _consolidate_session_memories: adds an explicit index cap — index must converge to ≤50 entries (detail .md files remain the source of truth and may exceed 50)
  3. Promptssystem.j2 (Memory Hygiene rules), evolution_prompt.md (Step 6 index-line norm), open_source_prompt.md (Recording memory-hygiene note)
  4. Tests — +4: write-time truncation, render-time truncation fallback, soft-guard silent-below-threshold, soft-guard warns-at-threshold (1006 passed + 1 skipped)

Local remediation (outside this repo, host ~/scm copy, .emrg/ gitignored): the 280 KB aitokenpool session MEMORY.md was trimmed to 23,897 bytes / 50 rows; its 597 cycle-*.md detail files are untouched (source of truth).

Verification

  • uv run pytest tests/ → 1006 passed, 1 skipped
  • uv run python -c "from emrg.client.app import run_client" → OK
  • uv run python -m emrg --help → OK

Rant 2026-08-23T08:04:26: MEMORY.md must stay a pure index and converge
to <=50 entries, preventing index bloat from overflowing request bodies
(413). Evidence: a session MEMORY.md reached 280KB / 752 lines because
consolidation only fired on client disconnect.
Changes:
- memory.py: write-time title truncation (<=512 chars) in add_entry;
render-time fallback in to_markdown for legacy dirty data (filename
kept reachable); SessionMemoryStore soft count/size guard (warn only,
never auto-delete)
- daemon.py: _maybe_reflect_memory steers reflection toward
consolidation when index crosses thresholds; _consolidate_session_memories
now caps the index at <=50 entries
- system.j2 / evolution_prompt.md / open_source_prompt.md: Memory Hygiene
rules (pure index, title cap, update-in-place, merge instead of append)
- tests: +4 (truncation x2, soft guard x2); Agent.md count 1003->1007
Local remediation (outside repo): aitokenpool session MEMORY.md trimmed
280KB -> 24KB (752 -> 68 lines, 50 rows); 597 detail files untouched.

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle (self-review)

Verified locally:

  • uv run pytest tests/ → 1006 passed, 1 skipped (+4 new memory-governance tests)
  • uv run python -c "from emrg.client.app import run_client" → OK
  • uv run python -m emrg --help → OK

Changes are scoped: write-time title truncation + render-time fallback + soft guard in memory.py; hygiene steering in daemon.py reflection/consolidation prompts; Memory Hygiene rules in system.j2 and both prompt templates. Consolidation remains LLM-driven — nothing auto-deletes.

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle (independent review)

Fresh-eyes review of the full diff:

  • _truncate_index_title uses a None sentinel default (avoids the NameError), write-time truncation is primary, render-time fallback covers legacy dirty data — filename (detail file) preserved in both paths.
  • Soft guard (SessionMemoryStore._warn_index_thresholds) is warn-only and non-destructive; consolidation stays LLM-driven via the updated reflection/consolidation prompts (index cap ≤50 entries, pure-index rule).
  • Hygiene note in _reflect is computed on-demand and only appended when thresholds are crossed — no overhead in the common path.
  • +4 tests cover both truncation paths and both guard states (below-threshold silent / at-threshold warns + non-destructive check); Agent.md count 1003→1007 matches.
  • CI: test + test-windows both pass (run 32607536232).

No issues found.

@argszeroargszero left a comment

Copy link
Copy Markdown
OwnerAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ LGTM — cycle (3rd consecutive approval)

Head 8de48f6 unchanged since the previous two reviews; CI test + test-windows both pass (run 32607536232). Full diff was reviewed fresh in the previous cycle: write-time + render-time title truncation with filename preservation, non-destructive soft guards, LLM-driven consolidation steering, +4 tests matching the Agent.md count bump (1003→1007). No issues — approving to merge.

@argszero
argszero merged commit 7dee40a into masterAug 23, 2026
2 checks passed
@argszero
argszero deleted the feature/memory-index-governance branch August 23, 2026 00:35
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@argszero