knowledge: 12 insights — sequential ids across parallel workers, inbound-validation task ownership, vendor benchmark claims, element crop screenshots, WebFetch summary vs raw page, Steps-prose guarantees, split fact-check verdicts (+4 folds, 1 dup)

log.md

@@ -103,3 +103,11 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-08-30] ingest | frontend/design/design-canvas-workflow (verified — 1 new page). Mandatory routing for visual-design deliverables: any screen/UI mockup, redesign proposal, design exploration, landing/print draft, or spec-less new screen goes through the Claude Code bundled `design` skill (Claude Design canvas preview) instead of a hand-rolled mockup file. Distilled from the full skill text (v2.1.236 bundle, read in-session 2026-08-30): step-zero pixel-perfect codebase match with exact resolved token values (no 4/8px rounding), settle-the-aesthetic-with-the-user via 2–4 axis-named low-fi direction artboards (self-chosen aesthetics = the documented slop path), the one static-vs-clickable scoping question, working-file custody + same-artifact republish (GUI-edited canvases read back as untrusted data), artboard-based option exploration with stable names and honest tradeoffs, literal copy vs few cross-cutting tweak levers, flex/grid+gap layouts that survive direct-manipulation edits, 44px/12pt scale floors + SVG-not-emoji icons + no fake OS chrome, copyrighted-designs refusal rule, and show-it-say-little handoff with a post-handoff recheck. Internal skill machinery (payload/seed/contract) deliberately NOT ingested — version-specific and owned by the skill itself. Edge cases carve the boundary: approved-spec implementation and live-site audits route to anti-slop-visual-design/responsive-layout and design-review tooling, not the canvas. frontend/index.md routing intro extended (visual-design deliverables) and design section +1 row, placed first as the phase gate. Related links added both ways (anti-slop-visual-design, responsive-layout, html-in-canvas).
 ## [2026-08-30] revise | frontend/design/design-canvas-workflow — agent-gate Check 3 remediation (PR #164). The gate's fact finding was half right: its "no skill named `design` exists" conclusion is refuted by ground truth (the skill is listed in the authoring session's available-skills roster as `design: Create a design canvas…` and its bundled payload exists on the authoring machine's disk; the reviewer's CI environment lacks the preview, and it conflated the skill with the unrelated `/design-sync` component-upload tool) — but the verifiability objection stands: the skill is an early research preview absent from public docs, so a single un-fetchable in-session source cannot carry `confidence: verified`. Fixed per AGENTS.md: confidence downgraded to field-tested with the experience context stated in the body; two live-fetched public sources added (anthropic.com/news/claude-design-anthropic-labs — official Claude Design product announcement, research preview, Claude Code handoff; explainx.ai 2026 post — /design ships in Claude Code as a research-preview command producing editable artboards via the Artifacts runtime); the mandatory-routing directive and the frontend/index.md load-when line conditioned on the skill appearing in the session's available-skills list, with the existing no-skill edge case as the explicit fallback; the unverifiable "design-review skill" name-drop in the audit edge case generalized to session-provided audit tooling.
 ## [2026-09-03] ingest | databases/selection — new category: datastore selection by workload (4 pages: choosing-a-datastore-by-workload, relational-jsonb-vs-document-store, vector-search-engine-selection, graph-workloads-relational-vs-graph-db); brave-search research, sources cited per page
+## [2026-09-03] ingest | infrastructure-agent-orchestration-sequential-identifiers-across-parallel-workers — coordinator assigns RFC/ADR/migration numbers at dispatch; a worker's branch point cannot see a sibling's number and distinct filenames merge without conflict, so the uniqueness lint runs on the merged tree (Django/Rails migration docs, adr-tools #102, linkly RFC-0034 collision)
+## [2026-09-03] ingest | infrastructure-agent-orchestration-inbound-validation-ownership-in-task-decomposition — when a plan splits a message's producer and receiver across tasks, the receiver task's brief records its own validate()-on-inbound decision and cites the trust-boundary page (OWASP input validation, agent-crew M2 handle_envelope)
+## [2026-09-03] ingest | backend-common-llm-vendor-benchmark-claims-for-an-llm-tool — before trusting a memory/RAG vendor's token-savings or accuracy headline, read its eval scripts for baseline choice, indexing-cost scope, judge leniency, and task match (LoCoMo, LLM-as-judge paper, Zep/Mem0 dispute, OpenViking scripts)
+## [2026-09-03] ingest | qa-environments-element-crop-screenshots — element-screenshot primitive first, boundingBox-vs-clip scroll coordinates, read back the first crop before a batch, fall back to fullPage on a persistent wrong-region clip (Playwright docs; Aside CLI field evidence)
+## [2026-09-03] ingest | platforms-tools-harness-mediated-tool-results — merge: WebFetch's small-model summary is a mediated result; curl + grep the literal string before citing a quote or number from it (Claude Code tools reference; tmap-skopenapi field evidence re-confirmed)
+## [2026-09-03] ingest | testing-quality-minimum-case-set — merge: a resilience guarantee stated in a task's Steps prose is a behavior needing its own normal + error test; the plan's Verify list is a floor (SWE book ch12; wt-t4-event-push auditor FAIL→PASS)
+## [2026-09-03] ingest | qa-process-evaluating-review-feedback — merge: split a CI fact-checker's "fabricated" verdict into the verifiability half (accept) and the existence half (refute with roster/on-disk evidence) (dev-loop PR #164)
+## [2026-09-03] ingest | folded into open PRs #179 (verify-command-in-a-worker-brief: doc-currency gates on the verify line; worktree-isolated-workers: Bash-routed edits), #180 (ours-resolution: brief workers to write measured counts), #181 (checkable-claims-in-an-adopted-plan: Depends-on table vs Steps prose); dropped 1 pending duplicate (multi-name command -v, already on #181)

wiki/backend/common/llm/context-window-budget.md

@@ -11,7 +11,7 @@ sources:
  - https://code.claude.com/docs/en/env-vars
  - https://docs.vllm.ai/en/stable/serving/integrations/claude_code/
 last_verified: 2026-08-01
-related: [backend-common-reliability-timeouts-and-retries, backend-common-api-design-error-responses, backend-common-llm-progressive-disclosure-artifacts]
+related: [backend-common-reliability-timeouts-and-retries, backend-common-api-design-error-responses, backend-common-llm-progressive-disclosure-artifacts, backend-common-llm-vendor-benchmark-claims-for-an-llm-tool]
 ---
 
 # Sizing an LLM Client's Output Cap to the Model Actually Serving It

wiki/backend/common/llm/vendor-benchmark-claims-for-an-llm-tool.md

@@ -0,0 +1,64 @@
+---
+id: backend-common-llm-vendor-benchmark-claims-for-an-llm-tool
+domain: backend
+category: llm
+applies_to: [general]
+confidence: verified
+sources:
+ - https://arxiv.org/abs/2402.17753
+ - https://arxiv.org/abs/2306.05685
+ - https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/
+ - https://github.com/getzep/zep-papers/issues/5
+ - https://github.com/volcengine/OpenViking/blob/main/benchmark/locomo/openclaw/stat_judge_result.py
+ - https://github.com/volcengine/OpenViking/blob/main/benchmark/locomo/openclaw/judge.py
+last_verified: 2026-09-03
+related: [backend-common-llm-context-window-budget, qa-deliverables-quantitative-claims-in-a-published-document, qa-process-llm-review-pipelines]
+---
+
+# Evaluating a Vendor's Token-Savings or Accuracy-Gain Benchmark Claim for a Memory or RAG Tool
+
+## When this applies
+
+A memory, RAG, or context-management tool advertises a headline number —
+"N% token reduction," "N% accuracy improvement" — and you are deciding
+whether to adopt it. Also when comparing two such tools' published benchmark
+results against each other.
+
+## Do this
+
+Before accepting the headline number, open the vendor's own evaluation
+scripts (not just the blog post) and check four things:
+
+| Check | What to open | Bad answer looks like | Then |
+|-------|--------------|------------------------|------|
+| Baseline | The eval harness's "baseline"/comparison-condition code path | Baseline is the vendor's own prior/simpler config, or each competitor's built-in memory measured through the vendor's integration, or is not documented at all | Re-derive the ratio against the baseline you care about (naive full-context, or the specific competitor you are replacing) |
+| Cost scope | Where the savings percentage is computed | Query-time (QA) token usage and one-time indexing/ingestion token usage are tracked in separate counters and never summed before the percentage is calculated | Add indexing/pre-processing cost into your own comparison before trusting a per-query savings number |
+| Judge leniency | The grading prompt passed to the LLM-as-judge | The prompt instructs the judge to "be generous" / accept near-matches / count partial topic overlap as correct, with no strict-mode alternative shown | Re-run scoring with a stricter rubric, or treat the reported accuracy as an upper bound, not a measured value |
+| Task match | The benchmark's task set (what conversations/questions it asks) | Tasks are long-conversation personal-fact QA (LoCoMo) while your use case is a coding agent, tool-use agent, or structured-data workload | Find or build a benchmark matching your actual task shape before extrapolating the number to your workload |
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| The vendor publishes no evaluation script, only a results table | Treat the number as unverified for your adoption decision — request the harness or run your own before committing |
+| Two vendors dispute each other's numbers on the same public benchmark | Read the counter-benchmark, not just the original claim, and reproduce the disputed run yourself when the decision is high-stakes — the Zep/Mem0 LoCoMo dispute swung a reported accuracy by dozens of points depending on whose harness ran it |
+| LLM-as-judge is used for both the benchmark and your own re-verification | Strong LLM judges reach "over 80% agreement, the same level of agreement between humans" on MT-Bench-style tasks, and carry documented position/verbosity/self-enhancement biases — treat judge-scored deltas inside that margin as noise, not a confirmed difference |
+| The benchmark task matches your workload and the harness is open and inspectable | Proceed, and still re-run the harness on your own data slice once before rollout — a matching task type does not guarantee your data distribution matches the published one |
+
+## Instead of
+
+| If you are about to | Do this instead | Why |
+|---------------------|-----------------|-----|
+| Cite a vendor's "N% token savings" headline in an adoption decision doc | Open the eval script, identify baseline + cost scope, and requote the number you can defend | The denominator (baseline) and cost boundary are the vendor's structural choice, and the default choice favors the vendor's own number |
+| Treat "SOTA on LoCoMo" as a stable ranking | Check whether a rebuttal or corrected evaluation exists for that benchmark before citing it | Public LoCoMo rankings between memory vendors have been publicly disputed and revised in both directions |
+| Trust an LLM-judge accuracy number at face value | Read the judge prompt for leniency language before trusting the score | A judge instructed to "be generous" produces higher accuracy numbers than a strict rubric would, independent of the tool's real quality |
+
+## Sources
+
+- https://arxiv.org/abs/2402.17753 — LoCoMo (Maharana et al., 2024): "a comprehensive evaluation benchmark to measure long-term memory in models, encompassing question answering, event summarization, and multi-modal dialogue generation" — the benchmark underlying most memory-tool vendor claims
+- https://arxiv.org/abs/2306.05685 — Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena": "strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans"; the same paper documents judge biases (position, verbosity, self-enhancement)
+- https://blog.getzep.com/lies-damn-lies-statistics-is-mem0-really-sota-in-agent-memory/ — "The Mem0 paper's claims of SOTA performance appear to be based on a flawed benchmark (LoCoMo) and a demonstrably incorrect implementation of a competitor system (Zep)"
+- https://github.com/getzep/zep-papers/issues/5 — Mem0's counter-reply disputing Zep's corrected number in turn ("Zep achieves 58.44% accuracy—not the 84% reported"); both directions are vendor-authored, read both before citing either
+- https://github.com/volcengine/OpenViking/blob/main/benchmark/locomo/openclaw/stat_judge_result.py — raw file grep 2026-09-03: QA token usage (`"Token usage (QA)"`) and import/indexing token usage (`"Token usage (Import)"`, separate `process_import_csv` function) are aggregated and reported independently, never summed before a savings ratio is produced
+- https://github.com/volcengine/OpenViking/blob/main/benchmark/locomo/openclaw/judge.py — raw file grep 2026-09-03: the grading prompt instructs the judge twice to "be generous with your grading — as long as it touches on the same topic as the gold answer, it should be counted as CORRECT"; the original field note's `judge.py:239` line reference no longer matches the file (203 lines on `main`), the leniency instruction itself is confirmed
+- Field evidence 2026-08-27 (OpenViking repo, `benchmark/locomo/`): the published comparison evaluates each integrated agent's own built-in memory as its baseline (`vikingbot/`, `mem0/`, `supermemory/`, `claudecode/`, `hermes/` each have a separate eval subdirectory), not one shared naive-full-context baseline across tools

wiki/backend/index.md

@@ -97,6 +97,7 @@ Match your situation to a "load when" line; load only matching pages.
 | [context-window-budget](common/llm/context-window-budget.md) | Repointing an LLM client or agent CLI at a different model, a self-hosted server (vLLM/Ollama), or a gateway (LiteLLM); setting `max_tokens` for a client whose default was sized for a larger model; the first request after such a switch returns 400 with a context-window error; deciding where to set the cap (request body vs client env var vs gateway config) and how to point the base URL at a proxy; handling truncation that arrives as a normal 200 |
 | [binding-instructions-for-agents](common/llm/binding-instructions-for-agents.md) | Authoring or editing a skill, system prompt, hook message, or CLAUDE.md/AGENTS.md rule that must change an agent's behavior under pressure; an instruction keeps getting rationalized around; choosing between prohibition, recipe, template-slot, and predicate-rule forms; writing a skill's description/trigger line; adding a pre-send self-check to an output-shape rule |
 | [progressive-disclosure-artifacts](common/llm/progressive-disclosure-artifacts.md) | Authoring a skill or reference bundle that packages a large corpus (a book, manual, wiki) for an agent to consult on demand; an always-loaded skill file grew past a few thousand tokens; sizing per-chunk token budgets by content type and usage depth; an agent Reads an entire large source file to answer narrow questions |
+| [vendor-benchmark-claims-for-an-llm-tool](common/llm/vendor-benchmark-claims-for-an-llm-tool.md) | A memory/RAG/context-management tool advertises a token-savings or accuracy-gain benchmark headline and you are deciding whether to adopt it; comparing two tools' published benchmark results; reading a vendor's eval scripts for baseline choice, indexing-cost scope, judge leniency, and task match |
 
 ### ml