feat(dashboard): reduce poll/render cost with leaner usage queries and UI refinements

[chriswritescode-dev]

Summary

Reduces dashboard poll/render cost with leaner server payload assembly and more efficient loop-session usage queries, plus UI refinements in the dashboard app. Refactors loop token-usage handling and loop status formatting, and adds aggregated helpers in the plans/section-plans repos.

Key changes

• src/dashboard/data.ts + src/dashboard/server.ts — leaner dashboard payload assembly on the poll path.
• src/dashboard/app/app.ts, components.ts, helpers.ts, render.ts — UI rendering refinements for the poll/render path.
• src/dashboard/app-bundle.ts — regenerated bundle.
• src/loop/token-usage.ts — token-usage accounting refactor.
• src/storage/repos/loop-session-usage-repo.ts, plans-repo.ts, section-plans-repo.ts — leaner usage queries and aggregated repo helpers.
• src/tools/loop.ts + src/utils/loop-format.ts — loop status tool and formatting updates.
• README.md — notes updated.
• Tests across app-dom, app-helpers, data, server, loop-format, loop-session-usage, loop-status-tool, and token-usage.

Files changed

22 files changed, +2142/-204

Summary by CodeRabbit

• New Features

  • Dashboard data now loads progressively by project and loop, improving responsiveness and reducing unnecessary detail.
  • Large loop, plan, findings, and picker lists are capped with “Show all” controls.
  • Usage reporting now includes token totals and cost breakdowns by model and role.
  • Dashboard navigation and tab content preserve state more reliably during refreshes.

• Bug Fixes

  • Active-loop checks now distinguish running loops from completed loops.
  • Configured data directories are consistently respected for database access.

• Documentation

  • Expanded API, dashboard, usage tracking, and milestone-limit documentation.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 227b0045-9711-448a-99b8-ae84581207b9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/dashboard-poll-render-cost

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

test/dashboard/data.test.ts (1)

748-798: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider adding a project-only scope case ({ projectId: 'p1', loopName: null }).

That's the repo-list view the dashboard actually requests when no loop is open, and no test pins its payload shape (findings/usage present, plan/sections/amendments absent).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/dashboard/data.test.ts` around lines 748 - 798, Extend the dashboard
data test around collectDashboardData to cover project-only scope with {
projectId: 'p1', loopName: null }. Assert that all loops in project p1 include
findings and usage, while plan, sections, and amendments remain absent; also
verify out-of-scope project data retains the expected empty findings, null
usage, and populated bugCount behavior.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/dashboard/app/app.ts`:
- Around line 88-95: Bound the fetch started in the load polling function with
an AbortSignal.timeout or equivalent manual AbortController so hung /api/data
requests are aborted after a finite duration. Pass the signal to fetch while
preserving the existing inFlight cleanup and coalescing behavior, allowing
finally to clear the request and the next interval to retry.
In `@src/storage/database.ts`:
- Line 94: Normalize dataDir at the configuration boundary before the mkdirSync
call, then reuse the normalized value for both directory creation and
resolveForgeDbPath. Update the surrounding database initialization flow so
whitespace-padded paths cannot produce different creation and connection
locations.
In `@test/utils/tui-client-stored-plan.test.ts`:
- Line 19: Update the resolveForgeDbPath mock to match production’s handling of
dataDir: trim whitespace and use the default '/tmp/forge-test-data-dir' path
when the value is empty or whitespace-only, while preserving the resolved
directory for non-empty input.
---
Nitpick comments:
In `@test/dashboard/data.test.ts`:
- Around line 748-798: Extend the dashboard data test around
collectDashboardData to cover project-only scope with { projectId: 'p1',
loopName: null }. Assert that all loops in project p1 include findings and
usage, while plan, sections, and amendments remain absent; also verify
out-of-scope project data retains the expected empty findings, null usage, and
populated bugCount behavior.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6db2d7bd-eba4-4a5e-9ca7-dca59614cf14

📥 Commits

Reviewing files that changed from the base of the PR and between aa3bc79 and a3dfe7e.

📒 Files selected for processing (61)

• AGENTS.md
• README.md
• docs/api/README.md
• docs/api/_media/loop-system.md
• docs/api/functions/createForgePlugin.md
• docs/api/functions/createParentSessionLookup.md
• docs/api/functions/createSessionDirectoryLookup.md
• docs/api/interfaces/CompactionConfig.md
• docs/api/interfaces/CreateParentSessionLookupOptions.md
• docs/api/interfaces/CreateSessionDirectoryLookupOptions.md
• docs/api/interfaces/PluginConfig.md
• docs/api/variables/VERSION.md
• docs/api/variables/default.md
• docs/loop-system.md
• scripts/dashboard.ts
• src/dashboard/app-bundle.ts
• src/dashboard/app/app.ts
• src/dashboard/app/components.ts
• src/dashboard/app/helpers.ts
• src/dashboard/data.ts
• src/dashboard/launch.ts
• src/dashboard/render.ts
• src/dashboard/server.ts
• src/hooks/plan-approval.ts
• src/index.ts
• src/loop/service.ts
• src/loop/token-usage.ts
• src/services/deterministic-decomposer.ts
• src/services/execution.ts
• src/storage/database.ts
• src/storage/index.ts
• src/storage/repos/loop-session-usage-repo.ts
• src/storage/repos/loop-transitions-repo.ts
• src/storage/repos/plans-repo.ts
• src/storage/repos/review-findings-repo.ts
• src/storage/repos/section-plans-repo.ts
• src/tools/loop.ts
• src/tools/plan-authoring.ts
• src/tui.tsx
• src/utils/format.ts
• src/utils/logger.ts
• src/utils/loop-format.ts
• src/utils/opencode-paths.ts
• src/utils/tui-client.ts
• src/utils/tui-loop-store.ts
• test/dashboard/app-dom.test.ts
• test/dashboard/app-helpers.test.ts
• test/dashboard/data.test.ts
• test/dashboard/server.test.ts
• test/hooks/plan-approval-dedupe.test.ts
• test/hooks/plan-approval-worktree-timing.test.ts
• test/loop-format.test.ts
• test/loop-session-usage-repo.test.ts
• test/loop-status-tool.test.ts
• test/loop-transitions-repo.test.ts
• test/loop/token-usage.test.ts
• test/plan-approval.test.ts
• test/review-findings-repo.test.ts
• test/tools/plan-authoring.test.ts
• test/utils/tui-client-stored-plan.test.ts
• test/worktree-log.test.ts

[coderabbitai]

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

A hung /api/data request permanently wedges polling for that scope.

inFlight.url is only cleared in the finally, and the coalescing guard on Line 90 drops every subsequent same-scope poll while it is set. If a fetch never settles (dropped connection, sleeping laptop resumed mid-request), the 5s interval becomes a no-op indefinitely — the UI keeps rendering stale data with no error. A scope change is the only escape.

An AbortSignal.timeout (or a manual controller) bounds this; the abort rejects, finally clears inFlight, and the next tick recovers.

🛡️ Proposed fix: bound the request

 const load = async () => {
const url = scopedDataUrl()
if (inFlight.url === url) return // coalesce overlapping same-scope polls
const gen = ++loadGen
inFlight.url = url
inFlight.gen = gen
try {
- const res = await fetch(url, { cache: 'no-store' })+ const res = await fetch(url, { cache: 'no-store', signal: AbortSignal.timeout(20000) })
if (gen !== loadGen) return

📝 Committable suggestion

‼️IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	constload=async()=>{
	consturl=scopedDataUrl()
	if(inFlight.url===url)return// coalesce overlapping same-scope polls
	constgen=++loadGen
	inFlight.url=url
	inFlight.gen=gen
	try{
	constres=awaitfetch('/api/data',{cache: 'no-store'})
	constres=awaitfetch(url,{cache: 'no-store'})
	constload=async()=>{
	consturl=scopedDataUrl()
	if(inFlight.url===url)return// coalesce overlapping same-scope polls
	constgen=++loadGen
	inFlight.url=url
	inFlight.gen=gen
	try{
	constres=awaitfetch(url,{cache: 'no-store',signal: AbortSignal.timeout(20000)})

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/dashboard/app/app.ts` around lines 88 - 95, Bound the fetch started in
the load polling function with an AbortSignal.timeout or equivalent manual
AbortController so hung /api/data requests are aborted after a finite duration.
Pass the signal to fetch while preserving the existing inFlight cleanup and
coalescing behavior, allowing finally to clear the request and the next interval
to retry.

[coderabbitai]

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Normalize dataDir before creating it.

resolveForgeDbPath trims the directory, but mkdirSync(dataDir) above still uses the raw value. A whitespace-padded configured path can therefore create one directory while opening the database under another path. Normalize the directory once at the configuration boundary and use that value for both operations.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/storage/database.ts` at line 94, Normalize dataDir at the configuration
boundary before the mkdirSync call, then reuse the normalized value for both
directory creation and resolveForgeDbPath. Update the surrounding database
initialization flow so whitespace-padded paths cannot produce different creation
and connection locations.