Design a persisted user-edit feedback loop for diagram quality

[anandpant]

Goal

Design the smallest durable feedback loop that turns a generated diagram and a user's approved edit into useful versioned evidence for generation and evaluation improvements.

Current v2 foundation

• The canonical runtime persists generated artifacts and provenance.
• Studio has stable project and diagram records plus an embedded editing route.
• Editor changes are currently downloaded locally rather than persisted as a new diagram version.
• There is no approval state, durable edit delta, or path from approved edits into maintained eval evidence.

Questions to answer

Product workflow

• What action records satisfaction: save, explicit approval, publish, or a version label?
• Is the first useful unit a diagram version, an approved artifact pair, or an eval example?
• Which metadata must remain visible to users and operators?

Version and provenance model

• How does an edited artifact retain its generated source artifact, prompt, typed IR, model/settings, and renderer version?
• Should the durable record store full immutable before/after artifacts, a derived semantic diff, or both?
• How are concurrent or branched edits represented without introducing collaborative event sourcing prematurely?

Learning loop

• How are approved pairs promoted into maintained scenarios or prompt examples?
• Which changes can be aggregated safely into renderer/layout improvements?
• What regression gates prevent one user's edit style from degrading general output?

Privacy and retention

• What diagram/prompt content may be retained for product improvement?
• What explicit consent, deletion, and access controls are required before reuse?

Deliverables

• A short design document describing the end-to-end workflow.
• A storage-neutral version/provenance model that fits the current Cloudflare artifact boundary.
• A recommendation for the first implementation slice.
• Explicit privacy, retention, and evaluation safeguards.

Recommended first slice

1. Persist an editor save as a derived immutable artifact.
2. Record source artifact, typed IR, generation metadata, and content hash.
3. Add explicit user approval to a version.
4. Export approved before/after pairs into a reviewable eval-candidate format.
5. Keep automated optimization and community templates out of the critical path.

Out of scope

• Convex-specific tables or assumptions.
• Automated model or prompt optimization.
• A community template marketplace.
• Full collaborative editing/event sourcing.

Current touchpoints

• apps/studio/src/lib/studio-projects-contract.ts
• apps/studio/src/lib/studio-projects.server.ts
• apps/studio/src/routes/diagrams_/$diagramId/edit.tsx
• packages/diagram-agent/
• packages/diagram-scenarios/
• docs/mcp-first-generation.md

Rewritten after the Sketchi v2 replacement in #217. The feedback-loop question remains useful; the Convex and marketplace assumptions do not.

[anandpant]

Triage 2026-07-23: the kernel — turning approved human edits into maintained eval evidence — is still unbuilt and worth keeping, but this issue needs a rewrite before implementation. Its factual foundation is now wrong: all apps/studio/* touchpoints are gone (now apps/playground + packages/studio/projects), and the claim that editor changes are only downloaded locally is false — the CLI store persists pulled edits atomically with full prior-revision snapshots and an authority model (apps/cli/src/pull.ts, apps/cli/src/storage.ts, landed via #250/#272). The genuinely open pieces are approval state and exporting approved before/after pairs into eval candidates, reframed around the CLI store and apps/eval-harness rather than Studio. Rescoping needed before this is actionable.

[anandpant]

Closing this ancient umbrella in favor of current, codebase-aligned slices: #282 covers durable Studio revisions and approval; #283 covers consent-aware export of approved before/after pairs into the eval workflow. The replacements account for the current apps/playground, packages/studio/projects, Code Mode artifact provenance, and CLI revision-store boundaries.