Review Stage: PlotPickle-native human-in-the-loop visual evaluation and agent feedback

## Developer Brief

### Purpose

Define a future PlotPickle-native **Review Stage**: a reusable human-in-the-loop visual evaluation and steering layer that lets PlotPickle agents show their work, evidence, uncertainty, alternatives, and proposed changes in a structured visual surface; lets a Human comment or decide directly against that work; routes the feedback back to the correct agent/context; and updates the review in place.

This issue captures the September 18, 2026 design discussion prompted by research into [Sideshow.sh](https://sideshow.sh/) and the open-source [modem-dev/sideshow](https://github.com/modem-dev/sideshow) project.

This is a **documented future capability**, not the next implementation priority. Do not begin implementation merely because this issue exists.

---

## Product decision

PlotPickle will **build its own native equivalent** rather than making Sideshow a permanent product dependency or embedding Sideshow as the long-term PlotPickle experience.

Sideshow and related Modem patterns are architectural/product references.

The PlotPickle implementation must use PlotPickle's own:

- project/canonical story authorities;
- Agent system and provider-routing boundaries;
- authentication and authorization;
- Story Map and 4 Act / 12 Sequence / 24 Block / 96 Mini-Block structure;
- Story-to-Screen workflow;
- DraftLens / Reader Simulation evidence;
- UAT Guide and developer validation workflow;
- Surface Grammar / current visual standards;
- provenance, approval and candidate-state rules;
- local/cloud capability boundaries.

The core idea being adopted is the **feedback loop**, not Sideshow's identity model, storage model, branding, or product shell.

---

## Reference pattern

Sideshow's useful pattern is:

```text
Agent works
  ↓
Agent publishes a visual surface
  ↓
Human sees the actual work/evidence
  ↓
Human comments, chooses, accepts, rejects, or asks for revision
  ↓
Feedback is delivered back to the agent
  ↓
Agent revises or replies
  ↓
Surface updates
  ↓
Loop continues until the Human is satisfied
```

The important shift is from:

```text
agent explains work in a long chat
```

to:

```text
agent shows the work + evidence + decisions in context
```

The PlotPickle Review Stage should preserve that principle.

---

## Why this matters to PlotPickle

PlotPickle is becoming increasingly visual, agentic, evidence-driven, and multi-stage.

A conventional static report is too passive.

The Review Stage should become a living evaluation surface that can answer two parallel questions using the same underlying architecture:

### Creator-facing

> What is happening to my story?

### Developer-facing

> What is happening to PlotPickle?

The same interaction model should work for both without merging their permissions, evidence sources, or domain-specific UI.

---

# A. Creator / Story Review Stage

After a meaningful story unit or workflow stage, PlotPickle should be able to assemble a visual review surface rather than returning only another large AI response.

A creator-facing Review Stage may show, when relevant:

- current story address: Act / Sequence / Block / Mini-Block / Scene / Beat / Shot;
- source screenplay evidence;
- story progress;
- characters currently active;
- Character Truth versus audience-visible screenplay evidence;
- character-state changes;
- locations;
- props / held objects / continuity-sensitive objects;
- setup / payoff state;
- reveals and concealment constraints;
- emotional movement;
- continuity findings;
- structural findings;
- Story Map changes;
- Storyboard candidates;
- Previs candidates;
- Reader Simulation / DraftLens findings;
- provenance of generated or inferred material;
- what agents changed;
- what agents are uncertain about;
- stale/downstream dependencies affected by a change;
- accepted versus candidate material.

The Review Stage may end with a deliberately small set of Human decisions, for example:

```text
DECISIONS NEEDED

1. Accept this interpretation
2. Reject and preserve current canon
3. Revise this specific item
4. Regenerate candidate
5. Return upstream to rewrite
```

Comments and decisions must remain attached to the exact story object/evidence they concern rather than becoming ungrounded chat text.

No Review Stage action may silently promote candidate material into accepted creative truth.

---

# B. Developer / UAT Review Stage

The same underlying system should support PlotPickle development and Human UAT.

The UAT Guide Agent should eventually be able to publish a structured Review Stage containing combinations of:

- current build / commit / branch / PR identity;
- feature or issue under test;
- workflow / screen under test;
- deterministic test results;
- CI evidence;
- screenshots;
- approved reference screenshots;
- visual differences;
- accessibility findings;
- Surface Grammar violations;
- navigation/redirect findings;
- terminal output;
- structured logs;
- code snippets or diffs;
- agent diagnosis;
- confidence / uncertainty;
- proposed fix;
- evidence after fix;
- Human decision request.

Illustrative structure:

```text
UAT REVIEW
Build / Commit: <identity>
Issue: <issue>

EXPERIENCE
Login        PASS
Dashboard    PASS
Learn        PASS
Plan         FAIL

VISUAL EVIDENCE
[current screenshot]
[reference screenshot]
[diff / mismatch]

AGENT FINDING
Navigation resolves to the wrong experience boundary.

CODE EVIDENCE
[diff]

TEST EVIDENCE
[terminal / CI / deterministic checks]

DECISION NEEDED
Accept fix
Reject fix
Comment
Request another run
```

The Human should be able to respond against the exact finding, screenshot, diff, test, or decision card.

This should complement GitHub rather than replace it. GitHub remains the durable engineering issue/PR/CI system; Review Stage is the visual Human/agent steering layer around the work.

---

## Review Stage surface model

The native PlotPickle implementation should support composable review parts rather than forcing every review into custom HTML.

Candidate part types:

- PlotPickle-native structured summary;
- Markdown/rich text;
- image / screenshot;
- before/after visual pair;
- visual diff;
- Mermaid / architecture diagram;
- code excerpt;
- unified diff;
- terminal output;
- JSON / structured state;
- test result;
- trace / agent activity summary;
- Story Map fragment;
- Block/Mini-Block card;
- storyboard/contact-sheet fragment;
- Previs evidence;
- decision card;
- comment thread.

A Review Stage is an ordered composition of parts tied to a stable review identity and underlying PlotPickle object identities.

The viewer/surface should supply standard chrome and visual grammar so agents do not repeatedly regenerate entire presentation frameworks.

---

## Interaction contract

Every Review Stage item should support enough identity and provenance to answer:

- What is being reviewed?
- Which PlotPickle object(s) does it refer to?
- Which agent produced the finding?
- Which evidence supports it?
- What source/version/build was examined?
- Is the item observation, inference, proposal, candidate, accepted state, or Human instruction?
- What decision is requested?
- Who responded?
- When was feedback delivered?
- Has that feedback been handled?
- What changed as a result?

Feedback delivery must be reliable. A Human comment should not disappear merely because an agent session ended, restarted, or temporarily disconnected.

Design toward explicit delivery/acknowledgement state rather than an ephemeral chat stream.

Conceptually:

```text
feedback-created
→ pending-delivery
→ delivered-to-agent
→ acknowledged
→ acted-on / answered
→ resolved
```

Exact state names are implementation-detail TBD.

---

## Agent contract

Any eligible PlotPickle agent should eventually be able to:

1. create a Review Stage;
2. add/update structured parts;
3. attach provenance and evidence;
4. request one or more Human decisions;
5. retrieve unresolved Human feedback;
6. acknowledge feedback;
7. post a revision/reply;
8. resolve an item without destroying its history.

Do not require every agent to support Review Stage.

Only agents with a meaningful Human-review boundary should publish into it.

---

## Event and data architecture direction

The Review Stage should be implemented as a PlotPickle domain capability, not as a second story authority.

Prefer an event-oriented model such as:

```text
ReviewSession
ReviewItem
ReviewPart
EvidenceRef
DecisionRequest
HumanFeedback
AgentResponse
Resolution
```

Names are provisional.

Critical constraints:

- references point to canonical PlotPickle entities; they do not duplicate canonical story/project state;
- review content may be regenerated, but Human decisions and provenance must remain durable;
- developer Review Stage may reference GitHub/CI/build artifacts without becoming their authority;
- creator Review Stage may reference PPF/story objects without becoming a parallel story store;
- all writes back into canonical state must pass the existing approval/command boundaries;
- local-only workflows must remain possible where current PlotPickle architecture allows local operation;
- cloud workflows must respect current capability/provider settings;
- stale evidence must be visibly marked when its source object/build changes.

---

## Future synthesis / product intelligence

The initial capability is **not** a product-analytics warehouse.

However, the architecture should leave room for a later evidence-synthesis layer inspired by the broader Modem concept:

```text
individual feedback
→ normalized evidence
→ clustered recurring themes
→ deduplicated findings
→ reporter / occurrence counts
→ supporting evidence
→ Human-reviewed product insight
```

Possible future developer questions:

- What screens repeatedly confuse Human testers?
- Which workflows generate the most UAT intervention?
- Which agent findings are repeatedly rejected?
- Which regressions recur?
- What requests are appearing across multiple testers?

Possible future creator questions:

- What continuity warnings recur across Blocks?
- Where does Reader Simulation repeatedly lose intended information?
- Which story addresses receive repeated revision?
- Which unresolved decisions are blocking downstream Story-to-Screen work?

This future synthesis must preserve the underlying evidence. Do not reduce Human feedback to an opaque AI-generated "summary truth."

---

## Relationship to existing PlotPickle architecture

Review Stage should integrate with, not replace:

- canonical PPF / project state;
- 4 / 12 / 24 / 96 structural addressing;
- Story Map;
- Writer-to-Screen convergence (#2165);
- Storyboard / Previs / Scene Workspace / Production convergence;
- DraftLens / Reader Simulation;
- existing Human approval boundaries;
- UAT Guide Agent;
- deterministic CI/UAT evidence;
- GitHub issues / PRs / Actions;
- current authentication and authorization;
- Surface Grammar / visual standards;
- local/cloud provider capability routing;
- Agent compute defaults / per-Agent overrides.

Where an existing owner already exists, Review Stage references that owner.

No parallel authority is permitted.

---

## Identity, permission and collaboration boundary

Do not copy Sideshow's identity/deployment assumptions directly into PlotPickle.

PlotPickle must eventually support its own authorization model for:

- creator versus developer surfaces;
- project membership;
- team/collaborator access;
- private project data;
- shared review access;
- read versus comment versus decision permissions;
- agent permissions;
- future community/collaboration boundaries.

A Review Stage must never become an authorization bypass into another project, account, developer surface, or private agent context.

---

## Visual/UX direction

Review Stage is a PlotPickle surface.

It must follow the current PlotPickle Surface Grammar, typography, spacing, keyboard/navigation requirements, accessibility rules and active visual system.

Do not reproduce Sideshow branding or interface chrome.

The useful Sideshow lesson is:

> show the Human the work itself, in a compact composable surface, and let the Human respond at the point of evidence.

The surface should prefer **one focal review context** rather than becoming another dashboard full of unrelated cards.

Progressive disclosure should keep raw logs, traces and technical detail available without making them the primary reading experience.

---

## Attribution and Sideshow contribution acknowledgement

PlotPickle should explicitly acknowledge Sideshow.sh / `modem-dev/sideshow` as an architectural/product inspiration for the Review Stage human-in-the-loop visual feedback pattern.

Reference project:

- Website: https://sideshow.sh/
- Source: https://github.com/modem-dev/sideshow
- License: MIT
- Current upstream copyright notice: `Copyright (c) 2026 Ben Vinegar`

### Attribution policy

Even if PlotPickle implements Review Stage independently and does not copy Sideshow source code, add an appropriate architecture/reference acknowledgement in PlotPickle's project documentation / third-party inspiration or acknowledgements area.

Suggested wording:

> PlotPickle Review Stage was informed in part by the open-source Sideshow project (https://github.com/modem-dev/sideshow), particularly its visual agent-work and Human feedback-loop concepts. Sideshow is licensed under the MIT License.

This acknowledgement must not imply that Sideshow, Modem, or Ben Vinegar sponsors, endorses, maintains, or is affiliated with PlotPickle.

### If Sideshow code is copied or adapted

If any Sideshow source code or substantial portion is later incorporated or adapted, preserve the upstream MIT copyright and permission notice as required by the license, including:

```text
MIT License
Copyright (c) 2026 Ben Vinegar
```

and the complete applicable MIT permission/disclaimer text in PlotPickle's third-party license notices/distribution as required.

Before implementation, re-check the upstream license and copyright notice at the exact revision actually used.

This issue is not legal advice; implementation should follow the license text in the adopted upstream revision.

---

## Recommended implementation sequence when this work is eventually scheduled

### Phase 0 — contract only

- define ReviewSession / ReviewItem / ReviewPart / Feedback / Decision contracts;
- define stable source references and provenance;
- define creator/developer permission boundaries;
- define delivery/acknowledgement semantics;
- no production UI yet.

### Phase 1 — developer/UAT prototype

Use the developer surface as the first proving ground because its evidence is easier to verify deterministically.

Support a small initial set:

- structured summary;
- screenshot/image;
- diff;
- terminal/test output;
- Human comment;
- decision request;
- agent reply/revision.

Connect initially to the UAT Guide workflow.

### Phase 2 — GitHub/CI evidence integration

Allow Review Stage to reference:

- issue;
- PR;
- commit;
- workflow run;
- failed job;
- screenshot/artifact;
- fix evidence.

Do not duplicate GitHub as the authority.

### Phase 3 — creator/story prototype

Bind the same Review Stage primitives to:

- Story Map address;
- Block/Mini-Block;
- source screenplay evidence;
- character/continuity evidence;
- DraftLens/Reader Simulation findings;
- Storyboard/Previs candidates.

### Phase 4 — decision/promotion integration

Allow accepted Human decisions to invoke existing PlotPickle commands/approval boundaries safely.

### Phase 5 — synthesis

Only after the core review loop is proven:

- normalize recurring feedback;
- cluster duplicates;
- preserve underlying evidence;
- expose Human-reviewed product/story intelligence.

---

## Acceptance criteria for the eventual feature

The issue is implementation-complete only when PlotPickle can demonstrate both of these stories with the same underlying Review Stage architecture.

### Developer acceptance story

> The UAT Guide can publish a visual review containing the failing experience, screenshot/reference evidence, deterministic test output and proposed fix. I can comment on the exact finding, the feedback reaches the correct agent once, the agent can revise/reply, and the review retains the history and build/issue provenance.

### Creator acceptance story

> PlotPickle can show me what its agents believe is happening in a specific part of my story, with the source evidence and candidate visual/structural consequences. I can accept, reject or comment on a specific item, and nothing becomes accepted story truth without my approval.

### Cross-cutting acceptance requirements

- no parallel project/story authority;
- no silent agent promotion into canon;
- durable feedback and provenance;
- exact object/build/story addressing;
- permission checks;
- stale-evidence handling;
- keyboard/accessibility compliance;
- Surface Grammar compliance;
- local/cloud boundaries respected;
- deterministic tests for feedback delivery/state transitions;
- Sideshow attribution/third-party notice requirement satisfied.

---

## Non-goals

This issue does **not** authorize:

- embedding Sideshow as PlotPickle's permanent UI;
- making Sideshow Cloud a required dependency;
- copying the Sideshow identity/security model;
- replacing GitHub;
- replacing canonical PPF/story state;
- replacing ordinary agent chat;
- exposing all agent chain-of-thought/internal reasoning;
- automatically accepting agent recommendations;
- turning every agent action into a review card;
- building a generic analytics warehouse in the first implementation;
- starting this work ahead of currently prioritized PlotPickle issues.

---

## Architectural principle to preserve

```text
SHOW THE WORK
     ↓
GROUND IT IN EVIDENCE
     ↓
ASK THE HUMAN
     ↓
DELIVER FEEDBACK TO THE RIGHT AGENT
     ↓
REVISE IN PLACE
     ↓
PRESERVE HISTORY + PROVENANCE
```

Review Stage should become the common PlotPickle Human/agent steering boundary for both story creation and product development while preserving the authority of the systems underneath it.

---

## Status

**Captured / deferred.**

This issue records the approved direction and developer brief.

Do not treat it as the next active implementation issue until the Human explicitly schedules it.


Review Stage: PlotPickle-native human-in-the-loop visual evaluation and agent feedback #2212

Description

Developer Brief

Purpose

Product decision

Reference pattern

Why this matters to PlotPickle

Creator-facing

Developer-facing

A. Creator / Story Review Stage

B. Developer / UAT Review Stage

Review Stage surface model

Interaction contract

Agent contract

Event and data architecture direction

Future synthesis / product intelligence

Relationship to existing PlotPickle architecture

Identity, permission and collaboration boundary

Visual/UX direction

Attribution and Sideshow contribution acknowledgement

Attribution policy

If Sideshow code is copied or adapted

Recommended implementation sequence when this work is eventually scheduled

Phase 0 — contract only

Phase 1 — developer/UAT prototype

Phase 2 — GitHub/CI evidence integration

Phase 3 — creator/story prototype

Phase 4 — decision/promotion integration

Phase 5 — synthesis

Acceptance criteria for the eventual feature

Developer acceptance story

Creator acceptance story

Cross-cutting acceptance requirements

Non-goals

Architectural principle to preserve

Status

Activity

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions