Skip to content

feat(metadata): per-group review metadata on FlowGroup - #14

Merged
jamesaphoenix merged 1 commit into
jamesaphoenix:mainfrom
jakob1379:feat/group-descriptions
Sep 2, 2026
Merged

feat(metadata): per-group review metadata on FlowGroup#14
jamesaphoenix merged 1 commit into
jamesaphoenix:mainfrom
jakob1379:feat/group-descriptions

Conversation

@jakob1379

Copy link
Copy Markdown
Contributor

Closes#13.

Group narrative previously existed only behind a manual Pass 1 call, keyed in a side-car object the consumer had to join against the group. Every FlowGroup now carries its own review metadata — group_type, risk, impact, complexity, review_focus, description, summary, invariant — answering how should I review this group rather than restating what changed.

Group review metadata in the info panel

How it works

A deterministic heuristic floor fills risk, group_type and impact for free on every analysis. description, invariant, review_focus and complexity are deliberately left empty without an LLM — a wrong invariant sends a reviewer hunting for a property that was never at stake.

An optional batched LLM pass runs over the final groups, after refinement ops are applied, and overrides what the model has an opinion on. It cannot ride inside RefinementResponse: apply_split mints ids at apply time and apply_merge reuses the first source id, so a model answering the refinement prompt cannot key metadata to groups that do not exist yet. Batches carry a read-only index of every group so CrossCutting stays judgeable, and never read each other's results, so output does not depend on completion order.

risk_score stays deterministic and remains the only input to review ranking. Risk is a label derived from it; the model may override the label, never the score.

Desktop fires both the metadata pass and Pass 1 from the analyze path, taking the button count from four to two — Refine and Analyze This Flow. The CLI stays explicit behind --describe, so diffcore analyze in CI never starts billing silently.

Pre-existing bugs fixed on the way

  • Nothing re-scored after refinement — merged groups sat at risk_score: 0.0 and sorted as the least risky change in the diff.
  • The streaming refine path never wrote its result back to last_analysis, so annotate_overview summarized pre-refinement groups.
  • RefinementConfig::max_iterations was parsed, validated, merged, and exposed as a desktop settings control, and read by nothing. Removed, along with the CLAUDE.md claim about an evaluator-optimizer loop that never existed.

Breaking

Pass1GroupAnnotation is removed; Pass 1 keeps only its PR-level overview. Desktop and the VS Code extension read the group's own fields instead. New FlowGroup fields are all #[serde(default)], so previously written analysis JSON still deserializes.

Verification

cargo test --workspace 26/26 binaries green · Playwright 225 passed / 2 skipped · VS Code extension 120 passed · nix build .#diff-core and .#diffcore-web clean.

Not verified: --describe has never made a live API call. The pass is unit-tested and the UI is E2E-tested against mocks, but no real model response has been through it — worth a look before this is trusted.

Design decisions and their rationale are in specs/group-metadata.md.

Group narrative previously existed only behind a manual Pass 1 call, keyed
in a side-car object the consumer had to join against the group. Every
group now carries its own review metadata: group_type, risk, impact,
complexity, review_focus, description, summary and invariant.
A deterministic heuristic floor fills risk, group_type and impact for free
on every analysis. An optional batched LLM pass runs over the final groups
— after refinement ops are applied, since refinement cannot key metadata to
group ids it has not minted yet — and overrides what the model has an
opinion on. risk_score stays deterministic and remains the only input to
review ranking.
Desktop fires both the metadata pass and Pass 1 from the analyze path, so
the button count drops from four to two: Refine and Analyze This Flow.
CLI stays explicit behind --describe.
Fixes found on the way, all pre-existing:
- Nothing re-scored after refinement, so merged groups sat at
risk_score 0.0 and sorted as the least risky change in the diff.
- The streaming refine path never wrote its result back to last_analysis,
so annotate_overview summarized pre-refinement groups.
- RefinementConfig::max_iterations was parsed, validated, merged, exposed
as a desktop settings control, and read by nothing. Removed, along with
the CLAUDE.md claim about an evaluator-optimizer loop that never existed.
Removes Pass1GroupAnnotation; Pass 1 keeps only its PR-level overview.
Desktop and the VS Code extension read the group's own fields instead.
Spec: specs/group-metadata.md
Refs: jamesaphoenix#13
@jamesaphoenix
jamesaphoenix merged commit 51deaf4 into jamesaphoenix:mainSep 2, 2026
1 check passed
@jamesaphoenix

Copy link
Copy Markdown
Owner

Thanks @jakob1379, going to do a new release now.

@jakob1379
jakob1379 deleted the feat/group-descriptions branch September 2, 2026 12:04
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fold a one-line group description into the refinement pass instead of a separate manual Pass 1

2 participants

@jakob1379@jamesaphoenix