You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Implements the roadmap's top "Now" item: bolt run --best-of "anthropic/claude-sonnet-4,openai/gpt-5" "task" fires the same task at every listed model in parallel (each in its own fresh session), then a judge model ranks the anonymized outputs and the winner's text is printed. The judge is --model when given, otherwise the first candidate.
Each candidate runs through the existing blocking session.prompt call, so the whole race is one Promise.all; the judge sees only labeled outputs (no model names) to avoid reputation bias, and its verdict is parsed from a machine-readable trailer:
// Pulls the last RANKING line out of the judge's reply. Returns the label// order, or undefined when the judge did not produce a usable verdict.exportfunctionparseVerdict(text: string,labels: string[]): string[]|undefined{constmatches=[...text.matchAll(/RANKING:\s*([A-Z](?:\s*>\s*[A-Z])*)/gi)]constlast=matches.at(-1)if(!last)returnundefinedconstorder=last[1].split(">").map((item)=>item.trim().toUpperCase())// ... validates it is a complete permutation of the candidate labels}
Design choices a reviewer can't infer from the diff:
The winner's text goes to stdout and everything else (progress notes, the ranked scoreboard with per-candidate session IDs) goes to stderr via UI, so --best-of stays pipeable exactly like a plain run. --format json emits one best_of JSON object instead.
Every candidate session survives the race: the scoreboard prints each session ID so any candidate (not just the winner) can be resumed with bolt run --session.
If the judge fails, times out, or returns an unparseable verdict, the first finished candidate wins with a warning; if only one candidate succeeds the judge is skipped; per-candidate failures don't abort the race. Exit code is 1 only when every candidate fails.
--best-of is rejected with --mini, --command, --session, --continue, and --fork (it always runs fresh sessions), capped at 8 models, and works with --attach since it goes through the same SDK client.
New logic lives in packages/opencode/src/cli/cmd/run/best-of.ts (lazily imported); run.ts only grows the flag declaration, validation, and a small branch in execute().
Also adds /yolo to the TUI. The permission auto-approve machinery already existed (permission.mode === "auto" auto-replies to permission.asked in context/sync.tsx before the dialog renders, toggled by /auto-approve); this PR adds yolo as a slash alias of that command and closes the missing half: while yolo mode is on, every prompt carries a system-level instruction so the model stops asking in prose for approval it would get automatically anyway:
system:
local.permission.mode==="auto"
? "Yolo mode is active: every permission request is granted automatically. Do not ask the user for permission or confirmation before acting; proceed directly."
: undefined,
The server appends the per-message system field after the base system prompt, so this is additive per prompt and turns off cleanly the moment the toggle flips back.
How did you verify your code works?
bun test ./test/cli/run/best-of.test.ts in packages/opencode: 11 tests covering candidate parsing (dedupe, nested model paths, limits, malformed entries), judge prompt construction, and verdict parsing (case/spacing tolerance, repeated rankings, incomplete or duplicated verdicts)
bun test ./test/cli/run/ (full run-command suite): 211 pass
bun typecheck in packages/opencode: clean
bunx tsgo --noEmit and bun test in packages/tui for the /yolo change: 198 pass
Screenshots / recordings
N/A (CLI; end-to-end output requires live provider credentials)
Checklist
I have tested my changes locally
I have not included unrelated changes in this PR
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is enabled.
Summary by CodeRabbit
New Features
Added a --best-of option to compare responses from multiple model candidates.
Runs candidates in parallel and selects the strongest response through automated ranking.
Supports human-readable and JSON output, with fallback handling when ranking is unavailable.
Validates candidate lists and prevents use with incompatible interactive or session modes.
Tests
Added coverage for candidate parsing, ranking, output selection, and invalid input handling.
The CLI adds --best-of for parallel runs across provider/model candidates. It validates candidates, ranks successful outputs with a judge model, supports JSON and human-readable output, and integrates the flow with existing run modes.
Changes
Best-of-N model runs
Layer / File(s)
Summary
Candidate and ranking protocol packages/opencode/src/cli/cmd/run/best-of.ts, packages/opencode/test/cli/run/best-of.test.ts
The module parses and validates candidates, builds judge prompts, parses ranking verdicts, and tests these behaviors.
Parallel execution and result selection packages/opencode/src/cli/cmd/run/best-of.ts
Candidate sessions run concurrently. Successful outputs are ranked, failures are reported, and the winner is emitted as human-readable or JSON output.
CLI option and execution wiring packages/opencode/src/cli/cmd/run.ts
The command adds --best-of, rejects incompatible modes, invokes runBestOf, propagates its exit code, and supplies undefined values from runMini.
Session2.prompt already defines this model as { providerID: string; modelID: string }. Let TypeScript infer the object so future shape changes produce a compile-time error.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/opencode/src/cli/cmd/run/best-of.ts` around lines 46 - 49, Remove
the unnecessary `as Model` assertion from the object returned by the unique.map
callback in Session2.prompt, allowing TypeScript to infer the `{ providerID,
modelID }` shape and catch future model-shape changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/opencode/src/cli/cmd/run.ts`:
- Around line 682-699: Update the best-of branch’s SDK selection around
runBestOf to resolve the attach directory before calling attachSDK: when
args.attach is set and directory is undefined, obtain the directory via the
existing current(sdk) fallback used by the single-session path. Pass the
resolved directory to attachSDK while preserving the existing SDK behavior for
explicit directories and non-attach runs.
In `@packages/opencode/src/cli/cmd/run/best-of.ts`:
- Around line 163-164: Update the fallback note in the best-of selection flow
around verdict and order to state that the first successful candidate in
submission order is retained, rather than referring to the first finished
candidate. Keep the selection logic unchanged.
- Around line 135-138: Update the total-failure branch guarded by done.length
=== 0 in the best-of command to honor input.json: emit one structured best_of
JSON object containing the failure details instead of relying on UI.error, while
preserving the existing human-readable error output for non-JSON runs and
returning 1.
---
Nitpick comments:
In `@packages/opencode/src/cli/cmd/run/best-of.ts`:
- Around line 46-49: Remove the unnecessary `as Model` assertion from the object
returned by the unique.map callback in Session2.prompt, allowing TypeScript to
infer the `{ providerID, modelID }` shape and catch future model-shape changes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
Push a commit to this branch (recommended)
Create a new PR with the fixes
ℹ️ Review info⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 0e201d89-f5dc-4025-8a67-40ee9f03397f
📥 Commits
Reviewing files that changed from the base of the PR and between 9d17df6 and 352220d.
We reviewed changes in 9d17df6...ce3afbe on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.
Some issues found as part of this review are outside of the diff in this pull request and aren't shown in the inline review comments due to GitHub's API limitations. You can see those issues on the DeepSource dashboard.
PR Report Card
Overall Grade
Focus Area: Reliability
Security
Reliability
Complexity
Hygiene
Feedback
Null assertions and early use of values
The non-null assertions and the costAlert usage-before-definition are both about assuming values exist in edge cases.
This matters because best-of-N and prompts are inherently about branching behavior; tightening those assumptions in one pass will likely make the flows more predictable under odd inputs.
Browser console usage and unused state
console usage in browser components and the unused moveLabelWidth both look like temporary debugging hooks.
Worth treating these as a single “debug leftovers” pass so the TUI surface stays clean and easier to reason about.
AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.
The reason will be displayed to describe this comment to others. Learn more.
`execute` has a cyclomatic complexity of 22 with "high" risk
A function with high cyclomatic complexity can be hard to understand and
maintain. Cyclomatic complexity is a software metric that measures the number of
independent paths through a function. A higher cyclomatic complexity indicates
that the function has more decision points and is more complex.
The reason will be displayed to describe this comment to others. Learn more.
The complexity is pre-existing in the run command handler; this PR only adds one small branch, and refactoring the whole function would be unrelated churn against the repo's keep-things-in-one-function style.
The reason will be displayed to describe this comment to others. Learn more.
Async function 'execute' expected a return value
Any code paths that do not have explicit returns will return undefined. It is recommended to replace any implicit dead-ends that return undefined with a return null statement.
The reason will be displayed to describe this comment to others. Learn more.
Unexpected function declaration in the global scope, wrap in an IIFE for a local variable, assign as global property for a global variable
It is considered a best practice to avoid 'polluting' the global scope with variables that are intended to be local to the script. Global variables created from a script can produce name collisions with global variables created from another script, which will usually lead to runtime errors or unexpected behavior. It is mostly useful for browser scripts.
The reason will be displayed to describe this comment to others. Learn more.
Unexpected function declaration in the global scope, wrap in an IIFE for a local variable, assign as global property for a global variable
It is considered a best practice to avoid 'polluting' the global scope with variables that are intended to be local to the script. Global variables created from a script can produce name collisions with global variables created from another script, which will usually lead to runtime errors or unexpected behavior. It is mostly useful for browser scripts.
The reason will be displayed to describe this comment to others. Learn more.
Unexpected function declaration in the global scope, wrap in an IIFE for a local variable, assign as global property for a global variable
It is considered a best practice to avoid 'polluting' the global scope with variables that are intended to be local to the script. Global variables created from a script can produce name collisions with global variables created from another script, which will usually lead to runtime errors or unexpected behavior. It is mostly useful for browser scripts.
The reason will be displayed to describe this comment to others. Learn more.
`runBestOf` has a cyclomatic complexity of 7 with "medium" risk
A function with high cyclomatic complexity can be hard to understand and
maintain. Cyclomatic complexity is a software metric that measures the number of
independent paths through a function. A higher cyclomatic complexity indicates
that the function has more decision points and is more complex.
The reason will be displayed to describe this comment to others. Learn more.
Cyclomatic complexity of 7 is modest for the orchestration entry point, and the repo style guide explicitly prefers keeping logic in one function unless helpers name a real reusable concept.
The reason will be displayed to describe this comment to others. Learn more.
Forbidden non-null assertion
Using non-null assertions cancels out the benefits of strict null-checking, and introduces the possibility of runtime errors. Avoid non-null assertions unless absolutely necessary. If you still need to use one, write a skipcq comment to explain why it is safe.
The reason will be displayed to describe this comment to others. Learn more.
Safe by construction: order is either a verdict validated as a complete permutation of the done labels or the done labels themselves, and ranked contains every settled label, so the lookup cannot return undefined. A fallback would be dead defensive code.
The reason will be displayed to describe this comment to others. Learn more.
Forbidden non-null assertion
Using non-null assertions cancels out the benefits of strict null-checking, and introduces the possibility of runtime errors. Avoid non-null assertions unless absolutely necessary. If you still need to use one, write a skipcq comment to explain why it is safe.
The reason will be displayed to describe this comment to others. Learn more.
Safe by construction: every label in order comes from done (a subset of settled), and ranked is built from all settled candidates, so the lookup cannot return undefined.
The reason will be displayed to describe this comment to others. Learn more.
Same as the sibling finding: the UI.Style token concatenation matches the codebase-wide convention, so changing it would reduce consistency without functional benefit.
The reason will be displayed to describe this comment to others. Learn more.
Concatenating UI.Style tokens with text (e.g. UI.Style.TEXT_DIM + text) is the established pattern throughout this file and run.ts; converting to a template literal here would be inconsistent churn.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue for this PR
Closes#64
Type of change
What does this PR do?
Implements the roadmap's top "Now" item:
bolt run --best-of "anthropic/claude-sonnet-4,openai/gpt-5" "task"fires the same task at every listed model in parallel (each in its own fresh session), then a judge model ranks the anonymized outputs and the winner's text is printed. The judge is--modelwhen given, otherwise the first candidate.Each candidate runs through the existing blocking
session.promptcall, so the whole race is onePromise.all; the judge sees only labeled outputs (no model names) to avoid reputation bias, and its verdict is parsed from a machine-readable trailer:Design choices a reviewer can't infer from the diff:
UI, so--best-ofstays pipeable exactly like a plainrun.--format jsonemits onebest_ofJSON object instead.bolt run --session.--best-ofis rejected with--mini,--command,--session,--continue, and--fork(it always runs fresh sessions), capped at 8 models, and works with--attachsince it goes through the same SDK client.New logic lives in
packages/opencode/src/cli/cmd/run/best-of.ts(lazily imported);run.tsonly grows the flag declaration, validation, and a small branch inexecute().Also adds
/yoloto the TUI. The permission auto-approve machinery already existed (permission.mode === "auto"auto-replies topermission.askedincontext/sync.tsxbefore the dialog renders, toggled by/auto-approve); this PR addsyoloas a slash alias of that command and closes the missing half: while yolo mode is on, every prompt carries a system-level instruction so the model stops asking in prose for approval it would get automatically anyway:The server appends the per-message
systemfield after the base system prompt, so this is additive per prompt and turns off cleanly the moment the toggle flips back.How did you verify your code works?
bun test ./test/cli/run/best-of.test.tsinpackages/opencode: 11 tests covering candidate parsing (dedupe, nested model paths, limits, malformed entries), judge prompt construction, and verdict parsing (case/spacing tolerance, repeated rankings, incomplete or duplicated verdicts)bun test ./test/cli/run/(full run-command suite): 211 passbun typecheckinpackages/opencode: cleanbunx tsgo --noEmitandbun testinpackages/tuifor the /yolo change: 198 passScreenshots / recordings
N/A (CLI; end-to-end output requires live provider credentials)
Checklist
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is enabled.Summary by CodeRabbit
New Features
--best-ofoption to compare responses from multiple model candidates.Tests