Compare model results
Run the same prompt with multiple models or agent harnesses in the Agents window, then compare their responses or implementations side by side. Use a folderless chat to compare ideas and plans, or a Git repository to compare code changes in isolated worktrees. Choose the strongest attempt or synthesize a combined result from the best parts of several attempts.
Model comparison is available only in the Agents window. It isn't available in the editor Chat view.
Prerequisites
Before you start, make sure you have:
- Met the Agents window prerequisites.
- Access to an eligible Agent Host harness.
For a folderless comparison, use a harness that supports quick chats. Folderless comparisons run independent chats without worktrees.
The models and harnesses available for comparison depend on your setup, credentials, and enterprise policies. Learn how to configure and choose an agent harness.
Configure a comparison
Model comparison is on by default. If Compare Models isn't shown for an eligible new session, check that sessions.chat.compareAgents.enabled is enabled in Settings.
-
In the Agents window, select New, and then select the Git repository for the task. To compare responses without a repository, start a quick chat instead.
-
Open the model picker and select Compare Models (scale icon).
-
Add between two and 10 models.
Choose from the models available to your agent harnesses. When available, you can mix Copilot, Anthropic Claude, and OpenAI Codex attempts. Copilot Fusion and its supported effort tiers are also available. Auto isn't available for comparisons.
For models that support thinking effort, add the same model more than once and select a different effort for each attempt.
-
Optionally select Configure Judge, and then choose a judge model to evaluate the attempts automatically.
-
To combine results later, select Configure Synthesis and choose a synthesizer model.
Configure the judge and synthesizer before you run the comparison. Without a judge, compare the attempts manually. Synthesis requires both a judge and a synthesizer.
-
Select Done. To compare attempts without a judge or synthesizer, select Done directly after choosing the attempts.
Run the same task across attempts
Enter one bounded prompt that gives every attempt the same goal and success criteria. For a repository comparison, use a coding task such as:
Add validation to the profile form so the display name is required and no longer
than 50 characters. Show an inline error, preserve the existing form styles, and
add focused tests for valid, empty, and overlong names. Don't change other forms.
For a folderless comparison, ask a question or request a plan, such as outlining the trade-offs between two API designs.
Select Run N Attempts, where N is the number of attempts you configured. All attempts start concurrently as independent sessions. The results appear in a tiled grid by default, where you can monitor each attempt and open its session.
For repository comparisons, each attempt works in its own Git worktree based on the repository's committed state. It doesn't include changes that are uncommitted in your original working tree. Git-ignored files and dependencies might also be absent. For worktree requirements, ignored files, and Allow all permission behavior, see Choose where an agent makes changes.
Repository comparisons started from Compare Models use worktree sessions with Allow all, which skips tool approval prompts. Worktrees separate edits, not the agent's access to your machine or network. Review approvals and permissions and agent sandboxing before you run a comparison.
Ask an agent to run a comparison
Ask an agent in an existing conversation to compare available models on a task and summarize the results. Enable sessions.chat.compareAgents.tools.enabled (off by default) and sessions.chat.compareAgents.enabled first. Comparisons inherit the conversation's workspace and permissions. Synthesis remains a separate, user-initiated action.
For example:
Compare two available models on the task 'Suggest tests for a command-line calculator'.
Use a judge to evaluate their answers.
Compare the results
After the attempts finish, compare them manually or use the judge scorecard:
- Review the responses and, for repository comparisons, changed files for each attempt.
- Check whether each result meets the prompt's constraints. For code changes, also check that the implementation follows the repository's conventions.
- For code changes, run the relevant tests and checks in promising attempts.
If you configured a judge, evaluation starts automatically after the attempts finish. The scorecard includes a recommendation, time and token usage for each attempt, and expandable rationale with validation evidence. Select Open Attempt N to inspect an attempt's full session.
Token values can be lower bounds rather than complete totals. The scorecard doesn't measure credit or currency costs.
A judge's assessment isn't proof that a result is correct. Verify the evidence. For code changes, review the code and run the project's tests before you integrate a result.
Synthesize a combined result
If you configured both a judge and a synthesizer, select Synthesize after reviewing the verdict. Add optional instructions to focus the synthesis on specific strengths, constraints, or corrections.
The synthesizer uses the judge verdict and strengths from the attempts to create a new grouped session. For repository comparisons, it produces a new implementation in another isolated worktree. For folderless comparisons, it combines the responses into a new answer. It doesn't modify the original attempts or your working tree.
Synthesis doesn't automatically merge code changes. Review and validate the combined result separately.
Continue with a result
Open the attempt or synthesized session that you want to continue. For a folderless comparison, continue the conversation in the chat you choose. For code changes, review all changed and untracked files, run the relevant tests, and make any needed revisions.
When a repository result is ready, follow the agent change review and worktree integration workflow. Comparison doesn't automatically merge, push, or create a pull request for the selected result.
Each attempt, judge evaluation, and synthesis is separate model work. Consider the additional model usage when you choose the number of attempts and whether to enable evaluation or synthesis.