Optimize against an eval

Optimization is a search over recipe changes (instructions, skills, tool selection, reasoning settings, agent YAML) scored by an eval suite you already have. Use it to raise a score, or to make a cheaper model match a stronger one. For one diagnosed failure, change the recipe and compare two runs instead.

You need: a scored eval suite (Add cases), the recipe in Git, and a coding agent set up with the CLI (Using a coding agent). Optimization runs locally and spends tokens on your provider accounts.

1. State the objective

Give the suite, the target, hard gates, and the cases to hold out:

Use Introspection to optimize this recipe so <candidate-model> matches
<reference-model> on the refund-policy eval. Gates: no regressed case, cost
at most the baseline's. Hold out the escalation cases. Up to two rounds of
four candidates.

Without a budget the agent uses up to three rounds of two candidates. It confirms the contract once, then runs without pausing.

2. What the search does

Each candidate is a Git commit in its own worktree, reviewed for benchmark gaming before it runs. Cases, judges, fixtures, credentials, and live state never change. Held-out cases run once, on the winner, to accept or reject it.

3. Read the report

Reference (<reference-model>):  43/52
Baseline (<candidate-model>):   33/52
Selected candidate:             43/52
Target:                         met

The winner is applied to your working tree; if none improves, nothing changes.

Verify

introspection eval compare <baseline-run-id> <winner-run-id>

Expect the target cases improved and no regressed cases. Try the winner with introspection local, then open a pull request and ship it.

Limits