Improve an agent
Find what goes wrong in production, turn it into cases you can rerun, prove a change fixes it, then ship.
observe → annotate → eval cases → evals → judges → experiment → ship
↑ │
└───────────────────────────────────────────────────────────────┘| Step | Page |
|---|---|
| Observe: find bad conversations, failing judgements, negative feedback | Conversations |
| Annotate: label production steps into datasets, assign reviewers | Label production steps |
| Eval cases: turn a turn into a rerunnable case with its expected result | Add cases to an eval |
| Evals: run before and after a change, compare case by case | Run and compare evals |
| CI: fail a pull request when a case regresses | Gate pull requests |
| Fakes: realistic MCP tools without real systems | Fake MCP servers |
| Judges: grade quality with a calibrated rubric, offline and live | Calibrate a judge |
| Experiment: split live traffic between versions | Run an experiment |
| Optimize: let your agent search recipe changes against an eval | Optimize |
| Harbor: task evals with a verifier, locally or on Introspection | Run Harbor tasks |
| Ship: merge and release the winner | Deploy a runtime |
Evals, calibration, and optimize run locally or in CI on your model keys;
online judges, experiments, and --commit-sha Harbor trials run on Introspection.
Or ask your agent: “Use Introspection to find this week’s worst refund failures, add them as eval cases, and fix the top one.”
Last updated on