Run and compare evals

introspection eval run runs a Vitest suite against your local recipe and saves the run under eval-runs/. eval compare lines up two runs case by case. This page uses CLI 0.52.0 or later.

You need: a suite (Add cases to an eval) and the model provider key your agent uses.

1. Run a baseline

On the unchanged recipe, from the recipe root:

introspection eval run -p evals/vitest

-p takes the suite directory or one .eval.ts file. It exits 0 even when cases fail. “No test files found” means the recipe’s Vitest config lacks the settings eval create names.

2. Score each case

Only a case with a score counts. A case with no judge, or whose expect() throws, is incomplete: it never counts as passed or regressed. Score with a YAML judge (recipeJudge) or a code judge built with createJudge, which can give partial credit:

import { expect } from "vitest";
import { createJudge, describeEval } from "vitest-evals";
import { createRecipeHarness, recipeCases, recipeJudge } from "@introspection-ai/vitest-evals";
 
const CodesRecall = createJudge<unknown, string, { expected: { codes: string[] } }>(
  "codes-recall",
  ({ output, expected }) => {
    const found = expected.codes.filter((code) => output.includes(code));
    return { score: found.length / expected.codes.length };
  },
);
 
describeEval("Billing codes", {
  harness: createRecipeHarness(),
  judges: [recipeJudge("cites-policy")],   // runs on every case
}, (it) => {
  it.for(recipeCases("billing-codes"))("$name", async ({ input, expected }, { run }) => {
    const result = await run(input);
    await expect(result).toSatisfyJudge(CodesRecall, { expected: expected as { codes: string[] }, threshold: 1 });
  });
});

Each case scores 0 to 1: the mean of its judges. It passes at 1.

3. Change one thing, run again, compare

introspection eval run -p evals/vitest
introspection eval compare <baseline-run-id>    # against the latest run
introspection eval view                         # open the report

Each case is improved, regressed, unchanged, new, removed, or incomplete (not measured: no score, or an error). Keep the model, cases and trials fixed: compare refuses two runs whose cases changed.

4. Keep a baseline

Commit the suite, run it, then promote that complete run and commit baseline.json so later runs compare against it. Promote refuses a run made with uncommitted changes:

introspection eval run -p evals/vitest
introspection eval baseline promote <run-id>   # writes baseline.json in the suite directory
introspection eval compare --baseline          # latest run vs the committed baseline

For the suite above that file is evals/vitest/baseline.json. To gate pull requests on it, see Gate pull requests.

Verify

The case you fixed is improved and every other case is unchanged.

introspection eval compare --baseline --query candidate.scores
introspection eval compare --baseline --query "{base:base.scores,candidate:candidate.scores}"

introspection eval run -p evals/vitest --trials 3
introspection eval compare <trial-1-run-id> <trial-2-run-id> --query "cases[?change!='unchanged'].id" -o table

introspection eval run -p evals/vitest --agent agent-small
introspection eval run -p evals/vitest --agent agent-large
introspection eval compare <small-run-id> <large-run-id>

Limits

Or ask your agent: “Run the refund-policy eval before and after this change and tell me what regressed.”