Add cases to an eval

An eval suite is a Vitest file plus a JSONL file of cases. Each case is a starting state (prompt, optional history) the current recipe answers afresh, plus anything you add, such as the expected result.

You need: a recipe directory, Node.js 24, and pnpm.

1. Create the suite

From the recipe root:

npm pkg set packageManager=pnpm@$(pnpm --version)   # CI reads the pnpm version from it
pnpm add -D vitest@^4.1.11 vitest-evals@^0.17.0 @introspection-ai/vitest-evals@^0.2.0
introspection eval create refund-policy                         # suite only
introspection eval create refund-policy --with-judge            # suite + a new judge of the same name
introspection eval create refund-policy --judge refund-grounding # suite using an existing judge

This writes evals/vitest/refund-policy.eval.ts and evals/vitest/refund-policy.eval.jsonl. Add eval-runs/ to .gitignore.

If pnpm stops with ERR_PNPM_IGNORED_BUILDS, run pnpm approve-builds.

When the recipe has no Vitest config, eval create also writes vitest.config.ts beside package.json (CLI 0.54.0 or later). If you already have one, it prints the settings to add instead: root (the config’s own directory), include: ["evals/vitest/**/*.eval.ts"], and testTimeout: 300_000.

Commit package.json, pnpm-lock.yaml, pnpm-workspace.yaml, vitest.config.ts, and the suite.

2. Add expected values

A row needs schema_version, a unique name, and input.prompt. Every other key is yours: put the expected result in expected, and per-case fixtures in input.

{"schema_version":2,"name":"final-sale","input":{"prompt":"Refund order 1842, it's a final-sale jacket."},"expected":{"decision":"decline"}}
{"schema_version":2,"name":"lease-17","input":{"prompt":"What is the monthly rent?","doc":"fixtures/lease-17.pdf"},"expected":{"rent":"4200"}}

The test receives the whole row:

it.for(recipeCases("refund-policy"))("$name", async ({ input, expected }, { run }) => {
  const result = await run(input);
  // grade result against expected with a judge: see Run and compare evals
});

Grade every case with a judge, not only expect(): a case with no score is incomplete and never counts as a pass or a regression. How to score a case.

3. Add a case from production

One turn per command, by trace ID or by conversation and turn number:

introspection eval add refund-policy --trace-id <trace-id> --case-name final-sale-2
introspection eval add refund-policy --from-conversation <conversation-id> --turn 3 --case-name final-sale-3

Without --case-name, the case is named after the trace ID. The row stores the turn’s prompt and the history before it, not the old answer. Add expected to it by hand. Adding the same turn twice is an error.

Steps you labelled carry the trace_id to use. To list them: introspection annotations list --label <dataset> --query "[].trace_id" -o table.

If the suite has a judge, --expected pass|fail|not-applicable also records a person’s verdict on the original answer as a judge example (--judge <name> for a differently named judge).

Verify

introspection eval run -p evals/vitest/refund-policy.eval.ts

Every new case appears in the report. Next: run and compare.

import { resolve } from "node:path";
import { createRecipeHarness } from "@introspection-ai/vitest-evals";
 
const harness = createRecipeHarness({
  renderPrompt: ({ prompt, doc }) =>
    doc ? `${prompt}\n\nThe lease is at ${resolve(String(doc))}` : prompt,
});

Limits

Or ask your agent: “Add trace <trace-id> to the refund-policy eval with the expected decision, plus two ordinary controls.”