Evaluate offline

This continues Learn from production. The approved hypothesis is that the refund-exceptions skill applies a generic return rule before checking the final-sale exclusion.

1. Choose the lightest faithful evaluation

A local service does not automatically require Harbor. Use the simplest runner that can execute the real behavior, reset state between cases, and expose the result. Choose Harbor when the environment or its isolation is itself part of what you are testing.

Keep offline evals in evals/ beside the recipe. That directory is yours: it is versioned with the behavior it measures but is not a recipe resource and is not validated or deployed by Introspection.

2. Create a durable Vitest eval

Scaffold one named suite from the recipe root:

introspection eval create refund-policy

This creates:

evals/vitest/refund-policy.eval.ts
evals/vitest/refund-policy.eval.jsonl

The TypeScript file owns the harness, assertions, judges, thresholds, and timeouts. The JSONL file owns the inputs. Keep multiple cases for one behavior in the same JSONL file; create another named suite when the harness or quality contract differs materially.

Add the original production turn:

introspection eval add refund-policy --trace-id <trace-id>

Or select it by conversation and one-based turn number:

introspection eval add refund-policy \
  --from-conversation <conversation-id> \
  --turn <turn-number> \
  --case-name final-sale-item

Each imported Vitest case contains the completed history before the selected turn and that turn’s user prompt. It deliberately omits the observed completion: the eval reruns the recipe from that starting point. The source metadata remains reviewable without entering the agent prompt.

Import restores conversation messages, not production files, services, credentials, or side effects. Supply deterministic fixtures for those boundaries and verify that the case still exercises the decision you intend to test.

3. Add representative cases

The source failure is necessary but not sufficient. Add a boundary case and ordinary controls that could regress:

A Vitest eval can load the JSONL cases and keep the exact decision check beside the recipe:

import { expect } from "vitest";
import { describeEval } from "vitest-evals";
import {
  createRecipeHarness,
  recipeCases,
} from "@introspection-ai/vitest-evals";
 
const expectedDecision = {
  "final-sale-item": "decline",
  "eligible-unopened-item": "approve",
  "missing-order-id": "need_information",
} as const;
 
describeEval("Refund exception decisions", {
  harness: createRecipeHarness(),
}, (it) => {
  it.for(recipeCases("refund-policy"))("$name", async ({ name, input }, { run }) => {
    const result = await run(input);
    const match = /^Decision:\s*([a-z_]+)/im.exec(result.output);
    expect(match?.[1]).toBe(expectedDecision[name as keyof typeof expectedDecision]);
  });
});

Add synthetic and sanitized cases directly to the same JSONL contract when production is not the right source. Ordinary Vitest assertions can protect exact contracts; a calibrated recipeJudge("name") can score semantic quality. Read the trace and output for every failure rather than trusting the aggregate.

Before editing the Recipe, run this exact suite against the unchanged behavior:

introspection eval run -p evals/vitest/refund-policy.eval.ts

Confirm that the untouched Recipe reproduces the observed failure, a known-correct result passes, a wrong result containing convenient keywords still fails, and harmless wording changes do not fail a semantic requirement. If the baseline already passes, fix the case or hypothesis before changing the Recipe.

4. Change one mechanism

Update only the refund-exceptions decision procedure so exclusions are checked before the general return window. Do not simultaneously change the model, tools, retry policy, prompt shape, and skill: you would no longer know what caused the result.

Run the identical suite against the working candidate:

introspection eval run -p evals/vitest/refund-policy.eval.ts

The result records the recipe commit and dirty state, model, runner, trials, scores, completion reason, token usage, and native artifacts. Compare the same runner, cases, model, agent, and trial count for baseline and candidate.

Each invocation writes eval-runs/<run-id>/run.json under the Recipe, with the native Vitest report at vitest/vitest.json. Open the latest report with introspection eval view, or select Vitest explicitly with introspection eval view -r vitest.

The expected checkpoint is:

final-sale failure: corrected
eligible control: unchanged
missing-information control: unchanged
high-value boundary: escalates

Read the traces behind that summary. A higher aggregate can hide one important regression or a scorer that rewards the majority label.

5. State only what the evidence supports

A replay is a directional engineering result, not a production-wide claim:

On the captured final-sale turn and selected controls, the candidate corrected the observed decision without an obvious regression. Production-wide impact remains unverified.

Keep the source conversation IDs, hypothesis, exact skill change, baseline and candidate artifacts, control cases, and remaining risks in the focused pull request.

Common failure pattern: changing the measurement with the agent

If the candidate edits its own eval cases, scorer, or expected labels, a green result says nothing. Land or review the measurement separately, establish the baseline first, and then compare the behavior change.

Done when

• The unchanged recipe reproduces the source failure.
• A known-correct reference passes and a known-wrong result fails.
• Baseline and candidate use the same relevant configuration.
• Failure, boundary, and ordinary control cases were reviewed at the trace level.
• The candidate fixes the observed behavior without damaging the controls, or the hypothesis was revised.

Next

• Optimize a recipe: search several recipe candidates against the stable eval objective when one diagnosed change is not enough.
• Calibrate judges: encode the risk as a standing measure when it is recurring and semantic.
• Experiments and shipping: ship the clear winner or compare bounded production traffic when offline evidence cannot decide.