Evaluate offline

This continues Learn from production. The approved hypothesis is that the refund-exceptions skill applies a generic return rule before checking the final-sale exclusion.

1. Choose the lightest faithful evaluation

A local service does not automatically require Harbor. Use the simplest runner that can execute the real behavior, reset state between cases, and expose the result. Choose Harbor when the environment or its isolation is itself part of what you are testing.

Keep offline evals in evals/ beside the recipe. That directory is yours: it is versioned with the behavior it measures but is not a recipe resource and is not validated or deployed by Introspection.

2. Capture the baseline before editing

Replay the original production turn against the unchanged recipe:

introspection eval run \
  --runner evalite \
  --path evals/refund-policy.eval.ts \
  --from-conversation <conversation-id> \
  --turn <turn-number> \
  --agent agent

Replay restores conversation state, not production files, services, credentials, or side effects. Supply deterministic fixtures for those boundaries and verify that the replay still exercises the decision you intend to test.

Before trusting the comparison, confirm:

• the untouched recipe reproduces the observed failure;
• a known-correct reference response passes the scorer;
• a wrong response containing convenient keywords still fails; and
• harmless wording or formatting changes do not fail a semantic requirement.

If the baseline already passes, the evaluation does not reproduce the bug. Fix the fixture or hypothesis before changing the recipe.

3. Add representative cases

The source failure is necessary but not sufficient. Add a boundary case and ordinary controls that could regress:

An Evalite suite can keep the cases and the exact decision check beside the recipe:

import { evalite } from "evalite";
import { createRecipeTask, recipeCases } from "@introspection-ai/evalite";
 
evalite("Refund exception decisions", {
  data: recipeCases([
    {
      input: {
        prompt: "Order 1842 is final-sale and inside 30 days. Review eligibility.",
      },
      expected: { decision: "decline" },
    },
    {
      input: {
        prompt: "Order 2207 is unopened, eligible, and inside 30 days. Review eligibility.",
      },
      expected: { decision: "approve" },
    },
    {
      input: { prompt: "I want a refund but do not have an order ID." },
      expected: { decision: "need_information" },
    },
  ]),
  task: createRecipeTask(),
  scorers: [
    {
      name: "Correct decision",
      scorer: ({ output, expected }) => {
        const match = /^Decision:\s*([a-z_]+)/im.exec(output.output);
        return match?.[1] === expected.decision ? 1 : 0;
      },
    },
  ],
});

This exact scorer protects the declared decision field. It does not prove that the evidence or draft is semantically good. Read the trace and output for every failure and use a calibrated judge only when that broader quality becomes a durable measurement need.

4. Change one mechanism

Update only the refund-exceptions decision procedure so exclusions are checked before the general return window. Do not simultaneously change the model, tools, retry policy, prompt shape, and skill: you would no longer know what caused the result.

Run the identical suite against the working candidate:

introspection eval run \
  --runner evalite \
  --path evals/refund-policy.eval.ts \
  --agent agent

The result records the recipe commit and dirty state, model, runner, trials, scores, completion reason, token usage, and native artifacts. Compare the same runner, cases, model, agent, and trial count for baseline and candidate.

The expected checkpoint is:

final-sale failure: corrected
eligible control: unchanged
missing-information control: unchanged
high-value boundary: escalates

Read the traces behind that summary. A higher aggregate can hide one important regression or a scorer that rewards the majority label.

5. State only what the evidence supports

A replay is a directional engineering result, not a production-wide claim:

On the captured final-sale turn and selected controls, the candidate corrected the observed decision without an obvious regression. Production-wide impact remains unverified.

Keep the source conversation IDs, hypothesis, exact skill change, baseline and candidate artifacts, control cases, and remaining risks in the focused pull request.

Common failure pattern: changing the measurement with the agent

If the candidate edits its own eval cases, scorer, or expected labels, a green result says nothing. Land or review the measurement separately, establish the baseline first, and then compare the behavior change.

Done when

• The unchanged recipe reproduces the source failure.
• A known-correct reference passes and a known-wrong result fails.
• Baseline and candidate use the same relevant configuration.
• Failure, boundary, and ordinary control cases were reviewed at the trace level.
• The candidate fixes the observed behavior without damaging the controls, or the hypothesis was revised.

Next

• Calibrate judges: encode the risk as a standing measure when it is recurring and semantic.
• Experiments and shipping: ship the clear winner or compare bounded production traffic when offline evidence cannot decide.