Judges & evals

Judges

A judge is an optional, recipe-owned quality criterion evaluated by a model. It asks a stable question about a conversation—for example, whether the agent followed an escalation policy or grounded its answer in approved evidence.

Judge definitions live as YAML files directly under judges/, are validated by introspection check with the rest of the package, and travel with the recipe so any runtime can grade conversations against the same rubric you wrote locally. Keep the human-approved calibration examples beside the definition so reviewers can see both the criterion and the evidence that it measures consistently.

judges/
  escalated-when-required.yaml
  escalated-when-required.calibration.jsonl

Calibration datasets

The .calibration.jsonl file beside a judge holds one judge_fixture schema-v1 object per line, produced by introspection conversations get --ids-file <file> --judge-fixtures. --judge-fixtures requires --ids-file, and it is what makes the export a calibration dataset: without it you get ordinary conversation bundles that judges eval rejects. Export at most 20 conversations per call.

Every exported row lands with "expected": null, so a fresh export is not a usable dataset until a human labels each row. Add only two top-level fields to each object:

• expected — pass, fail, or not_applicable. Required.
• split — train, dev, test, or null. Optional.

Merge them into the exported object rather than replacing it. judges eval validates the exported schema_version, record_type, ok, engine, source, snapshot_hash, and judge_input, and rejects a row when:

• engine.protocol_version or engine.contract_version changed;
• engine.version is missing or empty;
• judge_input.conversation_id and source.conversation_id no longer match;
• snapshot_hash no longer equals the sha256 of judge_input.

Because the hash is recomputed over the serialized judge_input, editing even one byte inside it invalidates the row. Never manufacture fixture provenance or adapt arbitrary JSON into this schema — if a source conversation cannot be committed, replay an authorized sanitized version and export a new fixture from that run.

Keep most rows on cases where the judge applies, labelled pass or fail. Label a row not_applicable only to check that the judge abstains on that shape of conversation; use a run gate to keep the rest of production traffic away from the judge in the first place.

Use a judge for a durable quality boundary worth measuring across production conversations or releases. Do not add one for every preference, and do not treat unreviewed model labels as ground truth. Calibrate the judge with domain owners before using its score to gate a release or compare candidates. Introspection operates deployed judges and stores their judgements; the recipe keeps the portable authored definition. See Judges and judgements for platform behavior.

Your own evals

Everything else you write to test the agent stays yours. The evals/ directory is not a recipe resource and nothing validates it, so lay it out however your runners prefer. What it gains by living in the repository is that the thing measuring your agent is versioned with the agent it measures, and travels with it.

my-agent/
├── agents/
├── judges/                    # validated with the package; travels with it
│   └── grounded-answers.yaml
├── evals/                     # yours. Not a recipe resource, never validated.
│   ├── brief-shape.eval.ts    #   lightweight scored checks
│   └── tasks/
│       └── acme-q2/           #   Harbor: a container, a verifier, a reward
├── SYSTEM.md
└── package.json

Use Evalite  when a JavaScript or TypeScript project can call the behavior directly. It can test a helper, a model call, or a complete agent. The Introspection adapter is the Evalite task: your eval still owns its cases and scorers, while the adapter runs each prompt through the locally selected runtime and recipe.

import { evalite } from "evalite";
import { createRecipeTask, recipeCases } from "@introspection-ai/evalite";
 
evalite("Brief includes a recommendation", {
  data: recipeCases([
    {
      input: { prompt: "Prepare the Acme Q2 brief as JSON." },
      expected: { recommendationRequired: true },
    },
  ]),
  task: createRecipeTask(),
  scorers: [
    {
      name: "Has a recommendation",
      scorer: ({ output }) => {
        try {
          const brief = JSON.parse(output.output);
          return typeof brief.recommendation === "string" &&
            brief.recommendation.length > 0
            ? 1
            : 0;
        } catch {
          return 0;
        }
      },
    },
  ],
});

Pin stable evalite, vitest, and @introspection-ai/evalite as development dependencies; the CLI uses the repository’s own node_modules/.bin/evalite rather than a global install.

Use Harbor  when the agent must work inside a controlled environment — editing a repository, calling tools, or changing service state. A Harbor task combines an instruction, an environment, and a verifier, authored in Harbor’s task format . Before it can compare anything, confirm that the untouched starting state fails and a reference solution passes.

introspection setup --target harbor
introspection eval run \
  --runner harbor \
  --path evals/tasks/acme-q2 \
  --agent <agent>

The evaluated model is read from the selected agent’s ai.model. See CLI → Offline evals for every option, and Learn from production → proving a fix locally for comparing a candidate against a baseline.

Only add cases that a domain owner has reviewed and approved, and keep evaluation changes separate from agent behavior changes: land the new cases, establish the baseline, then evaluate recipe changes against it.

Where each belongs

A judge defines a reusable grading question over completed conversations and ships with the recipe. An offline eval provides controlled tasks and expected outcomes and stays yours. See Learn from production → proving a fix locally for the authoring workflow and the Recipes documentation  for the format’s position on evals.