Judges

It asks a stable question about a conversation—for example, whether the agent followed an escalation policy or grounded its answer in approved evidence.

Judge YAML

A judge belongs with the agent it grades, so it is authored as a YAML file in the recipe’s git repository, directly under judges/:

customer-support/
├── SYSTEM.md
├── agents/*.yaml
├── skills/**/SKILL.md
└── judges/*.yaml      # one file per judge

A judge file declares its identity, the model it grades with, and the rubric. It can also include an optional on gate that decides whether the judge is relevant before making an LLM call.

name identifies the judge within the recipe and must be lowercase kebab-case: ASCII letters, digits, and single hyphens. introspection check validates judges along with the rest of the package.

name: billing-request-handling
description: "Did the assistant handle a billing request correctly?"
 
on:
  - event: message
    match:
      role: user
      text: /refund|invoice/i
 
llm:
  provider: openai
  model: gpt-5.6-luna
 
instructions: |
  Grade the assistant's handling of the billing request.
 
  Return pass when the response follows the billing policy, asks only for
  information needed to proceed, and gives a clear next step.
 
  Return fail when it makes an unsupported promise, exposes sensitive payment
  information, or leaves the user without a useful next step.
 
  Return not_applicable when the matched message is not actually about billing.
  Output { reasoning, verdict }.

name: useful-resolution
description: "Did the response move the request toward a useful resolution?"
 
llm:
  provider: openai
  model: gpt-5.6-luna
 
instructions: |
  Grade the assistant's response to the user's request.
 
  Return pass when it answers the request, asks for information genuinely
  needed to proceed, or clearly explains an appropriate next step.
 
  Return fail when it avoids the request, makes an unsupported claim, or leaves
  the user without a useful next step.
 
  Return not_applicable when the conversation contains no assistant response
  that can be graded.
  Output { reasoning, verdict }.

The gated judge runs only when a user message matches both fields in its matcher: role is user and text matches the regex. A judge can list several message, tool, or feedback matchers; matching any one of them makes the judge applicable. When none match, the judge does not run and no judgement is emitted. not_applicable is a verdict a judge may emit after it runs; a gate miss does not create that verdict.

The ungated judge evaluates every completed conversation selected for judging. Its rubric therefore owns the full applicability decision and should explain when to return not_applicable.

Calibration datasets

Keep the human-approved calibration examples beside the definition so reviewers can see both the criterion and the evidence that it measures consistently:

judges/
  escalated-when-required.yaml
  escalated-when-required.calibration.jsonl

The .calibration.jsonl file beside a judge holds one judge_fixture schema-v1 object per line, produced by introspection conversations get --ids-file <file> --judge-fixtures. --judge-fixtures requires --ids-file and supports only --format json, and it is what makes the export a calibration dataset: without it you get ordinary conversation bundles that judges eval rejects. Export at most 20 conversations per call. The export step needs a logged-in CLI (introspection login) and a deployed runtime with real production conversations to export from.

Every exported row lands with "expected": null, so a fresh export is not a usable dataset until a human labels each row. Add only two top-level fields to each object:

• expected — pass, fail, or not_applicable. Required.
• split — train, dev, test, or null. Optional.

Merge them into the exported object rather than replacing it. judges eval validates the exported schema_version, record_type, ok, engine, source, snapshot_hash, and judge_input, and rejects a row when:

• engine.protocol_version or engine.contract_version changed;
• engine.version is missing or empty;
• judge_input.conversation_id and source.conversation_id no longer match;
• snapshot_hash no longer equals the sha256 of judge_input;
• the same conversation appears on more than one line.

Because the hash is recomputed over the serialized judge_input, editing even one byte inside it invalidates the row. Never manufacture fixture provenance or adapt arbitrary JSON into this schema — if a source conversation cannot be committed, replay an authorized sanitized version and export a new fixture from that run.

Keep most rows on cases where the judge applies, labelled pass or fail. Label a row not_applicable only to check that the judge abstains on that shape of conversation; use a run gate to keep the rest of production traffic away from the judge in the first place.

This file format is the authoring artifact; the calibration procedure itself — running judges eval, reading disagreements, and iterating the rubric — is covered in Judges → Calibration, with a step-by-step workflow in Calibrate judges.

When to add a judge

Use a judge for a durable quality boundary worth measuring across production conversations or releases. Do not add one for every preference, and do not treat unreviewed model labels as ground truth. Calibrate the judge with domain owners before using its score to gate a release or compare candidates. Introspection operates deployed judges and stores their judgements; the recipe keeps the portable authored definition. See Judges and judgements for platform behavior.

Everything else you measure with

Judges are the only evaluation artifact the format validates and ships. Your own offline evals — Evalite suites, Harbor tasks, ordinary tests — live in an evals/ directory that is yours: versioned beside the behavior it measures, but not a recipe resource and never validated. See Evaluate offline for choosing a runner, authoring cases, and comparing a candidate against a baseline, and the Recipes documentation  for the format’s position on evals.