Judges

A judge is how you encode “good” as code: a version-pinned rubric that grades each conversation pass, fail, or not applicable, automatically, as conversations complete.

Observations tell you what the platform noticed. A judge is how you define what should have happened. It’s the evaluation primitive in the operator loop: write down a standard once, measure it continuously on real traffic, and act when the number moves.

What a judge is

A judge is a code-first, LLM-as-judge rubric. An LLM reads a completed conversation and grades it against your instructions, returning one of a fixed verdict vocabulary:

pass: the conversation met the standard.
fail: it didn’t.
not_applicable: the judge doesn’t apply to this conversation (a semantic abstention; it’s not counted against you).

You write only the rubric’s instructions; the set of verdicts is fixed by the platform. The model is always asked for a reasoning plus a verdict from that closed set, so it can never invent a "mostly_pass" and break downstream scoring. A response the model can’t grade surfaces as an explicit error rather than silently passing.

Judges are LLM trajectory judges: they grade semantic questions like “did the assistant clarify before researching?” or “was the answer actually helpful?”. They’re for nuanced, judgment-based standards, not deterministic structural assertions.

Judges are git-owned

A judge belongs with the agent it grades, so it’s authored as a YAML file in your recipe’s git repository, under judges/:


customer-support/
├── SYSTEM.md
├── agents/*.yaml
├── skills/**/SKILL.md
└── judges/*.yaml      # one file per judge

A judge file declares its identity, the model it grades with, an optional run-gate, and the rubric itself:


judge: useful_support_resolution
description: "Did the response move the support request toward a useful resolution?"
 
llm:
  provider: openai
  model: gpt-4.1-nano
 
instructions: |
  Grade the assistant's handling of a customer-support request.
 
  Return pass when it gives an actionable answer grounded in approved policy,
  asks for information genuinely needed to proceed, or clearly escalates a
  sensitive request and explains the next step.
 
  Return fail when it only repeats a policy, makes an unsupported promise, or
  leaves the customer without a useful next step.
 
  Return not_applicable when the conversation is not a support request.
  Output { reasoning, verdict }.

This judge has no on gate, so it evaluates every completed conversation and uses not_applicable for non-support traffic. Add a gate when applicability can be identified structurally—for example, because a particular support tool was called. A gate is a small matcher over message, tool, and feedback events; when it does not match, the result is not_applicable without an LLM call.

Because the definition lives in git, it inherits the recipe’s immutability contract: editing a rubric is a new commit, and every judge is pinned to an exact recipe version. That’s what makes a quality number meaningful: you always know exactly which rubric produced it.

Definition vs. operational state

A judge has two halves, owned by different people:

Half	Owned by	Examples
Definition	Git (pinned to a recipe version)	the rubric `instructions`, the `llm` configuration, the `on` gate, the judge’s name
Operational state	The operator (adjustable live)	`enabled` (an off-switch), `sample_rate` (how often to grade production traffic)

The definition is the intellectual property and the only thing that determines a verdict. Operational state is how you run the judge in production: turning it off, or dialing down how much production traffic it grades for cost, without touching git. Operational changes never alter what the judge measures, so they don’t disturb its trend line. Lower environments are graded in full; only production traffic is sampled.

When judges run

Judges execute automatically in the runtime sandbox when a conversation completes, at teardown, after the response has returned but while the sandbox still has time and network available. The sandbox grades the conversation in place and emits the result. There’s no separate batch job to schedule and nothing to trigger by hand.

Calibrate locally

Before enabling a judge on live traffic, assemble labelled fixtures from bounded conversation reads and run the same Rust judge engine locally:


introspection conversations get \
  --ids-file conversation-ids.txt \
  --output-file judge-fixtures.jsonl
 
OPENAI_API_KEY=... introspection judges eval \
  --judge judges/useful-support-resolution.yaml \
  --dataset judge-fixtures.jsonl

Label each fixture’s expected field as pass or fail; optionally assign train, dev, or test in split. The judge YAML owns its OpenAI-compatible model configuration, and the local model credential stays separate from the Introspection login. See CLI → Judge calibration.

Judgements

A judgement is one judge’s verdict over one conversation: the judge analogue of an observation. Judgements are immutable events on the same stream as everything else, and they’re tracked over time so you can do two things:

Trend quality across versions. Because every judgement carries the recipe commit it graded, you can watch a judge’s pass-rate move as your agent changes: a before/after across a deploy is a single query. A re-grade under an edited rubric is a new judgement, never a silent overwrite, so versions never blur together.
Keep a standing regression guard. Once a judge encodes a failure mode you’ve fixed, leave it running. If the rate regresses on a future version, you see it immediately instead of rediscovering the bug in production.

When comparing versions, pin a single rubric version. A judge’s identity is stable across rubric edits, so one judge can span several rubric versions. Mixing them in one rate is apples-to-oranges.

How it connects

Recipes own judge definitions: a judge is a recipe artifact, version-pinned to the same git commit as the agent it grades.
Conversations are what judges grade: one judgement per conversation per judge.
Experiments consume judgements as their scoring signal: a judge bound to an experiment’s runtime group grades every arm automatically, so you can compare versions on a measured pass-rate.