Judges

Observations tell you what the platform noticed. A judge lets you define what good behavior looks like, check it consistently on real conversations, and see whether that behavior improves or regresses over time.

What a judge is

A judge is a code-first, LLM-as-judge rubric. An LLM reads a completed conversation and grades it against your instructions, returning one of a fixed verdict vocabulary:

• pass: the conversation met the standard.
• fail: it didn’t.
• not_applicable: the judge doesn’t apply to this conversation.

You write only the rubric’s instructions and the set of verdicts is fixed by the platform. The model is always asked for a reasoning plus a verdict from that closed set. A response the model can’t grade surfaces as an explicit error.

Judges are git-owned

A judge belongs with the agent it grades, so it’s authored as a YAML file in your recipe’s git repository, under judges/ — one file per judge, declaring its identity, the model it grades with, the rubric, and an optional on gate that decides applicability before any LLM call:

name: useful-resolution
llm:
  provider: openai
  model: gpt-5.6-luna
instructions: |
  Return pass when the response moves the request toward a useful resolution.

Recipes → Judges is the authoring reference: the file schema, gate matchers, and worked examples.

Because the definition lives in git, it inherits the recipe’s immutability contract: editing a rubric is a new commit, and every judge is pinned to an exact recipe version. That’s what makes a quality number meaningful: you always know exactly which rubric produced it.

Recipe definition vs. live settings

A judge separates the standard in your recipe from the settings you may change while it runs:

The definition is the intellectual property and the only thing that determines a verdict. Live settings control how you use the judge in production: you can turn it off or reduce how much production traffic it grades without touching Git. Those settings never alter what the judge measures, so they do not disturb its trend line. Lower environments are graded in full; only production traffic is sampled.

Committing a judge file is the opt-in, so a judge is enabled at full sampling from its first deployment and keeps grading across later ones. The two settings are yours from then on: the sync that reads your judges/ directory writes the definition and its lineage and never touches enabled or sample_rate, so a sampling change you make survives every subsequent deploy.

CLI settings cannot edit the Git-owned definition. See CLI → Judges for the live controls and Calibrate judges for the fixture-to-calibration procedure.

When judges run

Judges execute automatically in the runtime sandbox when a conversation completes, after the response has returned but while the sandbox still has time and network available. The sandbox grades the conversation in place and emits the result. There is no separate batch job to schedule and nothing to trigger by hand.

Calibration

Before enabling a judge on live traffic, calibrate it against bounded, human-labeled conversation fixtures. Tune on train and dev, then evaluate the held-out test split without moving labels to improve the score.

Calibration does not create ground truth. It checks whether the exact rubric and model agree with labels your team already owns, which is why a domain owner approves every label and the judge never writes the standard it will be scored against. A judge definition and its calibration dataset are one measurement contract: changing the rubric or the model creates a different measurement and requires another calibration pass.

An invalid trial — a model error, malformed result, or missing context — is neither a pass nor an agent failure, and should be excluded rather than counted.

Reading a disagreement

Read every disagreement rather than counting them. Overall agreement can hide a judge that predicts the majority label every time, and the failures below all produce confident, consistent, wrong verdicts — so none of them appears as noise.

• Length. Longer answers score higher when the rubric does not say what completeness means. If your pass examples are visibly longer than your fail examples, the judge can reach your labels by counting.
• Order. When a judge compares two candidates, the position of each affects the verdict. Score each candidate on its own, or swap the order and keep only verdicts that survive both.
• Familiarity. A judge scores output that reads like its own generation style above output that does not, independently of quality.
• Correlated surface features. Anything that lines up with your labels by accident becomes the shortcut: a status header, a formatting convention, a hedging phrase. Balance the set so the label and the surface feature come apart.

The test is the same in each case: change only the suspected feature and see whether the verdict moves. If it does, the rubric is measuring the feature.

Calibrate judges walks through calibration step by step. See the CLI for every judge-calibration command option.

Judgements

A judgement is one judge’s evaluation of one conversation: the judge analogue of an observation. On the public event surface, a non-null result is the emitted verdict: pass, fail, or not_applicable. A null result exposes no verdict and is not evidence that the conversation failed the rubric. Rubric pass rates use pass and fail; not_applicable and missing verdicts are tracked separately. Judgements are immutable events on the same stream as everything else, and they’re tracked over time so you can do two things:

1. Trend quality across versions. Because every judgement carries the recipe commit it graded, you can watch a judge’s pass-rate move as your agent changes: a before/after across a deploy is a single query. A re-grade under an edited rubric is a new judgement, never a silent overwrite, so versions never blur together.
2. Keep a standing regression guard. Once a judge encodes a failure mode you’ve fixed, leave it running. If the rate regresses on a future version, you see it immediately instead of rediscovering the bug in production.

On the event surface, a judgement row carries its comparability key alongside the verdict:

{
  "event_name": "introspection.judgement",
  "conversation_id": "0198f2c1-4b6d-7f2a-9c3e-5a1d8e0b7c44",
  "payload": {
    "judge_id": "019f6909-a28e-7462-97c2-dbb29f0cf802",
    "definition_hash": "3f9a52e0c1d44b8a9d0e7c5b61a2f4d8e6b0c9a7f13e5d2c4b8a6f0e1d3c5b7a",
    "contract_version": "1",
    "result": "pass",
    "reasoning": "The assistant confirmed the refund policy before promising anything."
  }
}

How it connects

• Recipes own judge definitions: a judge is a recipe artifact, version-pinned to the same git commit as the agent it grades. Recipes → Judges is the authoring reference.
• Tasks & conversations are what judges grade: one judgement per conversation per judge.
• Experiments consume judgements as their scoring signal. Every arm must emit the judge ID and definition hash pinned by the experiment. Confirm that matching evidence also uses one contract_version so the comparison uses one stable evaluation contract.