# Judges

A judge encodes one durable behavioral standard, then measures completed production conversations consistently across recipe versions.

Observations tell you what the platform noticed. A judge lets you define what
good behavior looks like, check it consistently on real conversations, and see
whether that behavior improves or regresses over time.

## What a judge is

A **judge** is a code-first, LLM-as-judge rubric. An LLM reads a completed [conversation](/platform/conversations) and grades it against your instructions, returning one of a fixed verdict vocabulary:

- **`pass`**: the conversation met the standard.
- **`fail`**: it didn't.
- **`not_applicable`**: the judge doesn't apply to this conversation.

You write only the rubric's instructions and the set of verdicts is fixed by the platform. The model is always asked for a reasoning plus a verdict from that closed set. A response the model can't grade surfaces as an explicit `error`.

Judges are **LLM trajectory judges**: they grade semantic questions like
"did the assistant clarify before researching?" or "was the answer actually
helpful?". They're for nuanced, judgment-based standards, not deterministic
structural assertions.

## Judges are git-owned

A judge belongs with the agent it grades, so it's authored as a YAML file in your [recipe's](/recipes) git repository, under `judges/`:

```text
customer-support/
├── SYSTEM.md
├── agents/*.yaml
├── skills/**/SKILL.md
└── judges/*.yaml      # one file per judge
```

A judge file declares its identity, the model it grades with, and the rubric.
It can also include an optional **`on` gate** that decides whether the judge is
relevant before making an LLM call.

`name` identifies the judge within the recipe and must be lowercase
kebab-case: ASCII letters, digits, and single hyphens. `introspection check`
validates judges along with the rest of the package.

**With an on gate:**

```yaml
name: billing-request-handling
description: "Did the assistant handle a billing request correctly?"

on:
  - event: message
    match:
      role: user
      text: /refund|invoice/i

llm:
  provider: openai
  model: gpt-5.6-luna

instructions: |
  Grade the assistant's handling of the billing request.

  Return pass when the response follows the billing policy, asks only for
  information needed to proceed, and gives a clear next step.

  Return fail when it makes an unsupported promise, exposes sensitive payment
  information, or leaves the user without a useful next step.

  Return not_applicable when the matched message is not actually about billing.
  Output { reasoning, verdict }.
```

**Without a gate:**

```yaml
name: useful-resolution
description: "Did the response move the request toward a useful resolution?"

llm:
  provider: openai
  model: gpt-5.6-luna

instructions: |
  Grade the assistant's response to the user's request.

  Return pass when it answers the request, asks for information genuinely
  needed to proceed, or clearly explains an appropriate next step.

  Return fail when it avoids the request, makes an unsupported claim, or leaves
  the user without a useful next step.

  Return not_applicable when the conversation contains no assistant response
  that can be graded.
  Output { reasoning, verdict }.
```

The gated judge runs only when a user message matches both fields in its
matcher: `role` is `user` and `text` matches the regex. A judge can list several
`message`, `tool`, or `feedback` matchers; matching any one of them makes the
judge applicable. When none match, the judge does not run and no judgement is
emitted. `not_applicable` is a verdict a judge may emit after it runs; a gate
miss does not create that verdict.

The ungated judge evaluates every completed conversation selected for judging.
Its rubric therefore owns the full applicability decision and should explain
when to return `not_applicable`.

Because the definition lives in git, it inherits the recipe's immutability contract: editing a rubric is a new commit, and every judge is pinned to an exact recipe version. That's what makes a quality number meaningful: you always know exactly which rubric produced it.

## Recipe definition vs. live settings

A judge separates the standard in your recipe from the settings you may change
while it runs:

| Part | Where it lives | Examples |
| --- | --- | --- |
| **Definition** | Git (pinned to a recipe version) | the rubric `instructions`, the `llm` configuration, the `on` gate, the judge's name |
| **Live settings** | Introspection | `enabled` (an off-switch), `sample_rate` (how often to grade production traffic) |

The definition is the intellectual property and the only thing that determines a verdict. Live settings control how you use the judge in production: you can turn it off or reduce how much production traffic it grades without touching Git. Those settings never alter what the judge measures, so they do not disturb its trend line. Lower environments are graded in full; only production traffic is sampled.

CLI settings cannot edit the Git-owned definition. See [CLI → Judges](/sdk/cli#judges)
for the live controls and [Learn from production →
calibrating a judge](/guides/learn-from-production#7-turn-a-durable-risk-into-a-judge)
for the fixture-to-calibration procedure.

## When judges run

Judges execute **automatically in the runtime sandbox when a conversation completes**, after the response has returned but while the sandbox still has time and network available. The sandbox grades the conversation in place and emits the result. There is no separate batch job to schedule and nothing to trigger by hand.

## Calibration

Before enabling a judge on live traffic, calibrate it against bounded,
human-labeled conversation fixtures. Tune on `train` and `dev`, then evaluate
the held-out `test` split without moving labels to improve the score.

Calibration does not create ground truth. It checks whether the exact rubric and
model agree with labels your team already owns, which is why a domain owner
approves every label and the judge never writes the standard it will be scored
against. A judge definition and its calibration dataset are one measurement
contract: changing the rubric or the model creates a different measurement and
requires another calibration pass.

An invalid trial — a model error, malformed result, or missing context — is
neither a pass nor an agent failure, and should be excluded rather than counted.

### Reading a disagreement

Read every disagreement rather than counting them. Overall agreement can hide a
judge that predicts the majority label every time, and the failures below all
produce confident, consistent, wrong verdicts — so none of them appears as
noise.

- **Length.** Longer answers score higher when the rubric does not say what
  completeness means. If your `pass` examples are visibly longer than your
  `fail` examples, the judge can reach your labels by counting.
- **Order.** When a judge compares two candidates, the position of each affects
  the verdict. Score each candidate on its own, or swap the order and keep only
  verdicts that survive both.
- **Familiarity.** A judge scores output that reads like its own generation
  style above output that does not, independently of quality.
- **Correlated surface features.** Anything that lines up with your labels by
  accident becomes the shortcut: a status header, a formatting convention, a
  hedging phrase. Balance the set so the label and the surface feature come
  apart.

The test is the same in each case: change only the suspected feature and see
whether the verdict moves. If it does, the rubric is measuring the feature.

[Learn from production](/guides/learn-from-production#7-turn-a-durable-risk-into-a-judge)
walks through calibration step by step. See [CLI → Judge
calibration](/sdk/cli#judge-calibration) for every command option.

## Judgements

A **judgement** is one judge's evaluation of one conversation: the judge analogue of an observation. On the public event surface, a non-null `result` is the emitted verdict: `pass`, `fail`, or `not_applicable`. A null result exposes no verdict and is not evidence that the conversation failed the rubric. Rubric pass rates use `pass` and `fail`; `not_applicable` and missing verdicts are tracked separately. Judgements are **immutable** events on the same stream as everything else, and they're tracked over time so you can do two things:

The Judgements view tracks each standard over time and compares the same judge across services.

1. **Trend quality across versions.** Because every judgement carries the recipe commit it graded, you can watch a judge's pass-rate move as your agent changes: a before/after across a deploy is a single query. A re-grade under an edited rubric is a *new* judgement, never a silent overwrite, so versions never blur together.
2. **Keep a standing regression guard.** Once a judge encodes a failure mode you've fixed, leave it running. If the rate regresses on a future version, you see it immediately instead of rediscovering the bug in production.

Compare judgements only when the stable judge ID, definition hash, and
`contract_version` match. The judge ID follows the standard across edits, the
definition hash pins the authored rubric file, and `contract_version`
identifies the judgement contract that produced the event. Split a trend
when the definition hash or contract version changes.

Judge IDs, definition hashes, and contract versions together determine which
judgements form a comparable set. Missing verdicts are excluded from rubric
rates and tracked separately; they reduce coverage but do not change the
meaning of valid verdicts in that set.

## How it connects

- **[Recipes](/recipes)** own judge definitions: a judge is a recipe artifact, version-pinned to the same git commit as the agent it grades.
- **[Tasks & conversations](/platform/tasks-and-runs)** are what judges grade: one judgement per conversation per judge.
- **[Experiments](/platform/experiments)** consume judgements as their scoring
  signal. Every arm must emit the judge ID and definition hash pinned by the
  experiment. Confirm that matching evidence also uses one `contract_version`
  so the comparison uses one stable evaluation contract.

Judges

A judge encodes one durable behavioral standard, then measures completed production conversations consistently across recipe versions.

Observations tell you what the platform noticed. A judge lets you define what good behavior looks like, check it consistently on real conversations, and see whether that behavior improves or regresses over time.

What a judge is

A judge is a code-first, LLM-as-judge rubric. An LLM reads a completed conversation and grades it against your instructions, returning one of a fixed verdict vocabulary:

pass: the conversation met the standard.
fail: it didn’t.
not_applicable: the judge doesn’t apply to this conversation.

You write only the rubric’s instructions and the set of verdicts is fixed by the platform. The model is always asked for a reasoning plus a verdict from that closed set. A response the model can’t grade surfaces as an explicit error.

Judges are LLM trajectory judges: they grade semantic questions like “did the assistant clarify before researching?” or “was the answer actually helpful?”. They’re for nuanced, judgment-based standards, not deterministic structural assertions.

Judges are git-owned

A judge belongs with the agent it grades, so it’s authored as a YAML file in your recipe’s git repository, under judges/:


customer-support/
├── SYSTEM.md
├── agents/*.yaml
├── skills/**/SKILL.md
└── judges/*.yaml      # one file per judge

A judge file declares its identity, the model it grades with, and the rubric. It can also include an optional on gate that decides whether the judge is relevant before making an LLM call.

name identifies the judge within the recipe and must be lowercase kebab-case: ASCII letters, digits, and single hyphens. introspection check validates judges along with the rest of the package.

With an on gate


name: billing-request-handling
description: "Did the assistant handle a billing request correctly?"
 
on:
  - event: message
    match:
      role: user
      text: /refund|invoice/i
 
llm:
  provider: openai
  model: gpt-5.6-luna
 
instructions: |
  Grade the assistant's handling of the billing request.
 
  Return pass when the response follows the billing policy, asks only for
  information needed to proceed, and gives a clear next step.
 
  Return fail when it makes an unsupported promise, exposes sensitive payment
  information, or leaves the user without a useful next step.
 
  Return not_applicable when the matched message is not actually about billing.
  Output { reasoning, verdict }.

The gated judge runs only when a user message matches both fields in its matcher: role is user and text matches the regex. A judge can list several message, tool, or feedback matchers; matching any one of them makes the judge applicable. When none match, the judge does not run and no judgement is emitted. not_applicable is a verdict a judge may emit after it runs; a gate miss does not create that verdict.

The ungated judge evaluates every completed conversation selected for judging. Its rubric therefore owns the full applicability decision and should explain when to return not_applicable.

Because the definition lives in git, it inherits the recipe’s immutability contract: editing a rubric is a new commit, and every judge is pinned to an exact recipe version. That’s what makes a quality number meaningful: you always know exactly which rubric produced it.

Recipe definition vs. live settings

A judge separates the standard in your recipe from the settings you may change while it runs:

Part	Where it lives	Examples
Definition	Git (pinned to a recipe version)	the rubric `instructions`, the `llm` configuration, the `on` gate, the judge’s name
Live settings	Introspection	`enabled` (an off-switch), `sample_rate` (how often to grade production traffic)

The definition is the intellectual property and the only thing that determines a verdict. Live settings control how you use the judge in production: you can turn it off or reduce how much production traffic it grades without touching Git. Those settings never alter what the judge measures, so they do not disturb its trend line. Lower environments are graded in full; only production traffic is sampled.

CLI settings cannot edit the Git-owned definition. See CLI → Judges for the live controls and Learn from production → calibrating a judge for the fixture-to-calibration procedure.

When judges run

Judges execute automatically in the runtime sandbox when a conversation completes, after the response has returned but while the sandbox still has time and network available. The sandbox grades the conversation in place and emits the result. There is no separate batch job to schedule and nothing to trigger by hand.

Calibration

Before enabling a judge on live traffic, calibrate it against bounded, human-labeled conversation fixtures. Tune on train and dev, then evaluate the held-out test split without moving labels to improve the score.

Calibration does not create ground truth. It checks whether the exact rubric and model agree with labels your team already owns, which is why a domain owner approves every label and the judge never writes the standard it will be scored against. A judge definition and its calibration dataset are one measurement contract: changing the rubric or the model creates a different measurement and requires another calibration pass.

An invalid trial — a model error, malformed result, or missing context — is neither a pass nor an agent failure, and should be excluded rather than counted.

Reading a disagreement

Read every disagreement rather than counting them. Overall agreement can hide a judge that predicts the majority label every time, and the failures below all produce confident, consistent, wrong verdicts — so none of them appears as noise.

Length. Longer answers score higher when the rubric does not say what completeness means. If your pass examples are visibly longer than your fail examples, the judge can reach your labels by counting.
Order. When a judge compares two candidates, the position of each affects the verdict. Score each candidate on its own, or swap the order and keep only verdicts that survive both.
Familiarity. A judge scores output that reads like its own generation style above output that does not, independently of quality.
Correlated surface features. Anything that lines up with your labels by accident becomes the shortcut: a status header, a formatting convention, a hedging phrase. Balance the set so the label and the surface feature come apart.

The test is the same in each case: change only the suspected feature and see whether the verdict moves. If it does, the rubric is measuring the feature.

Learn from production walks through calibration step by step. See CLI → Judge calibration for every command option.

Judgements

A judgement is one judge’s evaluation of one conversation: the judge analogue of an observation. On the public event surface, a non-null result is the emitted verdict: pass, fail, or not_applicable. A null result exposes no verdict and is not evidence that the conversation failed the rubric. Rubric pass rates use pass and fail; not_applicable and missing verdicts are tracked separately. Judgements are immutable events on the same stream as everything else, and they’re tracked over time so you can do two things:

Trend quality across versions. Because every judgement carries the recipe commit it graded, you can watch a judge’s pass-rate move as your agent changes: a before/after across a deploy is a single query. A re-grade under an edited rubric is a new judgement, never a silent overwrite, so versions never blur together.
Keep a standing regression guard. Once a judge encodes a failure mode you’ve fixed, leave it running. If the rate regresses on a future version, you see it immediately instead of rediscovering the bug in production.

Compare judgements only when the stable judge ID, definition hash, and contract_version match. The judge ID follows the standard across edits, the definition hash pins the authored rubric file, and contract_version identifies the judgement contract that produced the event. Split a trend when the definition hash or contract version changes.

Judge IDs, definition hashes, and contract versions together determine which judgements form a comparable set. Missing verdicts are excluded from rubric rates and tracked separately; they reduce coverage but do not change the meaning of valid verdicts in that set.

How it connects

Recipes own judge definitions: a judge is a recipe artifact, version-pinned to the same git commit as the agent it grades.
Tasks & conversations are what judges grade: one judgement per conversation per judge.
Experiments consume judgements as their scoring signal. Every arm must emit the judge ID and definition hash pinned by the experiment. Confirm that matching evidence also uses one contract_version so the comparison uses one stable evaluation contract.