# Calibrate a judge

Teach a model to grade one quality the way your domain experts would, then use it in evals and on live traffic.

A **judge** is `judges/<name>.yaml` in your recipe: one question a model
answers about a conversation with `pass`, `fail`, or `not_applicable`. It runs
after the reply and never blocks it. **Calibration** checks that it agrees with
labels your experts approved. A grader that fails records `error` (status
`error`), which pass rates exclude. Use a judge for tone, grounding, or policy; use
code for anything deterministic.

## 1. Scaffold

```bash
introspection eval create refund-grounding --with-judge
```

This writes `judges/refund-grounding.yaml` with an empty working set
(`.eval.jsonl`) and held-out set (`.validate.jsonl`), and a Vitest suite in
`evals/vitest/` that uses the judge.

## 2. Write the rubric

```yaml filename="judges/refund-grounding.yaml"
name: refund-grounding
description: Does the refund decision cite the policy that controls it?
on:
  - event: message
    match: { role: user, text: /refund/i }
llm:
  provider: openai
  model: <model>
instructions: |
  Return pass when the decision names the policy clause it relies on.
  Return fail when it decides without one or cites the wrong clause.
  Return not_applicable when no refund decision was made.
```

`on` gates which conversations it grades. The scaffold's `llm.local` block
sets the model endpoint for local runs: delete it for `openai` or `openrouter`
(they read `OPENAI_API_KEY` or `OPENROUTER_API_KEY`), or fill in `base_url` and
`api_key_env` to use another OpenAI-compatible endpoint. All fields:
[CLI reference](/reference/cli/evals).

## 3. Label examples

One production turn per command, with a verdict a person chose:

```bash
introspection judges add refund-grounding --trace-id <trace-id> --expected fail
introspection judges add refund-grounding --trace-id <trace-id> --expected pass --validate
```

`--expected` takes `pass`, `fail`, or `not-applicable`. `--validate` writes
the held-out file. Aim for 20 to 40 working and 10 to 20
held-out examples, clear and borderline, from a [dataset](/improve/annotate)
or negative feedback. To list a dataset's trace IDs, then add each one:

```bash
introspection annotations list --label refund-mistakes --page-all --query "[].trace_id" -o table
```

## 4. Check agreement

```bash
export OPENAI_API_KEY=...        # or OPENROUTER_API_KEY, or llm.local.api_key_env
introspection judges eval refund-grounding
```

It reports accuracy, Cohen's kappa (near 0 means no better than chance), and
the rates at which it catches real failures and passes real passes. Read every
disagreement, fix the rubric rather than the labels, and repeat. Include
`fail` examples: without them the catch rate (and often kappa) is null, which
means not measurable, not perfect.

## 5. Validate once

```bash
introspection judges validate refund-grounding
```

Run it once, after freezing the rubric and model. It prints no pass mark,
so set your bar first. If it misses, revise and build new
held-out examples.

## 6. Use it in evals and in production

In a suite: `judges: [recipeJudge("refund-grounding")]`
([scoring](/improve/evals#2-score-each-case)). A `not_applicable` verdict
fails that case with an error, so gate cases to the judge's subject.

In production: commit the YAML and both JSONL files and
[ship](/ship). `introspection judges list` shows the
judge once the version builds. Each runtime gets its own copy of the judge, enabled with
sample rate 1.0. The sample rate applies to production; staging is always
fully graded. Later deploys keep your settings.

```bash
introspection judges list                                  # find the judge id per runtime
introspection judges update <judge-id> --sample-rate 0.25
introspection judges update <judge-id> --disable           # or --enable
```

Model keys for live judges

On a managed runtime, live judges run through Introspection's model gateway.
On a byok runtime they call the provider in the judge's `llm` block with your
key, so bind an `llm` endpoint for that host.

## Verify

`introspection judges get <judge-id> -o json` shows `enabled: true` and your
`sample_rate`. Read recent failures:

```bash
introspection events list --event-name introspection.judgement --filter judge_id=<judge-id> --lookback 7d \
  --query "[?payload.result=='fail'].{conversation:conversation_id,reason:payload.reasoning}" -o table
```

**Judges** in the sidebar charts pass rate per runtime. To re-grade a past
conversation with your local YAML after a rubric change (with your own key;
results land in `development`):
`introspection judges run --task-id <task-id> --judge refund-grounding`. Ship
the judge first: otherwise it's listed under `skipped` with
`reason: unregistered`, and the command still exits 0.

## Limits

| Not supported | Do this instead |
| --- | --- |
| Blocking a reply on a judge | Validate in a `tool_call` hook or a submit tool's schema. |
| Matching task metadata or tags in `on` | Disable the judge on runtimes it shouldn't grade. |

Or ask your agent: "Calibrate the refund-grounding judge on production examples I approve, then grade 10% of production."

Calibrate a judge

Teach a model to grade one quality the way your domain experts would, then use it in evals and on live traffic.

A judge is judges/<name>.yaml in your recipe: one question a model answers about a conversation with pass, fail, or not_applicable. It runs after the reply and never blocks it. Calibration checks that it agrees with labels your experts approved. A grader that fails records error (status error), which pass rates exclude. Use a judge for tone, grounding, or policy; use code for anything deterministic.

1. Scaffold


introspection eval create refund-grounding --with-judge

This writes judges/refund-grounding.yaml with an empty working set (.eval.jsonl) and held-out set (.validate.jsonl), and a Vitest suite in evals/vitest/ that uses the judge.

2. Write the rubric

judges/refund-grounding.yaml


name: refund-grounding
description: Does the refund decision cite the policy that controls it?
on:
  - event: message
    match: { role: user, text: /refund/i }
llm:
  provider: openai
  model: <model>
instructions: |
  Return pass when the decision names the policy clause it relies on.
  Return fail when it decides without one or cites the wrong clause.
  Return not_applicable when no refund decision was made.

on gates which conversations it grades. The scaffold’s llm.local block sets the model endpoint for local runs: delete it for openai or openrouter (they read OPENAI_API_KEY or OPENROUTER_API_KEY), or fill in base_url and api_key_env to use another OpenAI-compatible endpoint. All fields: CLI reference.

3. Label examples

One production turn per command, with a verdict a person chose:


introspection judges add refund-grounding --trace-id <trace-id> --expected fail
introspection judges add refund-grounding --trace-id <trace-id> --expected pass --validate

--expected takes pass, fail, or not-applicable. --validate writes the held-out file. Aim for 20 to 40 working and 10 to 20 held-out examples, clear and borderline, from a dataset or negative feedback. To list a dataset’s trace IDs, then add each one:


introspection annotations list --label refund-mistakes --page-all --query "[].trace_id" -o table

4. Check agreement


export OPENAI_API_KEY=...        # or OPENROUTER_API_KEY, or llm.local.api_key_env
introspection judges eval refund-grounding

It reports accuracy, Cohen’s kappa (near 0 means no better than chance), and the rates at which it catches real failures and passes real passes. Read every disagreement, fix the rubric rather than the labels, and repeat. Include fail examples: without them the catch rate (and often kappa) is null, which means not measurable, not perfect.

5. Validate once


introspection judges validate refund-grounding

Run it once, after freezing the rubric and model. It prints no pass mark, so set your bar first. If it misses, revise and build new held-out examples.

6. Use it in evals and in production

In a suite: judges: [recipeJudge("refund-grounding")] (scoring). A not_applicable verdict fails that case with an error, so gate cases to the judge’s subject.

In production: commit the YAML and both JSONL files and ship. introspection judges list shows the judge once the version builds. Each runtime gets its own copy of the judge, enabled with sample rate 1.0. The sample rate applies to production; staging is always fully graded. Later deploys keep your settings.


introspection judges list                                  # find the judge id per runtime
introspection judges update <judge-id> --sample-rate 0.25
introspection judges update <judge-id> --disable           # or --enable

Model keys for live judges

On a managed runtime, live judges run through Introspection’s model gateway. On a byok runtime they call the provider in the judge’s llm block with your key, so bind an llm endpoint for that host.

Verify

introspection judges get <judge-id> -o json shows enabled: true and your sample_rate. Read recent failures:


introspection events list --event-name introspection.judgement --filter judge_id=<judge-id> --lookback 7d \
  --query "[?payload.result=='fail'].{conversation:conversation_id,reason:payload.reasoning}" -o table

Judges in the sidebar charts pass rate per runtime. To re-grade a past conversation with your local YAML after a rubric change (with your own key; results land in development): introspection judges run --task-id <task-id> --judge refund-grounding. Ship the judge first: otherwise it’s listed under skipped with reason: unregistered, and the command still exits 0.

Limits

Not supported	Do this instead
Blocking a reply on a judge	Validate in a `tool_call` hook or a submit tool’s schema.
Matching task metadata or tags in `on`	Disable the judge on runtimes it shouldn’t grade.

Or ask your agent: “Calibrate the refund-grounding judge on production examples I approve, then grade 10% of production.”