Calibrate judges

This continues Evaluate offline. The candidate now handles the known final-sale case, but production needs an ongoing measure for a broader question: did the agent ground the refund decision in the applicable policy and evidence?

A judge is appropriate only because this quality is recurring, semantic, and important across releases. An exact decision schema or deterministic policy function should remain an ordinary test.

1. Agree on one decision

Write a question reviewers can answer consistently:

Did the agent ground its refund decision in the controlling policy and order evidence, avoid unsupported promises, and escalate ambiguity or protected exceptions?

Agree before authoring the judge on:

• what counts as pass, fail, and not_applicable;
• which ambiguous cases require escalation;
• the cost of a false pass versus a false fail; and
• who owns the final labels.

Calibration does not create ground truth. It measures whether one exact rubric and model agree with labels your domain owners already approved.

2. Select and label real conversations

Choose clear passes, clear fails, near-boundary cases, and ordinary controls from the same traffic mix. Include the original final-sale failure and corrected candidate runs, but do not let one pattern dominate the set.

Before export, confirm each conversation is safe to commit. Remove secrets and sensitive user data, or replay an authorized sanitized case and export that conversation instead.

Export at most 20 conversations per call:

introspection conversations get \
  --ids-file conversation-ids.txt \
  --judge-fixtures \
  --output-file judges/refund-decision-grounding.calibration.jsonl

Every row arrives with "expected": null. A human must add the approved label and optional split:

{"expected":"fail","split":"train"}

Use train while shaping the rubric, dev for repeated checks, and hold test out for the final decision. Preserve every other exported field exactly: the fixture’s snapshot hash covers its judge input, so editing the transcript invalidates the row.

The checkpoint is a reviewable dataset in which every row has an approved label and no unresolved sensitive content.

3. Write the judge beside its dataset

Describe observable outcomes, not preferred wording:

name: refund-decision-grounding
description: Did the agent ground the refund decision in policy and order evidence?
on:
  - event: message
    match:
      role: user
      text: /refund|return|chargeback/i
llm:
  provider: openai
  model: gpt-5.6-luna
instructions: |
  Grade the completed refund conversation.
 
  Return pass when the decision cites the controlling refund policy and order
  evidence, avoids unsupported promises, and escalates fraud, chargebacks,
  conflicting records, high-value exceptions, or policy ambiguity.
 
  Return fail when the decision ignores a controlling exclusion, invents order
  facts, promises that a refund was issued, or fails to escalate a protected
  exception.
 
  Return not_applicable when the conversation is not about a refund or return.
  Output { reasoning, verdict }.

The on gate avoids spending a model call on obviously unrelated traffic. A gate miss emits no judgement; not_applicable is a verdict the judge may return after it runs.

4. Calibrate locally

Tune against train, check repeated revisions against dev, then run the held out split once:

OPENAI_API_KEY=... introspection judges eval \
  --judge judges/refund-decision-grounding.yaml \
  --dataset judges/refund-decision-grounding.calibration.jsonl \
  --split train

Read every disagreement. Do not accept a high aggregate agreement score from a judge that predicts the majority label, fails all boundary cases, or confuses not_applicable with pass.

An acceptable checkpoint looks like:

clear passes: agree
clear fails: agree
boundary disagreements: reviewed and resolved in the rubric
held-out test: run once after the rubric is frozen

If the held-out result misses the agreed criteria, revise the judge and create a new held-out set. Never move or relabel test cases merely to improve the score.

5. Version and enable the measure

Commit the judge and approved dataset together. Changing the rubric or model creates a different measurement and requires another calibration pass. Ship them through the development lifecycle.

A judge starts grading as soon as a runtime version carrying it is deployed: the sync creates its registry row enabled, at a sample_rate of 1.0. Committing the file is the opt-in, so the setting to make here is how much production traffic you want graded:

introspection judges list
introspection judges update <judge-id> --sample-rate 0.25

--enable and --disable are for turning a judge off and back on later; a freshly deployed judge is already on.

Confirm new applicable production conversations emit judgements with the exact judge ID and definition hash you calibrated. This is also the evidence source a later experiment goal must reference.

Common failure pattern: letting the judge define truth

Do not ask a judge to label its own calibration set and then report agreement with those labels. Domain owners define the standard; the judge is only a versioned instrument for applying it consistently.

Done when

• The judge asks one narrow question with an agreed cost for each error type.
• Domain owners reviewed every fixture label.
• Fixtures are sanitized and safe to commit.
• Every disagreement on train and dev was read, not merely counted.
• The held-out split was run only after the rubric was frozen.
• The deployed judge emits the calibrated definition on intended traffic.

Next

• Experiments and shipping: ship the clear winner or point a bounded comparison at this exact live judge.