Evaluating with judges

Turn an observed failure mode into a calibrated, Git-authored rubric and retain the deployed judge identity needed by an experiment.

Observations describe what the platform noticed. A judge encodes what should have happened. Here, the recurring pattern is a customer asking for help and receiving a policy wall instead of a useful resolution.

What a judge produces

A judge reads a completed conversation and returns one verdict from a closed vocabulary:

pass: the conversation met the standard.
fail: it did not.
not_applicable: the rubric does not apply to this conversation; this is an abstention, not a failure.
error: evaluation could not complete. Errors remain visible and never become silent passes.

Each emitted judgement names the conversation, runtime and recipe version, judge identity, and exact definition hash. That lineage is what makes pass-rate comparisons across agent versions defensible.

Definition, execution, and guard

Layer	Where it lives	What changes it
Judge definition	Recipe Git commit	Instructions, model configuration, and optional `on` gate.
Judgement	Immutable event stream	One completed evaluation over one conversation.
Operational state	Live platform configuration	Whether the judge is enabled and how much production traffic it samples.

An optional on gate narrows the judged population—for example, to conversations that called a support tool. When the gate does not match, the result is not_applicable. In staging, judges run at conversation teardown after the user-facing response; in production, the live sample rate controls evaluation cost.

Calibrate before deployment because an LLM judge is still a model. Real conversations plus human pass/fail labels reveal whether the rubric agrees with the standard you intend. Tune on train/dev, then make the merge decision from a held-out test split.

1. Export immutable fixtures

Put one selected conversation ID on each line, with no JSON or header:

conversation-ids.txt


019...
019...

Export at most 20 in one call:


introspection conversations get \
  --ids-file conversation-ids.txt \
  --output-file judge-fixtures.jsonl

The CLI assembles one judge_fixture schema-v1 row per line. Do not rewrite its conversation, engine, or hash fields. For each row, add only:


{"expected":"pass","split":"test"}

Merge those two fields into the existing top-level object. expected must be pass or fail; split may be train, dev, test, or null. Labels are the human standard against which calibration is measured.

2. Write the judge

Create judges/useful-support-resolution.yaml inside the recipe package:


judge: useful_support_resolution
description: Did the response move the support request toward a useful resolution?
llm:
  provider: openai
  model: gpt-4.1-nano
instructions: |
  Grade the assistant's handling of a customer-support request.
 
  Return pass when it gives an actionable answer grounded in approved policy,
  asks for information genuinely needed to proceed, or clearly escalates a
  sensitive request and explains the next step.
 
  Return fail when it only repeats a policy, makes an unsupported promise, or
  leaves the customer without a useful next step.
 
  Return not_applicable when the conversation is not a support request.
  Output { reasoning, verdict }.

The rubric describes observable outcomes rather than preferred wording. That lets it grade many valid support styles without rewarding empty politeness. The field is llm, not model; the judge contract rejects unknown top-level fields.

3. Calibrate locally


export OPENAI_API_KEY='replace-me'
 
introspection judges eval \
  --judge judges/useful-support-resolution.yaml \
  --dataset judge-fixtures.jsonl \
  --split test > judge-report.json
 
jq '{fixture_count, metrics}' judge-report.json

The command exits non-zero if a fixture could not be evaluated. Choose acceptance thresholds before tuning, then revise the rubric against train/dev and evaluate the held-out test split once. Never move test labels merely to improve the score.

4. Push and verify the deployed definition

Commit the judge with the recipe change, push the branch, and open or update its pull request. The GitHub integration creates an immutable preview runtime from the PR head; an ordinary non-main branch push by itself does not. Identify that version, run it explicitly in staging, and wait for the conversation to complete:


introspection runtimes versions "$RUNTIME_ID" -o table
export CANDIDATE_RUNTIME_ID=<new-runtime-id>
 
for attempt in {1..60}; do
  introspection runtimes get "$CANDIDATE_RUNTIME_ID" > candidate-runtime.json
  IMAGE_STATUS=$(jq -r '.image_status' candidate-runtime.json)
  [ "$IMAGE_STATUS" = ready ] && break
  [ "$IMAGE_STATUS" = failed ] && { jq . candidate-runtime.json; exit 1; }
  sleep 5
done
test "$IMAGE_STATUS" = ready
 
introspection tasks create \
  --runtime-id "$CANDIDATE_RUNTIME_ID" \
  --environment staging \
  --subject judge-smoke-1 \
  --prompt "Run a request that matches this judge's on-gate." > judge-task.json
 
export JUDGE_TASK_ID=$(jq -r '.task.id' judge-task.json)
export JUDGE_RUN_ID=$(jq -r '.run.id' judge-task.json)
introspection tasks follow "$JUDGE_TASK_ID" --run "$JUDGE_RUN_ID" --since 0
introspection tasks get "$JUDGE_TASK_ID" > judge-task-final.json
jq -e '.status == "idle" or .status == "completed"' judge-task-final.json

Judging happens asynchronously at conversation teardown. Poll with a bounded window until a matching row exists:


for attempt in {1..24}; do
  introspection events list \
    --event-name introspection.judgement \
    --filter runtime_group_id="$RUNTIME_GROUP_ID" \
    --lookback 24h \
    --page-all > all-judgements.json
  jq --arg runtime "$CANDIDATE_RUNTIME_ID" \
    '[.[] | select(.runtime_id == $runtime)]' all-judgements.json > judgements.json
  [ "$(jq 'length' judgements.json)" -gt 0 ] && break
  sleep 10
done
test "$(jq 'length' judgements.json)" -gt 0
 
jq '[.[] | {
  judge_id: .payload.judge_id,
  definition_hash: .payload.definition_hash,
  result: .payload.result,
  conversation_id
}]' judgements.json
 
jq '[.[] | {judge_id: .payload.judge_id, definition_hash: .payload.definition_hash}] | unique' judgements.json
export JUDGE_ID=$(jq -r '.[0].payload.judge_id' judgements.json)
export JUDGE_DEFINITION_HASH=$(jq -r '.[0].payload.definition_hash' judgements.json)
test "$JUDGE_ID" != null
test "$JUDGE_DEFINITION_HASH" != null

If no row appears, first verify that the prompt matched the on gate and the task settled; then poll again. The first-row extraction is safe only when this recipe version has one enabled judge. If the unique list contains several definitions, the current CLI cannot map a judge name to its platform ID; resolve the intended definition in the dashboard or API. Retain both values so the experiment scores one exact rubric.

The CLI currently has local judges eval, but no judge registry or commands for enabled and production sample_rate. A judgement event is the only CLI path to discover a synced judge_id. Use the dashboard or API to manage the standing guard’s live operational state.

Done when

The held-out evaluation exits 0 and meets the preselected thresholds.
A deployed introspection.judgement names the candidate runtime, judge ID, and definition hash.
The result is plausible for the triggering conversation.

Experimenting & shipping: use the pinned judge definition as the experiment goal.

Judges: verdict and run-gate semantics.
CLI → Judge calibration: providers and local endpoints.