Experiments and shipping

This closes one continual learning loop. You have gathered a production signal, investigated its source, evaluated one focused change, and calibrated a judge when the risk deserved a standing measure. The final step is to distill the validated lesson into the agent recipe and return it to production.

1. Ship when offline evidence decides

If the failure, boundary cases, and controls clearly support the candidate, open a focused pull request and merge it through the normal Git workflow. The development lifecycle covers candidate runtime creation, staging proof, merge, and production verification.

For the refund example, offline evidence should usually decide. The final-sale exclusion is a known policy rule, not a product preference that needs customer traffic to resolve.

An experiment earns its complexity only when:

• credible candidates remain after faithful offline evaluation;
• before-and-after windows would receive materially different traffic;
• exposure to the candidate must be bounded; or
• the decision requires same-population evidence.

If none apply, deploy normally and monitor the calibrated judge by runtime version.

2. Confirm experiment prerequisites

An experiment goal points to an exact deployed judge. Every arm must emit the same judge ID and definition hash.

introspection judges list
introspection judges get <judge-id>

Verify that the judge is enabled and sampling enough traffic to measure the comparison. Then discover baseline and candidate versions from the same runtime group:

introspection runtimes list \
  --runtime refund-exception-agent \
  --output table

for await (const runtime of client.runtimes.list({
  runtime: "refund-exception-agent",
})) {
  const recipe = await client.recipes.get(runtime.recipe_id);
  console.log(runtime.id, recipe.git_commit_sha);
}

Record the exact runtime IDs and recipe commits. Do not compare unrelated runtime groups or definitions that emit different judges.

3. Create a bounded comparison

Suppose two grounded decision procedures remain credible after offline review. Create an experiment document that names the runtime group, production lane, judge goal, sampled fraction, and two immutable arms:

name: refund-exclusion-ordering
runtime: refund-exception-agent
environment: production
sample_rate: 0.2
goal_json:
  kind: composite
  direction: maximize
  components:
    - source: judge
      judge_id: <judge-id>
      weight: 1.0
arms:
  - runtime_id: <baseline-runtime-id>
    arm_label: control
  - runtime_id: <candidate-runtime-id>
    arm_label: variant

Create the draft from the authored document:

introspection experiments create @experiment.yaml

Then start routing with the SDK or CLI:

introspection experiments start <experiment-id>

const experiment = client.experiments(experimentId);
await experiment.start();

The first checkpoint is structural, not statistical:

• the experiment is running in production;
• both arms belong to refund-exception-agent;
• the sample rate matches the approved exposure;
• the goal points to the enabled, calibrated judge; and
• every arm emits the same judge definition hash.

4. Verify routing before reading outcomes

Send a representative task with a stable subject:

introspection tasks create \
  --experiment <experiment-id> \
  --subject customer-operations-001 \
  --prompt "Review order 1842 against the refund policy and draft the next response."

const experiment = client.experiments(experimentId);
const runner = await experiment.run({
  identity: { user_id: "customer-operations-001" },
});
const handle = await runner.tasks.start({
  prompt: "Review order 1842 against the refund policy and draft the next response.",
});

Inspect the returned task. Count it as experiment evidence only when metadata.experiment_arm_id is set. Subjects outside the sampled fraction use normal production and have no experiment arm.

Stable subjects keep sticky assignments. Repeating synthetic prompts can rehearse routing, but it does not substitute for production evidence.

5. Distill the result into the recipe

Read the experiment by arm, verify the traffic and judge definitions are comparable, and inspect representative conversations behind the aggregate. Stop when the approved decision rule is met or when the experiment can no longer answer the question safely.

Turn the selected result into the smallest reviewable recipe change, then ship it through Git. End or cancel the experiment according to whether its result should remain as the completed comparison:

introspection experiments end <experiment-id>
# Or, when the comparison is invalid or unsafe:
introspection experiments cancel <experiment-id>

await experiment.end();
// Or, when the comparison is invalid or unsafe:
await experiment.cancel();

Verify production resolves to the approved merged commit and that new conversations continue emitting the expected judge definition.

Common failure pattern: experimenting instead of deciding

Do not send deterministic policy corrections through live traffic merely because experimentation exists. Experiments are for genuine uncertainty that offline evidence cannot resolve, not a replacement for tests, review, or domain ownership.

Done when

• A clear local winner was merged without an unnecessary experiment.
• Any experiment answers a question that required bounded, same-population production evidence.
• Both arms use immutable versions from one runtime group and the same judge definition.
• Only tasks with an assigned experiment arm were counted.
• The selected version was merged, resolved in production, and verified against new conversations.

Next

• Develop and ship: repeat the normal release path for later recipe changes.
• Learn from production: begin the next loop from real user experience.
• Experiments: use the complete routing, goal, and lifecycle reference.