Experimenting & shipping

Use an experiment when sequential before/after windows would be confounded. Otherwise, ship the candidate and compare the same pinned judge across runtime versions.

When an experiment earns its complexity

Use a simultaneous experiment when:

• the before and after windows would see different traffic mixes;
• the candidate is risky enough that exposure should be bounded;
• or the decision needs same-population evidence rather than a directional trend.

If those conditions do not hold, a normal deploy plus a judge pass-rate split by runtime version is simpler and easier to explain.

How the comparison works

Every arm is an immutable runtime version from one runtime group and one environment lane:

Runtime group (one agent lineage)
├── Arm: baseline   → current runtime version
└── Arm: candidate  → proposed runtime version
        │
   eligible traffic ──▶ sticky subject assignment ──▶ one arm

Arms are symmetric: the experiment does not have a privileged control arm or operator-authored arm weights. Assignment is sticky by subject, so one user does not switch versions mid-relationship. On production, sample_rate sets the experiment’s total blast radius; subjects outside the sample continue to receive normal production and are not experiment evidence. Lower environments run at full sample.

The goal turns judgements into reward. A pinned judge ID and definition hash keep every accumulated result on one measurement standard. As evidence arrives, the platform updates each arm’s posterior and routing weights. Those results are advisory: the platform helps quantify the comparison, but it never merges code or changes production on its own.

Three decisions remain deliberately separate:

end experiment evidence collection
  → choose the version supported by the evidence
  → merge that recipe commit to activate production

end stores no winner and performs no deployment. That separation keeps “the test stopped,” “we selected a version,” and “production changed” independently auditable.

This guide expects RUNTIME_GROUP_ID, JUDGE_ID, and JUDGE_DEFINITION_HASH from the earlier guides.

1. Select two runtime versions

introspection runtimes versions "$RUNTIME_ID" > runtime-versions.json
jq '[.[] | {id,recipe_id,recipe_ref,environments,image_status}]' runtime-versions.json
 
export BASELINE_RUNTIME_ID=<current-production-runtime-id>
export CANDIDATE_RUNTIME_ID=<candidate-runtime-id>

Both IDs must belong to RUNTIME_GROUP_ID, have a ready image, and carry the bindings required in the experiment’s environment.

2. Create a valid experiment document

Save experiment.yaml, substituting the retained IDs:

name: support-agent-candidate
description: Test the candidate recipe against the production version.
runtime_group_id: REPLACE_RUNTIME_GROUP_ID
environment: staging
sample_rate: 1.0
arms:
  - runtime_id: REPLACE_BASELINE_RUNTIME_ID
    arm_label: baseline
  - runtime_id: REPLACE_CANDIDATE_RUNTIME_ID
    arm_label: candidate
goal_json:
  kind: composite
  direction: maximize
  components:
    - source: judge
      judge_id: REPLACE_JUDGE_ID
      judge_definition_hash: REPLACE_JUDGE_DEFINITION_HASH
      weight: 1.0

Arms are symmetric. Do not add control_arm_label, recipe IDs, or authored arm weights; those are not fields in ExperimentCreate. This staging example routes every explicit test subject, making the result easy to inspect. For a live production comparison, change the environment before creation and choose sample_rate from 0.10 to 1.0; then verify .task.metadata.experiment_arm_id is non-null because out-of-sample subjects use normal production and are not experiment evidence.

introspection experiments create --file experiment.yaml > experiment.json
export EXPERIMENT_ID=$(jq -r '.id' experiment.json)
test "$EXPERIMENT_ID" != null
introspection experiments start "$EXPERIMENT_ID"
introspection experiments get "$EXPERIMENT_ID" \
  --query '{id:id,status:status,sample_rate:sample_rate,arms:arms}'

Success means status is running and the response contains the two intended runtime IDs and the pinned judge definition.

3. Drive attributable traffic

An explicit experiment run requires a stable subject. Reusing a subject preserves its arm assignment; use distinct real identities for the experiment population.

import { IntrospectionClient } from "@introspection-sdk/introspection-node";
 
const client = new IntrospectionClient({
  token: process.env.INTROSPECTION_TOKEN,
});
 
const runner = await client.experiments(EXPERIMENT_ID).run({
  identity: { user_id: "user-001" },
});
 
const handle = await runner.tasks.start({
  prompt: "Help me resolve my support request.",
});
 
for await (const event of handle.stream()) {
  console.log(event.event, event.data);
}

introspection tasks create \
  --experiment "$EXPERIMENT_ID" \
  --subject user-001 \
  --prompt "Help me resolve my support request." > experiment-task.json
 
export EXPERIMENT_TASK_ID=$(jq -r '.task.id' experiment-task.json)
export EXPERIMENT_RUN_ID=$(jq -r '.run.id' experiment-task.json)
test "$(jq -r '.task.metadata.experiment_arm_id // empty' experiment-task.json)" != ""
introspection tasks follow "$EXPERIMENT_TASK_ID" --run "$EXPERIMENT_RUN_ID" --since 0
introspection tasks get "$EXPERIMENT_TASK_ID" > experiment-task-final.json
jq -e '.status == "idle" or .status == "completed"' experiment-task-final.json

Repeat with representative subjects and prompts. Synthetic repetition is useful for a staging rehearsal, but is not production evidence.

4. Inspect evidence and stop

const experiment = await client.experiments.get(EXPERIMENT_ID);
console.log(experiment);
 
for await (const judgement of runner.events.list({
  event_name: "introspection.judgement",
  runtime_group_id: RUNTIME_GROUP_ID,
  lookback: "7d",
})) {
  if (judgement.experiment_id === EXPERIMENT_ID) {
    console.log(judgement.payload.experiment_arm_id, judgement.payload.result);
  }
}
 
await client.experiments(EXPERIMENT_ID).end();
await runner.close();
// Or: await client.experiments(EXPERIMENT_ID).cancel();

introspection experiments get "$EXPERIMENT_ID" \
  --query '{status:status,posterior:posterior_json,weights:weights_json,arms:arms}'
 
introspection events list \
  --event-name introspection.judgement \
  --filter runtime_group_id="$RUNTIME_GROUP_ID" \
  --lookback 7d \
  --page-all > all-experiment-window-judgements.json
 
jq --arg experiment "$EXPERIMENT_ID" \
  '[.[] | select(.experiment_id == $experiment)]' \
  all-experiment-window-judgements.json > experiment-judgements.json
 
jq '[.[] | {
  arm_id: .payload.experiment_arm_id,
  result: .payload.result,
  judge_id: .payload.judge_id,
  definition_hash: .payload.definition_hash
}]' experiment-judgements.json

introspection experiments end "$EXPERIMENT_ID"
# Or: introspection experiments cancel "$EXPERIMENT_ID"

end records only that evidence collection ended. It accepts no winner and changes no production route.

5. Ship through Git, then verify with the CLI

Merge the recipe commit you selected into main. That merge/push is the production action: the connected GitHub integration creates the immutable runtime version and activates it for production. There is no CLI promote command.

After the integration processes the push:

introspection runtimes list --runtime support-agent > production-runtimes.json
jq '[.[] | select(.environments | index("production")) | {
  id,recipe_id,recipe_ref,environments,image_status
}]' production-runtimes.json
 
export PRODUCTION_RECIPE_ID=$(jq -r '[.[] | select(.environments | index("production"))][0].recipe_id' production-runtimes.json)
export PRODUCTION_RUNTIME_ID=$(jq -r '[.[] | select(.environments | index("production"))][0].id' production-runtimes.json)
introspection recipes list --name support-agent > recipes.json
jq --arg id "$PRODUCTION_RECIPE_ID" \
  '.[] | select(.id == $id) | {id,git_ref,git_commit_sha}' recipes.json
 
for attempt in {1..60}; do
  introspection runtimes get "$PRODUCTION_RUNTIME_ID" > production-runtime.json
  IMAGE_STATUS=$(jq -r '.image_status' production-runtime.json)
  [ "$IMAGE_STATUS" = ready ] && break
  [ "$IMAGE_STATUS" = failed ] && { jq . production-runtime.json; exit 1; }
  sleep 5
done
test "$IMAGE_STATUS" = ready
 

Run the production smoke task with the SDK or CLI:

import { IntrospectionClient } from "@introspection-sdk/introspection-node";
 
const production = new IntrospectionClient({
  token: process.env.INTROSPECTION_PRODUCTION_TOKEN,
});
 
const runner = await production.runtimes("support-agent").run({
  identity: { user_id: "production-smoke-1" },
});
const handle = await runner.tasks.start({ prompt: "Reply with the word ready." });
const reply = await handle.text();
if (reply.trim().toLowerCase() !== "ready") throw new Error(`Unexpected reply: ${reply}`);
await runner.close();

introspection tasks create \
  --runtime support-agent \
  --environment production \
  --subject production-smoke-1 \
  --prompt "Reply with the word ready." > production-task.json
 
export PRODUCTION_TASK_ID=$(jq -r '.task.id' production-task.json)
export PRODUCTION_RUN_ID=$(jq -r '.run.id' production-task.json)
introspection tasks follow "$PRODUCTION_TASK_ID" --run "$PRODUCTION_RUN_ID" --since 0
introspection tasks get "$PRODUCTION_TASK_ID" > production-task-final.json
jq -e '.status == "idle" or .status == "completed"' production-task-final.json

Do not call the rollout complete until the active production row names the merged commit and the production smoke run settles. Finally, wait up to four minutes for the same guard definition on that runtime:

for attempt in {1..24}; do
  introspection events list \
    --event-name introspection.judgement \
    --filter runtime_group_id="$RUNTIME_GROUP_ID" \
    --filter environment=production \
    --lookback 24h \
    --page-all > production-window-judgements.json
  jq --arg runtime "$PRODUCTION_RUNTIME_ID" \
     --arg judge "$JUDGE_ID" \
     --arg definition "$JUDGE_DEFINITION_HASH" \
    '[.[] | select(
      .runtime_id == $runtime and
      .environment == "production" and
      .payload.judge_id == $judge and
      .payload.definition_hash == $definition and
      .payload.result != null
    )]' production-window-judgements.json > production-guard.json
  [ "$(jq 'length' production-guard.json)" -gt 0 ] && break
  sleep 10
done
test "$(jq 'length' production-guard.json)" -gt 0

Done when

• The experiment used two versions from one runtime group and one pinned judge definition.
• end or cancel made the experiment terminal without being mistaken for deployment.
• The merged commit is the active production runtime.
• A production task and production judgement verify the shipped behavior and guard.

Related

• Experiments: routing, sampling, and posterior semantics.
• CLI → Experiments: lifecycle commands.
• Environments: Git-driven production activation.