Experiments

A judge already lets you trend quality across versions, and the simplest comparison needs no experiment at all: deploy a fix and watch the judge’s pass-rate move across deploys. Reach for an experiment when a sequential before/after would be confounded by changing traffic and you want a simultaneous comparison: both versions running at the same time, on the same population.

What an experiment is

An experiment routes live traffic across two or more arms within a single environment lane, then scores them by a goal: typically a candidate recipe against the current baseline, where you ship whichever wins.

Every arm is a runtime within one runtime group, and that shared lineage is what makes the comparison meaningful. Arms are symmetric: there’s no privileged baseline arm in the routing itself. The incumbent is simply whichever runtime currently serves production for the group; traffic outside the experiment is served by it normally and never enters the experiment.

Arms and routing

You define the arms; Introspection routes traffic between them with sticky per-subject assignment. A given subject (a user, an anonymous visitor, or a conversation, resolved in that order of preference) is assigned to one arm and stays on it for the life of the experiment. That stickiness keeps the comparison honest: a subject doesn’t flip between versions mid-relationship and muddy the signal.

Runtime group  (one agent's lineage)
├── Arm: baseline   → runtime rt_control
└── Arm: candidate  → runtime rt_candidate
        │
   incoming traffic ──▶ sticky per-subject assignment ──▶ one arm, for the experiment's life

Arms start at an even split and Introspection reallocates new assignments toward the better-scoring arm as evidence accumulates. On production you bound the experiment’s blast radius with a sample rate: the fraction of the group’s traffic the experiment is served to. Everyone else gets the normal production runtime. Lower lanes always run at full traffic.

Scoring by a goal

An experiment is scored by a goal, typically a judge’s pass-rate. You point the goal at a judge, and the platform reads that judge’s verdicts over each arm’s traffic to compute reward. Because a judge bound to the experiment’s runtime group already grades every arm automatically, scoring needs no extra wiring: the same judgement stream that trends your quality also scores your experiment, grouped by arm instead of by version.

A goal can carry a guard: a bound a component must stay within. An arm that breaches its guard is stopped regardless of its reward, so a candidate that wins on the headline metric but regresses on a safety bound can’t run away with the traffic.

The result is advisory: Introspection surfaces which arm is likely best and whether there’s enough evidence to call it, but it never ships on its own. You decide, and you ship by merging the winning recipe.

Lifecycle

An experiment moves through an explicit lifecycle:

Ending an experiment is not a deployment and does not choose or store a winner. The operator selects a version from the evidence, and production changes only when that recipe commit reaches the integrated repository’s main branch. Once an experiment is running, its arms and goal are frozen so accumulated evidence keeps measuring a single, stable objective.

Managing experiments

The CLI accepts the exact ExperimentCreate JSON or YAML document and exposes every lifecycle transition:

introspection experiments create --file experiment.yaml
introspection experiments start <experiment-id>
introspection experiments get <experiment-id>
introspection experiments end <experiment-id>
# or: introspection experiments cancel <experiment-id>

See CLI → Experiments and the API Reference for the scripted surfaces.

How it connects

• Runtimes are the arms: each arm is a runtime version within one runtime group, and the experiment runs inside one environment lane.
• Judges are the scoring signal: an experiment’s goal reads a judge’s pass-rate over each arm.
• Recipes are what you ship: choose from the experiment evidence, then merge that recipe commit to move production.