Experiments
A judge already lets you trend quality across versions, and the simplest comparison needs no experiment at all: deploy a fix and watch the judge’s pass-rate move across deploys. Reach for an experiment when a sequential before/after would be confounded by changing traffic and you want a simultaneous comparison: both versions running at the same time, on the same population.
What an experiment is
An experiment routes live traffic across two or more arms within one environment lane, then scores them by a goal. A common setup compares a candidate recipe with the current production version.
Every arm is a runtime within one runtime group, and that shared lineage is what makes the comparison meaningful. Arms are symmetric: there’s no privileged baseline arm in the routing itself. The incumbent is simply whichever runtime currently serves production for the group; traffic outside the experiment is served by it normally and never enters the experiment.
Arms and routing
You define the arms; Introspection routes traffic between them with sticky per-subject assignment. A subject (a user, an anonymous visitor, or a conversation, resolved in that order) keeps its assigned arm while that arm remains eligible for traffic. If an arm is stopped or its allocation reaches zero, the platform may reassign the subject to another eligible arm. This prevents new work from continuing on an arm that should no longer receive traffic.
Incoming traffic → sticky per-subject assignment
├── Baseline arm → runtime rt_control
Runtime group └── Candidate arm → runtime rt_candidate
(one agent's lineage)Arms start at an even split and Introspection reallocates new assignments toward the better-scoring arm as evidence accumulates. On production you bound the experiment’s blast radius with a sample rate: the fraction of the group’s traffic the experiment is served to. Everyone else gets the normal production runtime. Lower lanes always run at full traffic.
Run experiments in staging or production. A task that selects a runtime
by group and lane picks up a running experiment in those two lanes only.
Development always resolves to the exact runtime the caller asked for, because
the lane has no deployed environment assignment — so an experiment created on
development reports running and never routes any traffic to its arms. The
only way to reach an arm from development is to name the experiment and subject
explicitly on the task.
An arm can also remap the entry-point agent of a shared runtime, so two arms
can differ only by which agents/*.yaml entrypoint they run, not just by
recipe version.
Scoring by a goal
An experiment is scored by a goal, typically a judge’s pass rate. When you create the experiment, the platform pins the selected judge definition and reads matching judgements for each arm. Make sure every arm emits judgements for that exact definition. An arm with no matching judgements has no reward evidence, so its score cannot update. The platform verifies that arms belong to one runtime group, but it does not verify judge compatibility across the arms for you.
Before interpreting a comparison, also confirm that the matching judgements
carry one contract_version. The definition hash pins the authored rubric;
contract_version identifies the judgement contract that produced the event.
Only a non-null result is a verdict. Treat a null result as incomplete
evaluation data rather than proof that an arm failed the rubric. See
Judgements for the full comparability key.
Evidence from different judge definitions or contract versions is not one comparable set. Null results are excluded from the pass-rate denominator and tracked separately as missing evaluator results. They do not invalidate the remaining comparable non-null verdicts, although they reduce an arm’s valid coverage.
A goal can carry a guard: a bound a component must stay within. An arm that breaches its guard is stopped regardless of its reward, so a candidate that wins on the headline metric but regresses on a safety bound can’t run away with the traffic.
The result is advisory: Introspection shows which arm is likely best and whether there is enough evidence to make a decision. Nothing changes automatically. Review the evidence and merge the chosen recipe when you are confident in it.
Lifecycle
An experiment moves through an explicit lifecycle:
An experiment is created as a draft, which routes nothing. Three transitions move it from there, and two of them are terminal:
| Transition | From | To | What it does |
|---|---|---|---|
| start | draft | running | Makes routing live; the scorer begins updating arm allocation from incoming results. |
| end | running | ended | Concludes evidence collection; no winner is stored. |
| cancel | running | cancelled | Aborts the comparison. |
Running a task against the experiment — picking an arm for a subject and executing — is not a transition. It is ordinary task creation aimed at a running experiment, and it leaves the experiment’s state unchanged.
Ending an experiment is not a deployment and does not choose or store a winner. You select a version from the evidence, merge the chosen recipe through your normal Git workflow, and verify the production lane. Once an experiment is running, its arms and goal are frozen so accumulated evidence keeps measuring a single, stable objective.
At most one experiment can be running per runtime group per environment at a time. End the current one before starting another.
Run an experiment
Before starting, decide what result would make you choose or reject the candidate. Then create the comparison, send identifiable traffic through it, and review the evidence before ending or canceling it. Follow Learn from production → deciding for the steps. CLI → Experiments and the API Reference define the scripted lifecycle.
How it connects
- Runtimes are the arms: each arm is a runtime version within one runtime group, and the experiment runs inside one environment lane.
- Judges are the scoring signal: an experiment’s goal reads a judge’s pass-rate over each arm.
- Recipes contain the change: choose from the experiment evidence, then merge that recipe commit to move production.