Experiments

A judge already lets you trend quality across versions, and the simplest comparison needs no experiment at all: deploy a fix and watch the judge’s pass-rate move across deploys. Reach for an experiment when a sequential before/after would be confounded by changing traffic and you want a simultaneous comparison: both versions running at the same time, on the same population.

What an experiment is

An experiment routes live traffic across two or more arms within one environment lane, then scores them by a goal. A common setup compares a candidate recipe with the current production version.

Every arm is a runtime within one runtime group, and that shared lineage is what makes the comparison meaningful. Arms are symmetric: there’s no privileged baseline arm in the routing itself. The incumbent is simply whichever runtime currently serves production for the group; traffic outside the experiment is served by it normally and never enters the experiment.

Arms and routing

You define the arms; Introspection routes traffic between them with sticky per-subject assignment. A subject (a user, an anonymous visitor, or a conversation, resolved in that order) keeps its assigned arm while that arm remains eligible for traffic. If an arm is stopped or its allocation reaches zero, the platform may reassign the subject to another eligible arm. This prevents new work from continuing on an arm that should no longer receive traffic.

Arms start at an even split and Introspection reallocates new assignments toward the better-scoring arm as evidence accumulates. On production you bound the experiment’s blast radius with a sample rate: the fraction of the group’s traffic the experiment is served to. Everyone else gets the normal production runtime. Lower lanes always run at full traffic.

Scoring by a goal

An experiment is scored by a goal, typically a judge’s pass rate. When you create the experiment, the platform pins the selected judge definition and reads matching judgements for each arm. Make sure every arm emits judgements for that exact definition. An arm with no matching judgements has no reward evidence, so its score cannot update. The platform verifies that arms belong to one runtime group, but it does not verify judge compatibility across the arms for you.

Before interpreting a comparison, confirm the matching judgements form one comparable set — see Judgements for the comparability key and how null results are treated.

A goal can carry a guard: a bound a component must stay within. An arm that breaches its guard is stopped regardless of its reward, so a candidate that wins on the headline metric but regresses on a safety bound can’t run away with the traffic.

The result is advisory: Introspection shows which arm is likely best and whether there is enough evidence to make a decision. Nothing changes automatically. Review the evidence and merge the chosen recipe when you are confident in it.

Lifecycle

An experiment moves through an explicit lifecycle:

An experiment is created as a draft, which routes nothing. Three transitions move it from there, and two of them are terminal:

Running a task against the experiment — picking an arm for a subject and executing — is not a transition. It is ordinary task creation aimed at a running experiment, and it leaves the experiment’s state unchanged.

Ending an experiment is not a deployment and does not choose or store a winner. You select a version from the evidence, merge the chosen recipe through your normal Git workflow, and verify the production lane. Once an experiment is running, its arms and goal are frozen so accumulated evidence keeps measuring a single, stable objective.

Run an experiment

Before starting, decide what result would make you choose or reject the candidate. Then create the comparison, send identifiable traffic through it, and review the evidence before ending or canceling it.

An experiment is created from one document — an ExperimentCreate request, as YAML or JSON — naming the runtime under test, two to twenty arms (each a runtime version of that same group; list them with introspection runtimes versions <runtime-id>, which takes a runtime version id you already hold — not the group slug — and returns the version ids of that version’s group), and a goal whose components reference existing judge ids (from introspection judges list; judges are authored in the recipe repository and synced, never created over the API):

# experiment.yaml
name: shorter-prompt-vs-control
runtime: customer-agent
environment: production   # the lane the experiment routes in (the default)
sample_rate: 0.1          # serve the experiment to 10% of the group's traffic
goal_json:
  kind: composite
  direction: maximize
  components:
    - source: judge
      judge_id: 019f6909-a28e-7462-97c2-dbb29f0cf802
      weight: 1.0
arms:
  - runtime_id: 019f6909-a28e-7462-97c2-dbb29f0cf803
    arm_label: control
  - runtime_id: 019f6909-a28e-7462-97c2-dbb29f0cf804
    arm_label: variant

introspection experiments create @experiment.yaml   # makes a draft; nothing routes yet

Start the draft with the SDK or CLI:

introspection experiments start <experiment-id>

const experiment = client.experiments(experimentId);
await experiment.start();

environment defaults to production, and sample_rate (the in-sample fraction of the group’s traffic, 0.0–1.0) is selectable on production only — lower lanes always run at full traffic. When the evidence is in, introspection experiments end <id> keeps it and finishes the comparison; cancel stops it without one.

Follow Experiments and shipping for the decision procedure. CLI → Experiments and the API Reference define the scripted lifecycle.

How it connects

• Runtimes are the arms: each arm is a runtime version within one runtime group, and the experiment runs inside one environment lane.
• Judges are the scoring signal: an experiment’s goal reads a judge’s pass-rate over each arm.
• Recipes contain the change: choose from the experiment evidence, then merge that recipe commit to move production.