# Run an experiment

Compare versions of one runtime on the same live traffic when offline evals can't decide between them.

An **experiment** routes part of a runtime's traffic across **arms**, each a
version of that runtime, and scores them with a judge. Use it when evals
leave several candidates credible. With little traffic, use
[evals](/improve/evals).

**You need:** two or more versions of the runtime (a pull request's preview
counts), and an enabled [judge](/improve/judges) on every arm.

## 1. Find the versions and the judge

```bash
introspection runtimes list --runtime support-agent   # version ids
introspection judges list                             # judge ids
```

Every arm must ship the same judge file. An arm built with a different
`judges/` file never scores, and nothing warns you.

## 2. Write the draft

```yaml filename="experiment.yaml"
name: shorter-prompt-vs-control
runtime: support-agent
environment: production
sample_rate: 0.2
goal_json:
  kind: composite
  direction: maximize
  components:
    - source: judge
      judge_id: <judge-id>
      weight: 1.0
      guard: { min: 0.8 }        # optional: stop an arm whose pass rate on this judge falls below 80%
arms:
  - runtime_id: <version-id>
    arm_label: control
  - runtime_id: <version-id>
    arm_label: candidate
```

```bash
introspection experiments create @experiment.yaml
```

A draft routes nothing.

Field rules

| Field | Rule |
| --- | --- |
| `arms` | 2 to 20 versions of the same runtime. `agent_overrides: {<agent>: <other agent>}` lets arms differ by agent within one version. |
| `sample_rate` | Share of traffic that enters the experiment; the arms split that share adaptively. In production 0.10 to 1.0; other environments must use 1.0. |
| `goal_json` | At least one `source: judge` component with `weight` above 0. |
| `environment` | `production` (default), `staging` or `development`. Only production and staging route live traffic into it; a development experiment takes only `--experiment` tasks. |

## 3. Start it and check routing

```bash
introspection experiments start <experiment-id>
introspection tasks create --experiment <experiment-id> --subject test-user-001 \
  --prompt "Review order 1842 against the refund policy."
```

The test task's `metadata.experiment_arm_id` names its arm. Assignment is
sticky per subject: the runner's `user_id`, else `anonymous_id`, else
`conversation_id`. Weights start even and shift toward the better-scoring arm.

Which tasks can enter

Only tasks opened with an identity can enter: `--subject` on the CLI, or the
`identity` your backend passes when it opens a runner. A CLI task without
`--subject` gets the live version. Tasks started by Slack, an automation, or
an app sign-in session never enter. `--experiment` still applies
`sample_rate`: a subject outside the sample gets the live version, so you
can't put one tenant in by hand (use a runtime per tenant).

## 4. Read results

```bash
introspection experiments get <experiment-id>      # arms, posterior_json, weights_json
```

`posterior_json`, keyed by arm id: `p_best` (chance it is best), `n_total` (judged
conversations), `stopped` (guard tripped). A clear winner is one arm at
`p_best` ≥ 0.95 with every live arm at ≥ 100 judged conversations.

Per-arm judgement counts: save this as `arm-judgements.json`, then run
`introspection metrics query @arm-judgements.json -o table`.

```json filename="arm-judgements.json"
{"view":"judgements","metrics":[{"aggregation":"count"}],
 "dimensions":[{"field":"experiment_arm_id"},{"field":"judgement_result"}],
 "filters":[{"field":"experiment_id","operator":"eq","value":"<experiment-id>"}],
 "from_timestamp":"2026-06-01T00:00:00Z","to_timestamp":"2026-06-15T00:00:00Z"}
```

An arm with no judgements is unmeasured, not losing.

Cost per arm

Use the same query with `"view":"conversations"`,
`"metrics":[{"aggregation":"sum","measure":"total_cost_usd"}]` and
`"dimensions":[{"field":"runtime_id"}]`; it separates arms only when they are
different versions.

For business outcomes, store `metadata.experiment_arm_id` in your analytics.

## 5. End it and ship

```bash
introspection experiments end <experiment-id>      # keep the evidence
introspection experiments cancel <experiment-id>   # abort
```

Ending stores no winner and ships nothing: merge the winning change and
[ship](/ship) it.

## Verify

`experiments get` shows `running`, and new production tasks carry
`metadata.experiment_arm_id`.

## Limits

| Not supported | Do this instead |
| --- | --- |
| Fixed splits such as 50/50 | Routing is always adaptive (`beta_sample`); use `sample_rate` to bound exposure. |
| A cost or latency goal or guard | Judges only. Compare cost per arm afterwards (step 4). |
| Changing arms, goal or sample_rate while running | Only `description` can change. Cancel it and create a new one. |
| Two running experiments on one runtime and environment | `start` returns 409. End the running one first. |
| Slack-, automation- or sign-in-started tasks | Ship A, then B, and compare by `recipe_git_commit_sha`. |

Or ask your agent: "Set up an experiment on support-agent comparing production with PR 42's preview on 20% of traffic, and show me the draft first."

Run an experiment

Compare versions of one runtime on the same live traffic when offline evals can’t decide between them.

An experiment routes part of a runtime’s traffic across arms, each a version of that runtime, and scores them with a judge. Use it when evals leave several candidates credible. With little traffic, use evals.

You need: two or more versions of the runtime (a pull request’s preview counts), and an enabled judge on every arm.

1. Find the versions and the judge


introspection runtimes list --runtime support-agent   # version ids
introspection judges list                             # judge ids

Every arm must ship the same judge file. An arm built with a different judges/ file never scores, and nothing warns you.

2. Write the draft

experiment.yaml


name: shorter-prompt-vs-control
runtime: support-agent
environment: production
sample_rate: 0.2
goal_json:
  kind: composite
  direction: maximize
  components:
    - source: judge
      judge_id: <judge-id>
      weight: 1.0
      guard: { min: 0.8 }        # optional: stop an arm whose pass rate on this judge falls below 80%
arms:
  - runtime_id: <version-id>
    arm_label: control
  - runtime_id: <version-id>
    arm_label: candidate


introspection experiments create @experiment.yaml

A draft routes nothing.

Field rules

Field	Rule
`arms`	2 to 20 versions of the same runtime. `agent_overrides: {<agent>: <other agent>}` lets arms differ by agent within one version.
`sample_rate`	Share of traffic that enters the experiment; the arms split that share adaptively. In production 0.10 to 1.0; other environments must use 1.0.
`goal_json`	At least one `source: judge` component with `weight` above 0.
`environment`	`production` (default), `staging` or `development`. Only production and staging route live traffic into it; a development experiment takes only `--experiment` tasks.

3. Start it and check routing


introspection experiments start <experiment-id>
introspection tasks create --experiment <experiment-id> --subject test-user-001 \
  --prompt "Review order 1842 against the refund policy."

The test task’s metadata.experiment_arm_id names its arm. Assignment is sticky per subject: the runner’s user_id, else anonymous_id, else conversation_id. Weights start even and shift toward the better-scoring arm.

Which tasks can enter

Only tasks opened with an identity can enter: --subject on the CLI, or the identity your backend passes when it opens a runner. A CLI task without --subject gets the live version. Tasks started by Slack, an automation, or an app sign-in session never enter. --experiment still applies sample_rate: a subject outside the sample gets the live version, so you can’t put one tenant in by hand (use a runtime per tenant).

4. Read results


introspection experiments get <experiment-id>      # arms, posterior_json, weights_json

posterior_json, keyed by arm id: p_best (chance it is best), n_total (judged conversations), stopped (guard tripped). A clear winner is one arm at p_best ≥ 0.95 with every live arm at ≥ 100 judged conversations.

Per-arm judgement counts: save this as arm-judgements.json, then run introspection metrics query @arm-judgements.json -o table.

arm-judgements.json


{"view":"judgements","metrics":[{"aggregation":"count"}],
 "dimensions":[{"field":"experiment_arm_id"},{"field":"judgement_result"}],
 "filters":[{"field":"experiment_id","operator":"eq","value":"<experiment-id>"}],
 "from_timestamp":"2026-06-01T00:00:00Z","to_timestamp":"2026-06-15T00:00:00Z"}

An arm with no judgements is unmeasured, not losing.

Cost per arm

Use the same query with "view":"conversations", "metrics":[{"aggregation":"sum","measure":"total_cost_usd"}] and "dimensions":[{"field":"runtime_id"}]; it separates arms only when they are different versions.

For business outcomes, store metadata.experiment_arm_id in your analytics.

5. End it and ship


introspection experiments end <experiment-id>      # keep the evidence
introspection experiments cancel <experiment-id>   # abort

Ending stores no winner and ships nothing: merge the winning change and ship it.

Verify

experiments get shows running, and new production tasks carry metadata.experiment_arm_id.

Limits

Not supported	Do this instead
Fixed splits such as 50/50	Routing is always adaptive (`beta_sample`); use `sample_rate` to bound exposure.
A cost or latency goal or guard	Judges only. Compare cost per arm afterwards (step 4).
Changing arms, goal or sample_rate while running	Only `description` can change. Cancel it and create a new one.
Two running experiments on one runtime and environment	`start` returns 409. End the running one first.
Slack-, automation- or sign-in-started tasks	Ship A, then B, and compare by `recipe_git_commit_sha`.

Or ask your agent: “Set up an experiment on support-agent comparing production with PR 42’s preview on 20% of traffic, and show me the draft first.”