Production signals & evals

These command groups cover the work a running agent does and the evidence it leaves behind. They need a session from introspection login or, for most reads and task operations, a project API key.

This page is command reference. Use Learn from production and Evaluate offline for the ordered investigation and improvement workflows.

Choose the job before the command group:

See what the project’s agents have been doing before operating on any of it:

introspection tasks list -o table
introspection conversations list --lookback 24h -o table

Operate tasks and artifacts

Tasks

The CLI can create and operate tasks without writing an SDK program:

introspection tasks create --runtime <slug-or-group-id> --environment production [--prompt <text>]
introspection tasks create --runtime-id <runtime-id> --environment staging [--prompt <text>]
introspection tasks create --experiment <experiment-id> --subject <stable-id> [--prompt <text>]
introspection tasks prompt <task-id> --prompt <text>
introspection tasks stream <task-id> --run <run-id|current> --since <sequence>
introspection tasks resume <task-id> --interrupt-id <id> --payload '<json>' …
introspection tasks cancel <task-id>
introspection tasks list [--status <status>]… [--runtime-id <runtime-id>] [--limit <n>] [--next <cursor>]
introspection tasks get <task-id>
introspection tasks runs get <task-id> <run-id>
introspection tasks update <task-id> [--title <text>] [--metadata '<json>']
introspection tasks archive <task-id>
introspection tasks unarchive <task-id>
introspection tasks delete <task-id>

Task creation first mints a transient runner credential, which is never persisted.

--environment is required with both --runtime and --runtime-id, and selects which lane’s bindings and active version answer the task. With --runtime it takes staging or production only, because those are the two lanes with a lane-selected version; --environment development there is an error. --runtime-id takes all three, so a development task names the exact version with --runtime-id --environment development.

--agent <name> picks which agent in the recipe runs, matching the SDK’s agent_name; omit it for the recipe’s default agent. The name is the one in the agent’s own YAML, not the recipe slug.

tasks list returns one bounded page: --limit defaults to 100 and accepts 1 through 1000; follow next with --next for the rest. --status is repeatable and its values union, so --status running --status idle lists both. The recognised statuses are pending, queued, scheduled, running, idle, awaiting_user, cancelling, completed, failed, and cancelled.

create takes seven more selectors beyond the target and the prompt:

Tags both group work and, because access is set intersection, share it: a caller whose member tags intersect a task’s tags can read and write that task. See Tags before using them for anything but grouping. --identity-tag is narrowed server-side to tags the asserting agent member already holds, and applies only when the identity mints a new member. The CLI sets tags on create; filtering a list by tag and clearing tags on an existing task are SDK or REST operations today.

stream writes one JSON object per AG-UI event. --run defaults to current, which resolves the task’s active or latest run, and --since defaults to 0, a first attach; pass the highest sequence you already received to reattach. If the requested sequence fell outside the bounded replay window, the CLI passes the resume_gap event through so you can detect the missing segment.

resume verifies a complete response batch against durable metadata.pending_interrupts; use repeatable --cancel-interrupt <id> entries for cancelled answers. cancel stops only the active run and then prints the task’s resulting state.

The minimal task lifecycle is:

introspection tasks create \
  --runtime support-agent \
  --environment staging \
  --subject smoke-1 \
  --prompt "Reply with ready."
 
introspection tasks stream <task-id>
introspection tasks get <task-id>

tasks create returns one task object; use its id as <task-id>. No separate run record is needed, because --run current resolves metadata.active_run_id and falls back to metadata.latest_run_id. When you need an explicit <run-id> for tasks runs get, read it with tasks get <task-id> --query 'metadata.latest_run_id'. stream is always JSONL, regardless of -o, and ends when the run settles. If tasks get shows metadata.pending_interrupts, answer every pending ID in one resume call and stream the returned current run.

tasks get reports the task’s current state; tasks runs get reports one turn’s own.

Titles

A task is named from its prompt: the first line, cut on a word boundary. A task created without a prompt (a warm run) stays untitled until its first real turn supplies one, and later turns never rename it. tasks update --title wins over the derived name and survives every turn after it.

Archiving

archive hides a task from the default list without deleting anything: the transcript, files, and runs all survive, and unarchive puts it back. Prompting an archived task un-archives it. Neither verb prompts for confirmation; delete does.

Task metadata

--metadata on create and update carries the client metadata bag, which is how a task references things the flags do not cover: uploaded files, a fork source, a repository to clone. It also carries model and thinking_level on create, which override what the agent declares for that task alone — see selecting a model for one task. On update the object is merged into the existing bag rather than swapped for it. Reserved keys the platform owns are rejected rather than silently dropped.

Attaching files

To hand an uploaded file to an agent, upload it first and attach it by id with --file:

# --query still emits JSON, so a scalar arrives quoted; jq -r strips the quotes.
FILE=$(introspection files upload --file spec.md --file-type upload --query 'id' | jq -r)
 
introspection tasks create \
  --runtime support-agent \
  --environment staging \
  --prompt "Read spec.md and summarise it." \
  --file "$FILE"

The file is mounted read-only at /workspace/files/spec.md, and the agent is told the path, not the contents. The mount name comes from the file, so --file <id> is normally all you need; append =<path> to mount it elsewhere (--file "$FILE=specs/spec.md").

--file also works on prompt, so a file can be attached partway through a conversation. It lands in the workspace before that turn runs, without restarting the sandbox, and stays available on later turns:

introspection tasks prompt "$TASK" \
  --prompt "Now compare it to this one." \
  --file "$SECOND_FILE"

Two attachments that share a name both land: the first keeps the plain path and later ones are suffixed with their file id (spec.md, spec-019fd432.md). A file the platform cannot serve is reported to the agent as unavailable rather than failing the task.

Files

Files are durable, versioned artifacts. They outlive the sandbox that produced them, and a write creates a new version rather than replacing the old one:

introspection files list [--limit <n>] [--next <cursor>] [--filter KEY=VALUE]
introspection files get <file-id>
introspection files content <file-id> [--out <path>]
introspection files create --name <name> --content-file <path|-> [--mime-type <type>] [--metadata '<json>']
introspection files upload --file <path> [--name <name>] [--file-type <type>] [--metadata '<json>']
introspection files update <file-id> [--name <name>] [--metadata '<json>']
introspection files delete <file-id>
introspection files versions list <file-id> [--limit <n>] [--next <cursor>] [--filter KEY=VALUE]
introspection files versions get <file-id> <version-id>
introspection files versions create <file-id> --content-file <path|-> [--mime-type <type>] [--metadata '<json>']

create posts text content; upload posts multipart and is the only shape that carries binary. Creating with a name that already exists adds a version to that file rather than failing, which is how the platform models an edit. content writes the raw bytes, so redirect it or pass --out for anything that is not text. There is no restore verb: a write appends, it never rolls back.

Address versions create at the newest row in the chain. The new version number is computed from the row you name, so pointing it at an older version produces a duplicate number rather than the next one:

# The newest row is first in the default (newest-first) listing.
# jq -r unquotes the projected scalar; `--query` alone would keep the quotes.
HEAD=$(introspection files versions list <file-id> --query '[0].id' | jq -r)
introspection files versions create "$HEAD" --content-file next.md

delete removes the file and its whole version chain, and asks first.

Shares

A share grants read access to one file or conversation without giving away the project:

introspection shares list [--limit <n>] [--next <cursor>] [--filter KEY=VALUE]
introspection shares get <share-id>
introspection shares create --resource-type <file|conversation> --resource-id <id> \
  [--member <member-id> | --identity <identity-key>]
introspection shares delete <share-id>

Name --member or --identity to target one recipient; omit both to grant the whole project. The two are mutually exclusive, and you must own the resource you are sharing. Grants are read-only, and delete revokes the grant without deleting the resource.

Learn from production

Read production evidence

Start with the exact task and conversation before choosing an evaluation or experiment. These two reads establish what happened and whether the behavior recurs:

introspection conversations get <conversation-id> > conversation.json
introspection conversations list --lookback 24h --page-all -o table

Use Telemetry command reference below for event-family reads, byte-exact exports, and aggregate metrics queries.

Offline evals

Offline evals live beside the recipe. introspection eval run evaluates a local recipe through an authored Evalite  suite or Harbor  task. The runner owns cases, scorers, verifiers, and native artifacts. The CLI resolves the candidate, expands comparison variants, optionally hydrates production context, and returns one normalized record per run.

introspection eval run --runner <evalite|harbor> [--path <PATH>]
  [-r|--runtime <NAME>] [--agent <NAME>]
  [--trials <N>]
  [--from-conversation <CONVERSATION_ID> [--turn <N>] [--prompt <TEXT>]]
  [-- <RUNNER_ARGS>...]

Run an Evalite file, or omit --path to let Evalite discover the repository’s suite:

introspection eval run --runner evalite --path evals/support.eval.ts

Evalite, Vitest, and @introspection-ai/evalite must be local development dependencies. The CLI uses the repository’s own node_modules/.bin/evalite instead of installing or selecting a global Evalite.

Harbor requires --path and the toolchain installed by introspection setup --target harbor:

introspection eval run \
  --runner harbor \
  --path evals/tasks/acme-q2

Candidate selection follows local runtime discovery. Use --runtime when the workspace contains more than one manifest, and --agent to select an agent by the name declared in its YAML, including inherited variants. The selected agent’s ai.model supplies the evaluated model. Use --trials to repeat the selected agent against the same cases:

introspection eval run \
  --runner harbor \
  --path evals/tasks/acme-q2 \
  --agent triage \
  --trials 3

Each normalized result records the runtime, agent, YAML-resolved candidate model, observed model, recipe path, Git commit and dirty state, trial index, score or reward, token usage, completion reason, and native artifact paths. Results are JSON by default and support the CLI’s global output formats. Native artifacts stay under ~/.introspection/eval-runs/<evalrun-id>/. Evalite writes its JSON result, while Harbor writes its jobs, result, and trajectories.

Arguments after -- pass through to the selected runner. The CLI owns options that define the common comparison (--path, agent, trials, and artifact locations). Run introspection eval run --help for the installed-version contract.

--from-conversation hydrates the evaluated session with a real production conversation before launch — the turn-selection and hydration rules follow. See Evaluate offline for authored case examples and comparison guidance.

Replay production context

introspection eval run can hydrate a local run with real production context. --from-conversation reads the complete root-agent export for a conversation produced by the managed recipe runtime. The evaluation layer selects a canonical platform turn and passes its prior trajectory to introspection local, which hydrates the temporary Pi session immediately before launch. Replay the selected turn’s original user prompt:

introspection eval run \
  --runner evalite \
  --path evals/support.eval.ts \
  --from-conversation <conversation-id> \
  --turn 2

One trace is one platform turn; model and tool spans inside it are steps. Add --prompt to replace the selected turn’s prompt. To continue after the complete conversation, omit --turn and supply --prompt. The command rejects --from-conversation without either a replayable --turn or a continuation prompt. The result records the source conversation, trace-backed turn identity, prompt source, and snapshot hashes. Hydration restores only conversation state. It does not restore production files, services, credentials, or side effects, and it requires the normal project-scoped CLI login.

The CLI requests the turn-aware representation from GET /v1/conversations/{conversation_id}/export?agent=root. It hydrates only completed turns before the selected turn; the selected turn itself may be incomplete so a failed or interrupted turn remains replayable. Continuation without --turn requires every exported turn to be complete. Replay is addressed only by conversation ID, not task ID. A turn becomes complete when the managed runtime’s ended, parentless invoke_agent span is present; Error status still counts as completed. Generic GenAI conversations without that root wrapper are not replay-eligible.

Judges

Judge definitions remain git-owned recipe artifacts. The CLI reads deployed judges and changes only their operational state:

introspection judges list
introspection judges get <judge-id>
introspection judges update <judge-id> --disable
introspection judges update <judge-id> --enable --sample-rate 0.25

--sample-rate accepts a value from 0 through 1. Definition fields such as instructions, model, and run gates are not editable through this command.

Judge calibration

Export unlabeled fixtures from real conversations, label them by hand, then evaluate a judge against the labeled file through an OpenAI-compatible endpoint. The three steps are not one pipeline: the middle one is yours.

# 1. Export. Every row lands with "expected": null and "split": null.
introspection conversations get \
  --ids-file conversation-ids.txt \
  --judge-fixtures \
  --output-file judges/useful-support-resolution.calibration.jsonl
 
# 2. Label the JSONL by hand. `judges eval` fails on the first unlabeled row.
 
# 3. Evaluate.
OPENAI_API_KEY=... introspection judges eval \
  --judge judges/useful-support-resolution.yaml \
  --dataset judges/useful-support-resolution.calibration.jsonl

--judge-fixtures is what turns the export into fixture rows; without it the same command writes plain conversation bundles, which judges eval rejects. It requires --ids-file, is incompatible with --agent, and supports only --format json.

Export straight to the canonical path beside the judge: the definition and its approved dataset ship as one commit in the recipe repository.

The fixture file owns expected (pass, fail, or not_applicable) and optional split (train, dev, or test) labels; judges eval --split evaluates one split at a time. Add --include-reasoning to keep the model’s reasoning in the per-fixture output, and --concurrency to change how many fixtures are evaluated at once (default 8, from 1 through 64; lower it when the provider rate-limits).

The judge YAML owns the model configuration (see Calibrate judges for a complete definition). OpenAI defaults to OPENAI_API_KEY; OpenRouter defaults to OPENROUTER_API_KEY; a custom local endpoint is explicit under llm.local. The Introspection login credential is never used for the model call, and model credentials are not written into fixtures or reports.

Telemetry command reference

The CLI reads conversations, lists exactly one typed event family per request, reads a single event by id, or submits a bounded metrics query:

introspection conversations list --lookback 24h --page-all
introspection conversations get conv_123 > conversation.json
introspection conversations get conv_123 --summary-only
introspection conversations get conv_123 conv_456 --output-file conversations.jsonl
introspection conversations get \
  --ids-file conversation-ids.txt \
  --judge-fixtures \
  --output-file fixtures.jsonl
introspection conversations export conv_123 \
  --format trajectory \
  --output-file conv_123.trajectory.json
introspection events list \
  --event-name introspection.observation \
  --filter lens=task_resolution \
  --lookback 7d
introspection events get <event-id>
introspection metrics query @metrics.json

Conversation get takes one or more positional conversation ids, or an --ids-file of up to 20 ids; only --summary-only insists on exactly one id. For a single id it composes the summary, every item page, and the conversation’s feedback events into one bundle on stdout; --summary-only returns just the summary and, like --judge-fixtures, supports only --format json. For several ids, or with --output-file, it writes one JSONL row per id.

Those rows are conversation bundles unless you pass --judge-fixtures, which converts each one into a judge fixture: the transcript under judge_input, plus a snapshot_hash and empty expected / split labels for you to fill in, the form Calibrate judges builds a dataset from.

--format chooses the representation: json (default), arrow for the item rows as one Arrow IPC stream, or trajectory for the server-projected trajectory. --agent <ROOT_OR_ID> narrows the read to root or one exact agent invocation; a multi-conversation read accepts only --agent root. Add --force to replace an existing --output-file.

--page-limit (default 100) caps the feedback-event pages read for one conversation, and --max-items, --max-bytes, and --timeout-seconds bound the rest of that conversation’s traversal. Exceeding a bound fails the command rather than silently truncating the export.

An agent task defaults its conversation id to the task id, so unless the caller supplied one, a task id from tasks list can be passed here directly.

Byte-exact exports

conversations export forwards one server-side export stream unchanged, writing each chunk straight through. Use it when you want the server’s own bytes rather than the composed bundle get builds:

introspection conversations export <conversation-id>
  [--format json|arrow|trajectory] [--output-file <path>] [--force]
  [--agent <ROOT_OR_ID>] [--service-name <name>] [--operation-name <name>]
  [--lookback-days <1-365>] [--share-id <share-id>]
  [--start-date <iso>] [--end-date <iso>]

--format defaults to json; output goes to stdout unless --output-file names a path, and creation is exclusive unless you add --force. --share-id reads through a conversation share grant instead of the project’s own access, and the remaining flags narrow which records the server includes.

events get <event-id> reads one event by id across every family, so it needs no --event-name.

The conversations list and events list reads share one flag contract: --limit (default 100), --next, --sort, --order asc|desc (default desc), --start / --end, --lookback, and repeatable --filter KEY=VALUE. --lookback is mutually exclusive with --start / --end, and the valid --sort fields differ per command and per event family: see introspection events list --help.

For paginated conversation and event listings, add --format arrow --arrow-out <path>, with --force to replace an existing Arrow file. Event Arrow pages preserve the family-typed payload struct. --page-all follows opaque cursors to exhaustion; --page-limit bounds the number of pages.

metrics query forwards the JSON document unchanged. A complete count request looks like:

{
  "view": "observations",
  "metrics": [{ "aggregation": "count" }],
  "filters": [
    { "field": "runtime_group_id", "operator": "eq", "value": "019..." }
  ],
  "from_timestamp": "2026-07-11T00:00:00Z",
  "to_timestamp": "2026-07-18T00:00:00Z"
}

Experiments

Use an experiment only when offline evidence cannot decide and the comparison requires same-population production traffic. The CLI exposes the full lifecycle:

introspection experiments list [--runtime <slug-or-group-id>] [--environment <env>] [--status <status>]
introspection experiments get <id>
introspection experiments create @experiment.yaml
introspection experiments update <id> @experiment-update.yaml
introspection experiments start <id>
introspection experiments end <id>
introspection experiments cancel <id>
introspection experiments delete <id>

create @FILE accepts the exact ExperimentCreate JSON or YAML document and update <id> @FILE the exact ExperimentUpdate document; use @- for stdin. Create injects the active project when omitted, and the server stays the schema authority. delete is terminal and separate from end or cancel.

A minimal valid document carries only name, runtime, arms, and goal_json:

name: support-agent-candidate
runtime: support-agent
arms:
  - runtime_id: 019...
    arm_label: baseline
  - runtime_id: 019...
    arm_label: candidate
goal_json:
  kind: composite
  direction: maximize
  components:
    - source: judge
      judge_id: 019...
      weight: 1.0

goal_json.components needs at least one entry with source: judge, a judge_id from judges list, and a positive weight. Add environment (default production) and sample_rate when you want a lane other than the default or less than the group’s full traffic.

runtime_group_id is still accepted in existing create documents, but runtime is the canonical selector. All arms must belong to the named runtime group; they use runtime IDs and are symmetric, with no create-time control arm or authored weight. end takes no winner: it stops evidence collection and deploys nothing.

Related

• CLI setup & login: install, setup, login, and the output contract.
• Local recipe commands: author and validate what these commands run.
• Tasks & runs: run, cancel, resume, and stream semantics.
• Experiments & shipping: deciding from experiment evidence.
• Evaluate offline: authored cases and comparison guidance.