CLI

The CLI signs in as you through a browser approval flow. It is the operator surface for a project, where you author and administer the objects your agents run against: create the first runtime, define a recipe or an experiment, choose what staging serves, disable a bad version, and configure secrets. Production normally follows the repository’s default branch. See what each surface owns.

For step-by-step instructions, use the Guides to build, deploy, and improve an agent. Come back here when you need a command or flag.

Prerequisites

• Node.js 24+ 

Install

pnpm add --global @introspection-ai/cli

bun add --global @introspection-ai/cli

yarn global add @introspection-ai/cli

npm install --global @introspection-ai/cli

If pnpm reports that it has no global binary directory, run pnpm setup once and restart the shell before installing the CLI. That command configures pnpm’s own global directory and PATH; Introspection does not edit your shell profile.

Set up the local toolchain

In the agent-led flow, the coding agent runs setup when its selected workflow first needs Pi and Recipes; the user does not run it manually. If you are operating directly from a terminal instead, run the interactive setup yourself:

introspection setup

Setup converges the machine on one supported state:

• installs Pi with its official npm method when Pi is absent;
• asks an existing Pi below the Recipes minimum to update itself, while accepting newer Pi versions;
• installs or repairs the exact compatible Recipes extension.

Pi remains independent and user-owned: setup never downgrades a newer Pi or replaces a working installation, and when NVM already provides a compatible Node.js version it uses that without changing your shell’s default.

For automation, --check prints the plan without changing anything and --yes authorizes the complete plan, including Pi changes.

--target harbor prepares only the optional Harbor toolchain (uv, Harbor, and the Introspection Recipe agent) instead of the Pi/Recipes plan. Run it once before introspection eval run --runner harbor; --check prints the Harbor plan without installing anything:

introspection setup --target harbor --check
introspection setup --target harbor

Log in

introspection login
introspection whoami

Login uses the OAuth 2.0 Device Authorization flow; select the project on the browser approval page, and the server resolves that project’s development agent for you. The CLI stores the identity- and project-bound session at ~/.introspection/credentials.json with mode 0600 and refreshes it transparently. whoami validates the session and prints the Control Plane/Data Plane targets, granted scope, organization, project, member, token and session status, and expiry, without printing tokens. Run login again to switch project or development identity; project-scoped commands also accept --project.

The server always chooses the CLI capabilities: runtimes for platform operations and remotes for introspection dev. There is no capability flag. Project API keys and applications cannot request the CLI-only remotes scope.

introspection logout

Command map

Run introspection <group> --help for the version-installed flag contract. A bare introspection with no arguments prints a status view of where you are, not the command index; use introspection --help for the index.

Offline evals

Offline evals live beside the recipe. introspection eval run evaluates a local Recipe through an authored Evalite  suite or Harbor  task. The runner owns cases, scorers, verifiers, and native artifacts. The CLI resolves the candidate, expands comparison variants, optionally hydrates production context, and returns one normalized record per run.

introspection eval run --runner <evalite|harbor> [--path <PATH>]
  [-r|--runtime <NAME>] [--agent <NAME>]
  [--trials <N>]
  [--from-conversation <CONVERSATION_ID> [--turn <N>] [--prompt <TEXT>]]
  [-- <RUNNER_ARGS>...]

Run an Evalite file, or omit --path to let Evalite discover the repository’s suite:

introspection eval run --runner evalite --path evals/support.eval.ts

Evalite, Vitest, and @introspection-ai/evalite must be local development dependencies. The CLI uses the repository’s own node_modules/.bin/evalite instead of installing or selecting a global Evalite.

Harbor requires --path and the toolchain installed by introspection setup --target harbor:

introspection eval run \
  --runner harbor \
  --path evals/tasks/acme-q2

Candidate selection follows local Runtime discovery. Use --runtime when the workspace contains more than one manifest, and --agent to select an agent by the name declared in its YAML, including inherited variants. The selected agent’s ai.model supplies the evaluated model. Use --trials to repeat the selected agent against the same cases:

introspection eval run \
  --runner harbor \
  --path evals/tasks/acme-q2 \
  --agent triage \
  --trials 3

Each normalized result records the runtime, agent, YAML-resolved candidate model, observed model, Recipe path, Git commit and dirty state, trial index, score or reward, token usage, completion reason, and native artifact paths. Results are JSON by default and support the CLI’s global output formats. Native artifacts stay under ~/.introspection/eval-runs/<evalrun-id>/. Evalite writes its JSON result, while Harbor writes its jobs, result, and trajectories.

Arguments after -- pass through to the selected runner. The CLI owns options that define the common comparison (--path, agent, trials, and artifact locations). Run introspection eval run --help for the installed-version contract.

Replay production context

--from-conversation reads the complete root-agent export for a conversation produced by the managed Recipe runtime. The evaluation layer selects a canonical platform turn and passes its prior trajectory to introspection local, which hydrates the temporary Pi session immediately before launch. Replay the selected turn’s original user prompt:

introspection eval run \
  --runner evalite \
  --path evals/support.eval.ts \
  --from-conversation <conversation-id> \
  --turn 2

One trace is one platform turn; model and tool spans inside it are steps. Add --prompt to replace the selected turn’s prompt. To continue after the complete conversation, omit --turn and supply --prompt. The command rejects --from-conversation without either a replayable --turn or a continuation prompt. The result records the source conversation, trace-backed turn identity, prompt source, and snapshot hashes. Hydration restores only conversation state. It does not restore production files, services, credentials, or side effects, and it requires the normal project-scoped CLI login.

The CLI requests the turn-aware representation from GET /v1/conversations/{conversation_id}/export?agent=root. It hydrates only completed turns before the selected turn; the selected turn itself may be incomplete so a failed or interrupted turn remains replayable. Continuation without --turn requires every exported turn to be complete. Replay is addressed only by conversation ID, not task ID. A turn becomes complete when the managed runtime’s ended, parentless invoke_agent span is present; Error status still counts as completed. Generic GenAI conversations without that root wrapper are not replay-eligible.

See Learn from production → proving a fix locally for authored case examples and comparison guidance.

API keys and applications

api-keys manages project API keys for application/SDK callers. applications manages application identities, service-account secrets, identity-provider federations, and assertion keys. Neither can request the member-only remotes capability used by dev.

Project API keys are environment-bound credentials for SDK callers. create writes the one-time secret to a new mode-0600 file, acknowledges it only after the write succeeds, and never prints the secret:

introspection api-keys create \
  --name staging-sdk \
  --environment staging \
  --member-id <agent-member-id> \
  --secret-file ./staging-api-key
introspection api-keys create --secret-file ./ci-key --expires-at 2026-12-31T00:00:00Z
introspection api-keys list
introspection api-keys list --expiring
introspection api-keys update --key <key-id> \
  --name staging-sdk-2 \
  --member-id <agent-member-id>
introspection api-keys revoke --key <key-id>

--name defaults to cli and --environment defaults to production, so --secret-file is the only required flag on create. --expires-at takes an RFC 3339 timestamp; omit it for a key with no fixed lifetime. list returns active, non-expired keys; add --expiring to include keys with a future expiry or a grace-period revocation.

If the file write succeeded but acknowledgement failed, retry api-keys acknowledge --key <key-id>. Use revoke --expires-in <seconds> for a grace period.

Applications are organization-managed callers. Choose the type and environment that match the trust model, retain its id, then manage the application and its subordinate auth resources:

introspection applications create --type spa --name web \
  --environment staging \
  --redirect-uri https://app.example.com/callback \
  --allowed-origin https://app.example.com
introspection applications create --type native --name desktop
introspection applications create --type jwks --name backend
introspection applications create --type service-account --name worker \
  --agent-member-id <agent-member-id> \
  --allowed-scope runtimes
introspection applications list
introspection applications get <application-id>
introspection applications update <application-id> --name web-staging
introspection applications delete <application-id>

--environment is set at creation and defaults to production. A service account acts as --agent-member-id; when omitted, the Control Plane uses the organization’s system agent. update can replace that agent, or replace or clear redirect URIs, allowed origins, and allowed scopes.

Manage the auth resources appropriate to the application type:

# Service-account client secrets. The created secret is shown once.
introspection applications secrets create --app <application-id>
introspection applications secrets list --app <application-id>
introspection applications secrets revoke --app <application-id> \
  --secret <secret-id> [--expires-in <seconds>]
 
# Direct JWKS identity federation.
introspection applications idps create --app <application-id> \
  --issuer https://issuer.example.com \
  [--audience-client-id <audience>]
 
# Brokered SPA federation, which also needs the OAuth client.
INTROSPECTION_IDP_CLIENT_SECRET='...' \
introspection applications idps create --app <application-id> \
  --issuer https://issuer.example.com \
  --provider supabase \
  --client-id <oauth-client-id> \
  --scope openid --scope profile --scope email \
  --wait
 
introspection applications idps list --app <application-id>
introspection applications idps delete --app <application-id> --idp <idp-id>
 
# Assertion signing keys.
introspection applications keys create --app <application-id>
introspection applications keys list --app <application-id>
introspection applications keys delete --app <application-id> --key <key-id>

A jwks application needs only --issuer. A brokered spa application also needs --provider (for example supabase, auth0, okta, or zitadel), --client-id, and --client-secret — pass the secret through INTROSPECTION_IDP_CLIENT_SECRET to keep it out of shell history. --scope is repeatable and defaults to openid profile email, and --wait polls until the brokered federation finishes provisioning.

Protect the one-time service-account secret before persisting it; later secrets list never returns the value. Creating applications and federations can require an owner/admin member login.

Create a recipe

init scaffolds a runnable recipe, its .introspection/ runtime manifest, and a Git repository when you are not already inside one. It needs no Introspection account. Run it without arguments for prompts, or supply the recipe name and template for a non-interactive setup:

introspection init
introspection init support-agent
introspection init support-agent --template template-starter

The generated recipe is ready for introspection local. --template names a repository in the Introspection template catalog and defaults to template-starter.

Run a recipe locally

introspection local resolves a repository runtime manifest and runs its recipe with Pi. Use --agent to select a named agent. Everything after -- is forwarded to Pi unchanged.

introspection local [--runtime <NAME>] [--work-dir <DIRECTORY>] [--agent <NAME>]
                    [-p|--print <PROMPT>] [--mode <text|json>] [-- <PI_ARGS>...]

# Select the only runtime, or the one whose recipe contains this directory.
introspection local
 
# Select a runtime explicitly in a repository with several manifests.
introspection local --runtime support-agent
 
# Select an agent and run one prompt.
introspection local -r support-agent --agent triage \
  --print "Triage this refund request."

-p / --print <PROMPT> runs one prompt and exits. --mode requires --print and chooses between text (the final assistant text only) and json (Pi’s structured event stream). Both are the CLI’s own flags: supplying either natively and again after -- is a hard error.

The command starts at the current directory, or --work-dir <directory>, and walks upward to the nearest .introspection/ directory containing YAML runtime manifests. A manifest’s path must be relative to that repository root, stay inside it after symlinks are resolved, name an existing directory, and contain package.json. When selection is ambiguous, pass --runtime with the manifest display name or filename stem. --work-dir changes manifest discovery only: Pi still inherits the directory you invoked the CLI from.

Before Pi starts, the CLI preflights prerequisites and validates every recipe manifest in the discovered workspace, including sibling manifests in a monorepo. It installs nothing itself, so run introspection setup once when the preflight reports that Pi or Recipes is not ready. Model providers and Pi’s own sessions, trust, tools, and MCP stay under Pi’s configuration.

local runs entirely offline: no login, project, cloud binding, or snapshot is involved. The command inherits your terminal and environment and returns Pi’s exit status. Global --query is not supported; use --mode json for a structured one-shot run instead:

introspection local -r support-agent --print "Triage this." --mode json
 
# Run a recipe from a different workspace.
introspection local --work-dir ../support-agent

Cloud-connected development

Run from the Git working tree that contains your .introspection/<runtime-slug>.yaml manifest:

introspection dev

dev resolves the runtime from the Git repository (or --runtime), verifies that its cloud recipe is valid, checks declared MCP bindings, opens Runtime Preview, and serves local recipe changes to development tasks. The overlay is keyed by project, member, and exact recipe commit.

Run the application normally with its development-scoped credential; the SDK run() call has no environment option. Project API keys, service-account credentials, and federated customer sessions all reach the same agent overlay through their ordinary SDK calls, and federated tasks stay owned by the customer.

Different recipe commits can attach concurrently; a duplicate attachment for the same agent and commit fails with dev_slot_conflict. One root-level process can serve several recipe subpaths from a monorepo when their runtimes share a commit.

When several local sessions share a development runtime, use --as NAME to name the attachment, and set INTROSPECTION_DEV_TARGET in the application process to the exact target the intended session printed. Leave it unset when routing is unambiguous; the selector is development-only and needs a development-scoped credential. See Optional: route SDK traffic to one development session for the routing and failure rules.

Missing required and optional development bindings warn by default. Use --check-bindings to turn missing required bindings into a failing readiness check and exit without attaching:

introspection dev --check-bindings
introspection dev --runtime support-agent
introspection dev --runtime support-agent --as John
 
# Route a declared MCP server to a local process during development.
introspection dev --mcp contacts=http://127.0.0.1:8787/mcp

The --mcp name must match a server declared by the recipe. Your development process serves the local transport, but credentials still come from the matching development binding: the CLI never reads local credential files, uploads local secrets, or silently creates endpoints.

The rest of the flag contract:

Follow the Agent development lifecycle for the full path from local Pi testing to production.

Diagnose the local toolchain

introspection doctor is the read-only diagnosis of the CLI, Recipes, Pi, Node, and local recipe readiness. It changes nothing.

introspection doctor

The two checks answer different questions: setup --check renders the plan to reach a ready machine, doctor reports the state of the machine you are on.

Validate recipes

introspection check checks the runtime manifest, package metadata, agent YAML, and every declared agent, skill, extension, include, and glob boundary locally and offline.

# Discover every manifest under .introspection in the current workspace.
introspection check
 
# Validate a different workspace.
introspection check --work-dir ../support-agent
 
# Validate selected manifests. Repeat --path as needed.
introspection check \
  --path .introspection/support-agent.yaml \
  --path .introspection/refund-agent.yaml

Paths are resolved relative to --work-dir. The command exits nonzero when validation fails.

Recipe records

Cloud recipe records are immutable Git pins. Authoring them is operator work, so these commands have no SDK equivalent; the SDKs read recipes only.

introspection recipes list [--name <name>]
introspection recipes get <recipe-id>
introspection recipes create --name <name> \
  [--repository-id <id>] [--git-ref <ref>] [--git-sha <commit>] \
  [--sub-path <path>] [--slug <slug>]
introspection recipes update <recipe-id> [--name <name>] [--description <text>]
introspection recipes delete <recipe-id>

Runtime bootstrap normally creates the initial recipe pin for you; reach for recipes create only when operating the record directly. --name is its only required flag, and --slug overrides the stable slug otherwise derived from the name and short commit sha.

update changes the operator-facing name and description only. The Git pin (repository, ref, commit, sub-path) is immutable: pointing at different content means creating a new recipe. delete is a soft delete, and is refused while runtimes still reference the recipe.

Open the dashboard

open resolves the runtime for the current recipe and opens it in the Introspection dashboard. Select another runtime or a specific task when needed:

introspection open
introspection open --runtime support-agent
introspection open --task <task-id>
introspection open --print

The URL is always written to stdout. --print skips opening the browser.

Bindings

introspection bindings is a CLI umbrella over three flat Control Plane resources, plus one composed read over two of them. It is not a separate /v1/bindings API:

Each resource has its own verbs:

introspection bindings endpoints list|get|create|update|delete
introspection bindings variables list|create|update|delete
introspection bindings credentials list|get|create|update|delete
introspection bindings mcp list|connect|disconnect

Variables have no get: list already returns their values, while endpoint and credential reads deliberately withhold header values and secrets.

Endpoints, variables, and credentials can be project-wide, Runtime scoped (--runtime <slug-or-group-id>), or scoped to one exact Runtime version (--runtime-id <version-id>). On those three, --environment is optional and takes shared, development, staging, or production. Name a lane to scope the row to it, or shared to write the row that applies to every lane; on a list, omitting the flag leaves the listing unfiltered, while shared narrows it to the rows that carry no lane of their own.

bindings mcp list takes a different --environment: required, and only the three concrete lanes (development, staging, production). There is no shared there, because it resolves one Runtime version’s effective connections.

--runtime names the stable Runtime and --runtime-id one immutable version of it. Both are resolved server-side, so a version id passed to --runtime is rejected and told which flag takes it.

Endpoint create/update accepts a complete JSON or YAML document as positional @FILE (or @- for stdin); flags override document fields. Bindings shows a full endpoint document and the credential it references.

Keep secret values out of shell arguments: --from-env reads the value from an environment variable, and --from-file (a mode-0600 file) and --from-stdin are the other secret-safe sources. Credential reads expose has_secret and secret_set_at, never the value. Endpoint reads return the unresolved ${NAME} template and referenced credential names, never an expanded secret.

MCP connections

bindings mcp connect creates or upserts the concrete endpoint for a stable recipe MCP id. list composes recipe declarations with the endpoints applicable to one Runtime and environment, including required-but-missing and stale connections. Both resolve the recipe from the Runtime, so neither takes a recipe argument.

introspection bindings mcp connect \
  --mcp-server-id linear \
  --name Linear \
  --endpoint-url https://mcp.example.com/mcp
 
introspection bindings mcp list \
  --runtime <runtime-slug-or-group-id> \
  --environment production
 
introspection bindings mcp disconnect --endpoint <endpoint-id>

list requires --environment and either --runtime or --runtime-id. disconnect deletes the endpoint row named by bindings mcp list.

Repositories

introspection repositories registers the source repositories a project’s runtimes may work on:

introspection repositories list
introspection repositories add <owner/name>
introspection repositories remove <owner/name-or-id>

Registration is what makes a repository addressable. A sandbox asks for its GitHub credential by repository id (POST /v1/repositories/{id}/access-token takes no slug), so a repository that is not registered cannot be named in any request. It is not, by itself, a grant: a recipe still has to list the repository under runtime.github.repositories, and a task still has to ask for it. See Work with repositories.

You often will not need add first. Creating a recipe from a GitHub selector — which runtimes create and the dashboard both do — registers the repository its recipe is pinned in, so this command is for the additional repositories an agent works on.

add takes a full GitHub URL as readily as an owner/name slug, resolves the organization’s GitHub installation itself, and is idempotent — a repeat returns the existing registration. Pass --integration-id only if the organization has more than one active GitHub installation, which is refused rather than guessed. remove accepts either the id or the owner/name a recipe grant would name.

Removing a repository does not edit any recipe, but it is refused with 409 repository_in_use while a recipe is still pinned in it — including the one runtimes create registered it for. A recipe that merely grants it resolves to nothing, and its tasks launch without that checkout rather than failing.

Connectors

introspection connectors manages provider integrations and the customer connections beneath them. Every command acts on the project selected by introspection login, and accepts a connector’s id or slug wherever one is named — so connectors authorize slack-support works without pasting ids around. See Connectors for the model.

introspection connectors list
introspection connectors create --name <text> --provider <slug> --auth-mode <mode> …
introspection connectors get <connector>
introspection connectors update <connector> [--webhook-url <url>] [--client-secret <secret>] …
introspection connectors delete <connector>
introspection connectors authorize <connector> [--runtime <slug-or-id>] [--expires-in <duration>]
introspection connectors connections list --connector <connector>
introspection connectors connections get <connection-id> --connector <connector>
introspection connectors connections revoke <connection-id> --connector <connector>

Registering a provider app takes two passes, because the app’s delivery URL contains the connector id and so cannot be known until the connector exists:

# 1. Create the connector. Idempotent on the slug.
introspection connectors create --name "Slack support" \
  --provider slack --auth-mode oauth-stored \
  --scope chat:write --scope channels:read --api-host slack.com
 
# 2. Register the app with the provider against
#    {control-plane-host}/v1/webhooks/slack/{connector-id}
 
# 3. Feed back what the provider issued.
introspection connectors update slack-support \
  --webhook-url <URL> --client-secret <SECRET> --signing-secret <SECRET>

--client-secret and --signing-secret are also accepted on create, for a provider app that already exists. They are write-only: stored encrypted and never returned. Leaving one off update means unchanged — replacing it is the only way to rotate it, and there is no way to clear it. Creates and updates carrying a secret are sent so that a failed request does not echo it back.

connectors authorize is the command an operator actually reaches for: it prints the install link a customer opens.

introspection connectors authorize slack-support --runtime support-agent --expires-in 1h

The URL goes to stdout through the normal output path, so -o json and --query authorize_url work and it can be piped into whatever sends it; the expiry and the single-use warning go to stderr. It deliberately does not open a browser — the link is usually for someone else.

--expires-in takes seconds or a unit, so 1h and 3600 both work. The server allows 60s to 24h and defaults to 10 minutes, which suits following the link yourself; raise it when handing the link to someone who will open it later.

connections revoke destroys the provider token behind one connection: that subject is disconnected immediately and must re-consent through a fresh install link, and the approvals recorded against the connection are revoked with it. The CLI confirms before doing so. The connector and its other connections are untouched.

Runtimes

Runtime versions begin with the recipe manifest:

introspection runtimes create --manifest .introspection/<name>.yaml
introspection runtimes list [--runtime <slug-or-group-id>]
introspection runtimes get <runtime-id>
introspection runtimes versions <runtime-id>
introspection runtimes pin <runtime-id> [--environment staging]
introspection runtimes pin <runtime-id> --sha <sha>
introspection runtimes pin <runtime-id> --branch <main|pr/N>
introspection runtimes unpin <runtime-id>
introspection runtimes update <runtime-id> --yank [--reason <text>]
introspection runtimes update <runtime-id> --unyank
introspection runtimes delete <runtime-id>

create reads the canonical manifest and Git checkout, creates the immutable recipe pin through the active GitHub integration, and bootstraps the first runtime. The manifest filename is the runtime-group slug. Later versions remain Git-driven.

The first bootstrap must come from a reviewed production commit. Before running create, make sure:

• the worktree has no uncommitted changes;
• main is checked out and is the GitHub repository’s default branch;
• local main matches the pushed origin/main; and
• the repository is available through the project’s GitHub integration.

The resulting version immediately becomes active in both staging and production. create does not accept a feature branch or pull-request commit: if you started with a pull request, merge it first and update local main. Once the runtime group exists, later pull-request commits can become staging candidates through the normal Git-driven version flow.

pin steers staging; --environment defaults to and accepts only staging. With no flag it freezes staging on the named exact version, --sha freezes it on the version built from that recipe commit, and --branch follows future versions built from main or pr/N. A moving branch ref and a frozen commit are mutually exclusive, and unpin restores staging to main. Production normally follows the repository’s default branch and can be pinned only through the Control Plane API. See Runtimes & environments for the immutable-source versus movable-lane distinction.

update --yank withdraws a version from resolving for new work without deleting its history, and --unyank reverses it. delete removes the version from future use and evicts its baked images, so prefer --yank when you need a reversible withdrawal.

Tasks

The CLI can create and operate tasks without writing an SDK program:

introspection tasks create --runtime <slug-or-group-id> --environment production [--prompt <text>]
introspection tasks create --runtime-id <runtime-id> --environment staging [--prompt <text>]
introspection tasks create --experiment <experiment-id> --subject <stable-id> [--prompt <text>]
introspection tasks prompt <task-id> --prompt <text>
introspection tasks stream <task-id> --run <run-id|current> --since <sequence>
introspection tasks resume <task-id> --interrupt-id <id> --payload '<json>' …
introspection tasks cancel <task-id>
introspection tasks list [--status <status>]… [--runtime-id <runtime-id>] [--limit <n>] [--next <cursor>]
introspection tasks get <task-id>
introspection tasks runs get <task-id> <run-id>
introspection tasks update <task-id> [--title <text>] [--metadata '<json>']
introspection tasks archive <task-id>
introspection tasks unarchive <task-id>
introspection tasks delete <task-id>

Task creation first mints a transient runner credential, which is never persisted.

--environment is required with both --runtime and --runtime-id, and selects which lane’s bindings and active version answer the task. With --runtime it takes staging or production only, because those are the two lanes with a lane-selected version; --environment development there is an error. --runtime-id takes all three, so a development task names the exact version with --runtime-id --environment development.

--agent <name> picks which agent in the recipe runs, matching the SDK’s agent_name; omit it for the recipe’s default agent. The name is the one in the agent’s own YAML, not the recipe slug.

tasks list returns one bounded page: --limit defaults to 100 and accepts 1 through 1000; follow next with --next for the rest. --status is repeatable and its values union, so --status running --status idle lists both. The recognised statuses are pending, queued, scheduled, running, idle, awaiting_user, cancelling, completed, failed, and cancelled.

create takes six more selectors beyond the target and the prompt:

Tags both group work and, because access is set intersection, share it: a caller whose member tags intersect a task’s tags can read and write that task. See Tags before using them for anything but grouping. --identity-tag is narrowed server-side to tags the asserting agent member already holds, and applies only when the identity mints a new member. The CLI sets tags on create; filtering a list by tag and clearing tags on an existing task are SDK or REST operations today.

stream writes one JSON object per AG-UI event. --run defaults to current, which resolves the task’s active or latest run, and --since defaults to 0, a first attach; pass the highest sequence you already received to reattach. If the requested sequence fell outside the bounded replay window, the CLI passes the resume_gap event through so you can detect the missing segment.

resume verifies a complete response batch against durable metadata.pending_interrupts; use repeatable --cancel-interrupt <id> entries for cancelled answers. cancel stops only the active run and then prints the task’s resulting state.

The minimal task lifecycle is:

introspection tasks create \
  --runtime support-agent \
  --environment staging \
  --subject smoke-1 \
  --prompt "Reply with ready."
 
introspection tasks stream <task-id>
introspection tasks get <task-id>

tasks create returns one task object; use its id as <task-id>. No separate run record is needed, because --run current resolves metadata.active_run_id and falls back to metadata.latest_run_id. When you need an explicit <run-id> for tasks runs get, read it with tasks get <task-id> --query 'metadata.latest_run_id'. stream is always JSONL, regardless of -o, and ends when the run settles. If tasks get shows metadata.pending_interrupts, answer every pending ID in one resume call and stream the returned current run.

tasks get reports the task’s current state; tasks runs get reports one turn’s own.

Titles

A task is named from its prompt: the first line, cut on a word boundary. A task created without a prompt (a warm run) stays untitled until its first real turn supplies one, and later turns never rename it. tasks update --title wins over the derived name and survives every turn after it.

Archiving

archive hides a task from the default list without deleting anything: the transcript, files, and runs all survive, and unarchive puts it back. Prompting an archived task un-archives it. Neither verb prompts for confirmation; delete does.

Task metadata

--metadata on create and update carries the client metadata bag, which is how a task references things the flags do not cover: uploaded files, a fork source, a repository to clone. On update the object is merged into the existing bag rather than swapped for it. Reserved keys the platform owns are rejected rather than silently dropped.

Attaching files

To hand an uploaded file to an agent, upload it first and attach it by id with --file:

# --query still emits JSON, so a scalar arrives quoted; jq -r strips the quotes.
FILE=$(introspection files upload --file spec.md --file-type upload --query 'id' | jq -r)
 
introspection tasks create \
  --runtime support-agent \
  --environment staging \
  --prompt "Read spec.md and summarise it." \
  --file "$FILE"

The file is mounted read-only at /workspace/files/spec.md, and the agent is told the path, not the contents. The mount name comes from the file, so --file <id> is normally all you need; append =<path> to mount it elsewhere (--file "$FILE=specs/spec.md").

--file also works on prompt, so a file can be attached partway through a conversation. It lands in the workspace before that turn runs, without restarting the sandbox, and stays available on later turns:

introspection tasks prompt "$TASK" \
  --prompt "Now compare it to this one." \
  --file "$SECOND_FILE"

Two attachments that share a name both land: the first keeps the plain path and later ones are suffixed with their file id (spec.md, spec-019fd432.md). A file the platform cannot serve is reported to the agent as unavailable rather than failing the task.

Files

Files are durable, versioned artifacts. They outlive the sandbox that produced them, and a write creates a new version rather than replacing the old one:

introspection files list [--limit <n>] [--next <cursor>] [--filter KEY=VALUE]
introspection files get <file-id>
introspection files content <file-id> [--out <path>]
introspection files create --name <name> --content-file <path|-> [--mime-type <type>] [--metadata '<json>']
introspection files upload --file <path> [--name <name>] [--file-type <type>] [--metadata '<json>']
introspection files update <file-id> [--name <name>] [--metadata '<json>']
introspection files delete <file-id>
introspection files versions list <file-id> [--limit <n>] [--next <cursor>] [--filter KEY=VALUE]
introspection files versions get <file-id> <version-id>
introspection files versions create <file-id> --content-file <path|-> [--mime-type <type>] [--metadata '<json>']

create posts text content; upload posts multipart and is the only shape that carries binary. Creating with a name that already exists adds a version to that file rather than failing, which is how the platform models an edit. content writes the raw bytes, so redirect it or pass --out for anything that is not text. There is no restore verb: a write appends, it never rolls back.

Address versions create at the newest row in the chain. The new version number is computed from the row you name, so pointing it at an older version produces a duplicate number rather than the next one:

# The newest row is first in the default (newest-first) listing.
# jq -r unquotes the projected scalar; `--query` alone would keep the quotes.
HEAD=$(introspection files versions list <file-id> --query '[0].id' | jq -r)
introspection files versions create "$HEAD" --content-file next.md

delete removes the file and its whole version chain, and asks first.

Shares

A share grants read access to one file or conversation without giving away the project:

introspection shares list [--limit <n>] [--next <cursor>] [--filter KEY=VALUE]
introspection shares get <share-id>
introspection shares create --resource-type <file|conversation> --resource-id <id> \
  [--member <member-id> | --identity <identity-key>]
introspection shares delete <share-id>

Name --member or --identity to target one recipient; omit both to grant the whole project. The two are mutually exclusive, and you must own the resource you are sharing. Grants are read-only, and delete revokes the grant without deleting the resource.

Experiments

The CLI exposes the full experiment lifecycle:

introspection experiments list [--runtime <slug-or-group-id>] [--environment <env>] [--status <status>]
introspection experiments get <id>
introspection experiments create @experiment.yaml
introspection experiments update <id> @experiment-update.yaml
introspection experiments start <id>
introspection experiments end <id>
introspection experiments cancel <id>
introspection experiments delete <id>

create @FILE accepts the exact ExperimentCreate JSON or YAML document and update <id> @FILE the exact ExperimentUpdate document; use @- for stdin. Create injects the active project when omitted, and the server stays the schema authority. delete is terminal and separate from end or cancel.

A minimal valid document carries only name, runtime, arms, and goal_json:

name: support-agent-candidate
runtime: support-agent
arms:
  - runtime_id: 019...
    arm_label: baseline
  - runtime_id: 019...
    arm_label: candidate
goal_json:
  kind: composite
  direction: maximize
  components:
    - source: judge
      judge_id: 019...
      weight: 1.0

goal_json.components needs at least one entry with source: judge, a judge_id from judges list, and a positive weight. Add environment (default production) and sample_rate when you want a lane other than the default or less than the group’s full traffic.

runtime_group_id is still accepted in existing create documents, but runtime is the canonical selector. All arms must belong to the named runtime group; they use runtime IDs and are symmetric, with no create-time control arm or authored weight. end takes no winner: it stops evidence collection and deploys nothing.

Judges

Judge definitions remain git-owned recipe artifacts. The CLI reads deployed judges and changes only their operational state:

introspection judges list
introspection judges get <judge-id>
introspection judges update <judge-id> --disable
introspection judges update <judge-id> --enable --sample-rate 0.25

--sample-rate accepts a value from 0 through 1. Definition fields such as instructions, model, and run gates are not editable through this command.

Judge calibration

Export unlabeled fixtures from real conversations, label them by hand, then evaluate a judge against the labeled file through an OpenAI-compatible endpoint. The three steps are not one pipeline: the middle one is yours.

# 1. Export. Every row lands with "expected": null and "split": null.
introspection conversations get \
  --ids-file conversation-ids.txt \
  --judge-fixtures \
  --output-file judges/useful-support-resolution.calibration.jsonl
 
# 2. Label the JSONL by hand. `judges eval` fails on the first unlabeled row.
 
# 3. Evaluate.
OPENAI_API_KEY=... introspection judges eval \
  --judge judges/useful-support-resolution.yaml \
  --dataset judges/useful-support-resolution.calibration.jsonl

--judge-fixtures is what turns the export into fixture rows; without it the same command writes plain conversation bundles, which judges eval rejects. It requires --ids-file, is incompatible with --agent, and supports only --format json.

Export straight to the canonical path beside the judge: the definition and its approved dataset ship as one commit in the recipe repository.

The fixture file owns expected (pass, fail, or not_applicable) and optional split (train, dev, or test) labels; judges eval --split evaluates one split at a time. Add --include-reasoning to keep the model’s reasoning in the per-fixture output, and --concurrency to change how many fixtures are evaluated at once (default 8, from 1 through 64; lower it when the provider rate-limits).

The judge YAML owns the model configuration (see Learn from production → calibrating a judge for a complete definition). OpenAI defaults to OPENAI_API_KEY; OpenRouter defaults to OPENROUTER_API_KEY; a custom local endpoint is explicit under llm.local. The Introspection login credential is never used for the model call, and model credentials are not written into fixtures or reports.

Telemetry reads

The CLI reads conversations, lists exactly one typed event family per request, reads a single event by id, or submits a bounded metrics query:

introspection conversations list --lookback 24h --page-all
introspection conversations get conv_123 > conversation.json
introspection conversations get conv_123 --summary-only
introspection conversations get conv_123 conv_456 --output-file conversations.jsonl
introspection conversations get \
  --ids-file conversation-ids.txt \
  --judge-fixtures \
  --output-file fixtures.jsonl
introspection conversations export conv_123 \
  --format trajectory \
  --output-file conv_123.trajectory.json
introspection events list \
  --event-name introspection.observation \
  --filter lens=task_resolution \
  --lookback 7d
introspection events get <event-id>
introspection metrics query @metrics.json

Conversation get takes one or more positional conversation ids, or an --ids-file of up to 20 ids; only --summary-only insists on exactly one id. For a single id it composes the summary, every item page, and the conversation’s feedback events into one bundle on stdout; --summary-only returns just the summary and, like --judge-fixtures, supports only --format json. For several ids, or with --output-file, it writes one JSONL row per id.

Those rows are conversation bundles unless you pass --judge-fixtures, which converts each one into a judge fixture: the transcript under judge_input, plus a snapshot_hash and empty expected / split labels for you to fill in, the form Learn from production → calibrating a judge builds a dataset from.

--format chooses the representation: json (default), arrow for the item rows as one Arrow IPC stream, or trajectory for the server-projected trajectory. --agent <ROOT_OR_ID> narrows the read to root or one exact agent invocation; a multi-conversation read accepts only --agent root. Add --force to replace an existing --output-file.

--page-limit (default 100) caps the feedback-event pages read for one conversation, and --max-items, --max-bytes, and --timeout-seconds bound the rest of that conversation’s traversal. Exceeding a bound fails the command rather than silently truncating the export.

An agent task defaults its conversation id to the task id, so unless the caller supplied one, a task id from tasks list can be passed here directly.

Byte-exact exports

conversations export forwards one server-side export stream unchanged, writing each chunk straight through. Use it when you want the server’s own bytes rather than the composed bundle get builds:

introspection conversations export <conversation-id>
  [--format json|arrow|trajectory] [--output-file <path>] [--force]
  [--agent <ROOT_OR_ID>] [--service-name <name>] [--operation-name <name>]
  [--lookback-days <1-365>] [--share-id <share-id>]
  [--start-date <iso>] [--end-date <iso>]

--format defaults to json; output goes to stdout unless --output-file names a path, and creation is exclusive unless you add --force. --share-id reads through a conversation share grant instead of the project’s own access, and the remaining flags narrow which records the server includes.

events get <event-id> reads one event by id across every family, so it needs no --event-name.

The conversations list and events list reads share one flag contract: --limit (default 100), --next, --sort, --order asc|desc (default desc), --start / --end, --lookback, and repeatable --filter KEY=VALUE. --lookback is mutually exclusive with --start / --end, and the valid --sort fields differ per command and per event family: see introspection events list --help.

For paginated conversation and event listings, add --format arrow --arrow-out <path>, with --force to replace an existing Arrow file. Event Arrow pages preserve the family-typed payload struct. --page-all follows opaque cursors to exhaustion; --page-limit bounds the number of pages.

metrics query forwards the JSON document unchanged. A complete count request looks like:

{
  "view": "observations",
  "metrics": [{ "aggregation": "count" }],
  "filters": [
    { "field": "runtime_group_id", "operator": "eq", "value": "019..." }
  ],
  "from_timestamp": "2026-07-11T00:00:00Z",
  "to_timestamp": "2026-07-18T00:00:00Z"
}

Structured output

Data commands accept global output and projection flags:

introspection runtimes list -o table
introspection recipes list --query "[].id"
introspection bindings endpoints list \
  --query "[?kind=='mcp'].{id:id,name:name,url:base_url}" \
  -o table

• -o / --output: json (default), jsonc, table, or report. report renders the diagnosis that doctor and check produce; a command with no report of its own falls back to table.
• --query: a JMESPath  expression applied before formatting.
• --yes / -y answers every confirmation prompt, and --non-interactive makes a command fail with the missing flag rather than wait for a terminal.

Structured results go to stdout; progress and confirmations go to stderr, so JSON remains safe to pipe.

Success is exit 0; clap usage errors are exit 2, CLI pre-request read validation is exit 3, and runtime/API failures are exit 1. List commands return an array for the fetched page only: follow next/--next, or use --page-all where supported, rather than inferring exhaustiveness from one page. Task streams are replayable only within a bounded window, so treat resume_gap as incomplete history and refetch durable task or conversation state.

Upgrade

introspection upgrade
introspection upgrade --dry-run
introspection upgrade --format json
introspection upgrade --dismiss

--format json emits the upgrade plan as JSON and implies --dry-run, so it never changes anything. --dismiss silences the ambient notice until a release newer than the current latest ships.

The CLI checks the npm latest tag in the background, throttled to roughly once every 20 hours. Notices go to stderr and never corrupt structured output.

Related

• Recipes: the immutable, Git-backed behavior the CLI validates.
• Runtimes: runtime groups, versions, and lane routing.
• Bindings: endpoints, variables, and credentials.
• Tasks: run, cancel, resume, and stream semantics.
• Authentication: CLI login versus SDK API keys.