Evals & telemetry
Reads accept your login or an
API key; eval and judges eval run locally
beside the recipe. Workflows: Debug,
Evaluate, Calibrate judges,
Experiment.
Conversations
| Command | Does |
|---|---|
conversations list [--filter KEY=VALUE]... [list flags] | Conversation summaries (alias conv). |
conversations get <id>... [--ids-file <path>] [--summary-only] [--format json|arrow|trajectory] [--agent root|<id>] [--output-file <path>] | Summary, every item, and feedback in one bundle per id. |
conversations export <id> [--format ...] [--output-file <path>] [--service-name ...] [--operation-name ...] [--lookback-days 1-365] [--start-date ...] [--end-date ...] | One conversation’s export bytes, unchanged; date flags bound records within it. |
A task id works as a conversation id. get fails rather than truncates when
--max-items, --max-bytes, or --timeout-seconds is hit.
# The tool calls in one conversation
introspection conversations get <task-id> \
--query "items[?attributes.gen_ai.operation.name=='execute_tool'].{name: attributes.gen_ai.tool.name, args: attributes.gen_ai.tool.call.arguments}"
# Export a time range: list the ids, then fetch them into one JSONL file
introspection conversations list --start <iso> --end <iso> --page-all --query "[].id" -o table
introspection conversations get <id-1> <id-2> --output-file conversations.jsonl--ids-file <path> takes up to 20 ids, one per line, instead. --query does not
apply to get with several ids or --output-file, nor to export.
Filter conversations
--filter keys (repeat to AND; unknown keys are ignored):
| Key | Values |
|---|---|
conversation_id, conversation_ids, trace_id, annotation_id | Ids (trace_id is 32 hex). |
runtime_id, runtime_group_id, experiment_id, recipe_git_commit_sha | Version, runtime, experiment, commit. |
environment | development, staging, production. |
model, agent_name, service_name, service_names | What ran. |
status | Ok, Error. |
resolution | resolved, blocked, unresolved, pending. |
sentiment | positive, negative, mixed, neutral. |
owner_key | user:<id>, anon:<id>, member:<id> (there is no user key). |
metadata | key:value, repeatable. |
Events
introspection events list --event-name <family> [--filter KEY=VALUE]... [list flags]
introspection events get <event-id>list reads one family: introspection.observation, introspection.pattern,
introspection.pattern.assignment, introspection.observation_clustering.run,
introspection.judgement, introspection.feedback,
introspection.annotation, or introspection.repository.merge
(what each holds).
The CLI cannot list introspection.track, introspection.automation.triggered,
or introspection.automation.skipped; use the API
(GET /v1/events?event_name=<family>) or a metrics query on the events view.
introspection events list --event-name introspection.judgement \
--query "[?payload.result=='fail'].{conversation:conversation_id,reason:payload.reasoning}" -o table| List flag (conversations and events) | Meaning |
|---|---|
--limit, --next, --page-all, --page-limit | Page size (1 to 1000, default 100) and cursors. |
--start, --end / --lookback 24h | Absolute or relative window. |
--sort, --order asc|desc | Conversations: created, duration, turns, tool_calls, llm_calls, tokens, cost. Events: per family (events list --help). |
--format arrow --arrow-out <path> [--force] | Arrow output. |
Metrics
introspection metrics query @metrics.json # or @- for stdin{
"view": "conversations",
"metrics": [{ "measure": "total_cost_usd", "aggregation": "sum" }],
"dimensions": [{ "field": "model" }],
"from_timestamp": "2026-07-11T00:00:00Z",
"to_timestamp": "2026-07-18T00:00:00Z"
}The document is forwarded unchanged. Views, fields, and example queries: Metrics.
Annotations
introspection annotations list [--conversation-id <id>] [--trace-id <id>] [--span-id <id>] [--label <label>] [--assigned-to <email>] [--page-all]
introspection annotations create --trace-id <id> --span-id <id> [--label <l>... | --clear-labels | --reviewer <email>... | --clear-reviewers] [--comment <text>]Labels and reviewers are complete snapshots, and one call sets at most one of
them: set labels and reviewers in separate calls. A comment is appended.
introspection open --annotations opens the review queue.
Offline evals
| Command | Does |
|---|---|
eval create <name> [--with-judge | --judge <name>] | Writes evals/vitest/<name>.eval.ts and .eval.jsonl; --with-judge adds a judge with working and held-out examples. |
eval add <name> (--trace-id <id> | --from-conversation <id> --turn <n>) [--case-name <n>] [--expected pass|fail|not-applicable] [--judge <n>] | Adds one production turn as a case (no bulk add). With --expected, also adds a judge example. |
eval run [-p <path>] [-r <runtime>] [--agent <name>] [--trials <n>] [-- <vitest-args>] | Runs the suite against the local recipe; writes eval-runs/<run-id>/run.json (gitignore it; move it with an absolute INTROSPECTION_EVAL_RUNS_DIR). Exits 0 whenever the report was written, even when cases fail. --trials N makes N separate runs. |
eval run ... --from-conversation <id> [--turn <n>] [--prompt <text>] | Replays a production conversation’s history first (conversation state only, not files or side effects). |
eval view [<run-id>] | Opens the Vitest report. |
eval compare [<run-a>] [<run-b>] / --baseline | Case-by-case improved, regressed, unchanged, new, removed, incomplete, plus each check’s and judge’s mean (base.scores, candidate.scores). Refuses runs whose case fingerprints differ. |
eval compare ... --fail-on regressed|incomplete|score-drop | Prints the comparison, then exits 1 with a one-line reason when the condition holds (repeatable; CLI 0.52.0 or later). Without it, compare always exits 0. |
eval baseline promote [<run-id>] [--allow-dirty] / check [-p <path>] | Writes <suite dir>/baseline.json from a complete run of the whole suite, refusing a run made with uncommitted changes unless --allow-dirty / verifies baselines still cover their suites (runs nothing). |
# CI gate
introspection eval compare --baseline --fail-on regressed --fail-on incomplete
# Mean per check and judge, both sides
introspection eval compare --baseline --query "{base:base.scores,candidate:candidate.scores}"
# Flaky cases across two trials of one --trials run
introspection eval compare <trial-1-run-id> <trial-2-run-id> --query "cases[?change!='unchanged'].id" -o tablescore-drop fails when the candidate’s mean score is below the base’s over
the cases both measured.
A case row holds the input; expected and any other keys you add are passed to
the test unchanged:
{"schema_version":2,"name":"asks-for-account-id","input":{"history":[{"type":"message","role":"user","content":"Check my account."}],"prompt":"It is acct_123."},"expected":{"account":"acct_123"},"source":{"conversation_id":"...","trace_id":"...","turn":2}}A case counts as pass or regress only when a judge scored it (recipeJudge
or a createJudge code judge); an expect()-only case is incomplete.
vitest, vitest-evals, and @introspection-ai/vitest-evals must be dev
dependencies; the CLI says what to install when they don’t match. Writing
cases: Add cases to an eval.
Harbor
introspection setup --target harbor
introspection eval run --runner harbor -p <task-or-dataset> [-r <runtime> --commit-sha <sha>] [--retry-errors <n>] [--fresh-steps] [--agent harbor:oracle|harbor:nop]| Flag | Effect |
|---|---|
--commit-sha <sha> | Runs a pushed recipe commit (full 40-character SHA) in the runtime’s trial on the platform; needs -r and implies Harbor. Needs an owner or admin login; an API key alone can’t resolve the runtime. |
--retry-errors <n> | Reruns trials that failed on infrastructure, not on the agent, up to n times. |
--fresh-steps | Starts a new conversation at each step of a multi-step task. |
--agent harbor:oracle / harbor:nop | Runs the task’s own solution / sends no turn, to check the task itself. |
Walkthrough: Run Harbor tasks.
Fake MCP servers
introspection bindings mcp capture <server-id>... [--out fakes/mcp] [--api <file|url>] [--check]
introspection bindings mcp types <tools.json> --out contract.ts [--check]
introspection bindings mcp observe <server-id>... [-r <runtime>] [--lookback 30d] [--examples [<dir>]] [--check | --verify <responses.jsonl>]| Command | Does |
|---|---|
capture | Reads the bound server’s initialize and tools/list (never a tool call) and writes <out>/<server-id>/tools.json and manifest.json. |
types | Generates a TypeScript contract from tools.json. |
observe | Learns call shapes from the project’s traces of the live server into observed.json; --examples writes raw sampled calls outside the repository. |
--check | Re-derives and exits non-zero naming what changed; writes nothing. Use it as a drift gate. observe --check re-derives over the window the committed observed.json records and refuses --lookback. |
--verify | Checks a fake’s recorded answers against observed.json, and lists without failing the served tools it has no observation of. |
Walkthrough: Fake MCP servers.
Judges
| Command | Does |
|---|---|
judges list / get <id> | Deployed judges. |
judges update <id> (--enable | --disable | --sample-rate <0..1>) | Changes only operational state; definitions live in the recipe (Judges). |
judges add <name> (--trace-id <id> | --from-conversation <id> --turn <n>) --expected <pass|fail|not-applicable> [--validate] | Adds a judge example to judges/<name>.eval.jsonl (or .validate.jsonl). The verdict is human ground truth. |
judges eval <name> / judges validate <name> | Scores the judge on working / held-out examples: accuracy, Cohen’s kappa, TPR, TNR (fail is positive), confusion matrix. --concurrency (default 8). |
judges run --task-id <id> [--judge <name>]... | Grades one finished task with your provider key; results land in development. |
Judge YAML
judges/<name>.yaml, a direct child of judges/. Unknown fields are errors.
| Field | Required | Meaning |
|---|---|---|
name | Yes | Lowercase kebab-case, unique in the recipe. |
instructions | Yes | The rubric. |
llm.model | Yes | The grading model. llm.provider is openai or openrouter for local runs. |
description | No | The question in one line. |
type | No | online (default) grades runtime conversations; offline grades eval trials. |
on | No | A list of matchers; any one matching runs the judge. Events are message, tool, and feedback; regex values use /pattern/flags. |
llm.request | No | temperature, max_tokens, reasoning_effort. |
llm.transport | No | timeout_ms, max_retries, max_retry_delay_ms. |
llm.local | No | base_url and api_key_env for your own OpenAI-compatible endpoint. |
Judge example rows (<name>.eval.jsonl, <name>.validate.jsonl) hold
schema_version: 2, trajectory, optional system_instructions, and
expected (pass, fail, or not_applicable).
Judge calls use the endpoint in the judge YAML (OPENAI_API_KEY,
OPENROUTER_API_KEY, or llm.local), never your login. Dashboard datasets
hold labels, not verdicts: list them with annotations list --label <l> and
judges add each trace with your verdict.
Experiments
introspection experiments list [--runtime <slug>] [--environment <env>] [--status <draft|running|ended|cancelled>]
introspection experiments get <id> | start <id> | end <id> | cancel <id> | delete <id>
introspection experiments create @experiment.yaml
introspection experiments update <id> @update.yamlname: support-agent-candidate
runtime: support-agent
arms:
- { runtime_id: 019..., arm_label: baseline }
- { runtime_id: 019..., arm_label: candidate }
goal_json:
kind: composite
direction: maximize
components:
- { source: judge, judge_id: 019..., weight: 1.0 }create makes a draft; nothing routes until start. Arms are 2 to 20 versions
of one runtime; the goal needs at least one source: judge component (the only
source implemented). Traffic within the experiment is allocated adaptively;
there is no fixed split.
Optional environment (default production) and sample_rate. While running
only description changes. end deploys nothing. Route a task in with
tasks create --experiment <id> --subject <id>. See
Experiments.