Learn from production

An improvement can start with a thumbs-down, a product outcome, a recurring pattern, an observation, or a reported task. This guide is the whole loop: follow that signal back to the exact conversation, form a hypothesis, prove the fix locally, and — when local evidence genuinely cannot decide — measure the question in production.

Signal → Evidence → Hypothesis → Change → Measure → Decide

If the improvement cycle is new to you, read the learning loop first, along with how conversations capture individual runs and how observations and patterns reveal behavior across many runs.

1. Start with the strongest signal you have

A pattern label or negative rating tells you where to investigate, not what to change. Feedback describes an outcome. An observation describes something seen in one run. The conversation shows the behavior that produced it.

Observation synthesis and pattern clustering are asynchronous, so a completed conversation may appear before either aggregate does. On a fresh runtime, start from a known task. On an established one, start from the aggregate and drill down.

2. Follow the signal to a conversation

From a known task

When you already have the task — a smoke run, a bug report, the task your application recorded — use its metadata.conversation_id. That deterministic link is safer than taking the first row of a runtime-wide list, especially with concurrent traffic.

const task = await runner.tasks.get(TASK_ID);
const conversationId = task.metadata?.conversation_id;
if (!conversationId) throw new Error("Task has no completed conversation yet");

introspection tasks get <task-id>

If metadata.conversation_id is absent, read the rest of the task’s metadata before retrying: error on a failed task, completion_reason on a completed one. A task that failed before its agent ran never produces a conversation, and a task torn down by the idle window is completed without a result. In both cases the task row is the evidence. See task statuses and diagnosis.

From a recurring pattern

introspection.pattern is a current-state fold, not a historical stream: it accepts --end, but not --start or --lookback.

JavaScript SDK

const patterns = await runner.events.list({
  event_name: "introspection.pattern",
  runtime_group_id: RUNTIME_GROUP_ID,
  status: "active",
});
 
const patternId = patterns.records[0]?.payload.pattern_id;
const assigned = await runner.events.list({
  event_name: "introspection.observation",
  pattern_id: patternId,
  lookback: "30d",
});

Each pattern row carries payload.pattern_id, name, lens, and status; each observation carries lens, severity, and summary. No active pattern is expected on a fresh project or after one smoke task — an empty result is something to recheck or widen, never evidence that no signal exists.

3. Measure how big it is

An individual event proves something happened. A metrics query tells you whether it is isolated, recurring, improving, or concentrated in one runtime version. Count distinct conversations rather than raw observation rows, so one conversation producing observations through several lenses cannot inflate the result.

JavaScript SDK

const scope = await runner.metrics.query({
  view: "observations",
  metrics: [{ measure: "conversation_id", aggregation: "count_distinct" }],
  dimensions: [{ field: "lens" }],
  filters: [
    { field: "runtime_group_id", operator: "eq", value: RUNTIME_GROUP_ID },
  ],
  from_timestamp: START_ISO,
  to_timestamp: END_ISO,
});

Use the same window when comparing versions. If the question is about feedback, judgements, latency, cost, or product signals, choose the corresponding metrics view rather than forcing everything through observations. The full grammar is in Metrics.

4. Read the behavior in context

Open the entire conversation — messages, tool calls, errors, feedback, judge results — and find the earliest point where behavior diverges from what should have happened.

// Items arrive newest-first; the route takes no ordering parameter.
for await (const item of runner.conversations.items.list(conversationId)) {
  console.log(item.span_id, item.attributes.gen_ai?.output?.messages);
}

introspection conversations get <conversation-id>

Then compare it with other conversations behind the same pattern, a few ordinary conversations where the agent behaved correctly, and the runtime version, environment, and recipe commit behind each run.

The control sample matters. It distinguishes a recurring agent problem from one unusual request, an expired credential, a provider outage, or a bad upstream response.

5. Form a hypothesis

Write down what you think happened in a form the evidence can disprove:

When this situation occurs, the agent does this because this part of the system makes the decision. Changing this mechanism should produce this result without breaking these normal cases.

Choose the layer from the behavior you inspected:

Do not turn an infrastructure failure into a prompt rule. Do not treat the user’s initial explanation — or the pattern’s generated label — as the cause.

Before editing, summarize the evidence you read, the earliest meaningful divergence, the hypothesis, the smallest change you recommend, and what could regress. Get approval for that scope. If the evidence later points somewhere materially different, stop and agree on the new scope instead of silently expanding the pull request.

6. Change one mechanism and prove it locally

Capture current behavior before editing when the change needs a comparison. Make one coherent change in the right place, then run the affected cases, existing local evals, and a few normal cases in fresh Pi sessions. Read the traces behind the results instead of relying on an aggregate score.

Use the lightest evaluation that answers the question:

evals/ is a useful convention beside the recipe, but it is not part of the recipe format: Introspection does not validate or deploy that directory. Add a new eval only when the behavior is important, recurring, and difficult to protect with an ordinary test.

Score the result separately from running it. Use code for outcomes that can be checked exactly, and reviewed semantic scoring or human review when quality depends on meaning. A text check is not semantic merely because it is deterministic: if a verifier interprets meaning from text, prove that a wrong result containing every expected keyword still fails, and that a correct result with harmless formatting differences still passes. Keep both as regression coverage for the verifier.

A local service or database does not by itself require Harbor. Keep a host-process runner when it can start the real component, replace or replay only an external boundary, reset state between cases, and inspect the result. Choose Harbor when that environment or its isolation is part of the behavior under test.

Before comparing anything with a controlled task, confirm that the untouched starting state fails and a reference solution passes. A task that passes without the fix measures nothing.

Compare a candidate against the baseline

To try one recipe change without editing the base definition, add a small inherited agent. Its declared name — not its filename — is what --agent selects:

name: agent-candidate
from: agent
system_instructions:
  mode: append
  content: |
    For structured-output requests, validate every requested field before
    returning the final object.

Run the unchanged agent first, then repeat the identical command with the candidate:

introspection eval run --runner harbor --path evals/tasks/acme-q2 --agent agent
introspection eval run --runner harbor --path evals/tasks/acme-q2 --agent agent-candidate

--agent accepts one name, so each candidate is its own invocation. Change one coherent mechanism per candidate, and confirm both records use the same recipe revision, runner, task, model, and trial count. Agent inheritance is field-specific — capability arrays replace rather than append — so review the effective change before attributing a result to one instruction. Agent definitions covers what from: does and does not carry.

When a managed Recipe failure depends on conversation context, start with the smallest useful replay: run the original user prompt for that turn against the unchanged agent, then the same source conversation, turn, prompt, model, runner, scorer, and trial count against one candidate. Include an ordinary control and a relevant boundary case. CLI → Replay production context has the --from-conversation options and what hydration does not restore.

Keep the model, provider, recipe revision, and other relevant configuration fixed while comparing baseline and candidate. A replay supports a directional engineering decision, not a production-wide claim:

On this replay, the candidate corrected the observed behavior; the selected controls showed no obvious regression. Production-wide impact remains unverified.

If the candidate improves the reported cases but damages normal ones, revise the hypothesis rather than accepting the change. Keep the source conversation IDs, hypothesis, what changed, and remaining risks in a focused pull request.

7. Turn a durable risk into a judge

A fix you cannot measure will regress silently. When the behavior is a recurring semantic quality that an ordinary test cannot capture, encode it as a judge — one narrow question about a completed conversation — and leave it running as a standing regression guard.

Calibration does not create ground truth. It checks whether the exact rubric and model agree with labels your team already owns.

Choose one decision a reviewer can answer consistently, usually pass or fail, and agree up front on what counts as a false positive, what counts as a false negative, and which mistake costs more.

Select and label real conversations: clear passes, clear fails, near-boundary cases, and ordinary controls from the same traffic mix. A domain owner reviews every label before it becomes calibration data — never let the judge write the standard it will be scored against. Use train while shaping the rubric, dev for repeated checks, and hold test out for the final decision.

Confirm the fixtures can live in the repository before exporting. Remove secrets and sensitive user data, or replay an authorized sanitized conversation and export that instead.

Export authentic fixtures, at most 20 per call, beside the judge:

introspection conversations get \
  --ids-file conversation-ids.txt \
  --judge-fixtures \
  --output-file judges/useful-support-resolution.calibration.jsonl

--judge-fixtures requires --ids-file and is what makes the export a calibration dataset — without it you get ordinary conversation bundles that judges eval rejects. Every row lands with "expected": null, so the dataset is unusable until a human labels each row. Merge only the approved top-level expected (pass, fail, or not_applicable) and optional split into each exported object:

{"expected":"pass","split":"train"}

Preserve every other exported field exactly; the row’s snapshot_hash is the sha256 of its judge_input, so editing one byte inside it invalidates the row. The complete fixture contract is in Judges → calibration datasets.

Write the judge beside its dataset, describing observable outcomes rather than preferred phrasing:

name: useful-support-resolution
description: Did the response move the support request toward a useful resolution?
llm:
  provider: openai
  model: gpt-4.1-nano
instructions: |
  Grade the assistant's handling of a customer-support request.
 
  Return pass when it gives an actionable answer grounded in approved policy,
  asks for information genuinely needed to proceed, or clearly escalates a
  sensitive request and explains the next step.
 
  Return fail when it only repeats a policy, makes an unsupported promise, or
  leaves the customer without a useful next step.
 
  Return not_applicable when the conversation is not a support request.
  Output { reasoning, verdict }.

Calibrate locally. Tune on train, check on dev, then run the held-out split once:

OPENAI_API_KEY=... introspection judges eval \
  --judge judges/useful-support-resolution.yaml \
  --dataset judges/useful-support-resolution.calibration.jsonl \
  --split train

Read every disagreement rather than counting them; overall agreement can hide a judge that predicts the majority label every time. What a disagreement usually means lists the failure modes that produce confident, consistent, wrong verdicts. If the held-out result misses the criteria you chose, revise the judge and recalibrate with a new held-out set — never move or relabel test cases to improve the score.

Version the judge and its dataset together. They are one measurement contract, and changing the rubric or model creates a different measurement that needs another calibration pass. Move them to production through the agent development lifecycle.

8. Decide from the results

When the affected cases and controls clearly support the change, open the focused pull request and merge it through the normal Git workflow. The agent development lifecycle covers how the merged recipe version reaches production.

When credible candidates remain and local evaluation cannot choose between them, do not merge one on intuition — and do not reach for an experiment that a local test already answers. An experiment earns its complexity when:

• the before and after windows would see different traffic mixes;
• the candidate is risky enough that exposure should be bounded;
• or the decision needs same-population evidence rather than a directional trend.

If none of those hold, a normal deploy plus a judge pass-rate split by runtime version is simpler and easier to explain.

To run one, the judge that will score the comparison must already be deployed: an experiment’s goal points at an exact live judge, and every arm must emit the same judge ID and definition hash. Read those from the deployed recipe version:

introspection judges list
introspection judges get <judge-id>

Then select the baseline and candidate versions from the same runtime group and create the experiment. create takes the ExperimentCreate document with @, not a flag — CLI → Experiments shows a minimal valid one:

introspection runtimes list --runtime <runtime-slug> -o table
introspection experiments create @experiment.yaml
introspection experiments start <experiment-id>

Then send representative traffic with stable subjects, so each subject keeps its arm assignment:

introspection tasks create \
  --experiment <experiment-id> \
  --subject user-001 \
  --prompt "Help me resolve my support request."

Confirm the created task’s metadata.experiment_arm_id is set before counting the run as evidence; an unset arm means the subject fell outside the sample and received normal production. Synthetic repetition is useful for a staging rehearsal, but it is not production evidence.

Experiments covers arms, sticky assignment, goals, and the lifecycle in full, including how to read the evidence and when to stop.

Done when

• The improvement is tied to real feedback, observations, patterns, or conversations.
• The signal was measured over an explicit window, not inferred from one record.
• The hypothesis names the earliest divergence and where the problem likely lives.
• Normal conversations were checked alongside the failures.
• Baseline and candidate were evaluated with the same relevant configuration, and reviewed at the trace level.
• Any replay conclusion is scoped to the captured context and selected controls.
• A durable behavioral risk carries a calibrated judge, reviewed as one change with its dataset.
• The work ends in either a focused fix ready to merge or a bounded experiment for a question local evaluation could not answer.

Next

• Agent development lifecycle: move the merged recipe change through staging to production.
• Work with repositories: let the agent prepare the fix as a pull request against your source.

Related

• Learning loop: how production evidence becomes a better recipe.
• Observations & Patterns: event schemas, lifecycle, and the metrics grammar.
• Judges: judge, judgement, calibration, and run-gate semantics.
• Experiments: arms, routing, goals, and stopping rules.
• CLI: commands for conversations, events, metrics, judges, and experiments.