Learn from production

The guide follows a failure from the refund exception agent. The agent promised a refund for a final-sale item because it applied a general return rule. Before you change the prompt, you need to find what happened and how often it happens. You also need to identify the part of the agent that made the wrong decision.

The investigation is the first stage of continual learning. Feedback, observations, patterns, and judge results help you find a possible lesson and measure its scope. The agent has not learned the lesson yet. The lesson becomes part of the agent only after you test, review, and merge a change into the agent recipe. Introspection calls that final step distillation.

The examples put the CLI first for terminal investigations and include the equivalent JS SDK call for application code. You can find the same evidence in the dashboard. See Conversations, Observations and patterns, and the JS SDK.

1. Start with the strongest signal

Start with the clearest evidence that something went wrong. The evidence may be feedback, a product result, a judge failure, a pattern, an observation, or a reported task.

A negative rating or pattern label gives you a place to start. It does not tell you what to change. In the refund example, start with the task ID recorded with the complaint:

introspection tasks get <task-id>

const task = await runner.tasks.get(taskId);
console.dir(task, { depth: null });

Read the whole task record and note the following fields:

• The runtime group and exact runtime version
• metadata.completion_reason or metadata.error
• The environment and recipe commit
• metadata.conversation_id, if the agent ran

If the task has no conversation ID, the sandbox may have failed before Pi recorded a trace. Use the task record to diagnose the failure instead of assuming that the agent’s behavior caused it. See Task operations.

2. Read the exact conversation

Get the conversation identified by the task:

introspection conversations get <conversation-id> \
  --output-file refund-failure.json

const conversation = await runner.conversations.exportJson(conversationId);

Read the conversation in order. Check the user messages, evidence, model calls, tool results, errors, feedback, and judgements. Find the first point where the agent’s behavior differs from the behavior you expected.

For example, the refund conversation might contain the following events:

Policy read: final-sale items are not refundable
Order read: item 1842 is final-sale
Agent decision: approve refund under the generic 30-day return rule

The model received the correct policy and order data. A missing binding or a failed lookup did not cause the error. The decision procedure in the refund-exceptions skill caused it.

3. Check whether it recurs

One conversation proves that the failure happened once. Check more production results to learn whether it happens again, changes over time, or affects only one runtime version.

If Introspection has already created a pattern for the behavior, read the observations assigned to it:

introspection events list \
  --event-name introspection.pattern \
  --filter runtime_group_id=<runtime-group-id>
 
introspection events list \
  --event-name introspection.observation \
  --filter pattern_id=<pattern-id> \
  --lookback 30d

for await (const pattern of runner.events.list({
  event_name: "introspection.pattern",
  runtime_group_id: runtimeGroupId,
})) {
  console.log(pattern);
}
 
for await (const observation of runner.events.list({
  event_name: "introspection.observation",
  pattern_id: patternId,
  lookback: "30d",
})) {
  console.log(observation);
}

Introspection creates patterns and observations in the background, so a new task or runtime may not have them yet. If the command returns an empty list, search a longer period or start with tasks you already know about. An empty list does not prove that the behavior never occurs.

Count the distinct conversations affected during a specific period. Do not count the observations themselves because one conversation can produce several observations:

{
  "view": "observations",
  "metrics": [
    { "measure": "conversation_id", "aggregation": "count_distinct" }
  ],
  "dimensions": [{ "field": "runtime_id" }],
  "filters": [
    { "field": "runtime_group_id", "operator": "eq", "value": "<runtime-group-id>" }
  ],
  "from_timestamp": "<start-ISO-timestamp>",
  "to_timestamp": "<end-ISO-timestamp>"
}

introspection metrics query @refund-scope.json

const result = await runner.metrics.query({
  view: "observations",
  metrics: [
    { measure: "conversation_id", aggregation: "count_distinct" },
  ],
  dimensions: [{ field: "runtime_id" }],
  filters: [
    {
      field: "runtime_group_id",
      operator: "eq",
      value: runtimeGroupId,
    },
  ],
  from_timestamp: startTimestamp,
  to_timestamp: endTimestamp,
});

Record an exact finding, such as “Seven distinct production conversations in the last 30 days show the final-sale conflict, and all seven used runtime v3.” Save the time period, filters, and result for each runtime version with the investigation.

4. Compare failures with controls

Read several conversations from the same pattern. Then read some ordinary conversations where the agent behaved correctly. The successful conversations are controls that help you separate a repeated decision problem from an unusual request or bad order record. They can also show whether a stale credential, provider outage, or bad service response caused the failure.

Before you choose a fix, identify the type of failure at the first point where the actual behavior differs from the expected behavior:

Do not add a prompt rule to fix an infrastructure failure. Read the conversation before you accept the user’s explanation or a generated pattern name as the cause.

5. Form a falsifiable hypothesis

Write a hypothesis that the next evaluation can prove wrong:

When an order is marked final-sale, the agent approves a refund because the refund-exceptions skill applies the generic 30-day rule before checking the final-sale exclusion. Checking exclusions first should produce a grounded decline or escalation without changing ordinary eligible returns.

Before you edit the recipe, record the following information:

• Save the source conversation IDs and the period you measured.
• Describe the first wrong step and the part of the system that caused it.
• State the smallest change you plan to make.
• List representative failure, boundary, and control cases, along with behavior that the change could break.

Get approval for the planned scope. If later evidence points to another part of the system, stop and agree on a new scope before you expand the change.

Common failure pattern: editing before understanding

Adding “never refund final-sale items” to a general system prompt may fix one example while leaving the faulty procedure in place. The new rule can also change unrelated cases. First trace the failure to its cause, then change the smallest part of the agent that made the decision.

Done when

• You started with real feedback, a task, a judge result, or an aggregate of production results.
• You linked each failure to its task, conversation, runtime version, and recipe commit.
• You counted distinct conversations during a specific period and compared the failures with successful controls.
• Your hypothesis names the first wrong step, its cause, and the expected result. It also states which normal cases must continue to work.

Continue the continual learning loop

1. Evaluate offline explains how to change one part of the agent and replay failure, boundary, and control cases against the current version.
2. Calibrate judges explains how to create a lasting measure for a recurring quality problem when the risk requires one.
3. Experiments and shipping explains how to ship a clear winner or run a limited production comparison when the offline results do not identify a winner.