Observing & investigating

Completed tasks become immutable conversations. The platform then emits typed observation events and maintains a folded pattern catalog. Synthesis and clustering are asynchronous, so a newly completed task may appear before its observations or patterns do.

How the signal forms

Every task becomes a conversation: the replayable sequence of messages, model calls, and tool calls. Shortly after a conversation completes, the platform synthesizes sparse observations along a fixed set of lenses:

A pattern is a recurring cluster of similar observations within one lens and runtime group. Patterns turn many isolated runs into an investigation queue: a name, a count, and a set of concrete examples instead of a wall of transcripts.

Investigate top-down

For an established runtime, investigate from the aggregate signal down to the evidence:

1. Patterns        pick a recurring pattern (e.g. "asks for a refund, gets a policy wall")
2. Observations    inspect the observations grouped under it, filtered by lens / severity
3. Conversation    open one underlying conversation and replay it turn by turn

This path is reproducible because every hop uses structured IDs and filters. At the conversation, corroborating evidence—tool calls, observations, judge results, and user feedback—can be read against the same immutable timeline.

On a fresh runtime, one smoke task may not have produced a cluster yet. This guide therefore starts from the known conversation, waits briefly for its observations, and expands through a pattern when one exists. With representative traffic, the preferred investigation path remains top-down.

The examples continue from Connecting an agent. Each operation shows the JavaScript SDK first and the equivalent CLI flow second.

1. Find a conversation

const runner = await client.runtimes("support-agent").run({
  identity: { user_id: "ops" },
});
 
const page = await runner.conversations.list({
  runtime_id: RUNTIME_ID,
  limit: 100,
});
 
let conversationId = page.records[0]?.conversation_id;
if (!conversationId) throw new Error("No conversation found for this runtime");

introspection conversations list \
  --filter runtime_id="$RUNTIME_ID" \
  --lookback 24h \
  --page-all > conversations.json
 
jq '[.[] | {conversation_id,status,updated_at}]' conversations.json
export CONVERSATION_ID=$(jq -r '.[0].conversation_id' conversations.json)
test "$CONVERSATION_ID" != null

If the list is empty, confirm the task settled and that RUNTIME_ID is the exact version that handled it. Increase --lookback only after those checks.

2. Poll for observations, with a transcript fallback

Poll for at most four minutes. Observation synthesis is sparse, so zero rows is a valid result rather than a failed task.

let observations = [];
 
for (let attempt = 0; attempt < 24; attempt++) {
  const page = await runner.events.list({
    event_name: "introspection.observation",
    conversation_id: conversationId,
    lookback: "24h",
  });
  observations = page.records;
  if (observations.length > 0) break;
  await new Promise((resolve) => setTimeout(resolve, 10_000));
}
 
for (const observation of observations) {
  console.log(observation.payload.lens, observation.payload.summary);
}

for attempt in {1..24}; do
  introspection events list \
    --event-name introspection.observation \
    --filter conversation_id="$CONVERSATION_ID" \
    --lookback 24h \
    --page-all > observations.json
  [ "$(jq 'length' observations.json)" -gt 0 ] && break
  sleep 10
done
 
jq '[.[] | {
  observation_id: .payload.observation_id,
  conversation_id,
  lens: .payload.lens,
  severity: .payload.severity,
  summary: .payload.summary
}]' observations.json

Continue even when the array is empty; step 4 retrieves the durable conversation directly.

3. Expand through a pattern when one exists

introspection.pattern is a current-state fold, not a historical stream. It accepts --end, but not --start or --lookback.

const patternPage = await runner.events.list({
  event_name: "introspection.pattern",
  runtime_group_id: RUNTIME_GROUP_ID,
  status: "active",
});
 
const patternId = patternPage.records[0]?.payload.pattern_id;
if (patternId) {
  const assigned = await runner.events.list({
    event_name: "introspection.observation",
    pattern_id: patternId,
    lookback: "30d",
  });
  conversationId = assigned.records[0]?.conversation_id ?? conversationId;
}

introspection events list \
  --event-name introspection.pattern \
  --filter runtime_group_id="$RUNTIME_GROUP_ID" \
  --page-all > patterns.json
 
jq '[.[] | {
  pattern_id: .payload.pattern_id,
  name: .payload.name,
  lens: .payload.lens,
  status: .payload.status,
  last_detected_at: .payload.last_detected_at
}]' patterns.json

export PATTERN_ID=$(jq -r '[.[] | select(.payload.status == "active")][0].payload.pattern_id // empty' patterns.json)
 
if [ -n "$PATTERN_ID" ]; then
  introspection events list \
    --event-name introspection.observation \
    --filter pattern_id="$PATTERN_ID" \
    --lookback 30d \
    --page-all > pattern-observations.json
  PATTERN_CONVERSATION_ID=$(jq -r '.[0].conversation_id // empty' pattern-observations.json)
  [ -n "$PATTERN_CONVERSATION_ID" ] && export CONVERSATION_ID="$PATTERN_CONVERSATION_ID"
fi

No active pattern is expected on a fresh project or after one smoke task. Generate more representative traffic and return later; it does not block transcript investigation or judge calibration.

4. Retrieve the complete evidence bundle

for await (const item of runner.conversations.items.list(conversationId, {
  order: "asc",
})) {
  console.log(item.id, item.output_message);
}
 
await runner.close();

introspection conversations get "$CONVERSATION_ID" > conversation.json
jq '{
  conversation: .conversation.conversation_id,
  complete: .meta.complete,
  item_count: (.items | length),
  feedback_count: (.feedback_events | length)
}' conversation.json

The bundle contains the summary, every caller-visible item page, and feedback events. Read items in order to reconstruct the transcript and tool trajectory.

5. Count instead of downloading

Use metrics for an aggregate. Save this complete request as observation-count.json:

{
  "view": "observations",
  "metrics": [{ "aggregation": "count" }],
  "filters": [
    { "field": "runtime_group_id", "operator": "eq", "value": "REPLACE_RUNTIME_GROUP_ID" }
  ],
  "from_timestamp": "2026-07-11T00:00:00Z",
  "to_timestamp": "2026-07-18T00:00:00Z"
}

Replace the ID and timestamps, then run:

introspection metrics query @observation-count.json

Done when

• You inspected the synthesized observations, or confirmed that none appeared during the wait.
• When an active pattern exists, you can trace it to its assigned observations.
• You retrieved at least one complete conversation bundle behind the signal.
• You treated an empty asynchronous result as something to recheck or widen, not as proof that no signal exists.

Keep several representative CONVERSATION_ID values for judge calibration.

Patterns tell you what the platform noticed. Once you can state what the agent should have done instead, that expectation is ready to become a judge: a versioned standard you can calibrate, trend, and use to score an experiment.

Next

• Evaluating with judges: label those conversations and test a rubric locally.

Related

• Observations & Patterns: event schemas and lifecycle.
• CLI → Telemetry reads: pagination, Arrow, and typed families.