Observations & patterns

These layers let you move from “something went wrong” to “this happened 240 times this week,” then open the exact conversations behind the trend.

From activity to a recurring pattern

The layers build on one another:

1. Events record individual things that happened, including product feedback and judge results.
2. Observations summarize something notable in one completed conversation.
3. Patterns group similar observations across many conversations.

The question you are asking determines which read surface to use:

Events return individual structured records. Metrics aggregate telemetry into counts, trends, percentiles, and comparisons. Both are bounded and filterable, so an investigation can be repeated later with the same time range and filters. See Learn from production → following a signal for a practical walkthrough, JavaScript SDK → Read what happened for code, and the API Reference for every query field.

Events

An event records a durable platform signal about agent work or about the learning process built on top of it. Where a conversation preserves the detailed trajectory, events add interpretation and classification without changing that evidence.

Event reads expose durable platform signals connected to their source work and deployed recipe version. Feedback, judgements, pattern assignments, and clustering runs are returned as event streams. Observation and pattern results are current-state views: the platform folds their underlying telemetry into the latest state for each resource. This lets you compare signals across versions and experiments without implying that every event family is an immutable history.

The user-facing learning model is built from four kinds of signal:

A single conversation can accumulate several independent signals: a user leaves feedback, a judge produces a judgement, and the platform produces an observation from the same underlying work.

Observations

An observation is something the platform noticed about one completed conversation. It is generated automatically after a conversation becomes inactive: there’s nothing to wire up and nothing to enable. A conversation becomes eligible after 30 minutes without new chat activity, and the background scan normally runs every 10 minutes.

Every observation is recorded along exactly one of five lenses:

A conversation produces an observation on a lens only when there’s something worth noting there, and at most three per lens, so a conversation that genuinely struggled in more than one way can say so without padding. Each observation carries a one-sentence summary and a severity of low, medium, or high. Two lenses always carry one more field, because a dashboard charts share-of-conversations by it: task resolution carries resolution (resolved, blocked, or abandoned), and user sentiment carries sentiment (positive or negative).

Because observations are generated per completed conversation, a long-running conversation that completes, continues, and completes again is observed in segments: later turns become new observations rather than re-stating earlier ones. Trends are counted over distinct conversations so repeats don’t inflate them.

Patterns

A pattern is a named cluster of similar observations across many conversations. Where an observation is about one conversation, a pattern is about a recurring behavior or failure mode: it’s how “the agent keeps doing X” becomes a single thing you can point at, name, and track.

Patterns are mutable: as new observations arrive they’re sorted into existing patterns, and the cluster map is periodically regenerated, naming patterns, refining descriptions, minting new ones, and retiring stale ones. When a regenerated cluster remains close enough to an earlier one, it keeps that pattern’s identity so the trend stays continuous. A cluster that changes too much receives a new identity instead.

Patterns are scoped per organization, project, and lens, and normally by runtime group. Observations without runtime-group attribution are clustered in a separate project-level bucket rather than mixed into an agent lineage.

Metrics

A metrics request is one JSON document posted to POST /v1/metrics, submitted with introspection metrics query or runner.metrics. You choose a view — spans, conversations, events, judgements, observations, or patterns — then the measures to aggregate and the dimensions to group by.

spans is the per-model-call grain, so cost, token, and latency questions live there. Grouping by recipe_git_commit_sha attributes a regression to a recipe change, since that commit is the platform’s prompt version.

Field names per view are in the API reference.

The investigation path

Investigation runs top-down:

Patterns          → "agent_struggle: re-asks for confirmation it already has" (240×)
  → Observations  → the individual conversations that landed in this cluster
    → Conversations → the full, replayable transcript of any one of them

You start from a pattern (the aggregate behavior), drill into the observations that make it up, and open the underlying conversation replay to see exactly what happened. Each step narrows from “what is my agent doing at scale” to “show me the turn where it went wrong.”

Why this matters

Investigation gives you reproducible, filterable answers about agent behavior: “how many conversations hit this failure mode this week, broken down by version” returns the same structured result every time you ask, not a different ranked list.

Structured lenses and patterns mean the same query gives the same answer, you can filter and group on stable dimensions, and you can trend a behavior across runtime versions with confidence. Use those findings to decide what deserves a durable judge; experiments then compare versions using the judge’s emitted verdicts.

How it connects

• Tasks & conversations are the source: an observation is generated when a conversation completes, and the investigation path bottoms out in the conversation replay.
• Judges complement observations: observations are what the platform noticed automatically; a judge is what you decide to measure. Both live on the same event stream.
• Product feedback enters through feedback() in your application and is read back as introspection.feedback evidence. It is different from the live AG-UI events streamed while a task runs.