# Label production steps

Collect failures and good behavior from production into named datasets your team reviews together.

A **dataset** is an annotation tag such as `refund-mistakes`. Each step (span)
of a conversation you label with it is an **example**. Datasets point at
production data; they hold no copies and no verdicts. Use them to collect
material for [eval cases](/improve/eval-cases) and
[judge examples](/improve/judges).

## 1. Create the dataset

```bash
introspection tags create annotation refund-mistakes --color "#f97316" \
  --description "Refunds the agent got wrong"
```

Or **Datasets → Manage datasets** in the dashboard. Names are slugified.
Annotation tags are a separate catalogue from access tags (`tags list --kind
annotation` or `--kind access`).

## 2. Find the step

Open the conversation in the dashboard, or read its turns with
`introspection conversations get <conversation-id> -o json`; each item carries
its `trace_id` and `span_id` ([finding conversations](/observe/conversations)).

## 3. Label, comment, assign

In the dashboard, use **Annotate** on any step. From the CLI, each command
sets labels *or* reviewers, plus an optional comment.

```bash
introspection annotations create --trace-id <trace-id> --span-id <span-id> \
  --label refund-mistakes --comment "Promised a refund outside policy"

introspection annotations create --trace-id <trace-id> --span-id <span-id> \
  --reviewer alex@example.com --comment "Needs a judge"
```

- `--label` and `--reviewer` replace the step's whole set. Read the current
  labels first (`annotations list --span-id <span-id>`) and repeat them.
  `--clear-labels` and `--clear-reviewers` empty a set.
- Pass `--event-id <uuidv7>` and reuse it on a retry, so a retry never posts
  a comment twice.

## 4. Review as a team

**Datasets** in the sidebar lists each dataset; **Review queue** holds the
steps assigned to you. An item's review status is `backlog` (unassigned),
`assigned`, or `reviewed`. A step is `reviewed` once a person comments on it,
so a reviewer finishes an item by commenting.

```bash
introspection annotations list --label refund-mistakes --page-all -o json
introspection annotations list --assigned-to alex@example.com
```

Still unreviewed (no human comment yet):

```bash
introspection annotations list --assigned-to alex@example.com --page-all \
  --query "[?!has_human_comment].{trace:trace_id,span:span_id,conversation:conversation_id}" -o table
```

## 5. Pull in user feedback

Thumbs-down are candidates for review, not labels. List recent negative
feedback, then open each conversation and label the step:

```bash
introspection events list --event-name introspection.feedback --lookback 7d --page-all \
  --query "[?payload.name=='thumbs_down'].{conversation:conversation_id,comments:payload.comments}" -o table
```

Sending feedback from your app: [Patterns and feedback](/observe/patterns).

## 6. Export

Open a dataset in the dashboard and export it as JSONL: `trace_id`, `span_id`,
`conversation_id`, input, output, labels, latest comment. Next:
[turn examples into eval cases](/improve/eval-cases).

## Verify

`introspection annotations list --label refund-mistakes` returns your step.

## Limits

| Not supported | Do this instead |
| --- | --- |
| Bulk label or bulk assign | Run `annotations create` once per step. |
| Accept/reject workflow for agent outputs | Use a label per decision (`approved`, `rejected`). |
| Slack reactions as feedback | Not delivered; see [Slack limits](/patterns/slack#limits). |
| Annotating as your backend (API key or service account) | Annotations need a person's own sign-in: the CLI, the dashboard, or a review tool that signs each reviewer in and calls the Node SDK's `annotations.create` with their token. Otherwise send `feedback()`. |
| Fine-tuning from a dataset | Export JSONL and train elsewhere. |

Or ask your agent: "Label the last 20 thumbs-down refund conversations into refund-mistakes and assign them to Alex."

Label production steps

Collect failures and good behavior from production into named datasets your team reviews together.

A dataset is an annotation tag such as refund-mistakes. Each step (span) of a conversation you label with it is an example. Datasets point at production data; they hold no copies and no verdicts. Use them to collect material for eval cases and judge examples.

1. Create the dataset


introspection tags create annotation refund-mistakes --color "#f97316" \
  --description "Refunds the agent got wrong"

Or Datasets → Manage datasets in the dashboard. Names are slugified. Annotation tags are a separate catalogue from access tags (tags list --kind annotation or --kind access).

2. Find the step

Open the conversation in the dashboard, or read its turns with introspection conversations get <conversation-id> -o json; each item carries its trace_id and span_id (finding conversations).

3. Label, comment, assign

In the dashboard, use Annotate on any step. From the CLI, each command sets labels or reviewers, plus an optional comment.


introspection annotations create --trace-id <trace-id> --span-id <span-id> \
  --label refund-mistakes --comment "Promised a refund outside policy"
 
introspection annotations create --trace-id <trace-id> --span-id <span-id> \
  --reviewer alex@example.com --comment "Needs a judge"

--label and --reviewer replace the step’s whole set. Read the current labels first (annotations list --span-id <span-id>) and repeat them. --clear-labels and --clear-reviewers empty a set.
Pass --event-id <uuidv7> and reuse it on a retry, so a retry never posts a comment twice.

4. Review as a team

Datasets in the sidebar lists each dataset; Review queue holds the steps assigned to you. An item’s review status is backlog (unassigned), assigned, or reviewed. A step is reviewed once a person comments on it, so a reviewer finishes an item by commenting.


introspection annotations list --label refund-mistakes --page-all -o json
introspection annotations list --assigned-to alex@example.com

Still unreviewed (no human comment yet):


introspection annotations list --assigned-to alex@example.com --page-all \
  --query "[?!has_human_comment].{trace:trace_id,span:span_id,conversation:conversation_id}" -o table

5. Pull in user feedback

Thumbs-down are candidates for review, not labels. List recent negative feedback, then open each conversation and label the step:


introspection events list --event-name introspection.feedback --lookback 7d --page-all \
  --query "[?payload.name=='thumbs_down'].{conversation:conversation_id,comments:payload.comments}" -o table

Sending feedback from your app: Patterns and feedback.

6. Export

Open a dataset in the dashboard and export it as JSONL: trace_id, span_id, conversation_id, input, output, labels, latest comment. Next: turn examples into eval cases.

Verify

introspection annotations list --label refund-mistakes returns your step.

Limits

Not supported	Do this instead
Bulk label or bulk assign	Run `annotations create` once per step.
Accept/reject workflow for agent outputs	Use a label per decision (`approved`, `rejected`).
Slack reactions as feedback	Not delivered; see Slack limits.
Annotating as your backend (API key or service account)	Annotations need a person’s own sign-in: the CLI, the dashboard, or a review tool that signs each reviewer in and calls the Node SDK’s `annotations.create` with their token. Otherwise send `feedback()`.
Fine-tuning from a dataset	Export JSONL and train elsewhere.

Or ask your agent: “Label the last 20 thumbs-down refund conversations into refund-mistakes and assign them to Alex.”