Walkthrough: build, ship, improve

You run a store. Customers ask about refunds, and you want an agent to answer them in Slack. This page takes that agent through the whole loop: build, ship, catch a mistake, prove the fix, ship again. Each step links to the page with the details. It uses CLI 0.52.0 or later.

You need: Node.js 24, pnpm, Git, a GitHub repository, a Slack account that can create apps, and the CLI (npm install -g @introspection-ai/cli, then introspection setup).

1. Create the recipe and run it locally

A recipe is your agent as files in Git. Scaffold one, replace SYSTEM.md with the refund policy, and ask it a question on your machine. No account is needed yet.

introspection init refund-agent
cd refund-agent

You answer customer refund requests for our store. Be brief and polite.
 
- Unopened items returned within 30 days get a full refund.
- Opened items returned within 30 days get store credit.
- Ask for the order number if the customer has not given one.

introspection local --print "Order 1842 arrived unopened. The customer wants a refund. Draft the next response."

Run introspection local with no flags to chat with it interactively. To use another model, set ai.model in agents/agent.yaml.

More: Create a recipe · Develop locally

2. Deploy to staging and production

Log in, push the recipe, and create the runtime. Every later merge to the default branch builds a new version. An organization owner or admin must install the GitHub App once first.

introspection login
git remote add origin git@github.com:<owner>/refund-agent.git
git push -u origin HEAD
introspection runtimes create        # add --yes without a terminal
introspection tasks create --runtime refund-agent --environment staging \
  --prompt "Order 1842 arrived unopened. The customer wants a refund. Draft the next response."

Read the answer with introspection conversations get <task-id>.

More: Quickstart · Deploy your agent

3. Put it in Slack

Add the Slack channel package and declare it in package.json ({ "pi": { "channels": [{ "provider": "slack" }] } }), add channels to the agent’s tools, then merge. Create your Slack app and the Slack connector in Settings → Connectors as the Slack page describes. Then install it on your workspace for this runtime.

pnpm add @introspection-ai/recipe-channel-slack
introspection connectors authorize <connector-slug> --runtime refund-agent

Invite the app to your support channel with /invite @your-app and mention it.

More: Put your agent in Slack

4. Find the bad answer and label it

A customer asks to return a final-sale jacket, and the agent promises a refund. Find the conversation, read it to get the step’s trace_id and span_id, and label that step in a dataset your team reviews.

introspection conversations list --lookback 24h --filter environment=production -o table
introspection conversations get <conversation-id> -o json
introspection tags create annotation refund-mistakes --color "#f97316" --description "Refunds the agent got wrong"
introspection annotations create --trace-id <trace-id> --span-id <span-id> \
  --label refund-mistakes --comment "Promised a refund on a final-sale item"

The dashboard’s Annotate button on any step does the same.

More: Find a conversation · Label production steps

5. Turn it into an eval case with a judge

Create an eval suite with a judge of the same name, then add the labeled turn as a case. --expected fail also records that the original answer was wrong, as a judge example.

pnpm add -D vitest@^4.1.11 vitest-evals@^0.17.0 @introspection-ai/vitest-evals@^0.2.0
introspection eval create refund-policy --with-judge
introspection eval add refund-policy --trace-id <trace-id> --case-name final-sale --expected fail

In judges/refund-policy.yaml, replace the scaffolded tool gate with - event: message (a judge that does not apply fails the eval), fill in llm, and write the rubric: pass when the reply follows the policy, fail when it promises a refund the policy forbids. Add eval-runs/ to .gitignore, then commit everything so the worktree is clean: a baseline must come from a committed recipe.

git add -A && git commit -m "Add the refund-policy eval"
git status --short   # prints nothing

More: Add cases to an eval · Calibrate a judge

6. Fix the instructions and prove it

First run the suite on the unchanged recipe and promote that run as the baseline. Then add the missing rule to SYSTEM.md (“Final-sale items are not refundable; offer store credit only if the item is defective”), run again, and compare.

introspection eval run -p evals/vitest
introspection eval baseline promote <run-id>
git add evals/vitest/baseline.json && git commit -m "Baseline refund-policy"
introspection eval run -p evals/vitest
introspection eval compare --baseline --fail-on regressed

final-sale should now be improved and every other case unchanged. If any case regressed, the compare exits 1 and names the reason. Use the same line as a pull request gate.

More: Run and compare evals

7. Ship the fix and watch the judge

Commit SYSTEM.md and open a pull request. Point staging at its preview build, try it there, then merge: staging and production both move to the new version, and the judge starts grading production conversations.

introspection runtimes list --runtime refund-agent
introspection runtimes pin <version-id> --branch pr/<number>
introspection tasks create --runtime refund-agent -e staging --prompt "Refund order 1842, it's a final-sale jacket."
introspection judges list
introspection events list --event-name introspection.judgement --filter judge_id=<judge-id> --lookback 7d \
  --query "[?payload.result=='fail'].{conversation:conversation_id,reason:payload.reasoning}" -o table

Judges in the dashboard sidebar charts the pass rate per runtime. Each new failure is a candidate for step 4, and the loop starts again.

More: Test before production · Calibrate a judge · Usage, cost, and alerts

Limits