# Gate pull requests on evals

Fail a pull request when any eval case regresses or stops being measured.

Introspection doesn't block a release on eval results, and merging releases
the recipe, so make this job a required check. This page uses CLI 0.52.0 or later.

**You need:** a suite with a committed baseline
([Keep a baseline](/improve/evals#4-keep-a-baseline)), the files committed in
[Create the suite](/improve/eval-cases#1-create-the-suite), and repository
secrets for your agent's model provider and, if you use YAML judges, the judge's.

## 1. Add the workflow

```yaml filename=".github/workflows/evals.yml"
name: evals
on: pull_request

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm install -g @introspection-ai/cli
      - run: introspection setup --yes          # installs Pi and the Recipes extension
      - run: pnpm install --frozen-lockfile
      - run: introspection eval baseline check
      - run: introspection eval run -p evals/vitest
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}   # your agent's provider
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}         # YAML judges
      - run: introspection eval compare --baseline --fail-on regressed --fail-on incomplete
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-runs
          path: eval-runs/
```

`--fail-on` prints the comparison, then exits 1 with a one-line reason. Use
`--fail-on score-drop` to gate on the mean score instead. Run the job from the
recipe root; set `working-directory` if the recipe lives in a subdirectory.
`pnpm/action-setup` reads the pnpm version from `packageManager` in
`package.json`.

## 2. Set the secrets

| Secret | Used by |
| --- | --- |
| The key of the provider in the agent's `ai.model` (`anthropic/…` needs `ANTHROPIC_API_KEY`) | The recipe under test |
| `OPENAI_API_KEY` or `OPENROUTER_API_KEY` | YAML judges with `llm.provider` `openai` or `openrouter` |
| The variable named in a judge's `llm.local.api_key_env` | A judge pointed at your own endpoint |

The eval gate needs no Introspection login.

## 3. Update the baseline on purpose

When a change is meant to move scores, promote the new run in the same pull
request so reviewers see it. Commit the change first: promote refuses a run
made with uncommitted changes.

```bash
git add -A && git commit -m "Tighten the refund rule"
introspection eval run -p evals/vitest
introspection eval baseline promote <run-id>
git add evals/vitest/baseline.json
```

## Verify

Open a pull request that breaks one case. The `evals` check fails at the
compare step with `1 case(s) regressed`, and the `eval-runs` artifact holds the
run.

Pull requests that also edit cases

Compare refuses a baseline whose cases differ. Check out the base commit with
the new cases, run the suite and promote that run, then return to your branch,
run and promote again. To check MCP fakes for drift in CI, see
[Fake MCP servers](/improve/fakes#check-the-fakes).

## Limits

| Not supported | Do this instead |
| --- | --- |
| Platform-side release gates | Required checks in your Git host. |
| Eval history across releases | Keep the uploaded `eval-runs/` artifacts. |
| A cost budget per run | No native gate. Print a run's cost with `introspection eval run -p evals/vitest --query "[0].result.usage.cost_usd" -o table` and review it. |

Or ask your agent: "Add a GitHub Actions job that fails pull requests when the eval baseline regresses."

Gate pull requests on evals

Fail a pull request when any eval case regresses or stops being measured.

Introspection doesn’t block a release on eval results, and merging releases the recipe, so make this job a required check. This page uses CLI 0.52.0 or later.

You need: a suite with a committed baseline (Keep a baseline), the files committed in Create the suite, and repository secrets for your agent’s model provider and, if you use YAML judges, the judge’s.

1. Add the workflow

.github/workflows/evals.yml


name: evals
on: pull_request
 
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npm install -g @introspection-ai/cli
      - run: introspection setup --yes          # installs Pi and the Recipes extension
      - run: pnpm install --frozen-lockfile
      - run: introspection eval baseline check
      - run: introspection eval run -p evals/vitest
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}   # your agent's provider
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}         # YAML judges
      - run: introspection eval compare --baseline --fail-on regressed --fail-on incomplete
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-runs
          path: eval-runs/

--fail-on prints the comparison, then exits 1 with a one-line reason. Use --fail-on score-drop to gate on the mean score instead. Run the job from the recipe root; set working-directory if the recipe lives in a subdirectory. pnpm/action-setup reads the pnpm version from packageManager in package.json.

2. Set the secrets

Secret	Used by
The key of the provider in the agent’s `ai.model` (`anthropic/…` needs `ANTHROPIC_API_KEY`)	The recipe under test
`OPENAI_API_KEY` or `OPENROUTER_API_KEY`	YAML judges with `llm.provider` `openai` or `openrouter`
The variable named in a judge’s `llm.local.api_key_env`	A judge pointed at your own endpoint

The eval gate needs no Introspection login.

3. Update the baseline on purpose

When a change is meant to move scores, promote the new run in the same pull request so reviewers see it. Commit the change first: promote refuses a run made with uncommitted changes.


git add -A && git commit -m "Tighten the refund rule"
introspection eval run -p evals/vitest
introspection eval baseline promote <run-id>
git add evals/vitest/baseline.json

Verify

Open a pull request that breaks one case. The evals check fails at the compare step with 1 case(s) regressed, and the eval-runs artifact holds the run.

Pull requests that also edit cases

Compare refuses a baseline whose cases differ. Check out the base commit with the new cases, run the suite and promote that run, then return to your branch, run and promote again. To check MCP fakes for drift in CI, see Fake MCP servers.

Limits

Not supported	Do this instead
Platform-side release gates	Required checks in your Git host.
Eval history across releases	Keep the uploaded `eval-runs/` artifacts.
A cost budget per run	No native gate. Print a run’s cost with `introspection eval run -p evals/vitest --query "[0].result.usage.cost_usd" -o table` and review it.

Or ask your agent: “Add a GitHub Actions job that fails pull requests when the eval baseline regresses.”