# Run Harbor tasks

Evaluate the recipe on tasks with their own environment and a verifier that checks the end state, not just the reply.

A **Harbor task** is a directory with `task.toml`, `instruction.md`, an
`environment/` the agent works in, and `tests/` holding the verifier that
writes a reward. Use Harbor when success is a state you check (files written,
records changed, a repository passing its tests); use
[Vitest evals](/improve/evals) for conversational behavior.

**You need:** Docker for local runs, or a deployed runtime and an owner or admin login for platform runs.

## 1. Install the adapter

```bash
introspection setup --target harbor
```

## 2. Run on Introspection

Trials run in your runtime's sandbox, against a commit you pushed. Commit and
push first; the working tree is never evaluated.

```bash
git push
introspection eval run --runner harbor -p evals/refund-task \
  --runtime support-agent --commit-sha <full-commit-sha> --trials 3
```

`git rev-parse HEAD` prints the full SHA. `-p` is relative to the recipe.
Each trial uses the runtime's image with the recipe replaced by your commit. Platform trials share an evaluation lane, one
at a time by default, apart from production tasks ([Limits](/reference/limits)).

An owner or admin login can create trials; a member login can't. A project API
key alone can't start this run: `--commit-sha` resolves the runtime and
repository through your `introspection login`.

## 3. Or run locally

```bash
INTROSPECTION_HARBOR_ADAPTER=bootstrap introspection eval run --runner harbor -p evals/refund-task
```

`bootstrap` installs the CLI, Pi, and your recipe into whatever image the
task's `environment/` builds.

## 4. Read and compare results

```bash
introspection eval view
introspection eval compare <run-a> <run-b>
introspection eval baseline promote <run-id>       # writes baseline.json in the suite directory
```

Harbor runs compare, promote, and gate in CI exactly like Vitest runs, per
step for multi-step tasks.

## Verify

`introspection eval run ... --agent harbor:oracle` scores 1 on every task: the
verifier accepts a correct solution. Then run your recipe agent.

Run options

| Flag | Effect |
| --- | --- |
| `--agent <name>` | Run a recipe agent variant. `harbor:oracle` runs the task's `solution/`; `harbor:nop` sends nothing. Use them to test the verifier. |
| `--trials N` | N separate runs, each with its own `run.json` |
| `--retry-errors N` | Rerun trials that failed on infrastructure (no model response, expired login, sandbox never ready), up to N times. Agent failures and low scores are never rerun. |
| `--fresh-steps` | Start a new conversation at each step of a multi-step task. By default steps continue one conversation. |

Check MCP fakes in a verifier

A task can serve your [MCP fakes](/improve/fakes) as a sidecar on port 7353:
run `pnpm exec mcp-fakes bundle` in `fakes/` and commit `dist/sidecar.js`, add
a `fakes` service built from `fakes/` with `MCP_FAKES_SCENARIO` set to the
task's scenario, and bind each server to `http://fakes:7353/<server-id>/mcp`.
In the verifier, `introspection_harbor.verifier` reads the trajectory and
`introspection_harbor.fakes` reads each fake's state (`fake_state`,
`fake_initial_state`) and checks it (`load_manifests`, `state_changes`,
`write_attempts`).

## Limits

| Not supported | Do this instead |
| --- | --- |
| Evaluating uncommitted changes on the platform | Push a branch and pass its SHA. |
| `--retry-errors` with `--from-conversation` | Rerun failed trials separately. |

Or ask your agent: "Write a Harbor task for the refund flow and run it on support-agent against my pushed branch."

Run Harbor tasks

Evaluate the recipe on tasks with their own environment and a verifier that checks the end state, not just the reply.

A Harbor task is a directory with task.toml, instruction.md, an environment/ the agent works in, and tests/ holding the verifier that writes a reward. Use Harbor when success is a state you check (files written, records changed, a repository passing its tests); use Vitest evals for conversational behavior.

You need: Docker for local runs, or a deployed runtime and an owner or admin login for platform runs.

1. Install the adapter


introspection setup --target harbor

2. Run on Introspection

Trials run in your runtime’s sandbox, against a commit you pushed. Commit and push first; the working tree is never evaluated.


git push
introspection eval run --runner harbor -p evals/refund-task \
  --runtime support-agent --commit-sha <full-commit-sha> --trials 3

git rev-parse HEAD prints the full SHA. -p is relative to the recipe. Each trial uses the runtime’s image with the recipe replaced by your commit. Platform trials share an evaluation lane, one at a time by default, apart from production tasks (Limits).

An owner or admin login can create trials; a member login can’t. A project API key alone can’t start this run: --commit-sha resolves the runtime and repository through your introspection login.

3. Or run locally


INTROSPECTION_HARBOR_ADAPTER=bootstrap introspection eval run --runner harbor -p evals/refund-task

bootstrap installs the CLI, Pi, and your recipe into whatever image the task’s environment/ builds.

4. Read and compare results


introspection eval view
introspection eval compare <run-a> <run-b>
introspection eval baseline promote <run-id>       # writes baseline.json in the suite directory

Harbor runs compare, promote, and gate in CI exactly like Vitest runs, per step for multi-step tasks.

Verify

introspection eval run ... --agent harbor:oracle scores 1 on every task: the verifier accepts a correct solution. Then run your recipe agent.

Run options

Flag	Effect
`--agent <name>`	Run a recipe agent variant. `harbor:oracle` runs the task’s `solution/`; `harbor:nop` sends nothing. Use them to test the verifier.
`--trials N`	N separate runs, each with its own `run.json`
`--retry-errors N`	Rerun trials that failed on infrastructure (no model response, expired login, sandbox never ready), up to N times. Agent failures and low scores are never rerun.
`--fresh-steps`	Start a new conversation at each step of a multi-step task. By default steps continue one conversation.

Check MCP fakes in a verifier

A task can serve your MCP fakes as a sidecar on port 7353: run pnpm exec mcp-fakes bundle in fakes/ and commit dist/sidecar.js, add a fakes service built from fakes/ with MCP_FAKES_SCENARIO set to the task’s scenario, and bind each server to http://fakes:7353/<server-id>/mcp. In the verifier, introspection_harbor.verifier reads the trajectory and introspection_harbor.fakes reads each fake’s state (fake_state, fake_initial_state) and checks it (load_manifests, state_changes, write_attempts).

Limits

Not supported	Do this instead
Evaluating uncommitted changes on the platform	Push a branch and pass its SHA.
`--retry-errors` with `--from-conversation`	Rerun failed trials separately.

Or ask your agent: “Write a Harbor task for the refund flow and run it on support-agent against my pushed branch.”