Fake MCP servers
A fake stands in for one MCP server your recipe declares while an eval runs. It serves the tools captured from the real server, validates every call against their schemas, answers in the format its contract declares, and keeps state that your checks read afterwards. A scenario sets the situation the agent meets: the records each fake starts with and the calls that will fail.
You build fakes from your eval cases. Each case says which servers it uses,
which tools it needs, which records must exist and which failure it meets, and
the fake implements that and nothing more. Your coding agent does this work in
the fakes step of the create, improve and optimize workflows, so the
usual way to start is to ask it. This page shows the parts it builds, so you
can review them.
You need: CLI 0.74.0 or later, @introspection-ai/eval-simulate 0.6.0 or
later, @introspection-ai/vitest-evals 0.8.0 or later under Vitest
(introspection-harbor 0.11.0 or later under Harbor), and each MCP server
bound to your project (Bindings).
1. Capture the real server’s tools
Bind each server to the project yourself, then capture its contract:
introspection bindings mcp capture attio --out fakes/mcpCapture reads the server’s identity and its tool list and never calls a tool.
It writes tools.ts and starts manifest.json. The directory name must be
the server id your recipe declares, even when the vendor’s name differs. The
MCP tool list is the contract: the vendor’s REST API does not show which fields
its MCP server exposes. On your laptop, capture accepts plain http only for
loopback and http://host.docker.internal; a binding with a credential must
be https.
tools.ts is a TypeScript module whose default export is the captured
tools/list, as const, so TypeScript keeps every literal and each tool’s
argument types follow from its schema:
// Written by `introspection bindings mcp capture`. Do not edit.
// biome-ignore-all format: the CLI writes this file and reads it back
// prettier-ignore
export default { "tools": [ … ] } as const;Never edit it by hand. A formatter may rewrite it: the CLI reads the object
back and compares values, not text. A tools.json from an earlier CLI is still
read, with a warning, and deleted when capture writes tools.ts.
2. Learn real answers
If an agent in your project already calls the server, read how it does:
introspection bindings mcp observe attio --runtime support-agent --lookback 30dobserve writes observed.ts, in the same module form: which tools and
arguments the agent used, each tool’s result format (JSON, TOON, JSON after a
prose prefix, or plain text) and the schema of the decoded data, the errors
the server returned, and how paging worked. Every value is masked. An object
key is kept by name when it is a single lowercase-first word, such as
status, and not a uuid, an email, a date, a URL or a number; any other key
the contract lacks is masked, and the schema keeps what its values looked like
under x-masked-values. In a terminal, observe lists the kept keys the
contract lacks; check them for customer data before you commit. It reads only calls made to the live server, so answers
from a fake never feed back into it. Where no traces show a response, the fake
writes it from the tool’s description, and your agent names it in its report.
Nobody creates a record or sends a message on the real server to find out.
An observed.json, or an observed.ts written before CLI 0.59.0, is refused
by observe --check: run observe again for each server after upgrading.
--examples writes sampled raw calls to a folder outside your Git tree. They
are customer data: read them, but never commit them.
3. Set up the fakes package
Directory layout
fakes/
package.json # depends on @introspection-ai/eval-simulate and typebox
package-lock.json # what the Harbor sidecar image installs
Dockerfile # the Harbor sidecar image
mcp/
index.ts # export const fakes = { attio, resend }
attio/
tools.ts # from capture
observed.ts # from observe
answers.ts # each tool's successful answer
contract.ts # contractOf(tools, answers)
entities.ts # the fake's state type
handlers.ts # one callback per allowed tool
manifest.json # server, capture, entities, tool map
index.ts # exports manifest, contract, register, observed
scenarios/
index.ts # SCENARIOS and world(name, clock)
base.ts # the world every scenario starts from
blocked.ts # one module per scenario
tests/
setup.js # records every answer the tests receive
attio.test.js # the fake's contract tests
fakes.test.js # checkRecipeFakes
responses.jsonl # the recording, committedfakes/ is its own workspace package. Add it to pnpm-workspace.yaml
(packages: [fakes]), include fakes/ in your tsconfig.json, and add
fakes/ to .recipeignore, since a fake is never deployed:
{
"name": "fakes",
"private": true,
"type": "module",
"scripts": {
"test": "node --test --test-global-setup=tests/setup.js tests/attio.test.js && node --test tests/fakes.test.js"
},
"dependencies": { "@introspection-ai/eval-simulate": "0.6.0", "typebox": "1.3.34" },
"devDependencies": { "@modelcontextprotocol/client": "2.3.1" }
}Node runs the fakes’ TypeScript with type stripping, so nothing is built, and
the fakes import each other with .ts specifiers.
4. Describe what the server stores
A fake starts from the things the server stores, such as companies and deals,
written as types in entities.ts. The manifest says which tools read and
write each of them:
{
"server": { "name": "attio", "version": "0.1.0" },
"capture": { "at": "2026-10-15T09:00:00Z", "endpoint": "https://mcp.attio.com/mcp", "hash": "sha256:4f1c…" },
"entities": ["companies", "deals"],
"tools": {
"get-records-by-ids": { "kind": "stateful", "reads": ["companies", "deals"], "writes": [] },
"list-records": { "kind": "stateful", "reads": ["companies", "deals"], "writes": [] },
"update-record": { "kind": "stateful", "reads": ["deals"], "writes": ["deals"] }
}
}Capture writes server and capture, and keeps the rest when it runs again.
tools lists exactly the tools your recipe allows. A tool is stateful when
it reads or changes stored records, so a deal updated by one call shows up in
the next read. A tool may be static, always giving the same answer, only for
records no tool writes. The write map must agree with each tool’s
readOnlyHint annotation.
5. Declare each tool’s answer
A captured tool list says what a tool takes, and only rarely what it answers.
answers.ts declares each tool’s successful answer: the format the server
answers in (json, toon, prefixed_json with its prefix, or text) and
the TypeBox schema of the decoded value, taken from observed.ts where there
is one. contract.ts binds the two:
import { Type } from 'typebox';
const Record = Type.Object({ id: Type.String(), name: Type.String() });
export const answers = {
'get-records-by-ids': { format: 'toon', schema: Type.Object({ records: Type.Array(Record) }) },
'list-records': { format: 'toon', schema: Type.Object({ records: Type.Array(Record) }) },
'update-record': { format: 'toon', schema: Type.Object({ record_id: Type.String() }) },
};import { contractOf } from '@introspection-ai/eval-simulate/mcp';
import { answers } from './answers.ts';
import tools from './tools.ts';
export const contract = contractOf(tools, answers);
export type Contract = typeof contract;The type check fails when answers misses a tool, and contractOf throws for
an answer to a tool that was not captured. A tool that states an
outputSchema answers with that schema in its structuredContent and
declares no answer; declaring one fails too. The fake’s index.ts exports
manifest, contract, register and observed, which is
observedOf(observed.ts) where the server was observed.
6. Write handlers
Each handler is an ordinary MCP SDK tool callback, registered in captured
order. toolsOf(contract) gives config(name), the tool’s registration from
its captured schema, and answer(name, value), which encodes the value in the
tool’s declared format, its type checked against the declared schema. Here is
the one that changes a deal:
import { type McpServer, notImplemented, text, toolsOf } from '@introspection-ai/eval-simulate/mcp';
import { contract } from './contract.ts';
import type { State } from './entities.ts';
const { config, answer } = toolsOf(contract);
export function register(server: McpServer, state: State) {
server.registerTool('update-record', config('update-record'), async ({ object, record_id, values }) => {
if (object !== 'deals') return notImplemented(`updating ${object}`);
const deal = state.deals.find((item) => item.id === record_id);
if (!deal) return text(`The record with ID "${record_id}" was not found.`, true);
if (deal.id === state.failUpdate) return text('The CRM rejected the update.', true);
deal.stage = String(values.stage);
return answer('update-record', { record_id: deal.id }); // TOON, as answers.ts declares
});
// ...one registerTool per tool in the manifest, in captured order
}A handler answers in the real server’s form. It builds the result as data and
answer writes it: TOON with the encoder the real server’s TOON matches,
prefixed JSON with its prefix, and structuredContent for a tool that states
an output schema. text(...) is for a tool error. An argument value the
handler does not implement gets notImplemented(...), a tool error, instead of
a plausible answer. The fake enforces what the server enforces and not your
recipe’s rules, so a mail fake accepts a send your recipe should have asked
approval for, and your checks catch the mistake.
7. Write scenarios
A scenario is a module that changes the base world. It sets state, including failure conditions the handlers read, and never defines behavior:
import { labelId } from "@introspection-ai/eval-simulate/mcp";
import type { World } from "./index.ts";
// Updating the Acme deal is rejected by the CRM.
export default function blocked(world: World) {
world.attio.failUpdate = labelId("acme-growth");
}scenarios/index.ts exports SCENARIOS and world(name, clock), which
builds the base world when no name is given and throws on a name it does not
know, so a misspelt scenario never runs as the base world. Records are named by
readable labels, labelId turns a label into a stable id, every date is
computed from the clock, and all data is fictional.
8. Test the fake against its contract
The fake’s own tests call it with the official MCP client from
@introspection-ai/eval-simulate/mcp/testing, and record every answer they receive:
import { recording } from '@introspection-ai/eval-simulate/mcp/testing';
export const RECORD = new URL('./responses.jsonl', import.meta.url).pathname;
export const options = { clock: '2026-10-01T09:00:00.000Z', record: RECORD };
export const { globalSetup, globalTeardown } = recording(RECORD);import assert from 'node:assert/strict';
import { test } from 'node:test';
import { callTool, withFakes } from '@introspection-ai/eval-simulate/mcp/testing';
import { fakes } from '../mcp/index.ts';
import { world } from '../scenarios/index.ts';
import { options } from './setup.js';
test('an update the CRM rejects changes no deal', () =>
withFakes(fakes, world, 'blocked', async ({ attio, states }) => {
const before = structuredClone(states.attio.deals);
const result = await callTool(attio, 'update-record', {
object: 'deals', record_id: states.attio.failUpdate, values: { stage: 'Won' },
});
assert.equal(result.isError, true);
assert.deepEqual(states.attio.deals, before);
}, options));withFakes starts fresh fakes for the scenario and hands your function a
client for each. callTool throws when a successful answer is not the tool’s
contract answer, so every test also checks the answers it reads. Call every
tool observed.ts lists at least once: checkRecipeFakes reads the recording
against the real server’s answers. The world is synthetic and the clock fixed,
so responses.jsonl is committed and changes only when a fake does.
9. Bind the fakes in a Vitest case and check the result
import { fileURLToPath } from 'node:url';
import { loadManifests, writeAttempts } from '@introspection-ai/eval-simulate/mcp/verify';
import { createJudge, describeEval } from 'vitest-evals';
import { fakesHarness, recipeCases, type RecipeEvalInput } from '@introspection-ai/vitest-evals';
import { fakes } from '../../fakes/mcp/index.ts';
import { type World, world } from '../../fakes/scenarios/index.ts';
const manifests = loadManifests(fileURLToPath(new URL('../../fakes/mcp', import.meta.url)));
const NoCrmWrite = createJudge('no-crm-write', ({ toolCalls }) => ({
score: writeAttempts(toolCalls ?? [], { attio: manifests.attio }).length === 0 ? 1 : 0,
}));
const harness = fakesHarness<World, RecipeEvalInput & { scenario?: string }>({
fakes,
world,
scenario: (input) => input.scenario,
});
describeEval('CRM read-only questions', { harness, judges: [NoCrmWrite] }, (it) => {
it.for(recipeCases('crm'))('$name', async ({ input }, { run }) => { await run(input); });
});fakesHarness starts a fresh set of fakes on a free local port for each Pi
session, built from the case’s scenario, or the base world when the case
names none, and binds the session to them alone. Pass clock to fix “today”
for a case that depends on dates. writeAttempts returns the calls to tools
that write, which makes it an attempt check. To check state instead,
changedEntities(states, manifests) compares the fakeStates artifact’s
initial and final state and returns the written entities that changed, named
like attio.deals.
To check what the agent was told, read one server’s calls and type their answers by the fake’s contract:
import { answered } from '@introspection-ai/eval-simulate/mcp/answers';
import { piToolCalls, serverCalls } from '@introspection-ai/vitest-evals';
const calls = serverCalls(piToolCalls(run.artifacts?.piEvents), 'attio');
const deals = answered(fakes.attio, calls).filter((call) => call.name === 'list-records');serverCalls names each call by the server’s own tool name, and answered
decodes each successful answer by the contract. An answer outside the contract
throws, since the fake, not the agent, is wrong.
10. Run them in Harbor
Under Harbor all fakes run in one sidecar on port 7353. The image installs the
committed package-lock.json and runs eval-simulate serve, so nothing is built
or bundled:
FROM docker.io/library/node:22-slim
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
EXPOSE 7353
CMD ["node_modules/.bin/eval-simulate", "serve"]The task binds each server to http://fakes:7353/<server-id>/mcp, picks its
scenario with EVAL_SIMULATE_SCENARIO, and its verifier checks state with
introspection_harbor.fakes. With EVAL_SIMULATE_MANIFEST_DIR set, the sidecar
writes each fake’s manifest.json and answers.json there, so a verifier
reads one server’s calls with server_calls and their answers with
introspection_harbor.answers, as serverCalls and answered do under
Vitest. See Harbor.
Check the fakes
Every check in this table runs offline, so you can add each one to CI:
| Check | Fails when |
|---|---|
node --test fakes/tests/fakes.test.js, which runs checkRecipeFakes from @introspection-ai/eval-simulate/mcp/checks with recording | A fake doesn’t serve exactly your recipe’s allowed tools in captured order, a manifest’s tool map is incomplete or contradicts readOnlyHint, a static tool reads written data, a call outside a tool’s input schema is answered or changes state, two instances share state, a world is inconsistent, or an unknown scenario is accepted. With the recording, it also fails when a tool observed.ts lists has no recorded successful answer, or a recorded answer reads unlike the real server’s: another format (JSON where it answers TOON, say), another TOON layout, prefix or first line, a wrong type, or a key the server always sent left out once the tool was sampled 3 or more times. It warns, without failing, of each allowed tool observed.ts has no observation of |
| Your fake’s contract tests | A schema rejection, an empty or missing result, paging, a read after a write, or an injected failure behaves differently from the captured or observed contract, or a successful answer is outside the contract |
tsc --noEmit | answers.ts misses a tool or answers one that states an outputSchema, or a handler’s arguments or answer no longer match the captured schema |
Two checks compare against the live server and your project’s traces. They
need a teammate’s introspection login, because a project API key sees only
its own calls, so run them on a schedule from a teammate’s machine rather than
in CI:
| Check | Fails when |
|---|---|
bindings mcp capture attio --check | The live server’s tools or descriptions changed since capture |
bindings mcp observe attio --check | The agent’s real calls, over the window observed.ts records, no longer match it |
When the server changed, capture again, run the type check and the fakes’ tests, and rerun your evals.
Limits
| Not supported | Do this instead |
|---|---|
| Generating handlers automatically | Your coding agent writes them from the contract, observed.ts and your cases. |
| Calling the real server from an eval | Bind every server a case needs to its fake. |
bindings mcp types, observe --verify and --pool (removed in CLI 0.59.0) | Types come from tools.ts through contractOf; checkRecipeFakes({ recording }) checks the recorded answers. |
Or ask your agent: “Build MCP fakes for our attio server and use them in the crm eval.”