Skip to Content
Platform
ImproveFake MCP servers

Fake MCP servers

Run eval cases against fakes of your MCP servers that serve the real tool schemas, answer in the real formats, keep state you can check, and never touch production systems.

A fake stands in for one MCP server your recipe declares while an eval runs. It serves the tools captured from the real server, validates every call against their schemas, answers in the format its contract declares, and keeps state that your checks read afterwards. A scenario sets the situation the agent meets: the records each fake starts with and the calls that will fail.

You build fakes from your eval cases. Each case says which servers it uses, which tools it needs, which records must exist and which failure it meets, and the fake implements that and nothing more. Your coding agent does this work in the fakes step of the create, improve and optimize workflows, so the usual way to start is to ask it. This page shows the parts it builds, so you can review them.

You need: CLI 0.74.0 or later, @introspection-ai/eval-simulate 0.6.0 or later, @introspection-ai/vitest-evals 0.8.0 or later under Vitest (introspection-harbor 0.11.0 or later under Harbor), and each MCP server bound to your project (Bindings).

1. Capture the real server’s tools

Bind each server to the project yourself, then capture its contract:

introspection bindings mcp capture attio --out fakes/mcp

Capture reads the server’s identity and its tool list and never calls a tool. It writes tools.ts and starts manifest.json. The directory name must be the server id your recipe declares, even when the vendor’s name differs. The MCP tool list is the contract: the vendor’s REST API does not show which fields its MCP server exposes. On your laptop, capture accepts plain http only for loopback and http://host.docker.internal; a binding with a credential must be https.

tools.ts is a TypeScript module whose default export is the captured tools/list, as const, so TypeScript keeps every literal and each tool’s argument types follow from its schema:

fakes/mcp/attio/tools.ts
// Written by `introspection bindings mcp capture`. Do not edit. // biome-ignore-all format: the CLI writes this file and reads it back // prettier-ignore export default { "tools": [ … ] } as const;

Never edit it by hand. A formatter may rewrite it: the CLI reads the object back and compares values, not text. A tools.json from an earlier CLI is still read, with a warning, and deleted when capture writes tools.ts.

2. Learn real answers

If an agent in your project already calls the server, read how it does:

introspection bindings mcp observe attio --runtime support-agent --lookback 30d

observe writes observed.ts, in the same module form: which tools and arguments the agent used, each tool’s result format (JSON, TOON, JSON after a prose prefix, or plain text) and the schema of the decoded data, the errors the server returned, and how paging worked. Every value is masked. An object key is kept by name when it is a single lowercase-first word, such as status, and not a uuid, an email, a date, a URL or a number; any other key the contract lacks is masked, and the schema keeps what its values looked like under x-masked-values. In a terminal, observe lists the kept keys the contract lacks; check them for customer data before you commit. It reads only calls made to the live server, so answers from a fake never feed back into it. Where no traces show a response, the fake writes it from the tool’s description, and your agent names it in its report. Nobody creates a record or sends a message on the real server to find out.

An observed.json, or an observed.ts written before CLI 0.59.0, is refused by observe --check: run observe again for each server after upgrading.

--examples writes sampled raw calls to a folder outside your Git tree. They are customer data: read them, but never commit them.

3. Set up the fakes package

Directory layout

fakes/ package.json # depends on @introspection-ai/eval-simulate and typebox package-lock.json # what the Harbor sidecar image installs Dockerfile # the Harbor sidecar image mcp/ index.ts # export const fakes = { attio, resend } attio/ tools.ts # from capture observed.ts # from observe answers.ts # each tool's successful answer contract.ts # contractOf(tools, answers) entities.ts # the fake's state type handlers.ts # one callback per allowed tool manifest.json # server, capture, entities, tool map index.ts # exports manifest, contract, register, observed scenarios/ index.ts # SCENARIOS and world(name, clock) base.ts # the world every scenario starts from blocked.ts # one module per scenario tests/ setup.js # records every answer the tests receive attio.test.js # the fake's contract tests fakes.test.js # checkRecipeFakes responses.jsonl # the recording, committed

fakes/ is its own workspace package. Add it to pnpm-workspace.yaml (packages: [fakes]), include fakes/ in your tsconfig.json, and add fakes/ to .recipeignore, since a fake is never deployed:

fakes/package.json
{ "name": "fakes", "private": true, "type": "module", "scripts": { "test": "node --test --test-global-setup=tests/setup.js tests/attio.test.js && node --test tests/fakes.test.js" }, "dependencies": { "@introspection-ai/eval-simulate": "0.6.0", "typebox": "1.3.34" }, "devDependencies": { "@modelcontextprotocol/client": "2.3.1" } }

Node runs the fakes’ TypeScript with type stripping, so nothing is built, and the fakes import each other with .ts specifiers.

4. Describe what the server stores

A fake starts from the things the server stores, such as companies and deals, written as types in entities.ts. The manifest says which tools read and write each of them:

fakes/mcp/attio/manifest.json
{ "server": { "name": "attio", "version": "0.1.0" }, "capture": { "at": "2026-10-15T09:00:00Z", "endpoint": "https://mcp.attio.com/mcp", "hash": "sha256:4f1c…" }, "entities": ["companies", "deals"], "tools": { "get-records-by-ids": { "kind": "stateful", "reads": ["companies", "deals"], "writes": [] }, "list-records": { "kind": "stateful", "reads": ["companies", "deals"], "writes": [] }, "update-record": { "kind": "stateful", "reads": ["deals"], "writes": ["deals"] } } }

Capture writes server and capture, and keeps the rest when it runs again. tools lists exactly the tools your recipe allows. A tool is stateful when it reads or changes stored records, so a deal updated by one call shows up in the next read. A tool may be static, always giving the same answer, only for records no tool writes. The write map must agree with each tool’s readOnlyHint annotation.

5. Declare each tool’s answer

A captured tool list says what a tool takes, and only rarely what it answers. answers.ts declares each tool’s successful answer: the format the server answers in (json, toon, prefixed_json with its prefix, or text) and the TypeBox schema of the decoded value, taken from observed.ts where there is one. contract.ts binds the two:

fakes/mcp/attio/answers.ts
import { Type } from 'typebox'; const Record = Type.Object({ id: Type.String(), name: Type.String() }); export const answers = { 'get-records-by-ids': { format: 'toon', schema: Type.Object({ records: Type.Array(Record) }) }, 'list-records': { format: 'toon', schema: Type.Object({ records: Type.Array(Record) }) }, 'update-record': { format: 'toon', schema: Type.Object({ record_id: Type.String() }) }, };
fakes/mcp/attio/contract.ts
import { contractOf } from '@introspection-ai/eval-simulate/mcp'; import { answers } from './answers.ts'; import tools from './tools.ts'; export const contract = contractOf(tools, answers); export type Contract = typeof contract;

The type check fails when answers misses a tool, and contractOf throws for an answer to a tool that was not captured. A tool that states an outputSchema answers with that schema in its structuredContent and declares no answer; declaring one fails too. The fake’s index.ts exports manifest, contract, register and observed, which is observedOf(observed.ts) where the server was observed.

6. Write handlers

Each handler is an ordinary MCP SDK tool callback, registered in captured order. toolsOf(contract) gives config(name), the tool’s registration from its captured schema, and answer(name, value), which encodes the value in the tool’s declared format, its type checked against the declared schema. Here is the one that changes a deal:

fakes/mcp/attio/handlers.ts
import { type McpServer, notImplemented, text, toolsOf } from '@introspection-ai/eval-simulate/mcp'; import { contract } from './contract.ts'; import type { State } from './entities.ts'; const { config, answer } = toolsOf(contract); export function register(server: McpServer, state: State) { server.registerTool('update-record', config('update-record'), async ({ object, record_id, values }) => { if (object !== 'deals') return notImplemented(`updating ${object}`); const deal = state.deals.find((item) => item.id === record_id); if (!deal) return text(`The record with ID "${record_id}" was not found.`, true); if (deal.id === state.failUpdate) return text('The CRM rejected the update.', true); deal.stage = String(values.stage); return answer('update-record', { record_id: deal.id }); // TOON, as answers.ts declares }); // ...one registerTool per tool in the manifest, in captured order }

A handler answers in the real server’s form. It builds the result as data and answer writes it: TOON with the encoder the real server’s TOON matches, prefixed JSON with its prefix, and structuredContent for a tool that states an output schema. text(...) is for a tool error. An argument value the handler does not implement gets notImplemented(...), a tool error, instead of a plausible answer. The fake enforces what the server enforces and not your recipe’s rules, so a mail fake accepts a send your recipe should have asked approval for, and your checks catch the mistake.

7. Write scenarios

A scenario is a module that changes the base world. It sets state, including failure conditions the handlers read, and never defines behavior:

fakes/scenarios/blocked.ts
import { labelId } from "@introspection-ai/eval-simulate/mcp"; import type { World } from "./index.ts"; // Updating the Acme deal is rejected by the CRM. export default function blocked(world: World) { world.attio.failUpdate = labelId("acme-growth"); }

scenarios/index.ts exports SCENARIOS and world(name, clock), which builds the base world when no name is given and throws on a name it does not know, so a misspelt scenario never runs as the base world. Records are named by readable labels, labelId turns a label into a stable id, every date is computed from the clock, and all data is fictional.

8. Test the fake against its contract

The fake’s own tests call it with the official MCP client from @introspection-ai/eval-simulate/mcp/testing, and record every answer they receive:

fakes/tests/setup.js
import { recording } from '@introspection-ai/eval-simulate/mcp/testing'; export const RECORD = new URL('./responses.jsonl', import.meta.url).pathname; export const options = { clock: '2026-10-01T09:00:00.000Z', record: RECORD }; export const { globalSetup, globalTeardown } = recording(RECORD);
fakes/tests/attio.test.js
import assert from 'node:assert/strict'; import { test } from 'node:test'; import { callTool, withFakes } from '@introspection-ai/eval-simulate/mcp/testing'; import { fakes } from '../mcp/index.ts'; import { world } from '../scenarios/index.ts'; import { options } from './setup.js'; test('an update the CRM rejects changes no deal', () => withFakes(fakes, world, 'blocked', async ({ attio, states }) => { const before = structuredClone(states.attio.deals); const result = await callTool(attio, 'update-record', { object: 'deals', record_id: states.attio.failUpdate, values: { stage: 'Won' }, }); assert.equal(result.isError, true); assert.deepEqual(states.attio.deals, before); }, options));

withFakes starts fresh fakes for the scenario and hands your function a client for each. callTool throws when a successful answer is not the tool’s contract answer, so every test also checks the answers it reads. Call every tool observed.ts lists at least once: checkRecipeFakes reads the recording against the real server’s answers. The world is synthetic and the clock fixed, so responses.jsonl is committed and changes only when a fake does.

9. Bind the fakes in a Vitest case and check the result

evals/vitest/crm.eval.ts
import { fileURLToPath } from 'node:url'; import { loadManifests, writeAttempts } from '@introspection-ai/eval-simulate/mcp/verify'; import { createJudge, describeEval } from 'vitest-evals'; import { fakesHarness, recipeCases, type RecipeEvalInput } from '@introspection-ai/vitest-evals'; import { fakes } from '../../fakes/mcp/index.ts'; import { type World, world } from '../../fakes/scenarios/index.ts'; const manifests = loadManifests(fileURLToPath(new URL('../../fakes/mcp', import.meta.url))); const NoCrmWrite = createJudge('no-crm-write', ({ toolCalls }) => ({ score: writeAttempts(toolCalls ?? [], { attio: manifests.attio }).length === 0 ? 1 : 0, })); const harness = fakesHarness<World, RecipeEvalInput & { scenario?: string }>({ fakes, world, scenario: (input) => input.scenario, }); describeEval('CRM read-only questions', { harness, judges: [NoCrmWrite] }, (it) => { it.for(recipeCases('crm'))('$name', async ({ input }, { run }) => { await run(input); }); });

fakesHarness starts a fresh set of fakes on a free local port for each Pi session, built from the case’s scenario, or the base world when the case names none, and binds the session to them alone. Pass clock to fix “today” for a case that depends on dates. writeAttempts returns the calls to tools that write, which makes it an attempt check. To check state instead, changedEntities(states, manifests) compares the fakeStates artifact’s initial and final state and returns the written entities that changed, named like attio.deals.

To check what the agent was told, read one server’s calls and type their answers by the fake’s contract:

import { answered } from '@introspection-ai/eval-simulate/mcp/answers'; import { piToolCalls, serverCalls } from '@introspection-ai/vitest-evals'; const calls = serverCalls(piToolCalls(run.artifacts?.piEvents), 'attio'); const deals = answered(fakes.attio, calls).filter((call) => call.name === 'list-records');

serverCalls names each call by the server’s own tool name, and answered decodes each successful answer by the contract. An answer outside the contract throws, since the fake, not the agent, is wrong.

10. Run them in Harbor

Under Harbor all fakes run in one sidecar on port 7353. The image installs the committed package-lock.json and runs eval-simulate serve, so nothing is built or bundled:

fakes/Dockerfile
FROM docker.io/library/node:22-slim WORKDIR /app COPY package.json package-lock.json ./ RUN npm ci --omit=dev COPY . . EXPOSE 7353 CMD ["node_modules/.bin/eval-simulate", "serve"]

The task binds each server to http://fakes:7353/<server-id>/mcp, picks its scenario with EVAL_SIMULATE_SCENARIO, and its verifier checks state with introspection_harbor.fakes. With EVAL_SIMULATE_MANIFEST_DIR set, the sidecar writes each fake’s manifest.json and answers.json there, so a verifier reads one server’s calls with server_calls and their answers with introspection_harbor.answers, as serverCalls and answered do under Vitest. See Harbor.

Check the fakes

Every check in this table runs offline, so you can add each one to CI:

CheckFails when
node --test fakes/tests/fakes.test.js, which runs checkRecipeFakes from @introspection-ai/eval-simulate/mcp/checks with recordingA fake doesn’t serve exactly your recipe’s allowed tools in captured order, a manifest’s tool map is incomplete or contradicts readOnlyHint, a static tool reads written data, a call outside a tool’s input schema is answered or changes state, two instances share state, a world is inconsistent, or an unknown scenario is accepted. With the recording, it also fails when a tool observed.ts lists has no recorded successful answer, or a recorded answer reads unlike the real server’s: another format (JSON where it answers TOON, say), another TOON layout, prefix or first line, a wrong type, or a key the server always sent left out once the tool was sampled 3 or more times. It warns, without failing, of each allowed tool observed.ts has no observation of
Your fake’s contract testsA schema rejection, an empty or missing result, paging, a read after a write, or an injected failure behaves differently from the captured or observed contract, or a successful answer is outside the contract
tsc --noEmitanswers.ts misses a tool or answers one that states an outputSchema, or a handler’s arguments or answer no longer match the captured schema

Two checks compare against the live server and your project’s traces. They need a teammate’s introspection login, because a project API key sees only its own calls, so run them on a schedule from a teammate’s machine rather than in CI:

CheckFails when
bindings mcp capture attio --checkThe live server’s tools or descriptions changed since capture
bindings mcp observe attio --checkThe agent’s real calls, over the window observed.ts records, no longer match it

When the server changed, capture again, run the type check and the fakes’ tests, and rerun your evals.

Limits

Not supportedDo this instead
Generating handlers automaticallyYour coding agent writes them from the contract, observed.ts and your cases.
Calling the real server from an evalBind every server a case needs to its fake.
bindings mcp types, observe --verify and --pool (removed in CLI 0.59.0)Types come from tools.ts through contractOf; checkRecipeFakes({ recording }) checks the recorded answers.

Or ask your agent: “Build MCP fakes for our attio server and use them in the crm eval.”

Last updated on