Scripted checks for MCP servers
Write the calls you care about in YAML. mcp-probe runs them against your server and exits non-zero on failure.
git clone https://github.com/Kelvis123456/mcp-probe cd mcp-probe && npm ci && npm run build node dist/cli.js examples/everything.probe.yaml
PASS initialize handshake (1807 ms)PASS tools/list answers (95 ms)PASS every tool has a description and an object input schema (0 ms)PASS required tools exist (echo, get-sum, get-structured-content) (0 ms)PASS unknown tool is rejected, not ignored (16 ms)PASS call: echo returns the message (13 ms)PASS call: get-sum adds two numbers (6 ms)PASS call: get-sum rejects a string for a number (42 ms)PASS call: echo without its required argument fails (3 ms)PASS call: get-structured-content rejects an unknown location (24 ms)PASS server still responds after all calls (14 ms)All 11 checks passed
A refactor can change a tool without failing anything
An MCP tool can start returning different text after a refactor, or stop rejecting bad input. Nothing breaks until a client trips on it.
I wrote mcp-probe after shipping a few MCP tools with no cheap way to notice that. You describe the server and the calls you care about once. The tool starts the server, talks to it the way a client would, and prints a pass or fail line per check.
Declare the calls, read the report
Both files below are unedited copies from the repo’s examples/ folder. The probe targets the official reference server, pinned to 2.2.1, and all 11 checks pass.
# Probe for the official reference server (@modelcontextprotocol/server-everything, pinned in examples/package.json).# Run from the repo root: node dist/cli.js examples/everything.probe.yamlserver: command: node args: [examples/node_modules/@modelcontextprotocol/server-everything/dist/index.js, stdio] timeoutMs: 8000 tools: require: [echo, get-sum, get-structured-content] calls: - name: echo returns the message tool: echo args: { message: hello } expect: { equals: "Echo: hello" } - name: get-sum adds two numbers tool: get-sum args: { a: 40, b: 2 } expect: { text: { contains: "42" } } - name: get-sum rejects a string for a number tool: get-sum args: { a: "forty", b: 2 } expect: { error: true } - name: echo without its required argument fails tool: echo args: {} expect: { error: true } - name: get-structured-content rejects an unknown location tool: get-structured-content args: { location: Atlantis } expect: { error: true }
| Result | Check | ms |
|---|---|---|
| pass | initialize handshake | 475 |
| pass | tools/list answers | 22 |
| pass | every tool has a description and an object input schema | 0 |
| pass | required tools exist (echo, get-sum, get-structured-content) | 0 |
| pass | unknown tool is rejected, not ignored | 4 |
| pass | call: echo returns the message | 4 |
| pass | call: get-sum adds two numbers | 3 |
| pass | call: get-sum rejects a string for a number | 2 |
| pass | call: echo without its required argument fails | 1 |
| pass | call: get-structured-content rejects an unknown location | 2 |
| pass | server still responds after all calls | 2 |
That result says the reference server behaves the way the probe expects. It is not a claim that the probe finds bugs in it.
A changed expectation turns one line red
To see a failure, I changed the first golden value in that probe from Echo: hello to Hello: hello and ran it again. This is the real output. The process exited with code 1.
PASS initialize handshake (948 ms)PASS tools/list answers (48 ms)PASS every tool has a description and an object input schema (0 ms)PASS required tools exist (echo, get-sum, get-structured-content) (0 ms)PASS unknown tool is rejected, not ignored (8 ms)FAIL call: echo returns the message (8 ms)text differs from golden (expected "Hello: hello", got "Echo: hello")PASS call: get-sum adds two numbers (2 ms)PASS call: get-sum rejects a string for a number (2 ms)PASS call: echo without its required argument fails (2 ms)PASS call: get-structured-content rejects an unknown location (3 ms)PASS server still responds after all calls (4 ms)1 of 11 checks failed
Exit codes
- 0
- Every check passes.
- 1
- A check fails.
- 2
- The probe file is invalid, or the tool itself crashed.
A call that never answers is reported as a timeout. It does not count as an expected failure, because a hung tool is exactly what you want the probe to catch.
expect: { error: true } accepts both a thrown protocol error and a result with isError: true, since correct servers do either.
What it checks
Always, with no configuration
- initialize
- The handshake completes within the timeout.
- tools/list
- The server answers it. Pagination is followed, up to 100 pages. If it fails, the checks that need the list fail too instead of passing on an empty one.
- tool metadata
- Every tool has a non-empty description and an input schema of type object.
- unknown tool
- Calling a tool that does not exist fails instead of being accepted.
- ping
- The server still answers after all the calls have run.
From your probe file
- require
- The tools you list exist.
- calls
- Each call succeeds or fails, as you declare. Optionally its first text block must contain a string, match a regex, or equal a golden value. A call to a tool that is not in tools/list fails whatever you expect, so a typo cannot pass as an expected error. If the result has no text block, equals, contains and matches fail.
- strict files
- An unknown key in the probe file is an error that names the key. A matches pattern that is not a valid regex fails when the file is loaded.
- environment
- The server only gets the SDK's small default environment plus what you list under server.env. A relative server.cwd is resolved against the probe file's directory.
- CLI flags
- Unknown or incomplete flags exit with code 2 instead of being ignored.
What it does not do
- stdio only. No HTTP or SSE servers yet.
- Tools only. Resources, prompts and sampling are not probed.
- Golden values compare the first text block. Other text blocks and structuredContent are not looked at.
- Not on npm. Clone the repo to use it.
- Behavior, not strict conformance. It does not validate every message against the spec's JSON schema.
Tested, and audited once
The suite has 32 tests in 3 files. It starts a small real MCP server and runs the probe against it, including a server that hangs, one whose tools/list errors, one that pages its tool list, and one that throws protocol errors on every call.
An independent audit found checks that could pass when they should fail: probe files that silently dropped unknown keys, typo’d tool names satisfying an expected error, timeouts counted as expected failures, and the whole shell environment leaking into the server. All four are fixed. The reasoning is in docs/decisions.md. An audit is not a guarantee that no others remain.