Privacy and authorization

MCP Eval is read-only by default.

Capture journals store structured call metadata and shaped arguments, not raw

response bodies.

Human error text is reduced to the constant {message} plus a salted template

identifier; the original message is not stored in the journal.

Server labels, methods, tool names, keys, enum values, numeric and boolean

values, and registrable domains are retained only within their documented bounded grammars. Other hosts and string values are reduced to privacy-safe categories or length buckets.

Server stderr passes through to the client and is not journaled.
Probe records are tagged synthetic and use the same journal boundary. Raw

manifest arguments, response bodies, tool descriptions, sandbox descriptions, and raw errors are not stored in probe records or printed in summaries.

Manifest files can still contain sensitive operational inputs and are outside the share-safe boundary.

Capture records under <MCPEVAL_HOME>/store/ are content-minimized, but annotation notes are deliberate free-form user prose and are not automatically redacted. Before sharing any store records, you must manually review or remove every annotation note and inspect the remaining files. The fingerprint salt lives at <MCPEVAL_HOME>/.salt, outside store/, and must never accompany shared records. Do not share the entire capture root.

Mutation

Mutation requires two independent controls:

the manifest case uses "access": "mutating" and names a declared sandbox;
the operator passes --allow-mutation.

A missing or invalid manifest, undeclared sandbox, or missing flag never authorizes mutation. generate --confirm-read-only attests that an eligible tool is read-only and does not authorize mutation.

HTTP and authorization

HTTP endpoints are loopback-only by default. Remote endpoints require HTTPS

plus --allow-remote-http.

Optional authorization is read from MCPEVAL_HTTP_AUTHORIZATION, validated,

used in memory, and never persisted or printed.

The HTTP proxy may relay an incoming Authorization value in memory, but it

does not originate calls or grant mutation permission.

mcpeval serve refuses requests without a loopback Host, with a

non-loopback Origin, or without Content-Type: application/json. It lists its process-launching tools (run_probe, scaffold, score, verify_finding) only with --allow-spawn.

mcpeval shim-http applies the same Host and Origin checks to every

request and the Content-Type check to every POST before forwarding anything upstream.

What is recorded

For completed JSON-RPC calls and inbound server notifications, the journal keeps:

timestamps, recorder and event UUIDs, an opaque session token, and a

sequence number;

the validated server label and method;
tool names that satisfy the tool-name grammar;
latency, outcome, and shim overhead;
shaped params.arguments when present.

Server labels are 1-128 character ASCII labels, and methods use a bounded slash-separated grammar. A tool name is kept when it is non-empty ASCII alphanumeric plus _, -, ., or :, at most 128 bytes, whether or not tools/list declared it; any other name is recorded as unlisted. The store revalidates these fields before writing. There is no verbose or raw-payload mode.

Every unparseable frame produces a content-free record with only ts, session, seq, server, an unparsed/outbound or unparsed/inbound method, the unparsed outcome, shim_self_us, and kind. Calls go to calls-*.jsonl; annotations go to annotations-*.jsonl beside them.

Argument values are reduced to:

object keys, container sizes, and the shape of the first array item;
string length buckets and a uuid marker;
schema-declared enum values;
numbers and booleans as typed constants;
the public-suffix registrable domain of an HTTP(S) URL;
ip, localhost, or host for literal IP, localhost, or non-registrable

host values.

Each error message becomes the constant {message} plus template_id: the lowercase hex of the first 8 bytes of SHA256(salt || 0x00 || skeleton(message)). The skeleton removes UUIDs, quoted runs, URLs, paths, and digits first. The identifier only tells failures with the same underlying message apart from unrelated ones. Error codes keep scalar numbers, booleans, or identifier-shaped strings: a letter followed by letters, digits, _, ., :, or -, at most 64 bytes. Other string codes, and string layer or kind values, become length buckets.

The salt is created once at <MCPEVAL_HOME>/.salt, with mode 0600 on Unix. template_id is non-invertible only while the salt stays secret: the space of error-message skeletons is small enough to guess. mcpeval doctor --check-redaction prints the salt path as a must-not-share reminder.

Private corpus artifacts

Corpus collection and replay retain original and replay report bytes in operator-selected directories, separately from the capture journal. Target files can contain operational arguments and private paths, and retained reports or extension metadata can contain sensitive content. The corpus, target file, original reports, and replay bundle require private storage and manual review; they are not part of the mcpeval share envelope or automatically redacted.

Use a private parent with appropriate permissions or ACLs. Replay reserves a new evidence directory; on POSIX it creates owner-only directories and files. On Windows, permissions depend on the parent's ACLs. Do not commit raw evidence or place it in public workflow artifacts. Hashes bind bytes but do not redact low-entropy secrets, authenticate authors, or freeze changing files or services.

Offline checking and the corpus evidence action do not launch servers or require their credentials. The action emits only fixed messages, counts, and verdicts and uploads no reports. GitHub can log workflow inputs independently; use secrets for sensitive paths and control runner and storage access. See Corpus collection and replay for the workflow and CI consumption.

Evaluation resource limits

Stdio messages are limited to 4 MiB including the newline. The evaluator queues at most 16 incoming frames and retains at most 128 notifications totaling 4 MiB of encoded data. Exceeding these limits closes the evaluation transport; reports use the existing transport-error status without including message contents. The recording shim also refuses frames larger than 4 MiB.

Each evaluator pipe write is bounded by the configured transport timeout. Response waits use a fixed deadline, so progress messages and unrelated notifications cannot extend the wait. Client shutdown cancels and joins its I/O threads and terminates and reaps its direct child process. Descendant processes are outside this cleanup guarantee.

Standard input synthesis uses a conservative 64 KiB encoded-size budget and a 4096-node budget, with at most 256 items per array built from type constraints, 16 KiB per string built from type constraints, and eight levels of schema traversal. Values supplied by defaults, examples, enums, and constants also consume these budgets before cloning. Schemas exceeding a budget are reported as unsynthesizable rather than called.

Journal record limits

Call, annotation, and readiness-history writers limit each encoded JSONL record to 4 MiB, including UTF-8 bytes, JSON escaping, and the terminating newline. Sanitization and the existing annotation-note projection happen first. An oversized record is rejected before its journal file is opened, preserving existing journal bytes and avoiding an unreadable oversized append. Records are never truncated to meet the size limit.

Readiness-history writers validate any supplied measurement profile using the same rules as history readers before creating directories or opening a journal. Invalid profiles are rejected without echoing their values or changing existing history. Older readiness records without a measurement profile remain supported.

Encoding retains only a bounded record buffer. This does not bound caller-owned inputs or the in-memory sanitization of records constructed through the Rust API. The same per-record limit applies to indexing, trend loading, doctor, and share export.

Checking a capture store

mcpeval doctor (also doctor --check-redaction) checks every JSONL file under <MCPEVAL_HOME>/store/, including nested probe history. It streams physical lines under a shared journal lock, with a 4 MiB limit including the newline. Symlinks, unreadable entries, invalid UTF-8, and oversized lines stop the scan; they cannot produce a successful check. Findings report file paths and physical line numbers without printing matched content.

The scan preserves its text heuristics, including checking an unterminated final line; it does not validate JSON syntax. Annotation note fields remain exempt from automated detection, with non-empty notes counted for manual review. A passing scan is not proof that arbitrary metadata is safe to share.

Journal payload memory is bounded by the line limit. Traversal metadata and reported findings grow with the number of paths and flagged lines. Locks coordinate cooperating journal writers per file, rather than freezing the entire directory tree; use a private, quiescent copy when inspecting a store that could otherwise change during the sweep.

mcpeval share --dir <directory> snapshots every selected JSONL file under a shared journal lock, scans the exact exported bytes, and publishes the envelope only after the scan succeeds.

Nested JSONL files are checked too.
Symlinks and output paths overlapping the capture store are refused.
A flagged sweep exits 1 and publishes nothing. Invalid JSON records also

stop export.

mcpeval share --dir mcpeval-envelope

Attach the envelope directory to an issue; the salt is never in it.

Records are normalized, checked, and streamed into a staging directory beside the requested envelope; full journals are not retained in memory. Both source records and normalized output records are limited to 4 MiB, including their newline. A complete JSON object at EOF is accepted and receives a newline; an incomplete trailing record rejects export. Privacy and parsing failures discard staging and preserve the prior envelope. Source journals are unchanged.

Staging and the published envelope have owner-only directory permissions on Unix. On Windows, access depends on the parent's ACLs; use a private parent. Export requires temporary disk space proportional to the selected envelope. Directory traversal metadata still grows with the number of selected files.

The envelope contains JSONL records, typed annotation metadata, and a SHARE.md manifest.

Annotation prose is omitted by default; after manual review,

--include-annotation-notes explicitly includes it and prints a warning.

--include-probe-history includes readiness-trend history.
The fingerprint salt, databases (including authoritative lifecycle.db), and

manifests are excluded. Keep any file containing the salt separate.

--force replaces the entire prior envelope after successful preparation; a

refused export preserves it.

The redaction scan is a heuristic, not proof that arbitrary metadata is non-sensitive.