Testing

arcane-mcp claims every Arcane capability on three surfaces. Three layers of tests back that claim: unit tests that need nothing running, parity checks over the catalog, and a live suite that drives real Arcane instances.

make check       # generator up to date, 1:1 audit, vet, unit tests
make e2e-up      # start the Docker-in-Docker Arcane estate
make test-e2e    # live suite on every surface, then the coverage report
make e2e-down    # remove the estate, volumes included

All need Go 1.27.3 (GOTOOLCHAIN=go1.27.3 works with an older host Go); the live suite also needs Docker.

Unit tests

make test (or go test ./...) runs the unit suite. Arcane is faked with httptest servers; tests are table-driven and named TestUnit_Scenario_Expected. make cover measures coverage across internal/... and cmd/..., and CI fails below 90%.

Beyond per-package behaviour — argument binding, multipart encoding, response shaping, progress summaries, WebSocket collection, the terminal framing of container.exec — the unit suite checks the catalog as a whole:

• every action of the inventory is in the catalog and validates (ID shape, operationId, title and description, an object input schema, destructive implies mutating), and every related action it names exists;
• every action is registered on the meta and individual surfaces and resolvable by dynamic execute, and each surface registers the expected number of tools;
• the same call returns the same result on all three surfaces, a destructive call without confirm is refused on all three, and safe mode previews a mutation;
• read-only mode leaves no mutating action, exclusions by domain and by tool name remove what they name, and a withheld action reports why under any surface’s name for it.

Parity checks

Command	Fails when
go run ./cmd/gen_actions -check (make check-generated)	The generated declarations differ from what the vendored spec and api/*.yaml produce
go run ./cmd/audit_1to1 (make audit-1to1)	An Arcane OpenAPI operation or extra route has no action and no allowlist entry, an action covers no route, or a tool name exceeds 64 characters
go run ./cmd/gen_tool_reference -check (make check-tool-reference)	The tool reference on this site is out of date
go run ./cmd/audit_doc_names (make audit-doc-names)	An action ID or tool name written in served prose, the README, docs/ or a site page is not in the catalog, or a catalog figure written by hand (N actions, N domains, N operations, N meta-tools, one tool per action (N), in English or Spanish) in those pages or in a registry or marketplace listing (server.json, plugin.json, lhm.plugin.json, the package READMEs) differs from the catalog; pages a generator writes are left to its own check, and a figure about something else is listed with its reason in cmd/audit_doc_names/figures.go

make inventory writes docs/capabilities.md, the full table of Arcane routes, their actions and their tools.

The live end-to-end suite

The suite in test/e2e (build tag e2e) runs against a disposable Docker Compose estate of real Arcane v2.17.2 instances. Nothing in it touches the host’s Docker daemon: every Arcane instance manages a Docker-in-Docker daemon of the estate.

The estate

Service	Role
docker	Primary Docker-in-Docker daemon
docker-swarm	Second daemon, initialised as a single-node Swarm manager
arcane-a	Primary Arcane manager (127.0.0.1:13552); owns every agent environment
arcane-b	Independent second manager (127.0.0.1:13553), own database, users and keys
arcane-agent	Direct agent on docker
arcane-edge	Edge agent on docker, connected through Arcane’s edge tunnel
swarm-agent	Direct agent on docker-swarm

So manager a has four environments — local, agent, edge and swarm — covering each path Arcane has to a Docker daemon: its own client, HTTP to a direct agent, the edge tunnel, and a Swarm manager. Manager b exists for cross-manager scenarios.

make e2e-up (test/e2e/scripts/up.sh) goes from nothing to a provisioned estate in about half a minute with images cached: it starts the daemons and managers, initialises the Swarm, logs in to each manager and creates an API key, registers the three agent environments, starts the agents with their pairing keys, and waits until every environment is online and answers a container list. The result is test/e2e/.estate.json (git-ignored, mode 0600: it holds live keys). Ports bind to loopback only.

The estate README records how provisioning works and every gotcha measured against v2.16.1, v2.17.0, v2.17.1 and v2.17.2 — direct agents staying pending until a connection test, errors proxied from agents not being RFC 9457 problems, why the daemons are reached over TCP, how project bind mounts are made to resolve, and more.

Every scenario on every surface

Scenarios are written once against internal/surfacecall, which drives an in-process arcane-mcp server through any surface with one call shape, and each runs as three subtests: /dynamic, /meta and /individual. They cover containers and images, Compose projects and their lifecycle, Git repositories and GitOps, templates and variables, networks and volumes, vulnerability scans, environments and agents, registries, backups, jobs, webhooks, the event feed and diagnostics, users, roles, API keys, OIDC and passkey settings, and the Swarm cluster, nodes, services, stacks, configs and secrets — on the remote environments as well as the local one.

The ledger and audit_e2e_gaps

Every action a scenario calls is recorded, per surface, in test/e2e/.ledger.json. After the suite, make test-e2e runs go run ./cmd/audit_e2e_gaps, which compares the ledger with the catalog:

e2e coverage: N/461 actions on all three surfaces, P partial, U untestable (listed), M not exercised

An action counts as covered when at least one call succeeded on each of the three surfaces. Actions the estate cannot exercise are listed with a reason in test/e2e/untested.yaml — for instance a self-upgrade that would replace the server under test, or a feature that needs a containerd image store the estate’s daemons do not have — and the audit accepts those. Any other action that is partial or not exercised is reported, and the command exits 1. go run ./cmd/audit_e2e_gaps -list prints every action’s status.

The coverage record

make e2e-coverage-record folds the ledger of a complete, unfiltered run into the committed test/e2e/coverage-record.json: per action and surface, whether it succeeded, which tests exercised it and whether a failure path was checked. make check-e2e-coverage needs no estate: it scans the scenarios for the actions they call, compares them with the catalog, untested.yaml and the record, and checks the generated e2e coverage page. An action used only in a cleanup does not count as covered.

Through the real binary

By default the live suite builds the server inside the test process. E2E_TRANSPORT=stdio runs every scenario through the built arcane-mcp binary over its stdin and stdout instead, and E2E_TRANSPORT=http through arcane-mcp --http, with a Streamable HTTP session per scenario and surface: make test-e2e-live-stdio and make test-e2e-live-http. The scenarios, the ledger and the audit stay the same.

Transport tests

Two modules drive the real binary the way an MCP client does. They need no Arcane, no credentials and no Docker: calls that reach Arcane go to a fake served by the test.

Command	Module	What it checks
make test-e2e-stdio	test/e2e/stdio (tag stdioe2e)	Handshake, server info and protocol versions; the tool list on each surface against the catalog (read-only, exclusions); a call on each surface; the arcane://tools resources, and the data resources and guides read through the fake Arcane (Resources); stdout carries only JSON-RPC while logs go to stderr; configuration from .env, the environment and flags, in that order of precedence; refusal of a bad configuration with a message and a non-zero exit; confirm, read-only and safe mode; destructive calls confirmed through elicitation (decline and accept) with their audit records; session renewal after a 401; a clean exit on stdin EOF (after answering a call still in flight), SIGTERM and SIGINT
make test-e2e-http	test/e2e/http (tag httpe2e)	Session lifecycle (Mcp-Session-Id, calls, DELETE); the optional bearer token (ARCANE_MCP_HTTP_TOKEN) and per-client keys (X-Arcane-API-Key); the localhost Host check; how malformed requests and wrong methods are answered; isolated concurrent sessions; /healthz; shutdown on SIGTERM, idle, with a client connected, and with a call in flight that is allowed to finish

Each target builds the binary once into a temporary directory and passes it on as E2E_SERVER_BINARY.

Condition coverage and mutation testing

Beyond the statement coverage gate, make coverage-conditions PKG=./internal/toolutil (gobco) reports every boolean condition a package’s tests never evaluate both ways, and make coverage-mutants PKG=./internal/toolutil (gremlins) mutates the package and reports the mutants its tests let live. Both run on demand, locally or through the conditions workflow.

The conditions workflow also runs every Monday over a fixed list of packages, conditions only and report-only (no gate, no mutation run); dispatch it with mutants=true for mutation runs.

gremlins’ survivors are re-checked by hand, because gremlins v0.6.0 never runs a mutant in a case expression or a constant or package-level variable initializer (Go’s coverage profile has no block there, so it reads NOT COVERED), and a mutant that lives is worth confirming in a run of its own. make coverage-mutants-hand PKG=./internal/config REPORT=gremlins.txt, with the output of make coverage-mutants saved to gremlins.txt (its JSON report works too), applies each LIVED and NOT COVERED mutant alone in a scratch copy of the module, runs the package’s tests on it (go test -failfast, which runs go vet), and prints KILLED, LIVED, NOT VIABLE (it does not build, or go vet refuses it) or NOT APPLIED for each. A mutant is applied the way gremlins applies it, through the AST and go/printer, which adds the parentheses the new precedence needs: the first && of a && b && c turned into || is (a || b) && c, not a || b && c; MUTANT_WORKERS sets how many run at once.

A change that touches a package (decision D48) keeps its conditions evaluated both ways as they were or better, runs gremlins on the files it changed, and triages every survivor: killed by a new test, or recorded as equivalent with its reason in the record below. Not a literal zero, for the reasons above.

make coverage-conditions PKG=./internal/toolutil
make coverage-mutants PKG=./internal/toolutil CHANGED=auto MUTANT_REPORT=mutants.txt
make coverage-mutants-hand PKG=./internal/toolutil REPORT=mutants.txt

CHANGED=auto takes the package’s files changed against origin/main (else main; CHANGED=<ref> names another), FILES="a.go b.go" names them, and MUTANT_BLOCK_FILES=3 splits them, or the whole package, into blocks. In this block mode gremlins runs one block at a time, each with a build cache of its own that is deleted when the block ends (each mutant builds in a fresh copy of the module, so a whole package once added some 20 GB to the shared cache), a block does not start with less than 6 GB free, and one report covers every block, per file and in total.

The record

Measured on 2026-10-27 with gobco v1.3.4 and gremlins v0.6.0 (INVERT_LOGICAL on, two workers), before and after the tests of the W19 sweep. “After” is branch g7-mutants with main at f3660da merged in. Conditions are those evaluated both ways out of all; mutants are killed / lived / not covered / timed out as gremlins reports them, and every lived and not covered mutant was then applied again by hand (below).

Package	Conditions before	Conditions after	Mutants before	Mutants after (gremlins)	Left after the hand check
internal/toolutil	1709/1828	1931/1952	not run in full	956 / 50 / 139 / 6	19 lived, 2 not covered, all equivalent
internal/wiring	750/788	803/814	not run	353 / 34 / 33 / 18	17 lived, 1 not covered equivalent; 1 lived not killed
internal/config	430/460	458/462	189 / 30 / 69 / 0	218 / 4 / 68 / 0	3 lived equivalent; 1 lived left
internal/completions	160/176	176/176	68 / 9 / 4 / 0	76 / 1 / 4 / 0	1 lived equivalent
internal/prompts	92/94	93/94	51 / 6 / 0 / 0	56 / 1 / 0 / 0	none
internal/resources	117/124	123/124	82 / 2 / 5 / 0	84 / 1 / 4 / 0	1 lived equivalent
cmd/audit_workflow_pins	381/420	530/562	not run	215 / 10 / 29 / 3	6 lived, all equivalent
internal/toolutil (safe_preview.go, W20 H10)	1999/2022	1999/2022	not run	35 / 0 / 2 / 0	none
internal/prompts (workflows.go, W20 H10)	93/94	93/94	not run	0 / 0 / 0 / 0	none
cmd/reproduce_release (new, W20 H8)	—	131/146	—	47 / 5 / 38 / 0, then main.go after the review 19 / 2 / 30 / 0	1 lived, equivalent
cmd/audit_doc_names (W20 H12)	69/90	123/134	not run	70 / 18 / 0 / 0	none
cmd/audit_site_headers (W20 H11, new)	none	205/206	none	121 / 2 / 1 / 0	none

The toolutil and wiring runs before the sweep did not fit the build cache of the machine they ran on (toolutil stopped after 243 of its mutants), so only their condition figures have a “before”. toolutil’s “after” is two runs: the files the merge of main changed (action.go, confirmnote.go, dispatch.go, elicit.go, field_errors.go, overlay.go, proxyclose.go, render_hints.go, render_list.go, render_registry.go, safe_preview.go) were run on the merged code, the others on the code just before the merge, which they share. The completions, prompts and resources runs are from before the merge, which did not change their code; the one lived mutant of prompts is killed by a test added after its run. The core packages W13 measured (internal/arcane, internal/openapi, internal/tools/dynamic, internal/tools/livestream, internal/tools/actioncatalog) stand at 255/266, 373/408, 157/158, 153/156 and 32/32 conditions; their mutation figures are those of W13.

cmd/audit_workflow_pins was measured later, on 2026-11-01 for W20 H6 (D44’s addendum), with the same tools: “before” is main at 9f24130, “after” the branch w20-h6 with the review’s follow-ups. The mutants are those of the files the branch changed (CHANGED=auto: credentials.go, downloads.go, images.go, main.go, shape.go, workflow.go). By hand, 10 of the 29 not covered do not compile or go vet refuses them and the other 19 are killed; the 3 that timed out turn a loop’s i++ into i-- and never end, which counts as detected; 4 of the 10 lived are killed by tests added after the run (alias depth, merge keys past the first key, a workflow without jobs), and the 6 left are equivalent (below). An earlier run on the same branch had left 14 lived in code of W19 G9 that no test pinned (the second secret of a value, two scripts on one line, finding lines past a run block’s first line, the line splitter’s edges, the local forms of a global npm install); tests kill 12 of them, and the other 2 are among the 6.

The two W20 H10 rows changed only text (the safe-mode note and restart hint, and one prompt step), so their conditions are the same before and after by construction; the toolutil figure is the whole package on main at c0a8bf9 plus the change. Mutants are those of the touched file alone (FILES=): safe_preview.go’s 2 not covered are killed by hand, and workflows.go is data with no operator to mutate.

cmd/reproduce_release is new (W20 H8). Its tests cross-compile a fixture for six targets, which a fresh build cache per mutant would make slow and large, so gremlins ran with GOFLAGS=-short (those tests skip) on main.go and recipe.go, and every lived and not covered mutant was then applied by hand with the full tests: 3 do not compile, the rest are killed, 12 of them by tests added after the run (the -out directory, a rebuilt file that is not a Go binary, the line of a malformed checksum, the fat header’s bounds, and the universal layout checked against offsets computed in the test rather than by assembleFat). The review changed how the abbreviated commit is found (abbreviate), so main.go was run again the same way: 2 do not compile, 28 are killed by hand, one more by a test added after (a 16-character stamp), and one lives: i > 0 against >= in the loop that reuses the first target’s binary, which only builds that target a second time, to the same bytes. The 15 conditions never evaluated both ways are errors of git, the file system, a build after the first and YAML encoding once the inputs were read.

The hand check. Each lived and each not covered mutant was applied alone in a scratch copy of the module and the package’s tests run on it (make coverage-mutants-hand):

• Every mutant gremlins reported as living lives by hand too, unless a test added after its run kills it.
• Not covered is not untested. gremlins runs a mutant only where Go’s coverage profile has a block, and the profile has none for the expressions of a case clause or for constant and package-level variable initializers. By hand, toolutil’s 139 not covered mutants were killed but for 2 equivalent ones and a few go vet refuses; wiring’s 33 but for 1 equivalent and 4 that do not compile; config’s 68 but for 2 go vet refuses (x == a && x == b); completions’ and resources’ 4 each.
• The mutants that lived by hand were killed with new tests where they were not equivalent: more than 40 in toolutil, 16 in wiring, a handful in each other package.

W20 H1 (dot-segment refusal, D51), 2026-11-01. Same tools and settings, block mode on the files the change touched. Conditions: internal/toolutil 2017/2040 (main at the time: 1999/2022, the same 23 left; every new condition is evaluated both ways), internal/arcane 280/290 (one more than before, from a new SameOriginRedirect case), internal/config 484/490 (main: 479/486). Mutants: toolutil’s dispatch.go, path_args.go, audit.go and execute.go 124 / 1 / 7 / 1, then dispatch.go again after the review 50 / 0 / 6 / 0, all not covered killed by hand and the one lived removed with the guard it lived in; arcane’s client.go, ws.go and pathguard.go 115 / 2 / 0 / 0, one killed by a new test and Client.URL’s query boundary equivalent (below), pathguard.go after the review 8 / 0 / 0 / 0; config’s config.go 207 / 5 / 68 / 0, by hand 66 not covered killed and 2 go vet refuses, one lived (the trusted-proxies check with no proxies) killed by a new test, the other four the ones this record already gives as equivalent or left.

W20 H12 (catalog figures in audit_doc_names), 2026-11-01. Same tools and settings. Conditions: 69/90 on main at d399e34, 123/134 after (11 left, 10 fewer, all in the name scan that was there before: error branches no repository reaches, a usage note that is never empty, a dotted name followed by -). Mutants of the files the change touched (CHANGED=auto: figures.go and main.go): 70 / 18 / 0 / 0. Every one of figures.go is killed; the 18 that lived are in the name scan of main.go, older than the change (the tool-name pattern marks, isWordByte’s ranges, the empty usage note and the prompt messages), and new tests kill all 18 by hand.

W20 H11 (cmd/audit_site_headers, a new package), 2026-11-02. Same tools and settings, every file of the package (CHANGED=auto). The one condition left is the summary file’s Close error, which a test cannot provoke on a file it just opened for appending. Of the mutants, the not covered one turns the + of a constant URL into a -, which does not compile; one lived boundary (a z after the first character of a Permissions-Policy feature name) is killed by a new test, and the other (a member count turned negative, read only as zero or not) went with the counter, now a flag.

W20 H3 and H2 (HTTP-only settings on stdio, plain HTTP on the network, D52), 2026-11-02. Same tools and settings, block mode on the files the two items touched. Conditions: internal/config 514/520 (main: 484/490, the same 6 left), cmd/arcane-mcp 394/438 (main: 368/414; the new tests cover two of main’s 46, and a few in ready.go and tls.go come and go with timing between runs). Mutants: config’s config.go and httponly.go 231 / 5 / 68 / 0; by hand 66 not covered killed and 2 go vet refuses, one lived (plaintextFlag.String’s nil guard) killed by a new test, the other four the ones this record already gives as equivalent or left. cmd/arcane-mcp’s main.go and plaintext.go 74 / 11 / 11 / 7; by hand every lived and not covered one killed by new tests (the server’s timeouts, the startup line’s attributes, no drain line without a delay, a login serving clients without a key, the loopback Host check without a host list, shutdown waiting for a request and then closing the port, a wildcard bind judged before any lookup), except one in socket_windows.go (errors.Is(err, wsaeConnRefused) || … inverted), which is not measurable here: the file is built only for Windows, so no test on Linux, where these runs happen, compiles it. After the review (one-run error listing with --http, the unresolved-name refusal), on main merged in: conditions 510/516 in internal/config (the same 6 left) and 404/448 in cmd/arcane-mcp; config.go and httponly.go 229 / 4 / 68 / 0, by hand 66 not covered killed and 2 go vet refuses, the 4 lived the ones recorded above; plaintext.go 18 / 0 / 2 / 0, both not covered killed by hand.

What is left, and why.

• Equivalent mutants. Capacity hints (make(…, n±1)) in toolutil (three) and resources; slices cut to the length they already have (MaxListRows, columnSample, the 8192-byte head of a download) and a fill-up asked for zero columns; if a == b { a = b }-shaped boundaries: t < FetchTimeout in completions, ActionTimeout < Timeout in config, a second stage’s total and the total < current clamp in toolutil’s progress, seconds > MaxStreamSeconds; comparisons whose outcome cannot change: sameValue’s two guards, and its (aIsString || bIsString) && EqualFold(…) and (aNum || bNum) && af == bf, which differ only for a blank string or a 0 against a value of another type, never compared since fitsType runs first; networkOf’s err != nil (an invalid key has neither an IPv6 address nor 64 bits), a CutSuffix guard that only adds a duplicate candidate, a [ that cannot sit at the index of ](, a case s[i] < 0x20 the loop before it makes unreachable at 0x20, inSequence’s space, a line of length 0 that accept ignores; a log-only negation (NotifyProgress’s error); a progress interval of 0, which NewProgress never keeps, and the instant equal to the interval; r > unicode.MaxASCII against >=, where DEL is refused either way; the /48 count past its limit, which changes nothing once blocked; ties in the rate limiter’s fullest bucket; the rate limit’s read-ahead of blanks, which changes how much is read, never the count; the body capture going on with a truncated buffer or appending zero bytes, and an error text appending zero bytes; arcane’s len(query) > 0 against >= 0 in Client.URL, an empty query encoding to an empty query string; the stdio writer’s batch set, always set, and its messages, never empty; stdin’s drain with no time to wait, returning either way; a check for filepath.Separator beside the check for /, the same byte on Linux; the idle sweep’s tick, a quarter of the idle timeout, which only changes how soon an idle server is reclaimed.
• cmd/audit_workflow_pins: five loop bounds i+1 < len(n.Content) against <= over YAML mappings (the permissions block in writePermission, the services in jobImages, the jobs in workflowFromNode, mapGet, scalarMap), each behind a check that the node is a mapping, whose content is key/value pairs and never of odd length (the review of H6 found the writePermission one reachable by a sequence permissions:, which panicked under the mutant; such a block is now judged as able to write and reported before the loop); and step.action’s len(parts) < 2 against <=, where a reference of two parts gives owner/repo either way.
• config’s len(entries) == 0 || len(known) == 0 against && in the per-instance lists differs only for a value made of commas alone (ARCANE_API_KEY=,), which no configuration means; it is left rather than pinned.
• wiring’s s.writer != nil before flushed() at stdin’s end is a guard against a write still holding the stdout lock when the last answer is forgotten; a test that kills it would have to stop the scheduler at that instant, so it is left, as a known gap.
• Timed out (6 in toolutil, 18 in wiring) counts as detected: those mutants make a test loop or wait until its deadline, mostly stdin’s end in stdio.go.
• Conditions no input reaches: validHeaderName("") behind a check that the header is set; an instance list where an unnamed entry can only be the single one (len(entries) > 1, name != ""); an action that is destructive but not mutating (the safe-mode marks of prompts and next steps), which the catalog’s validation refuses; a Lookup that finds a name but returns no action; a next-step sentence naming an action that is not served, which is dropped before its safe-mode mark is computed; a text download with no field besides its content, when its type and size are always there; a field-error location that is empty while its argument is not, and a body binding past the case that takes every body location; safe-mode parameters that are always an object.
• Errors that cannot happen: writing a multipart body to a bytes.Buffer; encoding a JSON value just decoded, or a schema just reflected; re-encoding a request id the jsonrpc package has just decoded; netip.Addr.Prefix(64), which fails for no IPv6 address; a schema cache that only ever stores compiled schemas; a body capture whose probe ends neither at EOF nor past its limit.
• Defensive branches production never takes: the stdio writer’s batch and owed-answer sets are always set, and the SDK never writes an empty message; serveSDK and allowRate get a nil body only from a hand-built request; an apikey-mode caller’s key is never empty once the gate has read it; a trusted unix proxy without a header, which configuration refuses.
• elicit.go’s session without initialize parameters is the stateless HTTP transport’s, measured in internal/wiring (http_stateless_elicit_test.go), not in toolutil’s own tests. gobco reports SchemaFor’s panic branch as never taken, although TestSchemaFor_UnsupportedType_Panics takes it: gobco does not record that evaluation, in a generic function that panics right after it.

Arcane v2.17.2 sync (W20, 2026-11-01). internal/toolutil lost proxyclose.go: 1985/2008 conditions both ways against 1999/2022 on main, the same 23 not evaluated both ways (the 14 that left were all covered). gremlins on the files it changed (execute.go, request.go): 57 killed, 0 lived, 1 not covered (case hasText && hasB64 in decodeFileObject, killed by hand), 1 timed out (detected). The usage notes changed in eleven internal/tools/* packages and a comment in internal/openapi leave their conditions as they were. gremlins on those files: openapi 3 killed, tools/webhook 2 killed, tools/build 5 killed and 1 lived by hand, equivalent: err != nil || res == nil turned into && in fileGet differs only when ExecuteREST returns both a result and an error, or neither, which it never does. The other nine packages have no test files of their own (their actions are exercised through the catalog’s tests), and block mode refuses to measure them.

What a run costs. With two workers on four shared cores: about 1 hour 5 minutes for toolutil (1151 mutants) and 1 hour 12 minutes for wiring (438, slowed by 18 stdio mutants that each wait out the 300-second budget); 1 to 3 minutes for each of the other four. Every mutant is built in a temporary copy of the module, so nothing of the build cache is shared between mutants: about 18 MB of cache per mutant, some 20 GB for toolutil and 8 GB for wiring. The hand check takes 5 to 20 seconds per mutant.

Fuzzing

The code that reads untrusted input has native Go fuzz targets, each asserting a property rather than only the absence of a panic: the stdio filter (every line is passed whole or answered, and the SDK never ends the session), the request-id recovery of HTTP refusals, the Markdown renderer on arbitrary JSON (no control byte, ANSI sequence, tag or link survives, tables keep their shape, the structured content is untouched), the OpenAPI schema derivation, the binding of arguments to path, query, headers, cookies and body (they decode back to the arguments), the progress-stream summary and the live-stream frames (SSE events, log lines, terminal sequences, container.exec framing). make fuzz runs each target for 30 seconds (FUZZTIME=10m for longer, FUZZ=<name> for some of them); plain go test replays only their seeds and the inputs committed under testdata/fuzz.

Model evaluation

How well language models use these tools is measured against a corpus of natural-language tasks; see Model evaluation.

In CI

The CI workflow runs the unit tests with the race detector and the coverage gate, checks go mod tidy, compiles the e2e suite (go vet -tags e2e), runs the generator check, the 1:1 audit and the tool-reference check, cross-compiles for Linux, macOS and Windows on amd64 and arm64, runs golangci-lint and govulncheck, and builds this documentation site, and checks the e2e coverage record and the model-evaluation corpus and record offline. The fuzz workflow runs every fuzz target for 30 seconds on each push and for 20 minutes once a week. The weekly e2e workflow brings the Docker-in-Docker estate up on a GitHub runner, runs the live suite and uploads the folded coverage record.

No test recovers a hardware fault (a nil dereference caught by recover): on Windows, Go’s recovery from one can overwrite heap memory next to the goroutine stack on Intel hosts with AMX (golang/go#81238), which once made the Windows run fail with garbled schema text in an unrelated test. Tests that check panic recovery panic in software instead (panic(...), or a transport whose RoundTrip panics).

The documentation site’s headers

This site is a GitHub Pages build served through Cloudflare. Pages lets a site set none of its own response headers, and the build sets none either, so every security header comes from the Cloudflare response header rule for docs.jmrp.io, which only the owner can change. After each deploy, the headers job of the pages workflow runs go run ./cmd/audit_site_headers on the published root and on one deep page (es/reference/tools/), with the workflow’s read token only. A header that is missing or permissive fails the job, with the value to set as an error on the run. The site is already live by then, so a failure is a regression to fix in Cloudflare, not a deploy to roll back. A page that cannot be fetched is retried while the deploy propagates.

Header	Value the rule sets	Refused
Content-Security-Policy	default-src 'self'; script-src 'self' 'unsafe-inline' 'wasm-unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https://img.shields.io https://cursor.com https://kiro.dev https://files.lmstudio.ai; font-src 'self' data:; object-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'; report-uri https://jmrp.io/csp-report	No script directive, scripts from any host or 'unsafe-eval', plugins not 'none', no frame-ancestors or one that admits any host
Strict-Transport-Security	max-age=31536000; includeSubDomains; preload	No readable max-age, or under a year
X-Content-Type-Options	nosniff	Anything else
X-Frame-Options	DENY	A value that refuses no frame (ALLOW-FROM, ALLOWALL alone)
Referrer-Policy	strict-origin-when-cross-origin	unsafe-url, no-referrer-when-downgrade, or no policy a browser knows
Permissions-Policy	camera=(), microphone=(), geolocation=(), payment=(), usb=()	Empty, a member that does not parse, or a feature granted to every origin

All six were served on 2026-11-02 by the root, deep pages and the 404 page, so all six are required. go run ./cmd/audit_site_headers with no flags checks the published site from anywhere.