Model evaluation
Model evaluation asks whether a real model can use arcane-mcp to do a real thing to a real Arcane. It is a benchmark of tool use, not of prose: finding the right action, filling in its arguments, carrying the confirmation a destructive action needs, refusing what no action can do, and finishing without being corrected.
It exists because an MCP server is an interface for models. A tool can be right for a person and hard for a model — a description that does not say which of two actions to use, a parameter name a model only learns from a rejection — and none of that shows up in a unit test or in the live end-to-end suite. The design follows the model evaluation of gitlab-mcp-server, scaled down.
How a run works
Section titled “How a run works”A run puts every case of the corpus to every configured model, on every configured surface (dynamic and meta by default; individual on request):
- A disposable Arcane.
make modeleval-upstarts a Docker-in-Docker daemon and an Arcane v2.17.2 that manages it, both namedme-*and published on127.0.0.1:16000only. It is never the e2e estate: a model under evaluation sends whatever it decides to send, and nothing it does may reach containers another suite depends on. - Fixtures. Before each attempt the harness builds what the case needs (a running container, a volume, a Compose project, a user…) under names carrying a fresh token, and renders the prompt with those names.
- The server. An arcane-mcp server for the surface, built through the same wiring calls as the stdio binary. The model is shown exactly the tools that server lists. The individual surface runs only when asked for (
MODELEVAL_SURFACES=dynamic,meta,individual): its 461 tools do not fit in a request, so an attempt is shown every tool of the case’s domains plus distractors, up to 128, which measures choice within that slice rather than across the catalog. - The conversation. Tool calls go to the server verbatim and its answers go back. Nothing is repaired or retried on the model’s behalf; only a request the provider itself refused (rate limit, server error) is sent again. An attempt ends when the model answers in text, or at a turn cap of three turns per step on dynamic (two elsewhere) plus four.
- Teardown. The fixtures are removed, and so is anything the attempt created — anything carrying its token or that did not exist before it.
Every attempt is written to a shard: the prompt as sent, every provider turn with its token counts, and every tool call with the action it resolved to and what the server did with it. A shard holds observation and no verdict.
The corpus
Section titled “The corpus”Each case is a natural-language request, the fixtures it needs, and a key: the ordered steps a correct agent takes, each naming a canonical catalog action and the arguments whose values are compared. There are five kinds: read (every step reads), write (one non-destructive change), sequence (several steps in order, typically finding an ID the prompt does not give and then acting on it), destructive (a change the server refuses without confirm: true) and decline (something no action does, or a request too vague to act on safely, where the right answer is to say so or ask, and change nothing).
A prompt never names its own answer. An ordinary unit test renders every prompt on every surface and fails if it contains the case’s action ID, any surface’s tool name for it, its domain’s action names, any argument name the action’s schema admits (also as the phrase it reads as, so displayName catches “display name”), or any catalog action ID or tool name at all. A second test plants each kind of leak and proves the audit catches it. Where a word really is the request, the exception is written down with its reason. Every action the corpus names is checked against the catalog.
The corpus has 94 cases (30 read, 18 write, 15 sequence, 21 destructive, 10 decline) declaring 108 steps over 104 of the catalog’s 461 actions in 46 of its 52 domains. The full ledger, case by case with each prompt and key, is test/modeleval/corpus.md.
| Domain | Cases | Actions named |
|---|---|---|
activity | 2 | 2 |
api_key | 2 | 5 |
app_image | 1 | 1 |
auth | 1 | 1 |
backup | 1 | 1 |
build | 1 | 1 |
container | 9 | 9 |
dashboard | 1 | 1 |
diagnostics | 1 | 1 |
environment | 2 | 2 |
event | 1 | 3 |
federated_credential | 1 | 2 |
git_repository | 3 | 3 |
gitops | 4 | 5 |
image | 3 | 3 |
image_update | 1 | 1 |
job | 2 | 2 |
mobile_push | 1 | 1 |
network | 4 | 4 |
notification | 1 | 1 |
oidc | 2 | 2 |
passkey | 1 | 1 |
port | 1 | 1 |
project | 5 | 5 |
registry | 3 | 3 |
role | 4 | 3 |
s3_destination | 1 | 1 |
server | 1 | 2 |
settings | 2 | 1 |
stream | 1 | 1 |
swarm | 1 | 2 |
swarm_config | 1 | 1 |
swarm_node | 1 | 2 |
swarm_secret | 1 | 1 |
swarm_service | 3 | 3 |
swarm_stack | 1 | 1 |
swarm_task | 1 | 1 |
system | 1 | 1 |
template | 2 | 4 |
updater | 1 | 1 |
user | 5 | 5 |
variable | 2 | 3 |
volume | 5 | 5 |
volume_backup | 1 | 1 |
vulnerability | 1 | 1 |
webhook | 2 | 3 |
How an attempt is scored
Section titled “How an attempt is scored”Scoring is computed from the shards, against the corpus at HEAD, by one package shared by the harness and the publisher. The evaluation measures tool use, not Arcane: a step counts as reached when the server accepts the model’s call, whatever Arcane then answers. Precisely, a step is reached by the first call after the previous step that resolves to its action and that the server accepts (the input schema passed and, if destructive, the call carried the approval). Arcane failing an accepted call still reaches the step: the model did its part.
| Column | Numerator | Denominator |
|---|---|---|
| Pass | Attempts that reached every step in order with every compared argument right, changed nothing unrequested and finished on their own | Attempts scored |
| Reached | Steps reached | Steps declared |
| First time | Steps whose first call was the one that reached them | Steps reached |
| Arguments | Compared arguments with the right value | Arguments compared |
| Confirmation | Destructive steps reached with the approval | Destructive steps declared |
| Clean | Attempts with no successful mutation the key did not ask for | Attempts scored |
| Declined | Decline cases that changed nothing and answered in text | Decline cases scored |
| Completion | Attempts the model finished on its own | Attempts scored |
Each column is published as both halves, never as a bare percentage. Arguments are compared by value, not by name: a container may be named by its name, full ID or short ID, an ID by the value the fixture produced or the one an earlier step returned; an argument the prompt does not pin is not compared at all. Attempts that ended because the provider would not answer, the harness could not build the fixtures, or the run’s token budget ran out are excluded from every column and counted apart: none of them is the model’s doing.
What may be published
Section titled “What may be published”A fold (cmd/gen_model_results) reads a run’s shards, scores them, and adds a row per model and surface to the committed record, test/modeleval/results.json, unless a rule refuses it:
- fake-provider — the fake provider answers from the corpus’s own key; its figures prove the pipe works and say nothing about a model.
- filtered-run — a run over part of the corpus would read as a run over all of it.
- incomplete-provenance — every row says which commit, Arcane version, surface, model, request options, corpus, contracts and tool list it measured.
- stale-corpus — a run made against another corpus or other contracts is not scored against a key it was not asked.
- incomplete-run — a run that stopped early would publish the cases it happened to reach.
- dirty-tree — a published figure must point at a commit that exists, so a run made with uncommitted changes to tracked files is refused; commit before a paid run.
- duplicate-row — a row already published is replaced only when asked to.
None of the rules is about how well the model did. A dry run (make model-results-dry-run) sets aside only the fake-provider and dirty-tree rules and writes into dist/, with a banner on every page, so the publishing path can be rehearsed without spending anything; a fake run folded for real publishes nothing and says why, row by row. make check-model-results checks offline that the record and this page agree.
Running it
Section titled “Running it”make modeleval-up # the disposable Arcane (me-*, port 16000)MODELEVAL_MODELS=fake:perfect make modeleval # free: replays the key, must score 100%make modeleval-fake-variants # free: the fake's wrong variants, which must lose pointsMODELEVAL_MODELS=anthropic:<model> MODELEVAL_SPEND=yes make modeleval # paidmake model-results-record # fold dist/modeleval/run into the record and redraw this pagemake modeleval-downA real provider is called only with MODELEVAL_SPEND=yes, and a run stops at MODELEVAL_TOKEN_BUDGET billed tokens (two million by default). The Anthropic Messages API needs ANTHROPIC_API_KEY; any OpenAI-compatible chat completions endpoint works with openai:<model>, OPENAI_API_KEY, and OPENAI_BASE_URL for a local server. make modeleval-probe sends one request per model to check the provider accepts what the harness builds. Nothing here runs in CI.
Results
Section titled “Results”No results are published yet. No real model has been measured: the runs so far used the fake provider, which answers from the corpus’s own key and whose rows the publisher refuses by rule. To publish a measurement, start the evaluation’s Arcane (make modeleval-up), run the corpus against a real model (MODELEVAL_MODELS=anthropic:<model> MODELEVAL_SPEND=yes make modeleval), and fold its shards (make model-results-record). make model-results-dry-run shows how a run would be published without publishing it.