Operations at Scale

Rolling updates

1. Start every instance with --drain-delay at or above one balancer detection interval.
2. Roll one instance at a time. Under consistent hashing the callers pinned to it move to a neighbour, build a pool entry there, and come back when it returns.
3. Compare config_digest across the fleet after the roll, before declaring it done.

A build from a different version with the same settings reports the same config_digest, so an upgrade does not trip the comparison. That is deliberate: the digest answers “do these instances serve the same catalog”, not “are these instances the same build”. build answers the second.

/health and config_digest across a fleet

GET /health needs no credential and performs no GitLab round trip. It answers 200 while serving and 503 once shutdown was requested, with status, version, commit, build, config_digest, started_at and uptime_seconds.

for host in 10.0.0.11 10.0.0.12 10.0.0.13; do
  printf '%s ' "$host"
  curl -fsS "http://$host:8080/health" |
    python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["build"], d["config_digest"], d["status"])'
done

Every instance behind one balancer must report the same config_digest, or one of them serves a different catalog to whichever clients reach it and nothing else notices: the calls all succeed, and only the set of available actions differs. The digest covers tool surface, capability surface, meta parameter schema, tier and whether it was pinned, scope detection, read-only, safe mode and excluded tools. It does not cover TLS, addresses, rate limits or pool sizes, so it will not catch an instance that differs only in those.

/health deliberately does not test GitLab reachability, so a balancer must not read a 200 as “GitLab is up”, and there is no separate readiness endpoint.

Telemetry with many users

Telemetry is off by default and goes to a collector you configure. Three settings matter once the caller population is large:

• --telemetry-identity decides what is recorded about who made a call: none (the default, records nobody), pseudonymous (a keyed digest that correlates one caller’s calls without naming them), or full. Identity never reaches a metric under any policy.
• GITLAB_MCP_TELEMETRY_IDENTITY_KEY is what makes pseudonymous agree across replicas. Left empty, each process generates its own key, so one caller is a different pseudonym on each instance and a distinct-user count is meaningless across a fleet. Set it when replicas must agree, and keep it away from wherever the telemetry lands: GitLab user ids are small enough to enumerate against a known key, which makes the key the “additional information” that turns a pseudonym back into a person.
• --telemetry-tool-name defaults to auto, which keeps gen_ai.tool.name and gitlab_mcp.action as metric dimensions on the dynamic and meta surfaces and drops both on individual, where a thousand tool names would exhaust the SDK’s cardinality limit and collapse the long tail into one overflow bucket. The setting decides the two together, so a dashboard keyed on the action shows one series per gitlab_mcp.domain wherever they are dropped; the domain is kept under every value.

See OpenTelemetry for the full picture.

Fixed egress for GitLab allow-lists

GitLab applies its own rate limits and any IP allow-lists per source address. Instances behind a NAT gateway with a stable address are one caller to GitLab; instances with ephemeral public addresses are several unpredictable ones, and an allow-list cannot be written for them. Give the fleet a fixed egress address before you ask a GitLab administrator to allow-list it.

Secrets

There is no server credential to protect: in HTTP mode every request carries the caller’s own token and /health needs none. What the deployment does hold:

• The affinity salt. A file the balancer reads, mode 0600, not in the repository. It is a distribution function’s salt rather than a security primitive, but leaking it turns the routing key into a confirmable token fingerprint.
• The TLS private key, if the listener terminates TLS. Rotatable in place without a restart.
• GITLAB_MCP_TELEMETRY_IDENTITY_KEY, if pseudonymous telemetry is on. It has no flag on purpose: process arguments are readable through /proc by any local principal. GITLAB_TOKEN has no flag for the same reason.

Pooled tokens are held in memory for as long as their entry lives, because a client cannot call GitLab without one, and the pool is keyed by digest so a scan of its lookup structures yields no credential. Nothing is written to disk. Logs name a token by a keyed digest that changes with every restart, and carry none of its characters.

What to monitor

Signal	Where	Why it matters
config_digest equality across instances	/health	The only detector of an instance serving a different catalog
status and the HTTP code	/health	503 draining is the balancer’s cue, not a failure
build per instance	/health	Which version each instance is actually running
Peak resident set	Host metrics	The term that sizes the process is calls in flight, not credentials
Processor time per call	Telemetry	The ceiling is threads divided by this
Pool evictions	Logs	Frequent eviction means --max-http-clients is below the population
server pool: evicted an entry that was serving a subscription	Logs	A WARN carrying in_use=true and the max_size to raise. It is a signal, not an error: it fires only when every pooled entry was busy, so the pool had nothing quiet to take and a subscriber’s watch was ended
gitlab_mcp.credential_pool.entries against .capacity	Telemetry	How close the pool runs to its bound, without the flag’s value typed into the query
gitlab_mcp.credential_pool.evictions by reason	Telemetry	size_pressure_busy is the one to alert on: it is the pool taking a subscriber’s entry for want of room. The eight reasons are listed under Metrics
subscription refused: this process is at its watcher ceiling	Logs	A WARN naming the 512 watchers already running. Nothing to raise, since the ceiling is not configurable: it says this instance is saturated for subscriptions and callers are being refused
429 rate	Balancer logs	Every HTTP 429 this server answers is the authentication budget. The per-credential rate limiter never answers one: its refusals travel inside the response, as a tool result flagged isError or a JSON-RPC error -42900 (behaviour on exhaustion)
Authentication failures per address	Logs	Ten a minute blocks an address for every credential the deployment does not already hold; behind a proxy without --trusted-proxy-header and --trusted-proxies that address is the balancer’s, whoever sent the failures

The trap that bites hardest at scale

Without --trusted-proxy-header and --trusted-proxies, every caller’s authentication failures are charged to the balancer’s address, so ten bad tokens a minute from anywhere in the population block it for a minute. The callers the deployment is already serving stay served, and every credential it does not already hold is answered 429 until the window ends, which is every new colleague, every rotated token and every caller a balancer has just moved to that instance (the exemption). Configure both flags on any instance behind a proxy.

Frequently asked questions

What does config_digest actually compare?

Whether two instances serve the same catalog. It covers tool surface, capability surface, meta parameter schema, tier and whether it was pinned, scope detection, read-only, safe mode and excluded tools. It does not cover TLS, addresses, rate limits or pool sizes, so it will not catch an instance that differs only in those, and a build from a different version with the same settings reports the same digest on purpose: build is the field that answers which version each instance runs.

Can a balancer read a 200 from /health as GitLab being up?

No. /health performs no GitLab round trip and deliberately does not test reachability, and there is no separate readiness endpoint. It answers 200 while serving and 503 once shutdown was requested, with status, version, commit, build, config_digest, started_at and uptime_seconds, and it needs no credential.

Which monitoring signal bites hardest at scale?

Authentication failures per address. Without --trusted-proxy-header and --trusted-proxies, every caller's failures are charged to the balancer's address, so ten bad tokens a minute from anywhere in the population block it for a minute: every credential the deployment does not already hold is answered 429, whoever sends it, and only the callers it is already serving stay served. Configure both flags on any instance behind a proxy.