# HTTP endpoints Every path the agent serves on the veth, its query parameters, what it returns, how the token guards it, and the line formats of /snapshot and /stream. Source: https://docs.jmrp.io/mikroscope/reference/http/ The agent is an HTTP server and nothing else: it opens no outbound connection. Every path, parameter, status code and line format below is read from [`internal/agent/http.go`](https://github.com/jmrplens/mikroscope/blob/main/internal/agent/http.go), [`internal/agent/capture.go`](https://github.com/jmrplens/mikroscope/blob/main/internal/agent/capture.go), [`internal/agent/source.go`](https://github.com/jmrplens/mikroscope/blob/main/internal/agent/source.go) and [`internal/sample/sample.go`](https://github.com/jmrplens/mikroscope/blob/main/internal/sample/sample.go). ## Listen address The agent binds `ADDR:PORT`. `install` writes the agent's own veth address into `ADDR` and `9123` into `PORT`, so the default is `http://172.30.10.2:9123`. An agent started with `ADDR` unset binds `:PORT`, every address inside its container's network namespace. How a host reaches that address, directly, through the relay or through `--expose`, is on [Network access](https://docs.jmrp.io/mikroscope/install/reaching-the-agent/). The server gives a client 5 s to send its request headers and 30 s to take a response (`ReadHeaderTimeout` and `WriteTimeout` in [`internal/agent/agent.go`](https://github.com/jmrplens/mikroscope/blob/main/internal/agent/agent.go)). ## Authentication With `TOKEN` unset, every path is open. With it set, **every path but `/healthz`** needs: ```http Authorization: Bearer ``` The header's value is compared with the token after an optional `"Bearer "` prefix is removed, so a header holding the bare token is accepted too. A request without it, or with a different token, gets `401`, the body `token required` and `WWW-Authenticate: Bearer`. `/healthz` stays open on purpose: it carries the board string an operator is asked to send when their board has no port map, and a token should not be needed for that. The relay transport cannot present a token. `/tool fetch` runs on the router and sends no `Authorization` header ([`internal/transport/transport.go`](https://github.com/jmrplens/mikroscope/blob/main/internal/transport/transport.go)), so the relay reaches `/healthz` and nothing else on an agent that has one. `--expose` makes the token mandatory; [Expose on the LAN](https://docs.jmrp.io/mikroscope/security/expose/) says why. ## Endpoints | Method | Path | Token | Returns | | -------- | ---------------- | ----- | --------------------------------------------------------------------------------------------- | | `GET` | `/healthz` | no | JSON: liveness, sequence numbers, clocks, rate, slipped ticks, capability hash, board | | `GET` | `/capabilities` | yes | JSON: the kernel, the board, the sources, the device's own ceilings and each source's cadence | | `GET` | `/snapshot` | yes | NDJSON, then closes: the last N seconds, or up to N samples after a sequence number | | `GET` | `/stream` | yes | NDJSON, chunked and open: backfill from a sequence number, then live | | `GET` | `/sampler` | yes | JSON: what only the agent can count about itself — ticks, slips, triggers, captures | | `GET` | `/captures` | yes | JSON: the capture index | | `GET` | `/captures/{id}` | yes | NDJSON: one capture header line, then the sample lines verbatim | | `DELETE` | `/captures/{id}` | yes | `204`, and the capture's bytes return to the budget | | `POST` | `/capture` | yes | JSON: arms a capture now | Any other path is `404`; a known path with another method is refused by Go's router with `405`. ## `GET /healthz` | Field | Type | Meaning | | ------------------- | ------- | ------------------------------------------------------------------------------ | | `ok` | bool | always `true` when the agent answers | | `seq` | integer | sequence number of the newest sample in the ring; 0 when the ring is empty | | `oldest_seq` | integer | oldest sequence number still held; 0 when empty | | `wall_ns` | integer | the router's wall clock at the reply, ns since the epoch | | `mono_ns` | integer | the agent's monotonic clock at the reply, ns | | `uptime_s` | float | seconds since the HTTP server was set up | | `rate_hz` | integer | configured sampler rate | | `slipped` | integer | ticks whose read finished after the next tick was due | | `capabilities_hash` | string | eight hex digits over the kernel, the core count and the enabled sources | | `version` | string | the agent's build identity | | `board` | string | the device tree's model string; omitted when the board has none | | `boot_id` | string | the kernel's boot id, read at start; omitted when the kernel does not publish it | What the CLI reads from it: - `wall_ns`, against its own clock, to measure the skew; - `seq` and `oldest_seq`, to plan a backfill; - `capabilities_hash`, to notice that the kernel or the source set under it changed; - in `doctor`, `board` to classify kernel-log records, and `seq` with `oldest_seq` to read the whole ring, or its newest 10 000 samples when it holds more; - in `forward`, `boot_id`, to tell a router that rebooted from an agent whose container alone restarted: the container shares the router's kernel, so only a reboot gives it a new id ([`reboot`](https://docs.jmrp.io/mikroscope/sinks/detections/#reboot)). No CLI code reads `mono_ns`. The probe after `install` prints this reply as: ```text direct transport ok: agent , Hz, seq , slipped, round trip ``` `` is the CLI's own version, which it stamps into the agent it builds (`dev` when the CLI itself was built without one). ## `GET /capabilities` What the agent established at start about the kernel and the board, with no RouterOS API. [Device info](https://docs.jmrp.io/mikroscope/sinks/device-info/) is this object as the collector hands it to every sink. | Field | Meaning | | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `kernel` | the kernel version string from `/proc/version` | | `board` | the device tree's model; omitted when absent | | `ports` | kernel interface name to RouterOS default name, for a board in the port table; omitted otherwise | | `ports_from` | how that map was established; omitted with it | | `cores` | core count, from the first read of `/proc/stat` | | `user_hz` | `USER_HZ`, the tick counters' unit | | `sources` | source name to `true` when the file opened and the source is enabled: `stat`, `meminfo`, `loadavg`, `softnet`, `softirqs`, `interrupts`, `vmstat`, `psi`, `schedstat`, `self`, `yaffs`, `diskstats`, `slabinfo`, `kmsg`, `thermal`, `cpufreq`, `perf`, `buddyinfo`, `mtd` | | `namespaced` | what the agent deliberately does not read as router data: `net/dev`, `net/snmp`, `net/netstat`, `sys/net/netfilter/nf_conntrack_count` | | `cgroup` | `true` when cgroup2 `cpu.stat` is readable, so the self-cost is exact | | `privileged` | `true` when both `/proc/slabinfo` and `/dev/kmsg` opened | | `limits` | the device's own ceilings, read once at start (below) | | `cadences` | source name to `{"hz", "reason"}` for each level source (below) | | `hash` | the same value as `/healthz`'s `capabilities_hash` | `limits` has no JSON tags in the code, so its keys are the Go field names: `ThermalCriticalMilliC` (zone to the lowest critical trip, m°C), `ThermalPollingMS` (zone to its `polling_delay`), `CPUFreqMinKHz`, `CPUFreqMaxKHz`, `CPUFreqStepsKHz` (core to the ladder), `CPUFreqGovernor`, `CPUFreqRelated` (core to the cores that change frequency with it), `CgroupMemoryMaxBytes` and `ConntrackMax`. A ceiling the device does not publish is empty or 0. A cadence's `reason` is one of `rate` (read at the sampler rate), `declared` (the device publishes its own refresh cadence), `policy` (a `userspace` cpufreq governor means the clock cannot move without a write), `budget` (a measured parse cost), `change` (read every tick, stored on change) or `override` (`FLOOR_HZ`). Counters never appear here, because counters are never floored. [Source cadences](https://docs.jmrp.io/mikroscope/limits/source-floors/) explains each floor. ## `GET /snapshot` Writes NDJSON and closes. Two forms, chosen by whether `since` is present. | Parameter | Default | Accepted | Meaning | | --------- | ------- | ----------------- | -------------------------------------------------------------------------------------------------------- | | `seconds` | `1` | 1–3600 | without `since`: the newest `seconds × rate` samples the ring holds, oldest first | | `since` | absent | a sequence number | up to `max` samples with a sequence number above it, oldest first; `since=0` starts from the oldest held | | `max` | `20` | 1–10000 | with `since`: the most samples one reply carries | A value outside its range is `400` with a one-line reason. The `since` form is what `record` and `forward` pull, over both transports, and what `doctor` reads the ring with (`since` one below `oldest_seq`, or `seq` − 10 000 when the ring holds more than 10 000 samples, `max=10000`, direct transport only, with the bearer token when `TOKEN` is set). It adds two line kinds the `seconds` form does not: a gap line first when `since` is older than the ring, and a trigger line before each sample a capture condition fired on. `max` exists for the relay, because `/tool fetch output=user` returns at most 64 512 B and truncates the rest silently; the relay asks for at most 13 lines. > **A large /snapshot is not a free way to read the cost** > > A 60 s `/snapshot` at 10 Hz makes the agent hand over about 600 lines, around 1.9 MB, and the > `self.cpu_us` inside those samples includes the cost of serving them. Read the agent's cost from > the collector's `/metrics` instead: [Agent cost](https://docs.jmrp.io/mikroscope/cost/) has the procedure. ## `GET /stream` Chunked NDJSON with `Cache-Control: no-store`, held open until the client leaves or the server's 30 s `WriteTimeout` closes it, whichever comes first. - `since` absent or `0`: starts live. The first sample written is the next one produced after the request; the newest sample already in the ring is not written. - `since=N`: backfills every held sample after N first, then follows. A `since` older than the ring writes one gap line first. - The ring is checked every half period and every new sample is written as it arrives, with trigger lines before the samples they fired on. - Every 5 s a comment line `# heartbeat seq=` is written. A consumer skips lines that start with `#`. A `since` that is not a number is `400`. The server's 30 s `WriteTimeout` applies to every response, and Go's HTTP server does not lift it for a handler that keeps writing, so a `/stream` connection ends about 30 s after it opens however much it still has to say: a connection lives 30.01–30.06 s and carries 304–306 lines at 10 Hz. **A consumer of `/stream` has to reconnect**, with the last `seq` it saw as `since` so the gap is backfilled. The CLI does not depend on `/stream`: `record` and `forward` pull `/snapshot?since=`. ## Line types Apart from `/stream`'s heartbeat comment, every line `/snapshot`, `/stream` and `/captures/{id}` write is one JSON object and a newline. Four kinds exist, told apart by their first key: | Line | Where | Meaning | | ---------------------------- | ------------------------------ | ---------------------------------------------------- | | a sample, starting `{"seq":` | all three | one tick, below | | `{"gap":{"from":F,"to":T}}` | `/snapshot?since=`, `/stream` | samples F to T are no longer in the ring | | `{"trigger":{…}}` | `/snapshot?since=`, `/stream` | a capture condition fired on the sample that follows | | `{"capture":{…}}` | first line of `/captures/{id}` | the capture's header | A consumer that does not know a kind skips it, so a line kind it has never seen costs it nothing. ### Sample line Every numeric field is a **delta since the previous sample** unless the table says it is a level. A source the kernel does not have, or that the deployment cannot read, is **omitted**, never written as zero. The busy ticks of a core are `u + n + s + q + sq + st`; idle and iowait are not busy. | Field | Kind | Content | | ------------------ | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `seq` | — | sequence number, from 1 | | `mono_ns` | — | the agent's monotonic clock at the read | | `wall_ns` | — | the router's wall clock at the read, ns since the epoch | | `dt_ns` | — | the real interval since the previous sample; every delta is over this, not over the nominal period | | `cpu` | delta | one object per core: `u` user, `n` nice, `s` system, `i` idle, `w` iowait, `q` irq, `sq` softirq, `st` steal, in `USER_HZ` ticks | | `cpu_total` | delta | the same keys, from `/proc/stat`'s own `cpu` summary line | | `ctxt`, `intr` | delta | context switches and interrupts from `/proc/stat` | | `forks` | delta | the `processes` line of `/proc/stat` | | `procs_blocked` | level | tasks in uninterruptible sleep | | `psi` | delta | `cpu_some`, `mem_some`, `mem_full`, `io_some`, `io_full`, µs; only on a kernel with PSI | | `sched` | delta | per CPU `run_ns`, `wait_ns`; only on a kernel with `/proc/schedstat` | | `softnet` | delta | per CPU `p` processed, `d` dropped, `ts` time squeeze | | `softirq` | delta | softirq kind to an array of per-CPU counts | | `irq` | delta | the `IRQ_TOP_K` busiest interrupt lines this tick (8 by default): `id`, `name`, `cpu` array | | `irq_total` | delta | every interrupt line summed, the top-K's denominator | | `irq_err` | delta | the `Err` row of `/proc/interrupts`; omitted when zero | | `mem` | level | `/proc/meminfo` in kB, keyed by Go field name: `MemTotal`, `MemFree`, `MemAvailable`, `Buffers`, `Cached`, `Dirty`, `Shmem`, `Slab`, `SReclaimable`, `CommittedAS`, `Writeback`, `SUnreclaim`, `AnonPages`, `Mapped`, `KernelStack`, `PageTables`, `CommitLimit`, `Active`, `Inactive` | | `load` | level | `/proc/loadavg`: `Load1`, `Load5`, `Load15`, `Running`, `Total`, `LastPID` | | `vm` | delta | `/proc/vmstat` events: `pgfault`, `pgmajfault`, and when non-zero `pgscan_kswapd`, `pgscan_direct`, `pgsteal_kswapd`, `pgsteal_direct`, `pgalloc`, `pgfree`, `allocstall`, `compact_stall`, `oom_kill`, `pswpin`, `pswpout` | | `vmg` | level | `nr_free_pages`, `nr_dirty`, `nr_writeback`, `nr_slab_reclaimable`, `nr_slab_unreclaimable`, in pages | | `self` | mixed | `cpu_us` (delta, µs), `rss` (level, bytes), `cg_mem` (level, omitted when zero), `cg` (`true` when cgroup2 was read), and `throttled`, `throttled_us`, `oom_kill` (deltas); the last four are omitted when zero or false; `wake_ns` (how late the loop woke after its ticker, ns) and `read_ns` (how long reading every due source took, ns), each omitted when zero | | `thermal` | level | per zone `type`, `mc` (m°C) and `Celsius`; at the zone's declared cadence, every tick when no zone declares one, or at `FLOOR_HZ` | | `freq_khz` | level | per core, kHz; on change or on the 60 s heartbeat | | `thermal_critical` | level | zone to critical trip in m°C, on the rows that carry `thermal` | | `freq_max_khz` | level | core to cpufreq ceiling in kHz, on the rows that carry `freq_khz` | | `cgroup_mem_max` | level | the container's `memory.max` in bytes, on the first tick and once per heartbeat | | `flash` | mixed | per YAFFS device `dev`, `pw`, `pr`, `er`, `gcc`, `gc` (deltas) and `bad`, `free` (levels); a device with no operation and unchanged free chunks is omitted | | `disk` | mixed | per block device `name`, `r`, `rs`, `w`, `ws`, `io_ms` (deltas) and `inflight` (level); an idle device is omitted | | `slab` | level | cache to active objects; [needs `privileged=yes`](https://docs.jmrp.io/mikroscope/limits/privileged/); on change, at a budget floor | | `slab_limit` | level | cache to its published ceiling, today only `nf_conntrack` | | `perf` | delta | per hardware counter `name`, `cpu` array, and `enabled_ns`, `running_ns`; [needs `privileged=yes`](https://docs.jmrp.io/mikroscope/limits/privileged/), and only counters the CPU implements | | `buddy` | level | per zone `node`, `zone`, `free` array indexed by order; on change or heartbeat | | `mtd` | level | per partition `dev`, `name`, `corr`, `fail`, `bad`, `bbt`, `bitflip_threshold`, `ecc_strength`; [needs `privileged=yes`](https://docs.jmrp.io/mikroscope/limits/privileged/); kernel's since-boot counts | | `events` | — | kernel-log records this tick: `prio`, `lvl` (0 emerg … 7 debug), `fac`, `seq`, `us` (µs since boot), `msg`, and, when the text names a port, `iface`, `ros_iface` and `kind` (`link-up`, `link-down`, `stp-`, `own-address` or `other`) | | `events_dropped` | — | kernel-log loss events this tick, not records: one if the tick hit the 64-record cap, one per kernel ring overrun (which can stand for many records); while non-zero, `events` is a lower bound; omitted when zero | | `resets` | — | monotonic counters that went backwards without a 32-bit wrap this tick; omitted when zero | The perf counter names the agent tries to open are `cycles`, `instructions`, `cache-references`, `cache-misses`, `branch-instructions`, `branch-misses` and `bus-cycles`. The slab caches it keeps are `nf_conntrack`, `skbuff_head_cache`, `skbuff_fclone_cache`, `TCP`, `UDP`, `TCPv6`, `UDPv6`, `sock_inode_cache`, `dst_cache`, `ip_dst_cache`, `kmalloc-1k` and `kmalloc-2k`, where the kernel has them. ## `GET /sampler` What only the agent can count about itself, as JSON: ticks taken, ticks slipped, and what the trigger evaluator has fired, suppressed and refused, with what it is holding right now and the budget those captures pin. Every configured condition is present from the first read, at 0 until it fires — a counter that appears with its first event reads as a gap in the series rather than as a quiet router. The collector reads it at start and then on the same one-minute cadence it re-measures the clock skew on, and hands it to every sink. These figures are not per-tick, so they cannot ride in a sample. **The agent serves no `/metrics`.** The Prometheus exposition is the collector's, built from the samples and from this endpoint, and it carries every family the dashboards ask for: [Prometheus metrics](https://docs.jmrp.io/mikroscope/reference/metrics/). The agent is a sampler and a ring; it folds no sample into counters or histograms, which is memory it does not spend. ## Captures All four capture paths answer `404` with `captures disabled (CAPTURE_MB=0)` when the budget is 0. ### `GET /captures` | Field | Meaning | | -------------- | ----------------------------------------------------------------------------- | | `policy` | `first` or `last` | | `budget_bytes` | the pinned-bytes budget, `CAPTURE_MB` in bytes | | `bytes` | bytes the held captures pin | | `pending` | the capture still collecting its post-fire window; omitted when there is none | | `captures` | the held captures' headers, oldest first | | `triggers` | the configured conditions, each `name` and `threshold` | ### `GET /captures/{id}` One `{"capture":{…}}` header line, then the sample lines exactly as `/snapshot` would have written them, so no new parser is needed. The bytes served are counted in `mikroscope_capture_bytes_served_total`: a download runs on the same core as the sampler. An `id` that is not a number is `400`; an unknown one is `404` with `no such capture`. | Field | Meaning | | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | | `id` | capture number, from 1 | | `cause` | the condition's name as configured, or `manual` | | `condition` | the condition as configured, for example `busy>=0.95`; empty for a manual capture | | `field` | what was compared, for example `cpu[2].busy_ratio`; for a manual capture, the `reason` | | `value` | the value that tripped it | | `threshold` | the configured threshold | | `fire_seq` | the sample it fired on | | `fire_mono_ns`, `fire_wall_ns` | the two clocks at the fire | | `first_seq`, `last_seq` | the window held | | `samples`, `bytes` | its size | | `complete` | `false` when it holds fewer than `pre + post + 1` samples: the ring did not reach back far enough, or the agent stopped first | The trigger line in `/snapshot` and `/stream` carries `id`, `cause`, `field` (omitted when empty), `value`, `threshold`, `seq` and `wall_ns`. The agent keeps the last 64 of them. ### `DELETE /captures/{id}` Frees the capture's bytes and answers `204` with no body. Unknown or non-numeric ids answer as for `GET`. ### `POST /capture` Arms a capture at the newest sample, as if a condition had fired: ```sh curl -X POST "http://172.30.10.2:9123/capture?reason=queue-tree-applied" ``` `reason` defaults to `operator`. The reply is `{"id":N,"armed":true}`. If another capture is still collecting its window, or a manual capture fired within the refractory window, the reply is `409` and nothing is armed. The conditions, the budget and what a capture cannot show are on [Triggered capture](https://docs.jmrp.io/mikroscope/record/triggers/). ## See also - [Network access](https://docs.jmrp.io/mikroscope/install/reaching-the-agent/): the three ways a host gets to these paths. - [Prometheus metrics](https://docs.jmrp.io/mikroscope/reference/metrics/): what the collector's `/metrics` carries. - [Triggered capture](https://docs.jmrp.io/mikroscope/record/triggers/): the conditions behind `/captures`. - [Expose on the LAN](https://docs.jmrp.io/mikroscope/security/expose/): when the token becomes mandatory.