# Record, mark and plot Pull a window of full-rate samples to your machine, mark moments in it, add the router's own log and draw it as a deterministic SVG. Source: https://docs.jmrp.io/mikroscope/record/ Use `record` when you are about to change something on the router and want to see what the change does. It pulls the agent's samples to your machine at full rate; `mark` adds notes and the router's own log; `plot` draws the result. ## Commands and flags ```sh mikroscope record --for 5m --out cap # cap.jsonl, cap.csv, cap.markers.csv, cap.meta.json; type lines to mark mikroscope mark --out cap "queue tree applied" # a note stamped with the current time (see below) mikroscope mark --out cap --log-markers # the router's own log lines, over the API mikroscope plot --in cap # cap.svg, deterministic ``` The three commands share one flag set: each accepts every flag below, and the table says which command acts on it. | Flag | Default | What it does | | --------------- | ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--out` | `capture-` | Output prefix: `.jsonl`, `.csv`, `.markers.csv`, `.meta.json`. `mark` needs the prefix of an existing recording. | | `--for` | `0` | `record`: stop after this long. `0` records until Ctrl-C. | | `--from-start` | off | `record`: backfill everything the agent's ring holds before going live. | | `--poll` | `500ms` | How often to pull the agent's ring. | | `--batch` | `0` | Samples per pull. `0` is twice what one `--poll` interval produces at the agent's rate, at least 20; the relay caps a pull at 13. | | `--transport` | `auto` | `auto`, `direct` (HTTP to the veth) or `relay` (`/tool fetch` over the RouterOS API). | | `--log-markers` | off | `record`: adds the router log lines of the window once the recording stops. `mark`: adds the log lines of the recording's window. | | `--topics` | `system,interface,container` | Log topics kept as markers; add `firewall` or `script` when their lines are the story. | | `--router-tz` | `Local` | IANA zone the router's clock shows. RouterOS log times carry no zone. | | `--api` | `MIKROSCOPE_API_ADDR` | RouterOS API `host:port`, for the relay and for `--log-markers`. | | `--api-user` | `MIKROSCOPE_API_USER` | The API user. The password is read only from `MIKROSCOPE_API_PASSWORD`; there is no flag for it. | | `--token` | `MIKROSCOPE_TOKEN` | The agent's bearer token. | | `--port` | `9123` | The agent's HTTP port. | | `--subnet` | `MIKROSCOPE_SUBNET`, else `172.30.10.0/30` | The agent's /30; its address is `.2`. | | `--in` | none | `plot`: a recording prefix, or the path of a `.jsonl` file. | | `--svg` | `.svg` | `plot`: the output file. | | `--title` | the prefix | `plot`: the chart's title. | [First recording](https://docs.jmrp.io/mikroscope/start/walkthrough/) runs the three commands end to end. A measured recording, with its sample count, gaps and clock skew, is on the [evidence page](https://docs.jmrp.io/mikroscope/about/status/#campaign-record-2026-09-12). Recordings above 10 Hz and through the relay are [not measured](https://docs.jmrp.io/mikroscope/about/status/#not-tested). ## Transports | `--transport` | Path | On failure | | ---------------- | -------------------------------------------------------------------------------------------------------------- | ------------------ | | `auto` (default) | `direct` if `GET /healthz` on `http://<.2 of --subnet>:<--port>` answers, else `relay` | an error, below | | `direct` | plain HTTP from your machine to the veth, with `--token` as a bearer token | fails, no fallback | | `relay` | `/tool fetch output=user` on the router, over the binary API; the API user needs the `read,api,test` policies | fails, no fallback | - When `auto` has to fall back and `--api` or `--api-user` is missing, `record` stops with an error that names `--api`, `--api-user` and `MIKROSCOPE_API_PASSWORD` and suggests `install --expose`. A missing password shows as a login error instead, `api : …`. - The `--expose` hint is for other HTTP clients. `record` always dials the `.2` of `--subnet` and has no flag for the router's LAN address. - The router-side fetch sends no header, so an agent installed with a token can be recorded only over `direct`. - Each relayed call takes either about 3 ms or about 1 s. A reply is capped at 64 512 B, so a relayed pull asks for at most 13 samples, and a reply that reaches the cap is refused rather than parsed truncated. > **The relay tops out at about 26 samples a second** > > By arithmetic, from the code. The cap of 13 samples comes from > the 64 512 B fetch limit, a line charged at > the allocator size class of 3 456 B (the measured line > is 3 230 B) and 134 % headroom, so a full pull of charged-size lines is > about 45 kB and still fits with lines up to 134 % of that size. A full pull whose lines average > more than the fetch limit divided by the cap reaches the limit and is refused; line sizes at the > default per-source floors are [not measured](https://docs.jmrp.io/mikroscope/about/status/#not-tested). > A full pull is asked again at once, so the round trip of `/tool fetch`, not `--poll`, sets the > throughput: with about 1 s for half the calls and about 3 ms for the rest, that is on the order of > 26 samples a second, less when the slow calls cluster. Above that, use `direct`. [Network access](https://docs.jmrp.io/mikroscope/install/reaching-the-agent/) covers which path a network allows, and [API user](https://docs.jmrp.io/mikroscope/security/api-user/) the RouterOS user the relay needs. ### Pull loop 1. `record` reads `/healthz`. The agent's wall clock minus your machine's is the **clock skew**. It prints the skew on stderr with the agent's version, rate, newest sequence number, the oldest one the ring still holds, and the transport. 2. The recording starts live from the newest sample, or from the oldest the ring holds with `--from-start`. 3. Every `--poll`, it pulls `/snapshot?since=&max=`. While a pull comes back full it asks again, up to 100 times per poll, so a ring that got ahead is drained rather than followed at a fixed pace. 4. A pull that fails is logged as `pull: …` and the recording carries on; the next pull asks from the same sequence number. 5. When `--for` elapses or you press Ctrl-C, one last pull drains what arrived meanwhile. `record` prints how many samples it kept, their sequence range, the gaps, the markers, the transport and the files. ## Output files Every file is created with mode `0600`. An existing file with the same prefix is overwritten. | File | Contents | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `.jsonl` | Each sample line exactly as the agent sent it: raw tick and counter deltas, with every source the agent has. This is the recording; the other files are views of it. | | `.csv` | One wide row per sample, for a spreadsheet. The column set is sized from the first sample: one group per core and one per softnet queue. | | `.markers.csv` | `wall_ns,wall_utc,seq,kind,label`, one row per marker. | | `.meta.json` | Written at the start, so `mark` and `plot` can run later: `started_utc`, `skew_ns` (agent wall minus host wall), `agent` (its version), `rate_hz`, `transport`, and `capabilities` when the transport could fetch them. | The CSV holds a fixed subset of each sample. Everything else a sample carries — kernel-log events, PMU counters, temperatures, interrupts per line and the rest — is in the `.jsonl` only. | Columns | Content | | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | | `seq`, `wall_ns`, `wall_utc`, `dt_ns` | The sample's sequence number, agent wall clock, and real interval. | | `busy_total` | The mean of the per-core busy ratios, three decimals, computed by the CLI from the ticks. | | `c_busy`, `c_user`, `c_nice`, `c_system`, `c_idle`, `c_iowait`, `c_irq`, `c_softirq` | Per core: the busy ratio, then the raw tick deltas. | | `ctxt`, `intr`, `irq_total` | Context switches and interrupts from `/proc/stat`, and the delta summed over every row of `/proc/interrupts`. | | `softnet_processed`, `softnet_dropped`, `softnet_time_squeeze` | Per softnet queue, the receive path's deltas. | | `mem_free_kb`, `mem_available_kb`, `mem_cached_kb`, `mem_slab_kb` | Memory levels, in kB. | | `load1`, `threads_running`, `threads_total` | Load average and thread counts. | | `pgfault`, `pgmajfault` | Page-fault deltas. | | `self_cpu_us`, `self_rss_bytes` | The agent's own CPU time during the sample, in µs, and its resident memory at that moment, in bytes. | The busy ratio is the one number the CLI computes, and it inherits the kernel's floor: a tick is 10 ms, so one core in a 100 ms sample resolves to 10 % steps. [Resolution limits](https://docs.jmrp.io/mikroscope/limits/) explains why, and why the ratio is capped at 1. ## Markers A marker is a row of `.markers.csv`, of one of three kinds: | `kind` | Written when | | ------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `note` | You type a non-empty line while `record` runs in a terminal (it prints `type a line and press Enter to add a marker; Ctrl-C stops`), or run `mark --out "text"`. Nothing is read when standard input is not a terminal. | | `gap` | The agent reports that samples were no longer in its ring. The label is `samples .. lost`, and the summary `record` prints lists the same gaps. | | `log` | `--log-markers` adds a line of the router's own log (below). | Notes and gaps are stamped with your machine's clock plus the skew measured at the start, so they sit on the agent's time axis. `mark --out cap "text"` does the same with the skew stored in `cap.meta.json`: it stamps the current time in the agent's clock, and no flag sets another time. `mark` needs that file and an existing `cap.markers.csv`, and its row carries `seq` 0. > **Mark while the recording runs** > > `record` does not hold `.markers.csv` open: it opens the file, appends and closes it for > each marker, as `mark --out "text"` does from a second shell. Markers typed in one > terminal and added from another land in the file in the order they were made, on Linux, macOS > and Windows alike. A text `mark` on a recording that has already stopped stamps the current > time, which falls after the last sample, so `plot` draws nothing there. ## Router log markers `--log-markers` turns the router's log lines of the recording's window into `log` markers, labelled `: `. Use it when a transient you did not cause needs explaining: scheduler runs, errors, interface events. ```sh export MIKROSCOPE_API_ADDR=192.168.88.1:8728 MIKROSCOPE_API_USER=mikroscope MIKROSCOPE_API_PASSWORD=… mikroscope record --for 60s --out burst mikroscope mark --out burst --log-markers --router-tz Europe/Madrid ``` - `mark --log-markers` uses the window from `started_utc` in the meta file, shifted by its `skew_ns` into the agent's clock, to the last sample in the `.jsonl`, and prints how many lines it added, the window and the topics. `started_utc` is stored to the second. It appends without checking what is already there: running it twice adds the same lines again. - `record --log-markers` asks for the log once the recording has stopped, for the window from its start to its end in the agent's clock, and appends the lines through the same path `mark` uses, so the summary's marker total counts what the file holds. If the log request fails, `record` prints `log markers: …` and keeps the recording. The CLI asks the router only for the window, with a `?>time=` query starting one second before it, and only for the `time`, `topics` and `message` fields, because a router that logs a busy topic to disk can hold tens of thousands of rows. It then keeps the lines inside the window whose topics include one of `--topics`. An event logged by several logging actions on one topic arrives once per action, prefixed with the action's name (`[INFO]: …`, `[SYSTEM]: …`); the prefix is stripped and the duplicates are kept once. Log times carry no zone: - Over the API, RouterOS prints a full date and time ([verified](https://docs.jmrp.io/mikroscope/about/status/#verified-log-time-full-date)). The parser also accepts the shorter forms without a year or without a date, and fills them in from the end of the window. - Times are read in `--router-tz` and stamped as the agent's wall clock, which is the router's own clock, so no skew is applied. The default `Local` is your machine's zone: if the router's differs, pass it, or every log marker is off by the difference. - Log times have one-second resolution, so a log marker places its event to the second, not to the sample. ## Chart `plot --in cap` reads `cap.jsonl`, and `cap.markers.csv` when it exists, and writes `cap.svg` (or `--svg`). It prints the file name with its sample and marker counts. The same input always produces the same bytes. The chart is 1 200 units wide, with a title (`--title`, or the prefix) and a line giving the sample count, the duration and the core count. Three panels share one time axis, in seconds since the recording started: 1. **CPU busy per core, %** — 0 to 100, one line per core, each labelled at its end. The palette has eight colours, so the first eight cores are drawn. 2. **softnet per second, all CPUs** — `dropped` and `time_squeeze`, summed over every queue and bucketed into whole seconds. Drops are drawn in red. 3. **memory available, MiB** — `MemAvailable` per sample, with the y-axis bounded to the data. Markers on the chart: - Every marker is a dashed vertical line across all three panels, grey for notes and log lines and red for gaps, with a label chip above the first panel. - Chips are laid out in up to six rows. A label longer than 40 characters is shortened; when no row has room it is shortened further until one does, and a chip that would fall under 12 characters is dropped. Its dashed line still marks the instant. - Consecutive log markers, in time order, that fall in the same second are folded into one chip, `× : `. A note or gap between them breaks the fold, and notes and gaps are never folded. - Markers outside the recording's time span are not drawn. With a single sample the SVG says `not enough samples to draw`; with none, `plot` stops with `record: no samples` and writes no SVG. The palette is the chart's own, validated for its light background: adjacent-pair colour-vision-deficiency ΔE 9.1, normal-vision ΔE 22.9. There is no dark variant. `plot` also reads a capture the agent kept on a trigger: save `GET /captures/` to a `.jsonl` file and pass it to `--in`. The capture's header line carries no sequence number and is skipped. ## Trigger lines When a trigger fires on the agent, a `{"trigger":{…}}` line rides in the same pull as the samples, just before the sample it fired on. The collector recognises it; `record` does not. > **A trigger line in a recording** > > `record` reads a trigger line as if it were a sample with sequence number 0. The line is written > verbatim to the `.jsonl`, where `plot` skips it, and as a row of zeros with `seq` 0 to the `.csv`; > it is counted once in the number of samples `record` reports. If it is the first line of the > recording, the CSV header is sized for zero cores. `record` does not add it as a marker. To see > what fired during a window, read `/captures` on the agent. [Triggered capture](https://docs.jmrp.io/mikroscope/record/triggers/) covers what fires, and how to fetch the full-rate window the agent kept around it. ## See also - [First recording](https://docs.jmrp.io/mikroscope/start/walkthrough/): record, mark and plot, step by step. - [Triggered capture](https://docs.jmrp.io/mikroscope/record/triggers/): the samples around a condition, kept at full rate by the agent without a recording running. - [Network access](https://docs.jmrp.io/mikroscope/install/reaching-the-agent/): which of the direct and relay paths your network allows. - [API user](https://docs.jmrp.io/mikroscope/security/api-user/): the RouterOS user the relay and `--log-markers` need.