# Dashboards The five Grafana dashboards for MikroTik RouterOS kernel telemetry, one per store: how they get into Grafana, every section and panel, and what each store cannot carry. Source: https://docs.jmrp.io/mikroscope/dashboards/ Five Grafana dashboards, one per store Grafana can query (InfluxDB 3, Prometheus, PostgreSQL, Graphite and Elasticsearch), generated from one panel list in [`internal/dashboards/`](https://github.com/jmrplens/mikroscope/tree/main/internal/dashboards), with an alert-rule file beside three of them ([Alert rules](https://docs.jmrp.io/mikroscope/dashboards/alerts/)). `install` does not put them in Grafana: it writes only to the router. ## Get them into Grafana A dashboard reads the store the collector writes to ([Run the collector](https://docs.jmrp.io/mikroscope/sinks/)), and gets into Grafana one of three ways: - **From the collector**: `forward … --grafana ` creates the datasource and publishes the dashboard at every start, and `dashboards publish` does the same once ([details](https://docs.jmrp.io/mikroscope/dashboards/import-and-check/#letting-the-collector-do-it-forward---grafana)). - **With the CLI**: `dashboards import --store --datasource-uid `, against a datasource you already have ([details](https://docs.jmrp.io/mikroscope/dashboards/import-and-check/#importing-from-the-cli)). - **By hand**: upload the JSON in Grafana ([details](https://docs.jmrp.io/mikroscope/dashboards/import-and-check/#importing-by-hand)). [Set up in Grafana](https://docs.jmrp.io/mikroscope/dashboards/import-and-check/) has the datasource for each store and the check that runs every panel. ## Screenshots One capture per section of the InfluxDB dashboard, in the order the dashboard puts them. Each is a link to the file at full size. The captures show a demonstration database, not a router: the InfluxDB dashboard over a run of the canned fake agent the end-to-end suites use, written into a container store by the fill step of the suite in [`test/e2e/docker/`](https://github.com/jmrplens/mikroscope/tree/main/test/e2e/docker) and photographed by [`site/scripts/gen-dashboard-captures.mjs`](https://github.com/jmrplens/mikroscope/blob/main/site/scripts/gen-dashboard-captures.mjs). The host is called `rb5009` because the fake agent imitates that board's captured `/proc`, and the figures are whatever the fake publishes: read them as the shape of the page, never as a measurement. A panel that is empty in a capture is one the fake agent produces nothing for; on a real router with the API tier running, several of them fill in, and two whole sections that the store probe moved into "not available" come back. One capture per section of the InfluxDB dashboard, over a demonstration database filled by the fake agent, on [the page](https://docs.jmrp.io/mikroscope/dashboards/): - Overview (13) - CPU and scheduler (11) - Memory and load (9) - Connections (9) - Interface traffic (14) - Detections and captures (6) - Network receive path (9) - Forwarding cost (derived) (4) - Interrupts and softirqs (12) - Temperature and clock (10) - Kernel log (7) - CPU: how long a core stayed busy (3) - Memory: fragmentation (3) - Memory: reclaim and page faults (8) - Memory: detail and cross-checks (5) - Hardware counters (PMU) (11) - Flash wear (7) - NAND health (ECC) (2) - RouterOS API cross-checks — CPU and memory (9) - The observer (11) - The observer: sampler timing and self events (6) - This device (3) - Not available on this device (5) ## Panels per store Every panel is declared once, with its SQL and its PromQL side by side, so a panel added to one store is added to the other. Where a panel has no query for one store, the generator **drops it from that store's dashboard** rather than shipping it to read "No data" forever. A section all of whose panels are dropped emits no row at all. That is why the counts differ, and they differ in both directions. Most of the thermal and clock family, the flash wear section, the slab census and several memory panels have SQL and no PromQL, so they exist only on InfluxDB. Three things exist only on Prometheus, because none of them is written to InfluxDB: the trigger suppressions, the busy run still in progress and the age of each held reading, all exposition families on the collector's `/metrics`. The sampler's timing histograms and the capture budget are on the collector's `/metrics` too, but also have an InfluxDB form. Each panel below that is on one store only says which. Panels per section, per store: | Section (Grafana row) | InfluxDB 3 | Prometheus | PostgreSQL | Graphite | Elasticsearch | | --- | --- | --- | --- | --- | --- | | Overview | 13 | 13 | 12 | 10 | 6 | | CPU and scheduler | 11 | 9 | 11 | 1 | no row | | Memory and load | 9 | 9 | 9 | 7 | 5 | | Connections | 9 | 3 | 9 | 1 | 1 | | Interface traffic | 14 | 12 | 10 | 2 | no row | | Detections and captures | 6 | 4 | 6 | no row | no row | | Network receive path | 9 | 7 | 9 | 3 | 3 | | Forwarding cost (derived) | 4 | 4 | 4 | no row | no row | | Interrupts and softirqs | 12 | 10 | 12 | 2 | 2 | | Temperature and clock | 10 | 2 | 10 | 2 | 2 | | Kernel log | 7 | 6 | no row | no row | no row | | CPU: how long a core stayed busy | 3 | 4 | 3 | no row | no row | | Memory: fragmentation | 3 | 1 | no row | no row | no row | | Memory: reclaim and page faults | 8 | 6 | 8 | 2 | 1 | | Memory: detail and cross-checks | 5 | 3 | 5 | 1 | 1 | | Hardware counters (PMU) | 11 | 7 | 11 | no row | no row | | Flash wear | 7 | no row | 7 | 1 | 1 | | NAND health (ECC) | 2 | 2 | 2 | no row | no row | | RouterOS API cross-checks — CPU and memory | 9 | 9 | 9 | 1 | no row | | The observer | 11 | 9 | 10 | 3 | 3 | | The observer: sampler timing and self events | 6 | 7 | 6 | no row | no row | | This device | 3 | 3 | 3 | no row | no row | | Not available on this device | 5 | 5 | 5 | 3 | 3 | | **Total** | **177** | **135** | **161** | **39** | **28** | The counts are those of the committed files, which carry the compiled defaults. `dashboards import`, `dashboards publish` and `forward --grafana` ask the datasource what it holds first and can move panels into or out of the last row, and `check` checks the dashboard `import` would store; see [Not available on this device](https://docs.jmrp.io/mikroscope/dashboards/#not-available-on-this-device) below. ## Section order The sections are ordered by how often they are opened, not by taxonomy: 1. the four questions an operator arrives with: how busy the router is, how much memory is left, how many connections it holds and how much traffic is moving; 2. what the collector and the agent flagged; 3. the families that explain the first four when one looks wrong; 4. the deep tiers a reader opens deliberately; 5. what mikroscope costs the router it is measuring. Every section but the Overview ships **collapsed**. Grafana keeps a collapsed row's panels inside the row object and runs none of their queries until someone expands it, so the first render asks the store for the Overview's thirteen panels (twelve on PostgreSQL) and not for all 177. The defaults are a 3-hour range (`now-3h`) and a 5-minute refresh. A 15-minute window shows empty panels while the agent is stopped, and unreadable noise from 100 ms samples while it runs. The slow refresh is for an expanded section: the slab census and the PMU and per-sample cost heatmaps return one row per sample, and `maxDataPoints` does not apply to raw SQL. Set your own range and refresh for a live investigation. ## Overview Overview is the only section expanded by default. It shows whether the router is healthy now, and first whether its numbers can be trusted. - **Aggregates only.** No panel is a heatmap or a join across measurements, and every panel returns an aggregate. Three tiles ("Reboots in the window", "Sample continuity" and "Ticks never delivered, this window") compute theirs with a window function over the window's raw samples. - **Copies.** Most panels are copies of panels in a later section, because a Grafana panel belongs to exactly one row. The copies share their SQL. On Prometheus, the Overview's "OOM kills in the window" and the Observer's "Ticks never delivered, this window" differ from their counterparts: the Observer's carries a third query, the sampler's slipped ticks. - **Only here.** "Connections tracked right now", "Load average (1 min) against the core count", "Reboots in the window", "Detections in the window" and "Port errors in the window". On Prometheus, "OOM kills in the window" too: the Reclaim section's copy has no PromQL and is dropped. 1. **CPU busy per core**, **Memory in use, against the kernel's own total** ( `(MemTotal − MemAvailable) / MemTotal` over time, with dashed lines at 75 % and 90 %) and **Connections tracked right now** (the `nf_conntrack` slab's active objects; blank on an unprivileged container, and the tile says so). 2. **Interface throughput — rx above, tx below** (API tier; blank with `--api-mode off`, and the tile names the flag), **Die temperature by zone** and **Load average (1 min) against the core count**, where the core count is measured from the store rather than assumed. 3. **Sample continuity**, full width: each bin, never narrower than a minute, classified as continuous, ticks missing or agent restarted from the first difference of the sample sequence number. On Prometheus, which has no sequence number, the lane approximates this from the collector's gap counter and resets of the agent's sample counter, and its description says so. Every other panel on the dashboard should be read against this lane. 4. **Packets dropped in the kernel RX path (window total)**, **OOM kills in the window**, **Reboots in the window** and **Detections in the window** — each zero on a healthy device and colored on its own thresholds. 5. **Port errors in the window** (every typed MAC error on every port, summed; API tier, blank with `--api-mode off`) and **Ticks never delivered, this window**, which also counts agent restarts. The Overview is 31 grid units tall on InfluxDB, Prometheus and PostgreSQL, row header included (21 on Graphite, 14 on Elasticsearch). Below about 768 px wide, Grafana stacks each 24-column row into one column, so on a phone the Overview runs to several screens ([rendered heights](https://docs.jmrp.io/mikroscope/about/status/#the-dashboards)). ## Everyday sections ### CPU and scheduler `/proc/stat` ticks and the sampler's own cadence. Every per-core series comes from the data (`GROUP BY cpu`, `by (cpu)`), so a 2-core and an 8-core board each draw their own lines. - CPU busy per core - Busy per core, p95 of one-second means - Worst sample interval, relative to the window's median - Per-core busy as states — which core paid, and when - Where the ticks went — device share by mode - softirq share of busy time, per core - Busy-tick distribution per sample - Share of samples the tick counter called completely idle - Cycles retired in samples /proc/stat called idle — InfluxDB only - nice, irq and iowait ticks in the window - Tick accounting closes — InfluxDB only ### Memory and load The levels from `/proc/meminfo` and `/proc/loadavg`, averaged or maxed over a bin and never summed. Every "percent of RAM" divides by the kernel's own emitted total. - Memory in use, against the kernel's own total - Load average, all three windows - Runnable threads out of total - Memory by category, as a share of total - Free memory — three definitions, against the ceiling - Commit headroom — Committed_AS as a share of CommitLimit - Load average (1 min) per core - Threads on the whole router - Slab — the conntrack and route tables the netns hides ### Connections The router's connection table from the global slab allocator (`/proc/slabinfo`, privileged only), next to the API tier's count where it is polled. An empty section here means an unprivileged agent, not an idle router; the "Slab caches reporting" tile exists to say which. - Connection table, two ways — slab objects against the RouterOS API count - Connection churn floor — peak-to-trough swing inside each bin — InfluxDB only - Packet-buffer and large-allocation caches — InfluxDB only - Every slab cache, normalized to its own window minimum — InfluxDB only - Connection count distribution over time (nf_conntrack) — InfluxDB only - Slab census — latest, min, max and spread per cache — InfluxDB only - Connection table occupancy - Connections as RouterOS counts them (API poll) - Slab caches reporting (is this a privileged deployment?) — InfluxDB only ### Interface traffic `/interface/monitor-traffic`, the MAC's per-port counters and `/system/health` from the RouterOS API — the only per-interface counters mikroscope has, because the container's `/proc/net/dev` describes its own veth — beside the configuration the API tier reads at start and every `--labels-every`: what each interface is. The rates arrive already per second and are never summed, and neither are the per-port counters, because RouterOS counts a different thing on each type. A switch port counts its wire, hardware-forwarded frames included; the bridge counts its CPU side; a VLAN or a PPPoE counts what the CPU sent and received. A switch port and the bridge are therefore different planes, and neither is a subset of the other ([measured](https://docs.jmrp.io/mikroscope/about/status/#verified-port-planes)). The rate panels' interface set is whatever `--interfaces` selected, so an interface missing from one may be idle, unconfigured or not selected, and the three look the same. - Interface throughput — rx above, tx below - Interface packet rate — rx above, tx below - Mean packet size per interface - CPU cost of forwarding: cpu-load against the busiest interface's packet rate — InfluxDB only - Port errors per bin — typed, from the MAC counters - Where a port's receive bytes went — switched in hardware, CPU fast path, CPU slow path — InfluxDB only - Link flaps — link-downs per interface, per bin - Frame size mix over the window, per Ethernet port - Egress queue drops — the router's own transmit queue - Packets the interface itself dropped — rx and tx - What each interface is: type, role, bridge and label - Interface inventory and window summary - /system/health sensors, as RouterOS reads them - API tier coverage — which sources delivered, and when "What each interface is: type, role, bridge and label" is the table to read before two interface series are compared: one row per interface the router has — RouterOS type, interface lists, the bridge it is a port of, its comment, its default name and its MTU — from three configuration reads, never per poll. "Frame size mix over the window, per Ethernet port" is one row per port and carries no total: each `tx-rx-*` counter holds both directions of its own port, so a frame switched from ether1 to ether8 is counted on both, and a sum over ports counts every switched frame twice. "Interface inventory and window summary" carries no error columns, because `monitor-traffic` returns no error rate ([verified](https://docs.jmrp.io/mikroscope/about/status/#verified-monitor-traffic-keys)); the MAC's typed error counters have their own panel. ## Events ### Detections and captures Events, not levels: what the collector's [derive stage](https://docs.jmrp.io/mikroscope/sinks/derive/) and the agent's [triggered capture](https://docs.jmrp.io/mikroscope/record/triggers/) said about the window. Empty is the healthy state, and the event panels are marked known-empty so that `check` does not fail on it. - Detections per bin, by rule - Memory pressure state - Detections in this window — InfluxDB only - Trigger fires and suppressions per bin - Captures held on the agent, and the budget they pin - Trigger markers in this window — InfluxDB only In "Trigger fires and suppressions per bin", the suppressions are on Prometheus only: the collector renders them into its exposition and no store carries a field for them. ### Detection annotations A dashboard with detections in its window draws a vertical red dashed line, with a small triangle at the foot of the axis, across **every** panel at the instant of each one. They are annotations, not data: not a gap in the record, not a slipped tick, not a break in the series. They let you read whatever panel you are on next to what the collector flagged at that moment — a memory-stall regime on one core, a microburst on one queue, a link that flapped — without scrolling to the Detections section. Two layers ship with every dashboard: | Layer | Colour | Default | What each marker is | | -------------- | ------ | -------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | **detections** | red | on; off after an InfluxDB probe that finds no `mikroscope_detection` | one row of `mikroscope_detection`: `rule: message`, from the collector's derive stage | | **triggers** | orange | off | one capture marker from the agent: the condition that fired, the field and the value | Hovering a marker shows the message the row carries, so the line answers *what* as well as *when*. A rule that fires per key (a core, a port, a thermal zone) opens its message with that key, so the marker reads, for example, `microburst: cpu0: …`. The SQL does not read the `key` column: the InfluxDB sink writes it only when a detection has one, and six rules never do, so a store whose detections are all of those has no such column and a query naming it fails. The triggers layer is off by default so that a quiet dashboard stays quiet; turn it on when you are working with [triggered captures](https://docs.jmrp.io/mikroscope/record/triggers/). **Before the first detection.** An InfluxDB store has no `mikroscope_detection` table until the collector writes its first detection, and InfluxDB 3 refuses a query that names a missing table; no form of the query avoids that ([forms tried](https://docs.jmrp.io/mikroscope/about/status/#the-dashboards)). `dashboards import`, `dashboards publish` and `forward --grafana` ask the store first, and when the table is not there they ship the detections layer switched off, with its query kept, so you can switch it on later. The committed files and a [manual import](https://docs.jmrp.io/mikroscope/dashboards/import-and-check/#importing-by-hand) have nobody to ask, so the layer stays on: its query fails on every load and refresh without showing anything on the dashboard, and Grafana logs an error-level `Partial data response error` line each time ([observed](https://docs.jmrp.io/mikroscope/about/status/#the-dashboards)). The markers appear on their own once the first detection creates the table. On Prometheus the counter is simply absent and the query returns an empty result, so the layer stays on after a probe too. The PostgreSQL sink creates every table before its first insert, so there the query returns no rows until the first detection ([checked in the container suite](https://docs.jmrp.io/mikroscope/about/status/#the-test-suites)). On one Grafana version tested, the InfluxDB layer's query is never sent, with or without the table, and no marker is drawn ([known issue](https://docs.jmrp.io/mikroscope/about/status/#found-and-not-fixed)). **Turning them off.** Both layers are checkboxes in the submenu row under the dashboard title — the same row a dashboard's variables sit in, which on the InfluxDB, Prometheus and PostgreSQL dashboards holds nothing else. Click the layer's name to hide its markers. That is a view setting: it lasts for the session, and saving the dashboard keeps it. Nothing about the underlying rows changes, and the Detections section still counts them. Each store draws the layers from what its sink writes, in its own query language: - **InfluxDB and PostgreSQL**: one `mikroscope_detection` or `mikroscope_trigger` row per marker, with its message. - **Prometheus**: `increase(…[1m]) > 0` at a 1-minute step, so a marker there is the minute, not the instant. - **Elasticsearch**: a Lucene filter, `kind:detection AND host.keyword:$host` (or `kind:trigger`), over the event documents. A detection's text is its `message` and its tag the `rule`; a trigger document has no message, so its text is the `field` that crossed and its tag the `cause`. - **Graphite**: `aliasByNode($prefix.$host.detection.*, 3)` (or `.trigger.*`), the one point per event the sink writes. Grafana draws a marker at every non-null point and titles it with the series name, which is the rule or the cause. There is no message, because Graphite holds only numbers, and the marker is at the time of the point Grafana gets back: Grafana asks for at most 100 points, so several events close together can come back as one. The Elasticsearch and Graphite layers are checked by the unit tests against what each sink writes ([not run in Grafana](https://docs.jmrp.io/mikroscope/about/status/#not-tested)): the import check walks panels only. The section captures further up are taken with both layers off, because the canned fake agent that fills the demonstration database fires a detection every few seconds and twenty minutes of that photographs as a solid red wash. This is the same overview with the detections layer on, over a demonstration store carrying three of them — the density a real deployment produces, and the tile at the bottom right counts the same three: *The detections layer, on: one red dashed line per detection, on every panel at once.* ## Diagnostic sections ### Network receive path `/proc/net/softnet_stat`, which is global even inside the container's network namespace. No absolute packet-rate band survives here: the squeeze regime is banded as a multiple of the window's own median. - RX path: packets processed per second, per core - Squeeze rate: softirq budget exhaustions per second, per core - Squeeze pressure: budget exhaustions per 1 000 packets - Receive-path balance across cores - Packets dropped in the kernel RX path (window total) - Squeeze regime per core, as a multiple of this window's median - Burst distribution: packets per sample (all cores) — InfluxDB only - Squeeze against throughput (1 s points, whole window) — InfluxDB only - Packets per NET_RX poll, per core ### Forwarding cost (derived) The per-packet ratios and the fast-path share the collector derives across subsystems — PMU against softnet, softnet against interrupts, RouterOS port counters against each other. The fast-path share is the share of the traffic an interface hands the CPU, not a share of the wire: `fp-rx-byte` over `driver-rx-byte` on a switch port, over `rx-byte` on a software interface, with hardware-switched frames in neither number. That split is the "Where a port's receive bytes went" panel's, from the raw counters. The tx line is usually absent: `fp-tx-byte` can stay 0 on every interface after hundreds of GB transmitted ([observed](https://docs.jmrp.io/mikroscope/about/status/#verified-fp-tx-zero)), so the collector withholds the tx share while the cumulative counter is 0 rather than draw a 0 % it did not measure. Per bin the share is bytes-weighted — the bin's fast-path bytes over the bin's bytes — because an interface moving a few packets per poll swings 0–100 % from poll to poll. The Prometheus form is the collector's per-poll gauge and keeps that noise. - Cycles, instructions and cache misses per packet - Packets per device interrupt (NAPI coalescing depth) - Fast-path share of the traffic each interface hands the CPU - Sub-sample bursts: squeezed samples whose packet count looked ordinary ### Interrupts and softirqs `/proc/interrupts` and `/proc/softirqs` deltas. The device name is recovered from the raw `/proc/interrupts` text in the query, not matched against a driver name. - Interrupts per second by source (top-K) - Per-line interrupt load on the device with the most lines - Receive-path IRQ imbalance (max / mean across a device's lines) - Which core takes each interrupt - Top-K interrupt coverage - Interrupt mix right now - Top-K membership — which interrupt sources the agent was watching — InfluxDB only - NET_RX softirq invocations per core - Softirq invocations by kind (all CPUs) - NET_RX burst size distribution per sample — InfluxDB only - µs of softirq CPU per invocation - NET_TX and TASKLET softirq invocations per second ### Temperature and clock `/sys/class/thermal` and cpufreq. Every thermal threshold is measured from the zone's own critical trip point, read from the board: the headroom panel's thresholds sit at 10 °C and 5 °C of remaining margin below it. No panel carries a temperature the board did not publish. - Die temperature by zone - Temperature by source — kernel zones, and the RouterOS sensor when it is polled — InfluxDB only - Thermal slope, °C per minute by zone - Sensor step dwell — where each zone actually sat — InfluxDB only - Latest temperature by zone — InfluxDB only - Thermal headroom — how far each zone is from its own critical trip — InfluxDB only - Gap between the hottest and coolest thermal zone — InfluxDB only - Core clock per core — governor state — InfluxDB only - Is the clock pinned? — InfluxDB only - Clock-weighted work: effective Hz per core — InfluxDB only ### Kernel log `/dev/kmsg`, privileged only. The log is not absent when it is empty, it is silent, and silence is the healthy state. - Worst kernel-log severity in each bin - Kernel-log records per second by severity - Warning-or-worse kernel-log rate now - Kernel records in the window, by severity - Kernel records per sample — burst distribution against the per-tick cap — InfluxDB only - Port events from the kernel log, per port and kind - Port events in the window, per port Two of them read the kernel records that name a network port: - **Port events from the kernel log, per port and kind** counts them per bin, one series per port and kind. - **Port events in the window, per port** is the census beside it: one row per port, with the label, the role and a column per kind. How a port record is built and read: - **Kind.** Each record carries one: link-up, link-down, `stp-`, own-address (the bridge receiving a frame with its own MAC as the source, the layer-2 loop signature) or other. The agent classifies it as it reads `/dev/kmsg`; the collector classifies records from an agent that does not. - **Name and labels.** The collector replaces the board's default port name with the interface's current RouterOS name and attaches its comment and interface lists from the API tier's inventory. Without the API tier the record keeps the default name and gets no label. - **Cost and blind spot.** This per-port view comes free with the container: the kernel log costs the router nothing and dates each transition to the microsecond, where the RouterOS per-port counters need an API read every poll. It is blind to anything the kernel never hears of, the hardware-switched frames included. - **STP sequence.** A link-up is followed by stp-blocking, stp-learning and stp-forwarding on its bridge port: four records, not four faults. - **Known-empty.** Both panels are marked known-empty, because a quiet set of ports is the healthy state. - **Prometheus.** The Prometheus form reads the collector's `/metrics`, where every port record carries a `kind` because the collector classifies whatever reaches it unclassified. On InfluxDB the `mikroscope_kmsg` table is created by the first kernel record written, and its `kind` column by the first port record written with one. On a router whose kernel has not spoken since the collector started, the table does not exist and InfluxDB 3 refuses the query when it plans it; the collapsed section keeps that query from running until someone opens it. The code calls this a mitigation, not a fix. A probed publish, `dashboards import` or the collector's, routes those panels into the not-available row instead. ## Deep-dive sections ### CPU: how long a core stayed busy Contiguity, which no per-sample histogram can recover: a 2 s plateau and twenty scattered spikes bin the same. On InfluxDB the run lengths are a query over the raw samples; on Prometheus they are the upper edge of the largest populated bucket of `mikroscope_cpu_busy_run_seconds`, clamped at 60 s, so 60 reads as "longer than a minute", not as a measurement. - Longest run at or above 90 % busy, per cpu - Longest run at or above 50 % busy, per cpu - Busy run in progress right now — Prometheus only - Blocked tasks and forks ### Memory: fragmentation `/proc/buddyinfo`, the one memory number `/proc/meminfo` cannot give. - Free memory by block order (pages) — InfluxDB only - Free blocks per order (count) - Largest block order with any free block — InfluxDB only ### Memory: reclaim and page faults The `/proc/vmstat` deltas, summed or rated over a bin and never averaged — a separate section from the levels so the two kinds of reducer cannot be mixed. - Reclaim efficiency — pgsteal ÷ pgscan - Did the kernel have to reclaim at all? - Allocation distress — stalls and swap - OOM kills in the window — InfluxDB only - Page churn — allocate, free, and the net - Page faults — minor and major per second - Minor-fault bursts — distribution per sample — InfluxDB only - Context switches and all interrupts per second ### Memory: detail and cross-checks Writeback, the LRU, the small levels, and two panels that validate the instrument itself. These are read once per new device, not during an incident. - Writeback backlog — dirty pages and pages in flight - LRU balance — active vs inactive - Mapped, kernel stacks and page tables - vmstat pages vs meminfo kB — unit cross-check — InfluxDB only - Kernel stack per thread — InfluxDB only ### Hardware counters (PMU) `perf_event_open` from the privileged container. Every panel divides two raw counts; the clock-normalized ones divide by the frequency the kernel reported for that core in that sample. - IPC per core (instructions retired / cycles) - Beneath the tick floor: PMU cycles against /proc/stat busy ticks — InfluxDB only - Instructions retired while /proc/stat reported the core idle — InfluxDB only - Work the jiffie threw away (selected range) — InfluxDB only - Core cycles per bus cycle - Unhalted-cycle fraction of the clock, per core - Distribution of cycles per sample, as a share of the clock (all cores pooled) — InfluxDB only - Cache-miss rate per core (misses / references) - Cache misses per 1 000 instructions (MPKI), per core - Branch mispredictions per 1 000 instructions per core - PMU counters this CPU actually opened ### Flash wear `/proc/yaffs`, the only NAND wear signal on a RouterBOARD. InfluxDB only: the section has no row on the Prometheus dashboard. No panel can show a share of the partition, because the agent does not parse the partition's block range, so there is no denominator for one. - Flash page traffic per YAFFS partition (pages/s) - Write amplification — GC copies per page write - Flash housekeeping — erasures, garbage collections and GC copies per bin - YAFFS free-chunk drift within the window (chunks, relative to the first sample) - YAFFS partition state over the window - Bad blocks retired during the window, per YAFFS partition - Page writes and erasures per day, at the window's rate ### NAND health (ECC) The MTD ECC counters under `/sys/class/mtd`, privileged only — the flash's leading indicator, where the YAFFS bad-block count is the post-mortem. - ECC corrections since boot, per partition, against the bitflip threshold - Uncorrectable ECC failures and bad blocks, per partition ### RouterOS API cross-checks — CPU and memory `/system/resource` and `/system/resource/cpu`: the independent tier the kernel figures are checked against. RouterOS recomputes these once per second, so every field is 1 Hz at best and integer-quantized; no panel here claims a sub-second reading. - RouterOS cpu-load vs kernel busy — do the two tiers agree? - Cross-tier residual: cpu-load − kernel busy, distribution - Per-core load, RouterOS's own accounting - Per-core mean over the window: RouterOS load next to kernel busy - Per-core IRQ time as RouterOS accounts it, against kernel softirq+system - Per-core disk time (RouterOS) — max over the window - RAM used, as RouterOS accounts it - Free memory: RouterOS free-memory vs the kernel's two answers - RouterOS uptime ## Observer sections ### The observer What mikroscope costs the router it is measuring, and whether it was running. On InfluxDB, continuity is derived from the sequence number of every sample (Prometheus approximates it, as in the Overview), which can reveal missing ticks and restarts that the collector's gap record does not show ([example](https://docs.jmrp.io/mikroscope/about/status/#the-dashboards)). - Agent CPU cost against its 2 % budget - Observer effect: agent share of all busy CPU on the router - CPU per sample: mean and worst tick - Where the agent's cost actually lives (per-sample distribution over time) — InfluxDB only - Agent memory against the container cap - Headroom under the container memory cap - CPU budget used (window mean) - Ticks the agent took vs ticks the store received - Sample continuity - Ticks never delivered, this window - Gaps and restarts in this window — InfluxDB only ### The observer: sampler timing and self events The sampler's own smear — how late it woke and how long the read took — and the cgroup events the agent records about itself. The timings ride in each sample's `self` block as `wake_ns` and `read_ns`, so all three heatmaps exist on the InfluxDB and PostgreSQL dashboards as well as on Prometheus. - Tick interval distribution, relative to the nominal period - Wake latency: how late the sampler ran after its ticker - Read duration: how long every source took to read - Counter resets the agent saw - The container's own throttling and OOM kills - How each level source is read - Age of each held reading — Prometheus only ### This device The [device-info stream](https://docs.jmrp.io/mikroscope/sinks/device-info/): what the agent established about the board at start with no RouterOS API — identity, ceilings, the frequency ladder. - This device, as the agent established it - Thermal zones: the board's own trip points and polling cadence - CPU clock: range, ladder, governor and clusters ## Not available on this device The last row is titled "Not available on this device — measurements this kernel or board does not produce (open to read why)" and is collapsed. A panel whose measurement the store does not hold is moved out of its own section into this row, where its description says what it is waiting for. In the committed files — the compiled defaults, which are what plain `gen` writes and what a manual upload into Grafana gets — the row holds five panels: the PSI panel and the four block-device panels. On Graphite and Elasticsearch it holds three: the queue-depth and busy-percent panels have no query in either store's language, so those two dashboards leave them out. - Pressure stall (PSI), where the kernel exposes it - Block-device queue depth (requests in flight) - Block-device requests per second (reads and writes completed) - Block-device busy percent (io_s / wall time) - Block-device throughput (sectors → bytes per second) The PSI panel needs a kernel with `/proc/pressure`. The block-device panels need a block device that moves: the agent drops a block device whose reads, writes and in-flight count are all zero in a tick, so on a router whose listed devices all stay at zero no sink ever creates the table. The compiled defaults put both here because the device they were built against produces neither ([which device](https://docs.jmrp.io/mikroscope/about/status/#reference-hardware)). **On InfluxDB every query in this row ships switched off** (hidden, in Grafana's query editor), so opening the row runs nothing and each panel shows a short note instead of data ([rendered](https://docs.jmrp.io/mikroscope/about/status/#the-dashboards)). If your store does hold the measurement, switch the panel's queries back on in its editor, or publish the dashboard again with `dashboards import`, `dashboards publish` or a restart of a collector run with `--grafana` (InfluxDB and Prometheus; the other three cannot be probed), which asks the store and moves the panel back into its own section. **On the other four stores the row keeps its queries on**, because there a missing measurement is not an error. The PostgreSQL sink creates every table, `mikroscope_psi` and `mikroscope_disk` included, before its first insert, so the queries return no rows; Prometheus answers an absent metric with an empty result, and Graphite a path that matches nothing with an empty body. Elasticsearch answers with a line at zero: a `sum` over documents that lack the field is 0 in every bucket, even when another router in the same index has mapped the field ([checked in the container suite](https://docs.jmrp.io/mikroscope/about/status/#the-test-suites)). A zero there is a value, not an empty result, and it cannot tell a router that lacks the measurement from one whose values add up to zero: the row's title, not the line, is what says the measurement is not produced. The queries stay on so that a router that does produce the measurement fills the panel with no edit; on PostgreSQL, Graphite and Elasticsearch no probe could switch them back on. After a probe, on any store, a panel the probe found missing ships with its queries switched off, as on InfluxDB. Two sections are declared for these panels, **Pressure stall (PSI)** and **Block devices**, and ship no row while every panel in them is absent. On a kernel built with PSI, or a board with USB or eMMC storage that moves, `import`, or the collector at its next start, finds the measurement and the section appears in its place, queries on, with no edit to the generator. That takes a store `import` can probe, InfluxDB or Prometheus; the other three cannot be probed and keep the compiled row. `import` also works the other way: on a store that lacks a measurement the compiled defaults expect — or a field added after that store was first written — the panel moves into this row with its queries switched off, so it shows its explanation and not a red error badge. How the probe decides is on [Set up in Grafana](https://docs.jmrp.io/mikroscope/dashboards/import-and-check/#the-probe). ## Ratios, gaps and units - **Ratios are never the agent's.** The agent ships raw counters only, never percentages. A ratio on these dashboards is computed either in the panel's own query or by the collector's derive stage (the Forwarding cost section and the `mikroscope_derived*` families). The API tier's interface rates are the exception: they arrive from RouterOS already computed. - **Short holes are not drawn.** A line is broken where two neighbouring points are more than 5 minutes apart. The threshold has to exceed the widest bin a reader selects — at a 2-day range a 12-column panel's bin is about 4 minutes — so a hole shorter than 5 minutes is still drawn as an interpolated line. Read holes from Sample continuity, not from the shape of a line. - **Stat numbers are a fixed size.** A stat tile draws its value at 32 px and its title at 16 px instead of Grafana's automatic size, which fills the panel: on a phone, where every panel is full width, the automatic size turns each stat tile into a large block of the screen. - **No device is compiled in.** No title, query or threshold names a device, its core count, its interfaces or its memory size; the only fixed numbers are mikroscope's own budget targets (2 % of one core, 16 MiB). A panel description that quotes a measured figure names the device and the date it was measured on, and yours will differ. ## Files - dashboards/ - mikroscope-influxdb.json 177 panels, InfluxDB 3 (SQL) - mikroscope-prometheus.json 135 panels, Prometheus - mikroscope-postgres.json 161 panels, PostgreSQL / TimescaleDB - mikroscope-graphite.json 39 panels, Graphite - mikroscope-elasticsearch.json 28 panels, Elasticsearch - mikroscope-alerts-influxdb.yaml 14 rules - mikroscope-alerts-prometheus.yaml 15 rules - mikroscope-alerts-postgres.yaml 10 rules ```sh mikroscope dashboards gen # writes the eight files into ./dashboards, made if missing mikroscope dashboards gen --out /tmp/dash # or into another directory ``` - **Format.** Grafana's shareable export format: the datasource is a `${DS_MIKROSCOPE}` placeholder declared in `__inputs`, there is no `id`, and the `uid` is fixed, `mikroscope-`, one per dashboard, so a re-import updates the same dashboard in place instead of creating a second one. A test asserts that two generations of the InfluxDB dashboard are identical. - **Grafana version.** `__requires` declares Grafana 11.0.0. The panel options are written to the schema Grafana 13.2.1 expects: the xychart's mark, for one, moved between Grafana 11 and 13. The versions the dashboards have been checked on are on [Tested on](https://docs.jmrp.io/mikroscope/about/status/#the-dashboards). - **InfluxDB 3.** The queries are SQL, and every one is bounded by `$__timeFilter`, because InfluxDB 3 Core refuses unbounded scans. Where a column is named `cluster` the SQL quotes it, because `cluster` is a reserved word in DataFusion. - **Prometheus.** The queries read a Prometheus that scrapes the collector's `/metrics`, and that alone: the kernel tier recomputed from the samples, the collector's own derived and detection families, and what only the sampler can produce, which reaches the collector as data because the agent serves no exposition of its own. Prometheus 3 renders a histogram's zero bucket boundary as `le="0.0"`, so a query reading that bucket matches `le=~"0|0.0"`. The scrape job is on [Set up in Grafana](https://docs.jmrp.io/mikroscope/dashboards/import-and-check/#prometheus-one-scrape-job). ## See also - [Set up in Grafana](https://docs.jmrp.io/mikroscope/dashboards/import-and-check/): publish them from the collector, import them with the CLI or by hand, and check every panel. - [Alert rules](https://docs.jmrp.io/mikroscope/dashboards/alerts/): the rules generated beside the dashboards, and where each threshold comes from. - [Detection rules](https://docs.jmrp.io/mikroscope/sinks/detections/): the events behind the Detections section and the annotations. - [Prometheus metrics](https://docs.jmrp.io/mikroscope/reference/metrics/): what each Prometheus panel reads.