# Diagnose faults

The signature of each fault mikroscope has read on a MikroTik router, where it shows in the data, and the two checks to run before you trust a reading.

Source: https://docs.jmrp.io/mikroscope/playbooks/

Match what the data shows against the signatures below, then open the case study for the full
reading and the commands that reproduce it on your own device. Each signature is a change against
the idle shape: a kernel log that is normally silent carrying a steady stream, one core pinned while
the device total stays modest, one interrupt line rising several-fold.

## Before you test

Nothing the provoked case studies do can break the router's uplink or cut an administrator's path
to it. Keep that property on your own device:

1. Find the port you are connected through:

   ```routeros
   /interface/bridge/host/print where mac-address="<your machine's MAC>"
   ```

2. Leave that port, the WAN port and any port carrying a service alone.

## Fault signatures

Read the idle shape first. Without it, every other signature looks like an anomaly.

| Fault                                                                                     | Where it shows                          | Signature                                                                                                                                  |
| ----------------------------------------------------------------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| [Idle baseline](https://docs.jmrp.io/mikroscope/playbooks/idle/)                                              | per-core busy, `time_squeeze`, `events` | a low, steady busy floor on every core, a squeeze that is never zero, no kernel events                                                     |
| [Layer-2 loop](https://docs.jmrp.io/mikroscope/playbooks/loop/)                                               | `events` (the `kmsg` source)            | a steady stream of `events` at idle, spaced at the STP hello interval, while RouterOS's log and port monitor stay clean                     |
| [Port names](https://docs.jmrp.io/mikroscope/reference/port-names/)                                           | `events`                                | the kernel names a port by its own netdev name, which can differ from the RouterOS name                                                    |
| [CPU-bound core](https://docs.jmrp.io/mikroscope/playbooks/cpu/)                                              | per-core busy, temperature, frequency   | one core near 100 % while the device total sits near 100 % divided by the core count; the load hops between cores before it settles        |
| [Wake-up storm](https://docs.jmrp.io/mikroscope/playbooks/cpu/#when-the-cores-are-not-busy-but-the-kernel-is) | context switches, timer interrupts      | a context-switch rate several times its mean over the previous day, timer interrupts in bursts on one core at a time, busy columns flat     |
| [Packet flood](https://docs.jmrp.io/mikroscope/playbooks/packet-flood/)                                       | interrupts, softirqs, `time_squeeze`    | one interrupt line rising several-fold with a sharp onset and offset, its cost on the one core it is pinned to                             |
| [Flash wear](https://docs.jmrp.io/mikroscope/playbooks/flash-wear/)                                           | the `yaffs` source, MTD ECC counters    | `pw` and `er` deltas at idle that you did not cause (look first at logging actions set to `disk`); `corrected_bits` rising or any `ecc_failures` |
| [Conntrack without the API](https://docs.jmrp.io/mikroscope/playbooks/conntrack/)                             | the `nf_conntrack` slab cache           | the router's real connection count, where the container's own namespace reports 0                                                          |
| [Port losing frames](https://docs.jmrp.io/mikroscope/playbooks/port-errors/)                                  | the API tier's per-port MAC counters    | `rx overflow` on one port, in intervals that carry a small fraction of the link's capacity: a burst, not a load                            |

The wake-up storm has no case study of its own: its reading is part of the CPU-bound core study,
and the rule that watches for it is `mikroscope-wakeup-storm` in
[Alert rules](https://docs.jmrp.io/mikroscope/dashboards/alerts/).

## Check the data first

- **The sampler was not starved.** `mikroscope_slipped_total` should be 0 over the window you are
  reading. A slipped tick is one whose read finished after the next tick was due, and the sampler's
  own accounting is then the first thing to distrust.
- **The kernel log was kept whole.** `mikroscope_kmsg_dropped_total` counts loss events, not
  records: one per tick that hit the agent's cap of 64 records, and one per kernel ring overrun,
  which can stand for many records. While it is non-zero, the per-level counts in
  `mikroscope_kmsg_records_total` are a lower bound, and so is an event rate taken from them.

> **The commands on these pages carry no token**
>
> The `curl` examples talk to the agent at `http://172.30.10.2:9123` without credentials, which is
> how a default install answers. An agent started with a token — mandatory with `--expose` —
> returns 401 on every path but `/healthz` until each request carries the header
> `Authorization: Bearer <token>`.

To measure what the agent costs while you run a scenario, read the collector's `/metrics`, not a
snapshot: [Agent cost](https://docs.jmrp.io/mikroscope/cost/#measuring-it-on-your-own-device) has the procedure.

## See also

- [Agent cost](https://docs.jmrp.io/mikroscope/cost/): the budget, the measured cost and how to measure it on your router.
- [Resolution limits](https://docs.jmrp.io/mikroscope/limits/): why one core's busy percentage in a 100 ms sample moves
  in 10 % steps, and what reads beneath that.
- [Detection rules](https://docs.jmrp.io/mikroscope/sinks/detections/): the rules the collector runs over these same
  signals.
- [First recording](https://docs.jmrp.io/mikroscope/start/walkthrough/): a recording with markers, the other way to read
  a transient.