# Source cadences

Which kernel sources the agent reads and stores every tick and which less often, the named reason each reports, and FLOOR_HZ, which turns every floor off.

Source: https://docs.jmrp.io/mikroscope/limits/source-floors/

Not every source changes as fast as the sampler ticks, and storing a value more often than the
hardware refreshes it is storage for no information. The agent reads every counter on every
tick except the MTD ECC counters (every 10 s), slows a level source only for a reason it names,
and publishes each cadence so a consumer never has to infer it. `FLOOR_HZ` turns every floor off at once.

## Counters

CPU ticks, interrupts, softirqs, the vmstat counters, the PMU and the memory levels (which change
many times a second, [measured](https://docs.jmrp.io/mikroscope/about/status/#campaign-floors-overnight)) are read and shipped on
every tick. For a counter a delta of zero is real information, the core was idle, so there is
nothing to skip.

Three more are read every tick with their own rule:

- **NAND wear (`/proc/yaffs`) and block I/O (`/proc/diskstats`)** are counters and cheap to
  read. A device's row is stored only when its delta is non-zero: a router can list many idle
  devices, `nbd` among them, and a row per tick for each would be payload and nothing else.
- **The kernel log ([`/dev/kmsg`](https://docs.kernel.org/admin-guide/abi-testing.html#symbols-under-dev-kmsg))** is drained every tick and never slowed. An event's value
  is its timestamp, and a marker delayed by a second no longer lines up with the spike it
  explains.

## Level sources

A level source (a temperature, a clock frequency, a cache population) may be read or stored
less often than every tick, but only for a reason the agent names. It is never slowed because
it was seen changing slowly, since that would measure one night on one device. Each level
source reports its cadence and reason on `/capabilities` under `cadences`, on `/metrics` as
`mikroscope_source_cadence_hz{source,reason}`, and to every sink in the
[device-info stream](https://docs.jmrp.io/mikroscope/sinks/device-info/), so a consumer reads the true cadence of
a field instead of inferring it from the data.

The cadence reasons a source can report:

| `reason` | What it means |
| --- | --- |
| `rate` | read at the full sampler rate; nothing the device declares justifies less |
| `declared` | the device publishes its own refresh cadence, and reading faster returns the same value with new dither |
| `policy` | a setting says the value cannot move on its own: a `userspace` cpufreq governor |
| `budget` | a measured parse cost |
| `change` | read every tick, stored only when it moves |
| `override` | `FLOOR_HZ` is set, and every level source is on its one cadence |

`rate` describes the read cadence only, so `/proc/buddyinfo` reports `rate` although it is
stored on change. The MTD counters report `budget` although no parse cost was measured for
them. Both are how the code labels them, and neither matches the table above exactly.

## Cadence per source

| Source             | Read                                                    | Stored                  | `reason`                                                                                                       |
| ------------------ | ------------------------------------------------------- | ----------------------- | -------------------------------------------------------------------------------------------------------------- |
| thermal zones      | at the zone's declared `polling_delay`, else every tick | every read              | `declared`, or `rate` where the board declares none or the sampler rate is no faster than the declared cadence |
| `scaling_cur_freq` | every tick                                              | on change, or heartbeat | `change`, or `policy` under a `userspace` governor                                                             |
| `/proc/slabinfo`   | at about 6 Hz                                           | on change, or heartbeat | `budget`                                                                                                       |
| `/proc/buddyinfo`  | every tick                                              | on change, or heartbeat | `rate`                                                                                                         |
| MTD ECC counters   | every 10 s                                              | on change, or heartbeat | `budget`                                                                                                       |

A slower cadence is a whole number of ticks: the sampler rate divided by the floor, rounded to
the nearest integer and never less than one. So the reported rate is what actually happens,
not the nominal floor.

**Temperature.** A thermal zone declares its own [`polling_delay`](https://docs.kernel.org/driver-api/thermal/sysfs-api.html). At 1 000 ms the kernel
itself re-reads the sensor at 1 Hz, and at 10 Hz the agent reads every 10th tick. The sensor
quantises to steps of about 0.42 °C, and the raw reading dithers across
a step boundary tens of times a second. Storing on change would store that dither as signal;
sampling and holding at the declared cadence captures the real curve and drops the dither. On a
board that declares no cadence the zones are read every tick.

**CPU frequency.** A frequency step is a real DVFS transition, not noise, so the frequency is
read every tick and stored when any core moves. On a device with a pinned clock that stores
nothing after the first reading but the heartbeat; on a scaling device it catches every move.
Under a `userspace` [cpufreq governor](https://docs.kernel.org/admin-guide/pm/cpufreq.html#policy-interface-in-sysfs) the clock cannot move without a write, so the
reason reads `policy`.

**`/proc/slabinfo`** is the expensive one, the largest per-tick parse by an order of magnitude,
and that cost is the reason it is slowed at all. The 6 Hz it is slowed to is the rate its fastest
cache, `nf_conntrack`, was measured changing. It is stored on change. At 10 Hz that is every 2nd
tick (5 Hz); at 100 Hz, every 17th (about 5.9 Hz). How that amortisation shows in the per-sample cost is on
[Rate ceiling](https://docs.jmrp.io/mikroscope/cost/rate-ceiling/).

**`/proc/buddyinfo`** is about 100 bytes, among the cheapest files the agent reads, and has no floor
because none has been measured. The free lists churn with every allocation, so on a busy
router it will be stored on most ticks, and that is the measurement rather than noise.

**MTD ECC counters** are read every 10 seconds. That is not a measured floor: the counters
move on the scale of a device's life, and each read is six small sysfs files per partition, so
ten seconds is an arbitrary but generous choice. The code still labels that cadence `budget`,
although no parse cost was measured for it. They need `privileged=yes`.

The 6 Hz of `/proc/slabinfo` is the one measured floor, and it comes from one device over one idle
night ([Tested on](https://docs.jmrp.io/mikroscope/about/status/#campaign-floors-overnight)). A router with a scaling clock, a
busy dirty-page workload or a different board has different floors, so re-measure with
`FLOOR_HZ` equal to the sampler rate before trusting them on yours. The floors are named
constants in [`internal/agent/source.go`](https://github.com/jmrplens/mikroscope/blob/main/internal/agent/source.go).

## Heartbeat

Every source stored on change is also re-emitted about once every 60 seconds (on the first
read due after 60 s), so a value that sits still for an hour still has a recent row in any
store. The container's own `memory.max`, a constant, is re-emitted in the samples once per
heartbeat (every tick under `FLOOR_HZ`); the `limits` on `/capabilities` and the
[device-info stream](https://docs.jmrp.io/mikroscope/sinks/device-info/) carry it from start.

On `/metrics`, a floored gauge between emissions holds the last reading, because a level's
value between readings is the last one read, not nothing.
`mikroscope_source_age_seconds{source}` says how old that held reading is.

The change filter is re-armed after the sampler's baseline read, so the first sample carries
every source stored on change, and the floored gauge families reach `/metrics` with it rather
than a heartbeat later.

## `FLOOR_HZ` override

Every floor is overridable at once, without a rebuild:

```sh
mikroscope install --rate 10 --floor-hz 10   # writes FLOOR_HZ=10 into the agent's envlist
```

`FLOOR_HZ` is the agent's environment variable; `--floor-hz` is the deployment flag (`install`,
`upgrade`, and `plan` for the listing) that writes it into the container's envlist, only when
it is above zero. It is one
global setting in hertz, from 0 to 1000.

- `0`, the default, keeps the per-source floors above.
- Any `N > 0` puts the thermal zones, `/proc/slabinfo` and the MTD counters on one cadence of
  N Hz, and turns off the store-on-change filter for every level source, so nothing is held
  back and a capture sees every read. Every level source then reports the reason `override`.
- `N` equal to or above `--rate` reads and emits everything every tick. That is the
  configuration the floors were measured from, and the one to re-run before trusting any
  number on this page on your device.

Two things `FLOOR_HZ` does not change. The per-device row filter on `/proc/yaffs` and
`/proc/diskstats` stays: a device whose counters did not move still has no row, which loses
nothing because the delta was zero. And `scaling_cur_freq` and `/proc/buddyinfo` are read
every tick with or without it; with `N` below `--rate` they are emitted every tick, but their
reported cadence is the override's, not the sampler rate.

What it costs to read everything every tick, at 50 and 100 Hz, is on
[Rate ceiling](https://docs.jmrp.io/mikroscope/cost/rate-ceiling/).

## See also

- [Rate ceiling](https://docs.jmrp.io/mikroscope/cost/rate-ceiling/): the runs with and without `FLOOR_HZ`, and
  why per-sample cost falls as the rate rises.
- [Device info](https://docs.jmrp.io/mikroscope/sinks/device-info/): where each source's cadence and
  reason reach a store.
- [Environment variables](https://docs.jmrp.io/mikroscope/reference/environment/): `FLOOR_HZ` beside the agent's
  other settings.
- [Resolution limits](https://docs.jmrp.io/mikroscope/limits/): the floor no setting can move.