# Test suites The four test suites, unit, end to end, real stores and the virtual RouterOS lab: what each proves, how to run it, what it needs and when CI runs it. Source: https://docs.jmrp.io/mikroscope/reference/testing/ Four suites test mikroscope, and none of them needs a router. Run the one that covers your change before you open a pull request. CI runs the first three on every pull request. The lab runs weekly and on dispatch, so run it yourself before a pull request that touches what installs the agent. | Suite | Command | Needs | What it proves | | -------------------- | ----------------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------ | | Unit | `go test ./cmd/... ./internal/...` | Go | each package, against captured `/proc` trees, fakes and golden files | | End to end | `make test-e2e` | Go | both binaries as separate processes, and the exact bytes each sink sends | | Stores | `make test-e2e-docker` | Linux, Docker | a real store accepts those bytes and hands them back; every dashboard panel's query answers | | Virtual RouterOS lab | `make lab-up`, then `make test-lab` | Linux, Docker, `/dev/kvm` for x86_64 | the deploy verbs against a real RouterOS, with the router's `/export` back to its start after each one | `make test` runs the unit and end-to-end suites together. The stores and lab suites sit behind the build tags `dockere2e` and `labe2e`, so `make test` compiles neither, and `make lint` type-checks both so that they cannot rot. When each suite first ran, how long it takes and what it found are on [Tested on](https://docs.jmrp.io/mikroscope/about/status/#the-test-suites). ## Choose a suite | You changed | Run | | ------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | Any Go code | `make test` | | The sampler, the ring, the stream or a sink | `make test-race` as well | | A test that might have grown a dependency on the network | `make test-e2e-offline` (Linux) | | A sink, its encoding or its schema, or a dashboard | `make test-e2e-docker` | | `internal/router`, a deploy verb, the agent image or how the agent starts | `make lab-up`, then `make test-lab` | | The lab's driver (`cmd/mikroscope-lab`, `internal/lab`) | `go test ./internal/lab/... ./cmd/mikroscope-lab/`, then `make test-lab` | The static checks a pull request owes as well (`make analyze`, the agent's size budget, the site's gates) are in [`CONTRIBUTING.md`](https://github.com/jmrplens/mikroscope/blob/main/CONTRIBUTING.md). ## Unit suite ```sh go test ./cmd/... ./internal/... # the unit suite alone make test # every untagged package, the end-to-end suite included make test-race # every suite under the race detector make cover-check # fails below COVERAGE_MIN over cmd/ and internal/ ``` Each package tests itself against fixtures, never against a router: - The `/proc` and `/sys` parsers read a captured tree of a real router, [`testdata/proc/rb5009/`](https://github.com/jmrplens/mikroscope/tree/main/testdata/proc/rb5009). - `plan`, the install script and the listings are compared with golden files in [`internal/router/testdata/`](https://github.com/jmrplens/mikroscope/tree/main/internal/router/testdata), one set per case of `cases.json`. `make check-generated` fails when the site's copy of those scripts is stale. - The lab's driver runs against a fake Docker and a fake router: provisioning, the lock, the downloads and their checksums, the snapshot chain, the firewall rules, QEMU's command line and the CLI's refusals. | Job | What it runs | When | | ------------------------------------- | ------------------------------------------------------------- | ---------------------------- | | Cross-platform, Linux, macOS, Windows | the unit suite; on macOS and Windows the end-to-end suite too | every pull request | | Coverage | `make cover-check` | every pull request | | Race detector | `make test-race` | weekly, and before a release | ## End-to-end suite `make test-e2e` builds both binaries and drives them as separate processes. The agent reads the captured tree. The collector reads that agent, or a fake one that serves canned samples, and writes every sink to a receiver inside the test binary, which asserts what arrived byte for byte. | Sink | Receiver | | --------------------------------------------- | -------------------------------------- | | InfluxDB, Loki, OTLP, Elasticsearch, Telegraf | an HTTP capture server | | Graphite, Telegraf's socket modes | TCP and UDP listeners | | File, SQL | real files | | stdout | the process's own standard output | | Prometheus | a scrape of the collector's `/metrics` | It needs no router, no Grafana, no database, no container and no network: every address it binds or dials is on loopback. `make test-e2e-offline` proves that on Linux by running the suite in a network namespace with nothing but `lo`. It needs `unshare` and `setpriv`, and runs `unshare` under `sudo` unless it already has the capability. CI runs the suite and its offline form on every pull request, weekly and before a release, and runs the suite on macOS and Windows as well, where the executable's name, the files the file and SQL sinks write and how a child process is stopped can differ. ## Stores suite A capture server answers 204 to anything; a store does not. `make test-e2e-docker` starts eight stores and a Grafana with docker compose, runs the same collector against the same fake agent with every sink pointed at them, and asks each store its own question through its own API. The file sink's JSONL is the oracle: each store is compared with it value by value, not only by count. | Store | The question it is asked | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | InfluxDB 3 Core | SQL over HTTP: the tables, a row count per table, every `ctxt` value | | PostgreSQL 18 | the SQL sink's script through `psql`, and the connecting PostgreSQL sink into a second database: counts, the `ctxt` range, and whether both hold the same rows | | Elasticsearch 9 | `_search` with an aggregation by kind, and one whole document | | Loki 3 | `query_range` for this run's labels, and the text of each record | | Graphite | `metrics/find` for the tree, `render` for the points and their order | | OpenTelemetry Collector | what it decoded, written back out as OTLP/JSON | | Telegraf 1.39 | the line protocol it parsed: measurements, tags, field types, timestamps | | Prometheus 3 | a scrape of the exporter, against the exposition it served | The suite also publishes the five datasources with `forward --grafana`, and empties every store again with `uninstall --targets data`. ### Failures only a store shows - **InfluxDB 3** fixes a column as a tag or a field the first time it sees the table, and refuses a later write that disagrees. - **Carbon** answers nothing at all: a point older than its longest archive, or a name whisper cannot make a path of, is dropped in silence. - **Loki** answers 204 to a push that is not queryable until the chunk flushes, and rejects out-of-order entries per stream, in the body. - **Elasticsearch** infers a mapping from the first document it sees for a field, then rejects a later one that does not fit it, per document, inside a bulk request that still answers 200. - **Telegraf** is a real line-protocol parser: an unescaped space in a tag value, or a field with no type, is dropped and the batch is still 204. - **PostgreSQL** is the only thing that can say whether the SQL sink's script is valid SQL, whether the types it chose hold the values it emits, and whether its primary keys collide on a real run. ### Dashboards The same run imports all five dashboards into Grafana, each pointed at the store the run just filled, and runs every panel's query through Grafana's own `/api/ds/query`: the `dashboards check` you can run against your own store. A panel fails there for reasons no unit test reaches: a type an aggregate returns that the datasource plugin cannot decode, a macro the plugin escapes, a column the store does not have. On InfluxDB 3 a missing column is not an empty one: the query fails with `Schema error: No field named `, and the panel cannot render at all ([found this way](https://docs.jmrp.io/mikroscope/about/status/#the-test-suites)). ### Run the stores suite ```sh make test-e2e-docker # the whole suite; the stack comes up and goes down with it make e2e-docker-up # keep the stack up between runs go test -v -tags dockere2e -run TestLoki ./test/e2e/docker/ make e2e-docker-down # stop the stack and delete its volumes make test-e2e-docker-race # the same suite under the race detector ``` The harness reuses a stack it did not start, and leaves it up afterwards. Why each store is a real one and not a fake is in [`test/e2e/docker/README.md`](https://github.com/jmrplens/mikroscope/blob/main/test/e2e/docker/README.md). CI runs the suite on every pull request, weekly, on dispatch and before a release. > **Linux, for two of the nine** > > Prometheus and Grafana run on the host's network stack, because Prometheus is > the one sink that is scraped rather than pushed to, and it has to reach the > collector's exporter on the host. A container on a bridge network reaches the > host through the bridge's gateway, and that path goes through the host's > INPUT chain, which a default-deny firewall drops. The other seven services > are ordinary bridge-network containers. ## Virtual RouterOS lab `doctor`, `plan`, `install`, `upgrade`, `status` and `uninstall` need a RouterOS to be tested at all. The lab is MikroTik's Cloud Hosted Router (CHR), a real RouterOS, under QEMU in a Docker container. It is provisioned once into a clean snapshot, with the `container` package installed and `device-mode container=yes` confirmed, and put back to that snapshot in seconds. The full account, how it is wired and what CHR taught the installer, is [`test/lab/README.md`](https://github.com/jmrplens/mikroscope/blob/main/test/lab/README.md). | Lab | Runs as | Use it for | | ------ | -------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | x86_64 | CHR under KVM | every scenario, fast; CI runs it weekly and on dispatch | | arm64 | CHR emulated by QEMU (TCG), CPU model `cortex-a72` | the arm64 `container` package and agent image, and RouterOS choosing the image for arm64; slow | A third lab, RouterOS x86 installed from MikroTik's ISO (`make lab-up LAB_KIND=iso`, x86_64 only), is an opt-in recipe that CI never runs. It adds the PC install path, an `x86` board name and the x86 licence (a 24-hour trial, then a Level 1 registration or a paid licence per device), and nothing the agent reads that CHR x86_64 does not. The CLI runs inside the lab's LAN. `mikroscope-lab cli` starts it in the lab container's network namespace, where 172.30.0.0/16 routes to the lab router, so the agent's default address, 172.30.10.2, reaches the lab's agent and nothing else. From your own shell the same address leaves by your default route, toward whatever network that is. So every deploy verb, by hand (`make lab-cli`) or in the suite, goes through `mikroscope-lab cli`, which refuses `--router` and a `--subnet` outside the lab's routes. The lab's namespace has its own firewall: it opens no new connection to a private address outside the lab, and takes none from another container. ### Lab suite `make test-lab` builds the CLI and the agent image tars, then runs the suite in [`test/e2e/lab/`](https://github.com/jmrplens/mikroscope/tree/main/test/e2e/lab) against the running lab, holding the lab's lock for the whole run. Every scenario starts from a reset and the profile it names, takes the router's `/export` there as its baseline, and ends by comparing the two. | Scenario | What it does | What it asserts | | -------- | ----------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | S1 | `doctor` on a stock CHR, with the lists `none` and with the defaults | nothing missing with `none`; with the defaults only the interface list `LAN`, whose fix offers `--iface-list none` | | S2 | install by pulling the image from Docker Hub, `status`, `upgrade`, `uninstall` | the agent answers `/healthz` and `/capabilities`; the export is back to its baseline | | S3 | install and upgrade from the branch's own image tar | the agent reports the branch build's version, commit and date | | S4 | `plan --rsc` for both image routes, run with `/import` | `status` recognises the script's objects; `uninstall` leaves the baseline | | S5 | every golden script the lab can run, imported as the site hands it over | the agent answers on its own /30; `uninstall` with the case's flags leaves the baseline | | S6 | `--ephemeral` on a tmpfs disk, then a power cut | the container is left stopped with its root gone; `uninstall --ephemeral` leaves nothing at all | | S7 | a persistent install, 45 s for it to reach the disk, then a power cut | the agent answers again within 90 s: start-on-boot works | | S8 | `--expose` with a token | `/capabilities` answers 401 without the token and 200 with it; both firewall rules go with uninstall | | S9 | `uninstall` while a client reads `/stream`, ten times | every first attempt verifies the router clean | | S10 | the raw rules of MikroTik's advanced firewall, in list and range form | `doctor` names the rule that drops the agent's replies, and the list memberships that pass | | S11 | a foreign veth, a routed /30, a foreign envlist | `doctor` names each; `install` writes nothing; the `plan --rsc` script stops at its guard for the veth and the envlist | | S12 | two installs side by side | removing one leaves the other running and untouched | | S13, S14 | installs with other lists, and with `--expose`, uninstalled with no flag | everything they created is removed; a contradicting flag is refused | | S17 | `doctor` on a lab below the minimum RouterOS | `RouterOS 7.24 or later` is missing, and every other check still reads | | S18 | a pull from GHCR with no registry credential | the agent answers | | F4 | every install route, an install made by a released CLI, objects the user made first | `/export` equals the one before the install and `/file` holds no mikroscope path; the user's objects stay | | repeat | tar installs and uninstalls in a row, ten on x86_64 and three on arm64 | no install fails | The scenarios assert the fixed behaviour: none runs a second uninstall, and none allows a mikroscope path left on `/file`. Four more tests hold what the CLI must keep doing: the agent token on no command line of the host, RouterOS's words in every error, the probe reading the `running` flag, and `doctor`'s warning for start-on-boot with the root on a tmpfs disk. S7 waits 45 s between the install and the power cut. A cut made as soon as the agent answers can bring back a container that cannot start ([seen in the lab](https://docs.jmrp.io/mikroscope/about/status/#virtual-lab)), which is a question of its own; S7 asks only whether start-on-boot works. Only the scenarios that test a pull make the router pull. Docker Hub limits anonymous pulls per address, and a CI runner shares its address with whatever else ran from it, so every other scenario installs the branch's own tar from `make agent-tars`, which also tests the branch's agent. The pulls are anonymous unless the lab is given an account ([Pull as an account](https://docs.jmrp.io/mikroscope/reference/testing/#pulling-as-an-account)), and the scenarios about a router with no credential, S1, S18 and S5's Docker Hub and GHCR scripts, boot without it in any case. The suite drops every `MIKROSCOPE_*` variable before it runs anything, so a shell set up for a real router cannot steer it. ### Run the lab ```sh make lab-up # start the lab; the first run downloads RouterOS and provisions it make lab-status # the container, who holds the lock, what the router reports make test-lab # the whole suite, x86_64 make test-lab LAB_RUN='S09' # the tests a go test -run pattern matches make lab-cli ARGS='doctor --arch amd64' make lab-reset # back to the clean snapshot make roundtrip # doctor, install, status, upgrade, uninstall; /export compared make lab-down # stop it; the disks keep their state make lab-up LAB_ARCH=arm64 # the emulated lab: every target takes LAB_ARCH=arm64 ``` S17 needs a lab below the minimum RouterOS, which is a lab of its own: ```sh make lab-up LAB_ROS=7.23.7 make test-lab LAB_ROS=7.23.7 LAB_RUN=S17 ``` The lab needs: - Linux, and Docker allowed to give a container `NET_ADMIN` and `/dev/net/tun`. - Go, for the lab's driver and for the CLI and agent under test. - About 910 MB of disk for both architectures, 810 MB for one. - The loopback ports 220N, 800N, 870N and 910N free, with N = 1 for x86_64 and 2 for arm64, plus an instance's offset. - Network access: Docker Hub and Debian's mirrors for the lab image the first time, `download.mikrotik.com` once per RouterOS version, and Docker Hub for what the router pulls. - `/dev/kvm` for x86_64. Without it, `LAB_KVM=auto` falls back to emulation, many times slower, and `LAB_KVM=require` stops instead. arm64 is emulated on an x86 host whatever you set. Without a running lab every test of the suite skips, and `MIKROSCOPE_LAB_REQUIRED=1` makes that a failure. The lab's admin password and agent token are generated into `test/lab/.env` on first use, gitignored, and never printed. Every target runs the lab's driver, `bin/mikroscope-lab`, which `make lab-tool` builds from [`cmd/mikroscope-lab/`](https://github.com/jmrplens/mikroscope/tree/main/cmd/mikroscope-lab) and [`internal/lab/`](https://github.com/jmrplens/mikroscope/tree/main/internal/lab): a build-time tool that nothing ships. `bin/mikroscope-lab help` lists its verbs, and `test/lab/lab.sh` only builds and starts it, for a command written for the old script. The same static binary is the lab container's first process, which sets up its network and runs QEMU until the router powers off. One driver drives a lab at a time: every verb takes the lab's lock, and `bin/mikroscope-lab lock ` holds it for a whole session, as `make test-lab` and `make roundtrip` do. The two labs run side by side, and so can their suites, but not as two `make test-lab` in one checkout, because each rebuilds the agent tars the other may be reading. Build once, and start each suite under its own lab's lock: ```sh make build agent-tars lab-tool LAB_ARCH=x86_64 bin/mikroscope-lab lock go test -tags labe2e -count=1 -timeout 150m ./test/e2e/lab/ & LAB_ARCH=arm64 bin/mikroscope-lab lock go test -tags labe2e -count=1 -timeout 150m ./test/e2e/lab/ ``` ### Pull as an account The lab router pulls the agent image itself, about a dozen times per run of the suite, and Docker Hub limits anonymous pulls per address. Given a Docker Hub account, the router pulls as that account instead: ```sh # in a file of your own, mode 0600, outside the repository LAB_REGISTRY_USER= LAB_REGISTRY_TOKEN= ``` ```sh set -a; . ~/.config/mikroscope/lab-registry.env; set +a make lab-reset # every up and reset gives the router the credential ``` The lab only pulls, so on your own machine a read-only token is enough. At every `up` and `reset` the lab's driver writes `/container/config/set registry-url=… username=… password=…` to a file, copies it to the router, runs it with `/import` and deletes it, so the credential is on no command line and the cached snapshot never has it. Exported from a file, as above, it stays off `make`'s command line too, which the process table would show. Without the two variables the router pulls anonymously; one without the other is refused. The suite prints the user and the token as `` and ``, and one of its tests reads the host's process table every 2 ms through a reset to hold the driver to all of it. `registry-url` is `registry-1.docker.io`, with no scheme (`LAB_REGISTRY_URL` names another registry's host). RouterOS presents the username and password for a reference whose host is `registry-url` as written, and mikroscope writes the host into every reference; with `https://` in front, the same pull goes out anonymously ([verified](https://docs.jmrp.io/mikroscope/about/status/#verified-registry-url-host-match)). ### Lab settings Each setting goes on the `make` line or in the environment. | Setting | Default | What it does | | ----------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------- | | `LAB_ARCH` | `x86_64` | which lab: `x86_64` or `arm64` | | `LAB_ROS` | the Makefile's | the RouterOS version to download and run; a new version is a new provision | | `LAB_KIND` | `chr` | `iso` is RouterOS x86 from MikroTik's ISO, x86_64 only | | `LAB_KVM` | `auto` | `require` stops without `/dev/kvm`, `off` never uses it, `auto` uses it when it is there | | `LAB_RUN` | every test | a `go test -run` pattern for `make test-lab`, such as `S09` or `F4` | | `LAB_INSTALL_REPEAT` | 10 on x86_64, 3 on arm64 | how many installs the repeat test makes | | `LAB_S5_CASES` | every case the lab can run | S5's cases, by id, separated by commas | | `LAB_REMOTE_IMAGE` | the last release on Docker Hub | the image the pull scenarios pull | | `LAB_GHCR_IMAGE` | the same image on GHCR | the image S18 pulls | | `LAB_STATE_DIR` | this checkout's `test/lab` | where the lab's `.cache/` and `.env` live: point a second checkout at a lab another one runs | | `LAB_INSTANCE` | none | a second lab of an architecture, with its own container, ports, lock and disks | | `LAB_LOCK_WAIT` | as long as it takes | seconds to wait for another driver's lock before failing and naming the holder | | `MIKROSCOPE_LAB_REQUIRED` | unset | `1` turns a skip for want of a lab into a failure | | `LAB_REGISTRY_USER`, `LAB_REGISTRY_TOKEN` | unset: the router pulls anonymously | a registry account the lab router pulls with, given to it at every `up` and reset, never in the snapshot | | `LAB_REGISTRY_URL` | `registry-1.docker.io` | the registry that account belongs to, written as the host the references name | ### Lab in CI The workflow [`.github/workflows/lab.yml`](https://github.com/jmrplens/mikroscope/blob/main/.github/workflows/lab.yml) runs the same `make lab-up` and `make test-lab`, with `MIKROSCOPE_LAB_REQUIRED=1`, for both architectures, weekly on main and on dispatch. It never runs on a pull request or before a release: a run takes half an hour or more. Run `make test-lab` yourself before a pull request that touches what installs the agent: `internal/router`, `internal/image`, `internal/agent`, `cmd/mikroscope`, `cmd/mikroscope-agent`, `Dockerfile.agent`, the `Makefile`, the agent tar and round-trip scripts, the RouterOS scripts the site renders, and the lab, its driver and its suite. A dispatch takes `ref`, `arch` (`both`, `x86_64` or `arm64`) and `ros`. arm64 runs under emulation, which is slow, and depends on MikroTik's download server and on Docker Hub being up. The Actions cache keeps MikroTik's downloads per architecture and RouterOS version, checked against the SHA-256 sums the repository pins in [`test/lab/SHA256SUMS`](https://github.com/jmrplens/mikroscope/blob/main/test/lab/SHA256SUMS), and the provisioned router, keyed on the driver, the lab image and those sums. The snapshot carries no credential: admin has CHR's empty password and no key until the run's `make lab-up` gives it that run's own, so the `.env` and the ssh key never leave the runner. A failed or timed-out run uploads the console log, the container log, the lab's status, what an install left on the router and the test log, each lab credential replaced by its name, and never the `.env` or the ssh key. The router pulls as an account when the repository has two secrets, `DOCKERHUB_USERNAME` and `DOCKERHUB_TOKEN`, the token the release also pushes images with; the lab only pulls with it. They become `LAB_REGISTRY_USER` and `LAB_REGISTRY_TOKEN` for the steps that bring the lab up, run the suite and redact the failure report, and no other; `ci.yml` and `release.yml` pass the two by name. A pull request from a fork gets no secrets, and a repository without `DOCKERHUB_TOKEN` gets neither variable: both run anonymously. So does a dispatch that checks out another `ref`, which may be a fork's merge commit: the lab builds and runs that code, and the token stays out of its environment. ### Lab limits - **Virtual, not hardware.** No RouterBOARD, no flash, no sensors, no switch chip and no device tree. `cpufreq`, `mtd`, `psi`, `schedstat` and `thermal` are absent from the agent's `/capabilities`, and `status` cannot map kernel port names to RouterOS ones. Anything about a board still needs a real one. - **The free CHR licence caps what the router sends** at 1 Mbit/s per interface. A rate test above 10 Hz in the lab measures the licence rather than the agent ([the arithmetic](https://docs.jmrp.io/mikroscope/about/status/#virtual-lab)). - **Emulated arm64 figures are not costs.** Under emulation the guest's clock follows the host's, so every duration, every CPU figure (`cpu-load`, `self.cpu_us`, `read_ns`, `wake_ns`, the `dt_ns` spread, `slipped`), every interrupt, softirq and context-switch rate and every PMU count from the arm64 lab measures the host's emulation, not the core it emulates. - **No 32-bit ARM.** MikroTik publishes CHR for x86_64 and arm64 only, so the armv5 and armv7 agent meets RouterOS only on hardware. `make agent-smoke` starts its image under QEMU user-mode, which is not RouterOS. ## Manual checks No suite shows: - **A sink fed from a router.** The samples in every suite come from a fake agent, a captured tree or the lab's CHR, and the stores run on the test host, not across a router's veth. Which sinks have carried a router's samples is under [Feature status](https://docs.jmrp.io/mikroscope/about/status/#what-works-end-to-end). - **Anything about a board**: its flash, sensors, switch chip and CPU frequency, and what the agent costs on it. [Agent cost](https://docs.jmrp.io/mikroscope/cost/#measuring-it-on-your-own-device) says how to measure it on yours. ## See also - [Tested on](https://docs.jmrp.io/mikroscope/about/status/#the-test-suites): when each suite first ran, how long it takes, what the lab found, and what has not been tested. - [Run the collector](https://docs.jmrp.io/mikroscope/sinks/): what each sink writes. - [Set up in Grafana](https://docs.jmrp.io/mikroscope/dashboards/import-and-check/): the same `dashboards check` the stores suite runs, against your own store.