# CLI

Every verb of the mikroscope CLI and its flags, grouped by verb, with each flag's default, accepted values and the MIKROSCOPE_* variable that sets it, read from the Go source.

Source: https://docs.jmrp.io/mikroscope/reference/cli/

`mikroscope` takes a verb and its flags. The tables below give each flag's default, the values it
accepts and the environment variable that sets it, as [`cmd/mikroscope/`](https://github.com/jmrplens/mikroscope/tree/main/cmd/mikroscope)
and [`internal/router/options.go`](https://github.com/jmrplens/mikroscope/blob/main/internal/router/options.go) define them.

## Usage and exit codes

```sh
mikroscope <verb> [flags]
```

The verbs fall into five groups, each with its own flag set: the deployment verbs (`doctor`,
`plan`, `install`, `upgrade`, `uninstall`, `status`, `image`), the recording verbs (`record`,
`mark`, `plot`), the collector (`forward`), `dashboards`, and `version`. Flags are Go `flag` flags:
`-rate 50` and `--rate 50` are the same, and a boolean is turned off with `-privileged=false`.

| Status | When                                                                                                                                                                                                                                                                                |
| ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `0`    | the verb finished                                                                                                                                                                                                                                                                   |
| `1`    | the verb ran and failed (a missing prerequisite, a router error, a sink that could not be built, a panel with no data, a store `dashboards publish` could not publish), the verb is unknown, or a flag of `uninstall`, `record`, `mark`, `plot`, `forward` or `dashboards` failed to parse or validate |
| `2`    | no verb was given, or a flag of `doctor`, `plan`, `install`, `upgrade`, `status` or `image` failed to parse or validate; nothing was sent to the router                                                                                                                             |

Every deployment and connection flag is validated before anything connects, by one function,
`FinishFor` in [`internal/router/options.go`](https://github.com/jmrplens/mikroscope/blob/main/internal/router/options.go), so a bad value stops the verb with nothing
sent to the router and names the value it refused:

```text
$ mikroscope install --rate 500
mikroscope: rate must be 1-100 Hz, got 500
```

That exits 2; `uninstall --rate 500`, `record --port 0` and `forward --port 0` exit 1. `--triggers`
goes through the agent's own `ParseTriggers`, so an unknown condition, a bad threshold, a quote or a
semicolon fails the verb the same way. The agent parses `TRIGGERS` again when it starts, which is
what catches an envlist edited by hand on the router: the container exits with status `2` after a
`mikroscope-agent: bad configuration:` line in the router log, and `restart-policy=on-failure`
restarts it.

### Defaults

- A flag that names a variable in the tables below reads its default from `MIKROSCOPE_<KEY>`. A
  variable set to the empty string counts as unset, and a flag on the command line always wins.
- **Only the flags that name a variable have one.** `--rate`, `--buffer`, `--port`,
  `--memory-max`, `--mem-limit-mb`, `--capture-mb`, `--triggers`, `--floor-hz`, `--privileged`,
  `--ephemeral`, `--expose`, `--restart-max-count`, `--restart-interval`, `--start-on-boot`,
  `--container-name` and `--extract-timeout` are flags only.
- `status`, `upgrade` and `uninstall` take the flags that describe an install's shape from the
  install on the router when they are not given ([Install
  shape](https://docs.jmrp.io/mikroscope/reference/cli/#pass-the-same-shape-to-status-upgrade-and-uninstall)).
- The CLI does not read `.env` itself; export it first with `set -a; . ./.env; set +a`.
  [Environment variables](https://docs.jmrp.io/mikroscope/reference/environment/) lists every variable, including the
  credentials that have no flag at all.

## Deployment commands

The seven deployment verbs share the [deployment flags](https://docs.jmrp.io/mikroscope/reference/cli/#flags-of-the-deployment-verbs); a flag a
verb has no use for is accepted, checked and ignored. The flags of each verb's own are these:

| Verb        | What it does                                                                                                                                                                                                  | Writes to the router | Its own flags                                        |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | ---------------------------------------------------- |
| `doctor`    | Read-only preflight in one ssh connect; each failing check names its fix. Then reads the running agent's ring for a loop, STP churn, link flaps and softnet drops.                                            | no                   | none                                                 |
| `plan`      | Prints every object `install` would create, then stops; connects to nothing. With `--rsc` it writes a RouterOS script instead.                                                                               | no                   | `--rsc`, `--out`                                     |
| `install`   | Runs `doctor`, gets the image for the router's architecture, prints the listing, asks, writes the install manifest and then every object, then probes the agent from this host.                              | yes                  | `--dry-run`, `--yes`, `--no-doctor`                  |
| `upgrade`   | Reads the install's shape, gets a new image, lists the manifest, the container step and any step the router no longer holds, asks, writes them, re-creates the container step, then probes.                 | yes                  | `--dry-run`, `--yes`                                 |
| `uninstall` | Reads the install's manifest, removes everything the install created, newest first and the manifest last, then verifies by count and by tag. Lists and removes nothing without `--yes`.                     | yes                  | `--yes`, `--targets`, the sink and Grafana flags     |
| `status`    | Reads the install's shape and prints the ownership count of every step; if anything is installed, probes the agent and prints its health and board.                                                         | no                   | none                                                 |
| `image`     | Builds the agent image tar and writes it to `--out`, for side-loading by hand; connects to nothing.                                                                                                         | no                   | `--out`                                              |

### `doctor`

```sh
mikroscope doctor
```

It runs these checks, in this order, and exits 1 with `N prerequisite(s) missing; nothing was
written` when a `MISSING` line is printed; a `WARN` line changes nothing:

The checks doctor runs:

| Check, as printed | Passes when | The fix it names |
| --- | --- | --- |
| RouterOS 7.24 or later | `/system/resource` reports a `version` of 7.24 or later, read with or without a patch number and whatever the channel, as in `7.24 (stable)` or `7.25rc1 (testing)` | upgrade RouterOS to 7.24 or later (`/system/package/update`), and the `container` package with it |
| architecture has a container package | the router's `architecture-name` is `arm`, `arm64` or `x86_64`, the architectures MikroTik publishes a `container` package for | none: no agent can run on this router |
| the router picks the image's architecture | with `--remote-image`: always, naming the router's architecture, because RouterOS picks it from the image's multi-architecture index. A warning on `arm`, where the index holds both `linux/arm/v5` and `linux/arm/v7` and which one RouterOS pulls is not known | on `arm`, if the container stops with `Exec format error`, install from `mikroscope-agent-armv5.tar` with `--agent-tar` |
| architecture matches the --agent-tar image | with `--agent-tar`: the tar's own architecture is the router's (`amd64` for `x86_64`); `--arch` is not needed | download the release asset it names, `mikroscope-agent-<arch>.tar` |
| architecture read from the router | with neither image flag and `--arch` unset (or `auto`): always; `install` and `upgrade` build or load the image for the architecture doctor read | none |
| architecture matches --arch <arch> | with an explicit `--arch` and neither image flag: the router's `architecture-name` is the one `--arch` maps to (`arm64`, `arm`, `x86_64`) | re-run with the `--arch` it names, or leave `--arch` out so that `install` reads it from the router |
| container package installed and enabled | a `container` package exists with `disabled=no` | download the `container` package for this architecture and RouterOS version, upload it and reboot; when it is there and disabled, `/system/package/enable container` and reboot |
| device-mode container=yes | `/system/device-mode` reports `container=yes` | `/system/device-mode/update container=yes`, then confirm it as the console asks: on a router that says `update: please activate by turning power off or pressing reset or mode button`, press the reset or mode button or cut the power; on CHR, which says `update: turn off power in 5m to activate changes`, power the VM off and on again within 5 minutes |
| free memory ≥ <--memory-max> | `free-memory` is at least what `--memory-max` asks for, 64 MiB by default | free memory on the router, or ask for less with `--memory-max` |
| free memory leaves room for the pull | a warning, with `--remote-image` only: `free-memory` is at least `--memory-max` plus 16 MiB, room for RouterOS to pull and extract the image before the agent starts. How much a pull takes is not measured, so the margin is an estimate | install from a tar with `--agent-tar` if the pull fails |
| free flash ≥ <size> (image tar + extracted root) | without `--disk` or `--ephemeral`: `free-hdd-space` is at least twice the image plus 4 MiB. With `--remote-image` nothing is uploaded and the name ends in `(extracted root)`: the root the pulled image is extracted into, 7 MiB, plus 4 MiB | free flash, or install with `--disk tmpfs` or `--ephemeral` where a tmpfs disk exists |
| disk <disk> exists | with `--disk` or `--ephemeral`: a disk with that slot exists | `/disk/add type=tmpfs tmpfs-max-size=64M slot=tmpfs` for a RAM disk, or name an existing disk with `--disk` |
| disk <disk> has ≥ <size> free (image tar + extracted root) | with `--disk` or `--ephemeral`, once the disk exists: its free space is at least twice the image plus 4 MiB; with `--remote-image`, the 7 MiB root plus 4 MiB, as the flash check | free space on that disk, or give a tmpfs disk a larger `tmpfs-max-size` |
| disk tmpfs is RAM | with `--ephemeral`: the disk in slot `tmpfs` is of type `tmpfs` | free the slot for a tmpfs disk, or install with `--disk <slot>` without `--ephemeral` |
| start-on-boot suits a root in RAM | a warning, when the disk is a tmpfs disk: start-on-boot resolves to `no`, since a reboot empties the disk and a container started at boot has no root | pass `--start-on-boot no`, or `--ephemeral` |
| veth name <veth> is free or ours | no veth has that name, or the one that has it carries this install's tag | pick another `--veth` (and `--subnet`), or remove the veth by hand if it is a leftover of yours |
| envlist <name>-env is free or ours | no envlist has that name, or the one that has it holds this install's `MIKROSCOPE_TAG` entry | pick another `--name` |
| install manifest <disk/>mikroscope/<name>.manifest.txt is free or ours | no file is at the install manifest's path, or the one there holds this install's `tag=` line | move the file away, or pick another `--name` |
| container name <name> is free or ours | with `--container-name`: no container has that name, or the one that has it carries this install's tag | pick another `--container-name` |
| subnet <subnet> does not overlap a route | no route of the main table, active or not, lies inside the /30, and no connected network on another interface holds its router end. Routes that only contain the /30 (a default route, a wider prefix to a VPN), blackhole routes and the install's own veth are left out | pick another /30 with `--subnet` |
| interface list <list> exists | the `--iface-list` list (default `LAN`) exists. With `--iface-list none` doctor prints `interface list the veth joins` and passes: no membership is written | `--iface-list none` when no firewall rule needs the veth in a list (doctor offers it first then); otherwise `/interface/list/add name=…`, or pass the list your `in-interface-list=!…` drop rule uses |
| address list <list> | always: install adds the /30 to the `--addr-list` list (default `LANs`), which creates it when it is missing, and uninstall removes the entry. With `--addr-list none` doctor prints `address list the /30 joins`. Whether a rule needs the membership is the next row's question | none |
| no firewall rule drops the agent's replies | doctor reads every enabled rule of the chains the agent's replies meet, `/ip/firewall/raw` prerouting and `/ip/firewall/filter` forward and input, and walks each one as RouterOS does, first match wins, with the replies in the lists the plan joins: no rule drops them. A warning when a rule might, because it matches on something doctor does not judge (a destination, a mark, a rate), and when the replies to a LAN host pass but a rule may drop the ones to the router itself (filter input), which the relay transport needs | the `--iface-list` and `--addr-list` that let the replies through; when no list does (a `src-address=!<range>` rule, say), add an accept rule for `in-interface=<veth>` before that rule, or pick a `--subnet` inside the range |
| no firewall rule doctor reads is invalid or names a deleted list | a warning: no rule of raw prerouting or filter forward and input is marked invalid, which RouterOS passes over as if it were not there (it names an interface that was removed or is not ready, and `about` says which), and none names an interface list that was removed, which RouterOS keeps as the list's id (`in-interface-list=!*2000010`) and matches as an empty list. A rule changed a moment before reads invalid with no reason until RouterOS has applied it | fix or remove what an invalid rule names; set a deleted list again by name, since creating a list of the same name does not repair the rule. With no reason given, run `doctor` again |
| --lan-address <address> is the router's | with `--expose`: an interface of the router holds that address | pass the address the router has on its LAN, as `/ip/address/print` lists it |
| --lan-address is not on the uplink | a warning, with `--expose`: the interface that holds the address carries no default route, is in no `WAN` list, and shares no interface list with the interface that carries the default route | pass the router's LAN address: on the uplink the dst-nat would publish the agent on the Internet side |
| no registry credential meant for another registry | a warning, with `--remote-image` only: no `/container/config` username is set, or the host of `registry-url` is the host the image is pulled from, every spelling of Docker Hub counted as one. An empty `registry-url` with a username set warns. Doctor reads whether a username is set, never the name, and cannot read the password | `/container/config` holds one username for the whole device, and a credential from another registry can make the pull of a public image end in `auth error`. Install from a tar with `--agent-tar`, pass a `--remote-image` on the registry the username belongs to, or clear the username if nothing else needs it |
| the installed agent published on the LAN asks for a token | a warning, shown only when an install of this `--name` has a dst-nat on the LAN: its environment holds a `TOKEN`. Doctor counts the entries, never reads the value | `upgrade` with the same `--name` and `--token <secret>`; or remove the agent, the LAN rules and the container together with `uninstall --name <name> --yes` |
| the router does not answer DNS from its uplink | a warning: `/ip/dns` `allow-remote-requests` is off, or a rule of raw prerouting or filter input drops a UDP query to port 53 that comes in on the interface of the active default route, walked as for the trap check, first match wins. A warning that the queries may pass when a rule matches on something doctor does not judge, a source address list say. IPv6 is not read | a drop rule for UDP and TCP port 53 on the uplink before any accept that takes it, or `/ip/dns/set allow-remote-requests=no` if no LAN host uses the router as its resolver |
| nothing tagged for <name> that these flags do not select | a warning: in every menu an install writes to, the objects that carry the install's tag are no more than the plan for these flags selects. An install made with other flags (`--expose`, other lists, another `--subnet`) leaves more | run `status` and `uninstall` with no shape flag, so that they read the install manifest, or with the flags that install was given |

Standalone `doctor`, but not the doctor that `install` runs, then pulls the agent's ring once over
HTTP from the `--subnet` .2 address on `--port`, sending `--token`, and gives up after 3 s. Pass the
same `--subnet`, `--port` and `--token` the agent was installed with. The findings are defined in
[Requirements](https://docs.jmrp.io/mikroscope/install/prerequisites/#what-the-running-agent-shows) and never change the
exit status.

### `plan`

```sh
mikroscope plan
mikroscope plan --rsc --remote-image jmrplens/mikroscope-agent:1.6.1 --out mikroscope.rsc
```

- Without `--rsc` it prints the listing `install` would print, the same as `install --dry-run`,
  with the token masked as `(token)`.
- With `--rsc` it writes the whole install as a RouterOS script, to `--out` or to standard output.
  With `--out` it prints `<out>: N lines; review it, then paste it into the router's terminal or
  /import it`. Paste it only at the `] >` prompt: a script pasted while RouterOS is asking about the
  licence loses its first lines ([RouterOS script](https://docs.jmrp.io/mikroscope/install/script/#run-on-the-router)). With a token set, the script carries it in clear ([Installer
  safeguards](https://docs.jmrp.io/mikroscope/security/installer/#a-generated-rsc-script-is-a-credential)).
- `--arch auto` is `arm64` here, since nothing is asked of the router.

### `install`

```sh
mikroscope install --remote-image jmrplens/mikroscope-agent:1.6.1
```

1. `doctor`, in one connect, unless `--no-doctor`. A missing prerequisite stops it with nothing
   written.
2. With `--arch auto`, the architecture doctor read; `--no-doctor` without `--remote-image` makes
   one more connect to read it, and says so.
3. The image: built, read from `--agent-tar`, or none with `--remote-image`.
4. The listing, then `write the objects above to the router? [y/N]` unless `--yes`. Anything but
   `y` stops it with `not confirmed; nothing written`.
5. The install manifest, then every object, and `install done: N step(s) created`.
6. The probe from this host ([Network access](https://docs.jmrp.io/mikroscope/install/reaching-the-agent/#what-the-probe-after-install-tells-you)).

`--dry-run` stops after the listing and opens no connection.

### `upgrade`

```sh
mikroscope upgrade --remote-image jmrplens/mikroscope-agent:1.6.1
```

- One connect reads how the install was made, whether each step is there, the router's architecture
  and, with `--remote-image`, doctor's registry-credential check. `upgrade` runs no `doctor`.
- It refuses a router with no install (`nothing to upgrade: run install first`), and an install
  with a token when none is given.
- It lists the install manifest, the container step and any step the router no longer holds, asks,
  writes the manifest and the missing steps, removes and re-creates the container step, then
  probes. The network objects stay.
- The envlist is written from the flags given to `upgrade`: a tuning flag left out comes back at its
  default ([Install shape](https://docs.jmrp.io/mikroscope/reference/cli/#pass-the-same-shape-to-status-upgrade-and-uninstall)).
- `--dry-run` stops after the listing and opens no connection.

### `uninstall`

```sh
mikroscope uninstall                                             # lists the router objects
mikroscope uninstall --yes                                       # and removes them
mikroscope uninstall --targets all --influx … --grafana …        # lists everything
mikroscope uninstall --targets all --influx … --grafana … --yes  # and removes it
```

Without `--yes` it lists: with `--router` it reads the install's shape first, and without it lists
from the flags alone. `--targets` widens it past the router:

| Target      | What goes                                                                                                                                                                                                                        |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `router`    | everything `install` created: the objects in the install manifest, any other object with the install's tag, the container root, the manifest, and the `mikroscope` directory when nothing else is in it. The default            |
| `dashboard` | per store the sink flags name: the dashboard `mikroscope-<store>`, however it was imported, and the datasource `mikroscope-<store>`. Not the folder, an adopted datasource or provisioned alert rules |
| `data`      | the tables and indices the sinks wrote, and the files the file sinks wrote                                                                                                                                                       |
| `all`       | the three above                                                                                                                                                                                                                  |

The `dashboard` and `data` targets take the [sink](https://docs.jmrp.io/mikroscope/reference/cli/#sinks) and [Grafana](https://docs.jmrp.io/mikroscope/reference/cli/#publishing-to-grafana)
flags of `forward`, to find the stores. Of the Grafana flags, `uninstall` reads `--grafana` and
`--grafana-datasource-uid`, ignores the others, and refuses `--grafana-dry-run`: its dry run is
leaving out `--yes`. When a sink flag names a store, `dashboard` also needs `--grafana` (or
`MIKROSCOPE_GRAFANA_URL`, then `GRAFANA_URL`) and `GRAFANA_TOKEN`, and stops with `--targets dashboard needs --grafana …` without
a Grafana and `--targets dashboard needs GRAFANA_TOKEN` without the token. Given no sink flag,
`--targets dashboard` or `data` finds no store and prints `nothing of this is here to remove`; `all`
then acts on the router objects alone.

- **Nothing is removed without `--yes`**, the same `--yes` the deployment verbs take. A dropped
  table cannot be put back the way `install` puts back a router object.
- **The stores are listed before the router is touched.** The `dashboard` and `data` targets are
  listed first, so one that cannot be listed stops the verb before anything is removed, the router
  objects of `all` included.
- **A Grafana it cannot read stops it.** Only a 404 counts as not there. An unreachable Grafana, a
  refused token or a server error stops the verb with `asking Grafana whether dashboard
  mikroscope-<store> is there: …` and exit status 1, before anything is removed.
- **The tables are asked of the store, never compiled in.** A list inside the binary would be the
  measurements this version writes, and the ones worth removing are the ones nobody writes any
  more: what an earlier version collected, or a source switched off since. Everything under the
  `mikroscope_` prefix is claimed; nothing else is touched.
- **An adopted datasource is not removed.** One named in `--grafana-datasource-uid` was somebody
  else's before this ran and is somebody else's after.
- **One that will not go does not stop the rest.** Each failure is printed with what the store said
  and the removal carries on.

> **Three sinks store nothing this can remove**
>
> `--prom` is scraped rather than written to, so the series live in a Prometheus this has never
> heard of and age out with its retention. `--graphite` offers no delete at all: remove the whisper
> files by hand. `--sql` writes a file, and rows already loaded into a real database from it have to
> be removed there, or with `--postgres` pointed at that database. Each of them prints its own
> reason rather than being silently absent.

> **InfluxDB 3 deletes a table by renaming it**
>
> A table deleted through `/api/v3/configure/table` stays in `information_schema` under a name
> carrying the deletion instant, `mikroscope_cpu-<timestamp>`. Those are gone and are not offered
> again; without that filter an uninstall would list the same tables on every run, delete them
> successfully, and never empty. The delete asks for `hard_delete_at=now`, which the server accepts
> and schedules.

### `status`

```sh
mikroscope status
```

It prints what it read of the install, the ownership count of each step and, when anything is
installed, the agent's health, read directly with a 3 s timeout:

```text
install on the router (manifest mikroscope/mikroscope.manifest.txt): veth veth-mikroscope, subnet 172.30.10.0/30, lists LAN/LANs, port 9123, …
  1     install manifest mikroscope/mikroscope.manifest.txt
  1     veth interface veth-mikroscope
  …
agent: 1.6.1 (<commit>) built <time>, 10 Hz, seq 19 (oldest 1), up 2s, 0 slipped, 1ms round trip
```

### `image`

```sh
mikroscope image --arch arm64 --out mikroscope-agent-arm64.tar
```

It builds the agent image tar for `--arch`, `arm64` with `auto`, and writes it to `--out`, default
`mikroscope-agent-<arch>.tar`. It refuses `--remote-image`, because there is no tar to write.

### Install shape

`status`, `upgrade` and `uninstall` read how the install on the router was made before they build
their plan: from its install manifest, `mikroscope/<name>.manifest.txt` on its disk, or, for an
install without one, from the objects that carry its tag.

- A shape flag not given takes the value the install was made with: `--veth`, `--subnet`,
  `--iface-list`, `--addr-list`, `--disk` (and `--ephemeral`), `--port`, `--expose` with
  `--lan-address`, `--container-name` and `--start-on-boot`, and for `status` and `uninstall`
  `--remote-image`. `uninstall --yes` with no other flag removes an install made with any of them.
- Without a manifest, a list membership the router does not hold is not read as `none`: the flag,
  or its default, decides, and `upgrade` makes the membership again.
- A flag given, or set by its `MIKROSCOPE_*` variable, that contradicts the install is refused
  before anything is written, naming both values: `the install named mikroscope on the router does
  not match the flags: installed with --iface-list MYLAN, given LAN`. That holds for `--veth`,
  `--subnet`, `--iface-list`, `--addr-list`, `--disk` and `--ephemeral`, `--port`, and `--expose`
  with `--lan-address`. `--container-name`, `--start-on-boot` and `--remote-image` are filled in
  when not given and never refused: a flag given wins, and `upgrade` takes its image from its own
  flags only.
- `--name` selects the install, and is never read from the router.
- `upgrade` writes the envlist again from its own flags: an `upgrade` without the `--rate`,
  `--buffer`, `--mem-limit-mb`, `--capture-mb`, `--triggers` or `--floor-hz` you installed with
  writes the defaults in their place, and `--memory-max`, `--privileged`, `--restart-max-count`
  and `--restart-interval` likewise go back to theirs. It refuses an install whose envlist holds a
  `TOKEN` when no `--token` is given: `the install named mikroscope asks for a token and upgrade
  would write its envlist without one: pass --token (or MIKROSCOPE_TOKEN)`.

### Router writes

**What `install` writes to your router**

- the install manifest, a file `mikroscope/<name>.manifest.txt` on the install's disk that lists the options and every object below
- a veth
- one address
- one interface-list membership, unless `--iface-list none`
- one address-list entry, unless `--addr-list none`
- an envlist
- the image tar, deleted once the container is extracted, unless `--remote-image` has the router pull the image
- the container, and its root `mikroscope/<name>` on the same disk

Every object carries the comment `mikroscope:<name> (managed by mikroscope)`

`mikroscope plan` prints every command before anything is written.

`uninstall` removes by exact tag plus identity, never by pattern, and fails naming the step if anything remains.

**What `install --expose` adds**

- two firewall rules, tagged
- a token becomes mandatory
- `uninstall` and `status` find the two rules through the manifest and the tag, with or without `--expose`

Every object carries the comment `mikroscope:<name> (managed by mikroscope)`

`mikroscope plan` prints every command before anything is written.

**What `upgrade` replaces**

- the install manifest, written first every time, so an install made before there was one gets it
- a new image and the container
- the envlist, rewritten from the flags `upgrade` is given; it refuses an install with a token when no `--token` is given
- any other object of the install the router no longer holds, created again before the container
- network objects stay

Every object carries the comment `mikroscope:<name> (managed by mikroscope)`

`mikroscope plan` prints every command before anything is written.

**What `uninstall` removes**

- every object in the install's manifest, and any other object that carries its tag
- the container root `mikroscope/<name>`, with the container, or on the manifest's word when a root is left
- the manifest, last, and then the `mikroscope` directory when nothing else is in it
- never device-mode, the `container` package, `/container/config`, or a list, disk or rule the router had before

Every object carries the comment `mikroscope:<name> (managed by mikroscope)`

`mikroscope plan` prints every command before anything is written.

`uninstall` removes by exact tag plus identity, never by pattern, and fails naming the step if anything remains.

## Deployment flags

`doctor`, `plan`, `install`, `upgrade`, `uninstall`, `status` and `image` take every flag below.

### Connection

| Flag           | Default                     | Variable                                  | Accepted                                                                                                                                                                                                                       | Meaning                                                                                                                                                                                                                         |
| -------------- | --------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--router`     | none, required              | `MIKROSCOPE_ROUTER`                       | `user@host` or an ssh config alias                                                                                                                                                                                             | ssh target; every verb but `plan`, `install --dry-run`, `upgrade --dry-run`, `image` and an `uninstall` without `--yes` fails without it                                                                                        |
| `--ssh-port`   | empty (ssh config)          | `MIKROSCOPE_SSH_PORT`                     |                                                                                                                                                                                                                                | ssh port                                                                                                                                                                                                                        |
| `--ssh-key`    | empty (agent or ssh config) | `MIKROSCOPE_SSH_KEY`                      |                                                                                                                                                                                                                                | ssh identity file                                                                                                                                                                                                               |
| `--ssh-option` | none                        | `MIKROSCOPE_SSH_OPTIONS`, comma-separated | `Key=value`, repeatable; the key one of `StrictHostKeyChecking`, `UserKnownHostsFile`, `ConnectTimeout`, `HostKeyAlgorithms`, `PubkeyAcceptedAlgorithms`, `IdentitiesOnly`, `ServerAliveInterval`; the value `^[A-Za-z0-9_./~+:-]{1,256}$` | an ssh and scp option, placed before the CLI's own `-o BatchMode=yes -o ConnectTimeout=15`; the first flag replaces the variable's whole list. A router not yet in `known_hosts` needs `StrictHostKeyChecking=accept-new`, because the CLI runs ssh in batch mode |

### Router objects

| Flag            | Default                | Variable                 | Accepted                                                                            | Meaning                                                                                                                                                                  |
| --------------- | ---------------------- | ------------------------ | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--name`        | `mikroscope`           | `MIKROSCOPE_NAME`        | `^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$`                                                 | the install's name; tags every object as `mikroscope:<name> (managed by mikroscope)` and names the envlist, the root and the manifest                                    |
| `--veth`        | `veth-mikroscope`      | `MIKROSCOPE_VETH`        | `^[A-Za-z0-9][A-Za-z0-9_.-]{0,63}$`                                                 | veth interface name on the router                                                                                                                                        |
| `--subnet`      | `172.30.10.0/30`       | `MIKROSCOPE_SUBNET`      | an IPv4 /30 at its network address                                                  | the router takes `.1`, the agent `.2`                                                                                                                                    |
| `--port`        | `9123`                 | none                     | 1–65535                                                                             | agent HTTP port on the veth                                                                                                                                              |
| `--iface-list`  | `LAN`                  | `MIKROSCOPE_IFACE_LIST`  | same pattern as `--veth`, or `none`; not `all`, `dynamic` or `static`               | interface list the veth joins; `none` writes no membership ([Firewall lists](https://docs.jmrp.io/mikroscope/install/firewall/))                                                             |
| `--addr-list`   | `LANs`                 | `MIKROSCOPE_ADDR_LIST`   | same pattern as `--veth`, or `none`                                                 | address list the /30 joins, created by the entry when the list has none; `none` writes no entry                                                                          |
| `--disk`        | empty (internal flash) | `MIKROSCOPE_DISK`        | `^[A-Za-z0-9][A-Za-z0-9_-]{0,31}$`                                                  | RouterOS disk for the manifest, the image tar and the root: `tmpfs`, `disk1`, `usb1` …                                                                                  |
| `--ephemeral`   | `false`                | none                     |                                                                                     | forces `--disk tmpfs`, and `start-on-boot=no` unless `--start-on-boot` says otherwise: nothing written to flash, nothing survives a reboot                                |
| `--expose`      | `false`                | none                     | needs `--lan-address`; `install`, `upgrade` and `plan` also need `--token`          | dst-nat the agent port on the router's LAN address; adds two tagged firewall rules ([Expose on the LAN](https://docs.jmrp.io/mikroscope/security/expose/))                                    |
| `--lan-address` | empty                  | `MIKROSCOPE_LAN_ADDRESS` | an IPv4 address                                                                     | the router's LAN address for `--expose`                                                                                                                                  |

### Agent image

| Flag             | Default                      | Variable                  | Accepted                                                        | Meaning                                                                                                                                                                                                                                                                                                                            |
| ---------------- | ---------------------------- | ------------------------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--arch`         | `auto`                       | `MIKROSCOPE_ARCH`         | `arm64`, `arm`, `amd64` or `auto`                               | device architecture, used as `GOARCH` and in the image manifest. `auto` reads it from the router: `doctor`, `install` and `upgrade` take it from doctor's batch (`install --no-doctor` makes one connect for it, and none with `--remote-image`, where the router picks the architecture from the image's index); `plan`, `--dry-run` and `image`, which connect to nothing, take `arm64` |
| `--goarm`        | `5`                          | none                      | `5`, `6` or `7`                                                 | `GOARM` level, used only with `--arch arm`: 5 runs on every 32-bit ARM MikroTik ships, 7 does not run on EN7562CT boards ([Requirements](https://docs.jmrp.io/mikroscope/install/prerequisites/) lists them)                                                                                                                                          |
| `--agent-tar`    | empty (build the agent here) | `MIKROSCOPE_AGENT_TAR`    | a path to an agent image tar                                    | `plan`, `install`, `upgrade`, `image`: upload this tar instead of building one, so neither a Go toolchain nor a checkout is needed. The tar is checked first: one that is not a mikroscope agent image, or is built for another architecture than the router's or `--arch`, fails the verb naming the asset to download         |
| `--remote-image` | empty (upload a tar)         | `MIKROSCOPE_REMOTE_IMAGE` | a registry reference, `owner/name:tag` or `host/owner/name:tag` | `plan`, `install`, `upgrade`: the router pulls the image itself, so nothing is built or uploaded and no tar lands on the device. The whole reference goes into `remote-image=`, registry host included: none, `docker.io`, `index.docker.io` or `registry.hub.docker.com` becomes `registry-1.docker.io`, any other host is kept. `/container/config registry-url` is neither needed nor written, and the published image needs no registry login |

> **Three ways to get the agent image, one flag apart**
>
> With neither `--agent-tar` nor `--remote-image`, `plan`, `install`, `upgrade` and `image` run
> `go build -trimpath ./cmd/mikroscope-agent` with `CGO_ENABLED=0`, `GOOS=linux` and the
> architecture as `GOARCH` ([`internal/image/image.go`](https://github.com/jmrplens/mikroscope/blob/main/internal/image/image.go)), which needs a checkout of the
> repository and a Go toolchain on `PATH`; without one the build fails naming both flags and the Go
> version, before anything is listed. With `--agent-tar` they read and check the published tar. With
> `--remote-image` nothing is built or uploaded. `plan --rsc` gets no image on this machine at all:
> its script needs `--remote-image` or a tar already on the router. [Install
> methods](https://docs.jmrp.io/mikroscope/install/routes/) sets the routes side by side.

### Agent settings

| Flag             | Default                         | Variable           | Accepted                                                   | Meaning                                                                                                                                                                                                                                                                                        |
| ---------------- | ------------------------------- | ------------------ | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--rate`         | `10`                            | none               | 1–100                                                      | sampler rate in Hz (envlist `RATE_HZ`); 10, 50 and 100 Hz are lossless ([Rate ceiling](https://docs.jmrp.io/mikroscope/cost/rate-ceiling/))                                                                                                                                                                        |
| `--buffer`       | `60`                            | none               | 10–3600                                                    | ring buffer in seconds (envlist `BUFFER_S`)                                                                                                                                                                                                                                                    |
| `--mem-limit-mb` | `0` (derived)                   | none               | `0`, or 8–1024                                             | agent Go soft memory limit in MiB (envlist `MEM_LIMIT_MB`). `0` derives it from the ring: rate × buffer × 3 456 B × 2.5, rounded up, at least 16 MiB and at most ¾ of `--memory-max` while that still leaves room for the ring. That gives 16 at the defaults. A number you pass is used as is |
| `--capture-mb`   | `4`                             | none               | 0–256                                                      | triggered-capture budget in MiB (envlist `CAPTURE_MB`); `0` turns captures off                                                                                                                                                                                                                 |
| `--triggers`     | empty (the agent's default set) | none               | see [Triggered capture](https://docs.jmrp.io/mikroscope/record/triggers/)      | trigger conditions, comma-separated (envlist `TRIGGERS`)                                                                                                                                                                                                                                       |
| `--floor-hz`     | `0`                             | none               | 0–1000                                                     | one cadence for every level source, in Hz (envlist `FLOOR_HZ`); `0` keeps the per-source floors; equal to `--rate` reads and emits every source every tick                                                                                                                                     |
| `--token`        | empty                           | `MIKROSCOPE_TOKEN` | `^[A-Za-z0-9_.-]{0,128}$`                                  | bearer token the agent requires (envlist `TOKEN`); `install`, `upgrade` and `plan` need it with `--expose`, and `upgrade` refuses an install that has one when none is given; `doctor` also sends it to read the agent's ring. It reaches ssh on standard input, never on a command line      |

### Container settings

| Flag                  | Default                   | Variable | Accepted                              | Meaning                                                                                                                                                                                                                                    |
| --------------------- | ------------------------- | -------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--container-name`    | empty (RouterOS names it) | none     | `^[A-Za-z0-9][A-Za-z0-9_.-]{0,31}$`   | the container's `name=`; `doctor` checks that no other container holds it. Removals select by the tag, not by this name                                                                                                                    |
| `--memory-max`        | `64M`                     | none     | `^\d{1,6}[KMG]?$`                     | container cgroup `memory-max`, RouterOS syntax                                                                                                                                                                                             |
| `--privileged`        | `true`                    | none     |                                       | runs the container `privileged=yes`; `-privileged=false` opts out ([Privileged mode](https://docs.jmrp.io/mikroscope/limits/privileged/))                                                                                                                       |
| `--start-on-boot`     | `auto`                    | none     | `auto`, `yes` or `no`                 | `start-on-boot` of the container; `auto` is `no` with `--ephemeral` and `yes` without. `doctor` warns on `yes` with the root on a tmpfs disk                                                                                               |
| `--restart-max-count` | `5`                       | none     | 0–100                                 | `restart-max-count` of the container's `restart-policy=on-failure`                                                                                                                                                                        |
| `--restart-interval`  | `10s`                     | none     | `^\d{1,4}[smh]$`                      | `restart-interval` between those restarts                                                                                                                                                                                                  |
| `--extract-timeout`   | `120s`                    | none     | `^\d{1,4}[smh]$`, 10–600 s            | tar route: how long the container step waits for RouterOS to extract the image before it deletes the tar; on timeout it stops and says so, and the tar stays. The ssh deadline for one command, 3 minutes, grows to this and one minute more when that is longer |

Two settings of the container are not flags: `logging=yes` and `ignore-remote-image-change=yes`.
[Storage and container settings](https://docs.jmrp.io/mikroscope/install/layout/#the-container-settings) has all of
them.

### Run control

| Flag          | Default                       | Variable | Meaning                                                                                                                                          |
| ------------- | ----------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `--dry-run`   | `false`                       | none     | `install` and `upgrade`: print the listing and write nothing; no connection is opened                                                            |
| `--yes`       | `false`                       | none     | `install`, `upgrade`: do not ask before writing. `uninstall`: remove what it lists, which it does not do without this flag                        |
| `--no-doctor` | `false`                       | none     | `install`: skip the preflight checks; with `--arch auto` and no `--remote-image`, one connect reads the router's architecture                     |
| `--out`       | `mikroscope-agent-<arch>.tar` | none     | `image`: output path of the tar. `plan --rsc`: where the script is written; empty writes it to standard output                                   |
| `--rsc`       | `false`                       | none     | `plan`: write a RouterOS script that installs from the router itself, instead of the listing                                                     |

## record, mark and plot

The three verbs share one flag set with `forward`, so a flag that means something to a sibling is
accepted and ignored: `plot --for 5m` parses and does nothing. The table marks which verb reads each
flag. `--from-start` is the exception: only `record` registers it, and `forward`, `mark` and `plot`
refuse it as an unknown flag. `forward` always starts at the agent's newest sample
([`internal/forward/`](https://github.com/jmrplens/mikroscope/tree/main/internal/forward)).

| Flag            | Default                      | Variable              | Read by                     | Meaning                                                                                                                                                                                                                                                                                              |
| --------------- | ---------------------------- | --------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--out`         | `capture-<UTC time>`         | none                  | `record`, `mark`            | output prefix: `<out>.jsonl`, `.csv`, `.markers.csv`, `.meta.json`. `mark` needs the prefix of an existing recording                                                                                                                                                                                 |
| `--for`         | `0` (until Ctrl-C)           | none                  | `record`, `forward`         | run this long, then stop                                                                                                                                                                                                                                                                             |
| `--from-start`  | `false`                      | none                  | `record`                    | backfill everything the agent's ring holds before going live                                                                                                                                                                                                                                         |
| `--poll`        | `500ms`                      | none                  | `record`, `forward`         | how often the agent's ring is pulled                                                                                                                                                                                                                                                                 |
| `--batch`       | `0`                          | none                  | `record`, `forward`         | samples per pull; `0` is twice what one `--poll` produces at the agent's rate, never under 20. The relay caps a pull at 13. A pull repeats until it comes back short                                                                                                 |
| `--transport`   | `auto`                       | none                  | `record`, `forward`         | `auto` (direct, then relay), `direct` (HTTP to the veth) or `relay` (`/tool fetch` over the RouterOS API; each call returns at most 64 512 B and takes either about 3 ms or about 1 s; see [Network access](https://docs.jmrp.io/mikroscope/install/reaching-the-agent/)) |
| `--log-markers` | `false`                      | none                  | `record`, `mark`            | `record`: after recording, pull the router log over the API and append the matching lines to `<out>.markers.csv`. `mark`: add the log lines of the recording's window                                                                                                                                |
| `--topics`      | `system,interface,container` | none                  | `record`, `mark`            | log topics kept as markers with `--log-markers`                                                                                                                                                                                                                                                      |
| `--router-tz`   | `Local`                      | none                  | `record`, `mark`            | IANA zone the router's clock shows; RouterOS log times carry no zone                                                                                                                                                                                                                                 |
| `--in`          | none, required               | none                  | `plot`                      | recording prefix, or its `.jsonl` path                                                                                                                                                                                                                                                               |
| `--svg`         | `<in>.svg`                   | none                  | `plot`                      | output SVG                                                                                                                                                                                                                                                                                           |
| `--title`       | the prefix                   | none                  | `plot`                      | chart title                                                                                                                                                                                                                                                                                          |
| `--api`         | empty                        | `MIKROSCOPE_API_ADDR` | `record`, `mark`, `forward` | RouterOS API `host:port`, for the relay, `--log-markers` and the API tier                                                                                                                                                                                                                            |
| `--api-user`    | empty                        | `MIKROSCOPE_API_USER` | `record`, `mark`, `forward` | API user; its password comes only from `MIKROSCOPE_API_PASSWORD`                                                                                                                                                                                                                                     |
| `--token`       | empty                        | `MIKROSCOPE_TOKEN`    | `record`, `forward`         | bearer token the direct transport sends to the agent                                                                                                                                                                                                                                                 |
| `--port`        | `9123`                       | none                  | `record`, `forward`         | agent HTTP port                                                                                                                                                                                                                                                                                      |
| `--subnet`      | `172.30.10.0/30`             | `MIKROSCOPE_SUBNET`   | `record`, `forward`         | the agent's /30; its address is `.2`                                                                                                                                                                                                                                                                 |

`mark` takes the marker's text as its remaining arguments,
`mikroscope mark --out cap "queue tree applied"`, or `--log-markers` instead of text. `record` turns
every line typed on a terminal into a marker; when standard input is not a terminal it does not read
it.

`auto` tries `/healthz` over the direct transport first. If that does not answer it needs `--api`,
`--api-user` and `MIKROSCOPE_API_PASSWORD` to try the relay. It fails naming `install --expose` when
the direct transport does not answer and the relay is not configured; a configured relay that fails
reports `relay transport:` and its error. The relay does not carry the token: `/tool fetch` on the
router sends no `Authorization` header, so an agent with a token set can only be pulled over the
direct transport.

## forward

`forward` runs the collector: it pulls the kernel tier from the agent, polls the RouterOS API tier,
and writes both to every sink named. It reads `--for`, `--poll`, `--batch`, `--transport`, `--api`,
`--api-user`, `--token`, `--port` and `--subnet` from the table above, plus its own flags below. It
refuses to start without at least one sink.

### API tier

| Flag                | Default | Variable                | Meaning                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
| ------------------- | ------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--api-mode`        | `full`  | none                    | preset. `off` sets `--api-every 0` unless you set it yourself, so the collector opens no API-tier session; the relay transport still uses the API. `slow` runs the tier every 10 s with no `/system/health` and no conntrack count; `/system/resource`, `/system/resource/cpu`, `monitor-traffic` on `--interfaces` and the `--counters-every` port counters are still read. `full` (the flag defaults) reads everything. An explicit flag below wins over it |
| `--api-every`       | `1s`    | none                    | API-tier cadence; `0` disables the tier. `off` sets it to `0`, `slow` to `10s`                                                                                                                                                                                                                                                                                                                                                                                |
| `--interfaces`      | empty   | `MIKROSCOPE_INTERFACES` | comma-separated interfaces for `monitor-traffic`, one call for all                                                                                                                                                                                                                                                                                                                                                                                            |
| `--conntrack-every` | `0`     | none                    | ask the conntrack count this often; `0` never, because it is a table scan. One scan took 1.3 ms at 6 212 entries. `slow` sets it to `0` unless given explicitly                                                                                                                                                                                                                                        |
| `--counters-every`  | `10s`   | none                    | read every port's cumulative counters (typed errors, fast-path split, link-downs, frame sizes) this often; `0` never                                                                                                                                                                                                                                                                                                                                          |
| `--labels-every`    | `5m`    | none                    | re-read what each interface is (label from its comment, type, interface lists, bridge, MTU); read once before the first kernel pull and again this often; `0` takes the 5 min default                                                                                                                                                                                                                                                                         |
| `--no-health`       | `false` | none                    | skip `/system/health`. `slow` sets it                                                                                                                                                                                                                                                                                                                                                                                                                         |

Without `--api` and `--api-user`, `forward` runs the kernel tier alone. With them, a first dial that
fails logs `api tier: not connected yet, will keep trying: …` and the tier connects on the first
round the router answers; it is never disabled for the life of the process.

### Sinks

| Flag                | Default               | Variable                   | Meaning                                                                                                            |
| ------------------- | --------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `--file`            | empty                 | none                       | write the merged timeline as JSONL to this path (truncated at start)                                               |
| `--prom`            | empty                 | none                       | serve Prometheus `/metrics` on this address, for example `:9124`                                                   |
| `--influx`          | empty                 | `MIKROSCOPE_INFLUX_URL`    | InfluxDB 3 server, for example `http://host:8181`. A full write URL is still taken verbatim                        |
| `--influx-db`       | empty                 | `MIKROSCOPE_INFLUX_DB`     | the database `--influx` writes to; ignored when `--influx` already carries a write path                            |
| `--loki`            | empty                 | `MIKROSCOPE_LOKI_URL`      | Loki push URL; carries events (kernel log, gaps, API errors, detections, triggers, the device record), not samples |
| `--loki-tenant`     | empty                 | `MIKROSCOPE_LOKI_TENANT`   | `X-Scope-OrgID` for a multi-tenant Loki                                                                            |
| `--otlp`            | empty                 | `MIKROSCOPE_OTLP_URL`      | OTLP/HTTP metrics endpoint, for example `http://host:4318/v1/metrics`                                              |
| `--graphite`        | empty                 | `MIKROSCOPE_GRAPHITE_ADDR` | Graphite carbon plaintext listener, `host:port`                                                                    |
| `--graphite-prefix` | `mikroscope`          | none                       | first node of every Graphite metric path                                                                           |
| `--elastic`         | empty                 | `MIKROSCOPE_ELASTIC_URL`   | Elasticsearch or OpenSearch base URL for `_bulk`                                                                   |
| `--elastic-index`   | `mikroscope-%Y.%m.%d` | none                       | index name; `%Y`, `%m`, `%d` expand to the event's date                                                            |
| `--sql`             | empty                 | none                       | write PostgreSQL/TimescaleDB statements to this path, or to standard output with `-`; there is no database driver  |
| `--sql-hypertable`  | `false`               | none                       | also emit TimescaleDB `create_hypertable` calls in the SQL header; applies to `--postgres` too                     |
| `--postgres`        | empty                 | `MIKROSCOPE_POSTGRES_DSN`  | send the same statements to a running PostgreSQL. The one sink flag that may carry a credential; see below         |
| `--telegraf`        | empty                 | `MIKROSCOPE_TELEGRAF_URL`  | Telegraf listener: `http://host:8186/telegraf`, `tcp://host:8094` or `udp://host:8094`                             |
| `--stdout`          | empty                 | none                       | write to standard output as `lp` (InfluxDB line protocol) or `json` (NDJSON); any other value is refused           |
| `--host-tag`        | `router`              | `MIKROSCOPE_HOST_TAG`      | host tag or label on every point, in every sink                                                                    |
| `--queue-seconds`   | `60`                  | none                       | seconds of data each queued sink may hold before it drops the oldest batch; `0` or less takes 60                   |

No sink **token** is a flag, because a flag is visible in `ps` and in shell history: they come from
`MIKROSCOPE_INFLUX_TOKEN`, `MIKROSCOPE_LOKI_TOKEN`, `MIKROSCOPE_OTLP_TOKEN`,
`MIKROSCOPE_ELASTIC_AUTH` and `MIKROSCOPE_TELEGRAF_TOKEN`. A sink that was asked for and cannot be
built stops `forward` before it pulls anything.

`--postgres` is the one exception, on purpose. A PostgreSQL connection string is the shape every
PostgreSQL client takes, and pgx reads `PGPASSWORD`, `~/.pgpass` and the service file exactly as
`psql` does, so a DSN with **no** password in it works the way an operator already expects. The flag
exists so the DSN can say host, database, user and `sslmode` on the command line; the password
belongs in one of those three places, or in `MIKROSCOPE_POSTGRES_DSN` in an environment file. A
password written into the flag is readable by any process on the machine.

On exit `forward` prints the number of kernel samples, API samples, gaps and skew jumps, and for
each sink how many events it wrote, dropped and failed on. While it runs it logs a line on standard
error once a minute with the kernel, API, gap, trigger and detection counts, the agent-restart
count, the last sequence number, and each sink's written, dropped and error counts. When the API
tier has failed or reconnected it adds `api: N failed round(s), N reconnect(s)`.

The collector's Prometheus sink sizes its busy-tick histogram for 10 Hz whatever rate the agent runs
at (`promHistogramRateHz` in [`cmd/mikroscope/sinkflags.go`](https://github.com/jmrplens/mikroscope/blob/main/cmd/mikroscope/sinkflags.go)), so the bucket layout does
not change when it reconnects to an agent configured differently. [Prometheus
metrics](https://docs.jmrp.io/mikroscope/reference/metrics/) says what else follows from that constant.

### Grafana publishing

`forward` can create a datasource and publish a dashboard for each store it writes to, once, at
start, before the first sample. [`dashboards publish`](https://docs.jmrp.io/mikroscope/reference/cli/#dashboards) does the same once, with the
same flags, and exits without collecting. It is off unless `--grafana` is given, and the token is
`GRAFANA_TOKEN`: publishing without one would write as whoever an anonymous request is to that
server.

Each run, and in `forward` before the router is touched:

1. It checks the flags before any request, in a dry run too. It refuses a run with no sink that has
   a dashboard, a `--grafana-datasource-sslmode` outside Grafana's four modes, and a run of more than one
   store that sets `--grafana-datasource-url` or `--grafana-datasource-uid`. Then, unless it is a
   dry run, it needs `GRAFANA_TOKEN`.
2. It finds the folder `--grafana-folder` by title, or creates it:
   `folder "<title>" (<uid>) unchanged|created`.
3. For each store its sinks write to, in the order InfluxDB, Elasticsearch, Prometheus, PostgreSQL,
   Graphite, it creates the datasource `mikroscope-<store>`, corrects it (`updated`) or leaves it
   `unchanged`, or adopts the one named in `--grafana-datasource-uid` and leaves it as it is. It then
   asks the store which measurements it holds and imports the dashboard `mikroscope-<store>` over
   itself, into the folder.

`forward` logs each line on standard error behind a `grafana:` prefix; `dashboards publish` prints
them on standard output.

Every store is tried. One that fails is reported on a line of its own that names it,
`the datasource for <store>: …` or `the dashboard for <store>: …`, and the stores after it are
still published; only a folder Grafana refuses stops the run. `forward` logs
`grafana: could not publish, carrying on without it: <reason>` once per failed store and collects;
`dashboards publish` exits 1. Only `--influx`, `--elastic`, `--prom`, `--postgres` or `--sql`, and
`--graphite` have a dashboard: with none of them `dashboards publish` stops with `no sink this
builds a dashboard for is configured, so there is nothing to publish`, and `forward` logs it as its
`grafana: could not publish…` warning and collects.

| Flag                           | Default      | Variable                                                                  | Meaning                                                                                                                                                              |
| ------------------------------ | ------------ | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--grafana`                    | empty        | `MIKROSCOPE_GRAFANA_URL`; for `dashboards publish` and `uninstall`, then `GRAFANA_URL` | the Grafana to publish to; empty publishes nothing                                                                                                      |
| `--grafana-folder`             | `mikroscope` | `MIKROSCOPE_GRAFANA_FOLDER`                                               | the folder to publish into; `--grafana-folder ""` is Grafana's General folder, and an empty variable is ignored                                                     |
| `--grafana-datasource-uid`     | empty        | `MIKROSCOPE_GRAFANA_DATASOURCE_UID`                                       | adopt an existing datasource by uid instead of creating one, and leave it untouched; the only way to publish `--sql`. One store per run                             |
| `--grafana-datasource-url`     | empty        | `MIKROSCOPE_GRAFANA_DATASOURCE_URL`                                       | the address Grafana queries, `host:port` for PostgreSQL and a URL for the others: needed for `--prom` and `--graphite`, and it replaces the address `--influx`, `--elastic` or `--postgres` would give. One store per run |
| `--grafana-datasource-sslmode` | empty        | `MIKROSCOPE_GRAFANA_DATASOURCE_SSLMODE`                                   | `disable`, `require`, `verify-ca` or `verify-full` for the PostgreSQL datasource, and nothing else; empty takes the mode `--postgres` names when Grafana has it, and `disable` otherwise |
| `--grafana-dry-run`            | `false`      | none                                                                      | with `--grafana`, print what it would write, contact no Grafana and need no token; `forward` then stops before collecting, and refuses it without a Grafana; `uninstall` refuses it |

[`uninstall`](https://docs.jmrp.io/mikroscope/reference/cli/#uninstall---targets) reads only `--grafana` and `--grafana-datasource-uid` of these.

**One store per run** for the two datasource flags: each names a single datasource, and two stores
are two servers read by two plugins. A collector that writes to several stores, one of which needs
a flag, runs without it; that store is published on its own with `dashboards publish`, given only
its sink flag and its own `--grafana-datasource-url` or `--grafana-datasource-uid`.

A failure here is a warning to `forward` and not a refusal to start: the samples of an hour spent
not running cannot be recovered, and a dashboard can be published on the next restart. It deletes
nothing; `uninstall --targets dashboard` does. `--grafana-dry-run` runs before the router is
touched, so it is answerable with no router in front of it. [Set up in
Grafana](https://docs.jmrp.io/mikroscope/dashboards/import-and-check/#letting-the-collector-do-it-forward---grafana)
says which sinks can describe their own datasource and which have to be told.

## dashboards

```sh
mikroscope dashboards gen
mikroscope dashboards publish --influx http://influx:8181 --influx-db mikroscope --grafana http://grafana:3000
mikroscope dashboards import --store influxdb --datasource-uid <uid>
mikroscope dashboards check  --store influxdb --datasource-uid <uid> --window 1h --end <RFC 3339 time>
```

| Subcommand | What it does                                                                                                                                                                                                                                                              |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `gen`      | writes the five dashboards and the three alert files into `--out`, creating it when missing; the files are readable only by you. Contacts nothing                                                                                                                                     |
| `publish`  | what `forward --grafana` does at start, once, with no router and no collector: it takes the [sink](https://docs.jmrp.io/mikroscope/reference/cli/#sinks) and [Grafana publishing](https://docs.jmrp.io/mikroscope/reference/cli/#publishing-to-grafana) flags, creates, corrects or adopts each store's datasource and imports its dashboard, prints a line per object, and exits 1 when any store failed |
| `import`   | asks the datasource which measurements it holds, generates the dashboard and posts it to `/api/dashboards/import` with overwrite on, into the General folder; prints `imported: <url>`. Creates no datasource                                                            |
| `check`    | asks and generates as `import` does, runs every panel's query through `/api/ds/query`, prints one line per panel and exits 1 when any panel fails. Writes nothing to Grafana                                                                                             |

`gen`, `import` and `check` take the flags below. `publish` takes none of them, and no argument
either: its flags are `forward`'s sink and Grafana publishing flags.

| Flag               | Default         | Variable                                   | Read by           | Meaning                                                                                                                         |
| ------------------ | --------------- | ------------------------------------------ | ----------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `--out`            | `dashboards`    | none                                       | `gen`             | output directory for `mikroscope-<store>.json` (five) and `mikroscope-alerts-<store>.yaml` (three), created with its parents |
| `--store`          | `influxdb`      | none                                       | `import`, `check` | `influxdb`, `prometheus`, `postgres`, `graphite` or `elasticsearch`                                                             |
| `--grafana`        | empty           | `GRAFANA_URL`, then `MIKROSCOPE_GRAFANA_URL` | `import`, `check` | Grafana base URL; the token comes only from `GRAFANA_TOKEN`                                                                  |
| `--datasource-uid` | empty, required | none          | `import`, `check` | the datasource UID bound to `DS_MIKROSCOPE`                                                        |
| `--no-probe`       | `false`         | none          | `import`, `check` | do not ask the datasource which measurements it holds; use the compiled defaults                   |
| `--window`         | `15m`           | none          | `check`           | length of the query window                                                                         |
| `--end`            | now             | none          | `check`           | RFC 3339 instant the window ends at, to check against a capture that has already finished          |
| `--var`            | none            | none          | `check`           | set a dashboard variable, `name=value`, repeatable: `--var host=router`                            |

`import` and `check` stop with `import/check need --grafana, GRAFANA_TOKEN and --datasource-uid`
when one of the three is missing. Unless `--no-probe` is given, they first ask the datasource which
measurements it holds; if that question fails they warn and continue with the compiled defaults.
`check` prints one line per panel (`ok`, `none` for a known-empty panel, `FAIL`) and exits 1 if any
panel fails.

`publish` stops with `dashboards publish needs --grafana (or MIKROSCOPE_GRAFANA_URL or
GRAFANA_URL), with the token in GRAFANA_TOKEN` when no Grafana is named, and with
`dashboards publish needs the sink flags the collector runs with (…)` when no sink flag is given,
because each datasource is described from the sink that writes to it. Which Grafana variable each
subcommand reads first is in [Environment variables](https://docs.jmrp.io/mikroscope/reference/environment/#grafana).
[Set up in Grafana](https://docs.jmrp.io/mikroscope/dashboards/import-and-check/) puts the four to use.

## version

`mikroscope version` prints `mikroscope` and the build identity from the
package [`internal/version/`](https://github.com/jmrplens/mikroscope/tree/main/internal/version). It takes no flags.

## See also

- [Environment variables](https://docs.jmrp.io/mikroscope/reference/environment/): every `MIKROSCOPE_*` variable, the
  credentials with no flag, and the agent's own envlist.
- [Install with the CLI](https://docs.jmrp.io/mikroscope/install/): the deployment verbs in the order they are used.
- [Record, mark and plot](https://docs.jmrp.io/mikroscope/record/): what a recording is and how markers get into it.
- [Run the collector](https://docs.jmrp.io/mikroscope/sinks/): what `forward` does with the flags above.
