# systemd A hardened unit for the one process on the host that holds a GitHub token, and what each restriction is for. Source: https://docs.jmrp.io/ghchronicle/install/systemd/ ## The unit ```ini title="/etc/systemd/system/ghchronicle.service" [Unit] Description=ghchronicle, GitHub metrics collector After=network-online.target Wants=network-online.target [Service] Type=simple User=ghchronicle Group=ghchronicle EnvironmentFile=/etc/ghchronicle/ghchronicle.env ExecStart=/usr/local/bin/ghchronicle -config /etc/ghchronicle/config.yaml Restart=always RestartSec=30s # This is the only process on the host holding a GitHub token, so it gets # nothing it does not need. NoNewPrivileges=true PrivateTmp=true PrivateDevices=true ProtectSystem=strict ProtectHome=true ProtectKernelTunables=true ProtectKernelModules=true ProtectControlGroups=true ProtectClock=true ProtectHostname=true ProtectProc=invisible RestrictNamespaces=true RestrictRealtime=true RestrictSUIDSGID=true LockPersonality=true MemoryDenyWriteExecute=true SystemCallArchitectures=native SystemCallFilter=@system-service CapabilityBoundingSet= AmbientCapabilities= RestrictAddressFamilies=AF_INET AF_INET6 StateDirectory=ghchronicle ReadWritePaths=/var/lib/ghchronicle [Install] WantedBy=multi-user.target ``` ## Why it is hardened The threat model is short and it is the whole justification: **this is very likely the only process on the host holding a GitHub token with read access to every repository of an account.** A token is a bearer credential. Anything that can read this process's memory or its environment file has the account. So the unit gives the process exactly what it needs, which turns out to be almost nothing: an outbound TCP socket and one writable directory. | Directive | What it removes | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | | `CapabilityBoundingSet=`, `AmbientCapabilities=` | Every Linux capability. It binds no privileged port and owns no device | | `NoNewPrivileges=true` | Any path to gaining privileges through exec, including setuid binaries | | `ProtectSystem=strict` | Write access to the entire filesystem, except `ReadWritePaths` | | `ProtectHome=true` | Every home directory, which is where the interesting credentials on a host usually live | | `PrivateTmp=true`, `PrivateDevices=true` | Shared temporary files, and the physical device nodes | | `ProtectProc=invisible` | The ability to see other users' processes in `/proc`, so it cannot read another process's command line | | `RestrictAddressFamilies=AF_INET AF_INET6` | Unix and netlink sockets. It talks HTTPS and nothing else | | `MemoryDenyWriteExecute=true`, `LockPersonality=true` | The usual shellcode primitives | | `SystemCallFilter=@system-service` | Every syscall outside the ordinary service set, including the module and kernel-tuning ones | | `ProtectKernelTunables`, `ProtectKernelModules`, `ProtectControlGroups`, `ProtectClock`, `ProtectHostname`, `RestrictNamespaces`, `RestrictRealtime`, `RestrictSUIDSGID` | Every remaining route to changing the host from inside the service | `StateDirectory=ghchronicle` makes systemd create `/var/lib/ghchronicle` with the right ownership on start, so the state file has somewhere to live without a manual `mkdir` and a `chown` that someone will forget after a reinstall. Four files live there, not one. Beside `state.json` the sweep keeps its write ledger, `state-written.bin` by default, which is what stops an unchanged point being written again, and its cache, `state-cache.bin`, which is what lets a restart ask GitHub only for what changed; and the service holds `state-lock` for as long as it runs, which is how `-migrate -yes` knows not to change the stores under it. A backfill, or the reading back of a [migration](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations), that stops half way leaves its checkpoint there too. `ReadWritePaths` covers the directory, so all of them are already allowed. Put the state file or the ledger somewhere else and that path needs adding here, and the cache follows the state file wherever it goes. Losing the ledger costs one sweep of rewriting: [only what changed is written](https://docs.jmrp.io/ghchronicle/sinks/#only-what-changed-is-written); losing the cache, one sweep at full price: [the cache beside it](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it). ## Installing it 1. Create the user and the directories. ```sh sudo useradd --system --no-create-home --shell /usr/sbin/nologin ghchronicle sudo mkdir -p /etc/ghchronicle ``` 2. Put the configuration in place. - /etc/ghchronicle/ - config.yaml world readable, no secrets in it - ghchronicle.env mode 600, the tokens - /var/lib/ghchronicle/ - state.json created by the service - state-written.bin the write ledger, beside it - state-cache.bin the cache, beside it too - state-lock held by the service while it runs 3. Write the environment file, and nothing else in it. ```sh title="/etc/ghchronicle/ghchronicle.env" GITHUB_TOKEN=github_pat_... INFLUX_TOKEN=... ``` ```sh sudo chmod 600 /etc/ghchronicle/ghchronicle.env ``` Everything in `config.yaml` reads these through `${VAR}`, which is what lets the config be world readable and version controlled while the secrets are not. 4. Start it. ```sh sudo systemctl daemon-reload sudo systemctl enable --now ghchronicle sudo systemctl status ghchronicle ``` The first sweep runs every family, since none has run yet, except that the service starts at most one family of six hours or more a sweep: `traffic`, `stats`, `forks` and the rest of the eleven reach the store over the first two and a half hours, and only this once. See [the slow families take turns](https://docs.jmrp.io/ghchronicle/configuration/cadences/#the-slow-families-take-turns). ## What to watch The log says what was written and where. ```text level=INFO msg=written sink=influxdb family=repo points=934 unchanged=1955 level=INFO msg="rate budget" bucket=core remaining=4477 limit=5000 ``` [What the log says on a good day](https://docs.jmrp.io/ghchronicle/configuration/logging/#what-the-log-says-on-a-good-day) has every routine line. Three warnings are worth an alert: - **`rate limit reserve reached`** means a family was skipped to protect the budget. Once is fine; every sweep means the cadences are too fast for the number of repositories. - **`family failed everywhere, not marking it as run`** means every repository failed for one family, so it will be retried rather than treated as done. - **`migration pending`**, at every start after an upgrade, means a store still holds rows in a shape this release no longer writes, and the start left them: see [after an upgrade](https://docs.jmrp.io/ghchronicle/install/systemd/#after-an-upgrade). ```sh journalctl -u ghchronicle -f journalctl -u ghchronicle -p warning --since today ``` > **Check the sandbox after editing the unit** > > `systemd-analyze security ghchronicle` scores the unit and names anything the > sandbox is not covering. It is the fastest way to see that an edit quietly > removed a restriction. ## After an upgrade Replace the binary and restart the service. Under the default [`migrate: auto`](https://docs.jmrp.io/ghchronicle/configuration/#migrate), the start checks every store against the changes the new release carries and, before its first sweep, applies on its own each one that loses nothing, saying so at `WARN`. A change it leaves is a `WARN` at every start, naming the store, the reason and the two commands: [what a start does about it](https://docs.jmrp.io/ghchronicle/install/upgrading/#what-a-start-does-about-it). Run those as the service's own user and with its environment file, with the service stopped, since `-migrate -yes` refuses to run beside it: ```sh sudo systemctl stop ghchronicle sudo systemd-run --uid=ghchronicle --gid=ghchronicle --pipe --wait --collect \ --property=EnvironmentFile=/etc/ghchronicle/ghchronicle.env \ /usr/local/bin/ghchronicle -config /etc/ghchronicle/config.yaml -migrate sudo systemd-run --uid=ghchronicle --gid=ghchronicle --pipe --wait --collect \ --property=EnvironmentFile=/etc/ghchronicle/ghchronicle.env \ /usr/local/bin/ghchronicle -config /etc/ghchronicle/config.yaml -migrate -yes sudo systemctl start ghchronicle ``` The first prints the plan and changes nothing; the second applies it and reads back what it cleared. `systemd-run` reads the environment file as root, as the unit does, so it stays mode 600, and runs the binary as `ghchronicle`: measured on systemd 257, a command started that way saw the token of a file only root could read and ran as the user named. Run by root instead, `-migrate -yes` saves the files it rewrites beside the state file as root's, mode 600, `state.json` among them, and the service, which runs as `ghchronicle`, then stops at its start and names the file rather than start from a new one, which would forget a refill still owed: `chown ghchronicle:ghchronicle` it back. With `-once` run from cron rather than a service, comment the line out for the length of the two commands: a `-once` holds no lock while it sweeps, so nothing stops it running beside `-migrate -yes`. ## cron instead of a service `-once` runs a single sweep and exits, which is all a scheduler needs. ```text 0 * * * * /usr/local/bin/ghchronicle -config /etc/ghchronicle/config.yaml -once ``` Keep the state file on a persistent path even in this mode. Without it every run collects every family, whatever its cadence, and walks the stargazer list, the whole star history and the co-authored pull requests again; the cache file beside it is what lets a run ask GitHub only for what changed. Note that an hourly cron gives every family an hourly cadence at best, so the five that run every fifteen minutes, `actions`, `events`, `notifs`, `activity` and `ratelimit`, run four times less often: a workflow run's queue time is read after it is over, and a notification thread moved twice inside the hour shows only its second move.