# ghchronicle ghchronicle is an open-source tool that collects every metric GitHub exposes about an account and keeps each one with the date it happened. Source: https://docs.jmrp.io/ghchronicle/ Every point ghchronicle writes carries the date the thing happened, which is what makes a question about last July still have an answer. ## What it collects - [95 measurements](https://docs.jmrp.io/ghchronicle/collectors/measurements/) - [34 collectors](https://docs.jmrp.io/ghchronicle/collectors/) - [11 outputs](https://docs.jmrp.io/ghchronicle/sinks/) - [154 dashboard panels](https://docs.jmrp.io/ghchronicle/dashboards/) ## What it is GitHub answers most questions about the present and almost none about the past. The traffic API serves fourteen days and forgets. The activity feed keeps the last three hundred events, and none older than thirty days. Inbox notifications are kept for three months unless they are saved. Job logs are deleted after ninety days by default. **ghchronicle** sweeps those surfaces on a schedule and writes every observation as a dated point, into whichever database you already run. One static Go binary with no runtime dependencies. ## Who it is for - Anyone who already runs a time series database and a Grafana - Maintainers who want traffic and stars kept past GitHub's window - Teams who want merge times, review load and CI cost as history rather than as a number that resets - Anyone who wants their own data out of GitHub before GitHub drops it ## What it is not - Not a hosted service: it runs on your machine, with your token - Not a replacement for GitHub Insights, which answers about now - Not able to recover what GitHub has already dropped: a backfill walks stars, pull requests and runs back as far as the API answers, but traffic older than fourteen days, events past the last three hundred or older than thirty days and job logs past the repository's retention period, ninety days by default, are gone for everyone - Not a badge generator, though it can draw one ## What it looks like in use Three steps, and the first sweep lands in your database. ### Install A single binary, or a release, or the container image. ```sh go install github.com/jmrplens/ghchronicle/v2/cmd/ghchronicle@latest ``` ### Configure The smallest configuration that does something. A ${VAR} in a credential, an address or a path is read from the environment, so this file holds no secrets. ```yaml github: token: ${GITHUB_TOKEN} targets: user: your-login sinks: influxdb: url: http://localhost:8181 bucket: github ``` ### Collect What lands in the database. Note the timestamps: the star is dated 2024, not today, because that is when it was given. ```text gh_star,repo=parser,user=someone starred=1i 1731590400000000000 gh_traffic,repo=parser,kind=views count=142i,uniques=61i 1757376000000000000 gh_pull_request,repo=parser,number=318,state=MERGED churn=214i,seconds_to_merge=5820i 1757462400000000000 ``` ## How it is shaped Four ideas, and the second is the one everything else follows from. 1. **Sweep**: Each family of metrics has its own cadence, because they move at very different speeds: workflow runs every quarter of an hour, the contribution calendar every hour, the account's keys once a day. 2. **Date**: A point carries the moment the thing happened, not the moment it was collected. That is what makes re-collection converge instead of accumulating copies. 3. **Push**: Everything but the Prometheus exporter is pushed: to InfluxDB, PostgreSQL, Graphite, Elasticsearch, Prometheus through its OTLP receiver, OpenTelemetry, Loki, Telegraf, a file or anything Telegraf can reach, so it runs wherever it can reach them. 4. **Draw**: One dashboard specification, rendered once per store. The same panels whichever database you chose, and where a store cannot answer one honestly, the panel says so. ## Where to start - [Quickstart](https://docs.jmrp.io/ghchronicle/start/quickstart/): From nothing to a first sweep. - [The token](https://docs.jmrp.io/ghchronicle/start/token/): Which scopes, and why the automatic one is not enough. - [Dating a point](https://docs.jmrp.io/ghchronicle/how/dating/): The idea the rest of the design follows from. - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/): What each of the 11 can and cannot answer. --- # What it is What ghchronicle collects from GitHub, and the problem it exists to solve: traffic kept for fourteen days, an event feed of three hundred events and thirty days. Source: https://docs.jmrp.io/ghchronicle/start/ GitHub answers most questions about the present and almost none about the past. The [traffic API](https://docs.github.com/en/rest/metrics/traffic) serves fourteen days and forgets. The [activity feed](https://docs.github.com/en/rest/activity/events) keeps the last three hundred events, and none older than thirty days. The [notification inbox](https://docs.github.com/en/subscriptions-and-notifications/concepts/about-notifications#notification-retention-policy) drops every notification after three months, read or not, unless it was saved. [Job logs](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository) are deleted after ninety days by default. The star list will tell you when each star was given, but since July 2026 only to a repository's [admins and collaborators](https://docs.github.com/en/rest/activity/starring#new-access-restrictions), and only if you ask before the list gets long enough to be expensive to walk. None of it is archived anywhere unless you archive it. `ghchronicle` sweeps those surfaces on a schedule and writes every observation as a point stamped with the date the thing actually happened, so a year from now the question "how fast were we merging in July" still has an answer. ```sh ghchronicle -config config.yaml ``` It is one static Go binary with no runtime dependencies, and it pushes to every store it supports, so it runs wherever it can reach them: a server, a container, a scheduled workflow. > **The card is not the point** > > It can also draw an SVG summary card for a profile README. That is a side > feature. The reason the project exists is the ingestion. ## The six words the rest of this uses | Word | What it means here | | --------------- | ------------------------------------------------------------------------------------------------------ | | **family** | One collector, named in the configuration: `actions`, `stars`, `issues`. There are 34 | | **group** | A named set of families, for switching a whole area on or off: `ci`, `security`, `audience`. There are 8 | | **measurement** | One kind of row in the store, named `gh_*`: `gh_star`, `gh_workflow_run`. There are 95 | | **point** | One row: a measurement, its tags, its fields and the date the thing happened | | **sweep** | One pass over the families that are due, which is what the process does on a loop | | **backfill** | A run with `-backfill`, which walks the history instead of the increment | `ghchronicle -groups` prints the groups with their families, and `ghchronicle -config config.yaml -list` prints the repositories a sweep would cover. ## Where to go next - [Quickstart](https://docs.jmrp.io/ghchronicle/start/quickstart/): from nothing to a first sweep. - [The token](https://docs.jmrp.io/ghchronicle/start/token/): which scope buys which family. - [Dating a point](https://docs.jmrp.io/ghchronicle/how/dating/): the design idea everything else follows from. --- # Quickstart Install ghchronicle, point it at a database and run a first sweep of a GitHub account, and what that first sweep does that later ones do not. Source: https://docs.jmrp.io/ghchronicle/start/quickstart/ Six steps and a configuration file. The only decision worth thinking about before you start is which store keeps the history, and you can defer that by printing the points to the terminal first. ## The short way If you have a terminal, let it ask: ```sh ghchronicle -setup ``` It asks for a token, checks with GitHub who the token is, asks which account to collect and where to put the numbers, checks that the place answers, offers the dashboard and offers to set it up to run by itself. What it writes is a `config.yaml` that loads and, when you asked for a service, a unit, an agent or a scheduled task for whichever system this is. Credentials never go in the configuration. They go in a file beside it that only you can read, and the configuration refers to them by name. The installer above offers to run it as its last step, so on a new machine the two together are the whole of getting started. The rest of this page is what it writes, for a reader who would rather write it themselves or wants to know what they just agreed to. ## Zero to a first sweep 1. **Install the binary.** - **Script** ```sh curl -fsSL https://raw.githubusercontent.com/jmrplens/ghchronicle/main/install.sh | bash ``` On Windows, in PowerShell: ```powershell irm https://raw.githubusercontent.com/jmrplens/ghchronicle/main/install.ps1 | iex ``` - **Release** Take the archive for your platform from the [releases page](https://github.com/jmrplens/ghchronicle/releases) and put `ghchronicle` on your `PATH`. - **Container** ```sh docker pull ghcr.io/jmrplens/ghchronicle ``` - **Go** ```sh go install github.com/jmrplens/ghchronicle/v2/cmd/ghchronicle@latest ``` 2. **Create a token** at `https://github.com/settings/tokens` and export it. ```sh export GITHUB_TOKEN=github_pat_... ``` A classic token with `repo`, `read:packages`, `read:user`, `read:org`, `security_events`, `read:public_key` and `read:gpg_key` sees everything this collects. The [token page](https://docs.jmrp.io/ghchronicle/start/token/) explains which scope buys which family, and why the automatic `GITHUB_TOKEN` of an Action is not enough. 3. **Write the configuration.** Two decisions, and this is the whole file: ```yaml # config.yaml github: token: ${GITHUB_TOKEN} targets: user: your-login sinks: stdout: true # swap for influxdb once you have somewhere to put it ``` A `${VAR}` in a credential, an address or a file path is read from the environment at start-up, so the file itself holds no secrets and can be committed. Everything else has a default. The documented version, which comments every option there is, lives in the repository rather than in the install, so take it from there when you want to read the rest: ```sh curl -O https://raw.githubusercontent.com/jmrplens/ghchronicle/main/config.example.yaml ``` 4. **See what would be collected**, before spending any quota on it. ```sh ghchronicle -config config.yaml -list ``` That prints the repositories in scope. Forks and archived repositories are excluded by default. If something you expected is missing, this is the command that tells you. 5. **Run one sweep.** ```sh ghchronicle -config config.yaml -once ``` With `sinks.stdout: true` the points go to the terminal as line protocol instead of to a database, which is the cheapest way to see the shape of what you are about to store. 6. **Leave it running.** ```sh ghchronicle -config config.yaml ``` Each family then runs on its own cadence: workflow runs every fifteen minutes, the contribution calendar every hour, the account's SSH and GPG keys once a day. ## What the first sweep does that later ones do not Four things happen once, and they are why the first run is the expensive one. - The daily star history of every repository, whatever the token may see, is read back to the repository's first week, and the whole stargazer list of every repository whose list the token may read (since July 2026, only a repository's [admins and collaborators](https://docs.github.com/en/rest/activity/starring#new-access-restrictions)) is walked, page by page, so each of those stars carries the moment it was given and who gave it. After that, the history is one request per repository for its newest thirty weeks, usually a free 304, and the newest hundred stars of each readable list ride in one GraphQL query per ten repositories. - A month of workflow runs, so a fresh install does not chart a CI history that begins fifteen minutes ago. After that, twice the cadence, and never less than two hours. - The co-authored pull requests of the account's whole life, which `achievements` walks for the Pair Extraordinaire count: 35 queries and 23.7 MB over 2,315 merged pull requests, measured on 2026-09-27. After that a pass walks the days since, one page, and the whole history again once a week. - Every year's contribution calendar, if `every.families.history` is set, back to the day the account was created. After that, only the year in progress, rewritten at its cadence. Nothing is cached yet either, so every answer of the first sweep is paid in full, where a later one asks with the ETag the last answer came with and is mostly answered a free 304; see [the cache beside the state file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it). Expect a few thousand points from a first sweep of twenty repositories, and a few hundred from each one after. None of that is the history. The first sweep is a wider increment, and a dashboard at ninety days or two years then begins on the day you installed the collector: measured after a day of sweeps, about a fifth of the pull requests the repositories report, commits from the last thirty days only, and jobs for a tenth of the workflow runs. Run `ghchronicle -config config.yaml -backfill` once, before the service or right after it; [the backfill page](https://docs.jmrp.io/ghchronicle/how/backfill/) says what it reaches and what it costs. > **Keep the state file** > > `state_file` is what remembers where each family got to, and > [nine things live in it](https://docs.jmrp.io/ghchronicle/configuration/#state_file). Delete it > and the next sweep re-collects everything, which costs quota and nothing else > for seven of the nine. The other two are the commit each dependency diff > starts from, whose changes in the gap are not collected again, and what the > file records of each store, a refill a migration still owes among it, which > nothing then reads back. ## Where to go next - [Ways to install](https://docs.jmrp.io/ghchronicle/install/) is what turns the command above into something that keeps running: systemd, Docker or a scheduled Action. - [Dating a point](https://docs.jmrp.io/ghchronicle/how/dating/) is the design idea everything else follows from. - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/) decides which questions you will be able to ask later. - [Cost of a sweep](https://docs.jmrp.io/ghchronicle/api/cost/) is the measured price in API calls. --- # The token Which GitHub token scopes buy which ghchronicle families, and why the automatic GITHUB_TOKEN of a workflow is not enough. Source: https://docs.jmrp.io/ghchronicle/start/token/ Everything here is read access. The collector never writes to GitHub. What varies is how much of the account a given token is allowed to see, and a handful of scopes are worth understanding rather than just granting. ## Creating one 1. Go to `https://github.com/settings/tokens`. 2. Choose the kind of token and give it the scopes below. - **Classic** `repo`, `read:packages`, `read:user`, `read:org`, `security_events`, `read:public_key` and `read:gpg_key` covers everything this collects. - **Fine-grained** Read access to the repositories, including the repository permission Administration, which traffic needs, plus the account permissions for followers, gists, packages, plan, Git SSH keys and GPG keys. 3. Put it where the process can read it, and nowhere else. ```sh export GITHUB_TOKEN=github_pat_... ``` The configuration file refers to it as `${GITHUB_TOKEN}`, which is expanded from the environment at start-up. That is what lets the file be committed while the token stays out of it. ## What each scope buys | Scope | Without it | | ----------------------------- | ------------------------------------------------------------------------------------------------ | | Push access to the repository, and Administration (read) on a fine-grained token | Traffic is a 403. GitHub shows views and clones only to someone who can push to the repository | | `security_events` | Dependabot and code scanning alerts look exactly like a repository with the feature switched off | | `read:packages` | The container registry is invisible. Package versions cost one call per package | | `read:user` | Followers, contributions, gists and social accounts are missing | | `read:org` | Repositories owned by an organisation are not discovered | | `read:public_key`, `read:gpg_key` | The `keys` family writes nothing, and says nothing. No other scope implies either | Traffic is the one that surprises people, because being able to read a repository is not enough. GitHub shows views and clones only to someone who can push to the repository, and a fine-grained token also needs the [repository permission Administration (read)](https://docs.github.com/en/rest/metrics/traffic#get-repository-clones--fine-grained-access-tokens); a token that lacks either gets a 403 for that repository. The collector records that as "unavailable" and moves on, which is why the symptom is an empty traffic panel rather than a failed sweep. > **Unavailable is not an error** > > Anything the token cannot see is recorded as unavailable and skipped. A > repository with a feature switched off must not stop the sweep for the other > forty, so the log says `not available (403)` and the sweep continues. ## Why the automatic GITHUB_TOKEN is not enough A workflow gets a `GITHUB_TOKEN` for free. It is not enough for this, and the reason is worth being specific about rather than letting someone discover it as an empty dashboard. - **It cannot read traffic, not even its own.** Traffic needs push access to _every_ repository being collected, and the automatic token has it only for the repository the workflow is running in. Even there it gets a 403: GitHub also asks for the repository permission Administration, which a workflow cannot grant its automatic token. - **It has no `security_events`.** Dependabot and code scanning alerts are invisible to it. - **It has no `read:packages`.** The container registry is invisible to it. - **It is not a user.** Everything account-wide (followers, the contribution calendar, billing, notifications, the stars you gave) is about the person, not about a repository, and the automatic token is an installation, not a person. So a workflow needs a personal access token stored as a repository secret and passed in as the `token` input. See [GitHub Actions](https://docs.jmrp.io/ghchronicle/install/actions/). ## Running without a token Only for the three commands that ask GitHub nothing: `-backfill-status`, `-publish-dashboard` and `-uninstall`. Everything that collects, a card of public numbers included, needs one, and without it stops at start-up with `github.token is empty and GITHUB_TOKEN is unset`. What a token short of a scope costs is the table above: the part it cannot see is left empty, and the log says `not available (403)` for each of them, which is the collector reporting a permission, not a failure. --- # Compared with the alternatives ghchronicle beside GitHub Insights, repohistory, github-repo-stats, star-history and the Prometheus exporters: what each keeps, how long, and where. Source: https://docs.jmrp.io/ghchronicle/start/compared/ Several tools answer part of what ghchronicle answers: keeping GitHub traffic past fourteen days, drawing a star curve, or putting repository counts in Prometheus. They differ in what they cover, in whether a row carries the date the thing happened or the moment somebody looked, and in where the data ends up. This page sets eight of them beside ghchronicle, says where each cell comes from, and says when one of them is the better choice. ## What each one keeps | Tool | What it covers | How long it is kept | Dated, or a current value | Backfill | | ---- | -------------- | ------------------- | ------------------------- | -------- | | [ghchronicle](https://docs.jmrp.io/ghchronicle/collectors/) | 34 families: traffic, stars, forks, workflow runs with their jobs and steps, pull requests, issues, commits, the event feed, notifications, alerts, billing and more | as long as the store it writes to keeps it | dated: each point at the moment the thing happened; a top-ten list with no date at the start of its UTC day; a current state at the sweep | yes, `-backfill` walks every enabled family back as far as the API answers, but reads only the newest hundred webhook deliveries per hook and at most 500 failed job logs per repository; not traffic older than 14 days, the event feed past GitHub's ceiling, job logs older than 90 days, the names and exact times of the stars of a repository whose stargazer list the token cannot read (its stars are kept per day instead), or, from 2026-10-01, workflow runs older than the repository's retention setting | | [GitHub itself](https://docs.github.com/en/rest/metrics/traffic) | traffic: views and clones per day, the top ten referrers and paths; the event feed; the notification inbox; Actions runs, logs and artifacts | traffic 14 days; the event feed up to 300 events and 30 days; the inbox 3 months unless saved; Actions logs and artifacts 90 days by default, and from 2026-10-01 workflow runs too | views and clones by day; referrers and paths as one top-ten list for the whole 14 days | not applicable: nothing older than those windows is served | | [KipHub Traffic](https://github.com/farique1/KipHub-Traffic/blob/711ccf7558bae756eed14accdf1b418cbe17c7f3/README.md) | traffic only: views and clones per day, and GitHub's 14-day referrer list | as long as its local JSON file is kept | views and clones by day, merged into one file on every fetch | no: its README says to fetch more often than every 14 days | | [repohistory](https://github.com/repohistory/repohistory/blob/9dd4ac28b42d372cdb91668e379569b0cc69d694/README.md) | traffic: clones, views, referrers and popular pages; star history | from the first sign-in on | traffic stored per repository and date; stars at the date of each one for the 400 pages of 100 it reads at most, the oldest 40,000, and the rest spread evenly over the days after the last one read and marked estimated | traffic no, it starts at the first sign-in; stars from GitHub's stargazer list | | [github-repo-stats](https://github.com/jgehrcke/github-repo-stats/blob/ff48771cbb4324c3b77490bbfc26d6a5f3831cdb/README.md) | traffic: views and clones per day, top referrers and paths; stargazers and forks over time | "an arbitrarily long time frame", as daily snapshots kept in git | a daily snapshot of the traffic; stars and forks at the date of each one, and a daily snapshot of the star count that carries the curve past 40,000 | stars and forks yes, stars up to the oldest 40,000, a bound its source calls a GitHub API limit; traffic no | | [grafana/github-traffic](https://github.com/grafana/github-traffic/blob/f3ab4e1cb5fbeb52983fd67e93a3118e1630e96e/README.md) | views and clones of the newest day in the window, the star count, top paths and referrers, the rate limit | not stated: whatever the Prometheus that scrapes it keeps | current values, set on every run | none in its source | | [promhippie/github_exporter](https://github.com/promhippie/github_exporter/blob/a9208c3a257a6605a01cf0b18637dd4fa28b4153/README.md) | repositories, organisations and their seats, billing, self-hosted runners, admin statistics; workflow runs and jobs from a webhook | runs and jobs kept in its database for a purge window, 24 hours by default; the rest not stated | current values; the times of runs and jobs are exposed as timestamp values | runs and jobs arrive only by webhook, as they happen; otherwise not stated | | [githubexporter/github-exporter](https://github.com/githubexporter/github-exporter/blob/eedff7e7d54326c9607faa8b2ffc28a0140685de/README.md) | per repository: stars, forks, watchers, open issues, pull requests, size, release downloads; the rate limit | not stated: whatever the Prometheus that scrapes it keeps | current values | not stated | | [star-history.com](https://github.com/star-history/star-history/blob/608d0e39be441665c01cd6ac1fd01ca5ec1eaf39/README.md) | stars only | nothing is kept for you: every chart is drawn from GitHub's stargazer list | dated by the `starred_at` of a sample of stars, not of each one: under 15 pages of 100 it reads every page and keeps about fifteen evenly spaced stars (every star under 30); from 15 pages on (16 in its SVG API) it reads that many pages spread across the list and keeps the first star of each; a last point is today's total | the whole curve, on every draw | ## Where each one runs and writes | Tool | Runs as | Writes to | Self-hosted | | ---- | ------- | --------- | ----------- | | [ghchronicle](https://docs.jmrp.io/ghchronicle/install/) | one Go binary for Linux, macOS and Windows; Docker; a systemd unit; a GitHub Action | InfluxDB, PostgreSQL, Graphite, Elasticsearch, Loki, OpenTelemetry, Telegraf, a file or stdout, and Prometheus as current values only; a generated Grafana dashboard for five of them | yes | | GitHub itself | github.com: the Insights graphs and the REST API; the traffic graph of a private repository needs GitHub Pro, Team or Enterprise Cloud | nothing you own | not applicable | | KipHub Traffic | a Python script, with an optional Dear PyGui window, run by hand; last commit 2024-02-16 | a JSON file in a local `JSONs` folder | yes, on your own machine | | repohistory | a hosted service at repohistory.com, installed as a GitHub App, collecting once a day; free for one repository, paid for more | its own Supabase database; CSV export | the source is AGPL-3.0; running your own copy is not stated | | github-repo-stats | a GitHub Action, once a day, one repository per job | CSV files and an HTML and PDF report, committed to a branch | yes, in your own repository's Actions | | grafana/github-traffic | a Docker container, in Python, hourly by default; for a user or an organisation | a Prometheus endpoint on port 8001; Grafana dashboard 15000 | yes | | promhippie/github_exporter | a Go binary, DEB and RPM packages, containers; the webhook collectors need it reachable from the internet | a Prometheus endpoint; Chai (the default in most of its release builds), SQLite, MySQL or PostgreSQL for the webhook data | yes | | githubexporter/github-exporter | a Docker container, in Go, on port 9171 | a Prometheus endpoint | yes | | star-history.com | a hosted website, a Chrome extension, and an API that renders the chart as SVG | an image to embed | hosted; its README's development section runs it locally | The rows for the other tools are what each one's own README, documentation or source says at the commit linked, read on 2026-09-24, with "not stated" where it says nothing. Beyond the READMEs linked in the first table, the cells come from these files: - repohistory at `9dd4ac2` (2026-03-04): the [daily job](https://github.com/repohistory/repohistory/blob/9dd4ac28b42d372cdb91668e379569b0cc69d694/.github/workflows/cron.yaml), the [traffic writer](https://github.com/repohistory/repohistory/blob/9dd4ac28b42d372cdb91668e379569b0cc69d694/web/src/utils/update-traffic.ts), the [star reader](https://github.com/repohistory/repohistory/blob/9dd4ac28b42d372cdb91668e379569b0cc69d694/web/src/utils/repo/stars.ts), the [pricing page](https://github.com/repohistory/repohistory/blob/9dd4ac28b42d372cdb91668e379569b0cc69d694/landing/app/pricing/page.tsx), the [export route](https://github.com/repohistory/repohistory/blob/9dd4ac28b42d372cdb91668e379569b0cc69d694/web/src/app/api/export/%5BrepoId%5D/route.ts) and the [licence](https://github.com/repohistory/repohistory/blob/9dd4ac28b42d372cdb91668e379569b0cc69d694/LICENSE). - github-repo-stats at `ff48771` (2026-07-10): the [fetcher](https://github.com/jgehrcke/github-repo-stats/blob/ff48771cbb4324c3b77490bbfc26d6a5f3831cdb/fetch.py), whose star walk says it returns "at most the oldest 40.000 stargazers (a GitHub HTTP API limitation)", and which reads the star count on every run and keeps it when it has changed, "for building up a stargazer timeseries beyond said limit". - grafana/github-traffic at `f3ab4e1` (2026-05-25): its one [source file](https://github.com/grafana/github-traffic/blob/f3ab4e1cb5fbeb52983fd67e93a3118e1630e96e/github-traffic.py), which sets each traffic gauge from the last day of the window. - promhippie/github_exporter at `a9208c3` (2026-09-21): the [metrics](https://github.com/promhippie/github_exporter/blob/a9208c3a257a6605a01cf0b18637dd4fa28b4153/docs/partials/metrics.md), the [settings](https://github.com/promhippie/github_exporter/blob/a9208c3a257a6605a01cf0b18637dd4fa28b4153/docs/partials/envvars.md), the [usage guide](https://github.com/promhippie/github_exporter/blob/a9208c3a257a6605a01cf0b18637dd4fa28b4153/docs/content/usage.md), the [stores](https://github.com/promhippie/github_exporter/tree/a9208c3a257a6605a01cf0b18637dd4fa28b4153/pkg/store), the [default database](https://github.com/promhippie/github_exporter/blob/a9208c3a257a6605a01cf0b18637dd4fa28b4153/pkg/command/command.go) and the [release builds](https://github.com/promhippie/github_exporter/blob/a9208c3a257a6605a01cf0b18637dd4fa28b4153/.goreleaser.yaml). - githubexporter/github-exporter at `eedff7e` (2026-06-09): the [metrics list](https://github.com/githubexporter/github-exporter/blob/eedff7e7d54326c9607faa8b2ffc28a0140685de/METRICS.md) and the [metrics definitions](https://github.com/githubexporter/github-exporter/blob/eedff7e7d54326c9607faa8b2ffc28a0140685de/exporter/metrics.go). - star-history at `608d0e3` (2026-09-12): the [chart loader](https://github.com/star-history/star-history/blob/608d0e39be441665c01cd6ac1fd01ca5ec1eaf39/shared/common/chart.tsx), the [API client](https://github.com/star-history/star-history/blob/608d0e39be441665c01cd6ac1fd01ca5ec1eaf39/shared/common/api.tsx) and the [SVG server's constants](https://github.com/star-history/star-history/blob/608d0e39be441665c01cd6ac1fd01ca5ec1eaf39/backend/const.ts). - KipHub Traffic at `711ccf7` (2024-02-16): its README alone. The row for GitHub comes from GitHub's documentation, read the same day: [traffic in the REST API](https://docs.github.com/en/rest/metrics/traffic), [the traffic graph](https://docs.github.com/en/repositories/viewing-activity-and-data-for-your-repository/viewing-traffic-to-a-repository), [the event feed](https://docs.github.com/en/rest/activity/events), [the notification inbox](https://docs.github.com/en/subscriptions-and-notifications/concepts/about-notifications#notification-retention-policy) and [Actions retention](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#configuring-the-retention-period-for-checks-workflow-runs-commit-statuses-artifacts-and-logs-in-your-repository). The row for ghchronicle comes from this site: [what is collected](https://docs.jmrp.io/ghchronicle/collectors/), [dating a point](https://docs.jmrp.io/ghchronicle/how/dating/), [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/), [choosing a store](https://docs.jmrp.io/ghchronicle/sinks/) and [ways to install](https://docs.jmrp.io/ghchronicle/install/). Third-party tools change: check their pages before relying on a cell. ## Dated, or a current value The column that separates these tools most is the fourth one. A Prometheus exporter holds a gauge, and the history of a gauge is the samples Prometheus took of it, each stamped when it was scraped. grafana/github-traffic sets its views gauge from the newest day of the fourteen, so what Prometheus keeps is that number as it stood at each scrape, stamped with the time of the scrape rather than the day it counts. The tools that write files or rows can keep the day itself. KipHub Traffic, repohistory and github-repo-stats all store views and clones per day, which is the part of traffic that has a date, and github-repo-stats and repohistory draw stars from the date each one was given. ghchronicle applies the same rule to every surface it reads, and that rule is why its Prometheus side holds current values only: Prometheus refuses a sample dated days back, which [dating a point](https://docs.jmrp.io/ghchronicle/how/dating/#what-prometheus-structurally-cannot-hold) measures. ## GitHub's own windows Most tools on this page exist because of the second row. Traffic is fourteen days. The event feed serves up to three hundred events, none older than thirty days. The inbox keeps notifications for three months unless they are saved. Logs and artifacts are deleted after ninety days by default, and GitHub's documentation says that from 1 October 2026 the same retention setting also deletes workflow runs, checks and commit statuses, which until then were kept for 400 days or more. A public repository can set it between one and ninety days; a private one up to 400. What GitHub serves in those windows is complete while it lasts. For a question about the last two weeks, the Insights graphs and the REST API are the right tool. ## When another tool is the better choice - **A star chart for a README, and nothing else.** star-history.com draws it from GitHub's stargazer list on demand, with nothing to run and nothing to store. Since July 2026 GitHub serves that list only to a repository's admins and collaborators ([its starring page](https://docs.github.com/en/rest/activity/starring#new-access-restrictions)), and how star-history.com draws a chart for anyone else is not stated in the source pinned above. ghchronicle's own card is a side feature and needs a token and a run. - **Traffic for a few repositories, kept in git, with no database.** github-repo-stats is a daily workflow with one job per repository, commits its data to a branch and publishes an HTML and PDF report. ghchronicle needs a store somewhere it can reach. - **Traffic history without running anything.** repohistory is a hosted dashboard behind a GitHub App, free for one repository. - **A current count in a Prometheus you already run.** githubexporter/github-exporter and grafana/github-traffic are one container each and are scraped like any other target. If a current count is all you want, they are the smaller thing to run. - **Self-hosted runners, organisation seats or admin statistics.** promhippie/github_exporter has collectors for all three, and ghchronicle collects none of them. - **Traffic on one machine, with no server at all.** KipHub Traffic keeps it in a JSON file and draws it in a desktop window, provided somebody runs it every fortnight. > **Read, not run** > > None of the eight other tools was run for this page. Every cell in their rows > is what their own README, documentation or source says, read on 2026-09-24, > and none of it is a measurement. What each one costs in API calls, how it > behaves on an account of fifty repositories, and how its figures compare > with ghchronicle's for the same day were not measured. ## Where to go next - [Short answers](https://docs.jmrp.io/ghchronicle/start/questions/): traffic past fourteen days, stars over time, Prometheus, organisations, in a paragraph each. - [Dating a point](https://docs.jmrp.io/ghchronicle/how/dating/): the rule behind the fourth column. - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/): what each of ghchronicle's destinations keeps. --- # Short answers Twelve questions about keeping GitHub history with ghchronicle: traffic past 14 days, stars over time, backfill, Prometheus, organisations and cost. Source: https://docs.jmrp.io/ghchronicle/start/questions/ Each answer below is the short form of a page that has the detail, and links to it. Where an answer quotes GitHub rather than this project, it links to GitHub's own documentation. ## How do I keep GitHub traffic data longer than 14 days? Collect it more often than every fourteen days and store each day under its own date. GitHub's [traffic API](https://docs.github.com/en/rest/metrics/traffic) serves views and clones per day for the last fourteen days only. ghchronicle's `traffic` family rereads the whole window every six hours by default and writes each day at that day's date, so the rows converge rather than pile up, and a collector that was down for a week loses nothing. Older days cannot be recovered: GitHub never kept them. The detail is in [what is collected](https://docs.jmrp.io/ghchronicle/collectors/#audience) and [dating a point](https://docs.jmrp.io/ghchronicle/how/dating/). ## How do I track GitHub stars over time? Read GitHub's daily star history, which gives the stars a repository gained on each day, back to the week it was created, to anyone who can see the repository. ghchronicle's `stars` family reads it whole on its first sweep, for every repository it collects, and writes each day as a `gh_star_day` point, so the curve starts at the repository's first star, not on the day it was installed, and a repository whose stargazer list GitHub hides from the token still gets its daily curve. Where the token may read that list, the family also walks it with the star media type and writes each star as a `gh_star` point, dated to the second and naming who gave it. Since July 2026 [GitHub serves the list only to a repository's admins and collaborators](https://docs.github.com/en/rest/activity/starring#new-access-restrictions). The token always has that access on the account's own repositories and on those of an organisation its user administers. After the first sweep, ghchronicle asks for the newest thirty weeks of each history, one request per repository and usually a free 304, and the newest hundred stars of each list it may read, ten repositories to a GraphQL query. The history is GitHub's [`stargazers/history`](https://docs.github.com/en/rest/activity/starring#get-repository-star-history) endpoint: stars per day grouped by week, thirty weeks to a page, paging back to the repository's first week, thirteen pages for `cli/cli`, back to 2019. Its days are Pacific calendar days, and it counts today's stargazers, so an unstar takes a star off the day it was given. See [what is collected](https://docs.jmrp.io/ghchronicle/collectors/#audience) and [what GitHub will not give](https://docs.jmrp.io/ghchronicle/api/limits/). ## Can I collect GitHub history from before I installed it? Mostly, with one deliberate run of `ghchronicle -config config.yaml -backfill`. It walks every enabled family until the API runs out, or back to the bound `-backfill-since` sets, waits for the rate limit to reset instead of skipping, and, stopped half way, carries on from its checkpoint when run again. Three things no backfill reaches: traffic older than fourteen days, the event feed past its three hundred events or thirty days, and job logs older than ninety days, joined from 1 October 2026 by workflow runs past the repository's retention period. The walk, its checkpoint and its bound are on [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/). ## Can I archive GitHub notifications and the event feed? Yes, as long as the collector runs while they are still there. The `events` family keeps the account's activity feed, which GitHub serves up to three hundred events and none older than thirty days, and `notifs` keeps the inbox, which GitHub says holds notifications for three months unless they are saved. Both run every fifteen minutes by default, each item a point dated when it happened, and a Loki sink turns them into log lines. GitHub states both windows in its documentation on [the event feed](https://docs.github.com/en/rest/activity/events) and [the inbox](https://docs.github.com/en/subscriptions-and-notifications/concepts/about-notifications#notification-retention-policy). The families are in [what is collected](https://docs.jmrp.io/ghchronicle/collectors/#activity-and-cost), and the log lines in [Loki](https://docs.jmrp.io/ghchronicle/sinks/loki/). ## How long does GitHub keep workflow run history? From 1 October 2026, ninety days by default, according to GitHub's documentation. Logs and artifacts already had that default; from that date the repository's retention setting also deletes workflow runs, checks and commit statuses, which until then were kept for 400 days or more. A public repository can keep them between one and ninety days, a private one up to 400. ghchronicle's `actions` family writes every run with its jobs and steps, dated when it finished. GitHub's statement is under [retention for checks, workflow runs and artifacts](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository#configuring-the-retention-period-for-checks-workflow-runs-commit-statuses-artifacts-and-logs-in-your-repository). A [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/) walks every run GitHub still holds, back to `-backfill-since` when one is set, and expands each into its jobs, and into its steps while GitHub still serves them; from 1 October 2026 it can reach no further back than the repository's retention period, ninety days by default. ## Can Prometheus store ghchronicle's history? No. Prometheus stamps a sample when it is scraped and refuses one much older: measured against Prometheus 3.14 with a thirty-minute out-of-order window, a sample dated two days back came back as HTTP 400. So the Prometheus exporter, and the OpenTelemetry sink with `raw: false`, serve current values reduced from the dated rows, which suits alerting. The history belongs in InfluxDB, PostgreSQL, Graphite or Elasticsearch, alongside Prometheus if you run both. The measurement and the reduction are on [dating a point](https://docs.jmrp.io/ghchronicle/how/dating/#what-prometheus-structurally-cannot-hold). ## Which database should I store GitHub metrics in? One that keys a row by its timestamp, because that is what lets the same fourteen days be rewritten on every sweep without duplicates. InfluxDB, PostgreSQL, Graphite and Elasticsearch all keep the dated history, and ghchronicle generates a Grafana dashboard for each of them and for Prometheus, which keeps current values only. Running more than one sink is the normal arrangement: a history store for the charts, Prometheus for alerts. Each store is weighed on [choosing a store](https://docs.jmrp.io/ghchronicle/sinks/), and the dashboards are on [importing](https://docs.jmrp.io/ghchronicle/dashboards/). ## Why does stats/code_frequency always return 202? On a personal account, GitHub answers `stats/code_frequency` and `stats/contributors` with 202 and an empty body indefinitely. A 202 normally means the numbers are still being computed and a later request will get them; for these two, the next answer is another 202. ghchronicle does not call them: lines added and removed come from the commits collector instead, per commit, attributed to an author and dated to the commit. The ten-second check and the other endpoints that do not answer are on [what GitHub will not give](https://docs.jmrp.io/ghchronicle/api/limits/). ## Why does the GitHub traffic API return 403 with my token? Because traffic is gated on push access to the repository, not on being able to read it. GitHub [serves views and clones](https://docs.github.com/en/rest/metrics/traffic) only to someone who can push to the repository, and a fine-grained token also needs the repository permission [Administration (read)](https://docs.github.com/en/rest/metrics/traffic#get-repository-clones--fine-grained-access-tokens); without both, the answer is a 403. A workflow's automatic `GITHUB_TOKEN` cannot be granted Administration, so it gets a 403 even in the repository it runs in. ghchronicle records the 403 as unavailable and carries on, so the symptom is an empty traffic panel, not a failed sweep. Every scope and what its absence looks like is on [the token](https://docs.jmrp.io/ghchronicle/start/token/). ## Does it work with a GitHub organisation? For its repositories, yes: name the organisation under `targets.orgs` and its repositories are discovered and swept like the account's own, which needs the `read:org` scope. The account-wide families, the contribution calendar, the event feed, notifications, billing, packages and the outbound ones, need `targets.user` and describe a user. Organisation-only surfaces such as the audit log are not collected: the only organisation endpoint ghchronicle calls is the repository list. The keys are on [targets](https://docs.jmrp.io/ghchronicle/configuration/targets/), and what an organisation has that a personal account cannot see is on [what GitHub will not give](https://docs.jmrp.io/ghchronicle/api/limits/). ## Can it run in GitHub Actions without a server? Yes, apart from the store it writes to. The repository ships a composite Action, `jmrplens/ghchronicle@v2`, which downloads a release binary and runs one sweep, a backfill or a card render. It needs a personal access token, because the automatic `GITHUB_TOKEN` lacks push access to other repositories, [cannot read traffic even in its own](https://docs.github.com/en/rest/metrics/traffic#get-repository-clones--fine-grained-access-tokens), and is not a user. A hosted runner keeps no state file between runs, so unless the state is cached, which needs a configuration file, every run is a first run: it collects every family whatever its cadence, walks the stargazers, the whole star history and the account's co-authored pull requests again, and pays for every answer in full, with no ETag to be answered a free 304. The modes, the inputs and the cache step are on [GitHub Actions](https://docs.jmrp.io/ghchronicle/install/actions/). ## Will it use up my GitHub API rate limit? Not by default. The collector never spends the last 500 calls of a budget, or a fifth of the bucket's limit when that is smaller, and a sweep skips a family with a warning rather than crossing that line, so whatever else shares the token keeps working. Every response is cached by its ETag, and a 304 Not Modified costs no quota at all, which is what makes short cadences affordable. The brake is on [rate limits](https://docs.jmrp.io/ghchronicle/api/), and every family is priced on [cost of a sweep](https://docs.jmrp.io/ghchronicle/api/cost/). ## Where to go next - [Compared with the alternatives](https://docs.jmrp.io/ghchronicle/start/compared/): KipHub Traffic, repohistory, github-repo-stats, star-history and the Prometheus exporters, cell by cell. - [Troubleshooting](https://docs.jmrp.io/ghchronicle/reference/troubleshooting/): the messages that look like errors and are not, and the ones that are. --- # Ways to install Ways to install ghchronicle: a page per operating system, five ways to get the binary running, and what each one is good for. Source: https://docs.jmrp.io/ghchronicle/install/ One static binary with no runtime dependencies. How it gets onto the machine is the only decision, and it follows from where you want it to run. ## Your system, from download to service Releases carry binaries for Linux, macOS and Windows, on amd64 and arm64. The three pages below are the same walk for each system, taken all the way: which archive, how to check it, where the binary goes, how to run it once, and how to keep it running once you are done watching it. - [Linux](https://docs.jmrp.io/ghchronicle/install/linux/): A tar.gz, /usr/local/bin, and a hardened systemd unit. - [macOS](https://docs.jmrp.io/ghchronicle/install/macos/): The darwin archive, the quarantine attribute, and a launchd agent or daemon. - [Windows](https://docs.jmrp.io/ghchronicle/install/windows/): A zip, PowerShell and cmd, a scheduled task, and what is genuinely different there. ## Getting the binary - **Script** ```sh curl -fsSL https://raw.githubusercontent.com/jmrplens/ghchronicle/main/install.sh | bash ``` On Windows, in PowerShell: ```powershell irm https://raw.githubusercontent.com/jmrplens/ghchronicle/main/install.ps1 | iex ``` Either one works out the platform, takes the newest release, and refuses to install anything whose checksum is not the one the release published. - **Release** Take the archive for your platform from the [releases page](https://github.com/jmrplens/ghchronicle/releases). Archives are built for Linux, macOS and Windows, on amd64 and arm64, and ship as `tar.gz` (`zip` on Windows). ```sh tar -xzf ghchronicle_2.6.5_linux_amd64.tar.gz sudo install -m 755 ghchronicle /usr/local/bin/ ``` - **Go** ```sh go install github.com/jmrplens/ghchronicle/v2/cmd/ghchronicle@latest ``` Needs a Go toolchain, and builds from source at whatever the newest tag is. The `/v2` is part of the path: without it Go settles on 1.0.0, the last release whose module path it accepts, and says nothing. - **Container** ```sh docker run -v $PWD/config.yaml:/config.yaml:ro \ -v ghchronicle-state:/var/lib/ghchronicle -e GITHUB_TOKEN \ ghcr.io/jmrplens/ghchronicle -config /config.yaml ``` Distroless, static, and running as uid 65532. With `state_file` under `/var/lib/ghchronicle` the volume keeps the state, and it belongs to that user from the start, since the image's own directory does: [what has to be writable](https://docs.jmrp.io/ghchronicle/install/docker/#what-has-to-be-writable). - **Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: once config: .github/ghchronicle.yaml ``` A composite Action that downloads a release binary, so the runner needs no Go toolchain. ## Choosing where it runs The collector pushes to every store it supports, so it does not need to be reachable from anywhere. It needs to reach GitHub and to reach its databases. That is the only constraint, and it is what makes all four of these viable. Three of the four want a host you own: systemd, Docker and cron. GitHub Actions is the one that does not. The system pages above are where the scheduler lives, one per system: systemd or cron on Linux, launchd on macOS, a scheduled task on Windows. - [systemd](https://docs.jmrp.io/ghchronicle/install/systemd/): A long-running service on a host you own. The default choice: the state file persists, the schedule is the tool's own, and the unit can be locked down hard. - [Docker](https://docs.jmrp.io/ghchronicle/install/docker/): The same thing in a container. Mount a writable volume for the state file and any file sink. - [GitHub Actions](https://docs.jmrp.io/ghchronicle/install/actions/): No host at all. Good for the card and for a scheduled sweep; the state file does not survive between runs unless you cache it. - [cron](https://docs.jmrp.io/ghchronicle/install/systemd/#cron-instead-of-a-service): `-once` runs a single sweep and exits, which is all a scheduler needs. Keep the state file on a persistent path. ## The three run modes | Command | Does | | ------------------------------------------- | ---------------------------------------------------------- | | `ghchronicle -config config.yaml` | Runs forever, each family on its own cadence | | `ghchronicle -config config.yaml -once` | One sweep, then exits | | `ghchronicle -config config.yaml -backfill` | Walks every surface to its end, waiting for the rate limit | Plus two that write nothing: `-list` prints the repositories in scope, and `-card ... -card-only` renders the SVG without touching a database. > **The state file is the one thing that must persist** > > Whichever way it runs, keep `state_file` on a path that survives a restart, > in a directory the collector can write, since the cache file and the write > ledger live beside it. Without it every run collects every family whatever > its cadence says, and walks the stargazer list, the whole star history and > the co-authored pull requests again. ## Upgrading A new release goes over the old one the way the old one was installed, and the configuration and the state carry over. What the first hours after it look like, from 2.5.x to 2.6.x and to 2.6.1, is on [upgrading](https://docs.jmrp.io/ghchronicle/install/upgrading/). --- # Linux The whole path on Linux: the right archive, the signature, the PATH, a service, and building it yourself. Source: https://docs.jmrp.io/ghchronicle/install/linux/ One static binary, `CGO_ENABLED=0`, so there is no C library to install and no distribution to match. A release archive and a `PATH` entry is the whole install; everything below that is about doing it deliberately. ## The one-line install ```sh curl -fsSL https://raw.githubusercontent.com/jmrplens/ghchronicle/main/install.sh | bash ``` It works out the architecture, takes the newest release, checks the archive against the checksum that release published, and puts the binary in `/usr/local/bin` when that is writable. When it is not, it takes the first of `~/.local/bin` and `~/bin` that exists and is already on your `PATH`, and falls back to `~/.local/bin` when neither is. A checksum that does not match stops it without installing anything, and there is no flag to skip that step. When `cosign` 2.4.2 or newer is already on the machine it also verifies that the checksum file itself came from the release workflow, and a signature cosign does not confirm stops it, with what cosign said. An older cosign cannot read the signature bundle, so then, as with no cosign at all, it says that only the checksum was verified. It ends by running what it installed, so the version you see is the file it just wrote, and it says so when another `ghchronicle` earlier in your `PATH` still wins the name. Pin a version, or choose where it goes: ```sh curl -fsSL https://raw.githubusercontent.com/jmrplens/ghchronicle/main/install.sh | VERSION=2.6.5 BIN_DIR=~/bin bash ``` > **Reading it first is the whole point of a short script** > > Piping anything into a shell means running whatever that URL answers with > today. `curl -fsSL -o install.sh`, read it, then `bash install.sh`: it > is the same script, and it is short enough to read. The rest of this page is that same install done by hand, which is what to follow when you want to know exactly what landed where, or when you would rather not run a script you did not write. ## Pick the archive Release archives are named `ghchronicle__linux_.tar.gz`, with `` either `amd64` or `arm64`. `uname -m` answers which. | `uname -m` reports | The archive to take | | ------------------ | ------------------- | | `x86_64` | `linux_amd64` | | `aarch64` | `linux_arm64` | ```sh VERSION=2.6.5 arch=$(uname -m); case "$arch" in x86_64) arch=amd64 ;; aarch64) arch=arm64 ;; esac base=https://github.com/jmrplens/ghchronicle/releases/download/v$VERSION curl -fsSLO "$base/ghchronicle_${VERSION}_linux_${arch}.tar.gz" ``` > **Name the version rather than asking for the newest** > > The repository publishes a moving major tag beside the numbered releases, > because the Action is listed on the Marketplace and that listing needs one. > That tag's release carries **no files at all**, so never build a download URL > from the major tag: there is nothing behind it to download. Take the version > from the > [releases page](https://github.com/jmrplens/ghchronicle/releases) and write > it down, as above. ## Check what you downloaded Two files are published beside the archives: `checksums.txt`, which holds the SHA-256 of every archive, and `checksums.txt.sigstore.json`, which is a signature over that file. So the chain is one signature and a digest: verify the file, then verify the archive against the file. 1. Take the checksum file and its signature. ```sh curl -fsSLO "$base/checksums.txt" curl -fsSLO "$base/checksums.txt.sigstore.json" ``` 2. Check the archive against it. `--ignore-missing` is what lets one line of a twelve-line file be checked without the other eleven archives being present. ```sh sha256sum --ignore-missing -c checksums.txt ``` ```text ghchronicle_2.6.5_linux_amd64.tar.gz: OK ``` 3. Check the checksum file itself, if you have [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) 2.4.2 or newer. An older one cannot read the bundle and fails whatever the file says. ```sh cosign verify-blob \ --certificate-identity-regexp 'https://github.com/jmrplens/ghchronicle/.github/workflows/release.yml@refs/tags/.*' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --bundle checksums.txt.sigstore.json \ checksums.txt ``` ```text Verified OK ``` The signing is keyless: there is no public key to fetch and no private key for anyone to lose, because the identity being verified is the workflow that ran, recorded in a public transparency log. That is what the two `--certificate` flags say, and why they are not optional: without them cosign would confirm that somebody signed the file, which is not the question. Then let it write the configuration: ```sh ghchronicle -setup ``` It asks for a token and where the numbers go, checks each answer against the thing it names, and writes a `config.yaml` and, if you want one, a systemd unit. The installer above offers to run it as its last step. ## Put it on the PATH The archive holds three files and no directory, so extract it somewhere you meant to. - ghchronicle_VERSION_linux_ARCH.tar.gz - ghchronicle the binary - LICENSE - README.md - **For everyone** ```sh tar -xzf "ghchronicle_${VERSION}_linux_${arch}.tar.gz" ghchronicle sudo install -m 755 ghchronicle /usr/local/bin/ ghchronicle -version ``` - **For one account** ```sh mkdir -p ~/.local/bin tar -xzf "ghchronicle_${VERSION}_linux_${arch}.tar.gz" -C ~/.local/bin ghchronicle chmod 755 ~/.local/bin/ghchronicle ghchronicle -version ``` `~/.local/bin` is on the `PATH` of most distributions already. If `ghchronicle -version` answers "command not found", it is not on yours. `-version` answers with one line: the release number, then the commit and the build date of that release. ```text ghchronicle 2.6.5 (commit , built ) ``` ## Run it once The binary needs a configuration file and a token, and [the quickstart](https://docs.jmrp.io/ghchronicle/start/quickstart/) writes both in six steps. With those in place: ```sh ghchronicle -config config.yaml -list # what would be collected ghchronicle -config config.yaml -once # one sweep, then exit ``` ## Keep it running [systemd](https://docs.jmrp.io/ghchronicle/install/systemd/) is the arrangement this documentation treats as the default on Linux, and the unit there is hardened rather than minimal, because this is very likely the only process on the host holding a GitHub token with read access to every repository of an account. A service started on a state file nothing has written yet brings its eleven families of six hours or more into the store over its first two and a half hours, one a sweep, and only that once; the systemd page says why. A scheduler works too: [cron with `-once`](https://docs.jmrp.io/ghchronicle/install/systemd/#cron-instead-of-a-service) is a single line, at the cost of giving every family the same cadence. ## Build it from source Go 1.27.2 or newer is what the module declares. Nothing else is needed: the build sets `CGO_ENABLED=0`, so there is no compiler and no header package to find. - **go install** ```sh go install github.com/jmrplens/ghchronicle/v2/cmd/ghchronicle@latest ``` Lands in `$(go env GOPATH)/bin`, which is `~/go/bin` unless you moved it, and that directory has to be on your `PATH`. A binary built this way has no commit or build date to report: those two are stamped by the release build and by nothing else, and a module downloaded through the proxy carries no checkout to read them from. `-version` names what the build does record instead, the module version the go command fetched and the Go release that compiled it: ```text ghchronicle 2.6.5 (module v2.6.5, built with ) ``` The first version comes from the `VERSION` file the module embeds. - **make** ```sh git clone https://github.com/jmrplens/ghchronicle cd ghchronicle make build # into bin/ghchronicle make install # into GOBIN, stamped like a release build ``` `make build` stamps the version, the short commit and the commit date, so `-version` says exactly which tree it came from. `make` with no target lists every target there is. ## Where the files go The collector looks for a configuration file in no particular place: `-config` defaults to `config.yaml` **relative to the working directory**, and there is no search path behind it. So the path is a decision you make once and then pass on every invocation. What the rest of this documentation assumes: | File | Path | Mode | | ------------------ | ---------------------------------- | --------------------------- | | Configuration | `/etc/ghchronicle/config.yaml` | world readable, no secrets | | Tokens | `/etc/ghchronicle/ghchronicle.env` | `600` | | State, ledger, cache | `/var/lib/ghchronicle/` | written by the service user | `state_file` has a default of its own, `ghchronicle-state.json` in the working directory, with the write ledger beside it as `ghchronicle-state-written.bin` and the cache as `ghchronicle-state-cache.bin`. That default is fine for a first run in a directory you made, and wrong for a service, whose working directory is not something to rely on. Set it. --- # macOS The whole path on macOS: the darwin archive, quarantine, a launchd agent or daemon, and building it yourself. Source: https://docs.jmrp.io/ghchronicle/install/macos/ The same static binary as everywhere else, built for `darwin` on both Apple silicon and Intel. macOS is not a build target that is merely compiled and hoped for: every change runs the whole unit suite and the end-to-end suite on a macOS runner, beside Linux and Windows. ## The one-line install ```sh curl -fsSL https://raw.githubusercontent.com/jmrplens/ghchronicle/main/install.sh | bash ``` It works out the architecture, takes the newest release, checks the archive against the checksum that release published, and puts the binary in `/usr/local/bin` when that is writable. When it is not, it takes the first of `~/.local/bin` and `~/bin` that exists and is already on your `PATH`, and falls back to `~/.local/bin` when neither is. A checksum that does not match stops it without installing anything, and there is no flag to skip that step. When `cosign` 2.4.2 or newer is already on the machine it also verifies that the checksum file itself came from the release workflow, and a signature cosign does not confirm stops it, with what cosign said. An older cosign cannot read the signature bundle, so then, as with no cosign at all, it says that only the checksum was verified. It ends by running what it installed, so the version you see is the file it just wrote, and it says so when another `ghchronicle` earlier in your `PATH` still wins the name. Pin a version, or choose where it goes: ```sh curl -fsSL https://raw.githubusercontent.com/jmrplens/ghchronicle/main/install.sh | VERSION=2.6.5 BIN_DIR=~/bin bash ``` > **Reading it first is the whole point of a short script** > > Piping anything into a shell means running whatever that URL answers with > today. `curl -fsSL -o install.sh`, read it, then `bash install.sh`: it > is the same script, and it is short enough to read. The rest of this page is that same install done by hand, which is what to follow when you want to know exactly what landed where, or when you would rather not run a script you did not write. ## Pick the archive Release archives say `darwin`, which is the name of the system the Go toolchain uses; macOS is the name Apple uses for the same thing. `uname -m` answers which architecture. | `uname -m` reports | The machine | The archive to take | | ------------------ | ------------------ | ------------------- | | `arm64` | Apple silicon | `darwin_arm64` | | `x86_64` | Intel | `darwin_amd64` | ```sh VERSION=2.6.5 arch=$(uname -m); case "$arch" in x86_64) arch=amd64 ;; esac base=https://github.com/jmrplens/ghchronicle/releases/download/v$VERSION curl -fsSLO "$base/ghchronicle_${VERSION}_darwin_${arch}.tar.gz" ``` > **Name the version rather than asking for the newest** > > The repository publishes a moving major tag beside the numbered releases, > because the Action is listed on the Marketplace and that listing needs one. > That tag's release carries **no files at all**, so never build a download URL > from the major tag: there is nothing behind it to download. Take the version > from the > [releases page](https://github.com/jmrplens/ghchronicle/releases) and write > it down, as above. ## Check what you downloaded Two files are published beside the archives: `checksums.txt`, which holds the SHA-256 of every archive, and `checksums.txt.sigstore.json`, which is a signature over that file. 1. Take the checksum file and its signature. ```sh curl -fsSLO "$base/checksums.txt" curl -fsSLO "$base/checksums.txt.sigstore.json" ``` 2. Check the archive against it. macOS ships `shasum` rather than the `sha256sum` of a Linux box, so the line for your archive is selected and piped in, which works whatever version of `shasum` the system has. ```sh grep "darwin_${arch}.tar.gz$" checksums.txt | shasum -a 256 -c - ``` ```text ghchronicle_2.6.5_darwin_arm64.tar.gz: OK ``` 3. Check the checksum file itself, if you have [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) 2.4.2 or newer. An older one cannot read the bundle and fails whatever the file says. ```sh cosign verify-blob \ --certificate-identity-regexp 'https://github.com/jmrplens/ghchronicle/.github/workflows/release.yml@refs/tags/.*' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ --bundle checksums.txt.sigstore.json \ checksums.txt ``` ```text Verified OK ``` The signing is keyless: the identity being verified is the workflow that ran, recorded in a public transparency log, which is why the two `--certificate` flags are not optional. Without them cosign would confirm that somebody signed the file, which is not the question. Then let it write the configuration: ```sh ghchronicle -setup ``` It asks for a token and where the numbers go, checks each answer against the thing it names, and writes a `config.yaml` and, if you want one, a launchd agent in `~/Library/LaunchAgents`. The installer above offers to run it as its last step. ## Put it on the PATH The archive holds three files and no directory. - ghchronicle_VERSION_darwin_ARCH.tar.gz - ghchronicle the binary - LICENSE - README.md - **For everyone** ```sh tar -xzf "ghchronicle_${VERSION}_darwin_${arch}.tar.gz" ghchronicle sudo install -m 755 ghchronicle /usr/local/bin/ ghchronicle -version ``` `/usr/local/bin` is on the default `PATH` of every macOS install, on both architectures, because `/etc/paths` lists it first. - **For one account** ```sh mkdir -p ~/bin tar -xzf "ghchronicle_${VERSION}_darwin_${arch}.tar.gz" -C ~/bin ghchronicle chmod 755 ~/bin/ghchronicle echo 'export PATH="$HOME/bin:$PATH"' >> ~/.zprofile ``` `~/bin` is not on the `PATH` by default, hence the last line. `zsh` is the login shell on every supported macOS. > **If macOS refuses to run it** > > The binary carries no Developer ID signature and is not notarized, so a copy > that arrives with the quarantine attribute is refused with "cannot be opened > because the developer cannot be verified". The attribute is not carried > inside the archive: it is set by whatever process writes a file, and > inherited by the processes that one starts. A browser writes the `.tar.gz` it > downloads with the mark on it, `curl` does not, and `tar -xzf` run from > Terminal writes an unmarked binary either way. The case that bites is opening > the archive in Finder, because Archive Utility passes the mark on to what it > extracts. So look before you clear anything: > > ```sh > xattr -l "ghchronicle_${VERSION}_darwin_${arch}.tar.gz" # what a browser marked > xattr -l ghchronicle # the extracted binary > xattr -c ghchronicle # clear it > ``` > > `xattr -c` clears every extended attribute and succeeds when there are none. > `xattr -d com.apple.quarantine` does not: on a binary extracted from Terminal > it stops with `No such xattr: com.apple.quarantine`, which looks like a > broken instruction and is only the attribute never having been there. ## Run it once The binary needs a configuration file and a token, and [the quickstart](https://docs.jmrp.io/ghchronicle/start/quickstart/) writes both in six steps. With those in place: ```sh ghchronicle -config config.yaml -list # what would be collected ghchronicle -config config.yaml -once # one sweep, then exit ``` ## Keep it running with launchd launchd is what macOS has instead of systemd, and the choice it asks you to make first is agent or daemon. | | A LaunchAgent | A LaunchDaemon | | ---------- | ------------------------- | -------------------------- | | Lives in | `~/Library/LaunchAgents/` | `/Library/LaunchDaemons/` | | Runs as | you | `root`, or the `UserName` you give it | | Runs when | you are logged in | the machine is up, from boot | | Good for | a laptop you use | a Mac that stays on | The agent is the one to start with. It needs no `sudo`, and a collector that stops while the laptop's owner is logged out loses nothing that the next sweep does not pick up. ```xml title="~/Library/LaunchAgents/io.jmrp.ghchronicle.plist" Label io.jmrp.ghchronicle ProgramArguments /usr/local/bin/ghchronicle -config /Users/you/Library/Application Support/ghchronicle/config.yaml EnvironmentVariables GITHUB_TOKEN github_pat_... RunAtLoad KeepAlive StandardOutPath /Users/you/Library/Logs/ghchronicle.log StandardErrorPath /Users/you/Library/Logs/ghchronicle.log ``` 1. Protect the file before it holds a token, then load it. ```sh chmod 600 ~/Library/LaunchAgents/io.jmrp.ghchronicle.plist launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/io.jmrp.ghchronicle.plist ``` 2. See that it is up, and read what it says. ```sh launchctl print gui/$(id -u)/io.jmrp.ghchronicle tail -f ~/Library/Logs/ghchronicle.log ``` 3. After editing the file, unload it and load it again. launchd reads the property list once, when it is bootstrapped. ```sh launchctl bootout gui/$(id -u)/io.jmrp.ghchronicle launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/io.jmrp.ghchronicle.plist ``` > **The token sits in the property list** > > A launchd job does not read your shell profile, so `EnvironmentVariables` is > where the token has to be. That file is then the macOS equivalent of the > systemd environment file, and it deserves the same treatment: mode `600`, > never in a repository, and remembered when the token is rotated. `launchctl > setenv` is the alternative, and it is worse: it puts the token in every > process of the session. ### Or one sweep on a timer `StartInterval` is launchd's cron, in seconds, and `-once` is the mode that suits it. Replace `KeepAlive` with it and add `-once` to `ProgramArguments`: ```xml StartInterval 3600 ``` Keep `state_file` on a path that survives, in this mode above all. Without it every run collects every family, whatever its cadence, and walks the stargazer list, the whole star history and the co-authored pull requests again; the cache file beside it is what lets a run ask GitHub only for what changed. ## Build it from source Go 1.27.2 or newer is what the module declares. The build sets `CGO_ENABLED=0`, so the Xcode command line tools are not needed for it. - **go install** ```sh go install github.com/jmrplens/ghchronicle/v2/cmd/ghchronicle@latest ``` Lands in `$(go env GOPATH)/bin`, which is `~/go/bin` unless you moved it, and that directory has to be on your `PATH`. A binary built this way has no commit or build date to report: those two are stamped by the release build, and a module downloaded through the proxy carries no checkout to read them from. `-version` names what the build does record instead, the module version the go command fetched and the Go release that compiled it: ```text ghchronicle 2.6.5 (module v2.6.5, built with ) ``` - **make** ```sh git clone https://github.com/jmrplens/ghchronicle cd ghchronicle make build # into bin/ghchronicle make install # into GOBIN, stamped like a release build ``` The Makefile is written for GNU Make 3.81, which is the version macOS ships, so the stock `make` runs it. ## Where the files go The collector looks for a configuration file in no particular place: `-config` defaults to `config.yaml` **relative to the working directory**, and there is no search path behind it. macOS has no `/etc/ghchronicle` habit, so these are the conventional places rather than ones the tool knows: | File | For an agent | For a daemon | | ---------------- | --------------------------------------------------- | -------------------------------- | | Configuration | `~/Library/Application Support/ghchronicle/` | `/usr/local/etc/ghchronicle/` | | State, ledger, cache | `~/Library/Application Support/ghchronicle/` | `/usr/local/var/ghchronicle/` | | Log | `~/Library/Logs/ghchronicle.log` | `/usr/local/var/log/` | `state_file` has a default of its own, `ghchronicle-state.json` in the working directory, with the write ledger beside it as `ghchronicle-state-written.bin` and the cache as `ghchronicle-state-cache.bin`. A launchd job's working directory is not something to rely on. Set it. From 2.6.1 on a path setting takes a leading `~` as the home directory, so an agent's configuration can say `state_file: ~/Library/Application Support/ghchronicle/state.json`; an older release reads the `~` as a directory name, and wants `/Users/you/...` written out. --- # Windows The whole path on Windows: the zip, PowerShell and cmd, a scheduled task, and what is genuinely different there. Source: https://docs.jmrp.io/ghchronicle/install/windows/ Windows is a released platform, not an afterthought: every change runs the whole unit suite and the end-to-end suite on a Windows runner, and the source carries Windows-only code where the system behaves differently. What follows is written for PowerShell, with the `cmd` form beside it wherever the two differ. ## The one-line install ```powershell irm https://raw.githubusercontent.com/jmrplens/ghchronicle/main/install.ps1 | iex ``` It works out the architecture, takes the newest release, checks the archive against the checksum that release published, and puts `ghchronicle.exe` under `%LOCALAPPDATA%\Programs\ghchronicle`. A checksum that does not match stops it without installing anything, and there is no switch to skip that step. When `cosign` 2.4.2 or newer is already on your PATH it also verifies that the checksum file itself came from the release workflow, the same check `install.sh` makes, and a signature cosign does not confirm stops it, with what cosign said, so a machine that cannot reach Sigstore is not taken for a forged file. An older cosign cannot read the signature bundle, so then, as without `cosign` at all, it says that only the checksum was verified. It then adds that directory to **your** PATH, the per-user one, so `ghchronicle` works by name. Programs already open keep the PATH they started with, so open a new terminal. Windows reads the machine PATH before the per-user one, so another `ghchronicle` installed for everyone, or an older `go install` build, can keep answering to the name after this finishes; the script says so when it finds one. Nothing is written outside your profile and no elevation is asked for: a machine-wide install belongs to an installer with a UAC prompt, not to a script read off the network. To pin a version, choose the directory, or leave the PATH alone, the script takes parameters, which needs the slightly longer form because `iex` has nowhere to put them: ```powershell & ([scriptblock]::Create((irm https://raw.githubusercontent.com/jmrplens/ghchronicle/main/install.ps1))) -Version 2.6.5 -BinDir C:\tools -NoPathUpdate ``` > **Reading it first is the whole point of a short script** > > `irm ... | iex` runs whatever that URL answers with today. `irm https://raw.githubusercontent.com/jmrplens/ghchronicle/main/install.ps1 -OutFile install.ps1`, > read it, then `.\install.ps1`: it is the same script, and it is short enough > to read. The rest of this page is that same install done by hand, which is what to follow when you want to know exactly what landed where, or when you would rather not run a script you did not write. Then let it write the configuration: ```powershell ghchronicle -setup ``` It asks for a token and where the numbers go, checks each answer against the thing it names, and writes a `config.yaml` under `%APPDATA%\ghchronicle` and, if you want one, a scheduled task you register with `schtasks`. ## What is different here Read this first. Four of the five lines below are the reason a command copied from a Linux page does not work. - **The program** is `ghchronicle.exe`. Typing `ghchronicle` finds it, because `PATHEXT` lists `.EXE`. - **Runtime dependencies**: none. The binary is built `CGO_ENABLED=0`, so there is no C runtime to install. - **Stopping it** is Ctrl+C in its console. There is no signal to send it, and `taskkill /F` ends it without closing its sinks. - **Paths in the configuration**: a backslash inside a double-quoted YAML scalar is an escape. Use forward slashes. - **Running unattended** is a scheduled task. The binary is not a Service Control Manager service. ## Pick the archive Windows archives are `zip` rather than `tar.gz`, and they are named `ghchronicle__windows_.zip`, with `` either `amd64` or `arm64`. | `$env:PROCESSOR_ARCHITECTURE` | The archive to take | | ----------------------------- | ------------------- | | `AMD64` | `windows_amd64` | | `ARM64` | `windows_arm64` | That variable describes the **process**, not the machine. A 32-bit PowerShell on a 64-bit machine reports `x86` and leaves the machine's own architecture in `$env:PROCESSOR_ARCHITEW6432`; an x64 PowerShell running under emulation on an ARM64 machine reports `AMD64` and sets nothing else, which is the case that quietly hands you the wrong archive. If either could be you, ask the machine rather than the shell: ```powershell (Get-CimInstance Win32_Processor).Architecture # 9 is x64, 12 is ARM64 ``` ```powershell $version = "2.6.5" $arch = if ($env:PROCESSOR_ARCHITECTURE -eq "ARM64") { "arm64" } else { "amd64" } $base = "https://github.com/jmrplens/ghchronicle/releases/download/v$version" $zip = "ghchronicle_${version}_windows_${arch}.zip" Invoke-WebRequest -UseBasicParsing -Uri "$base/$zip" -OutFile $zip ``` `-UseBasicParsing` is not decoration. On Windows PowerShell 5.1 `Invoke-WebRequest` builds its result through the Internet Explorer engine unless told not to, and fails outright on a host where Internet Explorer was removed or never went through its first-run configuration, which covers Server Core and most hardened images. On PowerShell 7 the switch is accepted and does nothing. > **Name the version rather than asking for the newest** > > The repository publishes a moving major tag beside the numbered releases, > because the Action is listed on the Marketplace and that listing needs one. > That tag's release carries **no files at all**, so never build a download URL > from the major tag: there is nothing behind it to download. Take the version > from the > [releases page](https://github.com/jmrplens/ghchronicle/releases) and write > it down, as above. ## Check what you downloaded `checksums.txt` holds the SHA-256 of every archive, and `checksums.txt.sigstore.json` is a signature over that file. 1. Take the checksum file and compare the one line that is yours. ```powershell Invoke-WebRequest -UseBasicParsing -Uri "$base/checksums.txt" -OutFile checksums.txt $expected = (Select-String -Path checksums.txt -Pattern ([regex]::Escape($zip) + '$')).Line.Split(" ")[0] $actual = (Get-FileHash -Algorithm SHA256 -Path $zip).Hash if ($actual -eq $expected) { "OK" } else { "MISMATCH" } ``` `Get-FileHash` returns the digest in upper case and `checksums.txt` holds it in lower case. They still compare equal because PowerShell's `-eq` on two strings ignores case, which is the one place here where that default is convenient rather than a trap. 2. Check the checksum file itself, if you have [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) 2.4.2 or newer; an older one cannot read the bundle and fails whatever the file says. The command is the same one the release notes print, and the same one a Linux or macOS reader runs. ```powershell cosign verify-blob ` --certificate-identity-regexp 'https://github.com/jmrplens/ghchronicle/.github/workflows/release.yml@refs/tags/.*' ` --certificate-oidc-issuer https://token.actions.githubusercontent.com ` --bundle checksums.txt.sigstore.json ` checksums.txt ``` The backtick is PowerShell's line continuation, where a shell script uses a backslash. ## Put it somewhere and on the PATH The archive holds three files and no directory, so unpack it into a directory you made. - ghchronicle_VERSION_windows_ARCH.zip - ghchronicle.exe the binary - LICENSE - README.md - **For everyone** An elevated PowerShell, because both the directory and the machine `Path` need administrator rights. ```powershell $dir = "C:\Program Files\ghchronicle" Expand-Archive -Path $zip -DestinationPath $dir -Force $machine = [Environment]::GetEnvironmentVariable("Path", "Machine") [Environment]::SetEnvironmentVariable("Path", "$machine;$dir", "Machine") ``` - **For one account** No elevation needed, and nothing outside your profile is touched. ```powershell $dir = "$env:LOCALAPPDATA\Programs\ghchronicle" Expand-Archive -Path $zip -DestinationPath $dir -Force $user = [Environment]::GetEnvironmentVariable("Path", "User") [Environment]::SetEnvironmentVariable("Path", "$user;$dir", "User") ``` > **Two traps in writing Path back** > > Both snippets read `Path` from the scope they write to, never from > `$env:Path`. `$env:Path` is the process's own copy, which Windows built by > joining the machine list and the user list: write that back into either scope > and you have copied the other one into it, permanently, and it grows again > every time somebody repeats the command. > > The second trap belongs to the machine scope alone. > `[Environment]::GetEnvironmentVariable` expands `%SystemRoot%` and its > relatives while it reads, and `SetEnvironmentVariable` writes the result back > as a plain string, so a machine `Path` that held such entries comes back with > them baked in and its registry value changes kind from `REG_EXPAND_SZ` to > `REG_SZ`, for good. You cannot spot it in `$machine`, because the expansion > has already happened by then. Open System Properties, Environment Variables, > which shows the value unexpanded: if there is a `%VAR%` anywhere in the > machine `Path`, add the directory from that dialog rather than from the > snippet above. A new `Path` reaches only processes started afterwards, so open a new terminal before the check below. The current one keeps the environment it was given. ```powershell ghchronicle -version ``` It answers with one line: the release number, then the commit and the build date of that release. ```text ghchronicle 2.6.5 (commit , built ) ``` > **If Windows warns about the file** > > The binary carries no Authenticode signature: the release signs the checksum > file and the SBOMs, and nothing else. SmartScreen and Defender treat an > unsigned executable downloaded from the internet on their own terms, which > vary by version and policy and which this page will not guess at. If Windows > refuses to start it, the attribute to clear is the mark of the web: > > ```powershell > Unblock-File -Path "$dir\ghchronicle.exe" > ``` ## The token, and the rest of the environment A `${VAR}` in a credential, an address or a file path of the configuration file is read from the environment when the process starts, so the token never has to be in the file. - **PowerShell** ```powershell # This window only $env:GITHUB_TOKEN = "github_pat_..." # Persisted for this account [Environment]::SetEnvironmentVariable("GITHUB_TOKEN", "github_pat_...", "User") ``` - **cmd** ```bat rem This window only set GITHUB_TOKEN=github_pat_... rem Persisted for this account, and NOT visible in this window setx GITHUB_TOKEN "github_pat_..." ``` > **A persisted variable is a token in the registry** > > `SetEnvironmentVariable` with `User` or `Machine`, and `setx`, write to the > registry in clear text, where anything running as that account can read them. > That is the same bargain as a systemd environment file, without the file mode > to lean on. A scheduled task running as `SYSTEM` reads the machine scope, so > a token put there is readable by every service on the box: prefer the user > scope and a task that runs as that user. ## Run it once The binary needs a configuration file, and [the quickstart](https://docs.jmrp.io/ghchronicle/start/quickstart/) writes one in six steps. Save it as UTF-8, and mind the two Windows details below it. ```powershell ghchronicle -config config.yaml -list # what would be collected ghchronicle -config config.yaml -once # one sweep, then exit ``` If the binary is in the current directory rather than on the `PATH`, PowerShell needs it named as a path: `.\ghchronicle.exe`. A bare name is a command, and the current directory is not searched for commands. ### Paths in the configuration file YAML treats a backslash as an escape inside a **double-quoted** scalar and as an ordinary character everywhere else. So a Windows path in double quotes is not the path you wrote, and usually not valid YAML either: ```yaml state_file: "C:\ghchronicle\state.json" # ghchronicle: config.yaml: yaml: line N: found unknown escape character state_file: C:\ghchronicle\state.json # correct, plain scalar state_file: 'C:\ghchronicle\state.json' # correct, single quoted state_file: C:/ghchronicle/state.json # correct, and the one to prefer ``` Forward slashes are the simplest answer: Windows accepts them in a path, and they survive being quoted whichever way. ### The file itself has to be UTF-8 Windows PowerShell 5.1, the one that ships in the box, writes neither of the things a YAML parser wants, and it writes a different wrong thing depending on how you ask. `>` and `Out-File` produce UTF-16LE, which the parser reads as binary. `Set-Content` produces the system's active code page, usually ANSI, which parses while the file is pure ASCII and mangles the first accented character in it. PowerShell 7 defaults to UTF-8 without a byte order mark and has neither problem; on 5.1, be explicit: ```powershell Set-Content -Path config.yaml -Value $text -Encoding utf8 ``` `utf8` on 5.1 means UTF-8 **with** a byte order mark, which the value cannot express and the cmdlet does not warn about. The collector's parser reads past one, so the file works; a tool that reads the first bytes for itself may not. ## Keep it running There is no service mode. The collector is a console program: it does not talk to the Service Control Manager, so registering it with `sc.exe create` produces a service that Windows starts and then gives up on, reporting that it "did not respond to the start or control request in a timely fashion". Third-party service wrappers exist and this project neither ships nor tests one. The ordinary way to run something unattended on Windows is a scheduled task, and there are two shapes of it. - **One sweep on a timer** The Windows equivalent of cron, and the one to prefer: nothing has to be stopped, and a missed run costs one sweep. ```powershell $action = New-ScheduledTaskAction ` -Execute "C:\Program Files\ghchronicle\ghchronicle.exe" ` -Argument '-config "C:\ProgramData\ghchronicle\config.yaml" -once' $trigger = New-ScheduledTaskTrigger -Once -At (Get-Date) ` -RepetitionInterval (New-TimeSpan -Hours 1) ` -RepetitionDuration ([TimeSpan]::MaxValue) Register-ScheduledTask -TaskName ghchronicle -Action $action -Trigger $trigger ``` `-RepetitionDuration ([TimeSpan]::MaxValue)` is what spells out "for ever". Task Scheduler's own rule is that a repetition with no duration repeats indefinitely, so leaving it out is not a bug, but the cmdlet has no default of its own and the task is then registered carrying no duration at all. Two conditions come with this shape, and neither is cron's. An hourly task gives every family an hourly cadence at best, so the five that run every fifteen minutes, `actions`, `events`, `notifs`, `activity` and `ratelimit`, run four times less often, which is the same trade [cron makes on Linux](https://docs.jmrp.io/ghchronicle/install/systemd/#cron-instead-of-a-service). And `Register-ScheduledTask` here names no `-User` and no `-Principal`, so the task is registered under the calling account with the default logon type and runs **only while that account is logged on**. A cron job does not stop when you log out. For one that behaves the same way, register the task with a principal that has "run whether user is logged on or not" set, which is `New-ScheduledTaskPrincipal`. - **Running all the time** A task triggered at logon or at startup, with the collector left in its long-running mode so each family keeps its own cadence. ```powershell $action = New-ScheduledTaskAction ` -Execute "C:\Program Files\ghchronicle\ghchronicle.exe" ` -Argument '-config "C:\ProgramData\ghchronicle\config.yaml"' $trigger = New-ScheduledTaskTrigger -AtLogOn $settings = New-ScheduledTaskSettingsSet ` -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1) ` -ExecutionTimeLimit ([TimeSpan]::Zero) Register-ScheduledTask -TaskName ghchronicle -Action $action ` -Trigger $trigger -Settings $settings ``` `-ExecutionTimeLimit ([TimeSpan]::Zero)` is the one that matters: the default stops a task after three days, which for a process meant to run for ever is a restart nobody asked for. ```powershell Start-ScheduledTask -TaskName ghchronicle Get-ScheduledTaskInfo -TaskName ghchronicle # last run, last result ``` `Stop-ScheduledTask` ends the process rather than asking it to finish: it is `taskkill /F` by another name, and it closes no sink on its way out. In the timer shape there is nothing to stop, which is most of why it is the better default. > **Where the log goes** > > A task has no console, so the log has to be a file: set `log.file` in the > configuration. Standard output from a scheduled task is discarded, and > `Get-ScheduledTaskInfo` reports only the exit code. ## Build it from source Go 1.27.2 or newer, which is the version `go.mod` declares. `CGO_ENABLED=0` means no MSVC, no MinGW and no Windows SDK. - **go install** ```powershell go install github.com/jmrplens/ghchronicle/v2/cmd/ghchronicle@latest ``` Lands in `$(go env GOPATH)\bin`, which is `%USERPROFILE%\go\bin` unless you moved it, and that directory has to be on your `Path`. A binary built this way has no commit or build date to report: those two are stamped by the release build, and a module downloaded through the proxy carries no checkout to read them from. `-version` names what the build does record instead, the module version the go command fetched and the Go release that compiled it: ```text ghchronicle 2.6.5 (module v2.6.5, built with ) ``` - **From a checkout** ```powershell git clone https://github.com/jmrplens/ghchronicle cd ghchronicle go build -o ghchronicle.exe .\cmd\ghchronicle ``` `go build` rather than `make`: the Makefile is a GNU Makefile whose recipes are POSIX shell, so it wants Git Bash, MSYS2 or WSL. This one line is what `make build` does, minus the version stamping. ## Where the files go The collector looks for a configuration file in no particular place: `-config` defaults to `config.yaml` **relative to the working directory**, and there is no search path behind it. A scheduled task's working directory is not something to rely on, so give every path in the file absolutely. | File | For a machine-wide task | For one account | | ---------------- | -------------------------------- | ----------------------------------- | | Configuration | `C:/ProgramData/ghchronicle/` | `${LOCALAPPDATA}/ghchronicle/` | | State, ledger, cache | `C:/ProgramData/ghchronicle/` | `${LOCALAPPDATA}/ghchronicle/` | | Log | `C:/ProgramData/ghchronicle/` | `${LOCALAPPDATA}/ghchronicle/` | `${LOCALAPPDATA}` is written that way because `${VAR}` is the one form the collector expands, from the environment, as the process starts, and it does so in `state_file`, `sinks.dedupe_file` and `log.file` from 2.6.1 on. An older release expands no path, and reads `${LOCALAPPDATA}/ghchronicle/state.json` as a directory literally named `${LOCALAPPDATA}`; with one of those, write the path out. `%VAR%` is a shell notation and means nothing to the file: a `%LOCALAPPDATA%` copied into the YAML gives you a directory literally named `%LOCALAPPDATA%`, next to wherever the task happened to be working. It is `%LOCALAPPDATA%` in `cmd` and `$env:LOCALAPPDATA` in PowerShell that create the directory in the first place. The configuration's own path is the exception in the table: it is given on the command line, where the collector expands nothing, so a scheduled task's `-config` names it in full, `C:\Users\you\AppData\Local\ghchronicle\config.yaml`. A directory under `C:\ProgramData` is writable by its creator and readable by everyone, so create it elevated and then grant write to the account the task runs as. Leave the state file and the cache file beside it writable. The collector replaces both by writing a new file and renaming it over the old one, and NTFS refuses to replace a read-only file even when asked to replace it, so a read-only state file stops being saved and every sweep warns `state not saved`. The write ledger and the file sink's rotated files are the two it clears that attribute on itself, and a sweep does not fail over them. --- # systemd A hardened unit for the one process on the host that holds a GitHub token, and what each restriction is for. Source: https://docs.jmrp.io/ghchronicle/install/systemd/ ## The unit ```ini title="/etc/systemd/system/ghchronicle.service" [Unit] Description=ghchronicle, GitHub metrics collector After=network-online.target Wants=network-online.target [Service] Type=simple User=ghchronicle Group=ghchronicle EnvironmentFile=/etc/ghchronicle/ghchronicle.env ExecStart=/usr/local/bin/ghchronicle -config /etc/ghchronicle/config.yaml Restart=always RestartSec=30s # This is the only process on the host holding a GitHub token, so it gets # nothing it does not need. NoNewPrivileges=true PrivateTmp=true PrivateDevices=true ProtectSystem=strict ProtectHome=true ProtectKernelTunables=true ProtectKernelModules=true ProtectControlGroups=true ProtectClock=true ProtectHostname=true ProtectProc=invisible RestrictNamespaces=true RestrictRealtime=true RestrictSUIDSGID=true LockPersonality=true MemoryDenyWriteExecute=true SystemCallArchitectures=native SystemCallFilter=@system-service CapabilityBoundingSet= AmbientCapabilities= RestrictAddressFamilies=AF_INET AF_INET6 StateDirectory=ghchronicle ReadWritePaths=/var/lib/ghchronicle [Install] WantedBy=multi-user.target ``` ## Why it is hardened The threat model is short and it is the whole justification: **this is very likely the only process on the host holding a GitHub token with read access to every repository of an account.** A token is a bearer credential. Anything that can read this process's memory or its environment file has the account. So the unit gives the process exactly what it needs, which turns out to be almost nothing: an outbound TCP socket and one writable directory. | Directive | What it removes | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------ | | `CapabilityBoundingSet=`, `AmbientCapabilities=` | Every Linux capability. It binds no privileged port and owns no device | | `NoNewPrivileges=true` | Any path to gaining privileges through exec, including setuid binaries | | `ProtectSystem=strict` | Write access to the entire filesystem, except `ReadWritePaths` | | `ProtectHome=true` | Every home directory, which is where the interesting credentials on a host usually live | | `PrivateTmp=true`, `PrivateDevices=true` | Shared temporary files, and the physical device nodes | | `ProtectProc=invisible` | The ability to see other users' processes in `/proc`, so it cannot read another process's command line | | `RestrictAddressFamilies=AF_INET AF_INET6` | Unix and netlink sockets. It talks HTTPS and nothing else | | `MemoryDenyWriteExecute=true`, `LockPersonality=true` | The usual shellcode primitives | | `SystemCallFilter=@system-service` | Every syscall outside the ordinary service set, including the module and kernel-tuning ones | | `ProtectKernelTunables`, `ProtectKernelModules`, `ProtectControlGroups`, `ProtectClock`, `ProtectHostname`, `RestrictNamespaces`, `RestrictRealtime`, `RestrictSUIDSGID` | Every remaining route to changing the host from inside the service | `StateDirectory=ghchronicle` makes systemd create `/var/lib/ghchronicle` with the right ownership on start, so the state file has somewhere to live without a manual `mkdir` and a `chown` that someone will forget after a reinstall. Four files live there, not one. Beside `state.json` the sweep keeps its write ledger, `state-written.bin` by default, which is what stops an unchanged point being written again, and its cache, `state-cache.bin`, which is what lets a restart ask GitHub only for what changed; and the service holds `state-lock` for as long as it runs, which is how `-migrate -yes` knows not to change the stores under it. A backfill, or the reading back of a [migration](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations), that stops half way leaves its checkpoint there too. `ReadWritePaths` covers the directory, so all of them are already allowed. Put the state file or the ledger somewhere else and that path needs adding here, and the cache follows the state file wherever it goes. Losing the ledger costs one sweep of rewriting: [only what changed is written](https://docs.jmrp.io/ghchronicle/sinks/#only-what-changed-is-written); losing the cache, one sweep at full price: [the cache beside it](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it). ## Installing it 1. Create the user and the directories. ```sh sudo useradd --system --no-create-home --shell /usr/sbin/nologin ghchronicle sudo mkdir -p /etc/ghchronicle ``` 2. Put the configuration in place. - /etc/ghchronicle/ - config.yaml world readable, no secrets in it - ghchronicle.env mode 600, the tokens - /var/lib/ghchronicle/ - state.json created by the service - state-written.bin the write ledger, beside it - state-cache.bin the cache, beside it too - state-lock held by the service while it runs 3. Write the environment file, and nothing else in it. ```sh title="/etc/ghchronicle/ghchronicle.env" GITHUB_TOKEN=github_pat_... INFLUX_TOKEN=... ``` ```sh sudo chmod 600 /etc/ghchronicle/ghchronicle.env ``` Everything in `config.yaml` reads these through `${VAR}`, which is what lets the config be world readable and version controlled while the secrets are not. 4. Start it. ```sh sudo systemctl daemon-reload sudo systemctl enable --now ghchronicle sudo systemctl status ghchronicle ``` The first sweep runs every family, since none has run yet, except that the service starts at most one family of six hours or more a sweep: `traffic`, `stats`, `forks` and the rest of the eleven reach the store over the first two and a half hours, and only this once. See [the slow families take turns](https://docs.jmrp.io/ghchronicle/configuration/cadences/#the-slow-families-take-turns). ## What to watch The log says what was written and where. ```text level=INFO msg=written sink=influxdb family=repo points=934 unchanged=1955 level=INFO msg="rate budget" bucket=core remaining=4477 limit=5000 ``` [What the log says on a good day](https://docs.jmrp.io/ghchronicle/configuration/logging/#what-the-log-says-on-a-good-day) has every routine line. Three warnings are worth an alert: - **`rate limit reserve reached`** means a family was skipped to protect the budget. Once is fine; every sweep means the cadences are too fast for the number of repositories. - **`family failed everywhere, not marking it as run`** means every repository failed for one family, so it will be retried rather than treated as done. - **`migration pending`**, at every start after an upgrade, means a store still holds rows in a shape this release no longer writes, and the start left them: see [after an upgrade](https://docs.jmrp.io/ghchronicle/install/systemd/#after-an-upgrade). ```sh journalctl -u ghchronicle -f journalctl -u ghchronicle -p warning --since today ``` > **Check the sandbox after editing the unit** > > `systemd-analyze security ghchronicle` scores the unit and names anything the > sandbox is not covering. It is the fastest way to see that an edit quietly > removed a restriction. ## After an upgrade Replace the binary and restart the service. Under the default [`migrate: auto`](https://docs.jmrp.io/ghchronicle/configuration/#migrate), the start checks every store against the changes the new release carries and, before its first sweep, applies on its own each one that loses nothing, saying so at `WARN`. A change it leaves is a `WARN` at every start, naming the store, the reason and the two commands: [what a start does about it](https://docs.jmrp.io/ghchronicle/install/upgrading/#what-a-start-does-about-it). Run those as the service's own user and with its environment file, with the service stopped, since `-migrate -yes` refuses to run beside it: ```sh sudo systemctl stop ghchronicle sudo systemd-run --uid=ghchronicle --gid=ghchronicle --pipe --wait --collect \ --property=EnvironmentFile=/etc/ghchronicle/ghchronicle.env \ /usr/local/bin/ghchronicle -config /etc/ghchronicle/config.yaml -migrate sudo systemd-run --uid=ghchronicle --gid=ghchronicle --pipe --wait --collect \ --property=EnvironmentFile=/etc/ghchronicle/ghchronicle.env \ /usr/local/bin/ghchronicle -config /etc/ghchronicle/config.yaml -migrate -yes sudo systemctl start ghchronicle ``` The first prints the plan and changes nothing; the second applies it and reads back what it cleared. `systemd-run` reads the environment file as root, as the unit does, so it stays mode 600, and runs the binary as `ghchronicle`: measured on systemd 257, a command started that way saw the token of a file only root could read and ran as the user named. Run by root instead, `-migrate -yes` saves the files it rewrites beside the state file as root's, mode 600, `state.json` among them, and the service, which runs as `ghchronicle`, then stops at its start and names the file rather than start from a new one, which would forget a refill still owed: `chown ghchronicle:ghchronicle` it back. With `-once` run from cron rather than a service, comment the line out for the length of the two commands: a `-once` holds no lock while it sweeps, so nothing stops it running beside `-migrate -yes`. ## cron instead of a service `-once` runs a single sweep and exits, which is all a scheduler needs. ```text 0 * * * * /usr/local/bin/ghchronicle -config /etc/ghchronicle/config.yaml -once ``` Keep the state file on a persistent path even in this mode. Without it every run collects every family, whatever its cadence, and walks the stargazer list, the whole star history and the co-authored pull requests again; the cache file beside it is what lets a run ask GitHub only for what changed. Note that an hourly cron gives every family an hourly cadence at best, so the five that run every fifteen minutes, `actions`, `events`, `notifs`, `activity` and `ratelimit`, run four times less often: a workflow run's queue time is read after it is over, and a notification thread moved twice inside the hour shows only its second move. --- # Docker Running ghchronicle in Docker: the distroless image, what has to be mounted writable, and a compose file next to InfluxDB. Source: https://docs.jmrp.io/ghchronicle/install/docker/ Pick a stack, put two lines in a `.env` file beside it, and start it. ## InfluxDB and Grafana ```yaml # ghchronicle, with InfluxDB and Grafana. # # Put two lines in a .env file beside this one: # # GITHUB_TOKEN=github_pat_... # GITHUB_USER=your-login # # then `docker compose up -d`. Grafana is on http://localhost:3000, # admin and the password below, with the dashboard already in it: the # collector publishes it on start and points it at the store beside it. name: ghchronicle services: influxdb: image: influxdb:3-core # Without a credential, because a store that exists for the first time # when the collector first writes to it has nobody to have made one. command: - influxdb3 - serve - --node-id=node0 - --object-store=file - --data-dir=/var/lib/influxdb3 - --without-auth volumes: - influx:/var/lib/influxdb3 healthcheck: test: ["CMD", "curl", "-sf", "http://localhost:8181/health"] interval: 5s timeout: 3s retries: 30 grafana: image: grafana/grafana:12.3.0 environment: GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD:-ghchronicle} ports: - "3000:3000" volumes: - grafana:/var/lib/grafana healthcheck: test: ["CMD-SHELL", "wget -qO- http://localhost:3000/api/health || exit 1"] interval: 5s timeout: 3s retries: 30 ghchronicle: image: ghcr.io/jmrplens/ghchronicle command: ["-config", "/config.yaml"] restart: unless-stopped depends_on: influxdb: condition: service_healthy grafana: condition: service_healthy environment: GITHUB_TOKEN: ${GITHUB_TOKEN:?put your token in a .env file beside this} GRAFANA_PASSWORD: ${GRAFANA_PASSWORD:-ghchronicle} volumes: - state:/var/lib/ghchronicle configs: - source: ghchronicle target: /config.yaml configs: ghchronicle: # The doubled $$ are deliberate. A single $ is read by compose, which # would put the token into the file it hands the container; doubled, # compose writes the reference through and the collector expands it from # its own environment when it starts. content: | github: token: $${GITHUB_TOKEN} targets: user: ${GITHUB_USER:?put the account to collect in a .env file beside this} sinks: influxdb: url: http://influxdb:8181 bucket: github grafana: url: http://grafana:3000 user: admin password: $${GRAFANA_PASSWORD} publish_on_start: true state_file: /var/lib/ghchronicle/state.json volumes: influx: state: grafana: ``` ## InfluxDB ```yaml # ghchronicle, with InfluxDB. # # Put two lines in a .env file beside this one: # # GITHUB_TOKEN=github_pat_... # GITHUB_USER=your-login # # then `docker compose up -d`. Nothing is published to Grafana; # the store is yours to point one at. name: ghchronicle services: influxdb: image: influxdb:3-core # Without a credential, because a store that exists for the first time # when the collector first writes to it has nobody to have made one. command: - influxdb3 - serve - --node-id=node0 - --object-store=file - --data-dir=/var/lib/influxdb3 - --without-auth volumes: - influx:/var/lib/influxdb3 healthcheck: test: ["CMD", "curl", "-sf", "http://localhost:8181/health"] interval: 5s timeout: 3s retries: 30 ghchronicle: image: ghcr.io/jmrplens/ghchronicle command: ["-config", "/config.yaml"] restart: unless-stopped depends_on: influxdb: condition: service_healthy environment: GITHUB_TOKEN: ${GITHUB_TOKEN:?put your token in a .env file beside this} volumes: - state:/var/lib/ghchronicle configs: - source: ghchronicle target: /config.yaml configs: ghchronicle: # The doubled $$ are deliberate. A single $ is read by compose, which # would put the token into the file it hands the container; doubled, # compose writes the reference through and the collector expands it from # its own environment when it starts. content: | github: token: $${GITHUB_TOKEN} targets: user: ${GITHUB_USER:?put the account to collect in a .env file beside this} sinks: influxdb: url: http://influxdb:8181 bucket: github state_file: /var/lib/ghchronicle/state.json volumes: influx: state: ``` ## PostgreSQL and Grafana ```yaml # ghchronicle, with PostgreSQL and Grafana. # # Put two lines in a .env file beside this one: # # GITHUB_TOKEN=github_pat_... # GITHUB_USER=your-login # # then `docker compose up -d`. Grafana is on http://localhost:3000, # admin and the password below, with the dashboard already in it: the # collector publishes it on start and points it at the store beside it. name: ghchronicle services: postgres: image: postgres:18.6-alpine environment: POSTGRES_USER: ghchronicle POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-ghchronicle} POSTGRES_DB: ghchronicle volumes: # /var/lib/postgresql, not /var/lib/postgresql/data: the 18+ images # moved where they keep the cluster, and a volume on the old path is # refused outright with "in 18+, these Docker images are configured to # store database data in a subdirectory". - postgres:/var/lib/postgresql healthcheck: test: ["CMD-SHELL", "pg_isready -U ghchronicle"] interval: 5s timeout: 3s retries: 30 grafana: image: grafana/grafana:12.3.0 environment: GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD:-ghchronicle} ports: - "3000:3000" volumes: - grafana:/var/lib/grafana healthcheck: test: ["CMD-SHELL", "wget -qO- http://localhost:3000/api/health || exit 1"] interval: 5s timeout: 3s retries: 30 ghchronicle: image: ghcr.io/jmrplens/ghchronicle command: ["-config", "/config.yaml"] restart: unless-stopped depends_on: postgres: condition: service_healthy grafana: condition: service_healthy environment: GITHUB_TOKEN: ${GITHUB_TOKEN:?put your token in a .env file beside this} GRAFANA_PASSWORD: ${GRAFANA_PASSWORD:-ghchronicle} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-ghchronicle} volumes: - state:/var/lib/ghchronicle configs: - source: ghchronicle target: /config.yaml configs: ghchronicle: # The doubled $$ are deliberate. A single $ is read by compose, which # would put the token into the file it hands the container; doubled, # compose writes the reference through and the collector expands it from # its own environment when it starts. content: | github: token: $${GITHUB_TOKEN} targets: user: ${GITHUB_USER:?put the account to collect in a .env file beside this} sinks: postgres: dsn: postgres://ghchronicle:$${POSTGRES_PASSWORD}@postgres:5432/ghchronicle?sslmode=disable grafana: url: http://grafana:3000 user: admin password: $${GRAFANA_PASSWORD} publish_on_start: true datasource: sslmode: disable state_file: /var/lib/ghchronicle/state.json volumes: postgres: state: grafana: ``` ## PostgreSQL ```yaml # ghchronicle, with PostgreSQL. # # Put two lines in a .env file beside this one: # # GITHUB_TOKEN=github_pat_... # GITHUB_USER=your-login # # then `docker compose up -d`. Nothing is published to Grafana; # the store is yours to point one at. name: ghchronicle services: postgres: image: postgres:18.6-alpine environment: POSTGRES_USER: ghchronicle POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-ghchronicle} POSTGRES_DB: ghchronicle volumes: # /var/lib/postgresql, not /var/lib/postgresql/data: the 18+ images # moved where they keep the cluster, and a volume on the old path is # refused outright with "in 18+, these Docker images are configured to # store database data in a subdirectory". - postgres:/var/lib/postgresql healthcheck: test: ["CMD-SHELL", "pg_isready -U ghchronicle"] interval: 5s timeout: 3s retries: 30 ghchronicle: image: ghcr.io/jmrplens/ghchronicle command: ["-config", "/config.yaml"] restart: unless-stopped depends_on: postgres: condition: service_healthy environment: GITHUB_TOKEN: ${GITHUB_TOKEN:?put your token in a .env file beside this} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-ghchronicle} volumes: - state:/var/lib/ghchronicle configs: - source: ghchronicle target: /config.yaml configs: ghchronicle: # The doubled $$ are deliberate. A single $ is read by compose, which # would put the token into the file it hands the container; doubled, # compose writes the reference through and the collector expands it from # its own environment when it starts. content: | github: token: $${GITHUB_TOKEN} targets: user: ${GITHUB_USER:?put the account to collect in a .env file beside this} sinks: postgres: dsn: postgres://ghchronicle:$${POSTGRES_PASSWORD}@postgres:5432/ghchronicle?sslmode=disable state_file: /var/lib/ghchronicle/state.json volumes: postgres: state: ``` ## The collector on its own ```yaml # ghchronicle, on its own. # # Put two lines in a .env file beside this one: # # GITHUB_TOKEN=github_pat_... # GITHUB_USER=your-login # # then `docker compose up -d`. It writes what it collects to its own # log, which is enough to watch it work. Point sinks at a store of your # own when you have one. name: ghchronicle services: ghchronicle: image: ghcr.io/jmrplens/ghchronicle command: ["-config", "/config.yaml"] restart: unless-stopped environment: GITHUB_TOKEN: ${GITHUB_TOKEN:?put your token in a .env file beside this} volumes: - state:/var/lib/ghchronicle configs: - source: ghchronicle target: /config.yaml configs: ghchronicle: # The doubled $$ are deliberate. A single $ is read by compose, which # would put the token into the file it hands the container; doubled, # compose writes the reference through and the collector expands it from # its own environment when it starts. content: | github: token: $${GITHUB_TOKEN} targets: user: ${GITHUB_USER:?put the account to collect in a .env file beside this} sinks: stdout: true state_file: /var/lib/ghchronicle/state.json volumes: state: ``` ```sh docker compose up -d ``` The stack's `state` volume keeps the state and the cache from one start to the next, and it is the collector's to write from the first: the image carries the directory it is mounted on, owned by the uid the collector runs as, and Docker gives a new volume that owner. See [what has to be writable](https://docs.jmrp.io/ghchronicle/install/docker/#what-has-to-be-writable). The first sweep runs every family, since nothing has run yet, except that the collector starts one family of six hours or more a sweep: `traffic`, `stats`, `forks` and the rest of the eleven reach the store over the first two and a half hours, and only the first time. See [the slow families take turns](https://docs.jmrp.io/ghchronicle/configuration/cadences/#the-slow-families-take-turns). With Grafana it is on `http://localhost:3000`, user `admin`, password `ghchronicle` unless you set `GRAFANA_PASSWORD`. The dashboard is already there: the collector publishes it when it starts and points it at the store beside it, so there is nothing to import and no datasource to fill in. The rest of this page is the image itself, for a reader running it another way. ## One container, by hand ```sh docker run -d --name ghchronicle \ -v /etc/ghchronicle/config.yaml:/config.yaml:ro \ -v ghchronicle-state:/var/lib/ghchronicle \ -e GITHUB_TOKEN -e INFLUX_TOKEN \ -p 9605:9605 \ ghcr.io/jmrplens/ghchronicle -config /config.yaml ``` With `state_file: /var/lib/ghchronicle/state.json` in the configuration, which is where the example configuration puts it. The volume is what keeps the state and the cache across a new container, and Docker creates it on first use, owned by uid 65532 like the directory it is mounted on; [what has to be writable](https://docs.jmrp.io/ghchronicle/install/docker/#what-has-to-be-writable) says why each is needed. ## The image Built `FROM gcr.io/distroless/static-debian13:nonroot` over a static, `CGO_ENABLED=0` binary. There is no shell and no package manager in it, so code execution inside the container has nothing to pivot with, and the `nonroot` tag bakes in **uid 65532**, which keeps it off root even when the orchestrator sets no `securityContext` of its own. Two consequences worth knowing before you debug it: - `docker exec ... sh` does not work. There is no `sh`. Read the logs instead. - Anything the container writes must be owned by, or writable by, uid 65532. ## Verify the image Every released image from 2.5.1 on, which is every image a release page names, is signed with cosign, keylessly and by the same identity as the checksum file the other installs check: this repository's release workflow, run for a release tag, recorded in a public transparency log. With [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) 3: ```sh cosign verify \ --certificate-identity-regexp 'https://github.com/jmrplens/ghchronicle/.github/workflows/release.yml@refs/tags/.*' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com \ ghcr.io/jmrplens/ghchronicle:v2.6.5 > /dev/null ``` ```text Verification for ghcr.io/jmrplens/ghchronicle:v2.6.5 -- The following checks were performed on each of these signatures: - The cosign claims were validated - Existence of the claims in the transparency log was verified offline - The code-signing certificate was verified using trusted certificate authority certificates ``` The Docker Hub copy, `docker.io/jmrplens/ghchronicle`, verifies with the same command. What cosign checks is a digest, not the tag: without `> /dev/null` it also prints, as JSON, the digest it verified (`docker-manifest-digest`), and an image pulled as `ghcr.io/jmrplens/ghchronicle@sha256:...` is the one verified however a tag moves afterwards. `latest` is moved onto a release's digest last, once the release workflow has verified that digest's signature and run the image on both architectures, so it never names an image that was not checked; a release that fails before then leaves it on the previous one. Use cosign 3. The signatures are in the bundle format it writes, and an older cosign answers `no signatures found` for these images: measured, cosign 2.6.1 finds the signature only when given `--new-bundle-format`, 2.5.0 does not find it even then, and 2.4.2 does not know the flag. Images up to 2.5.0 carry no signature at all, so for them that answer is the true one. ## What has to be writable The config file is mounted read-only. Six things are not: | Path | Needed for | | ------------------- | --------------------------------------------------------------------------------- | | `state_file` | Always. Without a persistent path the stargazer walk repeats on every restart | | `-cache.bin` | Always. The cache, which sits beside the state file and has no setting of its own | | `sinks.dedupe_file` | The long-running service, with a sink that keeps it. A `-once` run opens none | | `-lock` | The long-running service, which holds it while it runs, and `-migrate -yes` | | `sinks.file.path` | Only with the file sink | | `log.file` | Only with a log file configured | The first three live in the same directory by default, and so does the lock, so one mounted directory covers them, and it has to be a directory. Each of the three is written beside itself and renamed into place, so a reader never sees half a file, and a file mounted on its own cannot be renamed over: measured with the 2.6.1 image built from its Dockerfile, a state file bind-mounted alone was never saved, and every sweep said so. ```text level=WARN msg="state not saved" err="rename /var/lib/ghchronicle/state.json.tmp /var/lib/ghchronicle/state.json: device or resource busy" ``` A directory the collector cannot write gives the same warning, with the error that fits, and one for the cache beside it: ```text level=WARN msg="state not saved" err="open /var/lib/ghchronicle/state.json.tmp: permission denied" level=WARN msg="cache file not saved" file=/var/lib/ghchronicle/state-cache.bin err="open /var/lib/ghchronicle/state-cache.bin.tmp: permission denied" ``` From 2.6.1 the image carries `/var/lib/ghchronicle` owned by uid 65532, and Docker fills a new named volume from the directory it is mounted over, owner included, so the directory the example configuration names needs nothing done to it: measured with a fresh volume, and with an empty one the 2.6.0 image had left to root, the collector wrote its state and its cache into both with no warning. With nothing mounted there it writes too, and the state goes with the container. Up to 2.6.0 the image had no such directory: nothing mounted there left one uid 65532 could not create, and a new named volume there was root's, which is where the two lines above came from. Two ways are left to end up with a directory the collector cannot write. A named volume mounted where the image has no directory of its own is created owned by root: hand it to uid 65532 once, with any image that has a `chown`, as in `docker run --rm -v :/v alpine chown 65532:65532 /v`. A host directory is mounted as it is, whoever owns it: `sudo chown 65532:65532` it on the host. A configuration that names no `state_file` at all writes to the container's working directory, which is writable and goes with the container. Without the state, every new container starts from nothing: the stargazer walk, the whole star history and the co-authored walk again, and every family at once. Without the cache, a whole sweep of asking GitHub again what it already knew: [the cache beside it](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it). Without the ledger, a whole sweep of rewriting, which is the one thing it exists to prevent: [only what changed is written](https://docs.jmrp.io/ghchronicle/sinks/#only-what-changed-is-written). ## The port `EXPOSE 9605` is the Prometheus exporter, and it is the only listener the process ever opens. Publish it only if you enabled the `prometheus` sink; every other sink is outbound. > **Bind the exporter to 0.0.0.0 inside a container** > > The example configuration listens on `127.0.0.1:9605`, which inside a > container means the container's own loopback and is unreachable from the host. > Set `sinks.prometheus.listen: 0.0.0.0:9605` and let `-p` decide who can reach > it. ## With compose - **Collector only** ```yaml title="compose.yaml" services: ghchronicle: image: ghcr.io/jmrplens/ghchronicle command: ["-config", "/config.yaml"] restart: unless-stopped environment: GITHUB_TOKEN: ${GITHUB_TOKEN} INFLUX_TOKEN: ${INFLUX_TOKEN} volumes: - ./config.yaml:/config.yaml:ro - state:/var/lib/ghchronicle volumes: state: ``` - **With InfluxDB** ```yaml title="compose.yaml" services: influxdb: image: influxdb:3-core volumes: - influx:/var/lib/influxdb3 ports: - "8181:8181" ghchronicle: image: ghcr.io/jmrplens/ghchronicle command: ["-config", "/config.yaml"] restart: unless-stopped depends_on: - influxdb environment: GITHUB_TOKEN: ${GITHUB_TOKEN} INFLUX_TOKEN: ${INFLUX_TOKEN} volumes: - ./config.yaml:/config.yaml:ro - state:/var/lib/ghchronicle volumes: influx: state: ``` The collector reaches the database by service name, so `sinks.influxdb.url` is `http://influxdb:8181`. ## After an upgrade Pull the new image and recreate the container. Under the default [`migrate: auto`](https://docs.jmrp.io/ghchronicle/configuration/#migrate), its start applies on its own, before its first sweep, every change the new release carries that loses nothing, and warns about the rest at every start with the two commands that apply them: see [Migrations](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations). The container takes the same flags as the binary, so those run in a one-off container of the same service, with its volume, its environment and its user, while the service is stopped: ```sh docker compose pull ghchronicle docker compose up -d ghchronicle docker compose logs ghchronicle | grep 'migration pending' docker compose stop ghchronicle docker compose run --rm ghchronicle -config /config.yaml -migrate docker compose run --rm ghchronicle -config /config.yaml -migrate -yes docker compose start ghchronicle ``` The first `run` prints the plan and changes nothing; the second applies it and reads back what it cleared. Both mount the volume the service mounts, so they find the lock it holds beside the state file: with the service still running, `-migrate -yes` refuses, naming the process by the number the service's own container gives it, and changes nothing. Measured with two containers on one named volume: a `flock` one of them held was refused to the other, and granted once the first had gone. A one-shot container a scheduler runs, below, holds no lock while it sweeps, so pause the scheduler for the length of the two runs. ## One sweep, then exit The container takes the same flags as the binary, so a scheduler can run it without a long-lived service. ```sh docker run --rm \ -v /etc/ghchronicle/config.yaml:/config.yaml:ro \ -v ghchronicle-state:/var/lib/ghchronicle \ -e GITHUB_TOKEN \ ghcr.io/jmrplens/ghchronicle -config /config.yaml -once ``` Mount the state volume in this mode too. It is what makes the second run cheap, and what keeps each family to its own cadence: without it every run collects every family. --- # GitHub Actions Running ghchronicle as a GitHub Action: the composite Action, its four modes, and the two things a hosted runner does not keep. Source: https://docs.jmrp.io/ghchronicle/install/actions/ The repository ships a composite Action, so a workflow needs no Go toolchain: it downloads a release binary and calls it. ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: once config: .github/ghchronicle.yaml ``` ## Inputs | Input | Default | What it does | | ----------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------- | | `token` | required | A personal access token. The automatic `GITHUB_TOKEN` is not enough | | `config` | `""` | Path to a configuration file. Omit to run with defaults built from `user`, whose state lasts one run and cannot be cached | | `user` | the repository owner | The account to collect when no config file is given | | `mode` | `once` | `once`, `backfill`, `card` or `migrate` | | `backfill-since` | `""` | Bound for `backfill`: a date, `90d`, `2y` or a Go duration. Empty means no bound | | `card` | `""` | Path of the SVG to write. Empty means no card | | `card-layout` | `summary` | One of the thirteen registered layouts | | `card-theme` | `auto` | `dark`, `light`, `auto`, or `both` for a light card and its `_dark` twin | | `card-fields` | `""` | Comma-separated fields. Empty means the layout's default | | `card-motion` | `once` | `once`, `loop` or `off`; `loop` changes only `terminal` and `ticker` | | `card-width` | `""` | Card width in pixels. Empty draws the layout at its own width; each one draws between two ends of its own, stated in its [section](https://docs.jmrp.io/ghchronicle/card/layouts/). Only `activity-heatmap` spends the room on data, a whole year of the calendar at its far end | | `card-speed` | `""` | How fast an animated layout plays, as a decimal from 0 to 1. Empty means 0.5, the pace every card has always been drawn at; below it the card is slower, above it faster, and every animated layout scales together. 0 is the slowest animation and not a still card, `card-motion: off` is | | `include-private` | `false` | `true` counts private repositories when no config file is given. See the warning below | | `version` | `latest` | The release to install: `latest` for the newest, or a release with or without its `v`, so `2.6.5` and `v2.6.5` are the same one. The major tag `v2` is what `uses:` takes, not a release, and is refused | ## The four modes - **once** One sweep against a configuration file you supply, which is how a workflow feeds a database. ```yaml name: Collect on: schedule: - cron: "*/30 * * * *" workflow_dispatch: jobs: collect: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: once config: .github/ghchronicle.yaml env: INFLUX_TOKEN: ${{ secrets.INFLUX_TOKEN }} ``` The config file refers to the database credentials as `${VAR}`, so they go in as secrets and never into the repository. - **card** One sweep and one SVG, and nothing written to any database. This is the only mode that needs no store configured at all. ```yaml name: Profile card on: schedule: - cron: "17 6 * * *" workflow_dispatch: permissions: contents: write jobs: card: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/github-stats.svg card-layout: github-stats card-theme: both - name: Commit if it changed run: | git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" git add generated/ git diff --staged --quiet || git commit -m "Update the profile card" git push ``` The SVG is byte-identical for the same input, so a day with no change produces no commit. - **backfill** Reaches as far back as GitHub allows and waits for the rate limit rather than stopping. Run it once, by hand. ```yaml name: Backfill on: workflow_dispatch: inputs: since: description: "A date, 90d, 2y, or empty for no bound" default: "2y" jobs: backfill: runs-on: ubuntu-latest timeout-minutes: 360 steps: - uses: actions/checkout@v7 - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: backfill backfill-since: ${{ inputs.since }} config: .github/ghchronicle.yaml ``` Give it a generous `timeout-minutes`: an unbounded backfill parks at every rate limit reset, and a job that is killed half way has spent the quota and kept part of the benefit. - **migrate** Brings the stores the configuration writes along after an upgrade: it runs `-migrate -yes`, which applies every change [the plan](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations) finds pending, the ones that need somebody's word too, and reads what it cleared again. Run it by hand, once, after reading what `-migrate` says about the same configuration. ```yaml name: Migrate on: workflow_dispatch: jobs: migrate: runs-on: ubuntu-latest timeout-minutes: 60 steps: - uses: actions/checkout@v7 - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: migrate config: .github/ghchronicle.yaml ``` It takes no card. A store holding rows of accounts this configuration does not collect, or whose rows could not be compared with it, is held back and the step fails, since only `-migrate-others` clears it, and that is a decision to take at a terminal, not in a workflow. Without a state file [restored between runs](https://docs.jmrp.io/ghchronicle/install/actions/#two-things-a-hosted-runner-does-not-keep), every run of the Action starts from a new one, so a store that cannot be asked, a SQL file, a Graphite or whatever is behind a Telegraf, has no history here and is never taken to need a change; with one restored, those follow its record as they would on a host. A job that timed out while reading the history again leaves nothing the next job can resume from, since `actions/cache` saves nothing for a job that did not succeed: run `mode: backfill` for the same configuration after it. In `once` and `backfill`, a start checks the stores the same way and warns about what is pending. On a new state file, which is every run without one restored, it applies nothing on its own even under `migrate: auto`: the state file goes with the runner, and with it the record that a refill is still owed, so a refill that failed or a job cancelled half way would leave a store cleared and nothing saying so. With a state file restored it applies what is safe to apply unattended, as a host does. The Action turns every line that says a change is pending, one was applied at the start, one failed or a refill is owed into an annotation on the run as the line is written, so it shows on the workflow's summary page rather than only in its log, a job cancelled half way included. ## A card in your profile README GitHub shows the README of the repository named after your account, `/`, at the top of your profile. The card lives in that repository as a file, the README points at it once, and a scheduled workflow replaces the file. The README itself is never rewritten. 1. Create a personal access token (see [the token](https://docs.jmrp.io/ghchronicle/start/token/)) and store it in `/` as the secret `GHCHRONICLE_TOKEN`. The automatic `GITHUB_TOKEN` will not do, even for public numbers: it is not a user, so the Action cannot list your repositories with it and the card comes out empty. 2. Add `.github/workflows/card.yml`: ```yaml name: Profile card on: schedule: - cron: "17 6 * * *" workflow_dispatch: permissions: contents: write # One run at a time: two that overlap would race each other to push. concurrency: group: profile-card cancel-in-progress: false jobs: card: runs-on: ubuntu-latest steps: - uses: actions/checkout@v7 - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: animated-counters card-theme: both card-motion: once - name: Commit if it changed run: | git config user.name "github-actions[bot]" git config user.email "41898282+github-actions[bot]@users.noreply.github.com" git add generated/ git diff --staged --quiet || git commit -m "Update the profile card" git push ``` 3. Paste this line into `README.md` wherever the card should appear, once: ```html My GitHub statistics ``` 4. Run the workflow by hand from the Actions tab the first time, so the files exist before the first scheduled run. **More than one card.** Repeat the `uses: jmrplens/ghchronicle@v2` step with another `card` path and layout, and paste one `` per card. Each step is a sweep of its own, and each draws every number again: a run that writes a card collects every family whatever its cadence says, and one in `card` mode writes nothing back to the state file. Several card steps can therefore share one `config`, and so one `state_file`, and each card is the whole account. That is also what each one costs: a card is a cold sweep of every family, so N cards are N sweeps of [the per-family price](https://docs.jmrp.io/ghchronicle/api/cost/). **A README that is not at the root.** This is for other repositories, since GitHub shows a profile's README only from the root of `/`. The paths in the `` are relative to the README, so a `docs/README.md` points at `../generated/card.svg`. > **Private repositories** > > Without a config file, stars, forks, languages, traffic and repository names > come from public repositories only, while contribution, commit and pull > request counts are account totals, as GitHub reports them on your profile. > With `include-private: true` the card adds the stars, forks, > languages and traffic of your private repositories, and the layouts that list repositories > **publish their names**: `repo-list` and `summary` (the Action's default > layout) list them by default, and so does any layout given `top_repos` in > `card-fields`. To count them without naming them, pick the fields yourself and leave > `top_repos` out, for example `card-fields: stars,forks,followers,contributions`. ## Two things a hosted runner does not keep > **The token must be a personal access token** > > Traffic needs push access to *every* repository being collected and, on a > fine-grained token, the repository permission Administration (read); the > automatic `GITHUB_TOKEN` cannot be granted Administration, so it gets a 403 > even in the repository the workflow runs in. It also has no > `security_events`, no `read:packages`, and is not a user, so nothing > account-wide works with it. See [the token](https://docs.jmrp.io/ghchronicle/start/token/). **The state file does not survive between runs.** Without it every run is a first run. Every family runs, whatever its cadence says, because nothing remembers when it last did. The full stargazer walk is done again, and every repository's daily star history is read whole, a page per thirty weeks of its life. `achievements` walks the account's merged pull requests whole for its co-authored count, which was 35 queries and 23.7 MB on an account with 2,315 of them. And with no [cache file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it) beside the state, nothing is asked with an ETag, so no answer is a free 304. On a small account that is a handful of calls; on an account with many stars, old repositories or a long history, cache it. Caching needs a `config:` file. Without one the Action writes a configuration of its own and puts the state in a new temporary directory under `RUNNER_TEMP` on every run, which no cache step can name. With one, point `state_file` at the home directory: ```yaml state_file: ~/.ghchronicle/state.json ``` and restore and save that directory with a step before the ghchronicle one: ```yaml - uses: actions/cache@v6 with: path: ~/.ghchronicle key: ghchronicle-state-${{ github.run_id }} restore-keys: ghchronicle-state- ``` A `~` in `state_file` is the home directory from 2.6.1 on. An older release reads it as written, a directory called `~` in the checkout that the cache step never sees, so with a `version` older than 2.6.1 write the path out: `/home/runner/.ghchronicle/state.json` on a GitHub-hosted Ubuntu runner. The directory then holds the cache file as well, which a `once` step writes and a `backfill` step only reads, so a run restored from it also asks GitHub with the validators the last one stored, and pays only for what changed. A `card` mode run reads a restored state file, which is what lets it skip both walks, and never writes one back, nor the cache file beside it: it delivers its points to the card and to no store, so nothing it learned may tell the next collection that a family is already done. What fills the `actions/cache` entry is a `once` step, with both files, or a `backfill` step, with the state file alone. No step keeps a write ledger there: a run that ends with its sweep opens none. ## Without the Action The same thing by hand, if you would rather not depend on it: ```yaml - uses: actions/setup-go@v7 with: go-version: stable - run: go install github.com/jmrplens/ghchronicle/v2/cmd/ghchronicle@latest - run: ghchronicle -config .github/ghchronicle.yaml -card profile.svg -card-only env: GITHUB_TOKEN: ${{ secrets.GHCHRONICLE_TOKEN }} ``` --- # Upgrading What the first hours after an upgrade from 2.5.x to 2.6.x look like, each of them once and on purpose, and what 2.6.1 changes in a store written before it. Source: https://docs.jmrp.io/ghchronicle/install/upgrading/ Install the new release the way the old one was installed, over it, and start the collector again. The configuration, the state file and the write ledger carry over as they are, and nothing has to be migrated by hand. What follows is what the first hours after the upgrade look like, so that none of it reads as a fault, and the few things worth doing about them. The dashboards are generated from the binary, so publish them again after an upgrade, with `-publish-dashboard` or `grafana.publish_on_start`: see [letting the binary do it](https://docs.jmrp.io/ghchronicle/dashboards/#letting-the-binary-do-it). ## Migrations A release sometimes changes what a stored row is keyed by. A tag that becomes a field is the usual case: every row written before the change is a series the rows written after it are not, so the store keeps both for ever, and a count over them grows by one for every item read on both sides of the change. The binary carries the list of every such change a store written by a 2.x release can still hold, each under an ID that names the release and the measurement, such as `2.6.1/gh_discussion_comment/is_answer`, and `-migrate` checks every configured store against it: ```sh ghchronicle -config /etc/ghchronicle/config.yaml -migrate ``` It prints one block per store and one line per change, and changes nothing: the stores are asked questions, GitHub is asked for the repository list alone, and the state file is read and never written. A store that can be asked decides for itself; one that cannot is decided by what the state file remembers of it. | Store | What decides | | ------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | InfluxDB 3 | Whether the old tag is a tag column of the live table, asked of the catalog. A table InfluxDB has set aside is listed under another name and is not asked | | InfluxDB 2 | Whether any row carries the old tag. The tag keys stay listed after a delete, so they are not asked | | PostgreSQL | Whether any row holds a value in the old tag's column, in the schema the sink writes to | | Elasticsearch | A count of the documents that carry the old tag. The mapping keeps a field after its documents are gone | | SQL file, Graphite, Telegraf | The release the state file records as the first to write the store | | Loki, Prometheus, OTLP, file, stdout | Nothing: none of them keeps a row whose identity a release could change | Each line starts with what the check found: - `not needed`: the store holds nothing of the old shape. - `pending`: it holds the old shape, or, for a store that cannot be asked, may hold it. The lines under it say why, what bringing it along would do there, which families would read the measurement again and from when, and what would not come back: the rows of repositories the configuration no longer covers, and of families it has switched off. For the comments, those are other people's comments on such a repository: the account's own come back through `outbound` wherever they are. The last line says whether applying it needs nobody's word: GitHub still serves the whole history, so every row the store holds would come back, the old rows would be set aside for at least 24 hours rather than deleted, and every row in the store is this configuration's. A store shared with another collector, one holding rows the refill would not bring back, one whose rows could not be compared with the configuration, an InfluxDB 2, whose only way is a delete, and every store that cannot be asked need somebody's word, and the line says which reason applies. A row that names no account belongs to a configuration of organisations alone, which writes no user: it is this configuration's only when this one has no `targets.user` either. - `note`: the rows are there and nothing can put them right, so the plan says what they mean and changes nothing. - `frozen`: nothing this configuration runs writes the measurement any more, so its rows are history and are left as they are. - `applied`: the state file records the change as applied to this store, and the store, where it can be asked, agrees. - `unreachable`: the store did not answer, so nothing about it is known. ### What a start does about it Every run that writes to the stores, the service, `-once`, `-backfill` and a card drawn beside the stores, checks them the same way before its first sweep, and what it does next is the [`migrate`](https://docs.jmrp.io/ghchronicle/configuration/#migrate) setting's: - `auto`, the default, applies on its own every pending change marked safe to apply unattended, and nothing else. It says so first, at `WARN`, with what it found, why the change exists and what it does in that store, including where the old rows are set aside; then it reads the measurement again from GitHub and only then sweeps. - `warn` applies nothing. A one-shot run, `-once` or `-backfill`, on a new state file applies nothing on its own under `auto` either, and warns instead. That is every run of the Action that does not restore its state file with `actions/cache`, whose state file goes with its runner, and with it the record that a refill is still owed: a refill that failed, or a job cancelled half way, would leave the store cleared and nothing anywhere saying so. The next run on a host, which has a state file by then, or `-migrate -yes`, applies it. A pending change a start does not apply is a `WARN` at every start, named by store, with the reason it was left and the two commands, copied from the configuration the run was given. One line in the log, wrapped here: ```text level=WARN msg="migration pending" sink=graphite measurement=gh_discussion_comment migration=2.6.1/gh_discussion_comment/is_answer why="is_answer was a tag, ..." not_applied="only whoever runs the Graphite host can remove its files" plan="ghchronicle -config /etc/ghchronicle/config.yaml -migrate" apply="ghchronicle -config /etc/ghchronicle/config.yaml -migrate -yes" first="stop this service: -migrate -yes refuses to run beside it" ``` There is no setting that applies the rest on its own: a change that could lose rows waits for somebody to read the plan and give the word. A note is said once at `INFO` and then at `DEBUG`, since nothing will ever change it, and so is a change whose measurement nothing the configuration runs writes any more. A store that does not answer within 30 seconds is a `WARN` too, and the sweep goes on without waiting longer. What a start finds not needed, and what it has applied, it records, and the next start takes the record's word for it: once every change is settled for a store, a start does not ask that store anything. `-migrate` does not take that word and asks again every time. ### Applying it by hand ```sh systemctl stop ghchronicle ghchronicle -config /etc/ghchronicle/config.yaml -migrate -yes systemctl start ghchronicle ``` Run it as the user the service runs as, with the environment the service has: it saves the state file and the files beside it, and a file root saves is one the service can no longer read. A run that finds the state file there and cannot read it, or cannot parse it, stops and names it rather than starting from a new one, which would forget a refill still owed: give the file back to the service's user. [systemd](https://docs.jmrp.io/ghchronicle/install/systemd/#after-an-upgrade) and [Docker](https://docs.jmrp.io/ghchronicle/install/docker/#after-an-upgrade) show how. Pause a cron job or a timer that runs `-once` as well, and let a `-backfill` that is running finish first. Neither holds the lock while it sweeps, and a SQL file two processes write at once is not one psql can replay. Their state file is safe either way: a run that saves it keeps what another process recorded of the stores since it read it, so a `-once` that ran across `-migrate -yes` does not put back the record it read before. `-migrate -yes` prints the same plan and then applies every pending change, the unsafe ones too, since `-yes` is the word the plan asked for. Each one is recorded in the state file as it is applied, so a stop half way costs nothing the next run cannot pick up: what was applied is not done twice. A line under the plan says what happened to each: - `applied`: done. Where the old rows are kept, it names them; where somebody else has to act, on a Graphite host or behind a Telegraf, it prints the commands to run there. - `failed`: the store refused, with its reason. The others still go ahead. - `held back`: the store holds rows of accounts this configuration does not collect, or whose rows it holds could not be compared with this configuration, and the line says which and why. Set aside, another's rows come back only when whoever collects them reads them again, so this takes `-migrate-others` as well as `-yes`. - `unreachable`: the store did not answer, so nothing was done there. - `refill`: what was read back from GitHub, from when and into which stores, or why reading it back did not finish: see [reading the history back](https://docs.jmrp.io/ghchronicle/install/upgrading/#reading-the-history-back). - `reconciled`: each copy of the old rows compared with what came back, and the items GitHub no longer serves. It exits 0 when everything pending was applied and read again, and 1 when anything was left, with the sentence that says how to go on. It refuses before it changes anything, and exits 1, with no GitHub token or no repository list, since nothing it clears could be read again, and while another process holds the state file: the service holds it for as long as it runs, beside the state file as `-lock`, and `-migrate -yes` names that process rather than changing the stores it writes. A service started while `-migrate -yes` runs waits for it to finish, and a second service on the same state file is refused. In the GitHub Action, `mode: migrate` runs `-migrate -yes`: see [the inputs](https://docs.jmrp.io/ghchronicle/install/actions/#inputs). ### What applying does in each store | Store | What applying does | | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | [InfluxDB 3](https://docs.jmrp.io/ghchronicle/sinks/influxdb/#what-a-migration-does-here) | Deletes the one table, which InfluxDB keeps as `-`, queryable, and purges itself 72 hours later, or never before 3.2 | | InfluxDB 2 | Deletes every row of the measurement in the bucket. Nothing is kept, so a start never does it on its own | | [PostgreSQL](https://docs.jmrp.io/ghchronicle/sinks/postgres/#what-a-migration-does-here) | Renames the table `-` in the sink's schema; ghchronicle drops it 24 hours later | | [Elasticsearch](https://docs.jmrp.io/ghchronicle/sinks/elasticsearch/#what-a-migration-does-here) | Blocks writes to the index, clones it to `-` and deletes it; ghchronicle deletes the clone 24 hours later | | SQL file | Writes `DROP TABLE IF EXISTS` for the measurement into the file, ahead of the rows written after it | | Graphite | Prints the commands that remove the old paths on the Graphite host | | Telegraf | Says what to do in the store behind it | Each store touches that one measurement and nothing else: the table, index or paths named exactly, in the database, bucket, schema or prefix the sink writes to. A measurement that looks like it, another collector's tables and another prefix's indices are never reached. A store that can be asked is only changed once it has been asked and found holding the old shape, and a dry run sends it nothing but questions. Where a store keeps the old rows aside, the plan names each copy under the store, with a `kept aside` line saying when it was set aside and when it goes, and the state file keeps it until then. ghchronicle purges its own copies once they have been kept 24 hours: the service after a sweep, any run at its next start, and `-migrate -yes` whenever it runs. A run on a new state file, every run of the Action that does not restore one among them, asks PostgreSQL and Elasticsearch for copies named the way a migration names them, since its state file cannot name them. InfluxDB 3 purges its own on a schedule of its own, which is read back from the system table of its `_internal` database: 72 hours after the delete by default (measured on 3.2.1 to 3.11.5), and it keeps the name in its catalog for its delete grace period after that, 24 hours by default. A server before 3.2 never purges one: the plan says so before applying, and `-migrate` asks the server for every copy it keeps that way, with the request that removes it once the server runs a release from 3.2 to 3.9 (see [before 3.2 the copy stays](https://docs.jmrp.io/ghchronicle/sinks/influxdb/#before-32-the-copy-stays)). Until a copy goes, undoing the change is on each store's page. A store that was cleared is written again whole. The [write ledger](https://docs.jmrp.io/ghchronicle/sinks/#only-what-changed-is-written) forgets the measurement in that store alone, every other measurement and every other store keeping what it remembers, and the [cache file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it) forgets what it claims about the families that write the measurement, their refusals and, when `actions` is among them, the workflow runs whose jobs it had written, so the sweeps after it ask everything a first sweep asks. ### Reading the history back A store that was cleared holds none of the measurement's history until it is read again from GitHub. Right after applying, the same run reads it back with one backfill of its own, the refill: the families that write the measurement, `discussions` and `outbound` for the comments, `security` for the alert items, and no other; writing that measurement alone; into the stores that were cleared and no other. ```text refill read discussions and outbound again, since 2023-11-14, writing gh_discussion_comment to elasticsearch, influxdb and postgres ``` - **Only what was cleared.** A family asks GitHub everything it always asks, and only the measurement cleared is written; the rest of what the family collects is in the stores already. Each store gets what was cleared in that store, so a store still holding a measurement in its old shape is never handed that measurement's history beside it. - **Only where it was cleared.** InfluxDB, PostgreSQL, Elasticsearch, the SQL file, after its `DROP`, and Graphite, at the new depth. Never Loki, the Prometheus exporter, OTLP, the file sink or stdout, which were not cleared and would hold every row of the refill a second time, nor a Telegraf, whose store is somebody else's: the plan names the `-backfill -families` that sends the history through it once that store has been cleared. - **As far back as the store held.** The day of the oldest row the store held, read before it was cleared: a bound later than that would lose the difference for good once the copy is purged. A store that cannot say how far back its rows go, the SQL file, Graphite, or an InfluxDB 3 Core that would not count them (below), is read back with no bound: what applying takes there is every row whatever its date, and `backfill.since` bounds what a backfill reaches, not what a store holds. One refill for several stores reads back as far as the furthest. - **Waiting, as a backfill waits.** It has a GitHub client of its own that waits for a spent rate limit to turn over, where a sweep skips, and `-backfill-retry` goes back for what it leaves as it does for a backfill. The containerised suite's whole `-migrate -yes`, clearing InfluxDB 3.11.2, PostgreSQL 18.6 and Elasticsearch 9.5.3 and reading the comments back from the fake GitHub, takes about 2 seconds; on the author's account, the `outbound` walk it needs took 33 seconds when 2.6.1's was done by hand. The refill is owed before a store is touched, and the state file records it then, under the store's `refill` key: a store cleared and not read back looks, to anyone who asks it, exactly like one that never held the old shape, and only that record says the history is still to come. A clear that failed and left the store as it was, which the store says, takes the debt back; one whose answer did not arrive, a proxy's 502 or a timeout, is looked at again, and when the store cannot say whether it was carried out the refill stays owed, which at worst reads the history once for nothing. It goes when the refill reaches the end of every family. A refill cut short, by a stop, a store that refused a write or GitHub not answering, keeps a checkpoint of its own beside the state file, named with `-refill.json`, apart from a backfill's so neither refuses or overwrites the other, and is resumed from there: by `-migrate -yes` run again, even with nothing left to apply, and by any start under `migrate: auto`. Under `migrate: warn` a start says it is owed, at every start, with the command. `-backfill-status` prints how far it has got, and `-migrate` lists it under its store as `refill owed`. #### What the service does meanwhile Under `migrate: auto` the service applies what is safe and reads it back after its sinks are built and before its first sweep, so its sweeps start when the refill ends. The Prometheus exporter is up in that time and holds nothing until the first sweep, as after any restart. A service stopped during the refill exits at once, keeping what it wrote, and its next start carries on. The write ledger remembers what the refill wrote, so the first sweep after it does not send those rows again. Under `migrate: warn` nothing is read back and the service sweeps as it always did, writing this release's shape beside the old one, and `-migrate -yes` refuses to run beside it. #### What GitHub no longer serves A refill brings back what GitHub still serves for the targets configured now. When the refill ends, each store that kept a copy of the old rows and can be asked, InfluxDB 3, PostgreSQL and Elasticsearch, compares the copy with the table, item by item: the `comment` of each comment, the `full_name` and `number` of each alert. What the copy holds and the table does not is what GitHub no longer served, a repository deleted or no longer covered, a comment deleted, an alert whose feature was switched off; the report and the log name a few, and those rows stay only in the copy until it is purged. From the containerised suite, whose seeded comment the fake GitHub does not serve: ```text reconciled gh_discussion_comment in postgres: 1 item in gh_discussion_comment-20260928T234418, 6 now; 1 GitHub no longer serves, whose rows are only in the copy until it is purged: 1 ``` Carrying them over is left to the reader, from the copy, while it is there: for an item whose old shape was two rows at one instant, which of the two was right is not something a program can tell offline. Measured read-only against production's InfluxDB 3.11.5, the copy InfluxDB kept of the table dropped by hand for 2.6.1 held 114 comments, and the table read back 121, none of the 114 missing. ### InfluxDB 3 Core's query file limit InfluxDB 3 Core refuses a query that would open more Parquet files than its `--query-file-limit`, 432 by default, and a table a sweep writes every ten minutes is past that in days. The check reads the catalog, which opens no file, so it still finds the old shape in such a table, and a table with no old tag is not counted at all. What it cannot do there is count the rows, which leaves the refill with no bound, nor read whose rows the table holds, which leaves the change needing your word and `-migrate -yes` holding it back for `-migrate-others`; the plan quotes the server's refusal. Measured on 3.11.2 with the limit lowered to 3: before this, the same table was `unreachable` at every start and `-migrate -yes` could not apply it at all. Raising the limit on the server, for the length of the migration, lets it answer everything. ### What the state file remembers of each store The `stores` key of [the state file](https://docs.jmrp.io/ghchronicle/configuration/#state_file) keeps, for every store a run writes to, where it points, the release that first wrote it and the release that last did. A first start, on a state file with no history, records the running release as the first writer of every store, so a fresh install has nothing to migrate. The first start after an upgrade from 2.6.1 or earlier records the first writer as unknown, which for a SQL file, a Graphite or a Telegraf reads as older than any change, so those show each change of a 2.x release as pending, or as a note where nothing can put it right, until it is applied. A sink pointed at another store starts a new record, and the one it leaves is kept while it still owes a refill or names a copy there, said at every start and by `-migrate` as `owed there`, and taken back if the sink points there again. A URL written another way, with a trailing slash, a capital or the default port, is the same store. A state file deleted after an upgrade takes the record with it, and the stores that cannot be asked are then taken to be the running release's; a refill still owed goes with it too, and nothing reads that history back until a `-backfill -families` of the families it names does. A change made before 1.0.0, the `state` and `reason` tags of the alert items, was never written by a release, so only a store that can be asked can show it. ## From 2.5.x to 2.6.x ### The first start pays in full, once 2.6.0 keeps what a sweep learned about GitHub in [a cache file beside the state file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it), and 2.5.x wrote none. So the first start finds nothing to read, says so at `debug` as `no cache file yet, the first pass of each family pays in full`, and asks everything once without an ETag. That is the price every 2.5.x restart paid: measured on the author's service on 2026-09-26, the first 38 minutes after a restart spent 1,092 charged `core` requests on passes that cost about 66 with the cache warm. From the second start on the file is there, and the log says `cache file read`. ### `achievements` walks the whole merged history, once The co-authored pull request count behind Pair Extraordinaire is kept in the state file from 2.6.0 on, and a 2.5.x state file holds none. The first `achievements` pass therefore walks every pull request the account has merged: 35 queries and 23.7 MB over 2,315 of them, measured on 2026-09-27. After that a pass walks the days since the count it keeps, and the whole history again once a week. ### `totals` runs first The pull request query is sized per repository from the counts `totals` reads, and 2.6.0 keeps those sizes in the cache file. The first sweep has none, so it runs `totals` before the pull requests whatever its cadence says. At its hourly cadence `totals` is usually due anyway after an upgrade, and then it simply runs first; only when it is not due, and the sweep is not a primed one, which runs every family, does the log say why it runs: ```text level=INFO msg="no page sizes remembered, running totals before the pull requests it sizes" ``` ### The slow families spread out A 2.5.x state file has the account's daily families marked at one instant and its twelve-hour ones at another, because that is when they ran, together. 2.6.0 starts at most one family of six hours or more a sweep, so the first day after the upgrade brings them in one a tick: over up to two and a half hours at the built-in cadences, and longer with `deps`, `history` or `joblogs` switched on. The log names who starts and who waits, under `slow families due together take turns`, and from then on each keeps its own time. See [the slow families take turns](https://docs.jmrp.io/ghchronicle/configuration/cadences/#the-slow-families-take-turns). ### `commented_elsewhere` drops The search behind `gh_account_total.commented_elsewhere` used to count the threads the account commented on in its own repositories too. From 2.5.2, whose changes shipped in 2.6.0, it leaves them out, as its two siblings already did, and the field keeps its name. So every store's series drops by the difference on the first `totals` sweep after the upgrade, from 125 to 55 on the account it was measured on. It is a correction, not a loss. ### PostgreSQL tables gain columns 2.6.0 adds fields to measurements whose tables an earlier release made, and a table an earlier release made has no column for them. The PostgreSQL sink that connects reads each table's columns the first time its process meets it and adds the missing ones with `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`. The SQL file sink cannot ask, so its file carries that statement for every field, and replaying it into a database an earlier release filled makes psql print a notice for every table and every column that is already there. They are expected: ```text NOTICE: relation "gh_repo" already exists, skipping NOTICE: column "stars" of relation "gh_repo" already exists, skipping ``` See [how the declarations arrive](https://docs.jmrp.io/ghchronicle/sinks/postgres/#how-the-declarations-arrive). ### The Stars column of Work elsewhere waits for `outbound` The "Work elsewhere" table joins `gh_upstream_repo`, a measurement 2.6.0 adds, for the stars of each repository. Until the first `outbound` pass of the new version writes it, which is within the hour, InfluxDB and PostgreSQL refuse the whole table rather than leave the column empty, and a range that ends before the upgrade has no stars to show in any store. Wait for that pass, or publish the dashboards after it; see [the data looks wrong](https://docs.jmrp.io/ghchronicle/reference/troubleshooting/#the-data-looks-wrong). ### A configuration copied from an older example keeps the old cadences Up to 2.6.0 the example configuration set every family under `every.families` to the built-in value of its release, so a `config.yaml` copied from it sets them all. One copied from 2.5.x keeps twelve of them slower than 2.6.0 runs them: `account`, `outbound` and `totals` at `12h`, `achievements` at `24h`, `stars`, `billing` and `analyses` at `6h`, `discussions` at `2h`, `deployments` at `1h`, and `activity`, `events` and `notifs` at `30m`. Nothing warns, since the warning is for a cadence four times faster than the built-in one. Deleting those lines is what picks up the current values; from 2.6.1 the example shows them commented out, so a copy of it sets none. See [every family, its group and its built-in cadence](https://docs.jmrp.io/ghchronicle/configuration/cadences/#every-family-its-group-and-its-built-in-cadence). ## To 2.6.1 ### Whether a comment is the accepted answer is a field `gh_discussion_comment` carried `is_answer` as a tag, and a maintainer accepts an answer days after the comment was written, so one comment read before and after that was two rows at the same instant. 2.6.1 writes no `is_answer` at all: the `answers` field every row already carried, 1 for the accepted answer and 0 for any other comment, says the same thing. A store written before 2.6.1 and since holds the measurement in two shapes until the measurement is dropped and filled again with a [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/). The dashboards read both shapes, one row per comment, and [the measurements page](https://docs.jmrp.io/ghchronicle/collectors/measurements/#how-to-read-the-tables) says what each store keeps and how to drop it. `-migrate` says which of the configured stores still hold the old shape, under `2.6.1/gh_discussion_comment/is_answer`: see [Migrations](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations). What the old shape still costs a reader is an answer accepted and taken back since: its old row says accepted, so both comment panels read it accepted until the store is brought along. Measured in the containerised suite with an answer taken back on the fake GitHub, in InfluxDB 3.11.2, PostgreSQL 18.6, Elasticsearch 9.5.3, Graphite 1.1.10-5 and the SQL file replayed into PostgreSQL: Answers elsewhere and Discussion answers read it accepted, one row for the comment, before the upgrade in every store; not accepted, still one row, in the first three after a start under `migrate: auto`, which leaves Graphite and the SQL file to their operator; and not accepted in all five after `-migrate -yes`, once the Graphite commands had run and the file had been replayed from where it was. ### A path setting is expanded `state_file`, `sinks.dedupe_file`, `log.file`, `sinks.file.path` and `sinks.sql.path` take a leading `~` as the home directory and a `${VAR}` from the environment, as credentials and addresses always did. A configuration that wrote either one used to get a directory named with those characters, under the working directory, and now gets the path it meant. Where that was the state file, the state the old release kept is in the directory with the odd name, and the new one starts without it: move the files across before the first start if the walks it saves are worth keeping. A `${VAR}` in a path that is unset or empty stops the start and names the key, rather than leaving the path without that part. See [`${VAR}` expansion](https://docs.jmrp.io/ghchronicle/configuration/#var-expansion). ### A Docker state volume needs no `chown` Up to 2.6.0 the image had no `/var/lib/ghchronicle`, so the volume a compose stack or a `docker run` mounted there was created owned by root, and the collector, uid 65532, saved neither its state nor its cache in it until the volume was handed over with a `chown`. From 2.6.1 the image carries the directory, owned by uid 65532, and Docker gives a new volume mounted there that owner. An empty volume an earlier image left to root is handed over the same way the first time a container of 2.6.1 is created on it, which `docker compose pull` followed by `docker compose up -d` does; one already handed over is left as it is. A host directory mounted there still needs its `chown`. See [what has to be writable](https://docs.jmrp.io/ghchronicle/install/docker/#what-has-to-be-writable). ### `achievements` walks the whole merged history once more The co-authored count behind Pair Extraordinaire now reads every commit of a pull request with more than a hundred, where 2.6.0 read the first hundred and counted the rest as a floor. That is a change to the rule the count is kept by, so the count a 2.6.0 state file holds is set aside and the first `achievements` pass after the upgrade walks every pull request the account has merged: 41 queries with the counts query, 41 points, in 1 minute 53 seconds over the account measured, on 2026-09-28. A `co-authored pull request count is a floor` warning that came back on every pass stops with it, unless a pull request's remaining commits really could not be read. See [`gh_achievement_progress`](https://docs.jmrp.io/ghchronicle/collectors/measurements/). > **What else changed** > > Every release's own notes, with what was measured and what was not, are in > [the changelog](https://docs.jmrp.io/ghchronicle/reference/changelog/). --- # The file The ghchronicle configuration: one YAML file, its credentials, addresses and paths expandable from the environment, and what each top-level block is for. Source: https://docs.jmrp.io/ghchronicle/configuration/ ghchronicle reads one YAML file, and the documented example in the repository is the place to start it from. ```sh cp config.example.yaml config.yaml ``` `config.example.yaml` documents every option in comments. It is the file to copy, and it is kept in step with the code: adding a setting means adding it there too. ## The blocks ```yaml github: # the token, the reserve, the timeout targets: # which repositories groups: # which categories of metric are collected at all sinks: # where the points go every: # how often things run: a default, per group, per family heartbeat: # how often the sweep loop wakes, for a test run log: # level, format, and an optional rotating file state_file: # what a restart remembers backfill: # the bound, when run with -backfill migrate: # what a start does about what an upgrade left in the stores grafana: # where the dashboard is published, when asked ``` Only `github.token` and one of `targets.user`, `targets.orgs` or `targets.repos` are genuinely required, plus at least one sink. Everything else has a default. ## `${VAR}` expansion A credential, an address or a file path is expanded from the environment at start-up. `${GITHUB_TOKEN}` becomes the value of that variable, or an empty string if it is not set, except in a file path, below. ```yaml github: token: ${GITHUB_TOKEN} sinks: influxdb: url: http://localhost:8181 token: ${INFLUX_TOKEN} ``` Expansion is per key, not over the whole file, and these are the keys it reaches: - credentials: `github.token`, `sinks.influxdb.token`, every value of `sinks.otlp.headers`, `sinks.telegraf.username` and `password`, `sinks.postgres.dsn`, `sinks.elasticsearch.api_key`, `username` and `password`, and `grafana.token`, `user` and `password`; - addresses: `github.base_url` and `web_url`, the `url` of `influxdb`, `loki`, `telegraf` and `elasticsearch`, `sinks.otlp.endpoint`, `sinks.graphite.addr`, `grafana.url` and `grafana.datasource.url`; - the identifiers of what Grafana already has: `grafana.dashboard_uid`, `grafana.datasource.uid` and `grafana.datasource.loki_uid`; - file paths: `state_file`, `sinks.dedupe_file`, `log.file`, `sinks.file.path` and `sinks.sql.path`, where a leading `~` is also the home directory, so `state_file: ~/.ghchronicle/state.json` is a file under it. Only `~` on its own or before a separator is read that way, `~name` is left as written, and a `~` where the process has no home directory is refused at start-up naming the key. So is a `${VAR}` in a path that is unset or empty, since the path left without it is not the one written: `state_file: ${STATE_DIRECTORY}/state.json` would be `/state.json`, with the ledger and the cache file beside it at the root of the filesystem. Before 2.6.1 no path was expanded, and that `~` in a `state_file` was a directory called `~` under the working directory. Every other value is read as written, so a `${VAR}` in `sinks.loki.tenant_id`, `sinks.influxdb.bucket`, `sinks.prometheus.listen`, a label, a prefix or a cadence reaches the collector as those characters. This is the whole reason the file can be committed. The configuration is the shape of the deployment and belongs in version control; the tokens are credentials and belong in an environment file with mode 600, or in a secret store. - /etc/ghchronicle/ - config.yaml world readable, version controlled - ghchronicle.env mode 600, never committed ## `github` ```yaml github: token: ${GITHUB_TOKEN} reserve_rate: 500 timeout: 30s # base_url: https://github.example.com/api/v3 # web_url: https://github.example.com ``` | Key | Default | Meaning | | -------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `token` | required | A classic or fine-grained personal access token | | `reserve_rate` | `500` | Calls never spent, so whatever else uses the token keeps working. A non-positive value falls back to the default | | `timeout` | `30s` | Per request. GraphQL over a large account can be slow. A Go duration; anything unparseable or not positive falls back to the default | | `base_url` | api.github.com | A GitHub Enterprise instance uses `https:///api/v3`. GitHub Enterprise Server does not serve the daily star history, so there [`gh_star_day`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#stars-and-forks) and the star-count panels drawn from it stay empty | | `web_url` | derived from the API | The site the profile page is on, read by `achievements` without the token. Needed only when `base_url` is a proxy in front of the API | The reserve is scaled per bucket; see [rate limits](https://docs.jmrp.io/ghchronicle/api/). ## `state_file` ```yaml state_file: /var/lib/ghchronicle/state.json ``` Nine things, and deleting the file costs a different one for each: - `last_run`, when each family last ran. Without it every family is due at once, so the next sweep is a full one, except that the service starts the families of six hours or more [one a sweep](https://docs.jmrp.io/ghchronicle/configuration/cadences/#the-slow-families-take-turns). - `first_saw`, when each repository's stargazer list was first walked whole. It is written only after a walk that came back without an error. Without it the one-off full walk of the stargazer list is done again. - `history_read`, when each repository's daily star history was last read whole. It is written only after a walk that reached the end of the history, not after one an error or a later page's 403 or 404 cut short. Without it that history is read whole again, a page per thirty weeks of the repository's life. - `last_head`, the commit each repository was on when the dependency diff last ran. Without it the dependency changes in the gap are gone: the next sweep has the photograph and no diff. - `last_full`, when each family that normally reads what changed last took the day's read: for `issues`, every open item and what moved since the day's read before it, recorded for each repository, so one that failed takes it again and no other does. Without it a repository reads as due, so the next sweep takes it, reaching back a month. - `last_notified`, where the inbox window was cut. Without it zero asks for the whole inbox. - `last_event`, the newest event the feed had. Without it empty reads the whole feed. - `coauthored`, the Pair Extraordinaire count the `achievements` family has settled, the last UTC day it covers, the version of the rule it was counted by and the day the whole history was last walked. Each pass walks only the pull requests merged since that day, and the whole history again once a week or when the rule has changed. Without it the next pass walks the account's merged pull requests whole, which on an account with 2,315 of them was 35 queries and 23.7 MB, where a pass that has it walks only the day in progress: the counts query and one page of the walk, two points, and 468 to 513 KB for each hourly pass the production proxy logged against the same account on 2026-09-27, a size that grows through the day with what is merged. - `stores`, for each store a run writes to, where it points, the release that first wrote it, the release that last did, the [migrations](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations) applied to it, the copies of old rows a migration set aside there, and the history a migration cleared there and has not read back yet. Without it a SQL file, a Graphite or a Telegraf is taken to be the running release's own, so what an earlier release left in it is no longer shown, and a store cleared and not read back looks like one that never held the old shape, so nothing reads its history back until a `-backfill -families` of the families the refill named does. Seven of the nine cost only quota, because what is collected again is keyed by measurement, tags and timestamp and overwrites what is already stored. `last_head` loses something: the dependency changes between the head it held and the next one are read from a range that nothing can name once the head is gone. So does `stores`: which release first wrote a store that cannot be asked what its rows are keyed by, and a refill a migration still owes, are questions nothing can answer afterwards. A file that is there and cannot be read or parsed is not taken for a new one: a run that writes to the stores stops and names it, since read as new it would forget a refill still owed, and its first save would put a new file in its place. A run as another user, root most often, is what leaves it unreadable; give it back to the user the collector runs as. A run that saves the file keeps what another process recorded of the stores since it read it, so two runs that overlap, a `-once` from cron across a `-migrate -yes`, do not undo each other's records of the stores. A run with `-card-only` writes none of the nine. Its points reach [the card](https://docs.jmrp.io/ghchronicle/card/) and no store, so a mark it left behind would make the next collection skip a family, or narrow a read, whose data went into a picture and nowhere else. It reads the file as any other run does. ### The other file a sweep remembers itself in ```yaml sinks: dedupe_file: /var/lib/ghchronicle/state-written.bin dedupe_horizon: 720h ``` | Key | Default | Meaning | | ---------------------- | ------------------------------------------- | -------------------------------------------------------------------------- | | `sinks.dedupe_file` | beside `state_file`, as `-written.bin` | The ledger of what has already been written. `off` disables it for every sink | | `sinks.dedupe_horizon` | `720h` | How long the ledger remembers a point nothing offers any more. A Go duration; anything unparseable or not positive falls back to the default | Losing it costs one sweep of rewriting and nothing else, which is exactly what a store that has been wiped and needs filling again wants. Both files want a persistent path: [only what changed is written](https://docs.jmrp.io/ghchronicle/sinks/#only-what-changed-is-written). ### The cache beside it ```text /var/lib/ghchronicle/state-cache.bin ``` Beside `state_file`, as `-cache.bin`, is what a sweep learned about GitHub that makes the next one cheap. There is nothing to configure: its place is the state file's. It holds four things: - the [conditional cache](https://docs.jmrp.io/ghchronicle/api/#etags-and-why-a-304-is-free): every answer asked for within twice the longest cadence the configuration runs, and never less than a day, 48 hours at the default cadences, with the ETag it came with, so a restart asks with `If-None-Match` and is answered a free 304; - the workflow runs whose jobs were written, so a restart does not list them again; - the [refusals](https://docs.jmrp.io/ghchronicle/api/#a-refusal-is-remembered-too), each until the end of its own day; - the page sizes the `totals` family gave the pull request query. All four used to live in the process, and every restart paid for them again. Measured on the author's service on 2026-09-26, the first 38 minutes after a restart spent 1,092 charged core requests on passes that cost about 66 with the cache warm. The runs are a claim that every store holds their jobs, so a start does not recall them where that is not known: where no write ledger remembers what the stores hold, which is the case once the ledger is deleted to fill a wiped store again, with `dedupe_file: off` or a store's own `dedupe: false`, and in a run that ends with its sweep; and where a destination has been added since the file was written. Their jobs are then listed and offered again with every other point. Two of those cases are news, a ledger the configuration keeps that reads empty and a destination added, and start-up says them: ```text level=INFO msg="the write ledger remembers nothing, listing the jobs of the runs the cache file remembers again" runs=666 level=INFO msg="a store was added since the cache file was written, listing the jobs of the runs it remembers again" stores=influxdb,loki,postgres written_to=influxdb,loki runs=666 ``` The others are how the configuration always runs, a store kept without a ledger or a run that ends with its sweep, and are said only at `debug`, as `not every store keeps a write ledger`. Every start that finds the file says what it read, at its first sweep, and one that finds none says so at `debug`: ```text level=INFO msg="cache file read" file=/var/lib/ghchronicle/state-cache.bin written=2026-09-27T15:27:41+02:00 answers=1170 runs=672 refusals=100 page_sizes=54 age=1s ``` It is written at most every five minutes while the collector runs, and once more when it stops: at the end of a `-once`, and when the service is asked to stop with `SIGTERM` or `SIGINT`, which is what `systemctl stop`, `docker stop` and Ctrl-C send. A process killed outright cannot, so `SIGKILL`, the kernel's out-of-memory killer, `docker kill`, `taskkill /F` and `Stop-ScheduledTask` lose what it learned since the last save, five minutes at most, and the next start asks those answers again. A save that fails says so, and so does a start that finds a file it cannot read: ```text level=WARN msg="cache file not saved" file=/var/lib/ghchronicle/state-cache.bin err="open /var/lib/ghchronicle/state-cache.bin.tmp: permission denied" level=WARN msg="cache file not read, the first pass of each family pays in full" file=/var/lib/ghchronicle/state-cache.bin err="..." ``` A run with `-card-only` reads it and does not write it, for the reason it leaves the state file alone. A `-backfill` reads it and does not write it either: the pages it walks are pages no sweep asks for, and kept they would crowd the sweeps' own answers out of the file. An upgrade keeps it: an answer is kept under the shape of what its collector reads, so the only answers asked for again in full are those of a collector that now reads something else. It is bounded. An answer larger than a megabyte is left out, and so is whatever comes past 64 MB, the least recently asked for first. Measured against the author's account of 37 repositories, one sweep of every family keeps 1,065 answers, 10.3 MB of them and 1.6 MB of file, none of them larger than 404 KB. Over days the cache in memory grows by the answers whose query carries a moving window: the author's service held 6,695 after five and a half days, about 99 MB, and the 2,517 of them asked for in the last 48 hours, which are what the file keeps, come to about 31 MB. It holds what GitHub answered about private repositories as well as public ones, and is written with mode 600, as the state file is. Deleting it costs one pass of each family at full price, the price every restart paid before it existed, and loses nothing. Every answer is asked for again, every refused feature is asked about again, which is also how to have a feature switched on today noticed before its day is out, the jobs of the newest runs are listed once more, and the first sweep runs `totals` before the pull requests, whatever its cadence says, so their pages are sized again. When `totals` was not due anyway, it says so: ```text level=INFO msg="no page sizes remembered, running totals before the pull requests it sizes" ``` A file that does not load in full, cut short, damaged or written in another format, is not read at all, with the warning above, and costs the same; the next save writes over it. It wants the same persistent path as the other two. ## `backfill` ```yaml backfill: since: 2y ``` Only applies to a run started with `-backfill`, and is overridden by `-backfill-since`. See [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/). A backfill keeps one more file beside `state_file`, `-progress.json`, which is where it records what it has already written so a stop costs one repository rather than the walk. There is nothing to configure: it appears when a backfill starts, it is removed when the walk reaches the end, and a sweep neither writes nor reads it. See [stopping one, and picking it up again](https://docs.jmrp.io/ghchronicle/how/backfill/#stopping-one-and-picking-it-up-again). ## `migrate` ```yaml migrate: auto ``` What a run that writes to the stores does, before its first sweep, about a store an earlier release left in a shape this one no longer writes. `-migrate` prints what each configured store holds of every such change; see [Migrations](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations). | Value | What a start does | | ------ | ------------------------------------------------------------------------------------------------------------------- | | `auto` | The default. Applies on its own a pending change that loses nothing, says so at `WARN`, reads back what it cleared, and what an earlier run cleared and did not finish reading, and warns about every other one at every start | | `warn` | Applies nothing and reads nothing back, and warns about every pending change, and every store still owed its history, at every start | A change loses nothing when all three hold: GitHub still serves the whole history, so reading it again brings back every row the store holds; the old rows are set aside for at least 24 hours rather than deleted, which InfluxDB 3, PostgreSQL and Elasticsearch do; and every row in the store is this configuration's, which the store is asked. There is no value that applies the rest on its own. A one-shot run, `-once` or `-backfill`, on a new state file applies nothing on its own either: that is every run of the Action without a state file restored, whose record of a refill still owed goes with its runner. Each warning names the two commands, `-migrate` for the plan and `-migrate -yes` to apply it, with the service stopped. Any other value is refused at start-up. The service also holds a lock beside the state file, `-lock`, for as long as it runs, which is what keeps `-migrate -yes`, and `-uninstall -yes` of the data or the state, from changing the stores under it. There is nothing to configure: it follows `state_file`, the operating system lets go of it however the process ends, and the file left behind only names the last process that held it. ## When it is wrong, it says so at start-up Configuration is validated before the first call is made, and the messages name the key and what it needs. | Message | Means | | --------------------------------------------------------------------- | --------------------------------------------------------------------- | | `github.token is empty and GITHUB_TOKEN is unset` | Exactly what it says | | `state_file: "${X}/state.json" names ${X}, which is unset, ...` | A path names a variable with no value: set it, or write the path out | | `targets: set at least one of user, orgs or repos` | Nothing to collect | | `sinks: enable at least one of ...` | A run that collects and discards is almost never what anyone meant | | `every.families.: unknown collector` | The name is not a family. The message lists the ones that exist | | `sinks.influxdb: url and bucket are required` | Each sink validates its own required keys and says which | | `sinks.sql.dialect: "mysql" is not postgres, the only dialect so far` | The value is not one of the accepted ones, and the message lists them | | `migrate: "off" is not auto or warn` | The same, for `migrate` | > **A cadence for a family that does not exist is fatal** > > `every` is checked against the list of known collectors at start-up rather > than being ignored. A typo there would otherwise mean a family silently > running at its default forever, which has bitten twice. ## The rest - [Targets](https://docs.jmrp.io/ghchronicle/configuration/targets/): which repositories, and the fork and archived defaults. - [Cadences](https://docs.jmrp.io/ghchronicle/configuration/cadences/): the `every` block, and what `0` means. - [Logging](https://docs.jmrp.io/ghchronicle/configuration/logging/): level, format and the rotating file. - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/): the `sinks` block, one page per store. - [Letting the binary publish the dashboard](https://docs.jmrp.io/ghchronicle/dashboards/#letting-the-binary-do-it): the `grafana` block. --- # Build the configuration A form that writes the config.yaml a normal run takes and the workflow step the Action takes, from the binary's own list of settings. Source: https://docs.jmrp.io/ghchronicle/configuration/builder/ Answer what your deployment is, and this writes the three things that run it: the `config.yaml` a normal run reads, the command that runs it, and the step a workflow gives the composite Action. All three come from one set of answers, and none of them asks for a credential. ## What the form allows and the parser refuses A form is a form: a destination is a checkbox and the keys it cannot resolve without are not, so a few clicks are enough to write a file that dies at start-up. The page warns about the ones its own generated list of settings can see, above the outputs, and names the rest here. Refusing the rest in the form would mean keeping a second copy of a parser rule on this page, which is the drift the whole arrangement exists to avoid. - A destination switched on with a required key left empty is refused by name, with an example: `sinks.loki: url is required, for example http://loki:3100`. The form warns about this one. - A credential field answered with the credential instead of the name of an environment variable is left out of the file. The form warns about this one. - A file with nothing in it is refused: `the file is empty`. - No account at all is refused: `targets: set at least one of user, orgs or repos`. - `heartbeat: 0` is refused: it must be positive, and leaving the key out is how the loop's tick is derived instead. - A cadence that is not a duration is refused where it is used: `every.families.repo: time: invalid duration "soon"`. - `sinks.elasticsearch.api_key` filled in beside `sinks.elasticsearch.username` is refused: `sinks.elasticsearch: set either api_key or username and password, not both`. - Writing `-` as `sinks.sql.path` while `sinks.stdout` is on is accepted and gives two writers one stream; pick one of them. > **What the form does with a credential** > > Where a setting is a credential, the form asks for the NAME of an environment > variable and writes `${NAME}` into the file. That is the expansion the binary > performs at start-up, and it is what lets the file be committed. An answer > that is not a variable name is not written at all: the form says so and > leaves the key out, rather than folding a pasted token into a reference to a > variable that will never exist. One field is not covered by that, > `sinks.otlp.headers`, which is free text because a header is not always a > credential: what you type there is written as it stands, so put a `${VAR}` > reference in it. The Action's token is a repository secret, which is what the > step references. ## The form This page is a form that writes a configuration. What follows is the inventory it offers, which is generated from the binary's own types, and the three outputs it starts from. - **The account and its API** - `github`: a block of settings - `github.token`: string, required, a credential, like `${GITHUB_TOKEN}` - `github.base_url`: string, like `https://github.example.com/api/v3` - `github.web_url`: string, like `https://github.example.com` - `github.timeout`: string, defaults to `30s`, like `30s` - `github.reserve_rate`: int, defaults to `500`, like `500` - **What to collect** - `targets`: a block of settings - `targets.user`: string, like `your-github-login` - `targets.orgs`: list, like `some-org, another-org` - `targets.repos`: list, like `someone/one-repo` - `targets.exclude`: list, like `someone/experiment-*` - `targets.include_forks`: bool, defaults to `false`, like `false` - `targets.include_archived`: bool, defaults to `false`, like `false` - `targets.include_private`: bool, defaults to `true`, like `true` - **Where the points go** - `sinks`: a block of settings - `sinks.influxdb`: a block of settings - `sinks.influxdb.url`: string, required, like `http://localhost:8181` - `sinks.influxdb.token`: string, a credential, like `${INFLUX_TOKEN}` - `sinks.influxdb.org`: string, defaults to `default`, like `default` - `sinks.influxdb.bucket`: string, required, like `github` - `sinks.influxdb.batch`: int, like `5000` - `sinks.influxdb.exclude`: list, defaults to `gh_job_log`, like `gh_job_log` - `sinks.influxdb.dedupe`: bool, defaults to `true`, like `true` - `sinks.prometheus`: a block of settings - `sinks.prometheus.listen`: string, defaults to `:9605`, like `127.0.0.1:9605` - `sinks.prometheus.path`: string, defaults to `/metrics`, like `/metrics` - `sinks.prometheus.no_prime`: bool, defaults to `false`, like `false` - `sinks.otlp`: a block of settings - `sinks.otlp.endpoint`: string, required, like `http://collector:4318/v1/metrics` - `sinks.otlp.headers`: map, a credential, like `Authorization: Bearer ${OTLP_TOKEN}` - `sinks.otlp.service`: string, like `ghchronicle` - `sinks.otlp.raw`: bool, defaults to `false`, like `false` - `sinks.otlp.batch`: int, like `2000` - `sinks.otlp.repeat`: string, like `1m` - `sinks.loki`: a block of settings - `sinks.loki.url`: string, required, like `http://loki:3100/loki/api/v1/push` - `sinks.loki.tenant_id`: string, like `tenant-one` - `sinks.loki.labels`: map, like `job: ghchronicle` - `sinks.loki.batch`: int, like `1000` - `sinks.loki.max_age`: string, like `1h` - `sinks.file`: a block of settings - `sinks.file.path`: string, required, like `/var/log/ghchronicle/points.lp` - `sinks.file.format`: string, one of `influx`, `json` - `sinks.file.max_bytes`: int, like `67108864` - `sinks.file.keep`: int, like `5` - `sinks.stdout`: bool, defaults to `false`, like `false` - `sinks.stdout_format`: string, one of `influx`, `json` - `sinks.telegraf`: a block of settings - `sinks.telegraf.url`: string, required, like `http://telegraf:8186/telegraf` - `sinks.telegraf.username`: string, like `telegraf` - `sinks.telegraf.password`: string, a credential, like `${TELEGRAF_PASSWORD}` - `sinks.telegraf.batch`: int, like `5000` - `sinks.telegraf.dedupe`: bool, defaults to `true`, like `true` - `sinks.graphite`: a block of settings - `sinks.graphite.addr`: string, required, like `graphite:2003` - `sinks.graphite.prefix`: string, defaults to `github`, like `github` - `sinks.graphite.batch`: int, like `1000` - `sinks.graphite.dedupe`: bool, defaults to `true`, like `true` - `sinks.sql`: a block of settings - `sinks.sql.dialect`: string, defaults to `postgres`, one of `postgres` - `sinks.sql.path`: string, required, like `/var/lib/ghchronicle/points.sql` - `sinks.sql.max_bytes`: int, like `67108864` - `sinks.sql.keep`: int, like `5` - `sinks.sql.dedupe`: bool, defaults to `true`, like `true` - `sinks.postgres`: a block of settings - `sinks.postgres.dsn`: string, required, a credential, like `${DATABASE_URL}` - `sinks.postgres.batch`: int, like `1000` - `sinks.postgres.dedupe`: bool, like `true` - `sinks.elasticsearch`: a block of settings - `sinks.elasticsearch.url`: string, required, like `http://elasticsearch:9200` - `sinks.elasticsearch.prefix`: string, defaults to `ghchronicle`, like `ghchronicle` - `sinks.elasticsearch.username`: string, like `elastic` - `sinks.elasticsearch.password`: string, a credential, like `${ES_PASSWORD}` - `sinks.elasticsearch.api_key`: string, a credential, like `${ES_API_KEY}` - `sinks.elasticsearch.batch`: int, like `1000` - `sinks.elasticsearch.dedupe`: bool, defaults to `true`, like `true` - `sinks.dedupe_file`: string, defaults to `ghchronicle-state-written.bin`, like `/var/lib/ghchronicle/state-written.bin` - `sinks.dedupe_horizon`: string, defaults to `720h`, like `720h` - **How often** - `every`: a block of settings - `every.default`: string, like `15m` - `every.groups`: map, like `ci: 1m` - `every.families`: map, like `deps: 24h` - **The run's own log** - `log`: a block of settings - `log.level`: string, defaults to `info`, one of `debug`, `info`, `warn`, `error` - `log.format`: string, defaults to `text`, one of `text`, `json` - `log.file`: string, like `/var/log/ghchronicle/ghchronicle.log` - `log.max_bytes`: int, like `67108864` - `log.keep`: int, like `5` - **The run** - `heartbeat`: string, like `15s` - `groups`: list, like `audience, account, repos` - `state_file`: string, defaults to `ghchronicle-state.json`, like `/var/lib/ghchronicle/state.json` - `backfill`: a block of settings - `backfill.since`: string, like `2y` - `migrate`: string, defaults to `auto`, one of `auto`, `warn` - `grafana`: a block of settings - `grafana.url`: string, like `http://localhost:3000` - `grafana.token`: string, a credential, like `${GRAFANA_TOKEN}` - `grafana.user`: string, like `admin` - `grafana.password`: string, a credential, like `${GRAFANA_PASSWORD}` - `grafana.folder`: string, like `GitHub` - `grafana.publish_on_start`: bool, like `false` - `grafana.dashboard_uid`: string, like `my-existing-dashboard` - `grafana.datasource`: a block of settings - `grafana.datasource.url`: string, like `http://influxdb:8181` - `grafana.datasource.uid`: string, like `ae3x9k2` - `grafana.datasource.sslmode`: string, like `require` - `grafana.datasource.loki_uid`: string, like `be7m1q4` - **config.yaml** Save this beside the binary, or at the path you give to -config, and run it with the command below. ```yaml github: token: ${GITHUB_TOKEN} targets: user: octocat sinks: stdout: true ``` - **The command that runs it** ```sh ghchronicle -config config.yaml -once ``` - **Workflow step** These answers need settings no input carries, so the step reads the file above. Commit it at that path. ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} config: .github/ghchronicle.yaml ``` ## What to do with each output There is not one command, which is why the form writes it. A configuration that names a destination is run with `-once`. A configuration that names none is refused by `-once`, with the parser asking for a sink you deliberately did not want, so it is run with `-card-only` and a path for the card instead. The command above changes with the answers, and so does the step. The step goes in the `steps:` of a workflow job. Answers no input can carry make the step read the file, and the file has to be committed at the path the step names; answers the four inputs can carry are written into the step itself, and then there is no file at all. A step with no file that names no destination is a card run, `mode: card` with a `card:` path, because that is the one thing the Action turns into `-card-only`. The two outputs default the opposite way on one setting, and the step is written so it says which one you are getting. `include-private` is off unless you ask for it, because a card that counts private repositories names them in a public README; `targets.include_private` is on unless you say otherwise, because the token already reaches them. So a file-less step always writes the input out rather than leaving it to a default that means the opposite of the file beside it. See [GitHub Actions](https://docs.jmrp.io/ghchronicle/install/actions/) for the workflow around it, and [the token](https://docs.jmrp.io/ghchronicle/start/token/) for what the secret needs to be able to do. ## The rest - [The file](https://docs.jmrp.io/ghchronicle/configuration/): what each block is for, and what happens when a value is wrong. - [Targets](https://docs.jmrp.io/ghchronicle/configuration/targets/): which repositories. - [Cadences](https://docs.jmrp.io/ghchronicle/configuration/cadences/): every family, its group and its built-in interval. - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/): one page per destination. --- # Targets Which repositories a sweep touches, and why forks and archived repositories are excluded by default. Source: https://docs.jmrp.io/ghchronicle/configuration/targets/ The `targets` block decides which GitHub repositories a sweep collects: an account's own, those of its organisations, and any named one. ```yaml targets: user: your-github-login orgs: [] repos: [] exclude: [] include_forks: false include_archived: false include_private: true ``` ## The keys | Key | Default | What it does | | ------------------ | ------- | --------------------------------------------------------------------- | | `user` | none | The account to collect. Its repositories are discovered automatically | | `orgs` | `[]` | Organisations to include as well | | `repos` | `[]` | Repositories to collect whatever the filters say | | `exclude` | `[]` | Shell globs, matched against `owner/name` | | `include_forks` | `false` | Whether forks are collected | | `include_archived` | `false` | Whether archived repositories are collected | | `include_private` | `true` | Whether private repositories are collected | At least one of `user`, `orgs` or `repos` must be set, or start-up fails with `targets: set at least one of user, orgs or repos`. `user` is also what makes the account-wide families possible. A configuration that names repositories alone has no login to hand to GraphQL, so the contribution calendar, the event feed, notifications, billing, packages and the outbound families are all skipped. ## Naming a repository overrides every exclusion `repos` is not another filter. It is a statement of intent, and it wins: naming `someone/thing` collects it even if it is a fork, even if it is archived, even if a glob in `exclude` would have matched it. ```yaml targets: user: acme repos: - someone-else/a-fork-i-actually-maintain exclude: - "acme/experiment-*" ``` `exclude` takes shell globs, so `someone/experiment-*` drops a whole prefix in one line. Naming one of the account's own repositories, a fork of yours for instance, costs no request to discover it: the listing already returned it, with the four things discovery wants to know about it. Only a name no listing returns, someone else's repository or one of an organisation not in `orgs`, is read on its own each time the list is rebuilt. ## Why forks and archived are off by default Both defaults exist for the same reason, which is the rate limit. - **A fork's traffic is almost always zero.** GitHub reports views and clones per repository, and for a fork that nobody visits, that is fourteen days of zeroes per sweep. It costs four calls per repository to learn nothing. - **An archived repository takes no more work.** Its stars, forks and watchers can still move, and those are read anyway, as below. Nothing else can, and collecting it spends the budget on rows that never move again. Turn either on when the assumption does not hold for you. A fork you actually develop in is a real repository with real traffic, and `repos` is the way to name that one without also collecting the forty bookmarks. Off is not invisible, on two counts. An archived repository still gets two rows from every `totals` sweep: `gh_repo_archived`, dated the instant it was archived, and `gh_repo_total`, its lifetime counts as they stand, stars and forks included, stamped at the sweep. The listing a sweep already pays for says which repositories are archived, and the `totals` family asks about all of them in one query per twenty five at its own cadence, so the _Repositories archived_ table exists from the first sweep, the account's star and fork totals count them, and the cost is a point per twenty five archived repositories per `totals` sweep. And a [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/) collects archived repositories in full whatever this key says, because their history is the account's history and one walk of it is enough. Forks stay as configured in both cases: an archived fork under `include_forks: false` has no row. > **Check the set before spending quota on it** > > ```sh > ghchronicle -config config.yaml -list > ``` > > That prints the repositories in scope and writes nothing. If something you > expected is missing, this is the command that tells you before a sweep spends > any quota on the wrong set. ## Private repositories `include_private` defaults to `true`, because a token that can see them was given deliberately and the point of the tool is to keep the history of the account, not of its public half. What is stored is the same measurements as for a public repository: counts, durations and names. Two things are worth knowing about what that means: - Repository and branch names appear as tag values, so they will be visible to anyone who can read the dashboard. - Only the _host_ of a webhook URL is stored, never the path, because the path usually carries a secret. Set it to `false` to collect only public repositories. ## The discovery cadence The repository list is rebuilt once an hour: at the first sweep an hour after the last listing, to within half a tick, the margin [a family's cadence](https://docs.jmrp.io/ghchronicle/configuration/cadences/#heartbeat-the-loops-tick-which-is-not-a-cadence) gets too. Repositories are created rarely and listing them costs a page per hundred, so a new repository can take up to an hour to enter the sweep. Before 2.6.1 the list had no margin and could outlive the sweep an hour after it by a few milliseconds, so at the quarter-hour tick a new repository waited up to an hour and a quarter about half the time. Naming it in `repos` does not change that; restarting the process does, and so does each `-once`, which lists them afresh. --- # Cadences The three layers of the every block, the built-in cadence of every family, the warning when a configuration speeds one up, and the heartbeat. Source: https://docs.jmrp.io/ghchronicle/configuration/cadences/ Each family of GitHub metrics is collected on its own interval, and the `every` block is where those intervals are changed. ```yaml every: default: 15m groups: ci: 1m feeds: 10m families: actions: 30s ``` ## The three layers Most specific wins. A family's own entry beats its group's, a group's beats `default`, and `default` beats the built-in value. Anything omitted falls through to the layer below it, so a file with only `families:` in it behaves exactly as it always did. The value is a Go duration: `30s`, `15m`, `2h`, `36h`. | Layer | Reaches | Written as | | ------------------ | ----------------------------------- | ----------------------- | | `every.families` | one family | `families: {keys: 24h}` | | `every.groups` | every family of one group | `groups: {ci: 1m}` | | `every.default` | every family neither above names | `default: 15m` | | the built-in table | every family none of the above name | nothing | They are three keys of a nested block rather than three flat ones because a flat map cannot hold this vocabulary. `security` and `account` are each both a family name **and** a group name, so `every: {security: 1m}` has two readings and no way to choose between them. Under `families:` the word is the family; under `groups:` it is the group; there is nowhere left for the question to arise. The same nesting is what makes `default` and `groups` safe as words: they are fields of a fixed structure, and an unknown key is refused at start-up, so no family can ever shadow a layer and no layer can ever shadow a family. > **The two broad layers never switch a family on** > > `default` and `groups` cannot reach a family whose built-in cadence is `0`. > `deps`, `history` and `joblogs` ship switched off and are enabled by naming > them under `families:` and by nothing else. `deps` alone is 1.8 MB of SBOM per > repository, and a default written to speed up the fast families must not also > switch on three families you never mentioned. It is the same rule `groups` > already follows. Turning it around gives the shortest way to collect a little: switch everything off with `default`, then name what you want back. ```yaml every: default: 0 families: traffic: 6h actions: 15m ``` ## Every family, its group and its built-in cadence The built-in values are not round numbers picked for tidiness. They came out of a costed audit, and the reason each one is what it is lives next to the number in the code. That is the source, and this table is pinned to it: a test in `internal/config` compares every group, every duration and every reason below against the code in both directions, so a family added, moved or re-costed without this table following it fails the build. `ghchronicle -groups` prints the same membership. | Group | Family | Default | Why that value | | ----------- | -------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `account` | `account` | `1h` | the contribution calendar moves with every contribution and the profile's counts with every follow and star, and a pass is one GraphQL point and seven REST requests, of which only the profile is charged: GitHub all but never answered it with a 304, once in 48 conditional reads, and the six package listings answer one until a package changes | | `account` | `achievements` | `1h` | the badges on the public profile page, which no API lists, and how far each tiered one is from its next tier; a pass transfers the page, 36 KB and off the budget, and the pull requests merged since the day before for two GraphQL points, so a tier reached in the afternoon shows in the afternoon, and the whole history is walked once a week, 24 MB | | `account` | `billing` | `1h` | the month in progress moves while continuous integration runs: it had changed at every one of 46 six-hourly reads measured, and a pass is two requests, that month and the one before, of which only the first is charged | | `account` | `history` | `0` | off until asked for by name: it walks every past year and the year so far, and the rows are idempotent | | `account` | `keys` | `24h` | an SSH or GPG key changes when somebody changes it, and what matters is its expiry date, not the hour it was noticed | | `account` | `outbound` | `1h` | stars given, work in other people's repositories and the answers accepted there, for about nine GraphQL points a pass, so the hour costs nothing that shows and an answer accepted this afternoon is on the dashboard this afternoon | | `account` | `profile` | `12h` | packages, gists and social accounts, all of them edited by hand | | `account` | `totals` | `1h` | the lifetime numbers move with every star, fork, merge and push, and a pass is one GraphQL point for the account, one per ten repositories and one per twenty-five archived ones set aside, and one request of the search budget | | `audience` | `forks` | `12h` | the whole list fits in one page, and a fork is a rare event | | `audience` | `stars` | `1h` | a star can land at any hour, and after the one full stargazer walk a pass is one GraphQL point per ten repositories for the newest hundred and one conditional request per repository for the daily history, which answered a free 304 to 184 of 185 measured | | `audience` | `traffic` | `6h` | the fourteen-day window is rewritten whole each time, so a missed sweep repairs itself on the next one | | `ci` | `actions` | `15m` | a workflow run is over in minutes, and its queue time is only worth watching while it is happening | | `ci` | `artifacts` | `1h` | artifacts appear with the run that made them and expire on a scale of days | | `ci` | `deployments` | `30m` | the surface a delivery dashboard reads, and the newest page is one GraphQL point per five repositories, which brought up to four new rows a pass measured, so the half hour costs little | | `ci` | `joblogs` | `0` | off until asked for by name: it is text rather than a measurement, it costs a request per failure, and it only makes sense with a log store attached | | `collector` | `ratelimit` | `15m` | free, and worth having at the resolution of the busiest family | | `feeds` | `activity` | `15m` | the repository log holds a hundred entries, which covered twenty-six hours on the busiest repository measured, and a pass is one or two conditional requests per repository, which answered a free 304 to 2,320 of 2,356 measured | | `feeds` | `events` | `15m` | one core request a pass, and the feed had moved in 43 of 46 half hours measured, so the quarter hour is how soon an event reaches the dashboard; the window, the last three hundred events of the past thirty days, is far wider than that | | `feeds` | `notifs` | `15m` | one core request a pass and twenty once a day for the whole inbox, and a thread shows only its latest move, so a move overtaken before the next read is never seen; half the passes measured brought a new row | | `repos` | `branches` | `24h` | branches are created and deleted all day, but the question the row answers is which are stale right now, which is a daily one | | `repos` | `deps` | `0` | off until asked for by name: the SBOM is 1.8 MB per repository and has its own budget of a hundred a minute | | `repos` | `inventory` | `24h` | four core requests per repository, for settings that change only when somebody changes them | | `repos` | `policyfiles` | `24h` | SECURITY.md, CODEOWNERS, dependabot.yml and FUNDING.yml move about once a quarter | | `repos` | `repo` | `1h` | stars, forks, languages and topics move slowly, and a pass is three REST requests per repository, most of them a free 304, and two GraphQL points per ten repositories | | `repos` | `rulesets` | `24h` | a ruleset is edited a few times a year and every version keeps its own date; both requests answer a free 304 until somebody edits one, the first pass after a restart included, since the ETag cache is kept beside the state file | | `repos` | `settings` | `6h` | webhooks, rulesets, environments and deploy keys change only when somebody changes them | | `security` | `analyses` | `1h` | GitHub prunes code scanning analyses and every scanned push adds some, and a pass is one conditional request per repository with code scanning, three of them charged in the median pass measured, the refusals of the rest remembered for a day | | `security` | `security` | `1h` | an alert is something to act on today, and the list of open ones is short | | `work` | `commits` | `1h` | one GraphQL point per repository whose default branch has a commit from the last two cadences, which 38 of 999 answers measured had, and one per twenty-five repositories, shared with issues and issueevents, to ask which, so the hour is how soon a push is charted | | `work` | `discussions` | `1h` | a discussion is answered over hours or days and few repositories have a forum, and a pass is two GraphQL points for each that does and nothing for the rest | | `work` | `issueevents` | `1h` | the timeline of what moved in two cadences, one GraphQL point for each repository where an issue or a pull request moved and nothing for the rest, so the hour is how soon a transition is worth seeing | | `work` | `issues` | `1h` | one or two GraphQL points per repository whose issues or pull requests moved in two cadences, nothing for the rest, and once a day a point or more for its open items and up to four a page for what moved in the day, so the hour is how soon a review or a merge is charted | | `work` | `planning` | `6h` | labels and milestones are edited by hand, a few times a week at most | | `work` | `stats` | `12h` | GitHub recomputes these slowly anyway, so asking more often returns the same numbers | ## The warning when a cadence is faster than the value is worth A broad layer is a cheap way to slow everything down and an expensive way to speed everything up. `default: 15m` asks GitHub for the account's SSH keys ninety-six times a day for a value that changes twice a year, and a group cadence does the same thing one level down: `work` holds families the audit measured at `1h` and at `12h`, so one number there is one number over six different answers. So every family a configuration collects **four times more often or more** than its built-in cadence is named at start-up, with the key that set it, both numbers, and the reason that number is what it is: ```text level=WARN msg="every.default sets keys to 15m against a built-in 24h, 96 times more often: an SSH or GPG key changes when somebody changes it, and what matters is its expiry date, not the hour it was noticed" level=WARN msg="every.groups.work sets stats to 1h against a built-in 12h, 12 times more often: GitHub recomputes these slowly anyway, so asking more often returns the same numbers" ``` It is a warning and never a refusal, and it is always per family, never per group. A line saying "the group `work` is too fast" would name nothing you can act on: the reason a cadence is what it is belongs to the family, so the warning has to as well. Four is the threshold because the built-in values stand on a ladder, `15m` `30m` `1h` `2h` `6h` `12h` `24h`, and the widest gap between two neighbouring rungs is three, from `2h` to `6h`. No family ships at `2h` since `discussions` moved to the hour, but the rung is still where one step down from `6h` lands. Four is therefore the smallest factor no single step down that ladder can reach. Moving one rung is a deliberate adjustment made by somebody looking at that family and stays quiet; four or more can only be a broad layer landing somewhere it was never chosen for, or a number typed without reading this table. Nothing warns for going slower. This is about waste, not about taste. ## Groups: collecting less than everything `every` sets how often a family runs. `groups` sets which families exist for this deployment at all. Omit the key and it collects for every group, which is the default and is what "every metric GitHub exposes" has always meant. ```yaml groups: [audience, account, repos, security] ``` Naming any group turns the rest off: their families are never requested and their measurements are never written. `ghchronicle -groups` prints the list, which is: | Group | Families | | ----------- | ---------------------------------------------------------------------------------------- | | `audience` | `forks`, `stars`, `traffic` | | `account` | `account`, `achievements`, `billing`, `history`, `keys`, `outbound`, `profile`, `totals` | | `repos` | `branches`, `deps`, `inventory`, `policyfiles`, `repo`, `rulesets`, `settings` | | `work` | `commits`, `discussions`, `issueevents`, `issues`, `planning`, `stats` | | `ci` | `actions`, `artifacts`, `deployments`, `joblogs` | | `security` | `analyses`, `security` | | `feeds` | `activity`, `events`, `notifs` | | `collector` | `ratelimit` | The top-level `groups` and `every.groups` are different questions about the same eight names: the first decides whether a group is collected at all, the second how often. A cadence under `every.groups` for a group the top-level `groups` leaves out is legal and warns, rather than failing: narrowing a deployment should not also mean pruning an `every` block you tuned last year. The two axes both have to say yes, and only one of them can say no. A group that is not named switches its families off whatever `every` says, and naming a group never resurrects a family whose cadence is zero. That is the whole of how a family ships switched off while the default is everything. Writing `groups: []` is refused rather than read as "collect nothing": omitting the key is how you ask for everything, so an empty list can only be a mistake. What GitHub deletes cannot be recovered later, whatever you do afterwards, and it sits in three different groups: the event feed, which keeps three hundred events and none older than thirty days, and the inbox, which keeps notifications for three months unless they are saved (`feeds`); the fourteen-day traffic window (`audience`); and job logs, deleted after the repository's retention period, ninety days by default, along with, from 1 October 2026, the workflow runs themselves (`ci`). Whatever GitHub dropped before ghchronicle read it does not exist anywhere. Everything else can be filled in later with `-backfill`. > **Panels for a group you switched off will show an error** > > A measurement that is never written does not exist as a table, and InfluxDB 3 > answers a query naming a table it does not have with an error rather than with > no rows. The dashboards are generated once, for the full set of metrics, so > every panel fed by a group you turned off shows that error. It is expected. > They are a demonstration of what the collector can chart, not a view that > reshapes itself around your configuration. PostgreSQL and Elasticsearch behave > the same way; Prometheus and Graphite have no schema to miss, so the same > panels there read No data instead. The dashboard sections are not aligned to the groups either, so a group switched off empties some panels of several sections rather than one section cleanly. `Stars and forks` reads `gh_repo` from `repos` as well as `gh_star` and `gh_star_day` from `audience`; `Code` reads `gh_workflow_run` from `ci` and `gh_repo_activity` from `feeds` alongside its own commits; `Inventory` reads from `repos` and `account`, and its licence panels read `gh_dependency_license`, which is family `deps` and ships switched off whether or not `repos` is selected. ## What `0` means `0` switches off every family the layer it is written on reaches. It is not "as often as possible" and not "use the default": those families never run, write nothing and cost nothing. ```yaml every: families: billing: 0 # no billing data at all artifacts: 0 # no artifact rows ``` Three families ship off and are enabled by giving them any duration, under `families:` and nowhere else: - **`joblogs`** is the tail of every failed job's log. It is text rather than a measurement, it costs a request per failure, and it only makes sense with a log store attached. The InfluxDB sink excludes it by default. - **`history`** walks every past year's contribution calendar, one GraphQL point per year, back to the day the account was created, and the year in progress from January to now. The past years never change and the rows are idempotent; the current year's row is a snapshot marked `partial`, so a daily cadence keeps it current and a single sweep leaves it frozen on the day it ran. - **`deps`** is the dependency graph. The SBOM is 1.8 MB per repository and has its own budget of a hundred a minute. A configuration that leaves nothing at all enabled says so at start-up rather than running an empty loop in silence. > **An unknown name is fatal at start-up** > > ```text > every.families.trafic: unknown collector (known: account, achievements, actions, activity, ...) > every.families.feeds: "feeds" is a group, not a family; every.groups.feeds is where a whole group's cadence lives > every.groups.actions: "actions" is a family, not a group; it is in group "ci", and every.families.actions is where its cadence lives > ``` > > Every name is checked against the known collectors and the known groups before > the first API call. A typo would otherwise mean a family silently running at its > default forever, and a name written under the wrong layer is told which layer it > belonged under. ## `heartbeat`: the loop's tick, which is not a cadence `heartbeat` forces how often the sweep loop wakes to ask which families are due. It gives no family an interval, it is compared against nothing in the table above, and it earns none of the warnings on this page. ```yaml heartbeat: 15s ``` Omit it and the loop ticks at the shortest cadence configured, held between one minute and one hour, which is what a real deployment wants: a family that runs every quarter of an hour is not delayed by one that runs every twelve hours. The upper end matters to a configuration whose cadences are all slower than an hour: the search for the shortest one starts at an hour, so the loop still wakes hourly and finds nothing due. Set it when you want the loop itself under control, which is almost always a test run. It is the only way to turn the loop faster than that one minute floor, because the floor exists to survive a mistyped cadence and an explicit heartbeat is not one. A family is due on the first tick at which its cadence has elapsed, to within half a tick. The loop reads the clock a few milliseconds after it wakes, by an amount that changes from tick to tick, and without that margin a family whose cadence is a whole number of ticks waited one tick more about half the time: until 2.6.0, which shipped the fix written for 2.5.2, the quarter hour families ran every 24 minutes on average and the hourly ones every 69. Half a tick absorbs that and never lets a family run a tick early. A heartbeat **longer** than the shortest cadence holds that family back, and start-up says so: ```text level=WARN msg="heartbeat is 1h and the shortest cadence is 15m (actions), so no family can run more often than every 1h" ``` ## The slow families take turns The running service starts one family whose cadence is six hours or more in each sweep, or the fewest that still keep every cadence when one cannot (below), and leaves any other that is due for the next tick. Families that run in the same sweep are marked with the same instant in the state file, so they come due together again at every cadence, for ever. A group like that forms whenever many families are marked at once: a fresh install, a `-once` or a backfill run before the service, a family switched on. Until 2.6.0 the production account's eight daily families ran in the first sweep of the UTC day, every day, and its five twelve-hour ones together in one sweep of their own. On 2026-09-26 the daily sweep took 294 billable `core` requests and 24.3 MB against 17.5 requests for the median sweep, and with the twelve-hour sweep 45 minutes after it, its hour was eight times the median hour in `core` requests. The calendar put them there, not the work. The family that starts is the one that has been due the longest, counted from the later of its last run and the last time the process let it start. The second half matters for a family whose every pass fails: a failed pass is not marked as run, so by its last run alone it would be the most overdue family on every sweep and would take every turn. Counted this way it goes behind the others. A family waits for each of the others at most once, so the worst wait is one tick for each other slow family, and only the last of a group that formed waits that long. At the quarter-hour tick the built-in cadences give: | Schedule | Families of 6h or more | Longest wait | | --------------------------------------------- | ---------------------- | ----------------- | | the built-in cadences | 11 | 10 ticks, `2h30m` | | with `deps` and `history` at `24h` | 13 | 12 ticks, `3h` | | with `deps`, `history` and `joblogs` at a day | 14 | 13 ticks, `3h15m` | That wait is paid once. Once two families have run in different sweeps they come due in different sweeps, and nothing moves them again: a simulated week of the thirteen, all due in one sweep at the start, had every family start exactly a cadence after its previous start from the first day on. The log says which family started and which are waiting, so a slow family missing from a sweep it was due in is accounted for: ```text level=INFO msg="slow families due together take turns" starting=planning waiting=settings,traffic,forks,profile,stats,branches,inventory,keys,policyfiles,rulesets ``` One a sweep holds while it can keep every cadence, and a configuration with a longer tick can have more slow families than that. `every.default: 6h` ticks hourly, because nothing in it runs more often, with thirty-one families at six hours, and one a sweep would start each of them every thirty-one hours. A sweep there starts the fewest that fit, six, and no family waits as long as its cadence. Only the service takes turns. `-once` runs every family that is due, because it has no next tick to leave one for: run once a day by a scheduler, it would leave it for a day. A backfill, a card and the first sweep of a service that primes the [Prometheus exporter](https://docs.jmrp.io/ghchronicle/sinks/prometheus/#scraping-it) run every family whatever the state file says. The primed sweep then records only the families that were due, so a restart does not put the others back on one instant, and each keeps the turn it had. ## Making them slower If `core` is the budget that runs short, lengthen `artifacts` and then `actions`, the two largest REST costs, which grow with how busy the repositories are. If it is `graphql`, lengthen `issues`, `issueevents` and `commits`, which read only the repositories where something moved and so cost more the more of them move. They are not the only ones that follow activity rather than size: `activity` is charged only for a repository whose log moved, and `outbound` a point more for each further hundred of what it reads. See [what scales with activity, not with size](https://docs.jmrp.io/ghchronicle/api/cost/#what-scales-with-activity-not-with-size). The symptom of cadences that are too fast is a warning every sweep: ```text level=WARN msg="rate limit reserve reached, family skipped" family=actions ``` ## Cadence is not resolution Lengthening a cadence does not coarsen the history, because the points are dated by the thing that happened rather than by the sweep. Collecting workflow runs every hour instead of every fifteen minutes still records each run at the second it finished. What a longer cadence risks is missing a window entirely: `events` reads the last three hundred events, `notifs` only the latest move of each thread and `activity` the last hundred entries of each repository's log, so whatever leaves one of them between two sweeps is gone. --- # Logging Level, format, the rotating file that never replaces standard error, and the two lines worth alerting on. Source: https://docs.jmrp.io/ghchronicle/configuration/logging/ ghchronicle logs to standard error, and the `log` block sets the level, the format and an optional rotating file. ```yaml log: level: info # debug, info, warn, error format: text # text or json file: /var/log/ghchronicle/ghchronicle.log max_bytes: 67108864 keep: 5 ``` ## The keys | Key | Default | Meaning | | ----------- | ---------- | --------------------------------------------- | | `level` | `info` | `debug`, `info`, `warn` or `error`; anything else behaves as `info` | | `format` | `text` | `text` for a human, `json` for a shipper; anything else is `text` | | `file` | none | A rotating file **as well as** standard error | | `max_bytes` | `67108864` | Rotate at 64 MiB. Anything not positive falls back to the default | | `keep` | `5` | How many rotated files to keep. Anything not positive falls back to the default | ## Both, never instead A configured `file` does not replace standard error, it is written in addition to it. Under systemd the journal is where anyone looks first, and a log file that silently took the journal's place would be a trap: `journalctl -u ghchronicle` would go quiet and the obvious conclusion would be that the service had stopped. Rotation is by size with numbered suffixes, so retention is a count rather than a date and two rotations in the same second cannot collide. The size counter is read from the file at start-up, so a restart does not reset it and let the file grow without bound. ## What the log says on a good day Taken from the author's service on 2026-09-27, a start and the lines of its sweeps: ```text level=INFO msg="ghchronicle running" tick=15m0s tick_from="shortest cadence" level=INFO msg="cache file read" file=/var/lib/ghchronicle/state-cache.bin written=2026-09-27T15:27:41+02:00 answers=1170 runs=672 refusals=100 page_sizes=54 age=1s level=INFO msg="repositories discovered" count=37 archived_aside=17 level=INFO msg="slow families due together take turns" starting=planning waiting=settings,traffic level=INFO msg=written sink=influxdb family=repo points=934 unchanged=1955 level=INFO msg="nothing moved since the window, repositories left unread" family=commits repos=34 level=INFO msg=written sink=influxdb family=commits points=17 unchanged=0 level=INFO msg="rate budget" bucket=core remaining=4477 limit=5000 level=INFO msg="points already written and not sent again" sink=influxdb skipped=54511 level=INFO msg="sweep finished" ``` The first two come once, when the process starts: the tick the loop wakes at, and what [the cache beside the state file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it) gave back. The list of repositories is rebuilt once an hour, and `archived_aside` counts the archived ones [set aside](https://docs.jmrp.io/ghchronicle/configuration/targets/#why-forks-and-archived-are-off-by-default), which still get their two rows. In `written`, `points` is what reached the store and `unchanged` what the write ledger left out because the store already holds it; see [what the written line counts](https://docs.jmrp.io/ghchronicle/sinks/#what-the-written-line-counts). A family of six hours or more that is due behind another one is named under `waiting`, see [the slow families take turns](https://docs.jmrp.io/ghchronicle/configuration/cadences/#the-slow-families-take-turns), and `commits`, `issues` and `issueevents` say how many repositories they left unread because nothing moved in them, see [asking first what moved](https://docs.jmrp.io/ghchronicle/api/cost/#asking-first-what-moved). A family that is not due yet simply does not appear. That is normal, and it is the first thing to check when a measurement seems to be missing: with a twelve-hour cadence, half a day of logs can legitimately never mention `stats`. ## The two lines worth an alert **`rate limit reserve reached`** means a family was skipped to protect the budget. ```text level=WARN msg="rate limit reserve reached, family skipped" family=artifacts ``` Once is fine. Every sweep means the cadences are too fast for the number of repositories. **`family failed everywhere, not marking it as run`** means every repository failed for one family. ```text level=WARN msg="family failed everywhere, not marking it as run" family=security ``` The family is deliberately not marked as done, so it is retried at the next cadence rather than being treated as complete. That is the line that separates "a feature is off on one repository" from "the token lost a scope". ## Debugging ```yaml log: level: debug ``` Debug adds these lines, and no more: - `loaded the written-points ledger`, with how many points it came back with, when the service starts; - `no cache file yet, the first pass of each family pays in full`, at the first start with a state file; - `not every store keeps a write ledger`, when the runs the cache file remembers are listed again because a store keeps no ledger or the run ends with its sweep; - `no targets.user, account-wide families skipped`; - `sink dropped old entries`, with how many a sink left out for being older than its horizon, the only way to see that a push was trimmed rather than rejected; - `card-only sweep, the state file is left as it was`; - `cache file saved`, with how many answers and runs it holds and how long it took; - `migration not needed` and `migration applied before`, at every start, for each change a store never needed or has had applied already; - `migration noted: nothing is changed` and `migration frozen`, at every start after the first, which says each at `INFO`; - `set-aside copy forgotten`, when a copy a migration set aside falls due and the state file stops naming it: `the store purges it itself` for one InfluxDB 3 purges on its own schedule, and `the store keeps it for good, and -migrate asks the store for it` for one it keeps for good. There is no per-request log: the client in `internal/ghapi` carries no logger, so which endpoint was called and which answer came back 304 are not visible at any level. That includes its one retry: a REST request answered 500, 502, 503 or 504 is asked again two seconds later, silently, and so is a job log's download from storage, so a `collector failed` naming one of those four has already failed twice. > **JSON for a shipper, text for a person** > > `format: json` emits one object per line, which is what Promtail, Vector or > Filebeat want. It is the same information; only the encoding changes. > Combining `format: json` with a `file` is the arrangement for a host that > already ships logs somewhere. ## Not to be confused with the job log collector `log` is the tool's own diary. `every.families.joblogs` is a _collector_: it fetches the last forty lines of every failed GitHub Actions job and turns them into points. They are unrelated settings, and the second one belongs in a log store rather than in a metrics database. See [Loki](https://docs.jmrp.io/ghchronicle/sinks/loki/). --- # The sweep What one ghchronicle sweep over the GitHub API does, in what order, and why each family of metrics has its own cadence. Source: https://docs.jmrp.io/ghchronicle/how/ A sweep is one pass over every family whose interval has elapsed, except that the running service starts one family of six hours or more in each sweep, more only where one could not keep every cadence, and leaves the others that are due for the next ticks: the [slow families take turns](https://docs.jmrp.io/ghchronicle/configuration/cadences/#the-slow-families-take-turns). It is an increment, not a rebuild: it asks for the little that can have changed since last time, writes what it got to every configured store, and records when each family ran. ## The shape of one pass ```mermaid %% Generated by site/scripts/gen-figures.mjs from internal/config/config.go and internal/run/runner.go flowchart TD T["Timer"] --> D["Discover repositories
(rebuilt hourly)"] D --> A["Account-wide families
account, totals, ratelimit, events, notifs,
billing, profile, outbound, history,
achievements, keys"] D --> R["Per-repository families
traffic, repo, branches, stars, issues,
issueevents, actions, artifacts, security,
stats, discussions, commits, activity,
analyses, forks, planning, joblogs, settings,
rulesets, inventory, deployments, policyfiles,
deps"] A --> B{"Budget above
the reserve?"} R --> B B -- "no" --> S["Skip the family
and warn"] B -- "yes" --> C["Collect"] C --> P["Dated points"] P --> H["Destinations that keep the date
InfluxDB, file, stdout, Telegraf, Graphite,
PostgreSQL to a file, PostgreSQL,
Elasticsearch"] P --> L["Loki
the twenty-two event renderings"] P --> RD["Reducer
current values"] RD --> G["Prometheus
OTLP, when raw: false"] C --> M["Mark the family as run
in the state file"] ``` Two things in that diagram are the whole design. Every collector produces _dated_ points, and the stores that cannot hold a date get them reduced to current values first, by the same reducer, before they ever see the data. That is the subject of [dating a point](https://docs.jmrp.io/ghchronicle/how/dating/). ## Why families and not one interval The surfaces move at wildly different speeds. Workflow runs finish every few minutes on a busy account. Labels and milestones are edited a few times a week. The list of forks changes a few times a year. One interval for all of them would either waste the rate limit on the slow ones or lose the fast ones, so each family carries its own. | Family | Default | Collects | | -------------- | ------- | ---------------------------------------------------------------------------------------- | | `actions` | 15m | Workflow runs, jobs, steps, the Actions cache | | `activity` | 15m | The repository activity log, where a force push is recorded | | `events` | 15m | The account event feed, which keeps only the last 300 | | `notifs` | 15m | The notification inbox | | `ratelimit` | 15m | What the collector has left to spend, in each budget | | `deployments` | 30m | Deployments and their environments, batched over every repository | | `account` | 1h | Profile, contribution calendar, contribution totals | | `achievements` | 1h | The profile badges, and the distance to each next tier | | `analyses` | 1h | Code scanning analyses, which GitHub prunes | | `artifacts` | 1h | Artifacts and their expiry | | `billing` | 1h | Usage per day, product, SKU and repository | | `commits` | 1h | Lines changed and signature state, per commit | | `discussions` | 1h | The forum half of a repository | | `issueevents` | 1h | The timeline of what moved: labels, assignments, transitions | | `issues` | 1h | Pull requests, issues and reviews, per item | | `outbound` | 1h | Stars given, and work in other people's repositories | | `repo` | 1h | Stars, forks, languages, topics, releases, rulesets | | `security` | 1h | Dependabot and code scanning alerts | | `stars` | 1h | Stars per day for every repository, and the stargazer walk once, then the newest hundred | | `totals` | 1h | The lifetime numbers, asked of GitHub rather than added up here | | `planning` | 6h | Labels and milestones | | `settings` | 6h | Webhooks and their deliveries, environments, deploy keys | | `traffic` | 6h | The whole 14-day window, rewritten | | `forks` | 12h | Who forked, and when | | `profile` | 12h | Packages, gists, social accounts | | `stats` | 12h | Commits per week, the punch card, the workflow definitions | | `branches` | 24h | Which branches are live and how stale each tip is | | `inventory` | 24h | What a workflow's own token may do, both secret stores, default code scanning | | `keys` | 24h | The account's SSH and GPG keys, and when each expires | | `policyfiles` | 24h | SECURITY.md, CODEOWNERS, dependabot.yml and FUNDING.yml | | `rulesets` | 24h | Every version of every ruleset's changelog | | `deps` | off | The dependency SBOM of each repository, and what changed | | `history` | off | Each year's contribution calendar, the current one included | | `joblogs` | off | The tail of every failed job's log | Setting any of them to `0` switches it off entirely. See [cadences](https://docs.jmrp.io/ghchronicle/configuration/cadences/). ## Discovery The repository list is rebuilt once an hour, by the first sweep that finds it an hour old, give or take half a tick: a sweep reads its clock a few milliseconds either side of the hour, and without that margin a repository created in between waited a tick more. Repositories are created rarely and listing them costs a page per hundred, so anything shorter spends quota to learn nothing. Forks and archived repositories are excluded by default, for the reason set out in [targets](https://docs.jmrp.io/ghchronicle/configuration/targets/). ## Failure is per repository, not per sweep A family that fails on one repository is logged and skipped; the sweep continues. This matters more than it sounds, because "failure" here is usually a feature being switched off: of fifty repositories, most have Dependabot off, and each of them answers 403. Treating that as an error would lose the other forty-nine. There is one exception, and it is deliberate. A family where _every_ repository failed is not marked as run. Marking it would hide the outage until the next cadence, which for the twelve-hour families is half a day. ```text level=WARN msg="family failed everywhere, not marking it as run" family=security ``` ## The state file `state_file` holds [nine things](https://docs.jmrp.io/ghchronicle/configuration/#state_file), and the two a sweep is judged by are when each family last ran and when each repository was first seen. The second is what makes the one-off full walk of the stargazer list happen once instead of on every sweep, as `history_read` does for the daily star history. It is written through a temporary file and renamed, so a crash mid-write cannot leave a truncated state that would trigger a full re-collection. Beside it the collector keeps what makes a sweep cheap rather than what makes it complete: the ETag cache, the workflow runs whose jobs were written, the refusals and the pull request page sizes, in [`-cache.bin`](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it). A restart reads it back and starts where the last process stopped, not from nothing, and deleting it costs one full-price pass of each family and loses nothing. > **Three runs collect every family whatever the state says** > > An exporter holds its samples in memory, so a restart empties it and it stays > empty until each family's cadence comes round, which for the twelve-hour ones > is half a day of a dashboard reading zero. Paying for one full sweep is the > cheaper mistake, so when the service feeds the [Prometheus > exporter](https://docs.jmrp.io/ghchronicle/sinks/prometheus/#scraping-it) its first sweep after > start-up runs every enabled family whatever the state file says, unless > `no_prime` is set. It records as run only the families that were due, so the > others keep the turn they had. `-once` starts no exporter and does not prime. > A [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/) runs every family too, because > reaching as far back as GitHub allows is the whole point of asking for one, > and so does a run drawing [a card](https://docs.jmrp.io/ghchronicle/card/), because every number > on the card comes from that one sweep and a family skipped as not due would > be a zero on the picture. ## The brake Before each family the collector checks the three rate buckets it actually spends from. If any of them is at or below its reserve and the window has not reset yet, the family is skipped and a warning is logged rather than the budget being spent to the last call. See [rate limits](https://docs.jmrp.io/ghchronicle/api/). A [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/) is the opposite intention: it waits for the window to turn over instead of skipping. --- # Dating a point Every point carries the moment the thing happened, and that single rule decides what the whole project can answer. Source: https://docs.jmrp.io/ghchronicle/how/dating/ **A point carries the date the thing happened, not the date it was collected.** That is the one rule everything else follows from. A workflow run is stamped when it finished. A star is stamped when it was given. A traffic day is stamped at that day's own date. A pull request is stamped when it closed. None of them is stamped at the instant the collector noticed. ## The three kinds of point Not everything GitHub reports has a date of its own, so there are three treatments and each measurement declares which one it gets. | Kind | Stamped at | Example | | --------- | ----------------------------- | --------------------------------------------------------------- | | **Dated** | the moment the thing happened | `gh_star`, `gh_workflow_run`, `gh_traffic`, `gh_commit` | | **Daily** | the start of the UTC day | `gh_traffic_referrer`, `gh_label`, `gh_milestone`, `gh_webhook` | | **Now** | the instant of the sweep | `gh_repo`, `gh_actions_cache`, `gh_dependabot_alert` | Daily is for a snapshot with no date of its own. GitHub returns the top ten referrers of the trailing fourteen days as one list with no day attached, so it is not a series. Stamping it at the instant of the sweep would write four copies a day and any query that summed them would report four times the traffic. Stamping it at the start of the UTC day means a day's sweeps rewrite one row. Now is for something that is genuinely a current state. The size of the Actions cache, the number of open alerts, the inventory of workflows: none of these happened at a moment, so pretending otherwise would be a lie with a timestamp on it. ## Why the rule exists: re-collection has to converge InfluxDB keys a point by measurement, tag set and timestamp. Three fields, one row. Write the same three again and the row is replaced, not added to. Written out, two sweeps six hours apart offer the same day twice, and the second replaces the first because the three keys are identical: ```text gh_traffic,owner=acme,repo=telemetry,kind=views count=220i,uniques=131i 1757203200000000000 gh_traffic,owner=acme,repo=telemetry,kind=views count=238i,uniques=140i 1757203200000000000 ``` ```text gh_traffic,owner=acme,repo=telemetry,kind=views count=238i,uniques=140i 1757203200000000000 ``` One row, carrying the newest number GitHub reported for that day. Stamp those two lines at the moment of collection instead and they are two rows, and every sum over them is wrong by however many sweeps have run. That is what makes the whole design work. GitHub's traffic window is fourteen days, and the collector rewrites _all fourteen_ on every sweep rather than trying to work out which day is new: GitHub keeps 14 days of traffic and the collector re-reads all of them 4 times a day. Dated as built, those 4 sweeps land on the same 14 rows and the newest count replaces the one before it. Stamped at the moment of collection, each sweep adds what it read to what is already there. | After a week, per repository | Dated, as built | Stamped at collection | | --- | --- | --- | | Rows written per sweep | 14 | 14 | | Rows in the store | 14 | 392 | | Copies of each day | 1 | 28 | | A sum of the month's views | the traffic | 28 times the traffic | The same property is what makes a backfill safe to run twice, and what lets you delete the state file without corrupting anything: re-collection rewrites rows it has already written. > **Every history store has this property** > > It is not InfluxDB-specific. The SQL sink's primary key is `(time, tag > columns)`, which is the same series key spelled as a constraint, and its > inserts end in `ON CONFLICT ... DO UPDATE` rather than `DO NOTHING`. > Elasticsearch derives the document id from the measurement, the tags and the > timestamp, and indexes rather than creates. Graphite writes into the whisper > slot the timestamp names. All four converge for the same reason. ## What Prometheus structurally cannot hold Prometheus stamps a sample at scrape time. It does not take a timestamp from the producer, and it rejects anything meaningfully older than now. This was measured rather than assumed. Against **Prometheus 3.14**, with `--web.enable-otlp-receiver` and `out_of_order_time_window: 30m`, a sample dated two days back comes back as **HTTP 400**. Half of what this collects is older than that on purpose: a star from 2020, a pull request merged in July, yesterday's traffic. So there is no configuration of Prometheus in which the dated history survives. Widening the out-of-order window moves the boundary; it does not remove it. > **Do not give the exporter timestamps** > > The obvious "fix" is to have the exporter emit each sample with the point's > own timestamp. Prometheus's exposition format allows it, and Prometheus will > refuse the ones that matter. The exporter deliberately drops the timestamp, > and the reduction below is why that is not a loss. ## The reducer, and what it makes of each measurement Because the store cannot hold the history, the reduction happens _before_ Prometheus or an OTLP backend with `raw: false` ever sees the data. `Summarize` gives each measurement one of four rules. | Rule | What it does | Used for | | ---------- | ------------------------------------------------------------- | ---------------------------------------------------------------------------- | | `keepLast` | The most recent point per label set wins | Snapshots: `gh_repo`, `gh_account`, `gh_release` | | `sum` | Every row in the batch is added up | Windows: views over the fourteen days | | `count` | The points become a count plus the mean of each numeric field | Dated items: pull requests become "how many merged" and "how long they took" | | `skip` | Nothing is served | History with no honest current value | A measurement with no rule is skipped rather than guessed at. That is the safe default, and it is deliberate: without it, a new collector could quietly flood an exporter with one series per star. A batch is reduced the way the stores hold it. Two points with the same measurement, tags and time are one row in InfluxDB, PostgreSQL, Graphite and Elasticsearch, so they are one row here too: `sum` adds it once, `count` counts and averages it once, and `keepLast` keeps the later of two readings stamped at the same instant. The row takes a field both points carry from the later one and keeps a field only the earlier one carried, as InfluxDB, PostgreSQL and Graphite do; Elasticsearch indexes the later point as the whole document, so a field only the earlier one carried is gone there. A tag with an empty value is no tag, as in InfluxDB, PostgreSQL and Elasticsearch; Graphite writes it as the node `none`, so there the point with it and the point without it are two paths. Across sweeps nothing more is needed, since each sweep's gauge replaces the last one and `total` remembers every item it has already counted. `sum` adds up every number but a price. The bill repeats a SKU's price on each of its rows, one per repository and day, so the reduction keeps the highest, which is the `MAX(price_per_unit)` the SQL dashboards read; added up, it read the price times the days billed. `count` averages each of their numbers except the identifiers. A field that joins one row to another, `run_id`, `workflow_id`, `pull_request`, `number`, `stack` and the rest, is a name and not a quantity: averaged over a count it becomes a number shaped exactly like the identifier it is made of and belonging to nothing, and `gh_deployment` published `run_id_mean` that way. Those fields are left out of the reduction, and so are the markers a point carries only so that it has a field at all, whose mean is 1.0 for ever. The reducer also publishes `total`, a running count of distinct items seen per series. That is what lets a Prometheus dashboard answer "per day" at all, through `increase()` over a monotonic counter, since it has no rows to count. ### The twelve that are never served Twelve of the measurements carry `skip`, so a Prometheus exporter and an OTLP backend with `raw: false` never see them. Nine are history, two are size, and one is text: | Measurement | Why | | -------------------------- | ------------------------------------------------------------------------------------- | | `gh_artifact` | History. One row per artifact ever, and none of them moves again | | `gh_commit_punchcard` | Size. One series per repository, weekday and hour | | `gh_commits_week` | History. The weekly commit series | | `gh_contribution_day` | History. The green calendar, one row per day | | `gh_contribution_day_repo` | History. The same calendar split per repository, which would mint a series per day | | `gh_job_log` | Text, not a number. It belongs in a log store | | `gh_package_version` | History. The publication date of every tag; the count of them is a field on `gh_package` | | `gh_release_asset` | Size. One series per file ever published | | `gh_release_published` | History. The publication date of every release; `gh_release` is already a series per release | | `gh_star_day` | History. Stars per day, revised backwards when a star given in the last thirty weeks is taken back; the count is `gh_repo.stars` | | `gh_traffic_path` | History. The per-day paths | | `gh_workflow_step` | History. The per-step timings | The two skipped for size are the ones worth knowing about, because they are real numbers rather than history: measured, together they were four fifths of the exporter's entire output. Both are drawn properly by the InfluxDB dashboard, and `gh_release` keeps the per-release download counts the assets were being read for. ## Which store keeps what | Store | The dated history | Why | | --------------------------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------- | | InfluxDB, PostgreSQL, Graphite, Elasticsearch | yes | The timestamp is part of the identity of a row | | Telegraf | as far as its outputs allow | It forwards the timestamps unchanged; an output that stamps at receipt loses them | | File and stdout | yes | The timestamp is in the line | | Loki | recent events only | Loki refuses an entry too far behind the newest in its stream. See [Loki](https://docs.jmrp.io/ghchronicle/sinks/loki/) | | OpenTelemetry | the backend decides | OTLP data points carry an explicit timestamp; whether it is honoured is not up to this tool | | Prometheus | no | Current values only, by the rules above | ## Three consequences worth knowing **A tag is a series, a field is a value.** Anything unbounded goes in a field. The Actions runner name looks like a good tag until you notice a hosted runner is named uniquely per run (`GitHub Actions 1000163135`), which would create a series for every job ever executed. It is a field. So is any value that moves after the row's own date: a tag is part of the row's identity, so a tag that changed would open a second row at the same instant and leave the stale one beside it for ever. A comment is accepted as the answer days after it was written, so since 2.6.1 that is the `answers` field of `gh_discussion_comment` rather than an `is_answer` tag. [The measurements page](https://docs.jmrp.io/ghchronicle/collectors/measurements/#how-to-read-the-tables) lists the others, and what a store written before 2.6.1 holds. **Weekly rows are anchored to the week, not to today.** `gh_commits_week` is stamped at the Sunday that starts each week. A sweep on Tuesday and one on Friday have to land on the same row, or every re-read writes a second copy of the year. `gh_star_day` is anchored the same way, each day the `week` GitHub returns plus its index. The day is GitHub's own calendar day in America/Los_Angeles, measured rather than documented, and the row is stamped at 00:00 UTC of that date, so a star given on a European morning can sit a day before the instant `gh_star` gives it. **Prometheus reserves some tag names.** A tag called `job` or `instance` collides with the scrape labels, and the OTLP receiver overwrites it with the service name. Workflow jobs are therefore tagged `job_name`. --- # Backfill One deliberate walk to the end of every surface, how far back it goes, and the three things no backfill can reach. Source: https://docs.jmrp.io/ghchronicle/how/backfill/ A backfill walks every GitHub surface back as far as the API answers, once, so the history starts before the first sweep. ```sh ghchronicle -config config.yaml -backfill ``` A sweep is an increment. A backfill is a walk. They are opposite intentions and the tool treats them as such. ## The difference in one table | | Sweep | Backfill | | --------------------------- | --------------------------------- | -------------------------------------------------- | | Pages per collector | the collector's own small default | until the API runs out, or the bound is reached | | When the reserve is reached | skip the family and warn | wait for the window to reset, then carry on | | Which families run | those whose interval has elapsed, the slow ones [taking turns](https://docs.jmrp.io/ghchronicle/configuration/cadences/#the-slow-families-take-turns) in the service | every enabled family, whatever the state file says | | When it is stopped | nothing to keep; the next tick sweeps | a checkpoint keeps its place, and the same command carries on | | How often | on a schedule, forever | deliberately, usually once | A normal sweep must never block. It protects the reserve so whatever else uses the same token keeps working, and it skips a family rather than sleeping. A backfill is run on purpose and the only thing that matters is that it finishes, so it parks until the budget is whole again. And a backfill that is stopped half way keeps its place, so the same command started again carries on from there rather than from the first repository. ## Stopping one, and picking it up again A walk of a real account is hours long, and a reason to stop it turns up: the author's own had been going six and a half hours when the binary underneath it had to be replaced. Stopping it used to cost the whole walk. The place is kept in a file beside the state file, named after it with `-progress.json` instead of `.json`. Nothing configures it. It is written after every repository rather than after every family, and that is measured rather than preferred: on an account of fifty nine repositories the account wide families take four minutes between them, and then `actions` takes two hours and three quarters and `commits` longer still, so a checkpoint at the end of each family would keep the four minutes and throw away the rest. What it records is what has been delivered. A repository goes into it only once its whole walk has reached every configured sink, which is what makes skipping it on the way back safe: there is no tail of its own pagination left anywhere. A repository whose collector failed, or whose rows a store refused, is not recorded, and the walk comes back to it. Coming back to it costs requests and nothing else, because a point carries the date the thing happened and re-collection rewrites the same rows. ```text level=INFO msg="backfill stopped, and what it had written is kept" file=/var/lib/ghchronicle/backfill-progress.json running_for=6h40m7s families_complete=9 family=actions repositories_written=23 resume="run the same command again" ``` The file itself is meant to be read, and says the same thing at more length: the families that finished and when, the family that was in flight, and which of its repositories are done. It is removed when the walk has covered every family it was asked for, so a checkpoint that exists is a walk with work left in it. Reaching the end without an error is not the same thing. A family truncated by a secondary rate limit is handed back as a pass rather than as a failure, and a family that failed on every repository is deliberately left unmarked; a walk can finish tidily with either of those behind it. The checkpoint stays, and the line at the end names the families still to do. An ordinary sweep keeps none of this. It writes no checkpoint and reads none: the state file it keeps is about cadences, and a half walked backfill has no business in it. ### The two things it refuses **A checkpoint it cannot read.** A machine that loses power can leave an empty or half written file behind. The run stops with a sentence naming it rather than starting the walk over in silence: what the file lists may be hours of somebody's history, and starting over is the answer that looks like success. **A checkpoint from a different walk.** The file records what the walk was asked for: the API it reads, the targets, the families that were enabled, the date bound exactly as it was spelled, and the stores it writes to. If one of those changed between the stop and the resume, the list of repositories not to walk again means something else, and the run stops and names the setting that changed. The stores are there because a record in this file means the rows reached every sink, and every sink means the ones that process had. Add a store while the walk is stopped and the resume skips everything the first half recorded, so the new store ends up holding the tail of the walk and nothing before it. Point an existing one somewhere else and the same hole opens in a different place. Only the settings you can read are compared: a store's password is never written into this file. The bound is the one that would go wrong quietly. A walk bounded at a year records its repositories as written; an unbounded resume would skip every one of them and leave a store that looks complete and stops a year back, with nothing in the rows to say so. Both refusals leave the file where it is. Put the setting back to carry on, or delete the file to walk again from the first repository. Replacing the binary is not one of them. That is the thing that stopped the walk this was written for, so a checkpoint written by another build is resumed, with a line saying which build wrote what it names. ### Seeing how far it has got ```sh ghchronicle -config config.yaml -backfill-status ``` It reads the checkpoint and prints it: when the walk began, how long ago it last recorded anything, how many families are complete out of how many, the family it was inside and how far into it, and the command that carries it on, with the `-backfill-since` and the `-families` it was started with. A migration's refill in progress is printed after it, with what it writes where. It asks GitHub nothing and writes nothing, so it is safe to run while a walk is going, and it needs no token: the credential is there because a sweep asks GitHub, and this asks nobody. When there is no walk in progress it says so and names the file it looked for. ### Going back on its own A pass can end with families left and no error at all. The usual reason is a few minutes of bad weather at the other end: a `502` on two repositories out of sixty leaves their family unmarked, and the walk ends tidily with one family short. ```sh ghchronicle -config config.yaml -backfill -backfill-retry 1h ``` waits an hour and goes back for whatever is left, and keeps doing that until there is nothing left. It costs almost nothing, because a resume walks only what the checkpoint does not already hold: two repositories out of sixty, not sixty. It stops on its own when **a pass records nothing new**. That is the whole rule, and it is deliberately not a list of which errors are worth retrying: a repository that was deleted fails the same way every hour, and a list of retryable statuses is wrong the moment GitHub answers something it did not answer before. A pass that gains nothing is a pass whose obstacle waiting does not clear. There is also a cap of ten passes, as a backstop rather than a knob. Without the flag nothing waits and nothing goes back, which is what it did before this existed. ## The cooldown When a bucket is at or below its reserve, the backfill waits. The wait is not guessed: every GitHub response says exactly when its window resets, so the collector sleeps until that instant plus a second of slack, and logs what it is doing. ```text level=INFO msg="rate limit reserve reached, waiting for the window to reset" family=commits wait=23m11s ``` Two guards on that wait. It is capped at one hour, so a clock skew or a stale header cannot turn into an unbounded sleep, and it has a floor of one second, so it cannot become a busy loop. A cancelled context ends the wait immediately, which is what makes `Ctrl+C` work during an overnight backfill. ## How far back `-backfill-since` on the command line, or `backfill.since` in the configuration file. Four spellings, because people reach for different ones: | Value | Means | | ------------------------- | ------------------------ | | `2024-01-01` | that date | | `90d` | ninety days ago | | `2y` | two years ago | | `720h` | a Go duration before now | | empty, `all`, `unlimited` | no bound at all | No bound means the walk stops only where the API does, however many hours or days that takes, pausing at every rate limit reset along the way. ```sh ghchronicle -config config.yaml -backfill -backfill-since 2y ``` ## Some families only ```sh ghchronicle -config config.yaml -backfill -families discussions,outbound ``` walks the families named and no other, into every configured store. It is what a store that lost one family's history wants: dropped by hand, or sent through a Telegraf whose store was cleared, without walking every other family it still holds. The names are the families `-groups` prints. One that is not a family, and the flag without `-backfill`, are refused with 2, as a command line that does not parse; a family the configuration switches off is refused with 1 before anything is asked, of GitHub or of the stores, the start's migration check included, since the walk would read none of it and end complete. Its checkpoint is the backfill's, and records the families it was asked for, so a backfill of other families, or of every family, refuses it and names the family that differs, and `-backfill-status` prints the resume line with the same `-families` in it. ### The refill of a migration A migration that cleared a store reads its history back with a backfill of its own, the refill: the families that write what was cleared, writing that alone into the stores it was cleared from; see [reading the history back](https://docs.jmrp.io/ghchronicle/install/upgrading/#reading-the-history-back). It keeps its checkpoint beside the backfill's, named with `-refill.json`, so a backfill in progress and a refill neither refuse nor overwrite each other's, and `-backfill-status` prints both. ## Run it once, first A new install should run a backfill before, or right after, starting the service. A sweep's first pass is a wider increment, not a history: a month of workflow runs, the star history in full, and the newest page of everything else. The points are dated, so every store is fine with that; a dashboard at ninety days or two years is not, because the history it draws begins on the day the collector was installed. Measured after a day of sweeps and no backfill, over repositories holding hundreds of pull requests each: pull requests, about a fifth of what the repositories report, because a sweep reads one page of fifty per repository however many it holds; issues, about half; commits, well under a tenth and none older than thirty days; jobs and steps for a tenth of the workflow runs, so the queue wait, the slowest jobs and the steps that fail were computed over that tenth. At two years, _Pull requests merged_ read a fifth of what _Pull requests merged, ever_ said. Stars and forks were complete, because the first sweep walks those to the end anyway, and so were releases, deployments and the alerts, because none of those repositories had more than the hundred of each that a sweep reads. ## What it reaches that a sweep does not - The whole commit history, rather than the last page. This is the one family where a backfill is qualitatively different rather than merely wider: without it the lines-changed series begins on the day you installed the collector. - Every workflow run GitHub still holds, each expanded into its jobs, and into its steps while GitHub still serves them. Measured on 24 September 2026, GitHub listed every job of a run 278 days old, but no steps for any run created before 12 April, about five and a half months back. Such a job, if it finished as success, failure or timed out, is written with no `steps` field rather than a 0, since it ran at least one. - The archived repositories, in full, whatever `include_archived` says. Their history is the account's history and it never moves again, which is exactly why a sweep leaves them out and why one walk of them is enough. Forks stay as configured. What does still move on an archived repository, its stars, forks and watchers, a sweep reads without a backfill: the listing it already pays for says which repositories are archived, and every `totals` sweep asks about all of them, one query per twenty five, for two rows each. One is `gh_repo_archived`, dated when the repository was archived. The other is its `gh_repo_total`, stamped at the sweep like a collected repository's, and that is the row the account's star and fork totals are read from. - Every pull request and issue, in pages of fifty, where a sweep reads what changed since the sweep before, and once a day every open item and what moved since the day before. - Every pull request and issue the account opened in other people's repositories that has since been merged or closed, in pages of a hundred ordered by when each one last moved, where a sweep reads each of the three closed states back to a cadence before the sweep before. That is enough for a sweep because a merge or a close moves the item to the top, however long ago it was opened, and it is one page unless more than a hundred items moved in that time; a sweep that read one page and no more lost the item that a hundred later updates, a bot locking old threads or a relabel, had pushed onto the second. The ones still open are bounded by neither a page nor a date: every sweep reads them all, since each one gets a row for every day it stays open. In 2.5.1 and earlier each of the five searches read its newest hundred and stopped, on a sweep and on a backfill alike. GitHub serves a thousand results of any search and no more, so an account past a thousand in one of those states keeps the thousand that moved most recently. That is said in the log, once per count, rather than left to show as two panels that disagree: ```text level=WARN msg="outbound search read fewer items than it counts, GitHub serves a thousand at most" kind=pull_request state=merged count=2860 read=1000 ``` - Every page of artifacts, of repository activity and of code scanning analyses, where a sweep reads five, two and one. - Every release, deployment and discussion, and every Dependabot and code scanning alert, where a sweep reads the newest page of each list. - Every issue and discussion comment the account left anywhere, and every answer of its own that was accepted, where a sweep reads the newest hundred comments of each kind and the newest five hundred accepted answers. - Every star the account gave, where a sweep reads the newest five hundred. - The co-authored pull requests behind Pair Extraordinaire's progress, walked over the account's whole life, where a pass adds the days since the last one and walks the whole history once a week. - Every page of every stargazer list, where a sweep reads the newest hundred stars of a repository whose list it has walked whole once. Up to 2.5.0 a first walk that failed was recorded as done, and a backfill read a recorded list by its first and last pages, so the stars in between were never read; a backfill now reaches them. - The whole inbox, read threads included, where a sweep reads the unread threads that moved and, once a day, twenty pages of fifty with the read ones. - Every month of billing GitHub still answers, until three empty months in a row, where a sweep reads two. - The newest hundred webhook deliveries per hook, where a sweep reads thirty. - At most 500 failed job logs per repository, none older than ninety days, once `joblogs` has a cadence; a sweep reads at most ten. Two things a sweep remembers, a backfill leaves alone. A 403 or 404 that a sweep [remembers for a day](https://docs.jmrp.io/ghchronicle/api/#a-refusal-is-remembered-too) is asked again, since a backfill consults no such memory. And it reads the [cache file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it) beside the state file but writes nothing to it: the pages it walks are ones no sweep asks for, and kept there they would push out the sweeps' own, so a service started after it would start colder than it stopped. ## What it does not switch on A backfill runs every enabled family whatever the state file says. It enables none of them. The three families that ship with a cadence of `0`, `deps`, `history` and `joblogs`, stay off unless they have been given one by name under `every.families`, and `-backfill` does not change that. `history` is the one that surprises people, because walking every past year of the contribution calendar is exactly what somebody asking for "the whole history" has in mind. Give it a cadence first: ```yaml every: families: history: 24h ``` See [cadences](https://docs.jmrp.io/ghchronicle/configuration/cadences/) for why `default` and `groups` cannot switch these three on either. ## Two endpoints that needed their own handling Both were found by running it, not by reading documentation. **Dependabot refuses page numbers.** It answers an error to `page=` outright and pages by cursor instead, so the alert walk is written against cursors. **The GraphQL gateway gives up on a hundred pull requests, and on fifty commits.** Asking for a hundred pull requests with their reviews in one query answers an **HTML 502** after about ten seconds, and so does asking for fifty commits of a busy repository with the checks each one carries: twelve pages of the author's own commit walks met it, in six repositories. Both walks halve their page size and retry on the same cursor, silently: neither collector carries a logger, so a backfill of a busy repository shows this only as a slower family, never as a line. The page is halved while it is larger than ten, so fifty is asked again at twenty-five, twelve and six. A page the gateway still gives up on at that smallest size, and a timeout in any walk with no smaller page to ask, is a failure: the rows read before it are written, the `collector failed` line says the query is too large for one request, and the repository is not recorded in the checkpoint, so a resume walks it again. Up to 2.6.3 the commit walk took the timeout for the end of the history and recorded the repository as walked. Only that answer is halved for: a `503`, or a `502` that comes back sooner than ten seconds, is not a query the gateway ran out of time on, and [the client asks it once more](https://docs.jmrp.io/ghchronicle/api/#an-answer-github-could-not-finish-is-asked-once-more) at the same size. > **Three things cannot be backfilled at any price** > > No amount of waiting changes these, and they are the reason the project exists > at all. > > - **The event feed keeps three hundred events**, and none older than thirty > days. Past that ceiling GitHub answers 422 "pagination is limited for this > resource", which the collector reads as the end of the data. > - **Traffic is fourteen days.** Anything older was never stored by GitHub. > - **Job logs are deleted after the repository's retention period**, ninety > days by default, and answer 410 afterwards. A backfill of `joblogs` asks > only for the last ninety days, whatever the setting. The runs they belong > to were kept for four hundred days or more; from 1 October 2026 GitHub > applies the same retention setting to the runs themselves: ninety days by > default, at most ninety on a public repository and up to four hundred on a > private one, so a backfill after that date reaches only as far back as > that setting. ## Running one safely It is idempotent. Points are keyed by measurement, tags and timestamp, so a backfill run twice rewrites the same rows rather than doubling them, in every store that keeps the history. What it costs is API quota and time. Two things worth doing first: run `-list` to confirm the repository set, and check that the store you are writing to is the one that keeps dates. Backfilling into Prometheus collects a great deal of history and then reduces all of it to a single current value. --- # What is collected The thirty-four families of GitHub metrics ghchronicle collects, what each one asks the API for, and the reason each exists. Source: https://docs.jmrp.io/ghchronicle/collectors/ Thirty-four families, ninety-five measurements. This page is what each family is _for_; the [measurements reference](https://docs.jmrp.io/ghchronicle/collectors/measurements/) is every tag and field. The families are also grouped, and `groups:` in the configuration switches a whole area on or off. The binary prints the grouping it actually uses, which is the one to trust: ```sh ghchronicle -groups ``` ```text account the account itself: its lifetime numbers, its profile, its keys, its spending and what it does in other people's repositories account, achievements, billing, history, keys, outbound, profile, totals audience who is looking at the projects, who starred them and who copied them forks, stars, traffic ci continuous integration and deployment: runs, jobs, steps, artifacts, caches and deployments actions, artifacts, deployments, joblogs ... ``` ## Audience **`traffic`** collects the only data GitHub genuinely throws away. Views and clones live for exactly fourteen days and then cease to exist anywhere. The whole window is re-read and rewritten on every sweep, each day stamped with its own date, so a collector that was off for a week loses nothing as long as it comes back inside the window. Referrers and popular paths are different: the API returns a top-ten snapshot with no dates at all, so they are stamped at the start of the UTC day and read as "who was sending traffic when we asked". **`stars`** reconstructs the star curve from its beginning, from two endpoints. GitHub's daily star history gives the stars a repository gained on each day, back to the week it was created, to anyone who can see the repository, so it is read for every repository the sweep collects and written as `gh_star_day`: whole on the first sweep, and after that one request for the newest thirty weeks, a free 304 unless one of their days gained or lost a star or a new week began. The daily star bars count it in every store but Prometheus, and in InfluxDB and PostgreSQL so does the star curve, so the entire history is there after the first run: a chart that goes back years, not one that starts the day the collector was installed. GitHub Enterprise Server does not serve the history, so there those panels stay empty. The stargazer list says who. Asked with the star media type it returns a `starred_at` per user, written as `gh_star`, walked in full once; after that only the newest hundred are read, since new stars land at the end, and they are read for every repository at once in one GraphQL query per ten. That is the difference between one point and 280 calls for a repository with 28,000 stars. GitHub has served the list only to a repository's admins and collaborators since July 2026, which the account always is on its own repositories and on those of an organisation it administers; any other repository has its stars counted per day and nobody named (see [what GitHub will not give](https://docs.jmrp.io/ghchronicle/api/limits/)). **`forks`** collects who forked and when. The repository snapshot carries a fork count, which says how many but never when or by whom. The list is walked through REST on the first sweep of a fresh install and in a backfill; after that the newest hundred of every repository ride in the same kind of batch the stars do. A fork row is not static the way a star is: it carries the fork's own stars and how long since it was pushed, so a repository the batch reports holding more than a hundred forks is walked through REST as well, which refreshes those on up to five hundred forks as it always did. ## Repositories **`repo`** collects what a repository is right now, plus the things that accumulate on GitHub's side: languages by bytes, topics, community health, and release downloads **per asset**, which is what tells a Linux build from a macOS one. Bytes per language matter because the dominant-language label cannot show a repository shifting from one language to another over time. **`settings`** collects the configuration that changes, and how well the parts of it that talk to the outside world are working. One thing here is a real time series rather than a snapshot: webhook deliveries carry a status code and a latency. **`rulesets`** collects the changelog of every ruleset, one row per saved version with an actor and a date, which is the only record GitHub keeps of the moment a protection was turned off. The ruleset itself, what it enforces and who may walk past it, is a daily snapshot in `repo`; this is the history that snapshot's `days_since_change` only summarizes. **`branches`** collects the live branch list, one row per branch carrying the age of its tip. Nothing else answers "which branches were abandoned": GitHub's own list sorts by name and forgets, so a branch whose last commit is four months old looks exactly like one pushed this morning. It deliberately does not ask which pull requests point at a branch, which is what makes that query expensive; the join belongs in the panel. **`inventory`** collects the three per-repository policy surfaces that change on a scale of months: what the `GITHUB_TOKEN` of a workflow is allowed to do, how old every stored secret is, and whether code scanning is switched on by GitHub rather than by a workflow of its own. Four core requests per repository a day. All three are settings rather than events, so they are stamped at the start of the UTC day and a change reads as the day the value moved. **`policyfiles`** records which governance files a repository carries, `SECURITY.md`, `CODEOWNERS`, `dependabot.yml` and `FUNDING.yml`, and when each of them last changed. Whether most of them exist today is already in `gh_repo_policy`; when they changed is nowhere else, and `.github/dependabot.yml` is in no other measurement at all, which is what makes "is this repository receiving dependency updates by either route" a question the data can answer. > **Webhooks fail silently** > > Measured, one hook had been answering 403 for seventy-eight of its last > hundred deliveries and nothing anywhere said so. Only the host of a webhook > URL is stored; the path usually carries a secret. ## Development **`issues`** collects pull requests and issues one by one, not as counts. A count of open pull requests says nothing about how the work actually goes; the interesting numbers are durations. How long until someone reviewed it, how long until it merged, how big the diff was, how many review rounds it took. One GraphQL query per repository covers both: a sweep walks what was updated in the last two cadences, ten at a time, and the first sweep of each UTC day reaches back to the day's read before it and also reads every open item, whatever it was last updated, which is the only read that rewrites an open pull request nobody touched. **`commits`** collects the commit history with its size and its signature. This is what replaces `stats/code_frequency`, which answers 202 with an empty body forever on a personal account. GraphQL gives lines added and removed per commit, attributed to an author and dated to the commit rather than to a week, and the signature comes with it for the same query. **`issueevents`** collects the transitions rather than the state. `issues` says what a pull request ended up as; this says when it was labelled, closed, reopened, renamed or had a review requested. A reopening exists in no other measurement. The repository-level list, `/issues/events`, is the reference for what an event is and how it is named, and it embeds the whole issue in every event: about a megabyte per page of a hundred, of which the collector keeps three per cent. So a sweep asks GraphQL for the timeline of the issues and pull requests updated in its window, ten items a page because the gateway was measured to drop timelines silently at twenty, and a backfill walks the per-issue endpoint, `/issues/{n}/events`, which is the same rows without the issue. Measured over a week of two repositories, the timeline agrees with the list on every event of every type it can name, field for field; a pull request in a stack is read through its own list because `added_to_stack` has no timeline type, and the one thing the list sees that this does not is a commit referencing an issue nobody has touched, three events in 2,217. The three of them ask only where something moved. One GraphQL query per twenty-five repositories first asks each when the head of its default branch was committed and when its newest issue and its newest pull request were updated, and a family asks nothing of a repository where nothing moved since its own window: on the account this was measured on, about half of the `issues` and `issueevents` queries and 96 per cent of the `commits` ones. The day's read of every open item of `issues` and a backfill are read everywhere. [Cost of a sweep](https://docs.jmrp.io/ghchronicle/api/cost/#asking-first-what-moved) has the count. **`deps`** collects the dependency graph, off by default. Two shapes of the same subject: the SBOM as a photograph, which is a licence histogram, and the difference between two commits, which says what entered and left and what advisory it carried. Only aggregates are stored, because one dependency bump is three hundred and seventy changes and six rows say the same thing. **`discussions`** collects the forum half of a repository. Discussions are invisible to every issue and pull request endpoint, and an answered question is a support cost that never appears in the issue numbers. A repository whose forum is switched off is never asked: the listing that discovered it already said so, and the query costs the same whether or not there is anything to page. **`planning`** collects labels and milestones. A milestone is the only place GitHub records an intention with a due date, and its completion percentage is computed server side. **`activity`** collects a repository's own activity log. It is the only place a force push is recorded: the public event feed does not distinguish one, and nothing else says that a branch was created or deleted or that a merge was a squash rather than a rebase. It is as perishable as traffic, a hundred entries covered twenty-six hours on the busiest repository measured. ## Continuous integration **`actions`** collects workflow runs as dated facts. A run belongs at the instant it completed, not at the instant we noticed, which is what makes "how long did CI take last Tuesday" answerable. Run-level timing hides where the time went: a run that takes twenty minutes because one job waited eighteen for a runner looks exactly like one that spent eighteen executing. Only the job level separates them, and only the job level names the runner and the steps, so the jobs of each run are expanded at one extra request per run. Once: the jobs of a finished attempt never change, so a run whose jobs this process already wrote is not listed again, and a re-run is a new attempt that is. An ordinary sweep reads the run list in pages of thirty and pages on while they are full of runs newer than its window; the first sweep after start and a backfill read pages of a hundred. **`artifacts`** collects what the workflows left behind, with sizes and expiry. **`joblogs`** collects the text a failed job printed. It is the one thing here that is a log rather than a measurement, and it answers the question a chart never can: not "the build failed" but why. Only failures, and only their last forty lines. Off by default. **`deployments`** collects the newest deployments of every repository, which is the surface a delivery dashboard reads. One GraphQL point per five repositories: five rather than ten because the gateway gives up on a query it cannot finish in about ten seconds, and this one asks for connections rather than plain numbers. ## Security **`security`** counts open alerts by severity and state, and records **explicitly which features are switched on**. That last part is why no data and no alerts are distinguishable: without it, a repository with Dependabot off looks exactly like one with nothing to fix. **`analyses`** collects the code scanning analyses themselves, not just the alerts. An alert says what is wrong now. An analysis says the scan ran, when, on which commit, with which version of the tool, and how many results it found, which is what answers "did the scan actually run on that release". GitHub prunes them, so they have to be captured while they are there. ## Account **`account`** is one GraphQL query and the cheapest thing in the project. It returns the full 366-day contribution calendar, every contribution total, the per-repository commit breakdown and the sponsors block for **one point of a five thousand point budget**. The same data over REST would be dozens of calls and would not include the calendar at all. **`profile`** collects packages, gists, social accounts and the follower graph. Packages come from REST deliberately: GraphQL reports zero packages for an account while REST lists them. **`outbound`** is the other half of everything else here. Every other family measures what the account owns; this measures what it reads and what it contributes to: the stars it gave, the pull requests it opened in other people's repositories, and those repositories themselves, how many stars each has and whether it is private. All of it is GraphQL, one point a page: the starred list, five issue searches, the two comment walks and the accepted answers the account wrote, which are read apart from the comments so that an answer accepted weeks after it was written is not missed. A sweep reads up to five pages of the starred list, every page of the two searches for what is still open, and the other three, which are ordered by what moved last, back to a cadence before the sweep before: their first page, unless more than a hundred items moved in that time. A backfill reads the starred list and those three to the end as well. **`totals`** asks GitHub for the numbers that are true since the beginning: pull requests merged ever, commits ever, issues opened ever, and the whole life of each repository. It is the one family that exists because of how a store reads rather than because of what GitHub offers. Every other measurement here is a row per fact, which is the right shape for "how many in July" and the wrong one for "how many ever": answering that from rows means scanning the whole table, and InfluxDB 3 Core refuses a query that would open more than its file limit, forty thousand where this was measured. Search reports a total for any query and GraphQL reports one for any connection, so one query of ten aliased searches, one REST search for the commit count and one batched query give a number that is one row and is right on the first sweep of a fresh install. The archived repositories the default filter sets aside are in it too, from the query that dates their archive, so their stars and forks stay in the account's totals. **`ratelimit`** is the only measurement the collector takes of itself: what is left in each of GitHub's fifteen independent budgets and when each resets. `GET /rate_limit` costs nothing at all, and without it a family skipped for want of budget looks exactly like a family with nothing to report. **`keys`** collects the account's own SSH and GPG keys: which have never been used, and when the key that signs every commit expires. **`stats`** collects commits per week and the hour-of-week punch card, the two `stats` endpoints that actually answer for a personal account. **`history`** walks every past year's contribution calendar, one GraphQL point per year, back to the day the account was created. Off by default because it only needs to happen once. **`achievements`** collects the badges on the public profile page, and it is the one family that does not come from the API at all: GitHub lists achievements in neither REST nor GraphQL, so the page is read every hour as an anonymous visitor, without the token and charged to no budget. Beside each badge it writes how far the account is from that badge's next tier, which does come from the API. The parser is strict on purpose: when GitHub redesigns the page the family writes nothing and says so once, rather than writing wrong numbers that would look exactly like right ones. ## Activity and cost **`events`** collects the account's activity feed, the most perishable surface GitHub has. It keeps the last three hundred events, none older than thirty days, and nothing else records that a repository was starred, forked, watched or pushed to at a given minute. **`notifs`** collects the inbox. Like the event feed it is a window, not a history: GitHub keeps inbox notifications for three months unless they are saved, and `per_page` is silently capped at 50 whatever is asked for. **`billing`** collects what the account actually spent, day by day, per product, SKU and repository, which is the only place that says which repository burned the minutes. Gross, discount and net are all kept rather than one being derived from the others, because the net is not always zero: on the account this was developed against it carries the monthly credit. > **A family that fails on one repository does not fail the sweep** > > Of fifty repositories, most have Dependabot switched off, and each of them > answers 403. That is recorded as "not enabled" and the sweep moves on. Only a > family where _every_ repository failed is left unmarked, so it is retried > rather than treated as done. --- # Measurements Every measurement ghchronicle writes from the GitHub API, its tags, its fields, and how each one is dated. Source: https://docs.jmrp.io/ghchronicle/collectors/measurements/ Ninety-five measurements. Each row says how a point is dated, because that is the thing that decides which questions it can answer. ## How to read the tables | Dating | Means | | --------- | ----------------------------------------------------------------------------------------------------------------------------------- | | **dated** | The point carries the moment the thing happened, so the history is real and re-collecting rewrites the same rows | | **daily** | A snapshot with no date of its own, stamped at the start of the UTC day so a day's sweeps converge on one row rather than piling up | | **now** | A current state, which only makes sense as "what is true at this moment" | Every measurement carries `owner`, `repo` and `full_name` as tags unless it is account-wide, in which case it carries `user`. The three travel together or not at all, and `repo` is always the short name, on somebody else's repository as much as on your own: `owner=golang`, `repo=go`, `full_name=golang/go`. That is what lets a filter written for one measurement answer in every other, and it is what the dashboards' repository variable is built from. The short name is not an identity, since two owners can use the same one, so a query that needs identity groups by `full_name` and one that needs a person groups by `owner`. That has not always been true of all of them. Twelve carried no `owner` and put the full name inside `repo`, and `gh_billing_usage` had a short `repo`, no `owner` and an `org`. The three shapes did not overlap at all, which is worse than overlapping badly: a filter built from one of them matched nothing whatsoever in the others, and a union of two counted every repository twice. They are one shape now. `org` is gone with them, and it loses nothing: `organizationName` is not a property of the billing endpoint this tool calls, so the tag could only ever hold `(none)`, and on the organisation form of that report it names the organisation whose report was asked for, which is the owner. `owner` carries it. **Upgrading from a release before this shape** is a breaking change to what is already stored, and it is larger than a seam. Read the four paragraphs below before upgrading a database you want to keep. **Your existing rows are not converted, and most of them are written again.** Rows already stored keep the old tags and nothing rewrites them, because in InfluxDB the tag set is part of a point's identity. What is easy to miss is that most of the twelve are dated at the item's own time and re-offered on every sweep, and the write ledger keys on the tag set. Change the tag set and every one of those points is a miss, so **the first sweep after the upgrade writes the whole retained history again, at the same timestamps, under the new tags, beside the old rows**. That is not a seam you can wait out. A sum over a range that covers the rewritten history **doubles immediately and stays doubled**, for as long as GitHub keeps serving those items. On the account this was developed against that was a full year of `gh_contribution_day_repo`, which a shipped panel sums, and several hundred rows each of `gh_issue_comment`, `gh_external_contribution` and `gh_discussion_comment`. Only `gh_event` and `gh_notification` behave like a seam, because they are windows GitHub forgets. **So there are two honest options, and waiting is not one of them.** Recreate the database, which is what the author did; or delete the old-shape rows yourself, which in InfluxDB 3 means dropping the thirteen tables, since a tag cannot be renamed in place and a delete predicate cannot name a tag the new rows do not carry. There is no migration and there will not be one: renaming a tag means rewriting every affected row under a new identity, which is a restore rather than an update. **The PostgreSQL sink stops loading until its tables are changed.** It emits `CREATE TABLE IF NOT EXISTS`, which does nothing against a table an earlier release created, so the `INSERT` that follows names `owner` and `full_name` columns that do not exist and an `ON CONFLICT` key that does not exist either, and psql rejects it for all thirteen measurements. Drop those tables and let the next file recreate them, or add the two columns and rebuild each primary key: ```sql ALTER TABLE gh_event ADD COLUMN IF NOT EXISTS "owner" TEXT NOT NULL DEFAULT '', ADD COLUMN IF NOT EXISTS "full_name" TEXT NOT NULL DEFAULT ''; ALTER TABLE gh_event DROP CONSTRAINT IF EXISTS gh_event_pkey; ALTER TABLE gh_event ADD PRIMARY KEY ("time", "action", "full_name", "owner", "ref_type", "repo", "type"); ``` **The rebuilt key must be exactly the key this tool would declare** for that measurement: `time` followed by the tag columns of the table above, in name order. Do not assemble it from the columns your table happens to have. An upgraded `gh_billing_usage` still carries an `org` column from the old shape that nothing writes any more and that is not part of the key; include it and the `ON CONFLICT` the sink emits matches no unique constraint, so the load fails again with a message that points nowhere. The check on your own work is the `CREATE TABLE` the sink writes for that measurement into a fresh file: its `PRIMARY KEY` list is the answer. Adding the columns keeps the old rows but does not convert them: they hold the empty string where the new rows hold an owner, so they double-count exactly as they do in InfluxDB. Dropping is the clean option in both stores. **Queries you wrote yourself need updating, and only some of them fail loudly.** `gh_billing_usage.org` no longer exists, and in InfluxDB 3 naming a column no row has written fails at planning, so that one tells you. A filter such as `gh_event WHERE repo = 'owner/name'` keeps parsing and quietly returns nothing; it becomes `full_name = 'owner/name'`. Most measurements also carry a `url` field: the page on GitHub for the thing the row is about, so a dashboard row that names an item can also open it. A `url` is absolute or absent, since the dashboards link to the value itself. Every measurement that has one is linked item by item from at least one table, with two kinds of exception. The rows of some are only ever counted or added up, into a curve, a bar, a stat or a table line per repository, reviewer, job or check, where a per-item url has no row to sit on: `gh_pull_request_review`, `gh_workflow_job`, `gh_event`, `gh_issue_event`, `gh_artifact`, `gh_contribution_day`, `gh_contribution_day_repo`, `gh_commit_check`, `gh_issue_comment`, `gh_billing_usage`, `gh_dependabot_alert`, `gh_code_scanning_alert`, and the account-wide `gh_account`, `gh_account_total`, `gh_contributions_total` and `gh_sponsors_listing`. And two carry a url no table links: `gh_release_published`, which no panel reads, since the releases table links each release from `gh_release`, and `gh_upstream_repo`, which _Work elsewhere_ joins for its Stars while each of its rows links the item rather than the repository. A tag GitHub leaves empty is written as `(none)`, one spelling on every measurement, and the same `(none)` goes into a few string fields that say nothing on most rows, a pull request's `decision` or an issue's `assigned_to`: InfluxDB 3 creates a column the first time a row carries it, and a query naming a column no row has written fails outright, so those are written on every row. The same convention writes `(ghost)` where a login belonged to an account that has since been deleted. The parentheses are the point of the spelling. GitHub answers `unknown` itself in a Dependabot alert's `relationship`, and `none` is a value in more than one of its enums, so a fallback spelled either way could not be told from an answer; `(none)` is never a value GitHub returns. One value does read `none` without them and means it: `gate` on `gh_commit` is the state of the commit's gate, and `none` is the state of a commit no gate ever ran on, beside `SUCCESS` and `FAILURE`. **A value that moves after the row's own date is a field, never a tag.** A tag is part of a row's identity, so a tag that changes after the fact opens a second series at the same instant and the stale row stays beside the new one for ever: measured after eleven hours of sweeps, one artifact in fifty had a row with `expired=false` and another with `expired=true` at the same timestamp, and a commit seen `PENDING` by one sweep and `FAILURE` by the next was counted twice. Every such tag is a field now, under a new name because InfluxDB 3 fixes a column as tag or field at first write: `gh_artifact.expired` is `live`, `gh_commit.checks` is `gate`, `state` on both alert items is `alert_state` and the code scanning `reason` is `resolution`, `gh_issue`'s `state_reason`, `assignee`, `milestone` and `parent` are `resolution`, `assigned_to`, `milestone_title` and `parent_issue`, `gh_pull_request.draft` and `review_decision` are `is_draft` and `decision`, `gh_discussion.answered` is `has_answer`, `gh_pull_request_review.state` is `review_state`, `gh_deployment.state` is `outcome` and `gh_notification.unread` is `is_unread`. `gh_discussion_comment.is_answer`, which 2.6.1 stopped writing, needed no new name: it is the `answers` field every row of that measurement already carried, 1 for the accepted answer and 0 for any other comment. `state` on a pull request, an issue and an external contribution stays a tag because its date moves with it: the open row is stamped at the start of the day and the closed row when it closed. The Prometheus exporter reads the demoted values back as labels; Graphite, which keeps no strings, cannot group by them and its panels say so. **A store written before 2.6.1 and since holds `gh_discussion_comment` in two shapes.** A comment an earlier release read before it was accepted and again after is two rows at one instant, one per value of `is_answer`. A comment an earlier release read gains one more row without the tag when a 2.6.1 sweep reads it again: a second for most, a third for one already held twice. The two dashboard tables over the measurement read one row per comment in every store, accepted when any of its rows says so, so an answer accepted before 2.6.1 and taken back since still reads accepted. Dropping the measurement and running a `-backfill` leaves one row per comment: the table in InfluxDB 3 and in PostgreSQL, the `github.discussion_comment` paths in Graphite, the `ghchronicle-gh_discussion_comment` index in Elasticsearch. The PostgreSQL sink that connects keeps writing into a table an earlier release made, conflicting on that table's own key, and its new rows hold the empty string for `is_answer`. The SQL file cannot know that key, so replayed into such a table its `gh_discussion_comment` statements are refused while the rest of the file loads, until the table is dropped. The Prometheus exporter keeps nothing across a restart and counts each comment once. `-migrate` says which of the configured stores hold the two shapes, under `2.6.1/gh_discussion_comment/is_answer`: see [Migrations](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations). ### From a row to a query Every row here is a table in the store, its tags are columns you filter and group by, and its fields are the numbers. Read against InfluxDB 3 in SQL mode, the three datings turn into three shapes of query. A **dated** measurement is history, so it is read over a range: ```sql SELECT time, "count" FROM gh_traffic WHERE kind = 'views' AND repo = 'telemetry' AND time > now() - INTERVAL '90 days' ``` A **daily** snapshot is one row per day, so the newest row is the answer and the difference between two days is the movement: ```sql SELECT time, downloads FROM gh_release_asset WHERE asset = 'ghchronicle_linux_amd64.tar.gz' ORDER BY time DESC LIMIT 30 ``` A **dated item** carries one row per thing that happened, which is what lets a question be asked of the items rather than of a count: ```sql SELECT date_trunc('week', time) AS week, count(*) AS merged, avg(seconds_to_merge) / 3600 AS hours FROM gh_pull_request WHERE state = 'MERGED' GROUP BY week ORDER BY week ``` The same three shapes work in the other history stores; the [dashboards](https://docs.jmrp.io/ghchronicle/dashboards/) carry one query set per store for every panel, which is the place to copy from. ### Every measurement, alphabetically Ninety-five, each link landing on the table it is in. [`gh_account`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_account_total`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_achievement`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_achievement_progress`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_actions_cache`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#continuous-integration) · [`gh_actions_cache_entry`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#continuous-integration) · [`gh_actions_policy`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#security) · [`gh_artifact`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#continuous-integration) · [`gh_artifact_total`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#continuous-integration) · [`gh_billing_usage`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#cost) · [`gh_branch`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_branch_protection`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_code_scanning_alert`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#security) · [`gh_code_scanning_alert_item`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#security) · [`gh_code_scanning_analysis`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#security) · [`gh_code_scanning_setup`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#security) · [`gh_collector_family`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_commit`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_commit_check`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_commit_punchcard`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_commits_week`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_contribution_day`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_contribution_day_repo`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_contribution_repo`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_contribution_year`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_contributions_total`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_dependabot_alert`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#security) · [`gh_dependabot_alert_item`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#security) · [`gh_dependabot_ecosystem`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_dependency`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_dependency_change`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_dependency_license`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_deploy_key`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_deployment`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_discussion`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_discussion_comment`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_environment`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_event`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#activity) · [`gh_external_contribution`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_fork`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#stars-and-forks) · [`gh_gist`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_issue`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_issue_comment`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_issue_event`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_job_log`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#job-logs) · [`gh_key`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_label`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_milestone`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_notification`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#activity) · [`gh_package`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_package_version`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_pinned_item`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_policy_file`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_profile_flag`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_pull_request`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_pull_request_review`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_rate_limit`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_release`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#repositories) · [`gh_release_asset`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#repositories) · [`gh_release_published`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#repositories) · [`gh_repo`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#repositories) · [`gh_repo_activity`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#continuous-integration) · [`gh_repo_archived`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#repositories) · [`gh_repo_community`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#repositories) · [`gh_repo_created`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_repo_language`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#repositories) · [`gh_repo_policy`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_repo_topic`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#repositories) · [`gh_repo_total`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_review_thread`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_ruleset`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_ruleset_rule`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_ruleset_version`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_secret`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#security) · [`gh_security_feature`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#security) · [`gh_security_setting`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#security) · [`gh_social_account`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_sponsors_listing`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_sponsors_tier`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_sponsorship`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_star`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#stars-and-forks) · [`gh_star_day`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#stars-and-forks) · [`gh_star_given`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#stars-and-forks) · [`gh_star_list`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#account) · [`gh_traffic`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#audience) · [`gh_traffic_path`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#audience) · [`gh_traffic_referrer`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#audience) · [`gh_upstream_repo`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#development) · [`gh_webhook`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_webhook_delivery`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) · [`gh_workflow`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#continuous-integration) · [`gh_workflow_job`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#continuous-integration) · [`gh_workflow_run`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#continuous-integration) · [`gh_workflow_run_total`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#continuous-integration) · [`gh_workflow_step`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#continuous-integration) ## Audience | Measurement | Dated | Tags | Fields | | --------------------- | ------------------------ | ---------------------- | ---------------------------------- | | `gh_traffic` | dated, one point per day | `kind` (views, clones) | `count`, `uniques`, `url` | | `gh_traffic_referrer` | daily | `referrer` | `count`, `uniques`, `url`, `referrer_url` | | `gh_traffic_path` | daily | `path` | `count`, `uniques`, `title`, `url` | GitHub serves fourteen days and the whole window is rewritten on every sweep, so a collector that was down for a day repairs itself on the next run. The referrers and paths are the top ten of that same window with no dates attached, which is why they are a snapshot rather than a series. ## Stars and forks | Measurement | Dated | Tags | Fields | | --------------- | -------------------------------- | -------------------------- | ------------------------------------------------------ | | `gh_star` | dated, when the star was given | `user` | `starred`, `url`, `user_url` | | `gh_star_day` | dated, one point per day: the Pacific day, at 00:00 UTC | | `stars` | | `gh_star_given` | dated | `user`, `language` | `stars`, `repo_stars`, `url` | | `gh_fork` | dated, when the fork was created | `by` | `forks`, `stars`, `seconds_to_push`, `advanced`, `url` | `gh_star_day` is the stars a repository gained on each day, read from GitHub's [star history](https://docs.github.com/en/rest/activity/starring#get-repository-star-history) for every repository a sweep collects, whatever the token may see of its stargazers. It names nobody, so it carries no tag of its own and no `url`: `gh_star` is the row per star, with who gave it and the second, and it exists only where the token may read the stargazer list, which since July 2026 means a repository's admins and collaborators. No panel reads both, so a repository with both is counted once, and the panels that count stars from rows read `gh_star_day`, so one whose list is hidden is counted all the same. A day is GitHub's own calendar day in America/Los_Angeles, and its row is stamped at 00:00 UTC of that date, as a `gh_contribution_day` row is stamped at 00:00 UTC of its own. Measured against the stargazer lists of nineteen repositories, 440 stars, the Pacific day matched every one, and the UTC day put 38 of one repository's 127 days wrong. So a star given on a European morning can sit a day before the date `gh_star` gives it. Each day is anchored to the week the API returns, never to the clock, no day after the sweep is written, and today may be written as 0 before it has begun in Pacific time, to be corrected by the next sweep. The count is the repository's current stargazers, each on the day they starred, so an unstar takes the star off the day it was given rather than the day it was taken back. A sweep reads the newest thirty weeks, one page, and writes every day of them, zeros included, so a day that drops from one to none is rewritten the next time. Older pages are read on a repository's first sight, once after upgrading to 2.5.0, and in a backfill; the state file records a whole read as `history_read`, and only after a walk that reached the end of the history, so a walk cut short, by an error or by a later page answering 403 or 404, is read whole again on the next sweep. Those pages write only the days that have stars: zero rows back to each repository's creation would multiply the files InfluxDB 3 opens, and a day older than thirty weeks that lost its only star therefore keeps it, the way `gh_star` keeps every star it ever saw. The sum usually sits at `gh_repo.stars` or a little below it, since that count also includes accounts GitHub no longer lists: one short on three of those nineteen repositories. A star given more than thirty weeks ago and taken back after the history was read can make it sit above: only a backfill that reaches back to its day lowers that day, and only while the day holds another star, so one that was its day's only star stays for good. Prometheus and an OTLP backend with `raw: false` never see it: an unstar lowers a past day, which no counter can do, and `github_repo_stars` already carries every repository's current count. GitHub Enterprise Server does not serve the star history: its REST API, checked against 3.21 and 3.22, has the stargazer list and no `stargazers/history`. There every repository answers it with a 404, which is read as nothing to collect, so `gh_star_day` stays empty, and so do the panels that count stars from it: Stars gained over time and Stars over time in InfluxDB and the SQL stores, Stars gained over time in Graphite and Elasticsearch, and Graphite's Recent stars. Recent stars still lists names from `gh_star` in the other stores, and Prometheus still counts from it. A history that answered 404 is not recorded as read, so a server that starts serving it has each repository's history read whole on the next sweep. `gh_star_given` is the outbound direction: what this account starred in other people's repositories. `advanced` on a fork separates a real derivative from a bookmark, which most forks are, and `seconds_to_push` says how long after the fork its last push came. It is negative when GitHub reports a push older than the fork itself. How long a fork has been idle is not stored, because the row is dated when the fork was created: it is that date subtracted from now, less `seconds_to_push`, and a query computes it. Stored, it was one day larger every day written on to a row dated years earlier, which meant "when the sweep ran" rather than anything about the fork, and every rewrite cost a file in the fork's own partition. ## Repositories | Measurement | Dated | Tags | Fields | | ------------------- | ----- | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | `gh_repo` | now | `language`, `visibility`, `license`, `archived`, `fork`, `default_branch` | `stars`, `forks`, `watchers`, `open_issues`, `size_kb`, `age_days`, `days_since_push`, `days_since_config_change`, `network`, `repo_id`, `is_template`, `has_pages`, `web_commit_signoff_required`, `allow_update_branch`, `pull_request_creation_policy`, `url` | | `gh_repo_language` | now | `language` | `bytes` | | `gh_repo_topic` | now | `topic` | `present`, `url` | | `gh_repo_community` | now | | `health_percentage`, `url`, `has_readme`, `has_license`, `has_contributing`, `has_code_of_conduct`, `has_issue_template`, `has_pull_request_template` | | `gh_repo_archived` | dated, when the repository was archived | | `archived`, `age_days_at_archive`, `url` | | `gh_release` | now | `tag`, `draft`, `prerelease` | `downloads`, `assets`, `age_days`, `url` | | `gh_release_asset` | daily | `tag`, `asset` | `downloads`, `size_bytes`, `digest`, `content_type`, `uploader`, `age_days`, `url` | | `gh_release_published` | dated, when published | `tag` | `published`, `prerelease`, `url` | `open_issues` is GitHub's field and GitHub counts pull requests in it. Use `gh_issue` to count issues. `gh_release_published` is when each release was published, to the second, and it is the one to group by for a calendar of releases or to read "the latest stable release" from, the newest row with `prerelease` false. `gh_release` stays stamped at the sweep, because its downloads move, and its `age_days` is floored to whole days counted back from the sweep: the date it reconstructs is a day late for any release published later in the day than the sweep ran. `published` is the integer 1 on every row, so counting releases is a sum. `prerelease` is a boolean field here, where it is a tag on `gh_release`: a pre-release is promoted by unticking the box on the published release, and as a tag a promotion that keeps the publication's date would write a second row at the same instant, which the sum would count twice. A release taken back to a draft and published again writes a second row if GitHub gives it a new `published_at`, so the exact count is the distinct `tag` values of each repository. A draft has no publication, so it writes no row here, and its `gh_release` row carries no `age_days`: GitHub sends a draft with `published_at` null, and the age measured from nothing held 106751 days on every draft row. The first sweep after an upgrade dates the page of releases it reads; `-backfill` dates the rest. The `url` on `gh_release_asset` is the asset's download address, not a page: following it fetches the binary. The assets are inventory, anchored to the start of the UTC day like the cache entries: stamped at the sweep, every asset was a fresh row every hour, which was 15 per cent of the whole database after eleven hours. One row per asset per day still answers "downloads per day", and the newest row is still the value. `gh_repo_archived` is the one row about a repository that carries a date rather than a state: dated at `archivedAt`, a clear-out is visible as the batch it was, and a live repository produces no row at all, so counting the rows is counting the archive. It does not need `include_archived`. The listing a sweep already pays for says which repositories are archived, and the `totals` family asks the date of all of them, beside their lifetime row, in one GraphQL query per twenty five repositories, on every `totals` sweep: a point per twenty five at that cadence, and the rows it rewrites are the same rows, which is what an exporter that keeps only what is rewritten needs; the listing cannot supply the date itself, since REST carries no `archived_at` and its `updated_at` was measured two seconds to eight minutes after the archive. An archived fork under the default fork rule is the one kind with no row. ## Development | Measurement | Dated | Tags | Fields | | -------------------------- | ----------------------------------- | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gh_pull_request` | dated when closed, daily while open | `number`, `state`, `author` | `is_draft`, `decision`, `title`, `labels`, `label_names`, `author_association`, `additions`, `deletions`, `churn`, `changed_files`, `commits`, `comments`, `total_comments`, `reviews`, `review_requests`, `review_threads`, `base_ref`, `head_ref`, `merged_by`, `merge_commit`, `mergeable`, `merge_state`, `stack`, `stack_size`, `stack_position`, `seconds_to_first_review`, `seconds_to_first_human_review`, `seconds_to_merge`, `seconds_open`, `url` | | `gh_pull_request_review` | dated, when submitted | `number`, `author`, `reviewer`, `bot`, `self` | `review_state`, `reviews`, `seconds_to_review`, `url` | | `gh_issue` | dated when closed, daily while open | `number`, `state`, `author` | `resolution`, `assigned_to`, `milestone_title`, `parent_issue`, `comments`, `reactions`, `labels`, `label_names`, `sub_issues_total`, `sub_issues_completed`, `pull_request`, `seconds_to_close`, `seconds_open`, `url` | | `gh_commit` | dated, when committed | `sha`, `author`, `branch`, `signature` | `gate`, `additions`, `deletions`, `churn`, `changed_files`, `commits`, `signed`, `oid`, `headline`, `url`, `pull_request`, `checks_total`, `checks_failed` | | `gh_commit_check` | dated, when the check finished | `sha`, `app`, `check`, `conclusion` | `checks`, `failed`, `url` | | `gh_issue_event` | dated, when it happened | `event`, `actor`, `kind`, `bot`, `label`, `milestone`, `requested_reviewer`, `review_requester`, `mentioned` | `events`, `number`, `title`, `url`, `commit_id`, `rename_from`, `rename_to` | | `gh_review_thread` | dated, when the thread's first comment was written | `thread`, `number`, `author`, `bot` | `path`, `comments`, `resolved`, `outdated`, `subject_type`, `resolved_by` | | `gh_discussion` | dated, when created | `category`, `answerable`, `author`, `number` | `has_answer`, `comments`, `replies`, `reactions`, `upvotes`, `closed`, `state_reason`, `seconds_to_answer`, `seconds_to_close`, `title`, `url` | | `gh_label` | daily | `label` | `issues`, `pull_requests`, `used`, `url` | | `gh_milestone` | daily | `milestone`, `state` | `progress`, `issues`, `pull_requests`, `days_to_due`, `seconds_to_close`, `url` | | `gh_external_contribution` | dated | `user`, `number`, `kind`, `state` | `contributions`, `merged`, `private`, `additions`, `deletions`, `changed_files`, `title`, `comments`, `seconds_to_merge`, `seconds_open`, `url` | | `gh_upstream_repo` | now | | `stars`, `forks`, `private`, `language`, `url` | `gh_commit` is what replaces `stats/code_frequency`, which returns 202 with an empty body forever on a personal account. `signature` is `unsigned` when there is no signature at all, which is a different fact from one that failed to verify. `gate` is the state of the whole gate on that commit, which is not the same claim as a workflow run having failed: a run says one job failed, the rollup says the commit came out red. It is a field because the verdict lands after the commit's own date. `gh_commit_check` holds only the checks that are not GitHub Actions, since everything Actions runs is already collected in far more detail. `seconds_to_first_review` counts any review, and on an account with review bots that is the bot: measured over 140 pull requests, its median was five seconds, because 132 were first reviewed by sourcery-ai or coderabbitai within a minute of opening. `seconds_to_first_human_review` is the wait for somebody else, over the first twenty reviews the query fetches: not a bot, and not the author. The author's reply in a review thread arrives as a review of state `COMMENTED` under their own name, and on this account it was the earliest non-bot review on every one of the 91 pull requests that had one, so a wait that counted it measured how fast the owner answers sourcery-ai. A pull request whose fetched reviews are all bots and the author's own replies carries no such field rather than a wrong one. A bot is a GitHub App (`__typename` Bot) or a login ending in `[bot]`; a deleted account is not one. `gh_pull_request_review.bot` draws the same line per review and `self` marks the author's own, so a reviewers table can leave both out or show them apart; `bot` is what `gh_review_thread.bot` already does for threads. `title`, `label_names` and `author_association` are fields because a title is unbounded and nine labels on one pull request are one row, not nine series. `labels` is the count and `label_names` the names joined by commas, absent when there are none, on pull requests and issues alike. `author_association` is `OWNER`, `MEMBER`, `COLLABORATOR`, `CONTRIBUTOR`, `FIRST_TIME_CONTRIBUTOR` or `NONE`: what separates an outside contribution from the owner's own work. `mergeable` and `merge_state` are written only while a pull request is open. A merged one keeps answering `CONFLICTING` long after it was merged, which is stale rather than false but reads as a repository full of conflicts. A sweep reads only what was updated in the last two cadences, so an open pull request nobody touches has its `seconds_open`, `mergeable` and `merge_state` rewritten once a day by the day's read of every open item rather than every hour, however long ago it last moved; anything that moves `updatedAt`, a review, a comment, a push, a close, is rewritten by the sweep that follows it. `stack`, `stack_size` and `stack_position` describe a stack of dependent pull requests and are absent on a pull request that is in none. `stack` is the stack's own number, not a member's, so the honest way to count deliveries is distinct `stack` values plus the rows carrying no stack fields at all. `review_requests` and `review_threads` are the two counts a reviewing flow is measured with, and `total_comments` counts every comment on the pull request rather than the ones in `comments`, which are the ones on the conversation. `sub_issues_total` and `sub_issues_completed` are how far an epic has got, from the checklist GitHub keeps on the parent. `parent_issue` is the other end of the same relation, on the child, and is `0` on an issue with no parent. `pull_request` is the pull request that closed the issue, `0` when none did. `gh_issue_event` is the transition rather than the state. `gh_issue` and `gh_pull_request` say what something ended up as; this says when it was labelled, closed, reopened, renamed or had a review requested. A reopening exists nowhere else. `mentioned` is the person a `mentioned` or `subscribed` event happened to, which GitHub files as the actor without saying who wrote the comment; here that person has a tag of their own and `actor` reads `(none)` on those two types, so the account named in "@coderabbitai" no longer shares a column with the app that reviews. An app is spelled the way REST spells it, with the `[bot]` suffix, on every measurement. `gh_discussion` counts comments and replies apart: comments answer the discussion, replies answer those, and GitHub's own number on the page is the two added together. `gh_external_contribution` is the work the account did in repositories it does not own, from five searches, one per `kind` and `state`: pull requests `merged`, `open` and `closed` unmerged, and issues `open` and `closed`. An item still open is rewritten at the start of every day it stays open, so a sweep reads both open states whole. A closed item is written once, stamped when it closed, so the three closed states are read by when each item last moved: a sweep reads each back to a cadence before the sweep before, which holds whatever merged or closed since then however long ago it was opened and however many other items moved in between, and is one page of a hundred unless more than a hundred did. A backfill reads on until the pages run out or reach `backfill.since`. GitHub serves a thousand results of any search and no more, so an account past a thousand in one state has the thousand that moved most recently, and the log says so at warning. `gh_account_total.pulls_merged_elsewhere` and `issues_elsewhere` are GitHub's own counts and are right whatever the cap. An item that was open and later closed keeps its `open` rows beside the closed one, since `state` is a tag. `private` says whether the repository the item went to is private. The searches run with the account's token, so an organisation's private repositories come back beside the public ones, and this field is what lets a page shown to others leave them out; a field and not a tag, so it added no second identity to rows already stored. A pull request carries `additions`, `deletions` and `changed_files`, the names `gh_pull_request` uses, and an issue carries none of the three. All of it comes on the nodes the searches already return, and measured on 2026-09-26 the query costs one point a page with it as without it. `gh_upstream_repo` is the repository side of the same searches: one row per repository they reached in the pass, stamped at the sweep, with its `stars` and `forks` as integers, `private` as a boolean, its primary `language`, absent when GitHub detects none, and its page. The star count is not on `gh_external_contribution` because that row is dated when the item closed, and a count that moves nearly every day would rewrite a row of the past each time. In InfluxDB and PostgreSQL the "Work elsewhere" table joins the newest of these rows inside the range onto each item as Stars, and in Prometheus its Stars is the count the exporter holds; Graphite and Elasticsearch cannot join one measurement onto another, and their tables have no Stars column. A repository gets a row whenever a search reads an item in it: each open state is read whole on every sweep, and each closed state from its most recent page, so a repository whose only items are closed and further back than that page keeps the row of the last sweep that read one of them, or of the last backfill. ## Continuous integration | Measurement | Dated | Tags | Fields | | ------------------------ | ----------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gh_workflow_run` | dated, when it finished | `workflow` (the file path), `event`, `conclusion`, `actor` | `duration_seconds`, `queued_seconds`, `attempt`, `success`, `run_id`, `run_number`, `pull_request`, `pull_requests`, `headline`, `head_repo`, `head_sha`, `head_branch`, `name`, `title`, `workflow_id`, `initial_actor`, `url` | | `gh_workflow_run_total` | now | | `runs` | | `gh_workflow_job` | dated, when it finished | `workflow`, `job_name`, `attempt`, `conclusion`, `runner_group`, `labels` | `duration_seconds`, `queued_seconds`, `steps`, `success`, `runner`, `run_id`, `head_sha`, `head_branch`, `url` | | `gh_workflow_step` | dated, when it finished | `workflow`, `job_name`, `attempt`, `step`, `conclusion` | `duration_seconds`, `step_number` | | `gh_workflow` | now | `workflow`, `path`, `state` | `active`, `age_days`, `days_since_change`, `url` | | `gh_artifact` | dated, when created | `artifact` | `live`, `size_bytes`, `retention_days`, `digest`, `run_id`, `head_sha`, `head_branch`, `url` | | `gh_artifact_total` | now | | `live_bytes`, `live_count`, `count`, `walked` | | `gh_actions_cache` | now | | `size_bytes`, `count` | | `gh_actions_cache_entry` | daily | `cache`, `ref` | `size_bytes`, `caches`, `key`, `days_since_use`, `age_days` | | `gh_repo_activity` | dated | `activity`, `actor` | `events`, `id`, `ref_name` | > **The workflow tag changed meaning** > > `workflow` used to hold the run's name and now holds the workflow's file path. > GitHub rewrites the name of anything dynamic, so a Dependabot run is named > after the bump it made and a code scanning run after the pull request that > triggered it: measured over three hundred runs of one repository, the name > took thirty-five values against seven paths, which is thirty-five series for > seven workflows. The human name is not lost, it is the `name` field, and > `gh_workflow` keys that same path to that same name, so a panel joins > `gh_workflow.path` to `gh_workflow_run.workflow` to print "CodeQL" again. > Nothing merges across the change: every series has a new identity, so rows > written before it sit beside the new ones rather than continuing them. `branch` was a tag on `gh_workflow_run` and on `gh_artifact`, and is now the field `head_branch` on both, and on `gh_workflow_job` as well. A branch is an identity, but not a reusable one: every pull request and every Dependabot bump mints a name that never comes back, so the tag grows without bound, and the bounded question a reader actually asks, whether this was a push or a pull request, is already the `event` tag. InfluxDB 3 also fixes a column as a tag or a field the first time it sees it and refuses every later write that disagrees, so keeping the name would have meant dropping both tables to publish a value the API itself calls `head_branch`. `attempt` is a tag on the job and on the step because the job listing is now asked for every attempt rather than the last one. Without it the two tries of a re-run are one series, told apart only by the second they finished in, and a flaky test cannot be distinguished from a broken one. `queued_seconds` on a run is written on first attempts only. GitHub keeps the run's `created_at` across re-runs, so on a second attempt the gap to `run_started_at` is the time a person took to press the button, not a runner queue: 0.4 s on average on first attempts against 1,747 s on second ones, measured. The queue of a retry exists only per job. `run_number` is the "#1483" GitHub shows and people quote; `run_id` is what the API keys by. `pull_request` is the number of the first pull request GitHub linked to the run and `pull_requests` how many it linked, both absent when it linked none. `headline` is the first line of the commit that ran, and `head_repo` is written only when the run came from another repository, which is what a fork's pull request looks like. `gh_workflow_run_total` is the run listing's own `total_count`, which is the whole history rather than the few hundred runs the walk sees. It is current state, so it is stamped now, and it is the only place "how many runs ever" can be answered without scanning the table. An ordinary sweep asks for the run list in pages of thirty rather than a hundred. The page is thirteen kilobytes a run, of which the collector keeps six hundred bytes, and at a hundred runs it was a megabyte and a half per active repository every quarter of an hour, forty six percent of everything a day downloads. The stores lose nothing: the walk still pages on while a page is full of runs newer than the window, up to seven pages, which is the two hundred and ten runs two pages of a hundred reached, and the first sweep after start and a backfill still ask for a hundred. What changes is the Prometheus exporter, which holds only what the last sweep collected and shows the newest thirty runs between builds rather than the newest hundred. The jobs of a run are listed once. The jobs of a completed attempt never change, and listing them again every sweep was a request per run in the window, nearly all of them 304s that cost no quota but a third of a second of waiting each, ninety six times a day. The collector remembers each attempt whose jobs it wrote, and keeps that memory with the ETag cache in [the file beside the state file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it), so a restart asks only for new runs and new attempts, as the sweep before it would have; a backfill lists every run regardless. A re-run keeps the run's id and is a new attempt, so it is listed again. The cap of twenty bounds what a sweep pays, not which runs get jobs: a window with more runs than that fills in twenty a sweep. A run is remembered only once every store has taken the pass that listed its jobs: a pass a store refused forgets its runs, and the next one lists and writes them again. The file keeps a run while the listing keeps returning it, and forgets it on the same horizon as an answer nobody asks for any more. A start that finds no write ledger to say what the stores hold, or a store added since, recalls none of them and lists their jobs again. A run's jobs outlive their steps. Measured on 24 September 2026, GitHub listed every job of a run 278 days old, with its times and its runner, and gave every run created before 12 April, about five and a half months back, an empty step list. A job that finished as success, failure or timed out ran at least one step, so when its list comes back empty it is written with no `steps` field rather than with a 0 that says it had none, and with no `gh_workflow_step` rows: a zero there would pull every mean of `steps` toward nothing as far back as a backfill reached. Every other job keeps `steps` as the length of its list, because its zero can be true: a skipped job runs no step (measured), and a cancelled one can stop before its first. `retention_days` is the retention an artifact actually got, which is rarely the configured default: eighty-eight of a hundred artifacts measured lived one day against a setting of ninety. `gh_repo_activity` is one row per activity type, actor and second. The branch is the field `ref_name`, one name per pull request and per Dependabot bump, and without it in the key the branches one push moved in the same second would be one row in every store, the last one written standing for all of them: eight force pushes in one second, measured. The entries that share a key are folded into one point, `events` counting them, `ref_name` naming every branch joined by commas, `id` the newest entry's, so a sum of `events` is the number of activities everywhere. Queue time only exists at the job level. The run-level figure folds the wait into the duration, and the job-level one includes waiting for a dependency, so a job that waits nineteen minutes for another job to finish is not evidence of a runner shortage. `runner` is a field, not a tag: a hosted runner is named uniquely per run, so as a tag it would create a series for every job ever executed. The workflow job tag is `job_name` rather than `job`, because `job` collides with the labels Prometheus adds at scrape time. `gh_artifact_total` carries three counts because its size is on none of the obvious ones. `count` is GitHub's own total for the repository and it counts the artifacts GitHub has already expired: measured on jmrplens/jmrp.io on 2026-09-17, page 40 of the listing was expired to the last row against a declared 29,405. `walked` is how far the five-page cap let the walk go. `live_bytes` is the size of the artifacts GitHub still holds among the ones walked, and `live_count` is how many those are, which is the count the size is over. When `walked` is below `count` the live figures are a floor rather than a total, which on that repository was short by a factor of fifty six. A walk that a failed page cut short still writes the row, from the pages before it, and its `walked` stops short of `count` in the same way; only a first page that failed writes none, because then there is no count to be short of. `gh_actions_cache` says a repository holds twelve gigabytes; `gh_actions_cache_entry` says which key holds them and which has not been touched for a week, which is what decides what GitHub evicts at the ten gigabyte ceiling. The `cache` tag is the key without its content hash, because the whole key is a series per build. A row of `gh_actions_cache_entry` is one cache on one ref for the day, with its entries summed into it: `caches` is how many there are, `size_bytes` their total, `days_since_use` and `key` those of the one used most recently, and `age_days` that of the oldest. Up to 2.5.2 it was a row per entry, and every entry of one cache on one ref had the same tags and the same day, so the store kept whichever was written last: on 2026-09-26 the fifteen CodeQL caches on main of jmrplens/jmrplens, 57.9 MB between them, were stored as one of 3.8 MB. No API lists a past day's caches, so those days stay as they are, and `-migrate` only notes them, under `2.6.0/gh_actions_cache_entry/sum`. The listing is read a hundred entries a page, up to ten pages, where it used to stop at the first: on 2026-09-27 two repositories of the account held 118 and 232 entries. It is read newest created first, an order a cache hit does not change. A pass that reads fewer entries than the listing said it held, because one was deleted between two of its pages, writes no `gh_actions_cache_entry` row, and nor does one whose later page failed: summed, a part of a cache would be written over the whole of it that the day's earlier passes stored, and the day's next pass writes the rows. The `gh_actions_cache` total, read before the listing, is written either way. ## Security | Measurement | Dated | Tags | Fields | | ----------------------------- | ------------------------ | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `gh_dependabot_alert` | now | `severity`, `ecosystem` | `open`, `url` | | `gh_dependabot_alert_item` | dated, when raised | `number`, `severity`, `ecosystem`, `package`, `ghsa`, `scope`, `relationship`, `manifest` | `alert_state`, `alerts`, `cvss`, `cvss_v4`, `epss`, `epss_percentile`, `cve`, `cwe`, `summary`, `vulnerable_range`, `first_patched`, `dismissed_reason`, `dismissed_by`, `dismissed_comment`, `seconds_to_detect`, `seconds_to_resolve`, `url` | | `gh_code_scanning_alert` | now | `severity`, `tool` | `open`, `url` | | `gh_code_scanning_alert_item` | dated, when raised | `number`, `severity`, `tool`, `rule`, `path`, `category`, `ref` | `alert_state`, `resolution`, `alerts`, `commit`, `line`, `cwe`, `seconds_to_resolve`, `url` | | `gh_code_scanning_analysis` | dated, when the scan ran | `tool`, `version`, `ref`, `category` | `analyses`, `results`, `rules`, `commit` | | `gh_security_feature` | now | `feature` | `enabled`, `open_alerts`, `alerts`, `url` | | `gh_security_setting` | now | `setting`, `status` | `enabled` | | `gh_code_scanning_setup` | daily | `state`, `query_suite`, `schedule` | `setups`, `languages`, `days_since_change` | | `gh_secret` | daily | `kind` (actions, dependabot), `secret` | `secrets`, `age_days`, `days_since_rotation` | | `gh_actions_policy` | daily | `permissions` | `policies`, `can_approve_pr` | The tags on the two item measurements cost nothing: both listings were already carrying them, and the series was always keyed by `number`, so they group rows that exist one per alert rather than multiplying them. All of them are always written, falling back to `(none)` when GitHub omits one. A tag written only sometimes gives the measurement two Graphite path depths, and the panels index their nodes from one fixed table. `alert_state` on both items and `resolution` on the code scanning one are fields, since an alert is dated when it was raised and both move when it closes; `resolution` reads `open` until then, so the column exists before any alert has closed. A build before 1.0.0 tagged both items with `state`, and the code scanning one with `reason`, so a store it wrote holds those rows beside today's: `-migrate` finds them, under `1.0.0/gh_dependabot_alert_item/state` and `1.0.0/gh_code_scanning_alert_item/state`, in the stores that can be asked. The fields are the opposite: `cve`, `cwe`, `first_patched`, `epss`, `epss_percentile` and `seconds_to_detect` are written only when the advisory carries them, because a missing EPSS score is not a score of zero. `cvss` and `cvss_v4` need the same guard for a different reason: GitHub always sends both keys and fills the one it lacks with `0.0`, and an advisory published with a v4 vector only was 78 of the 225 alerts of one repository, enough for "worst CVSS" over a severity group of them to read zero. Neither score is written unless it is above zero; a panel that wants one number per alert reads `COALESCE(cvss_v4, cvss)`. `summary` is the advisory's title, `vulnerable_range` the range it covers, which next to `first_patched` is the action to take, and `dismissed_reason`, `dismissed_by` and `dismissed_comment` say why a person closed an alert without fixing it; they exist only on an alert in state `dismissed`. `seconds_to_detect` is the gap between the advisory being published and the alert being raised here. It is negative when the alert came first, which happens when an advisory is written up after the fact. The two `cwe` fields cannot be joined. Dependabot writes `CWE-400` and code scanning writes `cwe-079`, both GitHub's own spelling, and neither is normalised here. `line` is the start line of the alert's most recent instance, and its zero is GitHub's own value for an alert about a whole file, not a missing reading. A Dependabot alert closes three ways, not two: `auto_dismissed_at` is how GitHub closes a development-dependency alert on its own, leaving the other two null. An alert closed that way used to be counted as still open forever. An alert still open carries no field for how long it has been open, and that is deliberate. The row is dated when the alert was raised, so the answer is now() less the row's own timestamp and a panel computes it when it is asked. Written by the collector instead, as `seconds_open`, it was true only at the instant of the sweep that wrote it and it moved on every sweep: a reader querying last week got whatever the last sweep decided, and every rewrite filed another parquet file in the partition of the alert's original date. Measured on 2026-09-17, the two alert families were writing about 234 files a day between them for 1,700 rows, whether or not GitHub had anything new to say. `gh_security_feature` exists so that no data and no alerts are distinguishable. Without it, a repository with Dependabot switched off looks exactly like one with nothing to fix. `enabled` is read from the first full page of the listing, which answers 403 when the feature is off and, for code scanning, 404 when nothing has been analysed yet. `open_alerts`, and `open` on `gh_dependabot_alert` and `gh_code_scanning_alert`, count every alert of the repository that is still open, however old. A sweep reads the newest page of each list, a hundred alerts in every state, and when that page comes back full the list is read again with `state=open`, to its end, and the counts are taken from that. An alert still open behind a hundred newer ones that were fixed is counted; up to 2.5.2 it was not. `alerts` is not the same number for both features. For code scanning it is the repository's total: when the page comes back full it is the last page GitHub declares for a page of one alert, which on one repository read 1,393 where the page had said 100. Dependabot's list pages by cursor and declares no last page, so there `alerts` is what the sweep read: 100 on a sweep means a hundred or more, and a backfill that walks the whole list writes the total. The two item measurements are the alerts the walk read either way, and complete after a backfill. ## Account | Measurement | Dated | Tags | Fields | | ------------------------ | ------------------------------------ | --------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gh_account` | now | | `followers`, `following`, `following_users`, `public_repos`, `gists`, `packages`, `projects`, `starred`, `watching`, `sponsors`, `sponsoring`, `account_age_days`, `pronouns`, `url` | | `gh_contributions_total` | now | | `calendar_total`, `commits`, `pull_requests`, `reviews`, `issues`, `repositories`, `restricted`, `repos_with_commits`, `repos_with_issues`, `repos_with_pulls`, `repos_with_reviews`, `url` | | `gh_contribution_day` | dated, one point per calendar day | | `contributions`, `level`, `url` | | `gh_contribution_year` | dated, end of the year; the year in progress daily | `year` | `contributions`, `commits`, `issues`, `pull_requests`, `reviews`, `repositories`, `restricted`, `repos_with_commits`, `repos_with_issues`, `repos_with_pulls`, `repos_with_reviews`, `partial` | | `gh_contribution_repo` | now | `kind` (commits, issues, pulls, reviews) | `contributions`, `commits`, `days`, `commits_dated`, `url` | | `gh_contribution_day_repo` | dated, the day the commits belong to | `private`, `own` | `commits`, `url` | | `gh_commits_week` | dated, the Sunday of its week | | `commits`, `owner_commits` | | `gh_commit_punchcard` | now | `weekday`, `hour` | `commits` | | `gh_package` | now | `package`, `type`, `visibility` | `versions`, `tagged_versions`, `age_days`, `days_since_update`, `url` | | `gh_package_version` | dated, when published | `package`, `type`, `visibility`, `tag` | `digest`, `published`, `url` | | `gh_gist` | now | `gist`, `public` | `files`, `comments`, `size_bytes`, `description`, `url`, `age_days`, `days_since_update` | | `gh_achievement` | daily | `achievement` | `name`, `tier_number`, `tier_name`, `present`, `image`, `url` | | `gh_achievement_progress` | daily | `achievement` | `name`, `count`, `tier_number`, `next_threshold`, `percent`, `page_tier`, `agrees`, `image`, `url` | | `gh_social_account` | now; the `orcid` row daily | `provider` | `url`, `present` | | `gh_pinned_item` | now | | `pinned`, `position`, `kind`, `stars`, `days_since_push`, `url` | | `gh_profile_flag` | now | `flag` | `enabled`, `message`, `age_days`, `url` | | `gh_sponsorship` | dated, when the sponsorship was made | `direction` (sponsor, maintainer), `sponsorable` | `sponsorship`, `active`, `one_time`, `privacy`, `tier`, `amount_cents`, `url` | | `gh_sponsors_listing` | now | | `has_listing`, `listing_name`, `listing_public`, `listing_age_days`, `tiers`, `monthly_income_cents`, `next_payout_cents`, `next_payout_date`, `sponsor_spend_cents`, `lifetime_received_cents`, `sponsorships_received`, `goal_kind`, `goal_title`, `goal_target`, `goal_percent`, `url` | | `gh_sponsors_tier` | daily | `tier` | `tiers`, `price_cents`, `one_time`, `retired`, `age_days`, `url` | | `gh_star_list` | daily | `list` | `lists`, `items`, `private`, `name`, `age_days`, `days_since_add`, `url` | | `gh_account_total` | now | | `pulls_opened`, `pulls_merged`, `pulls_open_now`, `pulls_merged_elsewhere`, `pulls_reviewed`, `issues_opened`, `issues_closed`, `issues_elsewhere`, `commented_elsewhere`, `commits`, `repositories`, `url` | | `gh_repo_created` | dated, when created | `fork` | `created`, `private`, `url` | | `gh_key` | daily | `kind` (ssh, gpg), `key` | `keys`, `age_days`, `days_since_use`, `never_used`, `days_to_expiry`, `verified`, `revoked`, `can_sign`, `emails`, `url` | | `gh_discussion_comment` | dated | `own`, `is_reply`, `author`, `comment`, `number` | `comments`, `answers`, `upvotes`, `title`, `reply_to`, `private`, `discussion_answered`, `discussion_answerable`, `discussion_closed`, `answered_by`, `answer_chosen_by`, `state_reason`, `category`, `seconds_to_answer`, `seconds_to_close`, `url` | | `gh_issue_comment` | dated | `own`, `number` | `comments`, `private`, `url` | `following` is the profile's own number, and it counts organisations as well as people. GraphQL's `following` connection counts only users, which on this account read four where the profile read nine, so more than half of it was invisible. Both are kept: `following` is what the profile page shows, `following_users` is the connection's people-only count. When the profile request fails the connection's count fills both, which is the tell that the profile was not reached. `versions` on a package is the count the package element declares, with the walked count as the fallback. `tagged_versions` counts named releases only. It used to count every container tag, and about half of those are the OCI referrers fallback tag GitHub publishes for each attestation and signature manifest: `sha256-` followed by the digest the row already carries. Nobody pulls one, there is a fresh one on every build, and excluding them halved the count on two packages, from a hundred and twenty six to fifty seven on one. `gh_package_version` no longer writes a row for one either. A package GitHub attaches to no repository writes `(none)` in all three of the tags that name one, rather than leaving them out and landing in a series with no repository column for a query to name. `gh_pinned_item` and `gh_profile_flag` are the profile page itself as data. A pin has no date of its own, so both are stamped now. A pinned gist is not in a repository at all: GitHub names it by its hash, so `repo` is that hash and `owner` is the account, which is the only owner a pin can have. The `kind` field says which of the two a row is. `position` is a field and not a tag: a repository that moves from slot two to slot three is the same pin, and as a tag every rearrangement would fork the series. `flag` is a closed list of eight: `hireable`, `developer_program`, `campus_expert`, `github_star`, `bounty_hunter`, `employee`, `sponsors_listing`, which is whether the account has a Sponsors profile at all, and `limited_availability`, which is the availability status carried as an eighth flag rather than a measurement of its own, with the `message` it displays and the `age_days` since that status was set. `age_days` is written on that row alone and is the age of the status message, not of the flag; GitHub says when none of the other flags was granted, so no row carries a date for them. `gh_sponsorship` is the only dated record of the money. `gh_account.sponsors` and `gh_account.sponsoring` are counts as of now that say neither when nor to whom, and `gh_sponsors_listing.lifetime_received_cents` is a total with no dates in it. Both connections are read with `activeOnly` off, which is what recovers a lapsed one. `sponsorable` is the other party, and it is the literal word `private` when the sponsorship hides it, in which case no URL is written rather than one being guessed at. `gh_sponsors_tier` is standing inventory, the way an SSH key is. Dating a tier at its creation would put all eight of them in 2021, outside every dashboard range, where they would read as "no tiers"; anchored to the start of the UTC day they converge on one row per tier per day, and `age_days` keeps the creation date recoverable. `gh_star_list` is the same shape for the same reason: the lists the account files its stars into, one row per list with how many it holds, anchored to the start of the UTC day. A list carries two dates, when it was made and when a star last went into it, and both survive as `age_days` and `days_since_add` rather than dating the row, which would put a list made in 2024 outside every dashboard range. The tag is the slug, which the list's page is addressed by; the display name is a field. Whether the slug outlives a rename is not verified, since checking it means renaming a list. It rides in the account query that was already being paid for: measured on 2026-09-11, eleven lists with their item counts added nothing to a cost of one. `gh_contribution_day` is the only place the green squares exist as data. With `every.families.history` set, it reaches back to the year the account was created, at one GraphQL point per year. Its `level` is the square's shade, GitHub's own quartile of the year as the 0 to 4 the profile draws, which is not a function of the count: on one account 83 contributions on one day and 52 on another were both the second quartile. The quartile is of the window asked for, the trailing twelve months for the sweep and the calendar year for `history`, so a day both write can change shade between the two, as it does on the profile when a year is picked. Which is why the dashboard's grid shades a day by its own count instead: measured on 2026-09-14, the profile page shades by the fifths of the busiest day of the window, a rule that reproduced all 366 of its squares from GitHub's own counts, where `level` disagreed with the page on 33 of those days. `gh_achievement` is the one measurement that does not come from the API. GitHub lists achievements nowhere in REST or GraphQL, so the family reads the public profile page, `https://github.com/?tab=achievements`, every hour as an anonymous visitor: no token travels to it and it is charged to no budget. One row per badge, stamped at the start of the UTC day: `name` is the badge, `tier_number` is the number on its label (1 with no label, 2 to 4 for x2 to x4) and `tier_name` the colour that goes with it (default, bronze, silver, gold); the number is not called `tier` because Elasticsearch maps a field name once across every measurement's index and `tier` is already a string on sponsorships. The parser is strict about the markup it accepts and holds each part of a card to the others, so when GitHub changes the page the family logs one warning and writes nothing until the parser is updated; the rows it wrote before stay, and a panel reading the newest row per badge goes stale rather than wrong. `image` is the badge image the page shows at that tier, for a panel to draw. The site the page is read from is derived from `github.base_url`. A `base_url` that is a proxy in front of the API has to name the site with [`github.web_url`](https://docs.jmrp.io/ghchronicle/configuration/#github), because the API host answers the page's url with a JSON 404, and the family refuses that rather than reading it as no badges. An account with no badge at all has no achievements tab: its url answers 404 while the profile answers 200, which is no rows and no warning, the same reading every family gives a 404. A 200 with no card in it is refused as a changed page rather than read as none. `gh_achievement_progress` is written by the same family beside the badges: one row per badge that has tiers (Pull Shark, Galaxy Brain, Starstruck, Pair Extraordinaire), whether or not the page shows it yet, saying how far the account is from the next tier. GitHub publishes neither the rule a badge is earned by nor the count it has reached, so the count is recomputed from the API and the thresholds are the community's, the Tiers table of [Schweinepriester/github-profile-achievements](https://github.com/Schweinepriester/github-profile-achievements) as read on 2026-09-12: Pull Shark counts merged pull requests anywhere and its tiers begin at 2, 16, 128 and 1024; Galaxy Brain counts the discussions whose accepted answer the account wrote, at 2, 8, 16 and 32; Starstruck takes the stars on the most starred repository of the account's own, forks left out, at 16, 128, 512 and 4096; Pair Extraordinaire counts merged pull requests in public repositories with a co-authored commit, one per pull request, at 1, 10, 24 and 48, cross-checked the same day against a hand count (the two counts differed by two, the difference falling in a range where the hand count ran past the thousand results a search pages, and the page showed the same tier either way; the co-authored pull requests of a private repository moved nothing). The single-tier badges and the two GitHub is still testing have no row: there is no next tier to measure against. `tier_number` is the tier the count implies (0 below the first threshold), `page_tier` the tier the profile page shows (0 when the badge is not on it) and `agrees` whether the two are the same; when they are, `next_threshold` is where the next tier begins (0 at the top) and `percent` the count against it (100 at the top). A row that disagrees is a rule the page contradicts, or a page GitHub has not recomputed yet, said once per process in the log, and it carries neither field, so no bar is drawn from a rule the page contradicts. Three of the counts are one GraphQL query; the fourth is a walk over the account's merged pull requests in public repositories with their commit messages, split by merge date where a range holds more than the thousand results a search will page, one point a page. Over a whole account's life that is a few dozen points and 24 MB (35 queries and 94 seconds over 2,315 pull requests, measured on 2026-09-27), so it is done once and then kept: the state file holds the count with the last UTC day it covers, and each pass walks only the pull requests merged since, one page a pass: 468 to 513 KB on that account for each hourly pass the production proxy logged on 2026-09-27, a size that grows through the day with what is merged. The day a pass runs on is still being merged into, so its pull requests are in that day's row and walked again by the next pass. The whole history is walked again once a week, because the count can go down (a repository made private or deleted takes its pull requests out of `is:public`), and whenever the rule the count was kept by has changed. A pass the API will not answer writes the badges and no progress rows, and says which read failed at warning, with the error: `achievement counts unavailable, no progress rows this pass` or `co-authored pull requests unavailable, no progress rows this pass`. It costs that pass and nothing more. The rows are the day's, so the pass an hour later writes them, and a walk cut short leaves the kept count as it was, for the next pass to walk its days again. A count the walk could not settle whole is written as a floor, and said once per process while its numbers stay the same: ```text level=WARN msg="co-authored pull request count is a floor" capped=false truncated=3 ``` `capped` is a day that alone held more than the thousand results a search pages, and `truncated` how many pull requests had more commits than a page of a hundred, no trailer in the ones read, and the rest of their commits unread because the query that asked for them failed, which is the most the count can be short by. Both are kept in the state file with the count, so a pass that adds to a floor still says it is one. A pull request with more than a hundred commits and no trailer in the first hundred is read on, a hundred commits at a time and ten pull requests to a query, one point each, until a trailer turns up or the commits run out. Up to 2.6.0 it was left there and counted as a floor: on the account measured, three pull requests of 143, 144 and 248 commits made every pass warn `truncated=3`, and reading the rest of their commits on 2026-09-28, two queries and 121 KB, found no trailer in any, so the count of 33 had been exact all along. The weekly whole walk pays those queries, and so does each pass that walks the day such a pull request merged. `gh_social_account` carries one more row than the social accounts listing: the homepage, under the provider `website`, from the `blog` of the profile. The ORCID iD the profile page shows is in no endpoint, so the `achievements` family, which already reads that page every hour, writes it from the page's vcard under the provider `orcid`, stamped at the start of the UTC day like the badges. The links the API does list (Mastodon, LinkedIn, Bluesky) are written by the `profile` family from the API and skipped on the page, so no account is written twice. The read is as strict as the badges': a page without the vcard is a changed page and one warning, never "no accounts". `pronouns` is the profile's pronouns line, `he/him`, as a field on the headline row, absent when the profile shows none. A field and not a tag: it is free text the owner can edit, and as a tag every edit would fork the account's one series. `gh_issue_comment` and `gh_discussion_comment` are read from the newest end of their connections, which list oldest first: a sweep's one page is the hundred newest comments. The account's accepted answers are then read on their own, the newest five hundred on a sweep and all of them in a backfill, so a comment accepted as the answer after it left that hundred is still written with `answers` 1 on the next sweep. A comment both reads return is written once. Both carry `private`, whether the repository the comment was left in is private, for the reason `gh_external_contribution` does: `own` does not say it, since the account's own repositories can be private or public and an organisation's are not the account's own at all. `gh_contribution_year` has one row per past year, dated the thirty-first of December, and one for the year in progress, asked for on every run from the first of January to now. That row is a snapshot, stamped at the start of the UTC day and marked `partial`, so a panel comparing years can tell a bar that is still growing from one that is finished; read it as the newest row per `year`. The first run of the next year replaces it with the final row dated December. `gh_account.packages` is counted from the REST listings the profile family walks, not from GraphQL's `packages` connection, which does not see the container registry and answered 0 for an account whose four packages are all containers. If a listing fails the GraphQL count stands. `gh_account_total` is the answer to "how many ever". Every other measurement here is a row per fact, which is the right shape for "how many in July" and the wrong one for a lifetime count. GitHub counts them itself: ten of them are one GraphQL query with a search alias for each, and `commits`, which GraphQL search cannot count, is a REST search of its own. So each number is one row and is right on the first sweep of a fresh install. The three fields that end in `_elsewhere` share one meaning of elsewhere: in a repository the account does not own, since the search qualifier `-user:LOGIN` leaves out the ones it does, so an organisation's repositories count as elsewhere even when the account belongs to it. `pulls_merged_elsewhere` is its pull requests merged there and `issues_elsewhere` the issues it opened there. `commented_elsewhere` is the issues and pull requests there that carry at least one comment of its own (`commenter:LOGIN -user:LOGIN`): threads, not comments, each counted once however many it left. One it opened counts only if it also commented, since the opening post is not a comment: measured on 2026-09-26, 33 of the 102 threads the account had opened elsewhere. In 2.5.1 and earlier it was `commenter:LOGIN -author:LOGIN`, which left out the threads the account opened and kept every one in its own repositories: 125 where the definition above gives 55 on the account this was read against, 103 of them at home and 54 of those Dependabot pull requests. A store holding rows from before the upgrade sees the series drop by that difference on the first sweep after it. `gh_dependency_change` writes a row on every sweep of the `deps` family, not only when a dependency moved: a range with no change, a head that did not move and the first sweep, which has no base yet, each write one row with `change` and `ecosystem` at `(none)` and both counters at zero. InfluxDB 3 creates a table at its first point and answers a query naming a table it has not seen with an error, so a measurement written only on a change did not exist until the first bump, and the panel over it was an error until then. The zero row costs no request and the panels leave the `(none)` series out. The three measurements about other people's repositories, `gh_discussion_comment`, `gh_issue_comment` and `gh_repo_created`, exist because a sweep over one's own repositories cannot see any of it. Each carries `own` so the two can be told apart. > **Why the weekly row is anchored to Sunday** > > `gh_commits_week` is stamped at the Sunday that starts each week, not at the > sweep. A sweep on Tuesday and one on Friday have to land on the same row, or > every re-read writes a second copy of the year. ## Activity | Measurement | Dated | Tags | Fields | | ----------------- | ------------------ | ------------------------------------------------------- | ------------------------------- | | `gh_event` | dated | `type`, `action`, `ref_type` | `events`, `public`, `commits`, `url` | | `gh_notification` | dated, last update | `reason`, `private`, `subject_type` | `is_unread`, `notifications`, `title`, `url` | Both are windows, not histories. GitHub keeps the last three hundred events of the past thirty days, and drops notifications after three months unless they are saved. What is captured is what was there when the sweep ran. `gh_event` carries no actor, since the feed is the account's own and the actor was the login on every row; `action` and `ref_type` are written on every row, `(none)` on a push. `is_unread` is a field: reading a thread does not move its `updated_at`, so the daily read with `all=true`, the one that lists a thread read without a reply, rewrites the same row rather than opening a second one beside it. A notification's `url` is derived from the API address of its subject, and a subject shape the mapping does not recognise is left without one rather than guessed at, so a good part of the rows carry no link. ## Configuration and delivery | Measurement | Dated | Tags | Fields | | ----------------------- | --------------------- | ----------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gh_webhook` | daily | `hook` (the id), `host`, `active` | `events`, `hooks` | | `gh_webhook_delivery` | dated, when delivered | `hook`, `host`, `event`, `status`, `code`, `ok` | `deliveries`, `duration_seconds`, `redelivery` | | `gh_ruleset` | daily | `ruleset`, `target`, `enforcement` | `rulesets`, `active`, `days_since_change`, `url` | | `gh_ruleset_rule` | daily | `ruleset`, `rule` | `rules`, `bypass_actors`, `bypass_always`, `bypass_sampled`, `ref_include`, `ref_exclude` | | `gh_ruleset_version` | dated, when saved | `ruleset`, `target`, `actor_type` | `versions`, `version_id`, `ruleset_id`, `actor_id`, `url` | | `gh_branch_protection` | daily | `pattern` | `rules`, `admin_enforced`, `allows_deletions`, `allows_force_pushes`, `blocks_creations`, `dismisses_stale_reviews`, `requires_approving_reviews`, `required_reviews`, `requires_code_owner_reviews`, `requires_commit_signatures`, `requires_conversation_resolution`, `requires_linear_history`, `requires_status_checks`, `requires_strict_status_checks`, `required_checks`, `requires_deployments`, `restricts_pushes`, `restricts_review_dismissals`, `url` | | `gh_branch` | daily | `branch`, `is_default` | `branches`, `oid`, `days_since_commit` | | `gh_deployment` | dated, when the deployment was created | `deployment`, `environment`, `task` | `outcome`, `deployments`, `deployment_state`, `success`, `superseded`, `creator`, `commit`, `ref`, `log_url`, `environment_url`, `run_id`, `seconds_to_status`, `seconds_live`, `url` | | `gh_policy_file` | dated, when the path last changed | `file` (dependabot, codeowners, security, funding) | `present`, `bytes`, `changes`, `path`, `blocks`, `ecosystems`, `url` | | `gh_dependabot_ecosystem` | dated, when dependabot.yml last changed | `ecosystem`, `interval` | `blocks` | | `gh_environment` | daily | `environment` | `environments`, `days_since_change`, `age_days`, `protection_rules`, `has_branch_policy`, `protected_branches`, `custom_branch_policies`, `can_admins_bypass`, `url` | | `gh_deploy_key` | daily | `key`, `read_only` | `keys`, `days_since_use` | | `gh_repo_policy` | now | | `security_policy`, `forking_allowed`, `discussions`, `issues`, `wiki`, `sponsorships`, `blank_issues`, `auto_merge`, `delete_branch_on_merge`, `merge_commit`, `rebase_merge`, `squash_merge`, `funding_links`, `issue_templates`, `pull_request_templates`, `branch_protection_rules`, `codeowners`, `codeowners_errors`, `vulnerability_alerts`, `url` | | `gh_repo_total` | now | `visibility`, `archived`, `fork` | `commits`, `stars`, `forks`, `watchers`, `issues`, `issues_open`, `issues_closed`, `pulls`, `pulls_open`, `pulls_merged`, `pulls_closed`, `releases`, `discussions`, `labels`, `milestones`, `branches`, `tags`, `size_kb`, `repo_id`, `age_days`, `days_since_push`, `url` | | `gh_dependency` | daily | `ecosystem` | `packages` | | `gh_dependency_license` | daily | `license` | `packages` | | `gh_dependency_change` | now | `change`, `ecosystem` | `packages`, `vulnerable`, `base`, `head` | | `gh_rate_limit` | now | `resource` | `limit`, `used`, `remaining`, `used_ratio`, `seconds_to_reset`, `own_cost`, `own_queries` | | `gh_collector_family` | now | `family`, `scope` (family, repo), `reason` | `repos`, `failed`, `points`, `error` | Webhooks fail silently. Measured, one hook had been answering 403 for seventy-eight of its last hundred deliveries and nothing anywhere said so. A 404 from branch protection does not mean unprotected: a repository can be governed entirely by rulesets, which that endpoint knows nothing about. `gh_ruleset_version` is the changelog behind `gh_ruleset`: one row per saved version of a ruleset, dated the moment GitHub saved it, with the actor that saved it. `days_since_change` only summarizes that history: a ruleset switched off on a Tuesday and back on the Friday after reads as "changed three days ago", and nothing else collected says a protection was ever absent. GitHub names the actor by id and type and not by login, so the row carries `actor_type` as a tag and `actor_id` as a field. The family is `rulesets`, daily: one list request per repository and one history request per ruleset, both with an ETag, so a day on which nobody edited a protection costs nothing from the budget. Measured on 2026-09-11 against the ruleset guarding the busiest repository measured: twenty versions across five months, 3 KB, one core request. Only the host of a webhook URL is stored. The path usually carries a secret. `gh_repo_policy` and `gh_repo_total` arrive in one batched GraphQL query that costs a single point for ten repositories, which is why settings that a REST sweep would price at a hundred and ninety eight calls are collected at all. `codeowners_errors` is the one that fails silently: a broken CODEOWNERS file stops requesting reviews and says nothing. `vulnerability_alerts` rides in that same query at no extra cost and is a second, independent reading of the switch `gh_security_feature{feature="dependabot"}.enabled` reports: one is the repository's own setting, the other is whether the listing actually answered. Two sources that disagree is the case worth seeing. `gh_repo_total` is where an archived repository's stars and forks are read from, on the Overview and in _Every repository, ever_, and `gh_repo` is not. One the default filter sets aside for being archived gets no `gh_repo` row from a sweep, since no family walks it, and has only the one a backfill wrote, but it is still starred, unstarred and forked, so the `totals` family writes its `gh_repo_total` on every sweep from the query that dates its archive: the same tags and fields a collected repository's row has, `archived` true, stamped at the sweep. Up to 2.5.1 only a backfill wrote it, once, and on 2026-09-26 one such row said 4 stars where GitHub said 3. A live repository's stars and forks on the Overview still come from `gh_repo`, the row the Inventory table and the card read too. It gets no `gh_repo_policy`. That query asks about twenty five repositories at a time: measured the same day, the gateway answered the lifetime row of fifty archived repositories once in 9.2 seconds and refused it twice after about eleven, and answered twenty five in under seven. The three dependency measurements are off by default. The SBOM is one call and a megabyte or two per repository, and only the aggregate is kept: a single dependency bump is three hundred and seventy changes, and what is stored is six rows. The commit a diff ends at, and the next one starts from, is read as the bare SHA of `HEAD` under the `application/vnd.github.sha` media type: forty bytes, where the one-commit listing it used to read was five and a half kilobytes. The answer carries an ETag and is asked for conditionally, so on a repository nobody pushed to the day's read is a free 304, as the listing's was. The SBOM is read only when that head moved: GitHub regenerates it on every request, so its ETag never matches and each read is charged from its own bucket, and a repository without a commit has the packages it had. `gh_rate_limit` and `gh_collector_family` are what the collector measures of itself. The first is what it has left to spend: GitHub runs fifteen independent budgets, and without it a family skipped for want of budget looks exactly like a family with nothing to report. The second is what each sweep managed to do. One row per family it ran, always, with how many repositories it was asked about (`repos`), how many of them it could not collect (`failed`) and how many rows it produced (`points`). For `commits`, `issueevents` and `issues`, `repos` also counts the repositories the [movement query](https://docs.jmrp.io/ghchronicle/api/cost/#asking-first-what-moved) found nothing new in, which the family left unread and wrote no row for. And one row more per repository it lost, naming that repository the way every other measurement names one and carrying `reason`, a bounded word for what stopped it (the HTTP status, `rate limited`, `query too large`, `canceled`), with the whole message in the `error` field. `scope` is what tells the two apart: `family` for the first kind, whose repository tags hold `(none)`, and `repo` for the second. `error` holds `(none)` too where there is no message, which is not decoration: the line protocol drops an empty string field, so a column written only on a failure would not exist at all until one happened, and a query naming it would be refused rather than answered with no rows. The rows that always arrive are the point of it. A family with no row at all in a sweep did not run in that sweep, which an empty panel could never say, and they are also what makes the measurement exist on an account where nothing has ever failed: a table InfluxDB has never been written to is not drawn empty, it is refused. One value of `family` is not a family. `discover` is the repository listing, which is not configurable and cannot be switched off, and it is here because every family depends on it: a sweep that cannot list the repositories runs none of them, and without this the page would show sixteen families that never ran and no reason for any of it. It writes a row only when it failed, because a listing that worked is already stated by every other row of the same sweep. This exists because of one measured failure. On 2026-09-16 `gh_workflow_run` and `gh_workflow_job` held nothing at all for the five busiest repositories of this account, each because one `/repos//actions/runs//jobs` call had answered `502` once and the runner had thrown away everything that family had already collected for that repository. The Continuous integration row of the dashboard was computed over an account missing its five busiest repositories, the Cost row on the same page reported one of them burning 27.6 K macOS minutes, and the only record of the cause was one line in a journal. The collector keeps what it gathered before a failure now, and it writes down what failed. `GET /rate_limit` reports the budgets and charges for none of them, which is what makes almost all of this free. Not every one it reports is true: measured with the token this runs under, the endpoint answered `graphql` as `used=0, remaining=5000` in the same minute GraphQL itself answered `used=162` and moved by one on every query, and the two do not even share a clock. So the `graphql` row is built from the `rateLimit` block GraphQL answers with, and the endpoint's version of it is dropped rather than published beside it. That reading is a request every fifteen minutes and costs nothing in points, measured. `graphql` is not the only bucket the endpoint invents. Measured on 2026-09-12, an SBOM request's headers said `dependency_sbom` used 1, remaining 99, reset in 59 s, and `GET /rate_limit` two seconds later said used 0, remaining 100, its reset sliding forward a second per call. Every REST answer names the bucket it charged in its headers, so a row is built from the newest headers the client saw whenever they are still inside their own window and say more was spent than the endpoint admits. The windows of `dependency_sbom` and `search` are one minute, so a `dependency_sbom` row that reads zero between runs of the `deps` family is a refilled bucket, not the defect. The `graphql` row can therefore be absent, where the others are written whenever the endpoint answers at all. It is not written when nothing has been read from GraphQL yet and no earlier reading is still inside its window: a missing row says "not measured" where a zero says "nothing spent", and the zero was the defect. `own_cost` and `own_queries` are on that row alone. `limit`, `used` and `remaining` describe the whole token's window, shared with whatever else holds it; these two are the part this process is answerable for. They count from process start, so a restart returns them to zero and a panel has to read them as a counter rather than a value. ## Job logs | Measurement | Dated | Tags | Fields | | ------------ | -------------------------------- | ----------------------------- | --------------------- | | `gh_job_log` | dated, when the line was printed | `workflow`, `job_name`, `run` | `line`, `head_branch` | Off by default: set `every.families.joblogs`. It is text rather than a measurement, so it is excluded from the InfluxDB sink by default and skipped by the Prometheus exporter; Loki is where it belongs. The exported dashboards carry a text panel, "Where failure output went", in the place the lines would take, since an importer may have no Loki; `cmd/publish_dashboard -loki ` publishes the dashboard with the lines drawn from Loki in that panel's place (see [the dashboards](https://docs.jmrp.io/ghchronicle/dashboards/panels/#delivery-and-access)). Only failed jobs, and only the last forty lines of each. A successful job's output is thousands of lines nobody will read, each log costs a request, and the tail is where a failure explains itself. GitHub deletes logs after the repository's retention period, ninety days by default, and answers 410 once they are gone; a backfill asks only for the last ninety days, whatever the setting, and at most 500 failed jobs per repository. `workflow` here is the same file path, through the same helper, so a log line joins to the run that printed it; and `branch` became the field `head_branch` for the same two reasons as on the run. `run` is deliberately a series per run, which is what a log line genuinely belongs to, and it is affordable only because this family is off by default. Colour codes are stripped and the byte order mark GitHub writes before the first timestamp is removed, so a search for a word does not fail because the word happened to be coloured. The failure list is asked only for the runs created in the thirty one days before the window opened, rounded down to the day. Thirty one days because a re-run keeps the created_at of its first attempt and GitHub allows one for thirty days: measured on 2026-09-11, the newest failure of the busiest repository measured was the third attempt of a run created two hours before it finished, and a margin the length of a job would have missed every re-run of a failure older than a morning. Unfiltered the list was the newest hundred failures the repository ever had, six hundred kilobytes per repository per sweep for a window of an hour that is nearly always empty; filtered, a month of failures, sixty eight rows and a megabyte decompressed on that busiest repository, a few rows or none on most. Rounding keeps the URL, and with it the ETag, the same across the sweeps of a day, which is what makes the repeat a free 304 rather than a charged 200 on a fresh URL; within the day the page changes only when a failure is created or re-run. The cut at the window itself is still made here, by when the run finished. ## Cost | Measurement | Dated | Tags | Fields | | ------------------ | -------------- | --------------------------------------- | --------------------------------------------------------------- | | `gh_billing_usage` | dated, per day | `product`, `sku`, `unit` | `quantity`, `price_per_unit`, `gross`, `discount`, `net`, `url` | `unit` is GitHub's own `unitType`, capitalised as GitHub sends it: `Minutes`, `GigabyteHours`, `AICredits`, `Requests`. It is passed through rather than normalised, and the panels that read minutes filter on the capital. There is no `org` tag. The only billing endpoint a personal account can read is its own, and that report has no `organizationName`: the field belongs to the organization report, which needs an organization to ask about. Checked against the published OpenAPI description and against the live endpoint, where none of 487 usage items carried the key. Written anyway it was `(none)` on every row ever collected, which is a column and a legend entry that only ever says there is nothing here. `repo` is `(none)` on a charge that belongs to no repository, which is what a Copilot seat is. That is a real row of the bill and not a repository, so the cost table by repository leaves it out; the spend totals above it include it. `net` is not always zero. On the account this was developed against it carries the monthly credit, which is why gross, discount and net are all stored rather than one being derived from the others. The row is stamped at the start of its day: GitHub's `date` arrives as the first billed minute of the day on half the rows, which would double the row had GitHub reported a different minute on the next read. ## Columns that exist only once written InfluxDB 3 creates a column the first time a row carries it, and a query that names a column no row has written fails at planning rather than answering null: the whole panel goes red. So a field written only when GitHub has a value for it does not exist on a database where that has never happened. The ones a fresh database is most likely to lack: `gh_discussion.state_reason`, `seconds_to_answer` and `seconds_to_close`; `gh_milestone.days_to_due` and `seconds_to_close`; `gh_ruleset_rule.ref_exclude`; and `gh_workflow_run.initial_actor`. The same rule covers every `seconds_to_*` that needs a closing, every `url` on an item GitHub sends no address for, the optional advisory fields on an alert, `checks_total` and `checks_failed` on a commit a gate ran on, `label_names` on an item with a label, `resolved_by` on a resolved thread, `queued_seconds` on a run's first attempt, `pull_request`, `pull_requests`, `headline` and `head_repo` on a run GitHub linked, described or took from a fork, `merged` on work elsewhere that was merged and `additions`, `deletions` and `changed_files` on work elsewhere that is a pull request, which an account that has only opened issues in other people's repositories has never written, and `language` on an upstream repository GitHub detected one in. Two more are worth naming, because the tables above list them beside fields that are always there: `gh_dependabot_alert_item.dismissed_comment`, written only when whoever dismissed an alert typed a reason, and `gh_event.commits`, written only on a push event. Neither column exists on the production database this documentation was checked against. `gh_label` writes only the labels somebody has used; `gh_repo_total.labels` is the declared count. One of these is named by a shipped panel, and it is the one most likely to be missing: `gh_pull_request.seconds_to_first_human_review`, which exists only once somebody other than the author and other than a bot has reviewed a pull request. On a database where that has never happened, the stat that reads it reports a schema error rather than No data, and it takes the five values beside it in the same panel with it. Nothing in a query can ask whether a column exists, so this is a property of the store rather than a defect to repair: the repair, if it bites, is one row of any kind carrying the field. How rare it is, measured on 2026-09-17 against the account this was developed on: fourteen rows in the whole store, over fourteen pull requests of four repositories, the newest raised on 2026-07-05, and none of them inside the last fortnight, against 703 pull requests of 825 in that fortnight that had a first review from a bot or from their own author. The field is not broken; it is the answer to a narrower question than a reader expects, which is why the panel is named for that question. The same reading applies to the two on `gh_workflow_run`. `initial_actor` is written only when GitHub's `actor` and `triggering_actor` differ, which is a re-run somebody else asked for: 0 of 10,201 runs in a fortnight here, and 0 of the 300 newest runs of three repositories checked against the API at the same time. `head_repo` is written only for a run that came from another repository: 18 of those 10,201, all from one fork's pull request. Both are correct and both are rare, which is what a field written only when GitHub has something to say looks like. --- # Choosing a store Where to store GitHub metrics: eleven stores, what each one can and cannot answer, and the one property that decides between them. Source: https://docs.jmrp.io/ghchronicle/sinks/ Eleven sinks, and running more than one is the normal arrangement. Every one of them pushes: the tool is meant to run wherever it is convenient and reach its stores from there, not to be scraped. The Prometheus exporter is the one exception, and it exists because Prometheus insists. ## The comparison | Store | Keeps | Good for | Config | | -------------------------------------------------- | -------------------------------------------- | ---------------------------------------------- | ------------------------------- | | [InfluxDB](https://docs.jmrp.io/ghchronicle/sinks/influxdb/) | the dated history | "how fast were we merging in July" | `url`, `token`, `org`, `bucket` | | [PostgreSQL](https://docs.jmrp.io/ghchronicle/sinks/postgres/) | the dated history, in a database it connects to | a Grafana user with a Postgres and no InfluxDB | `dsn`, `batch` | | [PostgreSQL, to a file](https://docs.jmrp.io/ghchronicle/sinks/postgres/) | the same tables, as SQL you pipe into psql | a load that happens later, or elsewhere | `dialect`, `path` | | [Graphite](https://docs.jmrp.io/ghchronicle/sinks/graphite/) | the dated history | a Graphite that is already there | `addr`, `prefix` | | [Elasticsearch](https://docs.jmrp.io/ghchronicle/sinks/elasticsearch/) | the dated history, as documents | search across everything collected | `url`, `prefix`, `api_key` | | [Prometheus](https://docs.jmrp.io/ghchronicle/sinks/prometheus/) | the current value | alerting, and a number on a wall | `listen`, `path` | | [OpenTelemetry](https://docs.jmrp.io/ghchronicle/sinks/otlp/) | either, depending on the backend | an existing collector pipeline | `endpoint`, `raw` | | [Loki](https://docs.jmrp.io/ghchronicle/sinks/loki/) | the events, as log lines | "what happened, in order" | `url`, `labels`, `max_age` | | [Telegraf](https://docs.jmrp.io/ghchronicle/sinks/telegraf/) | whatever its outputs keep | reaching anything Telegraf can reach | `url` | | [File and stdout](https://docs.jmrp.io/ghchronicle/sinks/file/) | line protocol or JSON | a shipper you already run, and a buffer | `path`, `format` | There is no sink for a specific hosted vendor, and that is deliberate. A managed backend is reached through one of the two sinks that exist to route onward: Telegraf, whose own outputs cover Datadog, New Relic, Wavefront, Azure Monitor and a hundred more, or OpenTelemetry, which most of them now accept directly. A sink per vendor is a key to rotate, an API to track and a test that needs a paid account, for a hop those two already make. ## The thing that decides everything A point carries the date the thing happened. A star is dated when it was given, a workflow run when it finished, a traffic day at that day's own date. InfluxDB keys a point by measurement, tag set and timestamp, so writing the same fourteen-day traffic window every six hours converges on the right answer rather than accumulating copies. That is what makes the whole backfill design work, and it is why InfluxDB is the sink that keeps history. Prometheus cannot do that. It stamps a sample at scrape time and rejects anything meaningfully older: measured against Prometheus 3.14 with the OTLP receiver enabled and a thirty-minute out-of-order window, a sample dated two days back comes back as HTTP 400. So the exporter reduces the per-item rows to current values before serving them. The full argument, and what the reduction does to each measurement, is on [dating a point](https://docs.jmrp.io/ghchronicle/how/dating/). ## Only what changed is written A sweep offers the same history every time: the fourteen day traffic window, every open pull request, the contribution calendar. Writing it all again is harmless to what the store holds, since a row is keyed by series and timestamp and simply overwrites, and it is how a collector that was down for a day repairs itself. It is not harmless to the store's files. InfluxDB 3 Core writes one Parquet file per partition per write request and never compacts them, and it refuses any query that would open more than its file limit. Measured before the fix below and with the limit then at ten thousand: `gh_notification` held barely more than one row per Parquet file, and a query over fourteen days came back with "Query would scan 10000 Parquet files, exceeding the file limit". Asking for a coarser interval does not help, because the limit counts the files the planner opens, before any aggregation. So the tool keeps a small ledger of what it has already written and sends only the points whose values have moved: ```yaml sinks: dedupe_file: /var/lib/ghchronicle/state-written.bin # default: beside state_file dedupe_horizon: 720h # forget a point nothing offers any more influxdb: dedupe: true # the default, here and for telegraf, graphite, sql and elasticsearch ``` The ledger stores two 64 bit hashes and a day per point, so a large account costs a few megabytes. It is keyed by sink, so a store that was unreachable still receives everything on its next write. Losing it, or setting `dedupe_file: off`, costs one sweep of rewriting and nothing else, which is exactly what a store that has been wiped and needs filling again wants. The jobs of the workflow runs the [cache file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it) remembers come with it: a start that finds no ledger lists them again. It is the second file worth putting on a persistent path, next to the [state file](https://docs.jmrp.io/ghchronicle/configuration/). A run that ends when its sweep does never opens the ledger at all. `-once`, `-backfill` and a card render write once and exit, so there is nothing to save and nothing to prune, and each of them offers the whole history again. That is what a backfill is for; it is also why a scheduled `-once` job, which is the shape [the Action](https://docs.jmrp.io/ghchronicle/install/actions/) runs in, writes every point every time. Where that matters, run the loop instead. The sweep log says what this saved: ```text level=INFO msg=written sink=influxdb family=events points=0 unchanged=300 ``` If the files have already accumulated, the writer fix stops the growth but does not remove them: raise `--query-file-limit` on the server, rewrite the affected tables, or move to InfluxDB 3 Enterprise, which compacts on its own and is free for home use. ## What the written line counts `points` is what that sink took, not what it was handed. Every sink here drops something of its own inside the write: InfluxDB skips the measurements [`exclude`](https://docs.jmrp.io/ghchronicle/sinks/influxdb/#exclude) names, Loki keeps only the measurements it has an event rendering for and drops the rest, the SQL, Graphite, Telegraf, file and stdout sinks skip a point that renders no row or line, and the two exporters skip a measurement they have no rule for. A third key appears when that happened: ```text level=INFO msg=written sink=loki family=repos points=0 unchanged=0 filtered=42 ``` `filtered` is what the sink did not write, and each sweep ends with one line per sink carrying the total since the run started, the way the ledger's own total beside it reads. Lines a store refused to parse are not counted as written either; they get a warning line of their own. Until this was counted the line said what the runner had handed over, so a sweep read `points=440` for a measurement InfluxDB excludes by default and has never held a row of, and reading Loki's lines told you nothing about what Loki stored. ## Start here - [You want the history](https://docs.jmrp.io/ghchronicle/sinks/influxdb/): InfluxDB is the reference implementation and the dashboard is built against it. PostgreSQL, Graphite and Elasticsearch keep the same facts in their own shapes. - [You want alerts](https://docs.jmrp.io/ghchronicle/sinks/prometheus/): Prometheus serves the current value of everything that has one. Run it alongside a history store rather than instead of one. - [You want to read what happened](https://docs.jmrp.io/ghchronicle/sinks/loki/): Loki turns twenty-two of the measurements into log lines that read as sentences and carry every tag after them in logfmt. - [You already have a pipeline](https://docs.jmrp.io/ghchronicle/sinks/telegraf/): Telegraf and OpenTelemetry hand the points to something that already knows where they should go. ## Running several Normal, and cheap: the collection happens once and the points are handed to every configured sink. The usual arrangement is one history store plus one of the current-value ones. ```yaml sinks: influxdb: url: http://localhost:8181 token: ${INFLUX_TOKEN} org: default bucket: github prometheus: listen: 127.0.0.1:9605 path: /metrics ``` > **One combination to avoid** > > `sinks.sql.path: "-"` and `sinks.stdout: true` both write to standard output, > and interleaving SQL statements with line protocol produces a stream that > neither psql nor Telegraf can read. Choose one. ## What every sink does with a failure A sink that fails is logged and the sweep continues; a database being down does not stop collection, and with the file sink configured the data is still on disk when it comes back. Two failures are reported specially rather than as errors, because they are partial successes: - **rejected lines**, where everything parseable was written and the refused lines are logged individually with the store's own reason. - **dropped entries**, which is Loki's age horizon leaving out what its out-of-order window would have refused, rather than losing the whole push. --- # InfluxDB The reference store for the dated history, why re-collection converges, and the one write error worth recognising. Source: https://docs.jmrp.io/ghchronicle/sinks/influxdb/ The InfluxDB sink writes every ghchronicle point, dated when it happened, to InfluxDB 2 or 3, and it is the store the dashboards are built against. ```yaml sinks: influxdb: url: http://localhost:8181 token: ${INFLUX_TOKEN} org: default bucket: github batch: 5000 # exclude: [gh_job_log] ``` ## The wire contract Line protocol posted to the **v2 write endpoint**, which InfluxDB 2 and 3 both serve, so one sink covers both. The token goes in an `Authorization` header, `org` and `bucket` in the query string. Precision is **nanoseconds**, because the traffic points are days and the workflow ones are seconds and one precision has to cover both. Batches default to 5000 lines per request. `batch` lowers that for a server with a smaller body limit. ## What it can answer that the others cannot Everything dated, which is most of the project: - The traffic of a particular Tuesday, months later. - The star curve since the first star, drawn from one row per repository and day. - The merge time of a pull request closed in July. - Lines added and removed per commit, per author, over years. InfluxDB keys a point by measurement, tag set and timestamp, so **rewriting a point that already exists is not a duplicate**. Replaying the same fourteen-day traffic window every six hours converges instead of accumulating, which is what makes the whole [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/) design work. The [dashboards](https://docs.jmrp.io/ghchronicle/dashboards/) are generated against this store first; the other four query sets are translations of it. ## Rewriting is free in rows and not in files InfluxDB 3 Core writes one Parquet file per partition per write request and never compacts them, and it refuses any query that would open more than its file limit. So the row that overwrites harmlessly still costs a file, and a sweep that offers the same history every six hours buys "Query would scan 10000 Parquet files, exceeding the file limit" a few weeks later. That is why the write ledger exists, why `dedupe` is on by default here, and why turning it off is a decision rather than a tidy-up: [only what changed is written](https://docs.jmrp.io/ghchronicle/sinks/#only-what-changed-is-written). ## `exclude` Names measurements this sink should not receive. It defaults to `gh_job_log`, which is text meant for a log store: writing thousands of lines of build output into a metrics database is a lot of storage for something nobody will query as a number. An excluded measurement is still collected and still offered to this sink, so the sweep log says what the database took rather than what it was handed: ```text level=INFO msg=written sink=influxdb family=joblogs points=0 unchanged=0 filtered=440 ``` `filtered` here is a measurement named above, or a point carrying no field the line protocol can render. It is [the same key every sink uses](https://docs.jmrp.io/ghchronicle/sinks/#what-the-written-line-counts). Until this was counted the line read `points=440` for a measurement the database has never held a row of. ## Column types are fixed on first sight > **InfluxDB 3 will not let a name change side** > > InfluxDB 3 fixes a column as a tag or a field the first time it sees it, and > rejects later writes that disagree. If a version of this tool ever moves a name > between the two, the table has to be dropped rather than repaired: > > ```http > DELETE /api/v3/configure/table > ``` > > This is the usual cause of `influx write: 400`. The delete does not destroy the > table at once: InfluxDB renames it `-`, keeps it queryable and > purges it 72 hours later by default, or never before 3.2 (see > [below](https://docs.jmrp.io/ghchronicle/sinks/influxdb/#before-32-the-copy-stays)), and the name takes new writes straight > away. For the changes the binary knows of, `-migrate -yes` sends it: see below. ## Rejected lines A write that comes back 400 is bisected: the sink halves the batch, retries, and narrows down to the individual lines the server refuses to parse. Those are logged one by one with the server's own reason, everything else is written, and the sweep reports a warning rather than a failure. ```text level=WARN msg="sink rejected some lines" sink=influxdb family=actions rejected=2 ``` That behaviour matters because a batch is five thousand lines. Failing the whole batch on one malformed value would lose four thousand nine hundred and ninety-nine good points. ## Grafana Use the **InfluxDB 3 datasource in SQL mode** for the shipped dashboard, and point it at the database the sink writes to. The dashboard file declares `DS_INFLUXDB` as an input, so importing asks you to choose your own datasource rather than carrying somebody else's uid. ```sql SELECT time, "count" FROM gh_traffic WHERE kind = 'views' AND repo = 'ghchronicle' ``` ## What a migration does here When a release changes what a measurement's rows are keyed by, [`-migrate`](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations) asks InfluxDB 3's catalog whether the old tag is a tag column of the live table, and applying the change deletes that one table: ```http DELETE /api/v3/configure/table?db=&table= ``` That delete is InfluxDB's own way of setting a table aside. **Measured against InfluxDB 3 Core 3.0.0 to 3.11.5**: the table is renamed `-`, the instant in UTC, for example `gh_discussion_comment-20261001T091004`, stays listed and answers queries under that name, and the next write under the old name creates the table afresh, in the new shape, even where a column changes from a tag to a field. The binary reads the name back from the catalog, prints it, and keeps it in the state file, with when the server has scheduled its hard deletion, read from the system table of its `_internal` database. That is 72 hours after the delete (measured on 3.2.1 to 3.11.5), and the server keeps the name in its catalog for its `--delete-grace-period` after that, 24 hours by default; where the time cannot be read, 3.2.0 among them, 72 hours is assumed, which is what 3.2.0 schedules too. A server before 3.2 never purges it: see [below](https://docs.jmrp.io/ghchronicle/sinks/influxdb/#before-32-the-copy-stays). ghchronicle leaves the copy to the server, and forgets it once the server's time for it has come. It never asks for it to go sooner: `hard_delete_at` is never sent, since measured on 3.11.5 `now` had not removed the rows eleven minutes later, and it takes the days to undo the change away; and from 3.10.0 on, a delete of the renamed table is answered with a 409, with or without it (measured on 3.10.0 to 3.11.5). Until then the copy's rows can be read with SQL and written back through the sink. InfluxDB 3 Core refuses a query that would open more Parquet files than its `--query-file-limit`, 432 by default, which a table written every ten minutes passes in days. The check reads the catalog, which opens none, and counts the rows only of a table that holds the old tag; a count Core refuses leaves the refill with no bound, and a refused read of whose rows the table holds leaves the change needing your word, the refusal quoted in the plan. See [the query file limit](https://docs.jmrp.io/ghchronicle/install/upgrading/#influxdb-3-cores-query-file-limit). The token has to be allowed to delete a table. A refused delete leaves the change pending, and the error names the same request to send by hand. InfluxDB 2 has no rename. Applying the change there deletes every row of the measurement in the bucket, over every instant it can hold, through `POST /api/v2/delete` with the predicate `_measurement=""`, and that is final. **Measured against InfluxDB 2.7.12**: it took that measurement and left `gh_discussion_comment_x`, the other measurements of the bucket and the same measurement in another bucket as they were. So a start never applies it on its own; `-migrate -yes` does. `-uninstall data` leaves out the tables InfluxDB 3 has already deleted, and says of each whether the server purges it or it stays: on a server before 3.2 every one stays, and on a later one each that a release before 3.2 deleted, asked of the server's system table, with the request that removes it where the release takes one (see [below](https://docs.jmrp.io/ghchronicle/sinks/influxdb/#before-32-the-copy-stays)). The rest are the server's to purge, and only a release from 3.10 on is said to refuse to be asked for that sooner. 3.2.0's system table cannot say which, and there the note says so. ### Before 3.2 the copy stays **Measured against InfluxDB 3 Core 3.0.0, 3.0.3 and 3.1.0**, none of which has a hard deletion: no deleter runs, `hard_delete_at` is taken and ignored, `_internal` has no `system.tables`, and a delete of the renamed table answers 200 and renames it once more, `--`. The copy stays, and answers queries, for as long as the server runs one of those releases. 3.2.0 is the first release with a deleter: 3.2.0 and 3.3.0 were measured scheduling a hard deletion and carrying it out as 3.4.0 and later do. The binary reads the release from `/ping`, and on a server before 3.2 says so where it matters: the plan says `set aside for good` before anything is applied, applying says it with the copy's name, and `-migrate` lists each copy such a server keeps as `kept aside ... for good`, asked of the server itself, so the list holds a copy the state file has forgotten, or a new state file never knew. `-uninstall data` says the same of every table it deletes there. Each ends in the request that removes the copy once the server runs a release that takes it: ```sh curl -X DELETE '/api/v3/configure/table?db=&table=&hard_delete_at=now' \ -H 'Authorization: Bearer ' ``` A copy made before 3.2 keeps no hard deletion time through an upgrade: measured with copies 3.0.3 and 3.1.0 made, opened by 3.2.0, 3.4.0, 3.9.13 and 3.11.5. So it stays after the upgrade too, until it is told to go. 3.2.0, 3.4.0 and 3.9.13 took the request for such a copy and dropped it from their catalog once their delete grace period had passed, 3.2.0 after renaming it once more; 3.10.0 to 3.11.5 answer it with a 409. So the request is sent while the server runs a release from 3.2 to 3.9, on the way up. From 3.2.1 on, `-migrate` lists the copies the system table shows with no hard deletion time, which is how such a copy looks there, and `-uninstall data` says each stays, with that request where the release takes it (measured on 3.4.0 with a copy 3.1.0 made). 3.2.0's system table has no such column, and answers the question with a 500, so on 3.2.0 `-migrate` lists none and `-uninstall data` says it cannot tell. Measured as well: 3.11.5, started on the data directory of a 3.0.3, began with an empty catalog, while 3.4.0 and 3.9.13 read the catalogs 3.0.3 and 3.1.0 had left. ## Where to go next - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/) compares InfluxDB with the others, and holds the write ledger every one of them shares. - [The dashboards](https://docs.jmrp.io/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Prometheus GitHub metrics in Prometheus: an exporter, not a pusher, and what the reduction to current values does to each measurement. Source: https://docs.jmrp.io/ghchronicle/sinks/prometheus/ The Prometheus sink is an exporter that serves the current value of each GitHub metric for Prometheus to scrape. ```yaml sinks: prometheus: listen: 127.0.0.1:9605 path: /metrics ``` An exporter, not a pusher: point a scrape at it. It is the one sink in the project that is not outbound, and it exists because Prometheus insists on pulling. ## What it serves Metric names are `github__`, with the tags as labels. ```text github_repo_stars{repo="ghchronicle",language="Go",visibility="public"} 283 github_workflow_runs_count{repo="ghchronicle",conclusion="success"} 412 ``` ## Can Prometheus keep the history? No, and not by choice: it cannot serve the dated history. Prometheus stamps a sample at scrape time and rejects anything meaningfully older: measured against Prometheus 3.14 with `--web.enable-otlp-receiver` and a thirty-minute out-of-order window, a sample dated two days back comes back as **HTTP 400**. Through this sink, GitHub's fourteen-day traffic window collapses to its most recent day, and the star history to the current total. That is worth stating plainly rather than hiding. Run it alongside a history store rather than instead of one; both can run at once and the collection happens only once. ## The reduction Before serving, `Summarize` reduces each measurement according to a rule. | Rule | What survives | | ---------- | ----------------------------------------------------------------- | | `keepLast` | The most recent value per label set. Snapshots | | `sum` | The rows added up. Windows, such as views over the fourteen days | | `count` | A count plus the mean of each numeric field. Dated items | | `skip` | Nothing | A measurement with no rule is skipped, so a new collector cannot quietly flood the exporter with one series per star. The batch is read the way the stores hold it. A point with the same measurement, tags and time as an earlier one is the same row, so it is added, counted and averaged once, as the history stores keep it once; [dating a point](https://docs.jmrp.io/ghchronicle/how/dating/#the-reducer-and-what-it-makes-of-each-measurement) says which of its fields each store keeps. A price is the one number a `sum` does not add: the bill repeats a SKU's price on each row, one per repository and day, so `github_billing_usage_price_per_unit` is the highest of them, the `MAX(price_per_unit)` of the SQL dashboards, and not the price times the days billed. Each mean is over the items that carried the field, not over the count. A collector leaves a field out when it has no honest value for it: a job with no start time has no `queued_seconds`, and a job GitHub no longer lists steps for has no `steps`. Counted as zeros, a sweep in which half the jobs had lost their steps would halve `github_workflow_jobs_steps_mean`. Three fields are the exception, because leaving them out is itself the answer: `merged` on an outside contribution is written only on a merge, `advanced` on a fork only when it has a push date, and `pull_requests` on a run only when it ran for any. Each of those is averaged over every item, so the mean of `merged` is the share of contributions merged rather than 1. The reducer also publishes `total`, a running distinct-item count per series. That is what lets a Prometheus dashboard say "per day" through `increase()`, since it has no rows to count. ## Two measurements are skipped for size The commit punch card is one series per repository, weekday and hour, and the release assets one per file ever published. Measured, together they were **four fifths of the exporter's entire output**. Both are drawn properly by the InfluxDB dashboard. Ten more are skipped for a reason other than size, nine of them as history and one as text, so twelve measurements in all never reach the exporter. The list, with the reason for each, is on [dating a point](https://docs.jmrp.io/ghchronicle/how/dating/#the-twelve-that-are-never-served). The exporter holds only what the last sweep collected, and for workflow runs that is the newest thirty per repository between builds: an ordinary sweep reads the run list in pages of thirty, and pages on only while a page is full of runs newer than its two hour window. The stores keep every run the walk ever saw; this is the one place the smaller page is visible. Workflow jobs are the other: a sweep carries the jobs of the runs it listed for the first time, so `gh_workflow_jobs` counts the jobs new to this process rather than those of the newest twenty runs, and a sweep in which no run finished carries none, leaving the last count standing until it goes stale a day later. The `total` beside it is unaffected, since it counts every distinct job the process has ever seen; the stores are unaffected too, since a job is written once and dated when it finished. > **A tag called job would collide** > > Prometheus adds `job` and `instance` labels at scrape time, and the OTLP > receiver overwrites a `job` attribute with the service name. Workflow jobs are > therefore tagged `job_name`. ## Scraping it ```yaml scrape_configs: - job_name: ghchronicle static_configs: - targets: ["127.0.0.1:9605"] ``` The exporter holds its samples in memory, so a restart empties it. That is why the first sweep after start-up runs every enabled family whatever the state file says: without it, a twelve-hour family would leave its panels reading zero for half a day. ```yaml sinks: prometheus: listen: 0.0.0.0:9605 path: /metrics no_prime: true # leave the first sweep on its ordinary schedule ``` `no_prime: true` switches that priming sweep off, for an account whose quota is tight enough that a full sweep on every restart is not affordable. The price is exactly the behaviour the priming exists to avoid: until each family's cadence comes round, the panels that read it have nothing, and for a twelve-hour family that is half a day of zeros. The stores are unaffected either way, since they keep what was collected before the restart. The priming sweep records as run only the families that were due, and the others run again when they would have without the restart. Recorded all at once, the families of six hours or more would be due together a day later and [take turns](https://docs.jmrp.io/ghchronicle/configuration/cadences/#the-slow-families-take-turns), and the last of them would pass the day after which the exporter drops a series. A series that has not been rewritten for **24 hours** is dropped, so a repository that leaves the sweep stops being reported as if it were still there. The horizon is not configurable. ## Errors at start-up, not in a goroutine ```text prometheus exporter: listen tcp :9605: bind: address already in use ``` Reported when the process starts rather than swallowed in a background goroutine, so a port clash cannot leave you with a running collector and a silently missing exporter. ## In a container The example listens on `127.0.0.1`, which inside a container is the container's own loopback and unreachable from the host. Use `0.0.0.0:9605` there and let the port publication decide who can reach it. ## Nothing to migrate The exporter holds today's values in memory, and a restart of the new release serves them in its shape, so [`-migrate`](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations) has nothing to do here. The Prometheus server keeps the old series until its retention, and they end where the new ones begin, so no query counts an item twice. ## Where to go next - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/) compares Prometheus with the others, and holds the write ledger every one of them shares. - [The dashboards](https://docs.jmrp.io/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # OpenTelemetry Sending GitHub metrics to an OpenTelemetry collector or to Prometheus: OTLP over HTTP with the JSON encoding, and why raw is false by default. Source: https://docs.jmrp.io/ghchronicle/sinks/otlp/ The OTLP sink pushes ghchronicle's metrics to an OpenTelemetry collector, or straight into Prometheus through its OTLP receiver. ```yaml sinks: otlp: endpoint: http://collector:4318/v1/metrics service: ghchronicle headers: Authorization: Bearer ${OTLP_TOKEN} raw: false batch: 2000 ``` ## The wire contract OTLP over HTTP with the **JSON encoding**, which every receiver worth using accepts on the same endpoint. That is a deliberate trade: JSON is larger on the wire, and it keeps a code generator, a protobuf runtime and their transitive dependencies out of the binary. Every sink but PostgreSQL, which needs its driver, speaks its protocol with the standard library alone. `endpoint` is the full URL of the metrics path, not a base. `headers` go with every request, which is where an API key or a tenant id belongs. `service` becomes the resource attribute the backend groups by. `batch` is how many data points go in one request, 2000 unless it is lowered for a receiver with a smaller body limit. Metric names use dots, following the OpenTelemetry convention: `github.workflow.run.duration.seconds`. A receiver exporting onward to Prometheus converts them itself; going the other way it cannot. ## `raw` - **raw: false (default)** Sends the same reduced current values as the Prometheus exporter, made by [the same reduction](https://docs.jmrp.io/ghchronicle/sinks/prometheus/#the-reduction), so each mean is taken over the items that carried the field. Safe with any backend, including Prometheus's own OTLP receiver, because nothing in the payload is older than the sweep. - **raw: true** Sends the dated points. Whether they survive is entirely the backend's decision: OTLP data points carry an explicit timestamp, so a store that accepts old ones keeps the history, and one that does not rejects them. > **Prometheus's OTLP receiver is one that does not** > > Measured: against Prometheus 3.14 with `--web.enable-otlp-receiver` and > `out_of_order_time_window: 30m`, a sample dated two days back comes back as > HTTP 400. With `raw: true` pointed at Prometheus, roughly half of every sweep > is refused. Use `raw: false` there, or send to a store that keeps the dates. ## `repeat` A gauge asserted once and then never again fades from the dashboards, and a family that runs every twelve hours would be a flat line with a dot in it. With `repeat` set, the sink keeps asserting the newest value of every series it has seen until the next sweep replaces it. ```yaml sinks: otlp: endpoint: http://collector:4318/v1/metrics repeat: 1m ``` This matters for a backend that answers an instant query from the last sample within a lookback window. ## A collector in front The usual arrangement is a collector that receives from here and exports onward, which is what makes this sink worth having at all: the tool speaks one protocol and the pipeline decides where the data ends up. ```yaml receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 exporters: prometheusremotewrite: endpoint: http://prometheus:9090/api/v1/write service: pipelines: metrics: receivers: [otlp] exporters: [prometheusremotewrite] ``` With that pipeline, keep `raw: false`: the exporter at the far end is Prometheus, and the constraint follows the data rather than the protocol. ## Nothing to migrate The sink pushes today's values, in the shape of the release that runs it, so [`-migrate`](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations) has nothing to do here. What the receiver keeps of an old series ends where the new one begins. ## Where to go next - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/) compares OpenTelemetry with the others, and holds the write ledger every one of them shares. - [The dashboards](https://docs.jmrp.io/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # PostgreSQL The two ways into PostgreSQL, INSERT statements you pipe into psql and a sink that connects, the schema both declare, and why the conflict clause updates rather than does nothing. Source: https://docs.jmrp.io/ghchronicle/sinks/postgres/ Two sinks put ghchronicle's dated points into PostgreSQL: one writes SQL to a file for a later load, the other connects and inserts. ```yaml sinks: sql: dialect: postgres path: /var/lib/ghchronicle/points.sql max_bytes: 67108864 keep: 5 ``` INSERT statements in PostgreSQL's dialect, written to a rotating file or, with `path: "-"`, to standard output. ```sh ghchronicle -config config.yaml -once | psql "$DATABASE_URL" ``` This is one of two ways into PostgreSQL, and the other one connects: ```yaml sinks: postgres: dsn: ${DATABASE_URL} batch: 1000 ``` `sinks.postgres` writes the same tables into a database that is running, in one round trip per batch, with the values sent as parameters rather than rendered into the statement. Nothing else about it differs: the schema below is the schema both write, deliberately, because the dashboards this project publishes query those tables and a second shape would make one of the two a lie. Which one you want is a question about when the load happens. The file is the answer when the database is somewhere this cannot reach, when the statements are meant to be read or held before they run, or when the reader is not PostgreSQL at all: they are ordinary SQL and another engine can take them. The connection is the answer when the database is right there, and it saves you the timer that pipes a file into psql. There is one more difference, and it is in Grafana rather than here. The connecting sink knows the server because it dials it, so `-publish-dashboard` can build the datasource out of the DSN; the file sink cannot, because it never connects and no host, port or user exists anywhere in its config. ## The schema is the contract - One table per measurement, named after it: `gh_repo`, `gh_traffic`, `gh_workflow_run`. - `time TIMESTAMPTZ NOT NULL`, the date the thing happened. - One `TEXT NOT NULL DEFAULT ''` column per tag, an empty string where the point had no value. - One column per field, typed from the value: `BIGINT` for an integer, `DOUBLE PRECISION` for a float, `BOOLEAN`, `TEXT`, and `TIMESTAMPTZ` for a field that is itself a time. - `PRIMARY KEY (time, )`. - Every identifier is double-quoted, because `user`, `type` and `state` are tag names here and reserved words there. That primary key is InfluxDB's series key spelled as a constraint, and it is what makes a rewrite of the fourteen-day traffic window converge instead of accumulate. ```sql CREATE TABLE IF NOT EXISTS "gh_traffic" ("time" TIMESTAMPTZ NOT NULL, "full_name" TEXT NOT NULL DEFAULT '', "kind" TEXT NOT NULL DEFAULT '', "owner" TEXT NOT NULL DEFAULT '', "repo" TEXT NOT NULL DEFAULT '', PRIMARY KEY ("time", "full_name", "kind", "owner", "repo")); ALTER TABLE "gh_traffic" ADD COLUMN IF NOT EXISTS "count" BIGINT; ALTER TABLE "gh_traffic" ADD COLUMN IF NOT EXISTS "uniques" BIGINT; ALTER TABLE "gh_traffic" ADD COLUMN IF NOT EXISTS "url" TEXT; INSERT INTO "gh_traffic" ("time", "full_name", "kind", "owner", "repo", "count", "uniques", "url") VALUES ('2026-09-07T00:00:00Z'::timestamptz, 'acme/telemetry', 'views', 'acme', 'telemetry', 41, 12, 'https://github.com/acme/telemetry/graphs/traffic') ON CONFLICT ("time", "full_name", "kind", "owner", "repo") DO UPDATE SET "count" = EXCLUDED."count", "uniques" = EXCLUDED."uniques", "url" = EXCLUDED."url"; ``` > **DO UPDATE, not DO NOTHING** > > Today's traffic row is rewritten with a higher count on every sweep. A row > frozen at its first value would be the one bug the whole dated-point design > exists to avoid. ## How the declarations arrive The `CREATE TABLE IF NOT EXISTS` is emitted the first time a measurement is seen in a file, or by a connecting sink's process, with the time and the tags the key is made of. Each field of the union that batch carries follows it as `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`, and so does a field that turns up in a later batch. The fields are never in the `CREATE TABLE`, because that statement does nothing to a table an earlier release made: a field the release did not write would reach the `INSERT` with no column, and PostgreSQL refuses the statement and the batch around it. Adding a field's column if it is missing means the same thing to a new table and an old one. A rotated file starts its declarations again, so any one file can be replayed on its own. So a file replayed into a database that already holds its tables makes psql print a notice for every declaration that finds its work done, one for the `CREATE TABLE` and one for each column the table already has: ```text NOTICE: relation "gh_traffic" already exists, skipping NOTICE: column "count" of relation "gh_traffic" already exists, skipping ``` Those are expected. An `ERROR` line is not. The connecting sink does not send every one of those `ALTER TABLE`s. It asks the catalog which columns a table already has the first time its process meets the table, and adds only the ones it lacks. PostgreSQL takes an `ALTER TABLE`'s exclusive lock before it checks `IF NOT EXISTS`, so one per field on every restart waited for each Grafana query reading the table and held up every query after it. **Measured against PostgreSQL 18.6** with a reader holding a table open: `CREATE TABLE IF NOT EXISTS` returned in under a millisecond, while an `ADD COLUMN IF NOT EXISTS` for a column already there waited until a one second `lock_timeout` refused it, and a `SELECT` behind it waited three seconds. A statement the server refuses is sent again on the next write rather than taken as done. A tag is different, because it is part of the key. One first seen after the table was declared, in the same file or the same process, cannot join the primary key without rewriting it, so it becomes a plain column. That only happens when a collector changes its tag set between sweeps. A table an earlier release made is another matter, and a release that changes a measurement's tags changes what loads into it. **Measured against PostgreSQL 18.6** on 2026-09-27: - A tag a later release adds gets no column from either sink, so every `INSERT` of that measurement is refused with `column ... does not exist`, and in the connecting sink the other rows of the same batch with it. - A tag a later release stops writing, as 2.6.1 stopped writing `is_answer` on `gh_discussion_comment`, leaves the file's `ON CONFLICT` naming a key the table does not have. psql refuses each of those statements with "there is no unique or exclusion constraint matching the ON CONFLICT specification", and the rest of the file loads. The connecting sink reads the table's own key from the catalog and conflicts on that, so it goes on writing, and its new rows sit beside the old ones with the empty string in the column no longer written, the two shapes InfluxDB holds as well. The clean way out of either is to drop the table and let the sweeps, and a backfill for the history, fill it again, which is what `-migrate -yes` does for the changes the binary knows of: see below. A collector with the connecting sink writes on across the drop. Its first write after it is refused with `relation ... does not exist`, since the sink declares a table once per process; the sink then forgets the tables of that batch, declares them again and sends the batch once more, which creates the table afresh. **Measured against PostgreSQL 18.6**: before this, every later write of the table and the rest of its batch were refused until a restart. [How to read the tables](https://docs.jmrp.io/ghchronicle/collectors/measurements/#how-to-read-the-tables) shows how to rebuild a key by hand instead. ## What a migration does here When a release changes what a measurement's rows are keyed by, [`-migrate`](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations) asks the connecting sink's database whether any row holds a value in the old tag's column, in the schema the sink writes to, and applying the change renames that one table in that schema, which keeps every row, index and key it had: ```sql ALTER TABLE ""."gh_discussion_comment" RENAME TO "gh_discussion_comment-20261001T091004"; ``` The name is the measurement, a dash and the instant in UTC, so it has to be quoted. The rename takes the same exclusive lock a drop does, which waits behind every Grafana query reading the table, so it is sent under a 5 second `lock_timeout`, three times; a table that stays busy longer leaves the change pending with the lock's reason. The sink's next write creates the table under the old name with the new key, whose index takes the name `gh_discussion_comment_pkey1` beside the copy's. **Measured against PostgreSQL 18.6**: the rename took 7 ms, touched no other schema, and a sink that had written the table before it wrote on after it. ghchronicle drops the copy once it has been kept 24 hours: after a sweep of the service, at the next start of any run, or at the next `-migrate -yes`, each with its own `lock_timeout`. `-uninstall data` lists it with the other `gh_` tables. Until then undoing the change is two statements: ```sql DROP TABLE "gh_discussion_comment"; ALTER TABLE "gh_discussion_comment-20261001T091004" RENAME TO "gh_discussion_comment"; ``` The SQL file cannot be asked, so the state file's record of the release that first wrote it decides, and applying the change writes the drop into the file, after everything already there: ```sql DROP TABLE IF EXISTS "gh_discussion_comment"; ``` The sink then forgets the table, so the next rows of that measurement declare it again after the drop. Replayed in order, the database loses the table in the old shape and gains it in the new one. **Measured with psql against PostgreSQL 18.6**: without the drop, the new rows are refused against the old key; with it, the file loads whole. The drop reaches whatever the file is replayed into, where nothing is kept aside, so a start never applies it on its own; `-migrate -yes` does. A file replayed on its own, without the one that holds the drop, still meets the old table, and a rotation that deletes that file before it was replayed takes the drop with it. With `path: "-"` the drop and the rows read again go to standard output, and `-migrate -yes` prints its plan and its report on standard error, so standard output is the SQL alone and takes the same pipe as `-once`: ```sh ghchronicle -config config.yaml -migrate -yes | psql "$DATABASE_URL" ``` **Measured with psql against PostgreSQL 18.6**: piped this way, the table 2.6.0 made was dropped and made again with the new key, holding the comments read again. Before, the plan went first on the same stream, psql read its first line as the start of a statement and lost the drop with it, and the target kept the old table while the state file recorded the change as applied. Run without the pipe, the SQL goes to the terminal and nowhere else, and the change is still recorded as applied. ## TimescaleDB Turn each table into a hypertable once it exists. The primary key already includes `time`, which is the one condition TimescaleDB puts on it. ```sql SELECT create_hypertable('gh_traffic', 'time', if_not_exists => TRUE); SELECT create_hypertable('gh_workflow_run', 'time', if_not_exists => TRUE); ``` Nothing in the dashboard changes. ## Setting it up 1. Choose the sink. The one that connects needs `sinks.postgres.dsn` and a database this machine can reach, and declares its tables on its first write. The file sink needs `sinks.sql.path`, a file or `-` for standard output. 2. With the file sink, load what it wrote. The connecting sink has nothing to load. ```sh psql "$DATABASE_URL" -f /var/lib/ghchronicle/points.sql ``` Or, for the streaming arrangement, run the collector with `-once` from a scheduler and pipe it straight in. 3. Point Grafana's PostgreSQL datasource at the database and import `ghchronicle-postgres.json`, or, with the connecting sink, let `-publish-dashboard` make the datasource out of the DSN and publish the dashboard. ```sql SELECT time, "count" FROM gh_traffic WHERE kind = 'views' AND repo = $repo ``` `dashboards/ghchronicle-postgres.json` has the same 154 panels as the InfluxDB one, with every query translated to PostgreSQL against this schema. The translation compares text by its bytes, as InfluxDB does: every text column a query sorts by, or takes the least or the greatest of, carries `COLLATE "C"`. Left to the database, the order is its collation's, and on a PostgreSQL built on glibc whose database was created under a locale such as `en_US.UTF-8` that one sets case and punctuation aside on its first pass. **Measured against PostgreSQL 18.6 on Debian**, whose database is created as `en_US.utf8`: before this, "(ghost)" sorted after "alice" and "VALID" after "unsigned", and three panels listed their rows in another order than the InfluxDB dashboard. The Alpine image agreed only because musl compares bytes whatever the locale is called. Nothing changes in the database: its columns keep the collation they were created with, and only the dashboard's queries say how to compare. ## Where to go next - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/) compares PostgreSQL with the others, and holds the write ledger every one of them shares. - [The dashboards](https://docs.jmrp.io/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Graphite Sending GitHub metrics to Graphite: the plaintext protocol over TCP, and the exact metric path, which is the dashboard's contract. Source: https://docs.jmrp.io/ghchronicle/sinks/graphite/ The Graphite sink sends ghchronicle's numeric fields to Graphite over TCP, each at the date the thing happened. ```yaml sinks: graphite: addr: graphite:2003 prefix: github batch: 1000 ``` The plaintext protocol over TCP: `path value timestamp`, one line per numeric field. String fields are skipped, because Graphite has no way to hold one. Graphite keeps a point at the time it was given, so the raw dated points go out as they are and the traffic window lands on its own days. Writing the same point twice fills the same slot of the same whisper file, which is exactly what a rewrite of the window wants. The connection is reopened on a failed write, once, before the write is reported as failed. ## The path is the dashboard's contract ```text ... ``` - Every tag is one node, holding its value. The nodes are ordered by the tag's **key**, alphabetically, never by the order a collector set them. - An empty tag value is written as `none`, so a measurement's depth never changes from one point to the next and `github.repo.*.*.*.*.*.*.*.*.*.stars` keeps matching. - A node keeps ASCII letters, digits, `_`, `-` and `:`. Everything else becomes `_`: the dot, the space, the comma, and the slash in `owner/repo`. A dot would split the node and a slash would nest a directory. So a `gh_repo` point tagged `archived=false default_branch=main fork=false full_name=acme/edge-cache language=Go license=MIT owner=acme repo=edge-cache visibility=public` with a `stars` field becomes: ```text github.repo.false.main.false.acme_edge-cache.Go.MIT.acme.edge-cache.public.stars 37 1757280000 ``` The dashboard names a series or a row by the node it is grouped under, so it shows a name as the path holds it: the `(ghost)` the collector writes for a deleted account reads `_ghost_`, the `Actions Linux` SKU `Actions_Linux`, the `Q&A` category `Q_A`, `another/project` `another_project`, and a cache key holding `go-1.27.1` holds `go-1_27_1`. Nothing turns them back on the way out, since an underscore in a node may have been one in the name as well, so the Graphite dashboard leaves them as they are, and the other four stores show each name as the collector wrote it. A field that shares its name with a tag is skipped, the same rule the line protocol applies: the tag wins, because it is the one that can be grouped by. > **A new tag changes the path depth** > > Adding a tag to a collector inserts a node into every path of that > measurement, so every existing dashboard target for it stops matching. This is > the cost of a store with no schema, and it is why `TAGS` in the dashboard > specification mirrors the tag set of each measurement. ## What it cannot answer Graphite keeps the dated points but has no rows. A series is a path and a number, so: - A table that needs several fields of one row cannot be built. The shipped dashboard reduces each series to the one or two numbers a row of it can carry, and its panel description names those and every column it drops. - A title or any other string is not a metric there at all, and the sink drops it; the panels that would show one say so. A boolean is kept as 1 or 0 like any number. - A panel cannot join two measurements, since each lives under its own path. Work elsewhere has no Stars column, and the community profile keeps the API's issue template flag rather than the count of templates, which lives under another measurement. - A target cannot ask about one window while it reads another. The Overview and Every repository, ever count an archived repository the default filter sets aside from its points of the last seven days, where the SQL stores ask whether the collector still writes it, so a range that ended more than a week ago leaves those repositories out. Everything time-shaped works normally, which is most of the dashboard. ## The dashboard `dashboards/ghchronicle-graphite.json` has the same 154 panels as the InfluxDB one, written against these paths with the default prefix. It needs **Graphite 1.1 or later** for the functions it uses, and any panel a series cannot carry says so in its description. Change `prefix` and the dashboard targets have to change with it, since the prefix is the first node of every path. Every chart over time adds its points up into buckets the width the range calls for, the width the SQL dashboards bin by: the range over a hundred, rounded the way Grafana rounds an interval and never under the chart's floor of a day, an hour or five minutes. The dashboard computes them in three hidden variables, `bucket_1d`, `bucket_1h` and `bucket_5m`, and each chart asks for 5,000 points, more than it has buckets, so graphite-web hands the buckets back as they are. Asked for fewer points than a series holds, graphite-web fits the series into bands and moves each point one step later as it does, and before 2.6.2 that put the newest hour of a chart past its right edge in the last hour before each band boundary. Measured against `graphiteapp/graphite-statsd:1.1.10-5` with one hour a step at 15:35 UTC, the star count a sweep had written in that hour came back stamped 16:00 at a hundred points, after the end of the range, and a chart whose one point was that one read "Data outside time range"; summed into the day's bucket it came back at 00:00, and the artifact storage in its six-hour bucket at 12:00. A bucket follows the dashboard's range, so a chart pinned to its own range, as two of the Code panels are, sums into days whatever the page is set to. Two charts sum into a fixed day instead, because their rows are already a day or a week apart and each belongs at its own date: the contribution calendar, a row a day, and the weekly commits, a row a week stamped at the Sunday GitHub starts the week on, with the days between the weeks left out. Seven days is not a bucket there: graphite-web counts one from the epoch, a Thursday, and every weekly bar stood three days early, the first week of a thirty-day range before the range began. Every other panel asks for 5,000 points as well. A table, a bar chart or a stat is not summarized: it reduces the points of the whole range to a number, and the fitting drops the first of them, one fewer than the steps it moves the first band's start by. Grafana asks a panel that names no number for as many points as it is wide, so before 2.6.2 a panel narrower than a range's points lost the first hour or hours of the range, depending on where the range began. Measured on the same Graphite, a point at 20:00 UTC read from 19:05 over thirty days summed to nothing at 500 points and to 1 at 5,000, and "Languages starred" drew a star given at that hour as 0 in a window 640 pixels wide. Such a panel holds a point per storage step, so 5,000 covers the 2,880 hours of the hundred and twenty days the containerised suite's schema keeps at an hour a step and the 4,380 days of the twelve years it keeps at a day. A retention that keeps more than 5,000 steps of a range, an hour a step for longer than a hundred and twenty days or anything finer, is fitted into bands again. A chart or a bar chart that names its busiest series and folds the rest into one called other does the same here as in the SQL dashboards: the busiest over the whole range are named, and other is every series less those, point by point, drawn only where something is left over. ## What a migration does here Graphite cannot be cleared from here. Nothing on carbon's ingest port deletes a path, and graphite-web deletes none that has no tags: **measured against graphite-statsd 1.1.10-5**, `/metrics/delete` answers 404 and `/tags/delSeries` answers `true` and leaves the series where it was. What a release left in another shape is decided by the state file's record of the release that first wrote the store, and applying the change is `-migrate -yes` printing, once, the commands for the Graphite host, and recording the change as applied, since whether they were run is something only whoever runs them knows: ```sh find /whisper/github/discussion_comment -mindepth 11 -name '*.wsp' -delete find /whisper/github/discussion_comment -type d -empty -delete ``` `` is carbon's storage directory, `/opt/graphite/storage` in the official image, and `github` is the prefix. A path is a node per tag and then the field, so the old shape of a measurement that lost a tag sits one level deeper than the new one, and `-mindepth` reaches that level and nothing above it. Measured in that image: the command took the two files of the old shape and left the new shape's and every other measurement's. The files are the host's and the order does not matter, since the two shapes are different files. A start never applies this on its own. ## Where to go next - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/) compares Graphite with the others, and holds the write ledger every one of them shares. - [The dashboards](https://docs.jmrp.io/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Elasticsearch The bulk API, one index per measurement, and a document id that makes a rewrite replace rather than duplicate. Source: https://docs.jmrp.io/ghchronicle/sinks/elasticsearch/ The Elasticsearch sink sends ghchronicle's dated points to Elasticsearch or OpenSearch in bulk, one document per point. ```yaml sinks: elasticsearch: url: http://elasticsearch:9200 prefix: ghchronicle api_key: ${ES_API_KEY} # or username and password batch: 1000 ``` The `_bulk` API, which Elasticsearch and OpenSearch share, so the same sink serves both, and Kibana or the OpenSearch dashboards on top of either. ## Credentials - **API key** ```yaml api_key: ${ES_API_KEY} ``` Sent as `Authorization: ApiKey`. - **Basic** ```yaml username: ghchronicle password: ${ES_PASSWORD} ``` Sent as basic auth. Not both. Configuring an API key alongside a username fails validation at start-up with `sinks.elasticsearch: set either api_key or username and password, not both`. ## The documents One index per measurement, named `-`: `ghchronicle-gh_repo`, `ghchronicle-gh_traffic`. One document per point, with the time as `@timestamp` in RFC 3339, `measurement`, and every tag and field as a top-level key, so nothing has to be unnested before it can be filtered on. ```json { "@timestamp": "2026-09-07T00:00:00Z", "measurement": "gh_traffic", "owner": "acme", "repo": "telemetry", "full_name": "acme/telemetry", "kind": "views", "count": 220, "uniques": 131 } ``` ## Why re-collection converges here too The document id is the **SHA-256 of the measurement, the tags that are set and the timestamp**, and the action is `index` rather than `create`. So writing the same fourteen-day traffic window every six hours replaces fourteen documents instead of adding fourteen more, which is the same convergence InfluxDB gives for free. > **A bulk request answers 200 even when items failed** > > Elasticsearch reports per-item verdicts inside a successful response. The sink > reads them, logs each refused document with the cluster's own reason, and > reports the count as a warning rather than a failure, because everything else > in the batch was written. ## No mapping is written The sink creates no index template. Dynamic mapping gives every string field a `.keyword` sub-field, which is what the dashboard aggregates on. If you want explicit mappings, create the index templates before the first write. Nothing in the sink depends on them; only the panels' choice of `.keyword` does. [`-migrate`](https://docs.jmrp.io/ghchronicle/sinks/elasticsearch/#what-a-migration-does-here) reads the mapping before it asks whose rows an index holds, so a template that maps strings as `keyword`, with no `.keyword` sub-field, is read from the field itself. ## The dashboard `dashboards/ghchronicle-elasticsearch.json` has the same 154 panels as the InfluxDB one, as Lucene filters and aggregations over **one** datasource pointing at `-*`, because each target names its own index in its query. Set the datasource's time field to `@timestamp`. A per-item table there is the newest documents themselves, or the newest document of each item where an item has several; everything else is a bucket aggregation. OpenSearch works through the same plugin. Two things a query cannot do there, and the panels that need them say so. It cannot join two indices, so Work elsewhere has no Stars column and the community profile keeps the API's issue template flag rather than the count of templates. And it cannot ask about one window while it reads another: the Overview and Every repository, ever count an archived repository the default filter sets aside from its documents of the last seven days, where the SQL stores ask whether the collector still writes it, so a range that ended more than a week ago leaves those repositories out. ## What a migration does here When a release changes what a measurement's documents are keyed by, [`-migrate`](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations) finds the documents of the old shape by counting the ones that carry the old tag, and applying the change sets the index aside in three steps. **Measured against Elasticsearch 9.5.3**: 1. The index stops taking writes, through `PUT /_block/write`. 2. It is cloned to `-`, the instant in UTC and in lower case, as an index name has to be, for example `ghchronicle-gh_discussion_comment-20261001t091004`, and the clone has to hold as many documents as the index before anything else happens. 3. The index is deleted, and the sink's next write creates it again, with a mapping of its own. Before the change is applied, the plan asks whose rows the index holds and which repositories they belong to, with a terms aggregation on whatever field the mapping makes aggregatable: the `.keyword` sub-field dynamic mapping gives a string, or the field itself where a template maps it as `keyword`. The buckets have to add up to every document that holds the field, or the answer is not taken: measured on 9.5.3, an aggregation on `user.keyword` over an index whose template mapped `user` as `keyword` answered no buckets and no error, which read as an index holding nobody else's rows, and a value past the sub-field's `ignore_above` is in no bucket either. A document without the field is a row that names nobody, which the plan says as such. A step that fails before the delete takes the write block off again, so a cluster that refused the clone goes on taking the sink's writes, and the change stays pending with the cluster's reason. A delete that fails is looked at again before anything is undone: measured through a proxy that answered 502 to a delete the cluster carried out, the index was gone, so the clone, the one copy of its documents, is kept and the change recorded as applied; only an index still there has its block taken off and its clone deleted. The API key needs the `manage` and `delete_index` privileges on the prefix's indices; a key that can only write is refused that way. The clone keeps the write block, and ghchronicle deletes it once it has been kept 24 hours: after a sweep of the service, at the next start of any run, or at the next `-migrate -yes`. Until then undoing the change is deleting the new index and cloning the copy back under its name. Every query of the shipped dashboard names its index whole, `_index:ghchronicle-gh_discussion_comment`, so no panel reads the clone: a panel's query over `ghchronicle-*` counted the one new document and not the clone's three. A Kibana data view over the same pattern without that filter does read it, for the day it is kept. ## Where to go next - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/) compares Elasticsearch with the others, and holds the write ledger every one of them shares. - [The dashboards](https://docs.jmrp.io/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Loki The twenty-two measurements that are events rather than numbers, and the age horizon that keeps a push from being refused. Source: https://docs.jmrp.io/ghchronicle/sinks/loki/ The Loki sink sends the GitHub activity that is an event rather than a number (a star, a release, a failed run) to Grafana Loki as log lines. ```yaml sinks: loki: url: http://loki:3100/loki/api/v1/push tenant_id: "" labels: job: ghchronicle max_age: 1h batch: 1000 ``` ## Measurements and events Some of what GitHub reports is a measurement and some of it is an event. "The repository has 148 stars" is a measurement. "Someone starred it at 03:03, this release was published, that workflow failed on main, this alert was raised" are events: each happened once, at a known moment, and what you want later is to read them in order and search them, not to average them. **Twenty-two measurements have an event rendering**: stars in both directions, forks, releases, published package versions, pull requests, reviews, review threads, issues, commits, workflow runs, job logs, repository activity, Dependabot alerts, code scanning analyses, the event feed, notifications, discussions, webhook deliveries, deployments, ruleset versions and external contributions. Everything else is a gauge in disguise and is not sent. A measurement with no rendering is dropped in silence, which is right for a gauge and wrong for an event nobody has got round to: deployments and review threads were dated events with no log line for months, and nothing said so. So every dated measurement now has to appear in one of two tables in `internal/sink/loki.go`, the renderings or the refusals, and each refusal carries the reason it is not a log line. A test fails on a dated measurement that appears in neither. It is not dropped in silence in the sweep log any more. This sink writes a fraction of what it is offered, so most of its lines carry [`filtered`](https://docs.jmrp.io/ghchronicle/sinks/#what-the-written-line-counts) and `points=0`, which is what it stored rather than what it was given. An entry older than `max_age`, or too far behind the newest entry of its own stream for Loki to accept, is counted there too and reported as a dropped-entries line. ## The line format Each line reads as a sentence first and carries every tag and field after it in logfmt, so the same line is greppable in a terminal and queryable in Grafana without keeping two copies of the data. ```text someone starred acme/telemetry full_name="acme/telemetry" user="someone" starred=1 ``` Two renderings changed in 2.6.0, and one of them again in 2.6.1. A line already in Loki keeps the text it was sent with: - An external contribution is a line when the item closes, at the moment it closed, and says what happened to the item and not who did it, since the row does not say who merged or closed it: `USER's pull request OWNER/REPO#N was merged` or `USER's pull request OWNER/REPO#N was closed without merging`, and `USER's issue OWNER/REPO#N was closed`. A closed row whose state and merged flag disagree reads `USER's contribution OWNER/REPO#N`. An item still open is no line at all from 2.6.1 on: its row is stamped at the start of every day it is seen open, which is a reading and not something that happened, and opening it is already a line of the event feed, with `action="opened"`. 2.6.0 sent that row as `... is open`, and only on a day an `outbound` pass wrote in the first hour of the UTC day. Every line used to say `USER merged OWNER/REPO#N`, open issues included. - A release reads `published release TAG of OWNER/REPO`, or `published prerelease TAG of OWNER/REPO`, at the moment it was published. It used to read `release TAG of OWNER/REPO, N downloads`, so a query filtering on `downloads` matches only the old lines. `batch` is how many entries go in one push, 1000 unless it is lowered. The stream label is `kind`, which is what you filter on first. ```text {job="ghchronicle", kind="workflow_run"} |= "failure" {job="ghchronicle", kind="job_log"} ``` > **Keep the other labels few** > > Loki indexes labels, and a high cardinality label costs far more than a wide > line. `labels` in the configuration is for the fixed ones that identify this > collector, not for anything that varies per point. ## `max_age`, and why the reason is not the obvious one Loki refuses an entire push when one entry predates `reject_old_samples_max_age`, a week by default, and half of what this collector produces is older than that on purpose: a star from 2020, a pull request from 2024. But the limit that actually bites is the other one. Loki also refuses an entry more than its out-of-order window behind the newest entry already in that stream, which is half of the ingester's `max_chunk_age`, one hour by default. **Measured against a real Loki 3**: once the stream held an entry from 19:14, one from 00:35 the same day came back as "entry too far behind". Against Loki 3.7.7 with its default limits, a stream holding an entry five minutes old took one 55 minutes old and refused one 75 minutes old. So the horizon is applied two ways: 1. against the wall clock, 2. against the newest entry that stream has been sent before. Inside one push nothing is behind, because the sink sends each stream oldest first and Loki judges each entry against the newest one before it: one push of entries from 23 hours, 12 hours and a minute ago into an empty stream was taken whole. What falls outside is left out and counted, at debug level, rather than costing the whole push. `max_age` defaults to one hour, which is Loki's default window. Raise it only if you have raised Loki's `max_chunk_age` to match, since the window is half of it. A release is the event that shows what the horizon costs, and one of the two streams that look further back. Its line is rendered from `gh_release_published`, at the moment the release was published. It used to come from `gh_release`, which is stamped at the sweep because its downloads move, so every repository pass pushed every release again. Measured on 2.5.1 in production, over 30.9 hours and 27 `repo` passes, that was 4,313 of the 10,467 lines the sink sent, 41 per cent, for the 2 releases published in those hours. Dated at the publication, a release is first seen by the `repo` pass after it, so one published just after a pass read its repository is a whole cadence old when the next pass writes, plus however late that pass runs: a tick it lost, the slower families that ran before it, a restart. With `repo` and `max_age` both at their default hour, `max_age` alone left such a release out, and every later pass only saw it older. So the release stream looks back the `repo` cadence plus `max_age`: two hours at the defaults, seven with `repo: 6h`. An external contribution is the other stream that does, for the same reason: its line is dated when the item closed, and the first pass to see it is the `outbound` pass after that, so the stream looks back the `outbound` cadence plus `max_age`, two hours at the defaults. Before 2.6.1 it had `max_age` alone, and a closing was sent only when the next pass wrote within the hour of it, which an hourly pass does not do for an item closed in the seconds after the last one read the searches, nor, when it runs late, for one closed in the minutes it is late by. Each lookback is capped at six days, a day short of the week `reject_old_samples_max_age` allows, and it never shortens `max_age`: one set past six days is these streams' horizon as it is every other's. Loki refuses an old line only for being behind a newer one in its stream, which the second check above still makes. A release published in the hour before a pass is sent by that pass and again by the next, the same line at the same instant, which Loki keeps once. A contribution is sent twice the same way, and its line carries the item's comment count and title, which can move in between: measured against Loki 3.7.7, a line sent again with `comments` moved from 1 to 2 was kept as a second line at the same instant. Under `-once` the cadence that matters is the schedule that runs the binary, so set `every.families.repo` and `every.families.outbound` to it. Both are in the metrics store either way. ## What Loki is not for The dated history. That is what a metrics store is for, and it is why the two run together rather than one replacing the other. A log answers "what happened recently, in order"; a time series answers "how much, over which period". ## Job logs belong here The `joblogs` family, off until `every.families.joblogs` gives it a cadence, collects the last forty lines of every failed GitHub Actions job. It is text rather than a measurement, so the InfluxDB sink excludes it by default and the Prometheus exporter skips it. Loki is where it belongs, and the query is: ```text {job="ghchronicle", kind="job_log"} ``` The exported dashboards do not show it, because a dashboard bound to one datasource cannot query two and an importer may have no Loki: they carry a text panel, "Where failure output went", with that query. Publishing the dashboard from the collector swaps that panel for the lines, newest first, filtered by the dashboard's repository variable where the store's variable can be read as a regular expression. A Loki sink whose address ends in `/loki/api/v1/push` is enough: a Loki datasource Grafana already has at that address, with the path dropped, is adopted, and otherwise the datasource is made from it. One writing anywhere else says so and takes a `grafana.datasource.loki_uid` instead, and so does a Grafana that reaches Loki by another address. A token that may not make the datasource costs this panel and nothing else: the run warns once and publishes every dashboard with the note. ## Nothing to migrate A stream is its job, its kind and the configured labels, and a point's tags go into the line, so a release that moves a tag changes no stream, and [`-migrate`](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations) has nothing to do here. A line is what was said when it was said, and Loki refuses entries older than its window anyway. ## Where to go next - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/) compares Loki with the others, and holds the write ledger every one of them shares. - [The dashboards](https://docs.jmrp.io/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Telegraf Sending GitHub metrics through Telegraf: line protocol posted to http_listener_v2, and the reason this is one sink instead of a hundred. Source: https://docs.jmrp.io/ghchronicle/sinks/telegraf/ The Telegraf sink posts ghchronicle's points to Telegraf, which can forward them to any of the outputs Telegraf supports. ```yaml sinks: telegraf: url: http://telegraf:8186/telegraf username: "" password: "" batch: 5000 ``` Line protocol posted to Telegraf's `http_listener_v2` input, as `text/plain`, with basic auth when a username is set. > **A URL with no path gets /telegraf** > > The listener answers 404 to `/` without saying why, so a URL with an empty > path is given the input's default path rather than being sent somewhere that > will silently fail. ## The matching input ```toml [[inputs.http_listener_v2]] service_address = ":8186" paths = ["/telegraf"] data_format = "influx" ``` ## Why this exists as one sink Telegraf has an output for Kafka, Datadog, New Relic, Graphite, Loki, Elasticsearch, Wavefront, Azure Monitor, Google Cloud Monitoring and a hundred more. Rather than grow a sink here for each of them, the points go to Telegraf as the line protocol it already speaks, and its own `[[outputs.*]]` blocks route them onward. That is the whole argument. It is one sink that reaches everything Telegraf reaches, and it costs this project no new dependency and no new wire format. ## What happens to the dates Telegraf keeps the timestamps as given, so the dated history survives as far as each output lets it. An output that stamps at receipt, or one that rejects old samples, loses it, exactly as those backends would if this tool wrote to them directly. So the question to ask is not "does Telegraf keep the history" but "does the output I configured keep it". Kafka and InfluxDB do. A Prometheus remote write does not. ## An example that fans out ```toml [[inputs.http_listener_v2]] service_address = ":8186" paths = ["/telegraf"] data_format = "influx" [[outputs.influxdb_v2]] urls = ["http://influxdb:8181"] token = "$INFLUX_TOKEN" organization = "default" bucket = "github" [[outputs.kafka]] brokers = ["kafka:9092"] topic = "github-metrics" ``` One sweep, both destinations, and the collector knows about neither. ## What a migration does here ghchronicle cannot see the store behind Telegraf, so what a release left there in another shape is decided by the state file's record of the release that first wrote it. Applying the change is `-migrate -yes` saying what to do there, dropping the measurement in that store and then reading the families that write it again through Telegraf with a backfill of those families alone, `-backfill -families`, and recording it as applied. A start never applies it on its own, and Telegraf is sent nothing: the refill that follows a migration writes only into stores ghchronicle cleared itself, and a Telegraf that writes to a file would keep every row of it a second time. ## Where to go next - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/) compares Telegraf with the others, and holds the write ledger every one of them shares. - [The dashboards](https://docs.jmrp.io/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # File and stdout A rotating file for a shipper you already run, the simplest durable buffer there is, and line protocol on standard output. Source: https://docs.jmrp.io/ghchronicle/sinks/file/ ## File ```yaml sinks: file: path: /var/log/ghchronicle/points.lp format: influx # or json max_bytes: 67108864 keep: 5 ``` For the setups that already run a shipper. Telegraf tails line protocol, Promtail and Vector tail JSON, and neither needs this process to know anything about their backend. It is also the simplest durable buffer there is: when the database is down, the file still has the data. ### Rotation By size, with numbered suffixes, so retention is a count and two rotations in the same second cannot collide. The size counter is read from the file on start-up, so a restart does not reset it and let the file grow without bound. - **influx** ```text gh_traffic,full_name=acme/telemetry,kind=views,owner=acme,repo=telemetry count=220i,uniques=131i 1788739200000000000 ``` - **json** ```json {"time":"2026-09-07T00:00:00Z","measurement":"gh_traffic","tags":{"full_name":"acme/telemetry","kind":"views","owner":"acme","repo":"telemetry"},"fields":{"count":220,"uniques":131}} ``` One object per line in JSON, which is the shape a line-oriented shipper wants. ## Standard output ```yaml sinks: stdout: true stdout_format: influx # or json ``` Line protocol on standard output, for piping into Telegraf or for seeing what would be written before pointing this at a database. It is the fastest way to answer "what does this actually collect". ```sh ghchronicle -config config.yaml -once | head -20 ghchronicle -config config.yaml -once | grep gh_workflow_run ``` `stdout_format: json` prints one object per line instead, the same shape the file sink writes. > **Do not combine it with the SQL sink on standard output** > > `sinks.sql.path: "-"` also writes to standard output. Interleaving SQL > statements with line protocol produces a stream neither psql nor Telegraf can > read. ## As a buffer The file sink is the answer to "what happens when the database is down". Run it alongside the real store: the write to the database fails and is logged, the sweep continues, and the points are on disk. Replaying them afterwards is a `curl` for InfluxDB, or `psql -f` for the SQL sink, and it converges rather than duplicating because the points carry their own timestamps. ```sh curl -s -XPOST "http://localhost:8181/api/v2/write?org=default&bucket=github&precision=ns" \ -H "Authorization: Token $INFLUX_TOKEN" \ --data-binary @/var/log/ghchronicle/points.lp ``` ## Permissions The dump is created `0600` inside a `0750` directory. A sweep of a private account puts private repository names, Dependabot severities and whole job log lines in it, so it is not created readable by the whole machine just because a shipper is coming to read it. ### Letting the shipper read it Telegraf, Promtail, Vector and Fluent Bit all read as their own user. None of them documents a required mode, and every one of them documents the same remedy, which is a group: their own answers say `usermod -aG adm`. So the grant is yours to make, and it is two commands: ```sh usermod -aG ghchronicle telegraf # the shipper joins the service's group chmod 0640 /var/log/ghchronicle/points.lp # or create it that way beforehand ``` The grant sticks. The sink never changes the mode of a file that is already there, and a rotation creates the replacement with the mode of the file it is renaming away, so the `chmod` is not undone the first time the dump fills up. > **The failure this prevents is a silent one** > > A shipper that cannot open the file it is tailing does not announce it, and > this process keeps writing to a handle it already holds. Neither end logs > anything, so the first symptom is a dashboard that stopped days ago. If you would rather the dump belonged to the shipper's own group, own the directory with it and set the setgid bit, which is what makes every file created in it inherit the group, rotations included: ```sh install -d -o ghchronicle -g telegraf -m 2750 /var/log/ghchronicle ``` That settles which group owns the dump, not what the group may do with it, so the `chmod 0640` above is still the other half. Making the directory yourself is worth doing whichever route you take: the one this process creates when the directory is missing is `0750` less whatever the process umask takes off it, and a shipper that cannot traverse the directory fails as quietly as one that cannot read the file. An ACL is not carried, because it belongs to the file it was set on and a rotation creates a new one, so grant through the group rather than through `setfacl` on the dump. What a rotation does carry is the mode, not the owner: this process cannot `chown` a file to a user it is not. The write ledger behind `sinks.dedupe_file` is the opposite case and gets the opposite answer: nothing but this process is meant to read it, so it is written `0600` and every save puts it back to `0600`. ### Where it can write at all Under the hardened systemd unit, the directory has to be in `ReadWritePaths`. In a container it has to be writable by uid 65532. Both are the same mistake in two clothes: the process is deliberately allowed to write almost nowhere. ## Nothing to migrate The file, and standard output, are a record of what was written when, in the shape of the release that wrote it, and [`-migrate`](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations) leaves them as they are. Replayed into a store, a file brings back every shape it holds, and the store then needs the migration again. ## Where to go next - [Choosing a store](https://docs.jmrp.io/ghchronicle/sinks/) compares the file sink with the others, and holds the write ledger every one of them shares. - [The dashboards](https://docs.jmrp.io/ghchronicle/dashboards/) says which of the five is drawn against which store, and what a panel a store cannot answer becomes. --- # Importing Five generated Grafana dashboards for GitHub metrics, one per store, and how to import each of them. Source: https://docs.jmrp.io/ghchronicle/dashboards/ Five dashboards, all in English, all in Grafana's shareable export format: the datasource is a `${DS_...}` placeholder and the `__inputs` block asks the importer to choose their own. | File | Panels | Store | | -------------------------------- | ------ | ----------------------------------------------------------------------- | | `ghchronicle-influxdb.json` | 154 | InfluxDB 3, queried with SQL | | `ghchronicle-prometheus.json` | 154 | Prometheus | | `ghchronicle-postgres.json` | 154 | PostgreSQL or TimescaleDB, from the sink that connects or the SQL file | | `ghchronicle-graphite.json` | 154 | Graphite, from the Graphite sink | | `ghchronicle-elasticsearch.json` | 154 | Elasticsearch or OpenSearch, from the Elasticsearch sink | The five hold the same panels in the same order. What differs is how many of them the store behind each one can answer. Each cell is the panels that store answers with a query, out of the panels in that section. A panel a store cannot answer ships as a text panel with the same title, so every dashboard has the same 154 panels; the 2 that are prose in all five are left out here. | Section | InfluxDB | PostgreSQL | Elasticsearch | Graphite | Prometheus | Panels | | --- | --- | --- | --- | --- | --- | --- | | Overview | 4 | 4 | 4 | 4 | 4 | 4 | | Lifetime | 5 | 5 | 5 | 5 | 5 | 5 | | Audience | 6 | 6 | 6 | 6 | 5 | 6 | | Stars and forks | 5 | 5 | 5 | 5 | 5 | 5 | | Contributions | 11 | 11 | 10 | 10 | 6 | 11 | | Pull requests and issues | 14 | 14 | 14 | 14 | 10 | 14 | | Continuous integration | 14 | 14 | 13 | 13 | 10 | 14 | | Code | 9 | 9 | 8 | 8 | 8 | 9 | | Planning and community | 10 | 10 | 10 | 10 | 8 | 10 | | Delivery and access | 13 | 13 | 13 | 13 | 13 | 13 | | Releases | 4 | 4 | 3 | 4 | 2 | 4 | | Security | 14 | 14 | 14 | 13 | 13 | 14 | | Cost | 6 | 6 | 6 | 6 | 6 | 6 | | Activity | 9 | 9 | 9 | 8 | 7 | 9 | | Inventory | 16 | 16 | 15 | 15 | 14 | 16 | | Profile and sponsorship | 8 | 8 | 8 | 8 | 8 | 8 | | The collector itself | 4 | 4 | 4 | 4 | 4 | 4 | | **Total** | **152** | **152** | **147** | **146** | **128** | **152** | ![The InfluxDB dashboard over ninety days of the demonstration database: the repository picker and the range across the top, the Overview with the ghchronicle badge and four tile groups reading 5 repositories with 350 stars and 51 forks, 37.5 thousand views with 21.1 thousand unique visitors and 19.6 thousand clones, 117 followers and 58 following with 4 sponsors and 2 sponsored, and 3.22 thousand contributions over 7.78 years, then the collapsed Lifetime header and the Audience section with views, unique visitors and clones per day, the top referrers, the top paths and clone amplification](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboard-influxdb-demo.png) The account in that capture is the invented one every capture in this documentation uses, `acme` and five repositories, described beside [what the panels show](https://docs.jmrp.io/ghchronicle/dashboards/panels/). Every section but the Overview ships collapsed, which is why Lifetime is a header there. ## Letting the binary do it The dashboard is built from the same code the binary carries, so the binary can publish it. Given a Grafana, it reconciles the datasource out of the sink it already writes to, asks that datasource whether it can actually be reached, and publishes the dashboard for every store it writes to. ```yaml grafana: url: http://localhost:3000 token: ${GRAFANA_TOKEN} ``` ```sh ghchronicle -config config.yaml -publish-dashboard ``` It asks GitHub nothing, so it needs no GitHub token, and it prints what it did to each datasource and each dashboard. > **A password from the machine is not copied** > > pgx reads libpq's environment and its password file, which is what the sink > wants, because it is the thing connecting. A datasource is different: it is > written into a Grafana other people can see, so only a password the DSN > itself carries is sent. One that came from `PGPASSWORD` or a `.pgpass` on > whichever machine ran the publish is named in a line and left where it is. > **What the token has to be allowed to do** > > Publishing a dashboard needs an Editor. Creating a datasource needs > `datasources:create` and correcting one needs `datasources:write`, and an > Editor has neither: a token without them gets as far as the dashboard and is > refused on the datasource, saying which permission is missing. So a service > account that manages its own datasource is an Admin, or an Editor granted > both; one that only publishes against a datasource named in > `grafana.datasource.uid` is an Editor, because adopting reads and never > writes. The Loki datasource never stops a publish: with a Loki sink, an > Editor adopts a Loki datasource Grafana already has at the sink's address, > reads a `ghchronicle-loki` an earlier run made as it is, with a warning that > it could not be rewritten, and when there is neither the create is refused, > one warning says so, and the failed job output panel alone is left as its > note. `grafana.datasource.loki_uid` names the one to read and avoids the > question. > **Grafana often reaches the store by another address** > > The address the collector writes to and the address Grafana queries are > frequently not the same one: a collector on the host writes to a published > port while Grafana in a container reaches the same store by its name on the > container network. Copying the sink's address across produces a datasource > Grafana accepts and cannot use, so the run asks it before publishing and > refuses on a datasource that does not answer. `grafana.datasource.url` is the > address to use instead. Three of the five sinks describe their own datasource with nothing else said, because what they write to is what Grafana queries: InfluxDB, Elasticsearch, and the PostgreSQL sink that connects, whose DSN carries the server, the database, the user and, when the DSN itself writes one, the password. Two more need the address and nothing else, because they write somewhere that is not where a query goes: the Prometheus sink is scraped rather than written to, and the Graphite sink speaks the ingest port while Grafana asks the web API on another port. Give those the address and the datasource is made the same way: ```yaml grafana: datasource: url: http://prometheus:9090 ``` What cannot be described at all is the SQL sink, the one that writes statements to a file: it never connects, so no host, port, user or password exists anywhere in its config. That one, and any datasource you would rather manage yourself, is named instead: ```yaml grafana: datasource: uid: ae3x9k2 ``` A datasource named that way is adopted and left exactly as it is, since correcting one this did not make would overwrite settings nobody asked it to have. `publish_on_start: true` does the same once when the collector starts, before the first sweep. It is off by default, and it is what keeps a server from quietly falling behind the binary feeding it: the dashboard is generated from the code, so updating the binary updates the dashboard. A failure there warns and the sweep goes on, because the metrics of an hour spent not running cannot be recovered and a dashboard published on the next restart can. ### What it makes, and under which names Nothing here is assigned by Grafana, so there is nothing to read back out of it and write into your config. Both uids are worked out from the store's name, and the same run twice writes to the same two places: | Store | Dashboard uid | Its datasource | | --------------- | --------------------------- | ------------------------------- | | `influxdb` | `ghchronicle-influxdb` | made, from the sink | | `elasticsearch` | `ghchronicle-elasticsearch` | made, from the sink | | `postgres` | `ghchronicle-postgres` | made, from the dsn | | `prometheus` | `ghchronicle-prometheus` | made, once you give the address | | `graphite` | `ghchronicle-graphite` | made, once you give the address | A datasource this makes takes the dashboard's uid, so both are `ghchronicle-`, and one you name yourself keeps whatever uid it has. A Loki datasource, when the sink's address explains where to find one, is `ghchronicle-loki`, unless Grafana already has a Loki datasource at that address and the sink has no `tenant_id`: that one is adopted and left as it is, since a Loki datasource with no tenant is its address and nothing else. With a tenant it is always made, because the tenant travels in a secret Grafana never hands back. The datasource is created the first time and corrected afterwards, and only the fields this writes are compared, so a timeout or a description you set on it yourself is left alone. One carrying a credential is written on every run, because Grafana reports which secrets are set and never their values: a token you rotate in the config cannot be seen from the outside, and writing it is the only way to be sure the datasource is not still using the old one. The dashboard is published with `overwrite`, so the second run updates the first rather than adding another. That is what makes `publish_on_start` safe to leave on. A run also names what it finds under a uid it no longer writes to. Changing which store the collector feeds is the case that leaves one: the old store's dashboard and datasource stay where they are, pointing at something nobody fills, and the new store's uid is a different string, so nothing overwrites them. The note says which uid and which store, and stops there. Deleting a dashboard unasked is not something a collector should do, even one it made: it may be the copy still being read, or one edited since. ### If the dashboard is already somewhere else Importing the JSON through the UI keeps the uid the file carries, so a dashboard imported that way is the one this writes over and there is nothing to do. It is only different if Grafana was asked to import it as new, or if the uid was changed by hand: then the generated uid is free, a publish makes a second dashboard beside the one being looked at, and the one open stops being the one updated. Name the existing one and it is written instead: ```yaml grafana: dashboard_uid: my-existing-dashboard ``` ### Taking it all away again `-uninstall` removes what this put in place, and only that. It takes the list of what to remove, and without `-yes` it removes nothing and prints what it would, because the alternative is one typed command that empties a store. ```sh ghchronicle -config config.yaml -uninstall all # says what would go ghchronicle -config config.yaml -uninstall all -yes # and then goes ``` | Target | What goes | | ----------- | -------------------------------------------------------------------------------------------------------------------------- | | `dashboard` | The dashboards it published, and the datasources it created | | `data` | Every table in the store whose name starts with `gh_` | | `state` | The state file, the dedupe ledger, the cache, the backfill and refill checkpoints and the lock file beside the state file | | `all` | The three above | With `-yes`, `data` and `state` take the lock beside the state file first, and refuse, naming the process, while the service or a `-migrate -yes` holds it: what they remove is what that process goes on writing, and the lock file is among the state files, so removed under a running service it would let the next `-migrate -yes` run beside it. A datasource named in `grafana.datasource.uid` or `grafana.datasource.loki_uid` is never removed, and neither is a Loki datasource it adopted: each was somebody else's before this ran and stays theirs. A target it does not recognise is refused whole, rather than the rest of the list being carried out without it. The tables are asked of the store rather than compiled in. A list inside the binary would be the measurements this version writes, and the ones worth removing are exactly the ones nobody writes any more: what an older version collected, or a family switched off since. Asking finds those. Not every store can be emptied from here, and the ones that cannot say why rather than staying silent, which would read as nothing to remove. Graphite offers no delete, so its whisper files go by hand. The Prometheus sink is scraped rather than written to, so nothing was stored to remove. The SQL sink's file is removed, but rows already loaded from it into a real database were loaded by you and have to go there: this sink emits statements and never connects. ## Importing from the UI 1. In Grafana, go to **Dashboards**, then **New**, then **Import**. 2. Upload the `ghchronicle-.json` for the store you are using. 3. Choose the datasource Grafana asks for. - **InfluxDB** The InfluxDB 3 datasource for the database the sink writes to, **in SQL mode**. - **Prometheus** The Prometheus that scrapes ghchronicle's exporter. - **PostgreSQL** The PostgreSQL datasource for the database the `postgres` sink writes to, or the one the SQL sink's statements were piped into. TimescaleDB is the same datasource with the TimescaleDB switch on; the queries do not change. - **Graphite** The Graphite the sink writes to. The paths assume the default prefix, `github`, and the functions need Graphite 1.1 or later. - **Elasticsearch** An Elasticsearch datasource whose index pattern is `ghchronicle-*` and whose time field is `@timestamp`. One datasource serves every panel, because each target names its own index in the query. OpenSearch works through the same plugin. ## Importing with the API Name the input the file declares: ```sh curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \ -d "{\"dashboard\": $(cat ghchronicle-influxdb.json), \"inputs\": [ {\"name\":\"DS_INFLUXDB\",\"type\":\"datasource\",\"pluginId\":\"influxdb\",\"value\":\"\"}], \"overwrite\": true}" \ "$GRAFANA/api/dashboards/import" ``` The inputs are `DS_INFLUXDB` (`influxdb`), `DS_PROMETHEUS` (`prometheus`), `DS_POSTGRES` (`grafana-postgresql-datasource`), `DS_GRAPHITE` (`graphite`) and `DS_ELASTICSEARCH` (`elasticsearch`). ## The uid Each file carries a fixed `uid` (`ghchronicle-`). That is deliberate for a repository import, where a stable uid means a stable URL and a re-import updates in place rather than duplicating. Importing two of these into one Grafana is fine, because the uids differ per store and cannot collide. ## They are generated, never hand-edited One list of sections and panels produces all five, so a panel that is fixed is fixed everywhere and a store that is added inherits the lot. What that means for you is the part worth knowing: a dashboard edited in Grafana is yours until the next publish overwrites it, so keep changes in a copy under a uid of your own and name it in `grafana.dashboard_uid`. How the generator works, and the checks that keep the committed files and the panels' own queries in step, are in [CONTRIBUTING](https://github.com/jmrplens/ghchronicle/blob/main/CONTRIBUTING.md). ## Publishing to the Grafana directory The files are already in the shape the directory requires: `__inputs` declares the datasource the importer must choose, `__requires` names the Grafana version and the plugin, and there is no `id` key, which the directory assigns on publication. Keep the listing names distinct, because five dashboards with the same title are indistinguishable in search results: "ghchronicle for InfluxDB", "ghchronicle for Prometheus", "ghchronicle for PostgreSQL and TimescaleDB", "ghchronicle for Graphite", "ghchronicle for Elasticsearch and OpenSearch". Publishing again against the same listing adds a revision rather than replacing it, so a regeneration that changes panels is a new revision of the same five listings, not five new listings. That is what a reader needs to know. The step by step, with the listing names, what each screenshot should show and what to do when a revision is rejected, is a maintainer's task and lives in [`dashboards/PUBLISHING.md`](https://github.com/jmrplens/ghchronicle/blob/main/dashboards/PUBLISHING.md) in the repository. --- # What they show The seventeen sections and their one hundred and fifty four panels, one capture each, and what changes when the store cannot answer. Source: https://docs.jmrp.io/ghchronicle/dashboards/panels/ One dashboard, rendered once per store. Seventeen sections, one hundred and fifty four panels, in the same places with the same titles whichever database you chose. One capture per section below, in the order the dashboard puts them. Every section but the Overview opens collapsed. Open, the first seven were twenty-three phone screens before the reader found out there were ten more; collapsed, the second screen of a phone is the index of the sixteen, each a tap away, and Grafana keeps what was opened in the URL. On a desktop it costs a click per section, and the first render asks for the Overview's panels rather than all of them. > **The section captures are of a demonstration database** > > Every section capture below is the InfluxDB dashboard over ninety days of a > demonstration database, generated so that every panel has something to draw. A > real account leaves several of them legitimately empty: no star this week, no > failed workflow in the window, no open milestone, and a capture of an empty > panel teaches nothing. The database was filled by a generator that is not part > of this repository, and it took every measurement, tag, field and column type > from the [measurements reference](https://docs.jmrp.io/ghchronicle/collectors/measurements/) and > from a live database, so the shape is real. Only the account is invented: > `acme` and five repositories that are obviously examples. The one capture that > is not of it is the last on this page, of the Prometheus dashboard, which > comes from the containerised end-to-end suite instead, and the text beside it > says so. No capture on this page is of a real account. A repository is told apart by its full name on every panel. Two owners can give a repository the same name, as an account and an organization it belongs to each keep a `.github`, and a query that grouped, joined, counted or took the newest row by the short name alone made the two one: "Open the longest" listed `alice/x#5` and `acme/x#5` as one pull request, and every chart per repository drew them as one series. Since 2.6.1 every store keys a repository by its full name, and the exporter keeps the full name on every gauge that names one. A table still shows the short name, so the two are two rows of one name, and the repository picker still filters by it. A chart in InfluxDB and PostgreSQL names a series by its short name unless two repositories in the range share it, and then by their full names; the other three stores name both by the short name. A tile that is a median or a share has no number in a range with nothing to measure, and since 2.6.2 it says what the range lacked rather than drawing an empty space under its label: "none merged" under the time to merge and the lines per pull request, "no issue closed", "none decided" under the success rate, "no runs" and "no jobs" under the run duration and the queue wait, "no commits" under the signed share, and "no deliveries" on the webhook failure rate. The words under a tile are drawn in the color of the text, not in that of the tile's lowest threshold, which is red for the success rate and the signed share: "none decided" is not a failed build. A count over such a range reads 0. A total has no number either, since the SQL stores' sum of no rows is null, and it says what the range lacked too: "no traffic" under the views and the two unique counts, "not read" under the stars and forks, the artifact storage and the cache, "no commits" under the lines added and removed, "no releases" under the download total, "none open" under the open alerts, and "no usage" under the spend. Before 2.6.2 a group of totals the range held none of, the traffic or the open alerts of a repository without any, was a panel with nothing in it at all, not even the names: Grafana sizes a tile's text by its value, and an empty one is drawn at no size. Over a range or a repository with nothing in it, every store draws the tiles the SQL stores draw, reading 0 or no value where they do: Graphite for a path it has never held, Elasticsearch for a range it holds no document in, and Prometheus for a query that finds no series. Before 2.6.2 such a tile left its group in those three, and over a repository with nothing in it eighteen tiles were missing from Graphite and twenty from Elasticsearch. Until 2.6.4 one case was left, in Elasticsearch: a value added up from the newest document of each repository, release or alert, the star and fork counts, the artifact storage and the cache, the download total and the open alerts, left its group when there was no such document, since the datasource fails outright on that aggregation over nothing, and the success rate, the run duration and the queue wait beside the two byte totals left it as well, because that panel added every value up and would have read a value of nothing as 0. Those four groups now add their values up in the panel's own transformations: each value arrives under the name of its tile, once per repository, release or alert where it is added up, the values of one name are added together, and a name nothing was read under has no value. A query that answers only the name, over any range, stands in for each total where nothing answers, so every group draws what the other stores draw. Over a range no sweep reached, the groups of the account's own snapshots, which the SQL stores read as the newest row of the range, say what the range lacked as well: since 2.6.4 every tile of Community, Account, Since the account began, Sponsorship and Repositories reads "not read" in every store, and so does each bar of Contribution mix (last year), which reads the snapshot the contributions in Account read. Until then the first four groups read "No data" in all five, as the SQL stores had no row to read, the repository count left its group in all five, taking the stars and the forks with it in Elasticsearch, and the bar gauge read "No data" in every store but Graphite, which drew its four names with nothing beside them. Before 2.6.2 Graphite, whose paths answer a range they hold nothing in with nulls, drew three of the groups as a panel with nothing in it, not even the names, and Elasticsearch drew the four names of Sponsorship with nothing beside them. A year with none of the four kinds of contribution has no mix to draw either. The bars read the newest snapshot of the range, as the Contributions tile does, so when that snapshot holds none of the four every bar reads "not read" beside a Contributions tile of 0, in every store and even where an older snapshot in the range held some. Until 2.6.4 the SQL stores and Graphite drew that older snapshot's mix there: the SQL stores left out every snapshot whose four kinds added up to 0, and Graphite's bars took the last share the range held, passing over the snapshot that has none. ## Overview A masthead rather than a panel: the mark, large and centered, the name under it, and under the name a button to this documentation and one to the source, on the page itself with no box around them. Then four groups of numbers: Repositories (repositories, stars, forks), Traffic in range (views, unique visitors, unique cloners), Community (followers, following, sponsors, sponsoring) and Account (contributions in the last year, account age, watching, stars given, gists, packages). A group is one stat panel of several values rather than a tile per number: on a desktop it reads as the row of tiles it replaces, and on a phone, where every panel is a column, sixteen tiles were four screens of single numbers and four groups are one. The repository variable beside them filters every section at once. Stars and forks add up the newest row per repository rather than every row in the range, because both are current state and a sum over the range would count every sweep. A live repository's row is its `gh_repo`, which a sweep writes every hour, and an archived one's is its `gh_repo_total`. A repository the default filter sets aside for being archived gets no `gh_repo` row from a sweep, so the picker stops offering it once the last backfill is behind it, but people still star and fork it and every `totals` sweep reads its counts again. With All selected the sums include it. The four stores that sum over the dashboard range can leave its row out of a range shorter than the `totals` cadence, an hour by default; Prometheus takes no range here, each sum being an instant query over what the running collector pushed last. An archived repository the configuration no longer collects is left out, as the picker leaves out a live one: a row a backfill wrote under an earlier configuration stays in the store, and on the account this was checked against one such archived fork added 8 stars and 3 forks, 393 and 106 where GitHub gave 385 and 103 for what the sweeps collect. The SQL dashboards count an archived row only for a repository with a `gh_repo_total` row in the last seven days, the picker's own window. Elasticsearch and Graphite cannot ask that of one window while reading another, so they count an archived repository from its rows of the last seven days, and a range that ended more than a week ago leaves the archived ones out there. Prometheus holds only what the running collector pushes, so it has no such row to leave out. Picking repositories counts those alone. Each repository counts once, by its full name: two owners' repositories of the same name are two, and one archived while the collector runs is not counted under both its live row and its archived one. The repository count beside the sums is GitHub's own count of the account's public repositories, which is a different set: it has the public forks the default filter leaves out, and none of the private repositories the sums include. The [card](https://docs.jmrp.io/ghchronicle/card/) adds up `gh_repo` alone, so its stars and forks leave out the archived repositories the default filter sets aside, which these sums count. On the account this was measured on, on 2026-09-27, the default filter set aside seventeen, holding 80 stars and 24 forks. ![The Overview row: the repository picker and the 90 day range across the top, the ghchronicle badge with its Docs and Source buttons, then four tile groups reading 5 repositories with 350 stars and 51 forks, 37.5 thousand views with 21.1 thousand unique visitors and 19.6 thousand clones, 117 followers and 58 following with 4 sponsors and 2 sponsored, and an account with 3.22 thousand contributions over 7.78 years](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/overview.png) The third traffic tile counts the people who cloned and not the clones. Continuous integration clones all day, so the clone count belongs beside the figure that explains it rather than in a headline: measured on the account this was read against, one repository was cloned 135,683 times in a fortnight by 1,807 cloners, and 186 K beside 2.89 K views reads as an audience. The count itself is two panels of the Audience section. The capture above predates that change and still reads "clones". Reads `gh_account`, `gh_repo`, `gh_repo_total`, `gh_traffic` and `gh_contributions_total`. ## Lifetime Six numbers that are true since the account began, in one panel, and one row per repository with its whole life in it: pull requests merged and reviewed ever, commits ever, issues opened, what was merged in other people's repositories and how many threads were commented on there. ![The Lifetime section: a tile group reading 1.18 thousand pull requests merged, 386 reviewed, 6.68 thousand commits, 148 issues opened, 27 merged elsewhere and 96 comments elsewhere, the table of every repository ever with commits, merges, issues, releases, stars, branches and tags, and below it the repositories created, the single archived repository and workflow runs ever as a bar per repository](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/lifetime.png) The last tile counts threads rather than comments, each issue or pull request once however many comments the account left on it, and only in repositories it does not own, the same elsewhere as the tile beside it. The capture above predates both: its tile reads "Comments elsewhere", and its 96 included threads in the account's own repositories. Every other section counts rows inside the dashboard range. These do not: GitHub counts them itself, in one search request or one GraphQL field each, and the collector stores the answer as a single row. That is why they are instant and why they are already right on the first sweep of a new install, which a running total kept by this tool would not be. It is also the shape a store can answer without reading everything it holds: InfluxDB 3 Core refuses a query that would open more than its file limit, forty thousand where this was measured, and "how many ever" from a row per fact is exactly that query. "Every repository, ever" lists every repository the picker holds, forks and archived ones included when the sweeps collect them, since that is what the title says. It is ranked by commits, which on an account with forks of busy projects means somebody else's history outranks everything the account wrote: 370,296 commits against 3,385 where this was read. So the fork and archived flags are columns, and the table opens sorted by the first of them and then by commits, which puts the account's own repositories on the first screen and the forks under them without leaving one out. With All selected it lists the archived repositories the default filter sets aside as well, with their current counts, whether or not the picker offers them: a sweep writes them no `gh_repo` row, which is what the picker lists, but the same row of each is read again on every `totals` sweep. One the configuration no longer collects is left out, as on the Overview, and Elasticsearch and Graphite apply the Overview's seven-day rule here too: a range that ended more than a week ago leaves out the archived repositories the default filter sets aside. Each row is the repository's newest in the range, one per full name, and not the largest value each column reached in it: stars, branches, tags, releases and open issues go down, and a table of peaks gave a repository 4 stars from a backfill's row nine days old where GitHub and its newest row said 3. A repository archived inside the range has rows from before the archive and after it, and is still one row in every store. Four of them flag it archived; Graphite keeps the commits alone, with no fork or archived column. Elasticsearch keeps an archived row only from the last seven days, so a repository archived inside a range that ended more than a week ago is listed there from its rows before the archive, as not archived. Under the table, the repositories created and the repositories archived, each dated when it happened rather than when a sweep noticed it, and the workflow runs each repository has ever had, from the run listing's own total. Reads `gh_account_total`, `gh_repo_total`, `gh_repo_created`, `gh_repo_archived` and `gh_workflow_run_total`. ## Audience Views, unique visitors and clones over time, the top referrers and the top paths. GitHub serves fourteen days and rewrites the whole window on every sweep, so the series extend as far back as the collector has been running rather than fourteen days. Referrers and paths carry no dates of their own, so they are the newest snapshot of that window rather than a series. ![The Audience section: views, unique visitors and clones per day as stacked bars per repository, the top referrers table led by Google with 250 views, the top paths table with each path's title, and the clone amplification table giving clones per cloner, around 1.3 in every repository](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/audience.png) > **Clones do not count people** > > Continuous integration clones a repository thousands of times for every human > visit. Measured on one repository, seventy three clones for every unique > cloner. The last panel of the section divides one by the other, which is the > only figure that separates adoption from machinery. Reads `gh_traffic`, `gh_traffic_referrer` and `gh_traffic_path`. ## Stars and forks Stars gained over time, the cumulative curve, stars by repository, the fifty most recent stars with the user and the moment, and forks over time. The eight repositories that gained the most in the range are named and the rest are `other`: of the seventeen that gained a star in two years here, nine gained between one and three, and seventeen legend entries hid a third of the plot. Stars gained over time counts `gh_star_day`, GitHub's daily star history, and so does the cumulative curve in InfluxDB and PostgreSQL; ghchronicle reads the history whole on the first sweep for every repository, except against GitHub Enterprise Server, which does not serve it. So the curve reaches back to the first star without the collector having run that long, and a repository whose stargazer list GitHub hides from the token is counted with the rest: since July 2026 GitHub serves that list only to a repository's [admins and collaborators](https://docs.github.com/en/rest/activity/starring#new-access-restrictions), and the history to anyone who can see the repository. Each day is GitHub's Pacific calendar day, so a star given on a European morning can sit a day before the moment Recent stars shows, and an unstar of a star given in the last thirty weeks is taken off the day it was given. The curve usually sits at the count GitHub shows or a little below it, since that count also includes accounts GitHub no longer lists, but a star given more than thirty weeks ago and later taken back can stay in it, so it can also sit above: only a backfill that reaches back to its day lowers that day, and only while the day holds another star. Recent stars names people, so it reads `gh_star`, and a repository whose list is hidden has its stars in the two panels above it and no names there. Prometheus still counts Stars gained and Recent stars from the stargazer list, so only where the token may read it, while its Stars over time is the current count of every repository; in Graphite and Elasticsearch the curve is the repositories' star count as each sweep read it, and starts the day the collector did. Forks over time is a running count as well in InfluxDB and PostgreSQL, from `gh_fork`, each fork dated when it was made, so it too reaches back past the day the collector started rather than beginning at the first snapshot. The other three draw the fork count each sweep read, from the day the collector or the exporter started. Both curves are drawn to both edges of the range: a running count over dated rows has a point only where a row is, so a fork curve over a quiet month was a line from the first fork to the last and blank on either side, which reads as collection having stopped. Each end carries a bucket of zero, so the line holds its value to the end of the range. ![The Stars and forks section over a range from mid-June to mid-September: Stars gained over time as daily bars stacked by repository for cli, docs-site, edge-cache, parser and telemetry, the busiest days at four; Stars over time as one line climbing slowly to 350, its legend reading a mean of 309 and a max of 350; Stars by repository as horizontal bars, telemetry 148, cli 89, parser 62, edge-cache 37 and docs-site 14; Recent stars as a table of user, moment and repository, the newest given on 12 September; and Forks over time climbing slowly to 51, with a mean of 47](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/stars-and-forks.png) Reads `gh_star_day`, `gh_star`, `gh_fork` and `gh_repo`. ## Contributions The contribution calendar as a series and then as the grid GitHub draws, a column per week and a row per weekday with a cell's shade a function of that day's own count; commits per week, the totals of the last year and the mix those totals make, the four kinds of contribution as shares of their sum, which is the radar the profile draws, beside the grid as the profile puts it; commits by hour of day and by weekday, commits by repository, and one row per past year. The grid is a status history panel over one field per weekday, because Grafana has no calendar panel and that is the one core panel that draws a grid of cells colored by value. It is GitHub's grid measured off the profile page on 2026-09-14 and copied: ten units wide by five draws the profile's own cell, ten pixels square in a 1920 pixel window, in a gutter of three to four pixels across and five to six down where the profile's is three both ways; the four greens are GitHub's dark theme read off that page the same day, over the gray of an empty day; three rows are named, Monday, Wednesday and Friday, as the profile names them; and the shade is the fifths of the busiest day of the year, the rule that reproduced all 366 of the profile's squares when applied to GitHub's own counts. The panel keeps that year whatever the dashboard range is set to, the way the profile's grid is always a year: at five years the same 53 columns would be 262 of six pixels, and at a week two bars. Three things a core Grafana panel cannot copy. The month name over the first week of each month: a status history builds its own x ticks, one every few columns, so a tick always lands on a week and the stride is a function of the panel's pixel width, which at three weeks writes the same month twice; the ticks name the Sunday each column starts on instead, which no two columns share. The square at every size: a cell is a fraction of the band its row gets, so the square survives at 1920 and again at 768, where the panel takes the whole width of a phone, and between the two the ten pixel height holds while the width narrows, eight at 1600, seven at 1280, five at 430; and where the panel sits on the dashboard it stretches into a tall bar when maximized, 27 by 67 pixels in a 1920 by 900 window and 27 by 84 in a taller one. And a tooltip naming the day and the count, since the cell is colored by the value it carries, so that value has to be the shade and a column is a week. The two punch cards are current state rather than history: every sweep writes the whole grid again, a point per weekday and hour, so a sum over a range would count every sweep. Each panel adds up every cell of each repository's newest grid in the range, by hour or by weekday, and what the panels are read for is the shape. Graphite keeps each cell as a series of its own, so a cell a rewritten history emptied keeps its last count there, where the SQL stores and Elasticsearch read only the newest grid. ![The Contributions section: contributions per day, the contribution calendar drawn as GitHub's own grid for the last year with its Mon, Wed and Fri labels, the contribution mix at 70.1 percent commits, commits per week with own commits overlaid, the totals table, the hour of day histogram peaking in the afternoon, the weekday bars, commits by repository, the five year table, commits per day by repository, and the bar chart splitting the commits the profile hides into yours and other people's, public and private](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/contributions.png) Reads `gh_contribution_day`, `gh_contribution_day_repo`, `gh_commits_week`, `gh_contributions_total`, `gh_commit_punchcard`, `gh_contribution_repo` and `gh_contribution_year`. ## Pull requests and issues Fourteen panels, and the section where the per-item collection pays for itself. Merged count, time to merge, time to review by someone else, issues closed, time to close and lines changed as one group of six values; then the same split by state over time, the largest merged pull requests, and the breakdowns by author, by repository and by reviewer. An item still open is written once per day for as long as it stays open, and those rows stay when it closes, so the dated panels count the open ones as distinct numbers under "Open that day" and draw them as a line beside the stack of what closed, and the "open the longest" tables read each item from its newest row. Each keeps the twenty-five open longest, the longest first; Prometheus, which holds no single pull request, lists the twenty-five repositories whose open pull requests have waited longest on average. Elasticsearch can keep the top values only inside each repository's bucket, so it keeps each repository's longest open and the table cuts them to twenty-five; before 2.6.1 it kept the twenty-five of each repository with the most rows, which in a repository with more open items than that could be any. ![The Pull requests and issues section: a tile group reading 467 pull requests merged, 10.3 hours to merge, 9.81 hours to a first review, 137 issues closed, 1.75 days to close one and 75 lines per pull request, pull requests and issues per day split by state, time to merge and pull request size over time, the largest merged pull requests with their titles, associations and labels, pull requests by author, the per repository table, the reviewer table led by review-bot at 209 reviews, reviews per day, the two open the longest tables, review threads per day split into bot and human, and the review debt table](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/pull-requests-and-issues.png) > **Read the review wait carefully** > > The tile counts the first review by somebody other than the author and other > than a bot, so on an account reviewed by bots alone it reads "no human > review" rather than the bots' few seconds. It is named for that, because "Time to first > review" over No data read as nothing having been reviewed at all: measured > here on 2026-09-17, 703 of 825 pull requests in a fortnight had a first > review and none of them had one from anybody else, and the field holds > fourteen rows in the whole store. Nine pull requests in ten had a bot review > inside a minute; the Reviewers table shows that, with each bot marked > as one and the author's own replies as one row. > **The lists of what to do next leave out what cannot be done** > > "Open the longest", "Open issues the longest", "Workflows that never ran" and > "Stale branches" are the four panels that read as a queue of work. On an > account with 17 archived repositories and 22 forks of other people's > projects, all four were won by rows nothing can be done to: the eight oldest > open pull requests were dependabot's in two archived repositories, every > workflow that had never run was in an archived one, and 1,083 of 1,208 > branches were a fork's. So the two SQL dashboards join each row's repository > to `gh_repo`, the one measurement carrying the fork and archived flags, and > leave the archived out of the first three and both the archived and the forks > out of the branches; the first three keep a Fork column, because a fork's > pull request is still one that can be merged. Prometheus, Graphite and > Elasticsearch have no join, so there the rows stay and each panel says so. Reads `gh_pull_request`, `gh_pull_request_review`, `gh_review_thread`, `gh_issue` and, for the archived and fork flags, `gh_repo`. ## Continuous integration Fourteen panels: run count, success rate over the runs that succeeded or failed, the canceled and skipped runs beside it, run duration, queue wait, artifact storage and cache size, all seven as one group; then the same figures over time, the workflows, the slowest jobs and the slowest steps, and then the six that say what to do about it: minutes spent on runs that failed, the workflows that keep failing, the steps that fail rather than the jobs, the workflows declared and never run, the artifacts created over time, and how much of the artifact storage was actually counted. ![The Continuous integration section: a tile group reading 1.51 thousand runs, a 90.2 percent success rate, 84 undecided runs, 15.2 minutes per run, 34 seconds of queue wait, 141 mebibytes of artifacts and 3.08 gibibytes of cache, runs per day by outcome, run duration and queue wait over time, artifact storage over time with a line per repository, and the workflows, slowest jobs, slowest steps, minutes spent on failed runs, repeatedly failing workflows, failing steps and artifact tables](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/continuous-integration.png) Queue wait is a job-level number. The run-level figure folds the wait into the duration, so a run that took twenty minutes because one job waited eighteen for a runner looks identical to one that spent eighteen executing. Every median of the InfluxDB and PostgreSQL dashboards is exact, so four jobs that waited 30, 35, 60 and 65 seconds read 47.5 in both, where the estimate InfluxDB used to answer with read 52. The 95th percentile of the run duration and the 90th of the time to merge are still estimates in InfluxDB, since InfluxDB 3 Core has no exact percentile before 3.9.0, and both panels say so. Two of these are the ones worth looking at first. "Workflows that keep failing" is not about flakiness: measured, two workflows had failed on every single run they ever had, dozens of runs apiece, and nobody had switched them off. It lists a workflow once it has failed more than three times in the range, a failure being any conclusion but success, and all five stores ask that same question. "Workflows that never ran" is the other side of the same question, and it needs both `gh_workflow` and `gh_workflow_run`, which is why it is the one panel here that only the two SQL stores can answer. "Artifact storage counted" exists because the total is a floor. GitHub reports how many artifacts a repository has, the collector records how many it actually walked, and when the second is smaller the live size is short: on one repository here, by a factor of fifty six. A sweep's walk stops at five pages, five hundred artifacts, and any walk stops sooner when a page of the listing fails, the pass keeping what it had read, so either can leave it short. It carries a third count, the live artifacts among the ones walked, because GitHub's own total includes the ones it has already expired and the size does not. The counts come from each repository's newest row, the one the tile reads, so a fuller walk earlier in the range cannot hide a short one. The tile at the top of the section is named for what it is over, and every panel that shows the size either shows those counts or says in its description that it is a floor. Reads `gh_workflow_run`, `gh_workflow_job`, `gh_workflow_step`, `gh_workflow`, `gh_artifact`, `gh_artifact_total`, `gh_actions_cache` and, for the archived and fork flags, `gh_repo`. ## Code Commits, lines added and removed, the share of signed commits, lines changed over time, repository activity by kind, commits by author and by signature, and the force pushes with who made them and on which branch. ![The Code section: a tile group reading 752 commits, 49.0 thousand lines added, 20.0 thousand removed and 55.6 percent signed over the last 90 days, lines added and removed per day, repository activity by kind, the commits by author table, commits by signature, the force push table, commits per day by gate state, the table of checks that are not Actions, and the table of commits sitting behind a red branch](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/code.png) `signature` separates `unsigned`, meaning there was no signature at all, from a signature that failed to verify. They are different facts and the bar chart keeps them apart. The last three panels are about the gate rather than the code. "Commits by gate state" is not the same claim as a workflow run failing: a run says one job failed, the rollup says the commit itself came out red, and on the measured account twenty seven of fifty commits on the default branch did. "Checks that are not Actions" holds what the two sections above cannot see, the code quality service and the dependency bot. And "Commits behind a red branch" joins the two by commit hash, which is the one panel that justifies storing `oid` and `head_sha` at all. > **This section keeps its own range** > > The six panels over `gh_commit` are pinned to ninety days whatever the range > picker is set to, and Grafana says so beside each title. One row per commit > is one Parquet file per commit in InfluxDB 3 Core, which refuses a query that > would open more than forty thousand: measured, a range of 270 days opened > just under that limit and answered, 300 days was refused, and the refusal > reaches the reader as an empty panel rather than as an error. Ninety days > opens a third of the limit, which leaves room for the history to keep > growing. Reads `gh_commit`, `gh_commit_check`, `gh_workflow_run` and `gh_repo_activity`. ## Planning and community Labels with how often each is used, milestones with their progress, forks gained over time, the fork list, discussions by category and whether they were answered, and beside that count the discussions themselves: the newest fifty one by one, with their category, comment count, whether each was answered and a link to each, whatever the range, since an account has a handful and a range of a month hid all but one of them under a row that read "Ideas". ![The Planning and community section: the labels table led by dependencies, the milestones table with progress bars, forks per day, the fork list with who forked and whether they pushed, the discussions table by category, the latest discussions with their answered state, issue transitions per day, comments left per repository, discussion answers, and the answers elsewhere table](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/planning-and-community.png) The fork list carries `advanced`, which is what separates a real derivative from a bookmark. Most forks are bookmarks. Three more panels are about the conversation rather than the plan. Transitions over time is when something was labelled, closed, reopened or renamed, which the state of an issue does not record: a reopening exists in no other measurement. The two comment tables count what was written anywhere, including in repositories the account does not own, which is where most of it happens: the comments left in other people's repositories outnumbered the discussions inside the account by more than an order of magnitude when this was measured. Beside the per-repository count of discussion comments, the comments themselves: every one left in a discussion of somebody else's repository, newest first, with whether the maintainer accepted it as the answer and a link to the comment in its thread. Both read one row per comment in every store, accepted when any of its rows says so, since a store written before 2.6.1 and since can hold a comment in more than one row, for the reason [How to read the tables](https://docs.jmrp.io/ghchronicle/collectors/measurements/#how-to-read-the-tables) gives. Reads `gh_label`, `gh_milestone`, `gh_fork`, `gh_discussion`, `gh_issue_event`, `gh_issue_comment` and `gh_discussion_comment`. ## Delivery and access The webhook failure rate, deliveries by status code, the endpoints ranked by failures, the rulesets and the deploy keys with how long each has gone unused. Under those, the configured webhooks, the environments, the stale branches, the branch protection rules, each ruleset's rules with who may bypass them, when each ruleset was changed, and the deployments over time and by environment. ![The Delivery and access section: a 20.1 percent webhook failure rate gauge, deliveries per hour by status code, the endpoint table showing legacy.example.net failing most of its deliveries, the rulesets table, the deploy keys table, the text panel about failed job output, the configured webhooks, the environments and stale branches tables, branch protection rules per repository, ruleset rules with their bypasses, ruleset changes, deployments per day by environment, and the deployments by environment table](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/delivery-and-access.png) Webhooks fail silently. Measured, one hook had been answering 403 for seventy-eight of its last hundred deliveries and nothing anywhere said so. Only the host of a webhook URL is stored, because the path usually carries a secret. One panel is text in the exported files, "Where failure output went", because the output of a failed job is text and belongs in a log store, and an importer may have none: a dashboard bound to one datasource cannot query two. Published to a Grafana with a Loki datasource, which the collector adopts or makes from a Loki sink or takes from `grafana.datasource.loki_uid`, the same panel draws the last lines of every failed job from Loki instead, newest first, with the workflow, job and run of each line in its logfmt tail. The repository variable is applied in the InfluxDB, PostgreSQL and Prometheus dashboards; the Graphite and Elasticsearch variables name the glob star as their All value, which is not a regular expression, so there the panel shows every repository and says so. The last two are inventories rather than traffic. "Webhooks configured" exists because the endpoint table is built from deliveries, so a hook that has never delivered anything appears in it nowhere, and an active hook with no traffic is exactly the interesting row. "Environments" asks the deploy key question about deployment targets: one environment here had not been touched in one thousand one hundred and seventy seven days. Reads `gh_webhook_delivery`, `gh_webhook`, `gh_ruleset`, `gh_ruleset_rule`, `gh_ruleset_version`, `gh_deploy_key`, `gh_environment`, `gh_branch`, `gh_branch_protection`, `gh_deployment` and, for the archived and fork flags of the stale branches, `gh_repo`. ## Releases Total downloads, downloads by release, and every asset with its size and its own download count. ![The Releases section: 11.6 thousand downloads over 14 releases, a bar chart of downloads per release tag, the release asset table with per asset downloads and sizes, and the table of downloads gained in the range](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/releases.png) The first three are current state. GitHub reports a running total per asset and never a history, so the series that a downloads-per-day panel would need does not exist to be collected. The fourth panel is what can be recovered from it: the difference between the first and last value inside the range, which is what each asset actually gained. A day of that on the measured account showed ninety seven per cent of the downloads going to one Linux binary and its checksum file, which is an installer rather than a person. Reads `gh_release` and `gh_release_asset`. ## Security Open Dependabot and code scanning alerts, the breakdowns by severity and by ecosystem, open alerts over time, the feature table, and the time taken to resolve an alert by severity. Under those, the oldest open alerts, the security settings, the default code scanning setup, the workflow token permissions and the secrets with when each was last rotated. ![The Security section: the open alerts tile reading 25 from Dependabot and 29 from code scanning, alerts by severity and by ecosystem, open alerts per severity over time, the security feature table with on and off per repository, code scanning runs per day by tool, the time to resolve table with each advisory and its CVSS, the resolved scanning alerts, scan results by tool, the oldest open alerts, the security settings and default code scanning setup tables, workflow token permissions, and the secret rotation table](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/security.png) The feature table is what tells "no alerts" apart from "the feature is switched off". Without `gh_security_feature`, a repository with Dependabot disabled looks exactly like one with nothing to fix. The alert counts are current state, rewritten on every sweep, so every panel that counts open alerts takes the newest row of each series and adds those up rather than summing the range. Summing the range counts each alert once per sweep: before this was fixed the tile read 3.61 thousand where the account had a few dozen. A series is a repository by its full name, so two owners' repositories of one name are two, as they are in the artifact and cache bytes of "Runs in range" and in "Downloads", which add up their newest readings the same way. The feature table and the four tables at the foot of the section, the settings, the code scanning setup, the token permissions and the secrets, are current state too, and read each repository's newest row, so a feature switched off or an alert fixed inside the range reads as it stands and not as it was at its highest. Elasticsearch does the same by narrowing each series to its newest document and reading that, and is the exception in one place, which says so in its description: its curve of open alerts takes the largest series of each severity in a bucket rather than adding them. Graphite is the exception in three, and says so in each: its settings, code scanning setup and token permissions name each row from the path, of which a setting's `status`, the setup's `state`, `query_suite` and `schedule`, and the `permissions` are part, so a change inside the range is two rows, the old one and the new. Two panels are new information rather than a different view. Time to resolve a code scanning alert comes from dates that were being downloaded and thrown away, so until now only the open count existed; of thirty four alerts on one repository, thirty were fixed and three dismissed, and all thirty three were invisible. Scan results by tool says what a scan found rather than that it ran, which is what explains a jump in the alert count. "Oldest open alerts" lists the alert rows that say open, which are not always the rows those counts are made of. A sweep writes a row for each of the newest hundred alerts of a repository, and a repository with more has its counts taken from a read of the open ones alone. So an open alert behind a hundred newer ones, raised before the collector started, is counted and not listed, and an alert fixed while it sat behind the newest hundred stays listed as open, until a backfill reads the whole list. Reads `gh_dependabot_alert`, `gh_dependabot_alert_item`, `gh_code_scanning_alert`, `gh_code_scanning_alert_item`, `gh_code_scanning_analysis`, `gh_security_feature`, `gh_security_setting`, `gh_code_scanning_setup`, `gh_actions_policy` and `gh_secret`. ## Cost Gross, the part covered by the plan, what was actually billed, Actions minutes, cost over time by product, minutes over time by SKU, and usage by repository. ![The Cost section: a tile group reading 360 dollars gross, 249 covered by the plan, 111 actually billed and 21.9 thousand Actions minutes, cost per day by product, minutes per day by SKU, the usage by repository table with SKU, quantity, unit and price, the cache entries by key, and the cache against the ceiling](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/cost.png) Gross, discount and net are all stored rather than one being derived from the others, because net is not always zero and the discount is where a monthly credit shows up. The price per unit is in the table for the same reason: it is what explains thirty thousand macOS minutes costing more than two hundred and forty thousand Linux ones. A repository can appear in that table and in no other panel, because the list that bills and the list that is swept are not the same list. A charge that belongs to no repository, which is what a Copilot seat is, is left out of that table in every store: it is a row of the bill and not a repository called (none). The spend totals above it include it. The two cache panels are about the ceiling. GitHub caps a repository at ten gigabytes and evicts the least recently used entry past it, so the bar is each repository against that cap and the panel is read for the distance left; the entry table says which key is being thrown away and which has not been touched for a week. Each row of that table is one cache of one repository as it stands: the rows of the last day in the range the collector read that repository's caches, one per ref, added up, so Entries counts the entries GitHub holds and not the rows the table read. A ref GitHub evicted keeps its last row in the store, and adding each ref's newest row in the range counted it: on the account this was checked against, one cache read 128 entries and 9.46 GiB over 108 refs where the newest day, and GitHub, held 44 and 3.08 GiB over 24. InfluxDB, PostgreSQL and Elasticsearch read each repository's newest day. Graphite cannot find it, so it reads the last UTC day of the range: its table is empty from midnight UTC until the day's first `actions` pass, and it shows the size alone, since a Graphite table keeps one column. Prometheus holds the value last pushed for each ref, so an evicted ref still counts there until the exporter drops it a day later, or, pushed over OTLP, until the process restarts. Reads `gh_billing_usage`, `gh_actions_cache` and `gh_actions_cache_entry`. ## Activity Events over time, events by type and by repository, notifications by reason and kind, notifications over time, and the work done in other people's repositories. Both dated panels bucket by the range rather than by a fixed hour or day. ![The Activity section: events per day by type, the donut of events by type led by PushEvent at 39 percent, events by repository, the notifications table by reason, notifications per day, the latest notifications, the work elsewhere table of pull requests and issues in other people's repositories, the languages starred chart, and the recently starred table](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/activity.png) Events and notifications are windows, not histories. GitHub keeps the last three hundred events of the past thirty days and drops notifications after three months unless they are saved, so what is stored is what was there when the sweep ran. The donut carries the share of each type in its legend and writes nothing on the slices, because Grafana leaves a label off a slice it does not fit and the thin slices never got theirs. The legend is a list under the chart rather than a column beside it, measured rather than chosen: under 992 pixels Grafana puts every legend under the chart and caps it at 35 per cent of the panel, whatever the panel asks for, and a legend placed beside the chart is drawn there as a column, one entry per line, that ended after seven entries on a phone; a list placed under the chart wraps, and at the height the pie has, the eleven types of a month all fit at 360 pixels. Each type is a slice in a color of its own and under its own name in every store: the query answers a row per type, and the panel makes each row a field, because Grafana colors a pie by field and a column of counts is one field, which drew every slice the same green. The last two panels are the mirror of the stars section: what this account starred rather than what was starred, by language and by project, dated when each star was given. Whether the projects are small or famous is a different question from how many there are, so the second table carries their own star counts. The Prometheus exporter keeps the language of each star and not its repository, so there the second table has a row per language, with the mean star count of the repositories starred in it. The table of work elsewhere carries the same number for the repositories the account contributed to: each one's star count at the newest sweep inside the range, from `gh_upstream_repo`, joined onto every item in the InfluxDB and PostgreSQL dashboards and beside every repository in the Prometheus one. Graphite and Elasticsearch cannot join two measurements in one panel and leave the column out, saying so. The `outbound` family writes that measurement since 2.6.0, so a range with no row of it, which is every range before the upgrade, has an empty Stars column. Until the first `outbound` pass of 2.6.0 writes it, InfluxDB and PostgreSQL have no table to join and refuse the whole panel rather than leave the column empty, which [the troubleshooting page](https://docs.jmrp.io/ghchronicle/reference/troubleshooting/#the-data-looks-wrong) covers. InfluxDB, PostgreSQL and Elasticsearch list each item once, from its newest row in the range, and Elasticsearch has no Opened column, since that date is the row's own less how long the item was open, a subtraction a bucket cannot make. Prometheus has a row per repository, and Graphite one per item and state, since the state is part of the path: an item seen open and then closed inside the range is two rows there. Reads `gh_event`, `gh_notification`, `gh_external_contribution`, `gh_upstream_repo` and `gh_star_given`. ## Inventory Code by language, the community profile score per repository, the repository table, the topics, the packages, the gists, and the container tags with the moment each was published. Under those, the repository settings, the account keys, the social accounts, the configuration changes, the policy files, the Dependabot ecosystems, and the dependencies by licence, by ecosystem and as they changed. ![The Inventory section: code by language, the community profile table, the repository table with stars, forks, size, age and licence, the topics, packages and gists tables, the container tags published, the repository settings and account keys tables, dependencies by licence, the social accounts, configuration changes, policy files and Dependabot ecosystems tables, and dependencies by ecosystem and dependency changes](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/inventory.png) `open_issues` in the repository table is GitHub's own field, and GitHub counts pull requests in it. `gh_issue` is what to count issues with. The community profile shows the boxes behind the score as well as the score, because the percentage hides which one is missing: on the measured account eight of thirty-five repositories have no licence. The issue template box is not GitHub's: the API's `issue_template` flag reports only the legacy single `ISSUE_TEMPLATE.md` file, not a templates directory, while the community page and `health_percentage` do count the directory, so the flag said "no" for every repository of an account whose repositories score 100 with four issue forms each (the gap is filed as community/community#207706). The column is the count of templates the repository actually has, forms and Markdown, from `gh_repo_policy.issue_templates`, joined to the profile row per repository in the InfluxDB, PostgreSQL and Prometheus dashboards; Graphite and Elasticsearch cannot join two measurements in one panel and keep the API's flag, saying so. The settings table is the same idea for what a repository allows, and `codeowners_errors` is the entry in it that fails silently, since a broken CODEOWNERS file stops requesting reviews and says nothing. The account keys table is where an SSH key that has never been used shows up, and where the expiry of the key that signs every commit is written down. The last panel is empty almost always, and that is the point: a row in "Configuration changes" means a repository was renamed, archived, made private, relicensed or had its default branch moved. Identity counts distinct `repo_id` values, which is the only way to tell a rename from a new repository, because GitHub publishes no rename history at all. Reads `gh_repo`, `gh_repo_language`, `gh_repo_community`, `gh_repo_topic`, `gh_repo_policy`, `gh_package`, `gh_package_version`, `gh_gist`, `gh_key`, `gh_social_account`, `gh_policy_file`, `gh_dependabot_ecosystem`, `gh_dependency`, `gh_dependency_license` and `gh_dependency_change`. ## Profile and sponsorship What the profile advertises and what Sponsors moves: the money in and out, the sponsorships in both directions, the tiers on offer, the pinned items, the profile flags, the star lists, and the achievement badges with the distance from each to its next tier. ![The Profile and sponsorship section: the four sponsorship figures, $1.28 K received over the lifetime, $65 a month, $32.5 at the next payout and $24 spent sponsoring, then the sponsorships table beside the tiers, the pinned items beside the profile flags, the star lists, the achievements led by Pull Shark at gold, and the achievement progress table with a bar per badge](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/profile-and-sponsorship.png) The four figures are the newest reading of a snapshot rewritten every sweep and not a sum over the range, which would report the lifetime total once per sweep. The lifetime total is the one that keeps the history: the monthly estimate goes to zero the moment the last sponsorship lapses, and the money that did arrive stops being visible anywhere else. What is spent sponsoring is not the spend in the Cost section, which is what GitHub charges for Actions, packages and storage; this is money that leaves for somebody else's work. The tables under it are inventories rather than histories, and they say so by naming their own window: a sponsorship made in 2021 is listed at the default thirty days, where a panel on the dashboard's range would draw an empty axis over a handful of rows spread across years. A sponsorship is dated the day it began and not the day of a payment, both connections are read with `activeOnly` off so a lapsed one is recovered, and the sponsorable is the literal word private when the other party is hidden, with no link guessed at. The tiers are anchored to the start of the UTC day for the same reason: dated at their creation they would all fall outside every range and read as no tiers at all. Pinned items carry the position as a field and not a tag, because a repository that moves from slot two to slot three is the same pin, and as a tag every rearrangement would fork the series. The repository filter at the top of the dashboard does not reach that panel: a pin can be a gist, which is named by its hash and is in no repository the filter knows. The achievements are read every hour from the public profile page, because no API lists them. The count behind Pair Extraordinaire, merged pull requests in public repositories with a co-authored commit, is a tally the state file keeps: each hourly pass adds the pull requests merged since the last day it covers, and the whole history is walked again once a week, which is when a count that went down, a repository made private or deleted, comes down in the table. Without a state file every start walks it whole. Next tier at is the community-observed threshold (Schweinepriester/github-profile-achievements) rather than a number GitHub publishes, so the last column says whether the profile page agrees with the tier the count implies; a row that disagrees is a rule the page contradicts, and it carries no target and no bar. Both achievement panels are empty until that family has run once. Reads `gh_sponsors_listing`, `gh_sponsorship`, `gh_sponsors_tier`, `gh_pinned_item`, `gh_profile_flag`, `gh_star_list`, `gh_achievement` and `gh_achievement_progress`. ## The collector itself What the collector has left to spend and what it managed to do: the budget of each of GitHub's fifteen independent rate limits over time, a table of all of them ordered by how much of each has been used, and under those two the row's own report on the sweep. ![The collector itself section: the rate budget used per bucket over the range, and the table of every bucket with its limit, most used and lowest remaining, led by core at 3134 used of 5000 and 1866 left](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboards/the-collector-itself.png) The chart shows only the buckets with more than thirty requests in them, because search has thirty a minute and would flatten the axis. Reading all of this costs nothing: `GET /rate_limit` is the one endpoint GitHub does not charge for. Without it, a family skipped because a bucket was spent looks exactly like a family with nothing to report. Most used and Lowest remaining in the table are the extremes of the range, the most any reading spent and the least any had left, so a bucket spent an hour ago still says so after it has refilled; Graphite keeps the lowest remaining alone. "Every family" and "What failed, and where" are the two panels below them, and the capture above predates both. The first lists every collector that ran in the range, what stopped it where something did, how many sweeps it ran in and the most repositories one of those sweeps asked it about, which for `commits`, `issueevents` and `issues` includes the repositories the [movement query](https://docs.jmrp.io/ghchronicle/api/cost/#asking-first-what-moved) found nothing new in and the family left unread; a family with no row there did not run at all. The reason is a column of its own so that the two kinds of failure sort apart: a spent search budget is not the 502 that cost a repository its history, and a family that met both has a row for each. The second is one row per repository one collector could not collect, newest first, with what GitHub answered. An empty second table is the good case, and it is drawn empty rather than refused because the first table's rows are written every sweep whether or not anything failed, so the measurement behind both exists from the first one. They are there because of a failure this page could not explain. On 2026-09-16 five repositories of this account held no workflow run or job at all, each because one call for the jobs of one run had answered `502` once and the collector had thrown away everything it had gathered for that repository. The Continuous integration row was drawn over an account missing its five busiest repositories while the Cost row on the same page reported one of them burning 27.6 K macOS minutes. Neither panel takes the repository variable: the family rows belong to no repository, and a repository a sweep could not collect may be one the variable does not list, since the variable is built from rows the same sweep writes. Reads `gh_rate_limit` and `gh_collector_family`. ## The Link column Every table whose rows are items on GitHub selects a column called Link, the row's `url`, and hides it: the link hangs on the table's first column, the name of the thing, and opens the row's own page in a new tab. On a phone the column at the far right was reached in two tables of twenty-eight, and the first column is always on the screen; for the same reason the column a table is sorted by is its second, and a date is shown to the minute rather than the second. A few tables link from another url the row carries: Recent stars opens the person who gave the star, Top referrers the host that sent the visitors, Deployments by environment has a second column, Live, for the environment's own address. Release assets keeps its file under Download, which fetches the binary, and adds the release page as its Link; a click on a bar of Downloads by release opens the release too. Where the page is one GitHub shows to the owner alone, the settings of a repository, its traffic graph, an alert, the link's hover title says so. The column reaches a store only where that store's query returns it: the two SQL stores select it by name and Elasticsearch buckets on `url.keyword`, so a document without one lands in an empty cell rather than out of the table. Prometheus carries no url and Graphite keeps no string, and each of their tables says in its description that the Link column of the InfluxDB dashboard is absent there. `cmd/check_dashboards` holds every link column of a rendered dashboard to that: the column has to come back, and hold nothing but absolute urls and empty cells, a null from the SQL stores or the empty string of that bucket. ## When a store cannot answer The stores cannot answer identical questions, and the panels say so rather than pretend. - **InfluxDB and PostgreSQL** hold a row per fact, dated when the fact happened, so they draw the traffic of a particular Tuesday, the star curve since 2018 and the merge time of a pull request closed in July. The PostgreSQL set is the InfluxDB SQL translated, because the SQL sink writes the same facts as tables, and it compares text by its bytes as InfluxDB does, so a list is in the same order in both whatever collation the database was created with. - **Prometheus** stamps every sample at scrape time, so the exporter reduces the per-item rows to current values plus, for the counted measurements, a monotonic `_total` of distinct items seen since the exporter started. `increase()` over that is how the "per day" and "over the range" panels are answered. - **Graphite** keeps the dated points but has no rows: a table there is one or two numbers per series reduced over the range, so a table that needs several fields of one row keeps the numbers a row of it can carry and names every column it drops, a string is not a metric there at all, and a boolean is kept as 1 or 0 like any number. - **Elasticsearch** keeps the dated documents, so a per-item table is the newest documents themselves, or the newest document of each item where an item has several, and everything else is a bucket aggregation, on the `.keyword` sub-field of each tag. Graphite and Elasticsearch share two limits the SQL stores do not have. Neither can join two measurements in one panel, so Work elsewhere has no Stars column there and the community profile keeps the API's issue template flag rather than the count of templates. And neither can ask about one window while reading another, so the archived repositories the default filter sets aside count on the Overview and in Every repository, ever from their rows of the last seven days, and a range that ended more than a week ago leaves them out. A chart or a bar chart the SQL stores fold, its busiest series named and the rest one called other, folds the same way in Graphite, and in Prometheus where the exporter holds the counts it would add up. Elasticsearch cannot, since a terms aggregation says nothing of the values it does not keep, and each panel there that draws every series says so. Each panel whose twin in another store is richer says so in one sentence of its description. ## Twenty-four panels have no Prometheus answer at all Top paths, the contribution calendar, the two commit punch cards and the two per-repository splits of the calendar, the largest pull requests, pull requests by author, the issues open the longest, the review debt, the slowest steps, the steps that fail, the workflows that never ran, the artifacts created over time, the commits that left the branch red, the latest discussions and the answers left elsewhere, release assets, what each asset gained, the oldest open alerts, events by repository, the latest notifications, container tags and the configuration that changed. The exporter either skips the measurement, drops the identity the panel is about, or the panel is a join between two of them. Where failure output went is a text panel in every store, this one included: it is a note about where to look, not a query. Three of those are joins or set differences and have no answer in Graphite or Elasticsearch either, for the same reason: an aggregation runs inside one index or one tree, and these need two. The calendar has none in either for a different one: the grid needs the week along one axis and the weekday along the other, and a date histogram or a summarize buckets by one interval. Each of the two loses a little more on its own account: Graphite the oldest open alerts and the latest notifications, because it stores numbers and not the strings those tables are made of, and Elasticsearch what each asset gained, which is the difference between the first and the last value of the range. They are still emitted, as text panels with the same title saying what they would show and why the store cannot, so the layouts stay identical. ![The Prometheus dashboard over ten minutes: the Overview with the octocat fixture's current values (42 repositories, 80 stars, 9 forks; 361 views, 54 unique visitors, 19 clones; 1.20 thousand followers; 1.11 thousand contributions over 15.6 years), then the Audience section in which Views, Unique visitors and Clones over time all read No data while Top referrers and Clone amplification carry rows and Top paths is the text panel saying the exporter skips gh_traffic_path, the fourteen collapsed section headers, and at the foot the collector's own Rate budget used drawn as a flat line at 0.06 per cent](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/dashboard-prometheus-e2e.png) That capture is of no account at all. It is the Prometheus dashboard of the containerised end-to-end suite (`test/e2e/docker`), where the collector sweeps the fake GitHub of `test/e2e/fakegh`, whose account is `octocat` with one repository, and Prometheus scrapes the exporter every five seconds. The range is the ten minutes it had been collecting when the capture was taken, which is what makes it worth showing: every number on the Overview, and both tables of the Audience section, is a current value, because a current value is all the exporter can serve; `Rate budget used` is a flat line, because a gauge repeated at every scrape is a flat line; `Top paths` is the text panel described above; and `Views`, `Unique visitors` and `Clones over time` read "No data" because they floor their bucket at a day and this Prometheus is minutes old. On a server that has been scraping for months those three draw, starting the day the exporter did. --- # Overview The ghchronicle card: a self-contained SVG of GitHub activity for a profile README, drawn from the same points the databases receive. Source: https://docs.jmrp.io/ghchronicle/card/ ghchronicle can draw a self-contained SVG card of an account's GitHub activity for a profile README, from the points it already collects. ```sh ghchronicle -config config.yaml -card card.svg -card-only ``` > **This is a side feature** > > The point of the project is the ingestion. The card exists because the numbers > were already there, and it is drawn from exactly the same points the databases > receive, so the card and the dashboard read the same numbers, with the one > exception [What it draws](https://docs.jmrp.io/ghchronicle/card/#what-it-draws) names. ## The flags | Flag | Does | | --------------------------------------- | ------------------------------------------------------------ | | `-card ` | Runs one sweep and writes the SVG there | | `-card-only` | Writes the SVG and nothing else, so no database is needed | | `-card-layout ` | One of the thirteen [layouts](https://docs.jmrp.io/ghchronicle/card/layouts/) | | `-card-theme ` | Both writes the light card and a `_dark` twin from one sweep | | `-card-motion ` | How an animated layout moves; the others ignore it | | `-card-fields ` | Comma-separated, in drawing order | | `-card-width ` | The layout's own width when left out; on `activity-heatmap` it buys weeks | | `-card-speed <0 to 1>` | How fast an animated layout plays; `0.5`, the default, is the pace it always had | | `-card-layouts` | Prints the layouts with their fields and widths, then exits | Without `-card-only` the card is written **as well as** everything the sinks would normally get, which is the arrangement for a host that is already collecting and wants a picture too. ## The card run and the state file A run given `-card` collects **every family**, whatever the cadences say, because every number on the card comes from the points of that one sweep and a family skipped as not due would be a zero on the picture. With `-card-only` it also leaves the [state file](https://docs.jmrp.io/ghchronicle/configuration/#state_file) exactly as it found it. Nothing that run collected reached a store, so nothing it learned may tell the next collection that a family is already done. It still reads the file, which is what lets it skip the one-off walks of the stargazer list and of the daily star history: what the state remembers makes the sweep cheaper, never the card smaller. So a card, a second card and a collection can all share one `state_file`, and each of them is the whole account. ## What it draws Stars, forks, followers and repository count; contributions over the last year; views and unique visitors over GitHub's fourteen-day traffic window; a sparkline of the contribution calendar; and the top repositories by stars with their language. Stars and forks are **summed from the repositories the sweep collected**, because GitHub's account endpoint reports neither total. That means excluding forks or archived repositories in `targets` is visible on the card too, which is the honest reading rather than a hidden discrepancy. Each repository counts once, by its full name, as the Overview counts it: two owners' repositories of the same name, a user's `.github` and an organisation's, are two repositories, and the top repositories can show that name twice. Up to 2.6.0 the card went by the short name, and one of the two replaced the other. That sum is the one place the card and the dashboard's [Overview](https://docs.jmrp.io/ghchronicle/dashboards/panels/#overview) part. The card adds up `gh_repo`, the row a sweep writes for each repository it collects. The Overview adds the archived repositories the default filter sets aside as well, whose counts every `totals` sweep reads into `gh_repo_total`, since people go on starring and forking them. On the account this was measured on, on 2026-09-27, the default filter set aside seventeen, holding 80 stars and 24 forks that the Overview counts and the card does not. ## The fields `-card-fields` takes a comma-separated list, in drawing order, from: `stars`, `forks`, `followers`, `repos`, `contributions`, `views`, `visitors`, `clones`, `commits`, `pull_requests`, `reviews`, `issues`, `languages`, `top_repos`, `sparkline`. A layout skips a field it cannot draw. An unknown name is an error that lists the valid ones, rather than a silent skip: a typo would otherwise remove a number and nobody would notice until the card was already committed. ```sh ghchronicle -config config.yaml -card card.svg -card-only \ -card-layout github-stats -card-theme dark \ -card-fields repos,stars,forks,followers,commits,pull_requests,languages ``` Empty means the layout's default set. ## Themes - **dark and light** One palette each, and the choice for a README. `-card-theme both` (or `card-theme: both` in the Action) writes the light card at the path given and the dark one beside it with `_dark` before the extension, from the same sweep, so the two can never disagree. Put both in a ``, the way GitHub documents showing a different picture per theme: ```html My GitHub card ``` This project's own README does exactly that with its cards. - **auto (default)** Both palettes in one file, behind a `prefers-color-scheme` query inside the picture. The query follows the reader's operating system rather than the theme they chose on the page, and a browser does not reliably evaluate it again inside an image: on GitHub, auto cards were seen switching palettes on a dark page after the tab was left and come back to. It suits a page with no theme of its own. For a README, use the two files. ## Three constraints it respects **Self-contained.** No external stylesheet, no webfont, no `` pointing at a URL, no script. GitHub's camo proxy serves README images from its own domain and blocks all of that, so anything external would simply not render. **Deterministic.** The same input produces a byte-identical file, so a scheduled job that commits the card does not produce a diff on every run. **Written atomically.** Rendered to a temporary file and renamed, so a reader watching the path never sees half a document. ## Motion Animation, where a layout has it, is CSS inside the SVG, and there is no script, ever. Every animation ends on the complete static card, so a renderer that ignores animation shows the finished state, and `prefers-reduced-motion` switches it off in every mode. | `-card-motion` | What the card does | | -------------- | --------------------------------------------------------------------------- | | `once` | Plays when it loads and settles. The default | | `loop` | The same, and whatever the layout has that never ends goes on for ever | | `off` | No animation at all, and a smaller file | ### A loop never replays An animation here reveals content: a number counts up to what it is, a bar grows to its share, a line draws itself. Playing that again would take back something the reader has already been shown, and a card that keeps hiding its own figures is worse than a still one. So `loop` does not mean "and again". The reveal plays once and settles in both modes, and the only thing that may go on for ever is motion that puts nothing on the card and takes nothing off it. Two layouts have such a thing: - `terminal`, whose cursor blinks at its prompt from the moment the window is drawn. The numbers still type themselves in once. Played `once` instead, the cursor waits for the last of them, blinks a couple of times and settles lit. - `ticker`, whose band of pills keeps scrolling. Nothing disappears; the same pills come round again. On every other layout `loop` draws exactly the card `once` draws, to the byte. It is accepted rather than refused, so a workflow can set it once and change `card-layout` freely. `once` is still the considerate default for a profile, and more so than before. A card that loops moves for everyone who opens the README, and neither of the two that can is paced: the cursor blinks and the band scrolls with no pause between passes, where the looping cards this replaced held still for seven seconds between plays. A README gives a reader no way to stop it. Their only escape is `prefers-reduced-motion`, which switches the animation off for them but is a setting for their whole machine, not a control over your card. That is the same reason the layouts page never shows a card looping until the reader presses the toggle (WCAG 2.2.2, Pause, Stop, Hide). ![The terminal layout played once: a terminal window whose lines of output each have a number typing itself in, with a lit block cursor at the prompt below them that blinks once the last number lands, and the page can also play it in a loop, where the typing still happens once and only the cursor goes on, blinking from the start](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-terminal.svg) - **Binary** ```sh ghchronicle -card-layout terminal -card-theme both -card-only -card card.svg -config config.yaml ``` The looping picture: ```sh ghchronicle -card-layout terminal -card-theme both -card-only -card card.svg -config config.yaml -card-motion loop ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: terminal card-theme: both ``` The looping picture: ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: terminal card-theme: both card-motion: loop ``` ### How fast it plays `-card-speed` is a decimal from 0 to 1, and 0.5 is the default. It is one number for the whole card: every animated layout scales by it, the two continuous motions with the reveals, so a card set slower has a band that takes longer to come round and a cursor that blinks more slowly at the same time. | `-card-speed` | What the card does | | ------------- | ---------------------------------------------------------------------- | | `0` | The slowest animation, twice as long as the default | | `0.5` | Exactly the card this renderer always drew, to the byte. The default | | `1` | The fastest, half as long as the default | There is one knob and not one per layout for the reason there is one width and not one per layout: the cycles here were paced against each other, and scaling them together is what keeps the pacing the motion was designed with. A speed outside the range is refused before the sweep runs, naming both ends. > **0 is the slowest animation, not none** > > A range that starts at zero reads like a switch, and this one is not. A card > drawn at `0` still animates, as slowly as this renderer will draw it. The one > that draws no animation at all is `-card-motion off`, and it is also the one > that makes the file smaller. `prefers-reduced-motion` is untouched by any of this: a reader who has asked their machine for less motion gets no animation to slow down or speed up. ## In a README ```html My GitHub statistics ``` The workflow that keeps it current, for a profile README or any other, is in [A card in your profile README](https://docs.jmrp.io/ghchronicle/install/actions/#a-card-in-your-profile-readme). ## There is no Go library The renderer lives in `internal/render`, and Go refuses that import from outside the module, so there is no way to draw a card from your own program by calling into this one. A program that wants an SVG runs the binary with `-card` and reads the file, the same way it would run any other command: see [calling it from a program](https://docs.jmrp.io/ghchronicle/reference/subprocess/). --- # Layouts Thirteen layouts in two visual families, with what each one draws by default, which of them animate and which of them can keep going. Source: https://docs.jmrp.io/ghchronicle/card/layouts/ The GitHub activity card comes in several layouts, and this command lists them. ```sh ghchronicle -card-layouts ``` Prints them with the fields each shows by default. Every layout below has a section of its own, stating its family, its motion, the width it is drawn at and those fields, over a picture of the card. Each card is shown in the palette this page is in, light or dark; every layout draws both, and `auto` puts both in one file. Where a layout animates, the animation plays once and settles on the complete static card, so every picture below is that settled frame and not a moment in the middle of one. How that is set, and what switches it off, is [Motion](https://docs.jmrp.io/ghchronicle/card/#motion). ## The two families The **chronicle** family is this tool's own look. The **github** family uses GitHub's Primer palette and monospace numbers so the card sits in a profile README as if GitHub had drawn it. > **The command under each card** > > Under each picture, **Command for** opens the command that draws that card > from your own account, and the same card as a step of the > [Action](https://docs.jmrp.io/ghchronicle/install/actions/#a-card-in-your-profile-readme). > `config.yaml` is your configuration, and `-card-theme both` writes two files, > `card.svg` and `card_dark.svg`, which is how a page or a README shows the card > in its own palette. On the two cards that loop, pressing the loop toggle adds > `-card-motion loop`. The numbers a card draws are chosen with `-card-fields`, > and every flag is on [the card](https://docs.jmrp.io/ghchronicle/card/#the-flags). ## summary The original card: title, two rows of numbers, a sparkline and the most starred repositories. - **Family**: chronicle - **Motion**: still - **Width**: 495 px, drawn from 300 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `repos`, `contributions`, `views`, `visitors`, `sparkline`, `top_repos` ![The summary layout: a title, two rows of large numbers, a contribution sparkline and a list of the most starred repositories](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-summary.svg) - **Binary** ```sh ghchronicle -card-layout summary -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: summary card-theme: both ``` ## github-stats GitHub's own box: a header band, rows of four monospace numbers and a language share bar with its legend. The widest of them, and the one that looks most native in a profile README. The numbers count up, and the bar grows in from its left edge once they have landed. - **Family**: github - **Motion**: plays once - **Width**: 800 px, drawn from 600 to 1200 - **Default fields**: `repos`, `stars`, `forks`, `followers`, `commits`, `pull_requests`, `views`, `clones`, `languages` ![The github-stats layout played once: a header band over rows of four monospace numbers counting up, and a horizontal language share bar growing in from the left beside its legend](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-github-stats.svg) - **Binary** ```sh ghchronicle -card-layout github-stats -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: github-stats card-theme: both ``` ## github-compact One row of monospace numbers under a thin header band. - **Family**: github - **Motion**: still - **Width**: 495 px, drawn from 300 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `repos`, `commits` ![The github-compact layout: a thin header band above a single row of monospace numbers](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-github-compact.svg) - **Binary** ```sh ghchronicle -card-layout github-compact -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: github-compact card-theme: both ``` ## badge-row A row of 20 pixel pill badges, one per number, for a README line. - **Family**: chronicle - **Motion**: still - **Width**: follows its content - **Default fields**: `stars`, `forks`, `followers`, `repos`, `contributions` ![The badge-row layout: a horizontal row of small pill badges, each with a label and a number](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-badge-row.svg) - **Binary** ```sh ghchronicle -card-layout badge-row -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: badge-row card-theme: both ``` ## wide-banner A full-width 60 pixel banner: login on the left, numbers spread across, the sparkline drawing itself behind them. - **Family**: chronicle - **Motion**: plays once - **Width**: 800 px, drawn from 500 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `contributions`, `sparkline` ![The wide-banner layout played once: a wide, short banner with the login on the left, numbers spread across and a sparkline drawing itself behind them](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-wide-banner.svg) - **Binary** ```sh ghchronicle -card-layout wide-banner -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: wide-banner card-theme: both ``` ## sparkline-hero The contribution sparkline is the whole card, with up to three numbers overlaid. The line draws itself on load. - **Family**: chronicle - **Motion**: plays once - **Width**: 495 px, drawn from 300 to 1200 - **Default fields**: `contributions`, `stars`, `followers`, `sparkline` ![The sparkline-hero layout played once: a large contribution sparkline drawing itself across the card with three numbers overlaid](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-sparkline-hero.svg) - **Binary** ```sh ghchronicle -card-layout sparkline-hero -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: sparkline-hero card-theme: both ``` ## language-ring A donut of language shares with the legend beside it and a row of headline numbers. Each slice draws itself around the ring after the one before it, and the legend appears when the donut is whole. - **Family**: github - **Motion**: plays once - **Width**: 495 px, drawn from 400 to 1200 - **Default fields**: `languages`, `stars`, `repos` ![The language-ring layout played once: a donut chart whose language slices draw themselves one after another, with a legend appearing beside it and a row of headline numbers](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-language-ring.svg) - **Binary** ```sh ghchronicle -card-layout language-ring -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: language-ring card-theme: both ``` ## repo-list The most starred repositories as the main content: language dot, stars and a bar per row, totals underneath. - **Family**: github - **Motion**: still - **Width**: 495 px, drawn from 300 to 1200 - **Default fields**: `top_repos`, `stars`, `forks`, `repos` ![The repo-list layout: one row per repository with a language dot, the star count and a proportional bar, with totals underneath](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-repo-list.svg) - **Binary** ```sh ghchronicle -card-layout repo-list -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: repo-list card-theme: both ``` ## activity-heatmap As much of the contribution calendar as the width holds, up to a year of it, as GitHub's green squares, with up to three numbers beside it. The week count is not a fixed number: the grid takes whatever room the numbers beside it leave, so it ends where the card does rather than stopping a third of the way short. At the width this layout declares that is twenty-three weeks, at its minimum sixteen, and `-card-width` at the far end its facts state draws the whole year the collector keeps. An account whose numbers reach seven digits takes a wider column for them and leaves the grid a week or two fewer, which is the same rule seen from the other side: at the width this layout declares, a million contributions is twenty-two weeks rather than twenty-three, and six digits still fits inside the labels. The far end is where the year lands for the three numbers this layout draws by default, so a card asked for fewer, or for numbers with shorter labels, reaches the year before it and has room to spare at the end: `-card-fields sparkline` draws its whole year well short of the far end. The weeks fade in from the left, the wave crossing the grid in 0.22 s however many weeks it holds, so the calendar fills in as a wave of the same length at every width. - **Family**: github - **Motion**: plays once - **Width**: 495 px, drawn from 400 to 891 - **Default fields**: `sparkline`, `contributions`, `commits`, `pull_requests` ![The activity-heatmap layout played once: twenty-three weeks of contribution squares in GitHub's green scale fading in from the left, with three numbers beside them](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-activity-heatmap.svg) - **Binary** ```sh ghchronicle -card-layout activity-heatmap -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: activity-heatmap card-theme: both ``` ## animated-counters Numbers that count up on load over a sparkline that draws itself, settling to the static card. - **Family**: chronicle - **Motion**: plays once - **Width**: 495 px, drawn from 300 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `repos`, `contributions`, `views`, `sparkline` ![The animated-counters layout played once: a grid of large numbers counting up over a contribution sparkline that draws itself](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-animated-counters.svg) - **Binary** ```sh ghchronicle -card-layout animated-counters -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: animated-counters card-theme: both ``` ## terminal A terminal window with the project's mark in its title bar, one line of output per number and one per repository. The numbers type themselves in, a line at a time, under a cover painted in the card's own background colour. The cursor at the prompt is the card's one piece of endless motion, and it does a different thing in each motion. Played `once` it sits lit while the numbers arrive, blinks a couple of times when the last one lands and settles lit, which is the state the finished card rests in. Under `loop` it blinks from the moment the window is drawn and never stops, because a cursor blinks for the reason a terminal is open and not for the reason a card is finished. The typing itself happens once either way. - **Family**: chronicle - **Motion**: plays once, or in a loop - **Width**: 495 px, drawn from 360 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `repos`, `contributions`, `top_repos` ![The terminal layout played once: a terminal window with the project's mark in its title bar, whose lines of output each have a number typing itself in under a lit block cursor that begins blinking when the last number lands, and the page can also play it in a loop, where the typing still happens once and the cursor blinks from the start and never stops](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-terminal.svg) - **Binary** ```sh ghchronicle -card-layout terminal -card-theme both -card-only -card card.svg -config config.yaml ``` The looping picture: ```sh ghchronicle -card-layout terminal -card-theme both -card-only -card card.svg -config config.yaml -card-motion loop ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: terminal card-theme: both ``` The looping picture: ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: terminal card-theme: both card-motion: loop ``` ## ticker A band of pills, one per number and one per repository, scrolling from right to left. The content is repeated end to end and the band moves by exactly one copy, so the picture at the end of a pass is the picture at its start and the loop has no seam. Played once, it makes a single pass and comes back to the beginning. The band scrolls at a fixed speed, so a card with more in it takes longer to come round than any other layout takes to settle. - **Family**: chronicle - **Motion**: plays once, or in a loop - **Width**: 800 px, drawn from 400 to 1200 - **Default fields**: `stars`, `forks`, `followers`, `repos`, `contributions`, `commits`, `views`, `top_repos` ![The ticker layout played once: a wide band of rounded pills, one per number and one per repository, scrolling from right to left under the account name, and the page can also play it in a loop](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-ticker.svg) - **Binary** ```sh ghchronicle -card-layout ticker -card-theme both -card-only -card card.svg -config config.yaml ``` The looping picture: ```sh ghchronicle -card-layout ticker -card-theme both -card-only -card card.svg -config config.yaml -card-motion loop ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: ticker card-theme: both ``` The looping picture: ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: ticker card-theme: both card-motion: loop ``` ## language-bars One full-width bar per language, each growing from its own left edge after the one above it, with the name and the share arriving once their own bar has stopped. The layout that shows the share of a language as its own line, where `language-ring` shows all of them in one donut and `github-stats` in one bar. - **Family**: github - **Motion**: plays once - **Width**: 495 px, drawn from 360 to 1200 - **Default fields**: `languages` ![The language-bars layout played once: one full-width bar per language growing from its left edge, one after another, with the language name and its percentage arriving behind each bar](https://raw.githubusercontent.com/jmrplens/ghchronicle/main/site/src/assets/card-language-bars.svg) - **Binary** ```sh ghchronicle -card-layout language-bars -card-theme both -card-only -card card.svg -config config.yaml ``` - **GitHub Action** ```yaml - uses: jmrplens/ghchronicle@v2 with: token: ${{ secrets.GHCHRONICLE_TOKEN }} mode: card card: generated/card.svg card-layout: language-bars card-theme: both ``` ## Motion Each section above states its layout's motion, and one fact decides what that line can say: a reveal is never replayed. A number that has counted up, a bar that has grown and a line that has drawn itself are not taken back, so `-card-motion loop` draws exactly the card `once` draws on every layout but the two whose motion ends nothing, `terminal`'s blinking cursor and `ticker`'s scrolling band. Those two are the ones with a loop toggle under their picture, and the reasoning is in [A loop never replays](https://docs.jmrp.io/ghchronicle/card/#a-loop-never-replays). ## Width Each layout declares the width it is drawn at and the two ends it refuses to go outside, and its own section above states all three. `-card-width` on the binary, and `card-width` on the Action, ask for another: anything between that layout's own two ends. A width outside them is refused before the sweep runs, naming both, and `-card-layouts` prints them. Left out, a card comes out at its layout's own width, which is what every card came out at before the flag existed. `badge-row` declares none of the three, because a pill row stretched to a fixed width would have gaps in it; its width follows its content, and the flag neither changes it nor is refused by it. The near end is where a column stops fitting. The far end is usually only a guard against a typo, because a layout given more room spreads the same content over it, and one asked for twenty thousand used to be drawn twenty thousand units wide. `activity-heatmap` is the one with a real one, and it is the reason the ends are each layout's own rather than one pair for all of them: it reads the width rather than only being sized by it, working out how many weeks of the contribution calendar fit in the room the width leaves, so the same card is sixteen weeks at its near end, twenty-three at the width it declares and the whole year the collector keeps at its far end. Past that there is no more calendar to draw, so the far end is exactly the width where the year lands and the card is never asked to fill space it has nothing for. ## Fields a layout cannot draw Each layout declares which fields it supports. Asking for one it does not, such as `top_repos` on `github-compact`, drops it silently. Asking for a name that is not in the vocabulary at all is an error listing the valid ones. Twelve of the fifteen fields are numbers, and every layout takes all twelve. Only the three that need room of their own are restricted: | Field | Drawn by | | ------------ | ----------------------------------------------------------------- | | `languages` | `summary`, `github-stats`, `language-ring`, `language-bars` | | `top_repos` | `summary`, `github-stats`, `repo-list`, `terminal`, `ticker` | | `sparkline` | `summary`, `github-stats`, `wide-banner`, `sparkline-hero`, `activity-heatmap`, `animated-counters` | `ghchronicle -card-layouts` prints the layouts with the fields each one draws by default. ## Where to go next - [The card](https://docs.jmrp.io/ghchronicle/card/) is what draws these, and how to put one in a README. - [Calling it from a program](https://docs.jmrp.io/ghchronicle/reference/subprocess/) is the supported way to get one out of another language. --- # Rate limits GitHub runs fifteen independent budgets; three of them matter here, and the brake is scaled to each. Source: https://docs.jmrp.io/ghchronicle/api/ GitHub does not have a rate limit. It has fifteen of them, and every response says which one it just charged in the `x-ratelimit-resource` header. ## The three that matter here | Bucket | Limit | Spent by | | --------- | -------------------- | ---------------------------------------------------------------- | | `core` | 5000 per hour | Every REST call | | `graphql` | 5000 points per hour | Every GraphQL query: the pull requests and issues, the account, commits, discussions, deployments and the rest, which the [cost table](https://docs.jmrp.io/ghchronicle/api/cost/) prices family by family, and the query that asks which repositories moved | | `search` | 30 per minute | The commit count of `totals`, and nothing else: GraphQL search has no COMMIT type | Two more are charged by one family each: `webhook_deliveries` (500 a minute) by the delivery list of every hook in `settings`, and `dependency_sbom` (100 a minute) by the SBOM in `deps`. Neither is one of the fifteen `/rate_limit` reports; they exist only in the headers of the endpoints that charge them, and both are named in the [cost table](https://docs.jmrp.io/ghchronicle/api/cost/) where they apply. The brake looks at the three above, not at whichever bucket was charged last, which matters for the reason below. ## The brake `github.reserve_rate` is how many calls are never spent. It defaults to 500. The collector stops a family rather than crossing that line, so whatever else uses the same token keeps working. ```yaml github: token: ${GITHUB_TOKEN} reserve_rate: 500 ``` The reserve is scaled to each bucket: a fifth of its limit, or the configured value, whichever is smaller. > **The scaling is a bug fix, not a refinement** > > Search allows thirty requests a minute. One call to it leaves "29 remaining", > and comparing that against a flat reserve of 500 read as exhausted, so every > remaining family in the sweep was skipped. Judging a thirty-request bucket by > a five-thousand-request reserve stops the collector dead. When a bucket is below its reserve, the collector waits for the reset the response already told it about, rather than sleeping a guessed interval and retrying. In a normal sweep the family is skipped with a warning: ```text level=WARN msg="rate limit reserve reached, family skipped" family=artifacts ``` Once is fine. Every sweep means the cadences are too fast for the number of repositories; see [cost of a sweep](https://docs.jmrp.io/ghchronicle/api/cost/) for what to lengthen first. ## ETags, and why a 304 is free Every response is cached by its ETag. A repeat request sends `If-None-Match`, and GitHub answers 304 Not Modified when nothing has changed. **A 304 costs no quota at all.** That is not an optimisation detail, it is the thing that makes short cadences affordable. Most of what this collects barely moves: a repository's language breakdown, its topics, its community profile, the list of workflows. Asking for them every hour would be unaffordable if each question cost a call. Asking whether they changed costs nothing. The consequence is that the measured cost of a sweep in the next page is an upper bound reached on the first sweep and after a change, not the steady-state figure. What the cache keeps beside each ETag is the value the collector decoded, encoded again, not the body GitHub sent. A page of a hundred workflow runs is 1.4 MB of which the collector keeps a few hundred bytes per run, so a sweep's cache holds about a ninth of what the raw bodies would, and a 304 decodes a ninth of the bytes. A test runs every REST collector twice against a fake GitHub that answers the repeat 304 and fails on the first point that differs, which is what keeps the replay the same answer as the original. An entry is keyed by the URL and the type that decoded it, so the one URL two collectors read differently, `GET /repos/{owner}/{repo}` (four flags for discovery of a repository named in `targets.repos` that no listing returned, forty fields for the repo family), has an entry per reader and neither is answered from the other's. The cache is bounded, and the bound is 256 MB. It is an LRU, and an entry is charged the bytes it holds plus two hundred for the map slot, the list element and the struct around them, so a cache full of small answers is accounted at what it really costs rather than at half of it. There is no key for it in the configuration file: a deployment whose live set does not fit raises it from Go with `SetCacheLimit`, and `CacheStats().Evicted` is the number that says whether it had to, because it stays at zero for as long as one sweep's live set fits. Two things follow. A limit below one sweep's live set is not a smaller cache but no cache, since every sweep would evict what the next one is about to ask for. And the bound is resident memory once the cache is full, which is the number to know before giving the process a container with a memory limit: it can hold that much of decoded bodies on top of its own footprint. The cache outlives the process. What was asked for within twice the longest cadence is kept in [a file beside the state file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it), written at most every five minutes while the collector runs and once more when it stops, and a restart reads it back, so its first sweep asks with the validators the last one stored. Before that file, the first sweep after a restart of the author's service made 130 requests and not one of them was answered 304. An entry is kept under a digest of everything its type decodes rather than under the type's name, so an upgrade that changes what a collector reads asks that collector's URLs again in full instead of answering them with a body stored without the new field. ## A refusal is remembered too A 403 or a 404 is how GitHub says a feature is switched off: Dependabot on a repository that does not use it, code scanning where it was never enabled, a forum on a repository with discussions off, the community profile of a fork, which GitHub does not serve at all (all 28 forks of the account it was measured on answered 404, and all 39 other repositories 200). That is not an error, and the collector turns it into "there is nothing here". What it is not is free. A refusal carries no ETag, so where a page that did not change costs nothing, a feature that is off is charged in full on every sweep, for ever. So a refusal is remembered for a day, keyed by family, repository and endpoint, and the sweeps inside that day ask nothing. Two consequences worth knowing: - Switch a feature on and it is noticed a day later at the latest, not on the next sweep. - Restarting the process does not ask again. The refusals are kept with the ETag cache in [the file beside the state file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it), each until the end of its own day, and deleting that file is how to ask again straight away. A spent budget is a 403 as well and is never remembered as one: the client types it apart precisely so that a rate limit is not read as a feature that is off. ## An answer GitHub could not finish is asked once more Four statuses say GitHub did not finish an answer: a `502` or a `504` is the gateway in front of it giving up on an answer that did not come back in time, a `500` is the application giving up on it itself, and a `503` is a server that could not take the request just then. All four are intermittent. Measured on the author's own sweeps between 2026-09-11 and 2026-09-29, of 457,098 REST requests GitHub answered 50 with a `502`, each after ten to eleven seconds, 2 with a `504` and 9 with a `500`. Seven of those `500`s were the artifact listing of the repository with the longest artifact history, after eight and a half seconds: the same slow listing, giving up at the application's own limit rather than at the gateway's. The size of the page is not the cause: asking for one artifact took as long as asking for a hundred. So a REST request that answers any of the four is asked once more, two seconds later, before the collector hears about it. Of the 21 `502`s and `504`s asked again that way since 2.6.0, 20 were answered. A `500` has been asked again that way only by hand, once, and answered `500` again; it is asked again all the same, because it is the same listing giving up one step further in, and a retry that fails costs what a `502`'s does. Three things hold for that second attempt: - **It is charged.** A `500` and a `502` each cost a request from `core`, even when asked conditionally, where a `304` costs none. So the retry goes through the brake like any other request, and a budget already at its reserve is not spent on it. - **It is conditional.** It carries the same `If-None-Match` as the first attempt, so a page that has not changed still comes back as a free `304`. - **It is the only one.** Each attempt can hold the family for ten seconds, and a request that fails twice is logged and left to the next sweep as before. What the collector read before it is kept either way. A failed job's log is two requests: the API answers with a redirect, which is charged, and object storage sends the text. When storage answers one of the four, storage alone is asked again, on the same signed URL and without the brake, because it is not GitHub's API and spends no budget. The API is not asked for a new redirect. A GraphQL query is asked once more the same way, two seconds later and through the brake, with one answer left out. GitHub documents its GraphQL timeout as a `502` or a `504` once a query has run for more than ten seconds, and that one is a query too large for one request. The rest of the four are asked again as they were, on the same page: a `500`, a `503`, and a `502` or a `504` that came back sooner than ten seconds, which cannot be that timeout. Measured on the same sweeps, of 66,821 queries GitHub answered 42 with a `502` after 10.45 to 11.20 seconds, 9 with a `504` after 11.03 to 11.23, and 3 with a `503` after 0.70 to 1.08 seconds. Each of those three cost a point and was a page of a commit history, which is why a quick failure is not read as a query too large: the commit walk took one, at the time, as the end of what it could read, and each of the three stopped its walk there and reported success. A query that fails twice is a failure the collector reports, and the next sweep asks again; so is an answer that is not JSON and is not that timeout, which is no GraphQL answer at all. What is not asked again: - **A refusal.** Any `4xx`, and a `501` or a `505`, is the answer the same request gets every time. A spent budget is the brake's to wait out. - **A request that got no answer at all.** The only kind measured was a name that could not be looked up, six times, three of them on the way to a job log's storage. Each took ten seconds to fail, which is the resolver asking every server twice before it gives up, so the lookup had already been asked again. No connection refused, reset or TLS handshake timeout appeared, and a request whose kept-alive connection closed before it was answered is sent again by Go's HTTP client itself. - **GitHub's GraphQL timeout.** A `502` or a `504` to a query that ran for ten seconds means a query too large to answer in them, which the same query would meet again. The pull request, commit and co-authored pull request walks ask again on the same cursor at half the page while it is larger than ten, so a page of fifty is asked again at twenty-five, twelve and six, and a query about several repositories at once is asked again with half of them, down to one. What still times out at the smallest, and every other walk or single query, is a failure the collector reports with the rows it read before it, never the end of what there is to read: a backfill does not record that repository as walked, and the next sweep reads it again. Up to 2.6.3 a walk with no smaller page to ask took the timeout for the end of its data and reported success, which on the author's account stopped twelve commit walks, all at a page of fifty; asked by hand, twenty-five of the same commits answered in under five seconds. ## The budget in the log ```text level=INFO msg="rate budget" bucket=core remaining=4354 limit=5000 ``` Worth an alert on the warning above rather than on this line. A budget that dips is normal; a family skipped every sweep is a configuration problem. ## A backfill inverts the rule A [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/) is the opposite intention and says so. When a bucket runs out it waits for the window to reset rather than giving up, because what it was asked for is the whole walk, and a family skipped for want of budget would be a family left to walk again. Stopped half way, it loses little: a checkpoint beside the state file keeps its place, repository by repository, and [the same command carries on from there](https://docs.jmrp.io/ghchronicle/how/backfill/#stopping-one-and-picking-it-up-again). --- # Cost of a sweep The measured price of one sweep, per family, and which families to lengthen when the budget is tight. Source: https://docs.jmrp.io/ghchronicle/api/cost/ Measured on 2026-09-11 with the real binary against the live API, every family switched on and every request logged with the `x-ratelimit-resource` and `x-ratelimit-used` headers of its own response. **Cold** is the first sweep of a fresh install: an empty ETag cache, a month of workflow runs. A restarted service no longer starts there: it reads its cache back from [the file beside the state file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it), and only one that has lost that file pays the empty cache again. **Steady** is the third sweep of the same process a few minutes later, when about four fifths of its requests were answered 304 and cost nothing; the second sweep is the one that pays the fill-in the actions row describes. A 304 is free, and it is only ever available to REST: GraphQL carries no ETag, so its column is the same in both sweeps. The figures are per repository where the family asks per repository, and per sweep where it asks about the account. `core` is the REST bucket, `pt` a GraphQL point. Families with a request that is not `core` say which bucket it went to: `search` (30 a minute), `webhook_deliveries` (500 a minute) and `dependency_sbom` (100 a minute) each have their own. A row the round of 2026-09-11 changed describes the requests the code makes now and says what the two sweeps measured where that differs. There is no total row on purpose: a total belongs to one account and ages with it, so multiply the per-repository rows by the number of repositories `-list` prints and you have your own. ## Per family | Family | Cadence | Cold | Steady | Notes | | ------------ | ------- | ------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | account | 1h | 1 pt, 7 core | 1 pt, 1 core | one query: the calendar, the totals, the pins, the sponsors block and the star lists; 66 KB, never compressed, never conditional; beside it the profile, for the organizations GraphQL's `following` leaves out, which GitHub all but never answered with a 304, once in 48 conditional reads from 2026-09-12 to 2026-09-27, and the six package listings GraphQL cannot see, which answer one until a package changes | | achievements | 1h | 1 page off budget, 1 pt, plus the co-authored walk | 1 page off budget, 2 pt | the badges come from the public profile page, read without the token and charged to no bucket. The progress rows beside them do come from the API: one counts query, and a walk over the merged pull requests whose co-authored commits no count exposes. The first pass walks the whole life of the account, and since search stops at a thousand results it splits any range that overflows into two, one point per page and one per split, and one per query that reads on past the first hundred commits of a pull request with more, ten pull requests to a query, which is tens of points on a long-lived account and one on a new one: 35 queries and 23.7 MB over 2,315 public merged pull requests, measured on 2026-09-27. The count it finds is kept in the state file with the day it covers, so every later pass walks only the pull requests merged since, one page, 468 to 513 KB for each hourly pass the production proxy logged on 2026-09-27 on the same account, a size that grows through the day with what is merged, where the whole walk had been 18 to 24 MB every day; the whole history is walked again once a week, since a repository made private or deleted takes its pull requests out of the count. The page and the two points every hour, the whole walk once a week | | totals | 1h | 1 search, 1 pt, plus 1 pt per 10 repos and 1 per 25 archived ones set aside | the same | ten of the eleven counts are one GraphQL query of ten aliases, 406 bytes at cost 1, measured identical to the REST answers of the same minute; only the commit count is still a search, charged every time; a refused counts query is asked again one count at a time, ten points on that sweep only, so one failed search loses one number as it did in REST | | ratelimit | 15m | 1 pt | 1 pt | `GET /rate_limit` is free; the point is the GraphQL half of the same question | | events | 15m | 3 core | 1 core | before the round: three pages of a feed that changes every time, so `If-None-Match` never matches. Now: the walk stops at the page that carries the newest event of the previous sweep, remembered as `last_event` in the state file, which is usually the first page: 1 core a sweep, 192 a day fewer at the quarter hour; a first sweep and a backfill still read the three | | notifs | 15m | 20 core | 1 core | before the round: twenty pages of fifty; one new notification shifts every page, so the ETag never matches, for six or seven rows that were new. Now: `since=` the newest `updated_at` seen minus two cadences (`last_notified` in the state file), which at the quarter hour is one page for a window that opens half an hour before the newest thread seen: 1 core a sweep, and the twenty once a day (`last_full`), on a backfill, and on the first sweep after an upgrade, read threads included (`all=true`): that bounds to a day whatever GitHub's filter left out of the windowed reads, and it is the read that closes a thread, since a sweep lists unread threads only and a thread read without a reply leaves that listing rather than coming back with its `unread` tag changed. about 1,800 core a day fewer at the quarter hour | | billing | 1h | 2 core | 1 core | one per month walked; the month in progress changes, the previous one is a 304. Measured from 2026-09-13 to 2026-09-26, six hours apart, the month in progress answered 200 to all 46 conditional reads and the one before 304 to all 46 | | profile | 12h | 4 core plus 1 per package | 0 core | the profile, the social accounts, the gists, the packages and one page of versions per package; all 304 after the first sweep | | outbound | 1h | 9 pt, plus 1 per further hundred stars given, items still open or moved since the sweep before, or accepted answers | the same | all GraphQL: the starred list is a query of a hundred a page (13.5 KB against 617 KB decompressed from REST), newest first, up to five pages on every sweep and every page on a backfill, the five searches one point a page (7 KB against 100 KB), the two comment walks one each, and the account's accepted answers one more, read on their own so an answer accepted after its comment left the newest hundred is still written (15 of them on 2026-09-26, one page, and five pages at most on a sweep). The two searches for what is still open are read to the end on every sweep, one page until an account has more than a hundred open of a kind, and the three for what was merged or closed back to a cadence before the sweep before, ordered by what moved last, which is one page each unless more than a hundred moved in that time; a backfill reads those to the end too, ten pages at most, since search serves a thousand results and no more. Nothing here has an ETag on either path, so the two columns are the same | | history | off | 1 pt per past year | the same | the account's whole calendar, once, at a point per year it has existed | | keys | 24h | 2 core | 0 core | the SSH and the GPG keys | | traffic | 6h | 4 core per repo | 0 core | views, clones, referrers, paths; the fourteen-day window is a 304 until it moves | | repo | 1h | 3 core per repo, 2 pt per 10 repos | 0 core for a repo that did not change, 2 pt per 10 repos | the repository, its community profile and one page of releases each; languages, topics, rulesets and branch protection ride in one GraphQL query per ten repositories. A fork has no community profile, and the 404 it answers, charged in full because it carries no ETag, is asked once a day rather than on every pass: fifteen forks were charged 404 of them in 30.9 hours, and with the memory are charged 15 a day | | branches | 24h | 1 pt per 14 repos | 1 pt per 14 repos | one query per fourteen repositories | | stars | 1h | 1 core per repo, plus 1 core per 30 weeks of its life | 1 pt per 10 repos, plus 1 core per repo, usually a free 304 | the full walk through REST the first time a repository is seen, then the newest hundred of every repository in one GraphQL query per ten, about a kilobyte per repository; a restart with a state file starts at that query. Before the round the last page was asked of every repository on every sweep, all but one of them 304. The daily star history came after the round and is not in its figures: 1 core per repository, a 304 unless a day of the last thirty weeks gained or lost a star or a week began; its first read is a page per thirty weeks of the repository's life, 180 pages for the nineteen starred repositories of the account measured, once | | issues | 1h | up to 4 pt a page of a month of moves, 1 pt or more for open items | 1 to 2 pt per repo that moved | before the round: a page of fifty with ten review threads each, 8 points, which the gateway refused with a 502 once a sweep on the busiest repository. Now: 2 points of GraphQL per repo for what changed in two cadences, ten at a time (1 point where the repository holds five or fewer). The first sweep of each UTC day also reads every open item of every repository, however long ago it moved, in pages sized from the open counts of the last totals, and what moved since the day's read before it, a month back on a new state file, in pages of up to twenty-five (5, 10, 20, 25 or 50 cost 1, 2, 3, 4 or 9, asked with `dryRun` on 2026-09-30). Until 2.6.5 the day's read was one page of the fifty items of any state updated last, which missed an open item once fifty others had moved past it, and met the gateway's ten seconds on the busiest repository every day. Since 2.6.0 the read of what changed asks only the repositories where an issue or a pull request was updated in its window, which one query for all of them says first (see [Asking first what moved](https://docs.jmrp.io/ghchronicle/api/cost/#asking-first-what-moved)); the read of every open item is made in every repository | | issueevents | 1h | several pt per repo updated in the month, plus 1 core per stacked pull request in it | 1 pt per repo that moved, 0 to 1 core | before the round: one page of a hundred events each, up to a megabyte per repository because every event embeds its whole issue; a 304 on all but the repository that moved. Now: one GraphQL query per repository, the timeline of the ten most recently updated issues and ten pull requests for the events of the last two cadences (1 pt, 2 to 6 KB, a second page only when more than ten items moved), plus one core per pull request in a stack, whose `added_to_stack` event the timeline cannot name; thirty days on a first sweep, once, which is minutes rather than seconds where the pull requests are nearly all stacked, because almost every one updated in the month takes the per-issue road for its `added_to_stack`; a steady sweep is a point per repository that moved and one such read at most. And from a cadence before the last run after a gap, so a stopped process does not leave its hours out of the series. Measured against the list over a week of two repositories: every event of every type agrees, field for field, except a commit referencing an issue nobody has touched, 3 of 2,217, which does not move the issue and so is not asked for. A backfill walks `/issues/{n}/events` per item instead of the list, 1.4 KB compressed per twelve events against 45 KB per event. Since 2.6.0 a sweep asks only the repositories where an issue or a pull request was updated in its window (see [Asking first what moved](https://docs.jmrp.io/ghchronicle/api/cost/#asking-first-what-moved)) | | actions | 15m | a month of runs per repo, plus 1 core per run | 1 core per repo that had a run, plus 1 per run completed since | before the round: pages of a hundred runs reaching a month back, a job list per run, and the caches and workflows of every repository, which was three fifths of the cold sweep's bytes, and those job lists again on every sweep, nearly all of them 304. Now: pages of 30, one on a quiet repository and up to 7 while they come full of runs newer than the window, plus one per run not yet expanded, jobs listed once per attempt, at most 20 new runs a sweep; pages of 100 on the first sweep and in a backfill. Measured over three sweeps of one process, the second is the one that pays the fill-in, because the page of thirty is a new URL for every repository and holds runs the first sweep's twenty did not cover; from there a sweep costs one page per repository that had a run and one job list per run completed since, so what it costs is how many runs the account completes. The cache listing is read in pages of a hundred entries, up to ten, so only a repository past a hundred pays for a second, and a page is a 304 while the listing has not changed: from 2026-09-25 12:58Z to 2026-09-27 01:39Z the account's listings answered 304 to 3,059 of 3,256 requests | | artifacts | 1h | up to 5 core per repo | 0 to 5 core per repo | a busy repository fills its pages, and all five of them are charged again whenever an artifact was added or expired: 5 on one sweep, 0 on the next | | security | 1h | 2 core per repo | 0 core | Dependabot and code scanning alerts; most are the 403 of a repository with Dependabot switched off or the 404 of one without code scanning, and a refusal carries no ETag, so each was charged again on every sweep before the round remembered them. Now: a refusal is answered from memory for a day, per family, repository and endpoint, so the steady figure is 0 core on the second sweep and one request per refusal once a day. The rest are conditional requests, all 304. Only a list whose first page comes back full, in a repository past a hundred alerts of that kind, asks for more: its open alerts, a page per hundred still open, and for code scanning a page of one alert whose last page is the total. Measured on 2026-09-26 that is two lists of this account and three requests a sweep, conditional like the rest and a 304 while nothing moves | | stats | 12h | 3 core per repo | 0 core | participation and the punch card; GitHub recomputes them slowly and answers 304 | | discussions | 1h | 2 pt per repo with a forum | the same | before the round: fifty threads with twenty comments and twenty replies each, 11 points, asked of every repository, including every one with no forum. Now: 2 points per repo with a forum, the ten most recently updated threads with the same twenty comments and twenty replies each (comments come oldest first, so a shorter page there would stop recording the eleventh comment of a thread); a repository whose forum is off is never asked | | commits | 1h | 1 pt per repo with a commit in the month | 1 pt per repo that moved | the last two cadences of the default branch, since 2.6.0 only in the repositories whose head was committed in them (see [Asking first what moved](https://docs.jmrp.io/ghchronicle/api/cost/#asking-first-what-moved)); a month back on the first sweep, which is a few hundred kilobytes for a busy repository; where the gateway gives up on that page of fifty, as it did on the busiest repository measured, it is read in pages of twenty-five | | activity | 15m | 2 core per repo | 0 to 2 core per repo | the repository log, a hundred entries per page; charged only for a repository whose log moved, 2 on one sweep and 0 on the next | | analyses | 1h | 1 core per repo | 0 core | most are the 403 and 404 of repositories without code scanning, charged again each sweep before the round; now remembered for a day, so the steady sweep is conditional requests and 0 core | | forks | 12h | 1 core per repo | 1 pt per 10 repos | one page each through REST on the first sweep of a fresh install and in a backfill, then the newest hundred of every repository in one GraphQL query per ten, under a kilobyte per repository; a repository the batch reports holding more than a hundred forks is walked through REST as well, since a fork row's stars and last push move and the batch cannot refresh the rows past its page. Before the round: one request per repository a sweep, all 304 | | planning | 6h | 1 pt per repo | 1 pt per repo | labels and milestones | | joblogs | off | 1 core per repo, 1 blob per failed job | 0 core | the failed runs of the last hour per repository, asked of a list filtered to the month a re-run can reach back to, and one blob per failed job from object storage, outside the API's quota; the filter's URL changes once a day, so a day costs one charged page per repository and the rest are 304 | | settings | 6h | 2 core per repo, 1 webhook_deliveries per hook | 0 core, 1 webhook_deliveries per hook that moved | webhooks, environments and deploy keys; the deliveries of every hook are charged to their own bucket | | rulesets | 24h | 1 core per repo plus 1 per ruleset | 0 core | one list per repository and one history per ruleset, both with an ETag; none of it charged on a day nobody edited a ruleset, when the steady sweep asks the same questions and every one of them is conditional | | inventory | 24h | 4 core per repo | 0 core | the workflow token policy, the secrets, the code scanning setup; the refusals were the 403 of repositories without it, now remembered for a day, which at this cadence is the next sweep anyway; the rest are conditional requests, all 304 | | deployments | 30m | 1 pt per 5 repos | 1 pt per 5 repos | one query per five repositories, the newest hundred deployments each | | policyfiles | 24h | 1 pt per 5 repos | 1 pt per 5 repos | one query per five repositories, the history of four paths each | | deps | off | 1 core and 1 dependency_sbom per repo | 0 core, 1 dependency_sbom per repo that received a commit | one commit read and one SBOM per repository; the SBOM has its own bucket and most were 404, now remembered for a day. GitHub regenerates a SBOM on every read and its ETag never matches, so the photograph is taken only when the head moved since the last sweep (the head itself is a free 304 when it did not): 0 dependency_sbom on a repository without a commit, one per repository that received one, and one of the SBOM reads timed out on GitHub's side on each of the three sweeps measured | What weighed before the round was not REST but GraphQL: `issues` and `discussions` were 88 % of the points, because both queries were sized for a backfill and asked on every sweep. The only search is the commit count of `totals`, one a sweep against a budget of thirty a minute. The round of 2026-09-11 lowered several of these rows, and the first measurement is the baseline it is measured against: the job lists a run has already had expanded are not asked for again, the runs page is thirty on an ordinary sweep, `discussions` is asked only of repositories with a forum and for ten threads, `pulls` is sized to the repository and windowed to two cadences, notifications and the event feed stop at what the previous sweep saw, a 403 or 404 is remembered for a day instead of being charged again every hour, and the newest stars and forks, the starred list, the outbound searches and ten of the eleven totals counts moved from REST to GraphQL, the same rows for a point apiece instead of a request apiece. Measured again after the round, three sweeps of one process on the evening of the same day, the steady sweep charges about a third of the `core` and a third of the GraphQL points it charged before, moves half the bytes on the wire, and answers about four fifths of its requests with a 304. Projected to a day at the cadences of the time that is roughly an eighth of the REST calls and a third of the points, plus one job list per workflow run completed, which is the one term that grows with how busy the repositories are rather than with how many there are. Two things did not fall. The cold sweep charges more `core` than before, because the first sweep of issue events now reads the whole month through the timeline and, where pull requests are stacked, one per-issue list for nearly every pull request updated in it: minutes, once per process. And a sweep that runs every family at once still takes minutes rather than seconds, because its conditional requests cost a third of a second each and its GraphQL queries nine tenths, one after the other; the sweeps production runs are the 15 minute one and the hourly one, each a fraction of the whole. The cadences of 2.6.0 run ten families more often, `events`, `notifs` and `activity` every quarter of an hour, `deployments` every half hour, and `stars`, `billing`, `analyses`, `account`, `totals` and `discussions` every hour, because their extra passes cost next to nothing. Costed from the request log of the production process from 2026-09-25 12:58Z to 2026-09-26 19:50Z, 37 repositories, each family's first pass left out so that every pass counted had a warm ETag cache, they add about 250 `core` requests and 500 GraphQL points a day, 0.2 and 0.4 per cent of the two daily budgets, and 22 search requests. The points are mostly `deployments`, eight a pass there, and `totals`, six; the `core` requests mostly `activity`, `events` and `analyses`, which are charged only for what moved, since a 304 is free: in the same log, a `core` request answered 304 left `x-ratelimit-used` where the request before it in the same rate window had put it 11,692 times out of 11,771. Two more families run every hour since the same release, each priced in its own row above: `outbound`, which ran every twelve, and `achievements`, which ran once a day while every pass walked the whole merged history. ## Asking first what moved `commits`, `issueevents` and the incremental pass of `issues` read only what moved since a window of their own, one GraphQL query per repository, and GitHub charges a query for the page it asks for, not for what comes back. So a sweep that runs any of the three first asks every repository, twenty-five to a query at one point each, when the head of its default branch was committed and when its newest issue and its newest pull request were updated, and each family leaves unread a repository where nothing moved since the start of its own window. That is the condition under which the read comes back empty, not a guess at it: measured on 2026-09-27, `history(since:)` holds a head committed at the very second it names and nothing one second later. A repository the query does not answer for is read as it always was, and neither the day's read of every open item of `issues` nor a backfill asks it at all. No row of the table above carries the query, since it belongs to none of the three: it is one point per twenty-five repositories on every sweep that runs any of them. A family that leaves repositories unread says how many, once a pass, and its [`gh_collector_family`](https://docs.jmrp.io/ghchronicle/collectors/measurements/#configuration-and-delivery) row counts them in `repos` beside the ones it read, with no rows written for them: ```text level=INFO msg="nothing moved since the window, repositories left unread" family=commits repos=34 ``` A query that failed, or answered for some repositories and not the others, is said at warning, and every repository it gave no answer for is read as if it had not been asked: ```text level=WARN msg="which repositories moved could not be asked, reading those it did not answer for" answered=36 repos=37 err="graphql: the gateway gave up (502); the query is too large for one request (36 repositories were asked successfully)" ``` From the request log of the production process between 2026-09-25 12:58Z and 2026-09-26 19:50Z, 27 passes of each family over 37 repositories, before the query existed: 961 of the 999 `commits` answers held no commit, and 468 of the 968 incremental `issues` answers and 486 of the 1,005 `issueevents` answers held no item at all, 2,077 points in all, 43 per cent of the 4,786 the process spent on GraphQL. For those 37 repositories the query is two points a sweep, 54 over the 27 sweeps, so it saves 2,023 points, about 66 an hour: 42 per cent of all the GraphQL points and 52 per cent of the three families'. That is a floor. An answer is counted empty when it is under 300 bytes, the budget block and an empty list; a repository whose issues and pull requests all predate the window answers with a page of them anyway, is not counted here, and is skipped all the same. ## GraphQL is the cheap one, by a wide margin One query returns the full 366-day contribution calendar, every contribution total, the per-repository commit breakdown and the social counts, for **one point of a five thousand point budget**. The same data over REST would be dozens of calls and would not include the calendar at all, because the calendar exists nowhere else. That is why the account family can run every hour and still cost almost nothing, and why the expensive families are the REST ones that scale with how busy the repositories are. ## What scales with activity, not with size Everything else costs a roughly fixed number of calls per repository. These do not: - **`actions`** costs a page of thirty runs, up to seven while they come full of runs newer than the window, plus one request per run whose jobs this process has not written yet, at most twenty a sweep. A repository with continuous integration on every push generates runs continuously; a quiet one costs the one page, answered 304, and nothing else. - **`artifacts`** walks up to five pages per repository, and a busy repository fills them and has them charged again whenever an artifact was added or expired. - **`activity`** is charged only for a repository whose log moved: the page of one that did not is a free 304. - **`commits`**, **`issueevents`** and the incremental pass of **`issues`** read only the repositories where something moved in their window, a point or two each, after [asking which](https://docs.jmrp.io/ghchronicle/api/cost/#asking-first-what-moved) for one point per twenty-five repositories. The day's read of every open item of `issues` is made in every repository, busy or not. - **`outbound`** costs a point more for each further hundred of what it reads: stars given, items still open or moved since the sweep before, and accepted answers. > **What to lengthen first** > > If `core` is the budget that runs short, lengthen `artifacts` and then > `actions`; if it is `graphql`, lengthen `issues`, `issueevents` and > `commits`. Each of them costs more the busier the repositories are, so > lengthening its cadence buys back a variable amount rather than a fixed one. ## Switching a family off Set its interval to `0`. ```yaml every: families: artifacts: 0 joblogs: 0 ``` A family that is off writes nothing and costs nothing. Its dashboard panels go empty, which is the honest reading. See [cadences](https://docs.jmrp.io/ghchronicle/configuration/cadences/). ## Reading the arithmetic for your own account A first sweep is the expensive one: the full stargazer walk, the whole daily star history of every repository, a month of workflow runs, the co-authored pull requests of the account's whole life, which `achievements` walks again once a week (35 queries and 23.7 MB over 2,315 public merged pull requests, measured on 2026-09-27), and (with `every.families.history` set) every past year's contribution calendar. After that, multiply the per-repository rows above by the number of repositories `-list` prints, and divide the hourly budget by the cadence. A [card](https://docs.jmrp.io/ghchronicle/card/) runs every family whatever the cadences say, because every number it draws comes from that one sweep, so N cards are N sweeps, and a workflow drawing three of them pays three. Its GraphQL is a whole sweep's every time, since GraphQL has no ETag. Its REST requests are the cold column's only the first time: every run reads the cache file beside its `state_file` and asks with the validators stored there, so from the second card on a page that did not change is a free 304. A run whose state file is not kept from one run to the next pays them cold every time: a container with nothing mounted where the file goes, or the Action given no `config:`, which starts each run with a state directory of its own. A `-card-only` run reads the cache file and never writes it, so it is warm only when another run keeps that file. The signal that the sum came out wrong is a warning, not a guess: ```text level=WARN msg="rate limit reserve reached, family skipped" family=actions ``` Once in a while is fine. Every sweep means the cadences are too fast for the number of repositories. --- # What GitHub will not give The endpoints that are verified not to work on a personal account, written down so nobody rediscovers them. Source: https://docs.jmrp.io/ghchronicle/api/limits/ Every entry here was checked against the live API from a personal account, and most of it is not in GitHub's documentation, which describes what an endpoint should return rather than what it does. It is written down so nobody spends an afternoon finding it out again, and so that a missing panel can be told apart from a broken collector. ## Why do two statistics endpoints never answer? `stats/code_frequency` and `stats/contributors` answer **202 with an empty body, indefinitely**, on a personal account. A 202 normally means "still being computed, ask again", and for these two the next answer is another 202. They are deliberately not called. The lines added and removed that `code_frequency` would have given come from the commits collector instead, per commit rather than per week, attributed to an author and dated to the commit. `stats/participation` and `stats/punch_card` do work and are used. Ten seconds is all it takes to see it on your own account: ```sh curl -s -o /dev/null -w '%{http_code}\n' \ -H "Authorization: Bearer $GITHUB_TOKEN" \ https://api.github.com/repos/OWNER/REPO/stats/code_frequency # 202, for ever curl -s -o /dev/null -w '%{http_code}\n' \ -H "Authorization: Bearer $GITHUB_TOKEN" \ https://api.github.com/repos/OWNER/REPO/stats/participation # 200 ``` ## Billing | Endpoint | Answer | | --------------------------------------- | -------- | | `/settings/billing/actions` | 410 Gone | | `/settings/billing/packages` | 410 Gone | | `/settings/billing/shared-storage` | 410 Gone | | `/user/settings/billing/usage` | 404 | | `/users/{login}/settings/billing/usage` | works | Only the last form works for a personal account, and it returns full RFC 3339 timestamps in a field its documentation describes as a date. ## Organisation and enterprise only Custom repository properties, classic projects, cost centres and the audit log. A personal account cannot see any of them, however the token is scoped. ## Endpoints that answer, but say nothing - **`workflows/{id}/timing`** returns 200 with an always-empty `billable` object. It looks like the source for per-workflow minutes and is not. - **The stargazer list, to anyone but the repository's admins and collaborators.** [Since July 2026](https://docs.github.com/en/rest/activity/starring#new-access-restrictions) GitHub serves it to no one else. REST answers 404, and GraphQL's `stargazers` answers an empty list with a `totalCount` of 0 while `stargazerCount` beside it still gives the real number: measured on `cli/cli` and `octocat/Hello-World` with a token holding every scope. ghchronicle therefore writes `gh_star`, who starred and when to the second, only where the token has that access, which it always has on the account's own repositories and on those of an organisation the account administers; one named in `targets.repos` or reached through `targets.orgs` without it has its stars counted per day from the history below, and nobody named. - **`/user/installations`** returns 403 without a GitHub App. ## The star history github.com serves to anyone [`stargazers/history`](https://docs.github.com/en/rest/activity/starring#get-repository-star-history) answers where the stargazer list does not. It serves anyone who can see the repository, unauthenticated too: the stars given per day, grouped by week, thirty weeks to a page, which is both the default and the most `per_page` allows (a smaller one is honoured, a larger one is cut to thirty), paging back to the repository's first week (measured on `cli/cli`: thirteen pages, back to 2019). It names no stargazers, so it cannot stand in for `gh_star`, which is one row per star at the instant it was given and says who gave it. It can stand in for the count, and ghchronicle reads it for every repository it collects, writes it as `gh_star_day` and draws from it the panels that count stars by day in every store but Prometheus, which cannot hold a history and still counts `gh_star`. Three things about it were measured rather than read in its documentation. The days are GitHub's calendar days in America/Los_Angeles, under a `week` labelled as Sunday 00:00 UTC: against the stargazer lists of nineteen repositories, the Pacific day matched all 440 stars and the UTC day did not. It counts the repository's current stargazers, each on the day they starred, so an unstar takes a star off a day in the past rather than the day it happened. And a 304 answer to it carries no `Link` header, where a 200 carries `next` and `last`, so the walk reads no `Link` at all and stops at a page shorter than thirty weeks or an empty one. ## GraphQL is wrong about packages GraphQL reports zero packages for an account while REST lists them. The pretty query is simply wrong here, so packages come from REST, at one call per package for its versions. ## Traffic goes stale rather than empty A repository with no traffic does not return an empty window. GitHub keeps returning the last fourteen days that _had_ data, so the window can end weeks ago. The collector records what it is told. > **This is why gh_security_feature exists** > > Several of the entries above have the same shape: an endpoint that answers > nothing is indistinguishable from a feature that is switched off, which is > indistinguishable from a repository with nothing to report. > `gh_security_feature` records explicitly which features are enabled, so "no > alerts" and "no data" stop looking alike on a dashboard. ## What this means for a sweep None of the above is treated as a failure. `ghapi.UnavailableError` (403 or 404: the feature is switched off) and `ghapi.NotReadyError` (202: GitHub is still computing) both mean "there is nothing here", and the sweep continues to the next repository. The activity feeds have their own version of this: past their ceiling GitHub answers **422 "pagination is limited for this resource"**, which is read as the end of the data rather than as an error. ## Where to go next - [Rate limits](https://docs.jmrp.io/ghchronicle/api/) is the other half of this: what the calls that do work cost, and why a 304 costs nothing. - [Cost of a sweep](https://docs.jmrp.io/ghchronicle/api/cost/) prices every family. --- # The command line The ghchronicle command line: the flags, what each one does, and which of them print something and exit. Source: https://docs.jmrp.io/ghchronicle/reference/cli/ The binary takes these flags and no subcommands. Everything else is in the configuration file, because a schedule is not something to retype. ```sh ghchronicle -config /etc/ghchronicle/config.yaml ``` ## Every flag | Flag | Default | What it does | | ----------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `-setup` | off | Ask what a working configuration needs, check each answer against the thing it names, and write it | | `-config` | `config.yaml` | Path to the configuration file | | `-once` | off | Run one sweep and exit instead of scheduling; a family that is not due by its cadence is still skipped | | `-list` | off | Print the repositories that would be collected, and which are set aside, then exit | | `-version` | off | Print the version, the commit and the build date, then exit | | `-groups` | off | Print the groups with the families in each, then exit | | `-backfill` | off | Reach as far back as each surface allows, waiting for the rate limit to reset rather than stopping | | `-backfill-since` | none | Bound the backfill: a date (`2024-01-01`), a duration (`720h`), days (`90d`) or years (`2y`) | | `-backfill-status` | off | Print how far the backfill in progress, and a migration's refill, have got, then exit; it asks GitHub nothing, writes nothing, and needs no token | | `-backfill-retry` | `0` | After a backfill, or a migration's refill, ends with families left, wait this long and go back for them, until a pass records nothing new; `0` does not go back | | `-families` | none | With `-backfill`, walk only these families, comma separated, the names `-groups` prints; a name that is not a family, or the flag without `-backfill`, is refused with 2 | | `-publish-dashboard` | off | Publish the Grafana dashboard and the datasource it reads from, then exit; it needs the `grafana` section, asks GitHub nothing and needs no token | | `-uninstall` | none | Remove what this put in place and exit: `dashboard`, `data`, `state`, `all`, comma separated; it lists and removes nothing without `-yes` | | `-yes` | off | Go ahead with `-uninstall` or `-migrate` rather than only listing what it would do | | `-migrate` | off | Print, for every configured store, what an earlier release left there in a shape this one no longer writes and what bringing it along would take, then exit; it changes nothing without `-yes`, and with it applies every pending change and reads what it cleared again | | `-migrate-others` | off | With `-migrate -yes`, also apply a change to a store holding rows of accounts this configuration does not collect, or whose rows could not be compared with it; without `-migrate` it is refused with 2 | | `-card` | none | Run one sweep and write a summary SVG to this path; that sweep runs every family, whatever the cadences say | | `-card-only` | off | With `-card`, write the SVG and nothing else: no sink is needed, none is written to, and the state file is left as it was | | `-card-theme` | `auto` | `dark`, `light`, `auto`, or `both`: the light card at `-card` and the dark one beside it with `_dark` before the extension, from one sweep | | `-card-motion` | `once` | `once`, `loop` or `off`; `loop` changes only `terminal` and `ticker` | | `-card-layout` | `summary` | Which of the thirteen [layouts](https://docs.jmrp.io/ghchronicle/card/layouts/) to draw | | `-card-fields` | none | Comma-separated fields the card shows, from [the fields](https://docs.jmrp.io/ghchronicle/card/#the-fields); empty means the layout's own default | | `-card-width` | the layout's | Card width in pixels. Each layout draws between two ends of its own, which its [section](https://docs.jmrp.io/ghchronicle/card/layouts/) and `-card-layouts` both state; `badge-row` ignores it, its width following its pills | | `-card-speed` | `0.5` | How fast an animated layout plays, as a decimal from 0 to 1. `0` is the slowest animation and `1` the fastest; `0.5` is the pace every card has always been drawn at. `0` is not a still card, `-card-motion off` is | | `-card-layouts` | off | Print the layouts with the fields and the widths each draws, then exit | ## The five that print and exit `-version`, `-groups`, `-card-layouts`, `-list` and `-backfill-status` answer and stop. The first three need no token and no configuration; `-list` reads the configuration and asks GitHub which repositories the targets resolve to, which is the cheap way to check a change before spending quota on a sweep. `-backfill-status` reads the configuration to find the [backfill checkpoint](https://docs.jmrp.io/ghchronicle/how/backfill/#seeing-how-far-it-has-got) and prints how far that walk has got, and the refill of a migration after it when one is in progress; it asks GitHub nothing, so it needs no token. ```sh ghchronicle -version ghchronicle -groups ghchronicle -card-layouts ghchronicle -config config.yaml -list ghchronicle -config config.yaml -backfill-status ``` ## What an upgrade left in the stores ```sh ghchronicle -config config.yaml -migrate ``` `-migrate` checks every configured store against the list of changes the binary carries, each a measurement an earlier release wrote under a key this one no longer uses, and prints one block per store: what each change finds there, the evidence, what bringing the store along would take and what would be read again from GitHub. It changes nothing. The stores are asked questions, GitHub is asked for the repository list alone, and the state file is read and never written, so it exits 0 whatever it finds. Without a token it still runs and says what it could not compare. [Migrations](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations) says how each store is decided and what each line means. When something is pending, the plan ends with the command that applies it and with what a start does about it under the [`migrate`](https://docs.jmrp.io/ghchronicle/configuration/#migrate) setting. That command, and every other one ghchronicle prints for you to run next (the resume line of `-backfill-status`, the warnings of a start, the last line of `-setup`), names the configuration as it was given, quoted when a shell would read part of the path: in single quotes on Linux and macOS, and in double quotes on Windows, the one form PowerShell and `cmd` both read as a single argument. ```sh ghchronicle -config config.yaml -migrate -yes ``` With `-yes` it prints the same plan and then applies every pending change, the ones marked as needing somebody's word too, reads what it cleared again from GitHub, and says under the plan what happened to each. Reading it again is one backfill of the families that write what was cleared, writing that alone into the stores it was cleared from; `-backfill-retry` goes back for what it leaves, and where a store kept a copy of the old rows, the report compares the two and names what GitHub no longer serves. One cut short keeps a checkpoint of its own, and the same command resumes it, even with nothing left to apply: [reading the history back](https://docs.jmrp.io/ghchronicle/install/upgrading/#reading-the-history-back). A store holding rows of accounts this configuration does not collect, or whose rows could not be compared with it, is held back unless `-migrate-others` is given as well. It needs a token and the repository list, and it takes the state file for as long as it runs: with the service running it refuses, naming the process, and changes nothing, so stop the service first, and pause a cron job running `-once`. With the SQL sink on standard output, the plan and the report go to standard error, so standard output is the SQL alone. What it applies is recorded as it goes, so running it again after a failure carries on without doing anything twice. | Exit | When | | ---- | --------------------------------------------------------------------------------------------------------------------- | | `0` | `-migrate`, whatever it finds; `-migrate -yes` when everything pending was applied and read again, or nothing was | | `1` | `-migrate -yes` left something: a store that refused or did not answer, one held back, a refill that did not finish; or it refused to start | | `2` | A command line that does not parse, `-migrate-others` without `-migrate` among them | ## The three ways to run a sweep ```sh ghchronicle -config config.yaml # the loop: each family on its cadence ghchronicle -config config.yaml -once # one sweep, in the foreground, then exit ghchronicle -config config.yaml -backfill # the history walk, once ``` The loop is what a service runs. `-once` is what a scheduled job runs, and it is also the quickest way to see what a configuration change does. A [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/) is a different intention and says so: it walks every surface to the end and waits for a spent budget to refill rather than giving up. ```sh ghchronicle -config config.yaml -backfill -backfill-since 2y ``` `-families` narrows a backfill to the families named, into every configured store: [some families only](https://docs.jmrp.io/ghchronicle/how/backfill/#some-families-only). ```sh ghchronicle -config config.yaml -backfill -families discussions,outbound ``` ## The card, in one line ```sh ghchronicle -config config.yaml -card profile.svg -card-only \ -card-layout github-stats -card-theme dark ``` `-card-only` is the one combination worth remembering: it makes a run that writes no points, so it needs no sink configured and a configuration that would otherwise be refused at start-up is accepted. [The card](https://docs.jmrp.io/ghchronicle/card/) has the layouts and the fields. `-card-width` is the one flag that changes what a card says rather than only how it looks, and on one layout only. [`activity-heatmap`](https://docs.jmrp.io/ghchronicle/card/layouts/#activity-heatmap) spends the room on data: sixteen weeks of the contribution calendar at its near end, twenty-three at the width it declares, and the whole year the collector keeps at its far end, which sits exactly where the year lands so the card is never asked to fill space it has nothing for. Every other layout spreads the same content over whatever width it is given, so widening one of those buys proportions and not information, and its far end is only a guard against a typo. A width outside a layout's two ends is refused before the sweep runs, naming both, and `-card-layouts` prints them for every layout. ```sh ghchronicle -config config.yaml -card calendar.svg -card-only \ -card-layout activity-heatmap -card-width 700 ``` ## The speed, and what its slow end is not `-card-speed` is one number for the whole card. Every animated layout scales together, the continuous motions with the rest: the ticker's band takes longer to come round and the terminal's cursor blinks more slowly at the same setting. It is one knob and not one per layout because the motion the card has was paced against itself, one layout's cycle chosen beside another's, and a reader who finds the band slow finds the typing slow with it. `0.5` is the middle of the range and is exactly the card this renderer has always drawn, to the byte, so leaving the flag out and asking for `0.5` are the same command, and each end reaches the same distance from it: `0` draws the animation twice as long as the default, `1` half as long as the default. > **0 is the slowest animation, not none** > > A range that starts at zero reads like a switch, and this one is not. At `0` > the card still animates, as slowly as this renderer will draw it. What draws > a card with no animation at all is `-card-motion off`. ```sh ghchronicle -config config.yaml -card slow.svg -card-only \ -card-layout ticker -card-motion loop -card-speed 0.25 ``` ## Where the rest lives Everything that is not in that table is a configuration key, not a flag: [the file](https://docs.jmrp.io/ghchronicle/configuration/) is the map of them. --- # Troubleshooting The messages that look like errors and are not, the ones that are, and the data that looks wrong and is not. Source: https://docs.jmrp.io/ghchronicle/reference/troubleshooting/ ## Things that look like errors and are not **`not available (403)` or `(404)`.** The feature is switched off for that repository, or the token cannot see it. Dependabot, code scanning, discussions and the dependency graph all answer this way when disabled. The collector records the fact and moves on: a repository with a feature off must not stop the sweep for the other forty. If it is _every_ repository rather than one, it is the token. Traffic needs push access and, on a fine-grained token, the repository permission Administration (read); alerts need `security_events`. See [the token](https://docs.jmrp.io/ghchronicle/start/token/). **A feature switched on, and nothing collected from it.** Code scanning enabled this morning, Dependabot turned on, a forum opened: the collector asked before you did it, was refused, and remembers the refusal for a day rather than paying for it on every sweep. It is noticed a day later at the latest, a restart included; deleting [the cache file beside the state file](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it) asks again straight away. See [a refusal is remembered too](https://docs.jmrp.io/ghchronicle/api/#a-refusal-is-remembered-too). **`still being computed by GitHub (202)`.** GitHub computes the `stats/*` endpoints asynchronously and answers 202 with an empty body while it works. The next sweep usually gets the numbers. Two of them never do. On a personal account `stats/code_frequency` and `stats/contributors` return 202 with an empty body indefinitely, which is why this project does not call them: the lines added and removed come from the commits collector instead. **`pagination is limited for this resource` (422).** The end of an activity feed, not a failure. GitHub serves three pages of the event feed and refuses the fourth. **A repository with no traffic showing a window that ended weeks ago.** GitHub keeps returning the last fourteen days that _had_ data, not the last fourteen days. The collector records what it is told. **`cache file not read, the first pass of each family pays in full`.** The cache file beside the state file was cut short, damaged or written in another format, so it is not read at all, and the next save writes over it. It costs one pass of each family at full price and loses nothing; see [the cache beside it](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it). **`which repositories moved could not be asked, reading those it did not answer for`.** The query that tells `commits`, `issues` and `issueevents` which repositories to read failed, or answered for some of them. Every repository it said nothing about is read as if it had not been asked, so nothing is missed and the pass costs what it did before the query existed; see [asking first what moved](https://docs.jmrp.io/ghchronicle/api/cost/#asking-first-what-moved). **`outbound search read fewer items than it counts, GitHub serves a thousand at most`.** GitHub serves a thousand results of a search and no more, and the account has more than that in one of the states `outbound` searches, merged pull requests for instance. The thousand that moved most recently are kept, and the line is said once per count; see [what a backfill reaches that a sweep does not](https://docs.jmrp.io/ghchronicle/how/backfill/#what-it-reaches-that-a-sweep-does-not). **`NOTICE: column ... already exists, skipping` from psql.** Replaying the SQL file sink's file into a database that already has its tables. The file declares every table and every field column it writes, since it cannot ask which exist, and PostgreSQL says so for each one that does, with a `relation ... already exists` for the table. Nothing failed. **A family that never appears in the log.** It is not due yet. With a twelve-hour cadence, half a day of logs can legitimately never mention `stats`. A family of six hours or more can also be due and waiting its turn behind another one, which the log names under `waiting` in `slow families due together take turns`; see [the slow families take turns](https://docs.jmrp.io/ghchronicle/configuration/cadences/#the-slow-families-take-turns). ## Things that are errors **`github.token is empty and GITHUB_TOKEN is unset`.** Exactly what it says. **`every.families.: unknown collector`.** The name is not a family. The message lists the thirty-four that exist, in one parenthesis after the colon. **`groups: is empty`.** `groups: []` would collect nothing at all. Omit the key to collect everything, which is what it means when it is absent. **`groups[N]: "" is not a group`.** The name is not a group. The message lists the ones that exist, and `ghchronicle -groups` prints each with its families. **`groups[N]: "" is a family, not a group`.** Families and groups are both lowercase nouns from the same table, so this is an easy one to hit. The message names the group the family is in, which is probably what you wanted, and points at `every.families.`, which is where a single family's cadence lives. **`sinks: enable at least one of ...`.** A run that collects and discards is almost never what anyone meant. `-card-only` is the exception and needs no sink at all. **`prometheus exporter: listen tcp :9605: bind: address already in use`.** Reported at start-up rather than swallowed in a goroutine, so a port clash cannot leave you with a running collector and a silently missing exporter. **`influx write: 400`.** Almost always a column type collision. InfluxDB fixes a column as a tag or a field the first time it sees it and rejects later writes that disagree. If a collector changed which one a name is, the table has to be dropped: `DELETE /api/v3/configure/table`. **`family failed everywhere, not marking it as run`.** Every repository failed for one family, so it will be retried rather than treated as done. One repository failing is normal; all of them is the token, the network or an outage. **`collector failed` naming a `500`, a `502 Bad Gateway`, a `503` or a `504 Gateway Timeout`.** GitHub did not finish the answer to a request, at its gateway or in the application, or the storage a job log is read from did not. The client asks once more two seconds later and says nothing about it, so a line like this is a request that failed twice, and the next sweep asks again. A GraphQL query is asked again the same way, but for one answer: a `502` or a `504` that took GitHub's ten seconds to come back is a query too large, which the collectors that have a smaller page ask again with it. A line that says the query is too large for one request is one that timed out even at the smallest page, or in a walk with no smaller page to ask. What was read before it is written, a backfill does not record that repository as walked, and the next sweep reads it again. **`state not saved` or `cache file not saved`.** The user the collector runs as could not put the file in place. Each of the two is written under a temporary name in the directory the state file names and then renamed into place, so either that directory cannot be written, or it does not exist and cannot be created, or the file itself cannot be renamed over. Nothing a sweep learns then survives a restart: the next start collects every family and walks the stargazer list, the whole star history and the co-authored pull requests again. In a container this is a host directory mounted without handing it to uid 65532, a new volume mounted anywhere but `/var/lib/ghchronicle`, or any new volume under an image before 2.6.1, none of which that uid owns, or a state file mounted on its own, which cannot be renamed over; see [what has to be writable](https://docs.jmrp.io/ghchronicle/install/docker/#what-has-to-be-writable). **`the state file ... cannot be read` or `does not parse`, and the run stops.** The state file is there and this run could not read it, or it is not whole. A run that writes to the stores does not take it for a new one, which would forget the refills a migration still owes and put a new file in its place on its first save. A file root left behind, after `-migrate -yes` or a backfill run with `sudo`, is the usual cause: `chown` it back to the user the collector runs as. Moved aside instead, the next run starts from a new one, and what it recorded, a refill still owed among it, is gone; see [the state file](https://docs.jmrp.io/ghchronicle/configuration/#state_file). **`rate limit reserve reached, family skipped`.** Once is fine. Every sweep means the cadences are too fast for the number of repositories. If the bucket that runs short is `core`, lengthen `artifacts` and then `actions`; if it is `graphql`, `issues`, `issueevents` and `commits`. See [what scales with activity, not with size](https://docs.jmrp.io/ghchronicle/api/cost/#what-scales-with-activity-not-with-size). **`refill owed: a store a migration cleared does not hold that history yet`.** A migration cleared the measurement named in that store and reading its history back from GitHub did not finish: the run was stopped, a store refused a write, or GitHub did not answer. The store now looks like one that never held the old shape, so only the state file knows. Under `migrate: warn` a start says this and sweeps; `-migrate -yes`, named in the line, reads it back from where it stopped, and so does any start under `migrate: auto`. See [reading the history back](https://docs.jmrp.io/ghchronicle/install/upgrading/#reading-the-history-back). ## The data looks wrong **`reconciled` at `WARN`, with `not_served`.** After reading a cleared measurement back, ghchronicle compared the copy of the old rows with the table and found items GitHub no longer serves: a repository deleted or no longer covered, a comment deleted, an alert whose feature was switched off. `first` names a few, and `only_in` is the copy that still holds them, until it is purged: a day after the migration for PostgreSQL's and Elasticsearch's, and when the server's schedule says for InfluxDB 3's, 72 hours by default, or never on a server before 3.2. Carrying them over is by hand, from that copy, while it is there. **A number is a multiple of the sweep count.** Something that is a snapshot is being summed over time. Referrers, paths, labels and milestones are snapshots of a window with no date of their own; they are stamped at the start of the UTC day so a day's sweeps rewrite one row, and the dashboard takes the newest rather than the sum. **Median time to review by someone else reads No data.** The panel reads `seconds_to_first_human_review`, which leaves out review bots and the author's own replies; on an account where nobody else reviews, no pull request carries it and the tile is honestly empty. It does not mean nothing was reviewed. The bots' speed is in the Reviewers table, where each one is marked as a bot: measured, nine pull requests in ten had a bot review inside a minute. **Clones are enormous compared with views.** Continuous integration clones a repository thousands of times for every human visit. One repository measured here took more than a hundred clones for every view. `clones` does not count people. **`open_issues` disagrees with the issue count.** That field is GitHub's, and GitHub counts pull requests as issues in it. The `gh_issue` measurement is the one that counts issues. **Artifact storage looks too small.** Compare `walked` against `count` in `gh_artifact_total`. When they disagree the live size is a floor: the repository has more artifacts than the page cap walked, or a page of the listing failed on that sweep and the row was written from the pages before it. It can also be smaller than expected for a second reason: `count` is GitHub's total and includes the artifacts it has already expired, while the size is over the live ones alone, which `live_count` counts. **A panel answers with a schema error rather than No data.** InfluxDB 3 creates a column the first time a row carries it, so a query naming one that no row has written fails at planning. A release that adds a field does that to an existing database until the family that writes it has swept once: `gh_fork.seconds_to_push` and `gh_artifact_total.live_count` were two, which the Forks and "Artifact storage counted" panels name, and the `forks` family runs every twelve hours. A release that adds a measurement a panel joins does the same to the whole panel, in PostgreSQL as well, where the table rather than the column is missing: the "Work elsewhere" table joins `gh_upstream_repo` for each repository's stars, and until the first `outbound` pass of the new version writes it, which is within an hour, both stores refuse the table rather than leave its Stars column empty. Either wait for that pass or publish the dashboards after it. A `-once` does not hurry it: it runs only the families that are due, and `outbound` is not until an hour after its last pass. The other columns a database can lack are listed under [Columns that exist only once written](https://docs.jmrp.io/ghchronicle/collectors/measurements/#columns-that-exist-only-once-written). **The traffic chart only goes back fourteen days.** That is a first sweep. The window is rewritten day by day on every sweep, so the series extends as the collector keeps running. It cannot be backfilled: GitHub never stored anything older. **A panel says "Query would scan 10000 Parquet files".** InfluxDB 3 Core writes one file per partition per write request and never compacts them, so a store fed by a version of this tool older than the write ledger holds its rows in far more files than it needs. Widening the panel's interval does not help: the limit counts the files the planner opens, before any aggregation. What helps is the ledger, which is on by default and stops the growth, and then one of three things for what has already accumulated: `--query-file-limit` raised on the server, the affected tables rewritten, or InfluxDB 3 Enterprise, which compacts on its own and is free for home use. See [only what changed is written](https://docs.jmrp.io/ghchronicle/sinks/#only-what-changed-is-written). **A Prometheus panel shows one flat line.** That is the store, not the data. The exporter serves current values, so the fourteen-day traffic window collapses to its most recent day and the star history to the current total. See [dating a point](https://docs.jmrp.io/ghchronicle/how/dating/). ## Why is nothing being written? Run one sweep in the foreground and read what it says. Then check, in order: 1. that `-list` prints the repositories you expect, 2. that the sweep log says `written`, 3. that the sink is reachable. ```sh ghchronicle -config config.yaml -list # the repositories, and which are set aside ghchronicle -config config.yaml -once # one sweep in the foreground, then exit journalctl -u ghchronicle -f # under systemd ``` `debug` adds these to that and nothing else: the size of the written-points ledger at start-up, a first start with no cache file, a start listing jobs again because a store keeps no ledger, the account-wide families skipped for want of a `targets.user`, the entries a sink left out for being too old, a card-only sweep leaving the state file alone, each save of the cache file, the migrations a start finds not needed or applied before, the ones noted or frozen that the first start said at `INFO`, and each copy a migration set aside that the state file forgets because the store purges it or keeps it for good. There is no per-request log at any level. See [debugging](https://docs.jmrp.io/ghchronicle/configuration/logging/#debugging). ```yaml log: level: debug ``` A family that is not due yet simply does not appear. > **Deleting the state file costs quota, and two things more** > > It remembers nine things, and seven of them cost only quota when they go: what > is collected again is keyed by measurement, tags and timestamp and overwrites. > `last_head` is the commit each dependency diff started from, and without it > the next sweep has the photograph and no diff. `stores` is what the file > records of each store, a refill a migration still owes among it, and without > it nothing reads that history back until a `-backfill -families` does. See > [the state file](https://docs.jmrp.io/ghchronicle/configuration/#state_file). The cache file > beside it costs quota and nothing more: see [the cache beside > it](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it). ## Why does Loki drop entries? Look for the debug line counting them. Loki refuses an entry more than its out-of-order window behind the newest entry already in that stream, half of the ingester's `max_chunk_age` and so one hour by default, so the sink leaves the older ones out rather than losing the whole push. Raise `max_age` only alongside Loki's own `max_chunk_age`. See [Loki](https://docs.jmrp.io/ghchronicle/sinks/loki/). ## Publishing the dashboard **`Permissions needed: datasources:create`.** The token publishes dashboards and cannot make datasources. An Editor can do the first and not the second, so either give the service account that permission, make it an Admin, or let it adopt a datasource you made yourself by naming it in `grafana.datasource.uid`, which only ever reads. Correcting a datasource that is already there asks for `datasources:write` instead, with the same answers. That stops the run for the store's datasource, which every panel reads. For the Loki one it is only a warning, below. **`datasource ... does not answer: connection refused`.** Grafana reached the address and nothing was listening, which almost always means the address is right for the collector and wrong for Grafana. A collector on the host writes to a published port; Grafana in a container reaches the same store by its name on the container network, and `127.0.0.1` there is the container. Put the address Grafana would use in `grafana.datasource.url`. The run refuses rather than publishing, because a dashboard against a datasource that cannot be reached draws nothing and says nothing about why. **`asking datasource ... whether it answers: ... datasources:query`.** The token may not query the datasource, which is not the same thing as a datasource that cannot reach its store, and the address is not what to change. Measured on Grafana 13, a service account with no role gets this and an Editor does not, so give it the Editor role. **`Invalid API key`.** The token reached Grafana as the characters `${GRAFANA_TOKEN}` rather than as a token, which means the variable is not set where the collector runs. In a service unit that is an `EnvironmentFile` the unit does not read, or a variable named in the file and not exported. **The datasource is there and has no password.** Only a password the DSN itself writes is sent. One that pgx found in `PGPASSWORD` or a `.pgpass` on the machine that published is named in a line and left there, because a Grafana is shared and a credential the configuration never mentioned is not one to copy into it. **`the dsn asks for sslmode=prefer and Grafana's datasource has no such mode`.** libpq's default is to try TLS and carry on without it, and the datasource either insists or refuses. It was given `disable`; set `grafana.datasource.sslmode` to `require`, `verify-ca` or `verify-full` if that is wrong for your server. **Two dashboards, and the one you are looking at stops changing.** The generated dashboard has its own uid and every publish overwrites it, so this only happens when yours lives somewhere else: imported as new, or renamed by hand. Name the one you read in `grafana.dashboard_uid`. **`... is still in Grafana and no ... sink is configured here`.** Something this made is under a uid nothing writes to any more, which is what changing store leaves behind. It is a note and nothing was touched; delete it in Grafana when you are done with it. **`no sink this builds a dashboard for is configured`.** Dashboards exist for five stores, and the sink you have is not one of them. A forwarding sink has no dashboard of its own because the dashboard belongs to wherever it forwards to, and `file`, `stdout` and `loki` are not metric stores. **`warning: the Loki datasource ghchronicle-loki could not be set up`.** The token may not make the Loki datasource the failed job output panel reads, and the line names the permission it lacks: `datasources:create` for an Editor, since the datasource is not there yet. Nothing else waits on it: that one panel is left as the note the exported files carry, and the dashboards are published as long as the store's own datasource is. Name a Loki datasource Grafana already has in `grafana.datasource.loki_uid`, or grant the permission. The line lists the Loki datasources Grafana has, with their addresses, because the one to name is often there already under an address the collector does not push to, such as Loki's name on Grafana's container network. 2.4.0 and 2.5.0 stopped the whole publish here instead, with `the Loki datasource: writing datasource ghchronicle-loki`, and `loki_uid` is the fix for those too. **`warning: the Loki datasource ghchronicle-loki is used as Grafana has it`.** An earlier run with a token that could write datasources made it, and this token may not rewrite it: the line names `datasources:write`. A sink with a `tenant_id` asks for that rewrite on every start, because the tenant is a secret Grafana never hands back to compare, and so does a datasource whose address was changed by hand. The panel reads it as it is, so nothing is lost unless the sink has moved since. Grant the token `datasources:write` to keep its address and tenant in step with the sink, or set `grafana.datasource.loki_uid` to `ghchronicle-loki` to read it without trying. **`datasource ... (loki) adopted`.** Grafana already had a Loki datasource at the address the sink's push endpoint comes from, so the panel reads that one and no second one was made. Adopting only reads, so a token that may not create datasources gets this far. A sink with a `tenant_id` is never matched this way: the tenant travels in a secret Grafana never hands back, so nothing can tell whether the datasource at that address reads the same tenant, and one is made instead. **The failed job output panel is still a note.** That panel becomes the log lines when there is a Loki sink whose address ends in `/loki/api/v1/push`, which is what says where the query endpoint is. A sink writing anywhere else says so and leaves the panel alone; name a Loki datasource in `grafana.datasource.loki_uid`. A token that may not make the datasource leaves it too, with the warning above. **Nothing was published and nothing failed.** `publish_on_start` is off unless you turn it on, and it is skipped for `-once`, `-backfill` and `-card`, because turning it on in a service unit did not mean rewriting the dashboard on every run of a scheduled card render. **The collector started and the dashboard did not appear.** A publish that fails at start-up warns and the sweep goes on. The metrics of an hour spent not running cannot be recovered and a dashboard published on the next restart can, so the log line is the place to look. ## Taking things away **`-uninstall` printed a list and removed nothing.** That is the default. Add `-yes`. **The datasource survived the uninstall.** One named in `grafana.datasource.uid` is adopted, not made, so it was somebody else's before this ran and stays theirs. **`unknown -uninstall target`.** The whole list is refused rather than the part that parsed, so a typo removes nothing. The targets are `dashboard`, `data`, `state` and `all`. **The rows are still in the database after `-uninstall data`.** Three stores cannot be emptied from here and say so: Graphite has no delete, the Prometheus sink was scraped rather than written to, and the SQL sink's file goes but rows already loaded from it into a real database were loaded by you. **Tables or indices named `gh_...-20261001T091004`.** A migration set the measurement's old rows aside under that name, the measurement, a dash and the instant in UTC: PostgreSQL's renamed table, Elasticsearch's clone, lower case there, and the name InfluxDB 3 gives a table it deleted. None of the shipped panels reads one. ghchronicle purges PostgreSQL's and Elasticsearch's once they have been kept 24 hours, and InfluxDB 3 purges its own on its own schedule, 72 hours after the delete by default, except one a release before 3.2 deleted, which is never purged, not even after an upgrade (see [before 3.2 the copy stays](https://docs.jmrp.io/ghchronicle/sinks/influxdb/#before-32-the-copy-stays)); `-migrate` lists them under their store as `kept aside`. `-uninstall data` removes PostgreSQL's and Elasticsearch's with everything else, and leaves out InfluxDB 3's with a note saying which of them stay, since the server refuses a delete of a table it has already deleted, or, before 3.2, renames it once more. ## After an upgrade **`migration pending`, at every start.** A store holds rows of a measurement in a shape this release no longer writes, and this start did not bring it along. `not_applied` says why: `migrate: warn`, or a reason the change needs somebody's word, such as an InfluxDB 2, whose only way is a final delete, a SQL file, a Graphite or a Telegraf, where ghchronicle keeps nothing aside, rows of accounts this configuration does not collect, rows reading it again would not bring back, or rows whose accounts or repositories could not be read, which an InfluxDB 3 Core past its query file limit refuses to say; or a one-shot run on a new state file, every run of the Action without one restored, which applies nothing on its own. `plan` and `apply` are the two commands, and `first`, on the service, says to stop it before the second. It is said at every start until the store is brought along: see [Migrations](https://docs.jmrp.io/ghchronicle/install/upgrading/#migrations). **`applying a migration before the first sweep`.** Not an error. Under `migrate: auto` the start found a change it can apply without losing anything, and is applying it: `found` is what the store holds, `action` where the old rows go, and `refill` what is read back from GitHub. It comes once per store and change, followed by `refill starting`, `refill complete` and `reconciled`, and then the first sweep. **`migration noted: nothing is changed` or `migration frozen`.** Not errors either. A note is a change nothing can put right, such as the `gh_actions_cache_entry` rows dated before 2.6.0, and a frozen measurement is one nothing this configuration runs writes any more. Each is said once at `INFO` and then at `DEBUG`. **`migration check failed: the store did not answer`.** The store was asked at start-up whether it holds an old shape, and refused or did not answer within 30 seconds. Nothing was recorded and nothing changed; the sweep goes on, and the next start asks again. **`migration failed`.** The store refused what applying asked of it, and the change stays pending; the other stores went ahead. `err` says why, and the usual reason is a credential that may write and may not delete: InfluxDB 3's token has to be allowed to delete a table, and the error carries the same delete as a `curl` to run by hand; PostgreSQL's user has to own the table, as it does when its own sink made it; an Elasticsearch key needs `manage` and `delete_index` on the indices of its prefix. PostgreSQL also gives up after three waits of 5 seconds each behind a query that holds the table, which a long Grafana query can be. Run the same command again once the cause is fixed: what was applied is recorded and is not done twice. **`refill did not finish, and is still owed`.** Reading back what was cleared stopped: the run was stopped, a store refused a write or GitHub did not answer. `resume` says how to go on. The walk keeps its checkpoint beside the state file, `-refill.json`, and `refill owed`, under [things that are errors](https://docs.jmrp.io/ghchronicle/reference/troubleshooting/#things-that-are-errors), is what a later start says of it. **`the store may have been cleared, so the refill stays owed`.** Clearing the store failed in a way that does not say whether it was carried out: a proxy's 502, a timeout, a connection that broke after the store did what it was asked. The refill was recorded as owed before the store was touched and stays owed, so the history is read back whether or not the clear happened; the change is asked about again at the next start. At worst that reads the history once for nothing. **`the state file records a refill owed to ... where it no longer points`.** A migration cleared a store, the refill did not finish, and the sink now points at another store. The record is kept, and the refill is read the next time the sink points there; until then nothing reads it. `-migrate` shows it under the store as `owed there`. **`not reconciled: the copy and the table could not be compared`.** The history was read back, and the copy of the old rows could not be read, or was already due to be purged. What came back is in the table; only the list of the items GitHub no longer serves is missing, and while the copy is there it can be read by hand. **`set-aside copy not purged` or `set-aside copies not listed`.** A day after a migration set it aside, ghchronicle could not drop PostgreSQL's copy or delete Elasticsearch's clone, or could not ask the store which copies it holds. It tries again after every sweep and at every start; `aside` names the copy, which can also be removed by hand. InfluxDB 3 purges its own on its own schedule, or, before 3.2, never. **`the cache file still claims what the cleared store held`.** A migration could not rewrite the cache file to forget the refusals, and the workflow runs already written, of the families it reads back, so those families may go on skipping what the cleared store no longer holds. With the service stopped, deleting `-cache.bin` costs one sweep at full price: [the cache beside it](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it). **`-migrate -yes` answers `process N (ghchronicle ..., service, since ...) holds ...-lock`.** The service, or another `-migrate -yes`, holds the state file, and nothing was changed. Stop it, or let it finish, and run the command again. In Docker, N is the process as the service's own container numbers it. **`waiting for the process holding the state file to finish`.** The service was started while `-migrate -yes` ran on the same state file. It waits for it to end and then starts on what it left. **`another ghchronicle service keeps the same state file`.** A second service was started on a state file a running one holds, and refused, since two services saving one state file undo each other's marks. Stop one, or give each a `state_file` of its own. **`the state file cannot be locked, so -migrate -yes cannot tell this service is running`.** The service could not create or open `-lock` beside the state file: the directory is not writable, or a run as another user, root most often, made the file. The service runs without the lock, and `-migrate -yes` then refuses to run at all, since nothing would keep the service from starting under it half way. Run it as the service's user: [after an upgrade](https://docs.jmrp.io/ghchronicle/install/systemd/#after-an-upgrade). ## The PostgreSQL that connects **`could not reach its database`.** The sink connects on the first write rather than at start-up, so this is the first sweep and not the configuration being read. The DSN takes either shape libpq does, and the message is the driver's. **A column is missing rather than the row.** A field with no value does not get a cell: an empty string is left out, and a float that is not a number is stored as NULL. The column appears the first time a point carries something for it. **A field changed type and the insert fails.** A column is created with the type of the first value seen and is never altered afterwards, so a field that was a number and is now a string has nowhere to go. Rename the field, or drop the table, and the collector's next write declares it again. No restart is needed: the sink declares a table once per process, and the first insert after the drop is refused because the table is not there, so the sink forgets the tables of that batch, declares them again and sends the batch once more. **The SQL file's `gh_discussion_comment` statements are refused.** With `there is no unique or exclusion constraint matching the ON CONFLICT specification`, replayed into a table a release before 2.6.1 made. That table keeps `is_answer` in its key, which 2.6.1 no longer writes, and the file cannot read a table's key the way the connecting sink does, so its upserts name a key the table does not have. The rest of the file loads. `-migrate -yes` writes the drop of that table into the file, ahead of the rows it reads again, so the file replayed in order loads whole: see [what applying does in each store](https://docs.jmrp.io/ghchronicle/install/upgrading/#what-applying-does-in-each-store). By hand, drop the table, as [the measurements page](https://docs.jmrp.io/ghchronicle/collectors/measurements/#how-to-read-the-tables) describes, and fill it again with a backfill. **The file sink and this one disagree.** Apart from the key above, they cannot: both render the same schema from the same code, and `cmd/check_postgres` plans every panel query against it. If the tables differ, one of them was loaded from an older version. --- # Release history The current ghchronicle release and its date, read at build from VERSION and CHANGELOG.md, what that file records, and where every release is published. Source: https://docs.jmrp.io/ghchronicle/reference/changelog/ The current release of ghchronicle is 2.6.5, tagged on 2026-09-30. Neither is typed on this page: the number is read from the repository's `VERSION` file when the site is built, the date from the heading that number carries in `CHANGELOG.md`, and the site does not build when that heading is missing. Every install command that names a version takes it from the same file. ## What the changelog records [`CHANGELOG.md`](https://github.com/jmrplens/ghchronicle/blob/main/CHANGELOG.md) says what changed in each release, why, the measurement behind it, and the part nobody verified. Where a claim in it was measured, it says on what. It has one section per version, newest first, each headed by the version and the day its tag was pushed, and from 2.5.1 on a release's section is its notes on GitHub word for word; the release page adds the pull and verification commands and a link to the commits. One section has no tag behind it: 2.5.2 was never released on its own, and its changes shipped in 2.6.0. It is in English, like the whole repository outside this site. What the first hours after moving from one release to the next look like is on [upgrading](https://docs.jmrp.io/ghchronicle/install/upgrading/). The site links the file rather than copying it, so there is one history to keep. Versions follow [semantic versioning](https://semver.org/), and a binary names its own with `ghchronicle -version`, one of [the five flags that print and exit](https://docs.jmrp.io/ghchronicle/reference/cli/#the-five-that-print-and-exit). ## Where each release is - [GitHub Releases](https://github.com/jmrplens/ghchronicle/releases) has every numbered tag with its notes and its files: an archive for Linux, macOS and Windows on amd64 and arm64, an SPDX SBOM for each archive, and `checksums.txt` over all of them. The checksum file and every SBOM carry a keyless cosign signature, and [checking what you downloaded](https://docs.jmrp.io/ghchronicle/install/linux/#check-what-you-downloaded) has the commands. A release named after a major tag, such as `v1`, is there for the Action's Marketplace listing and carries no files. - The container image, `ghcr.io/jmrplens/ghchronicle`, is tagged with each release's version behind a `v`, and with `latest`, for `linux/amd64` and `linux/arm64`; `latest` moves onto a release's image only once the release workflow has verified and run it. From 2.5.1 on it carries a keyless cosign signature by the same workflow identity as the checksum file, and [verify the image](https://docs.jmrp.io/ghchronicle/install/docker/#verify-the-image) has the command. - [The releases feed](https://github.com/jmrplens/ghchronicle/releases.atom) is the same list as Atom, for a feed reader. - The Action follows the major version: `jmrplens/ghchronicle@v2` resolves through a `v2` tag that the release workflow moves onto each 2.x release once its files are published. See [GitHub Actions](https://docs.jmrp.io/ghchronicle/install/actions/). --- # Calling it from a program There is no Go library. What there is instead is one sweep, NDJSON on standard output, and good reasons not to call it in a loop. Source: https://docs.jmrp.io/ghchronicle/reference/subprocess/ > **There is no Go library** > > Every package this tool has lives under `internal/`, and Go refuses that > import from outside the module. The subprocess route on this page is not a > workaround for something you have missed; it is the interface. ```text main.go:6:2: use of internal package github.com/jmrplens/ghchronicle/v2/internal/collect not allowed ``` What exists instead is a binary that runs one sweep, writes one JSON object per line to standard output and exits. Anything that can spawn a process and read a pipe can use it, in any language, and it gets the same points a database would rather than a second pass over the API. ## A configuration for one question ```yaml github: token: ${GITHUB_TOKEN} reserve_rate: 500 targets: repos: [acme/telemetry] sinks: stdout: true stdout_format: json state_file: /tmp/ghchronicle-adhoc.json log: level: warn ``` Two things about `targets` are worth getting right the first time. **Name repositories, not the account.** `repos` is an inclusion, not a filter: adding `user: acme` beside it does not narrow anything, it discovers the whole account and collects that as well. `-list` is the cheap way to check before spending quota on it. ```sh ghchronicle -config adhoc.yaml -list ``` ```text acme/telemetry ``` **Leaving `user` out switches off eleven families for free.** The account-wide collectors (`account`, `totals`, `ratelimit`, `events`, `notifs`, `billing`, `profile`, `outbound`, `history`, `achievements` and `keys`) have no login to ask about and are skipped, which is most of what an application asking about one repository does not want to pay for. The log goes to standard error, always and whatever the level, so standard output carries data and nothing else. That is what makes the pipe safe to parse without filtering it first. ## The command ```sh ghchronicle -config adhoc.yaml -once > points.ndjson 2> sweep.log ``` `-once` runs a single sweep and exits. It starts no exporter and holds no port, so it does not collide with a long-running instance on the same host. ## What comes out Three lines of it, from the demonstration account every example here uses: ```json {"time":"2026-09-08T16:08:17.651862791Z","measurement":"gh_repo","tags":{"archived":"false","default_branch":"main","fork":"false","full_name":"acme/telemetry","language":"Go","license":"apache-2.0","owner":"acme","repo":"telemetry","visibility":"public"},"fields":{"age_days":1290,"days_since_push":0,"forks":21,"network":21,"open_issues":9,"repo_id":1043778215,"size_kb":18422,"stars":148,"url":"https://github.com/acme/telemetry","watchers":11}} {"time":"2026-09-07T00:00:00Z","measurement":"gh_traffic","tags":{"full_name":"acme/telemetry","kind":"clones","owner":"acme","repo":"telemetry"},"fields":{"count":94,"uniques":71,"url":"https://github.com/acme/telemetry/graphs/traffic"}} {"time":"2026-09-08T16:08:17.651862791Z","measurement":"gh_release","tags":{"draft":"false","full_name":"acme/telemetry","owner":"acme","prerelease":"false","repo":"telemetry","tag":"v2.4.0"},"fields":{"age_days":22,"assets":4,"downloads":1840,"url":"https://github.com/acme/telemetry/releases/tag/v2.4.0"}} ``` Four keys, and they are the same four for every point: | Key | What it holds | | ------------- | ---------------------------------------------------------------------- | | `time` | The date the thing happened, RFC 3339. Not the time of the sweep | | `measurement` | What kind of thing this is, always prefixed `gh_` | | `tags` | Strings, and only strings. Together they identify the series | | `fields` | The values: numbers, booleans, and the occasional string such as `url` | The `time` distinction is the whole design, and it shows in those three lines. The traffic point is dated `2026-09-07T00:00:00Z` because that is the day those 94 clones happened, and every sweep for the next fortnight will offer it again with the same date. The repository point is stamped at the moment of the sweep, because "148 stars" is true now and has no other date to carry. The release carries its age in a field for the same reason. Every measurement and every field is listed in [Measurements](https://docs.jmrp.io/ghchronicle/collectors/measurements/). `stdout_format: influx` prints InfluxDB line protocol instead, which is the better choice when the receiving end already speaks it. ## Narrowing it to what you want Even against one repository a sweep runs twenty-one families, the twenty-three per-repository ones less `deps` and `joblogs`, which ship switched off. `every` switches them off, and `default` is the layer that does it in one line: put everything to `0`, then name back what you want. ```yaml every: default: 0 families: traffic: 6h ``` That run costs five API calls, one to resolve the repository named in `targets` and four for the traffic family, and prints 48 objects: 28 `gh_traffic`, 10 `gh_traffic_path` and 10 `gh_traffic_referrer`. > **A default of 0 never switches the off ones on** > > `deps`, `history` and `joblogs` ship with a built-in cadence of `0`, and > neither `default` nor `every.groups` can reach a family that ships switched > off. Naming it under `families:` is the only way to turn one on, so the > configuration above collects traffic and nothing else, the dependency graph > included. A family is not a measurement, which is the other half of this: `repo` alone emits `gh_repo`, `gh_repo_language`, `gh_repo_topic`, `gh_release` and several more. Switching the family off is what saves the API calls; filtering the stream is what saves you reading them. ```sh ghchronicle -config adhoc.yaml -once | grep '"gh_traffic"' ghchronicle -config adhoc.yaml -once | jq -c 'select(.measurement == "gh_repo") | {repo: .tags.full_name, stars: .fields.stars}' ``` A family name spelled wrong is a start-up error, and the message lists every name there is, so there is no need to keep a copy of the list anywhere. ## Reading it from another program Any language, since the contract is a pipe and a line of JSON. Python here, because a Go program cannot import this and would be running the same subprocess. ```python import json import subprocess proc = subprocess.Popen( ["ghchronicle", "-config", "adhoc.yaml", "-once"], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True, ) stars = {} for line in proc.stdout: point = json.loads(line) if point["measurement"] == "gh_repo": stars[point["tags"]["full_name"]] = point["fields"]["stars"] if proc.wait() != 0: raise SystemExit("the sweep failed; run it again without stderr suppressed") print(stars) ``` ```text {'acme/telemetry': 148} ``` Read the pipe as it fills rather than after the process exits. The sink flushes once per family, so a consumer sees the traffic points while the commit collector is still working, and a sweep over a whole account takes minutes. Waiting for the end also means holding all of it in memory: over a handful of busy repositories the `actions` family alone can produce more than ten thousand objects in a single sweep. Sending the log to `DEVNULL` is right for a program that only needs to know whether the sweep succeeded. It is the wrong default while you are still writing that program: keep it, and read it. ## Two smaller questions - **-list** Prints the repositories in scope, one full name per line, then the archived ones the filter sets aside, each followed by `(archived: its archive date and lifetime totals; a backfill collects the rest)`, and costs the discovery calls and nothing else. ```sh ghchronicle -config adhoc.yaml -list ``` - **-card** Still runs a full sweep, but writes a self-contained SVG with no database configured at all. The card is about an account rather than a repository, so it needs a config with `targets.user` in it: the one above has none, and gets `render: card has no login` and a 1. See [the card](https://docs.jmrp.io/ghchronicle/card/). ```sh ghchronicle -config account.yaml -card summary.svg -card-only ``` ## Exit codes | Code | Means | | ---- | -------------------------------------------------------------------------------------------------------------- | | 0 | The sweep ran | | 1 | It could not start, or could not list the repositories: no config file, an invalid one, a token GitHub refused | | 2 | An unknown flag. This is Go's flag package, and it has already printed the usage | Zero answers "did the sweep run", not "did everything work". A collector that fails is logged and the sweep continues, because a repository with a feature switched off must not stop the sweep for the other forty. ```text level=ERROR msg="collector failed" family=artifacts repo=acme/parser err="/repos/acme/parser/actions/artifacts?per_page=100&page=2: 502 Bad Gateway: {\n \"message\": \"Server Error\"\n}" ``` A 500, 502, 503 or 504 in such a line has already failed twice: the client asks once more, two seconds later, before it gives up on a REST request or on a job log's download from storage, and logs neither attempt. A caller that needs to know about that has to read standard error. There is no exit code for it, deliberately: on a large account some family fails somewhere most sweeps, and a status that reported it would be permanently red. Killing the process usually gets a 1 as well. `SIGTERM` and `SIGINT` cancel the sweep, what is already on the pipe stays there, and the run ends with `ghchronicle: context canceled`. Usually, because the code depends on where the signal lands: during discovery it is a 1 carrying the cancelled request's own error instead, and in the last repository of a family it is logged as a collector failure and the sweep still finishes with a 0. A caller with its own deadline should treat a kill it ordered as "incomplete" rather than read the exit code for it. ## Calling it in a loop is the wrong shape A sweep costs API calls: four per repository for traffic, three per repository plus two GraphQL points per ten repositories for the repository family, one GraphQL point per repository with a new commit for commits and one to two per repository with an issue or a pull request that moved for issues, after one point per twenty-five repositories that asks which. [Cost of a sweep](https://docs.jmrp.io/ghchronicle/api/cost/) has the measured table. The budget is 5000 REST calls an hour for the whole token, shared with everything else that uses it, and the collector brakes before spending the last `reserve_rate` of them: it stops a family rather than crossing that line, and says so in the log. So a program that calls this on every request, or on a tight timer, is answered with an empty sweep and a warning rather than with fresher numbers. Two things follow, and the second one surprises people. **Cache the answer.** These numbers move on the order of hours. GitHub's own traffic window updates once a day. **Expect the second run to print nothing.** A family is collected only when its cadence says it is due, and `-once` marks it as run in the state file. Two sweeps a minute apart therefore give a full stream and then an empty one, exit code 0 both times, which reads like a bug and is the brake working. At `info` the log says so by omission: ```text level=INFO msg="cache file read" file=/tmp/ghchronicle-adhoc-cache.bin ... level=INFO msg="repositories discovered" count=1 archived_aside=0 level=INFO msg="rate budget" bucket=core remaining=3949 limit=5000 level=INFO msg="sweep finished" ``` No `written` line, because nothing was due. Deleting the state file collects everything again, including the one-off walk of every star, so point `state_file` somewhere the application controls and leave it there. The cache file beside it, `/tmp/ghchronicle-adhoc-cache.bin` here, keeps the ETags, so what has not changed since is answered a free 304; deleting that as well is what costs the full price. > **When it is not ad hoc** > > If the program wants a continuous feed rather than an answer now, stop > spawning it: run it as a service and give it a [sink](https://docs.jmrp.io/ghchronicle/sinks/). > The file sink writes this same JSON to a rotating file for anything that tails > it, and the push sinks reach a store the program can query without touching > GitHub at all. --- # The test layers Three layers, only one of them free: the contract test of the bytes, the containerised stores that accept them, and the dashboards' own queries. Source: https://docs.jmrp.io/ghchronicle/reference/testing/ The tests come in three layers, and they are worth telling apart because only the first one is free. Two of them start nine containers, and anyone deciding whether to wait for that deserves to know what it buys. | Layer | Command | Docker | Time | What it proves | | ------------------ | ---------------------- | ------ | -------------- | -------------------------------------------------------------------- | | L1, the contract | `make test` | no | about 5 s | the exact bytes each sink puts on the wire | | L2, the stores | `make test-e2e-docker` | yes | 53 s warm | a real store accepts those bytes, and keeps the date of the event | | L3, the dashboards | the same target | yes | included above | the five dashboards' own queries answer against what the sinks wrote | L1 runs on every push. L2 and L3 are one suite behind the `dockere2e` build tag, so `go test ./...` never starts a container; in CI they run weekly, on demand, and as a release gate. ## L1: the bytes on the wire `test/e2e` builds the real binary, runs it against a fake GitHub whose fixtures live in `test/e2e/testdata` and whose route table is `test/e2e/fakegh`, and points every sink at an `httptest` capture server. Then it asserts the bytes: the line protocol, the `_bulk` envelope, the SQL statements, the Graphite path, the OTLP payload. The fake also prices its answers the way api.github.com does: one ETag per REST fixture and none on a GraphQL answer, a 304 charged nothing for a request that presents it, one for every other answer on the API, nothing for the object storage a job log redirects to, and the budget block on every GraphQL answer. That is what lets `TestTheSecondSweepIsPricedByTheCache` run two sweeps in a single process and hold that every URL answered 200 the first time came back 304 the second, that the second sweep charged fewer than half the core requests of the first, and that `own_cost` on the `gh_rate_limit` row is the number of queries the process made rather than zero. It searches the way GitHub does, too. Each of the five outbound searches asks for one kind in one state, `is:pr is:merged` or `is:issue is:open`, and the fake serves it only the items of its fixture those qualifiers select, counting the others out of `issueCount`. A kind or a state it does not know, an `is:`, `type:` or `state:` it has no rule for, fails the test; the author, the owners left out and the order are the fixture's own already, and it reads past them. Until 2.6.2 it answered every search with every item, so a sweep wrote the one merged pull request five times, once as an open issue, and the dashboards drew those rows. A fixture never writes out a recent date. It spells one as an offset the fake resolves when it serves the file, `"@DAYS_AGO_12@T00:00:00Z"`, beside the `@NOW@`, `@TODAY@` and `@SOON@` it already resolved, and `@DAYS_AHEAD_n@` for whatever has to still be in the future. A dashboard asks for the last day, week or month, so a written-out date is correct only until it falls out of that window, and nothing announces the day it does: the release of 2.4.0 failed with four panels of every store answering nothing, twelve hours after the same suite had passed, because the day one pull request merged on had crossed `now-30d` in between. A test that has to name one of those days asks `fakegh.DaysAgo`, and a date written out by hand fails `TestNoFixtureWritesOutARecentDate` on the spot. That is the right test for a format, and it is fast enough to run while a sink is being changed. What it cannot catch is anything the receiver has an opinion about. A capture server answers 204 to everything. It has no column types, no mapping, no query planner and no schema. ## L2: the stores themselves `test/e2e/docker` starts the real stores in containers, runs one sweep from the same fake GitHub into all of them, then reads each store back and asserts the value, the tags and above all the timestamp. The dating rule is the product of this tool: a star is stamped when it was given, a workflow run when it finished, a traffic day at that day's own date. No capture test can prove a store kept the date of the event rather than the date of the sweep, because storing it is the store's job. These are the defects this layer exists for, each of them real: **InfluxDB fixes a column's type on first sight.** InfluxDB 3 decides that a column is a tag or a field the first time it sees it and refuses every later write that disagrees: `400 invalid column type for column 'owner', expected iox::column_type::tag`. A capture server answers 204 and notices nothing. This has already cost a database wipe, and reproducing it was the first thing the containerised stack was used for. **Elasticsearch's dynamic mapping decides whether the dashboards can aggregate.** One panel pulls a url through a `top_metrics`, which normally needs a keyword field rather than a text one. Two separate audits recorded that as unverifiable for want of a real Elasticsearch. This layer answers it, by indexing through `_bulk` and reading back the mapping the cluster built for itself. **PostgreSQL has to accept the DDL.** The SQL sink emits statements rather than speaking the wire protocol, so until now nothing ever had them parsed. The suite pipes them through `psql`, inserts, and plans the panels' queries against the schema the sink created rather than against one transcribed from InfluxDB. **Graphite paths have to have the depth the dashboards index.** The dashboards address path nodes by position, and the agreement between those positions and what the sink writes was kept by a hand-maintained table that nothing checked. Here the sink writes to carbon and the render API is asked for the path back. ## L3: the dashboards' own queries With the stores loaded, the five generated dashboards are run through Grafana's `/api/ds/query`, which is the path `cmd/check_dashboards` takes against a live Grafana. Every datasource is provisioned at boot with a fixed uid and the harness mints a service account token, so a panel query goes through Grafana exactly as it would for a person looking at the dashboard. That is what turns three manual checkers into something CI runs, and it is what settles the Elasticsearch question above: a panel that cannot aggregate returns no frame. Each panel is posted with its range resolved to the two instants a browser sends, and asked three questions, and every stat and gauge two more below: did the datasource refuse it, did a panel over something the sweep wrote inside the range answer with anything, and do the five stores draw the same thing. For the third, each answer is replayed through what Grafana does between the query and the screen, the Prometheus datasource's own reshaping of a table, the panel's transformations and its field overrides, and every pair of stores is compared on what a reader would see: a tile's number, its unit and the words it shows for nothing, a table's rows over the columns both stores draw, and a bar's name and length. A table is held as well to one order of the columns both stores draw, which rows matched over those columns cannot show: on the 2.6.2 branch 28 Prometheus tables and 4 Graphite ones headed them in another order than the SQL stores, "Every bucket" with Most used last. The review of 2.6.1 found eleven differences by putting the dashboards side by side, among them a table that drew seven rows for one repository and stat tiles that had lost their units. Nine of them lived in the dashboards, and run against that release's dashboards this fails on eight. The ninth was four columns Elasticsearch's "Open the longest" went without, which no comparison of values can see. What a store can hold at all is its own description's business, so another assertion holds every column the SQL stores draw in a table to being drawn by each other store that draws the table, or named in that store's own words about the panel: on the 2.6.2 branch 33 tables of the other three stores lacked a column their descriptions did not name, and each now draws it or says why not. Every stat and gauge is then asked about nothing, twice: about a repository no sweep wrote anything for, and over thirty days no point of any sweep falls in, four hundred days back or more. Over no rows a SQL count is 0 and anything else is null, which a tile draws as the words its panel gives a value that is not there, and every store is held to drawing the tiles the SQL stores draw, with the same words. The range is asked because the account's own snapshots do not follow the repository picker: over a range four hundred days back Graphite drew three stat groups as panels with nothing in them, not even the names. A tile a store cannot draw over nothing is listed in `tilesLeftOverNothing`, in `test/e2e/docker/tiles_over_nothing_test.go`, under words of that store's own description, and held to them as `dashboardsDiffer` is. InfluxDB and PostgreSQL run one statement over the same rows, so they are also held to drawing the rows of a table or a bar chart in one order, which a comparison of the rows as a set cannot see: before the statements named their tie-breakers, twelve of the 49 panels both draw with more than one row put two of them the other way round in one run, and eleven in the next. `TestEverySQLListOrdersItsRowsCompletely`, in `internal/dashboards`, holds every list's `ORDER BY` to naming what tells two of its rows apart, so that the next tie does not wait for a run that happens to draw it. Two of the rules held of what is drawn are held of the specification as well, since a table the fixture leaves empty in both SQL stores is never drawn at all: every column the SQL stores draw that another store does not is named in that store's own description, and a chart the SQL stores fold into other folds in each other store or says it does not (`TestEveryColumnAStoreLacksIsNamedInItsDescription` and `TestEveryStoreFoldsTheRestIntoOtherOrSaysItDoesNot`). And a Prometheus query that is wrong only for series the fixture never makes, a label value nobody wrote or a series that stood still over the range, is put to promtool inside the stack's own Prometheus over series written for it (`TestPrometheusAnswersSeriesTheFixtureHasNoneOfAsTheSQLStoresDo`): until it was, the Signed commits of an account that never signs read "no commits" beside a count of 57. A store that draws a panel differently on purpose says why in its own description of the panel, and the difference is listed in `dashboardsDiffer`, in `test/e2e/docker/dashboards_agree_test.go`, under those words. The test fails when the words are no longer in the description, when the stores have come to draw the panel alike, and when a store the entry names draws the panel as the stores it does not name, so an entry can neither outlive its reason nor excuse a store that needs no excuse. What the harness itself causes is absorbed where it arises rather than listed: the exporter's half minute of history, values the collector computes from its own clock across three sweeps a minute apart, values a query computes from `now()` across stores asked seconds apart, and names Graphite holds as path nodes. A separate assertion fails on any panel, time series included, that draws a field under the name its datasource gave it, such as `p50.0 seconds_to_merge`. ## What none of them catch Every layer runs against the fake GitHub, so nothing here notices GitHub changing a payload, retiring an endpoint or throttling differently. That is what `ghchronicle -once` against a real token is for. They also prove nothing about a store the suite does not start. The answer covers InfluxDB 3 Core, PostgreSQL 18, Elasticsearch 9, Graphite 1.1, Prometheus 3, Loki 3, the OpenTelemetry collector and Telegraf, at the pinned versions. OpenSearch, TimescaleDB and anything behind the Telegraf or OTLP hop are still an inference from the format. There are five more things, and none of them is a layer. Each is switched on by an environment variable and skipped when it is absent, so an ordinary `go test ./...` stays offline. **The store you actually run.** `test/live` pushes a handful of points at a Loki or an OpenTelemetry collector named in `GHC_LIVE_LOKI` or `GHC_LIVE_OTLP`. It answers the one question containers cannot: whether your instance accepts them. ```sh GHC_LIVE_LOKI=http://localhost:3100 go test ./test/live/ ``` **The real API, end to end.** `GHC_E2E_LIVE=1` runs `TestLiveAPI` against GitHub itself rather than the fake, with a real `GITHUB_TOKEN`, sweeping the account named in `GHC_E2E_USER`. ```sh GHC_E2E_LIVE=1 GHC_E2E_USER=octocat GITHUB_TOKEN=ghp_... go test ./test/e2e/ -run TestLiveAPI ``` **What a sweep costs in cache.** `GHC_LIVE_CONFIG` points `TestLiveSweepCacheFootprint` at a configuration file and sweeps the account it names, reporting the entries and the bytes the conditional-request cache holds after each sweep. Those are the figures the 256 MB bound and the [cost of a sweep](https://docs.jmrp.io/ghchronicle/api/cost/) rest on, and this is how to reproduce them for your own account. `GHC_LIVE_DUMP=1` adds the per-URL list to standard output. ```sh GHC_LIVE_CONFIG=config.yaml go test ./internal/ghapi/ -run TestLiveSweepCacheFootprint -v ``` **One repository, one family.** `cmd/probe` runs the collectors against a single repository and prints the line they would write, writing nothing anywhere. `GHC_DUMP=` prints every point of that family in full, which is the fastest way to see what a collector actually produces. ```sh go run ./cmd/probe owner/name GHC_DUMP=actions go run ./cmd/probe owner/name ``` **The pictures of the card.** `GHC_CARD_GALLERY` names an existing directory and `TestCardGallery` renders one card per layout into it, from the fake GitHub rather than from anybody's account. That is where the pictures on [the layouts page](https://docs.jmrp.io/ghchronicle/card/layouts/) come from, and a layout that changes shape is one command away from a set that agrees with it. The account is the base fixtures with `test/e2e/testdata/gallery/` laid over them: a year of contributions, GitHub's whole fourteen days of traffic, five repositories to rank and one of them in six languages, which the smaller account every other suite asserts on cannot give a picture. A fixture named `~` there answers for that one repository, and any other repository borrows hello-world's. Each layout comes out of one sweep under `-card-theme both` as two files, `card-.svg` in the light palette and `card-_dark.svg` in the dark one, which is what the site's `ThemeImage` and the README's `` read. The two layouts that loop come out a second time under `-card-motion loop`, as `card--loop.svg` and its `_dark` twin. Only those two: on every other layout `loop` draws the same card as `once`, so a looping picture of one would be a second copy of the first under a name that promises something else. Which layouts they are is the registry's `Loops`, and the gallery reads it rather than keeping its own list. ```sh mkdir -p /tmp/cards GHC_CARD_GALLERY=/tmp/cards go test ./test/e2e/ -run TestCardGallery ``` `make check-gallery` renders the gallery into a scratch directory and fails, naming every difference, if the committed set no longer matches it byte for byte; `make gallery` regenerates it in place. CI's "Generated artifacts" job runs the check on every pull request. ## Running the stack Docker with the compose plugin, and room for the images. Then: ```sh make test-e2e-docker ``` Up, run, down on every path including a failing assertion, and then a check that `docker ps` shows nothing of the project left. A suite that leaves nine containers behind on a failure is a suite nobody runs twice. > **Nothing of this listens outside loopback** > > Every port is published on `127.0.0.1`, on a free port Docker picks out of > 49200 to 49299, never a store's default: 9200, 8086, 5432, 2003, 9090, 3100, > 3000 and 4318 belong to whatever else is on the machine. The harness reads the > chosen ports back with `docker compose port`, and the project is named > `ghchronicle-e2e` in the compose file as well as on every command, so nothing > here can reach a container it did not start. Boot, measured cold with the images already pulled: Elasticsearch 29 s, Loki 21 s, Grafana 13 s, the Graphite render API 10 s, InfluxDB 8 s, PostgreSQL 6 s, the rest 6 s. The stack is ready in 30 s; the target end to end, teardown included, is 53 s. ## Debugging with the stack up The reason to fail an assertion is to go and look at the store, and a suite that tore the store down first cannot be looked at. So the two halves are separate targets, and the harness reuses a stack it finds already running and leaves it running. 1. Start the stores and leave them up. The command prints the port each service ended up on. ```sh make e2e-docker-up ``` 2. Run the suite, or one test of it, as many times as it takes. ```sh go test -count=1 -tags dockere2e -timeout 30m -v ./test/e2e/docker/ ``` `GHCHRONICLE_E2E_KEEP=1` also stops the test binary tearing down a stack it started itself, which is what you want when a single `-run` is failing. 3. Ask the store what it thinks, then tear it down. ```sh make e2e-docker-logs SERVICE=influxdb make e2e-docker-down ``` With the ports from step 1: ```sh # What InfluxDB thinks each column is. This is the answer to a 400 on write. curl -s "http://127.0.0.1:/api/v3/query_sql?db=ghchronicle" \ --data-urlencode "q=SELECT * FROM information_schema.columns WHERE table_name = 'gh_repo'" # The mapping Elasticsearch built for itself. curl -s "http://127.0.0.1:/ghchronicle-*/_mapping?pretty" # What the SQL sink actually created. psql "postgres://ghchronicle:ghchronicle@127.0.0.1:/ghchronicle" -c '\d+ gh_repo' # The Graphite path, node by node. curl -s "http://127.0.0.1:/metrics/find?query=github.repo.*" ``` Grafana is at the port it published, with `admin` and `admin`, and every datasource is already provisioned, so a panel query can be pasted into Explore and run by hand. ## Three things the stack had to be told Each of these silently produced a wrong answer before it was found, and each is in the compose file or its configuration with the measurement beside it: - **Carbon drops a point older than its longest archive without saying so.** A star dated 2020 vanished under a six year retention and the write was reported as accepted. The retention is `1d:12y` for that reason. - **Carbon's default `MAX_CREATES_PER_MINUTE` is 50**, fewer paths than one sweep creates, so most of a first sweep would be dropped. - **Loki answers a push with 204 and will not serve it until the chunk is flushed.** The test polls rather than asking once, and `chunk_idle_period` is 5 s. > **A firewalled machine needs one rule to scrape the exporter** > > The Prometheus exporter is the one sink that is scraped rather than pushed to, > so the scrape has to cross from the container back out to the host. Where the > default INPUT policy is deny, no container on any bridge reaches any host > port. One narrow rule is enough: allow TCP 49300-49399, the range > `scrapePortLow` and `scrapePortHigh` bound in `sweep_push_stores_test.go`, > from the container address space and from nothing else. Under ufw that is > > ```sh > ufw allow proto tcp from 172.16.0.0/12 to any port 49300:49399 > ``` > > Without it the harness reports `ErrExporterUnreachable` and the Prometheus > assertions skip with a reason instead of failing, which means they have never > run on that machine. A CI runner needs no rule and neither does an ordinary > workstation. A machine that adds the rule runs those assertions for the first time, which is when `promNeedsHistory` in `dashboards_test.go` starts to matter: the exporter is alive only for the length of the test, so every timeseries panel and every panel built on `increase()` is held to nothing and only the instant panels are asserted. ## In CI `.github/workflows/e2e.yml` runs `make test-e2e-docker` and has three ways in: manual dispatch with an optional ref, a weekly schedule on main, and `workflow_call`, so a release pipeline gates a tag on it with one line rather than a copy of the job that drifts from the original. It is not a required check on a pull request: nine containers and around 10 GB of image is too much for every push, and L1 is what covers every push. The weekly run is the point of the schedule. Nothing else in the repository ever starts a container, so without it the suite would run only when somebody remembered it, which is how a suite ends up broken for weeks with nobody knowing. The same suite also runs under the race detector, in `.github/workflows/race.yml`: weekly, at every release beside the E2E gate, and by hand. The harness builds the collector with `-race` and starts it with `GORACE=halt_on_error=1`, so a race inside the collector fails the test that started it, with the report. Locally it is `make test-e2e-docker-race`.