# The sweep

What one ghchronicle sweep over the GitHub API does, in what order, and why each family of metrics has its own cadence.

Source: https://docs.jmrp.io/ghchronicle/how/

A sweep is one pass over every family whose interval has elapsed, except that
the running service starts one family of six hours or more in each sweep, more
only where one could not keep every cadence, and leaves the others that are due
for the next ticks: the [slow families
take turns](https://docs.jmrp.io/ghchronicle/configuration/cadences/#the-slow-families-take-turns).
It is an increment, not a rebuild: it asks for the little that can have changed
since last time, writes what it got to every configured store, and records when
each family ran.

## The shape of one pass

```mermaid
%% Generated by site/scripts/gen-figures.mjs from internal/config/config.go and internal/run/runner.go
flowchart TD
    T["Timer"] --> D["Discover repositories<br/>(rebuilt hourly)"]
    D --> A["Account-wide families<br/>account, totals, ratelimit, events, notifs,<br/>billing, profile, outbound, history,<br/>achievements, keys"]
    D --> R["Per-repository families<br/>traffic, repo, branches, stars, issues,<br/>issueevents, actions, artifacts, security,<br/>stats, discussions, commits, activity,<br/>analyses, forks, planning, joblogs, settings,<br/>rulesets, inventory, deployments, policyfiles,<br/>deps"]
    A --> B{"Budget above<br/>the reserve?"}
    R --> B
    B -- "no" --> S["Skip the family<br/>and warn"]
    B -- "yes" --> C["Collect"]
    C --> P["Dated points"]
    P --> H["Destinations that keep the date<br/>InfluxDB, file, stdout, Telegraf, Graphite,<br/>PostgreSQL to a file, PostgreSQL,<br/>Elasticsearch"]
    P --> L["Loki<br/>the twenty-two event renderings"]
    P --> RD["Reducer<br/>current values"]
    RD --> G["Prometheus<br/>OTLP, when raw: false"]
    C --> M["Mark the family as run<br/>in the state file"]
```

Two things in that diagram are the whole design. Every collector produces
_dated_ points, and the stores that cannot hold a date get them reduced to
current values first, by the same reducer, before they ever see the data. That
is the subject of [dating a point](https://docs.jmrp.io/ghchronicle/how/dating/).

## Why families and not one interval

The surfaces move at wildly different speeds. Workflow runs finish every few
minutes on a busy account. Labels and milestones are edited a few times a week.
The list of forks changes a few times a year. One interval for all of them would
either waste the rate limit on the slow ones or lose the fast ones, so each
family carries its own.

| Family         | Default | Collects                                                                                 |
| -------------- | ------- | ---------------------------------------------------------------------------------------- |
| `actions`      | 15m     | Workflow runs, jobs, steps, the Actions cache                                            |
| `activity`     | 15m     | The repository activity log, where a force push is recorded                              |
| `events`       | 15m     | The account event feed, which keeps only the last 300                                    |
| `notifs`       | 15m     | The notification inbox                                                                   |
| `ratelimit`    | 15m     | What the collector has left to spend, in each budget                                     |
| `deployments`  | 30m     | Deployments and their environments, batched over every repository                        |
| `account`      | 1h      | Profile, contribution calendar, contribution totals                                      |
| `achievements` | 1h      | The profile badges, and the distance to each next tier                                   |
| `analyses`     | 1h      | Code scanning analyses, which GitHub prunes                                              |
| `artifacts`    | 1h      | Artifacts and their expiry                                                               |
| `billing`      | 1h      | Usage per day, product, SKU and repository                                               |
| `commits`      | 1h      | Lines changed and signature state, per commit                                            |
| `discussions`  | 1h      | The forum half of a repository                                                           |
| `issueevents`  | 1h      | The timeline of what moved: labels, assignments, transitions                             |
| `issues`       | 1h      | Pull requests, issues and reviews, per item                                              |
| `outbound`     | 1h      | Stars given, and work in other people's repositories                                     |
| `repo`         | 1h      | Stars, forks, languages, topics, releases, rulesets                                      |
| `security`     | 1h      | Dependabot and code scanning alerts                                                      |
| `stars`        | 1h      | Stars per day for every repository, and the stargazer walk once, then the newest hundred |
| `totals`       | 1h      | The lifetime numbers, asked of GitHub rather than added up here                          |
| `planning`     | 6h      | Labels and milestones                                                                    |
| `settings`     | 6h      | Webhooks and their deliveries, environments, deploy keys                                 |
| `traffic`      | 6h      | The whole 14-day window, rewritten                                                       |
| `forks`        | 12h     | Who forked, and when                                                                     |
| `profile`      | 12h     | Packages, gists, social accounts                                                         |
| `stats`        | 12h     | Commits per week, the punch card, the workflow definitions                               |
| `branches`     | 24h     | Which branches are live and how stale each tip is                                        |
| `inventory`    | 24h     | What a workflow's own token may do, both secret stores, default code scanning            |
| `keys`         | 24h     | The account's SSH and GPG keys, and when each expires                                    |
| `policyfiles`  | 24h     | SECURITY.md, CODEOWNERS, dependabot.yml and FUNDING.yml                                  |
| `rulesets`     | 24h     | Every version of every ruleset's changelog                                               |
| `deps`         | off     | The dependency SBOM of each repository, and what changed                                 |
| `history`      | off     | Each year's contribution calendar, the current one included                              |
| `joblogs`      | off     | The tail of every failed job's log                                                       |

Setting any of them to `0` switches it off entirely. See
[cadences](https://docs.jmrp.io/ghchronicle/configuration/cadences/).

## Discovery

The repository list is rebuilt once an hour, by the first sweep that finds it
an hour old, give or take half a tick: a sweep reads its clock a few
milliseconds either side of the hour, and without that margin a repository
created in between waited a tick more. Repositories are created rarely and
listing them costs a page per hundred, so anything shorter spends quota to
learn nothing. Forks and archived repositories are excluded by
default, for the reason set out in [targets](https://docs.jmrp.io/ghchronicle/configuration/targets/).

## Failure is per repository, not per sweep

A family that fails on one repository is logged and skipped; the sweep
continues. This matters more than it sounds, because "failure" here is usually
a feature being switched off: of fifty repositories, most have Dependabot off,
and each of them answers 403. Treating that as an error would lose the other
forty-nine.

There is one exception, and it is deliberate. A family where _every_ repository
failed is not marked as run. Marking it would hide the outage until the next
cadence, which for the twelve-hour families is half a day.

```text
level=WARN msg="family failed everywhere, not marking it as run" family=security
```

## The state file

`state_file` holds [nine things](https://docs.jmrp.io/ghchronicle/configuration/#state_file), and
the two a sweep is judged by are when each family last ran and when each
repository was first seen.

The second is what makes the one-off full walk of the stargazer list happen
once instead of on every sweep, as `history_read` does for the daily star
history. It is written through a temporary file and renamed,
so a crash mid-write cannot leave a truncated state that would trigger a full
re-collection.

Beside it the collector keeps what makes a sweep cheap rather than what makes it
complete: the ETag cache, the workflow runs whose jobs were written, the
refusals and the pull request page sizes, in
[`<name>-cache.bin`](https://docs.jmrp.io/ghchronicle/configuration/#the-cache-beside-it). A restart
reads it back and starts where the last process stopped, not from nothing, and
deleting it costs one full-price pass of each family and loses nothing.

> **Three runs collect every family whatever the state says**
>
> An exporter holds its samples in memory, so a restart empties it and it stays
> empty until each family's cadence comes round, which for the twelve-hour ones
> is half a day of a dashboard reading zero. Paying for one full sweep is the
> cheaper mistake, so when the service feeds the [Prometheus
> exporter](https://docs.jmrp.io/ghchronicle/sinks/prometheus/#scraping-it) its first sweep after
> start-up runs every enabled family whatever the state file says, unless
> `no_prime` is set. It records as run only the families that were due, so the
> others keep the turn they had. `-once` starts no exporter and does not prime.
> A [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/) runs every family too, because
> reaching as far back as GitHub allows is the whole point of asking for one,
> and so does a run drawing [a card](https://docs.jmrp.io/ghchronicle/card/), because every number
> on the card comes from that one sweep and a family skipped as not due would
> be a zero on the picture.

## The brake

Before each family the collector checks the three rate buckets it actually
spends from. If any of them is at or below its reserve and the window has not
reset yet, the family is skipped and a warning is logged rather than the budget
being spent to the last call. See [rate limits](https://docs.jmrp.io/ghchronicle/api/).

A [backfill](https://docs.jmrp.io/ghchronicle/how/backfill/) is the opposite intention: it waits for
the window to turn over instead of skipping.