# Diagnostics & crash reports

This page explains Bookclerk's opt-in crash and error reports and its
opt-in usage check-ins: how to turn them on, what they contain, and how
logging verbosity works. The second half describes the collector service
that receives them, for the people who run it.

## Share reports

Reporting is off until you turn it on. The first-install setup asks,
after you start a new library or join one as a branch (by invite or by
signing in to the library), and the operator can change it any time under **Settings → Server Settings →
Diagnostics** in the web app, or in `config.toml`:

```toml
[diagnostics]
share_reports = true
```

A change from Settings applies without a restart; an edit to
`config.toml` applies on the daemon's next config reload or restart.

A build sends reports to the collector address baked in when it was
built. Release builds in CI pick it up on their own. To build with it
yourself:

```sh
BOOKCLERK_DIAGNOSTICS_COLLECTOR_URL="https://diagnostics.bookclerk.app" cargo build --release -p bookclerk-cli
```

To send reports somewhere else, set `collector_url` in `[diagnostics]`, or
`BOOKCLERK_DIAGNOSTICS_COLLECTOR_URL` at runtime.

> [!NOTE]
> The collector is served on `https://diagnostics.<DOMAIN>` when the
> `DOMAIN` Actions variable is set (`bookclerk.app` upstream). A fork
> without a domain gets its Cloudflare account's workers.dev URL instead.
> The deploy stores that URL in the Actions variable
> `DIAGNOSTICS_COLLECTOR_BASE_URL` (organization first, then repository).

### When a report is sent

With sharing on, a report is sent on a panic, an ERROR burst, a WARN
burst, a burst of slow responses, a daemon or CLI job failure, or a CLI
command failure. These keys tune it:

| Key | Default | Meaning |
| --- | --- | --- |
| `upload_on_crash` | `true` | Send the ring buffer from the panic hook |
| `upload_on_error_burst` | `true` | Send when ERROR events cross `error_burst_threshold` |
| `error_burst_threshold` | `10` | ERROR events inside the window that trigger a report |
| `error_burst_window_secs` | `60` | Sliding window for the error burst, in seconds |
| `upload_on_warn_burst` | `true` | Send when WARN events cross `warn_burst_threshold`, and when slow responses do (`slow_burst`) |
| `warn_burst_threshold` | `20` | WARN events (or slow responses) inside the window that trigger a report |
| `warn_burst_window_secs` | `60` | Sliding window for the warn burst, in seconds |
| `ring_buffer_capacity` | `200` | Most recent redacted events kept for a report (see [the ring buffer](#local-verbosity-vs-ring-buffer)) |

Slow responses are kept apart from faults. The daemon logs a request that
takes longer than a second (three seconds for static files and unmatched
paths) as a WARN on the target `bookclerk::slow`, with its method, route
template, status and `latency_ms`. Those warnings count toward their own
`slow_burst` trigger, not `warn_burst`, so a starved machine and a real
fault arrive as different kinds of report. A server error (5xx) is a WARN
named "server error response" with the same fields, and counts toward
`warn_burst`.

### Your report history

Each report the collector accepts gets a report ID (see
[Report IDs](#report-ids)) and an access code. The install keeps the last
200 reports it sent, with when and why each was sent and its code, in
`$BOOKCLERK_FILES_DIR/diagnostics/reports.json`. The file is readable by
its owner only, and the codes are registered with log redaction, so they
never show in logs or in a later report.

To follow up on a report, search the project's GitHub issues for its ID:
the issue the dashboard files for it, and its summary comment, name the
IDs it tracks. Or give the ID to a maintainer, who can open the report
itself.

The operator sees the history under **Settings → Server Settings →
Diagnostics**, where each ID has a copy button and a link that searches
the issues for it. From the command line:

```sh
bookclerk diagnostics reports
```

The API is `GET /api/diagnostics/reports` (operator only), newest first.

### Reading a report back

The access code is the credential for one report. The ID alone opens
nothing, so anyone who reads a GitHub issue can see the IDs it tracks but
not pull the reports.

- **View** in the history (or `bookclerk diagnostics view <report-id>`)
  reads the report back from the collector: the redacted log lines as they
  were sent, the GitHub issues tracking it, whether each is open, and the
  fix with the first release that has it, once one is recorded.
- Besides the log, a report carries the install's version and platform,
  the commit it was built from (when the build knew it, from
  `BOOKCLERK_BUILD_COMMIT` or the checkout's `git rev-parse HEAD`), and on
  Linux the CPU and memory limits of its cgroup next to the machine's CPU
  count. The member dashboard shows both on the report page.
- **Delete** (or `bookclerk diagnostics delete <report-id>`) removes the
  collector's stored copy. Its one-line summary (version, system and type
  of error, no log) and its issue links stay, so the report still counts
  toward the issue it joined. The history marks it deleted.
- If neither the web app nor the CLI can start, open
  `https://diagnostics.bookclerk.app/lookup` (your collector's `/lookup`)
  and paste the ID and code from `diagnostics/reports.json`. The page shows
  the same report and has the same delete button.

The daemon makes these calls on the operator's behalf with
`GET /api/diagnostics/reports/{id}` and
`DELETE /api/diagnostics/reports/{id}` (operator only), so the web app's
CSP and the collector's closed CORS stay as they are. The hosted demo
lets a visitor view the reports their copy sent, but not delete them,
because other people may be working on the demo's bug.

Reports sent before the collector issued codes, or past its daily index
cap, have none: their rows show the ID only. A collector without the
dashboard database issues no codes and answers lookups with `503`.

## Local verbosity vs ring buffer

What you see in the logs and what goes into a report are set separately.
The design behind this split is in
[ADR: Logging and tracing](/docs/adr/observability/index.md).

- **stderr and the OS facility** (journald, macOS `os_log` under the
  subsystem `app.bookclerk`, Windows Event Log) honor `BOOKCLERK_LOG`,
  then `RUST_LOG`, then the daemon default `bookclerk=info,warn`. The CLI is quiet by default (`off`). Use `-v`,
  `-vv` or `-vvv`, or `BOOKCLERK_LOG`.
- The **diagnostics ring buffer keeps Bookclerk's own events at TRACE and
  above, and every dependency's at INFO and above**, so crash and burst
  reports include deep context even when the console is quiet, without a
  library's protocol frames (HTTP/2, TLS) pushing it all out.
- **Warnings and errors outlive the ring.** The last 64 of them are kept
  beside it, so a report still carries the warnings that triggered it after
  newer events have filled the ring.
- **Events carry their span's fields.** An event inside a request span
  carries that span's fields as `span.field` (for example
  `http.request.method` and `http.request.route`), except the concrete
  request path, which stays local.
- **Events don't repeat their span's fields.** An event field with the
  same name and value as one of its span's fields is dropped from the
  report, so `http.request.method` appears once per line.
- Records from the `log` crate arrive under their own target and module,
  not as `log`.
- **The WebSocket library's own log records reach neither.** It traces
  every frame's payload in hex and the request head of each handshake,
  and a frame or a header can carry a bearer token in a form the
  redaction would not recognize. Bookclerk's own events about its sockets
  are logged as usual.

> [!TIP]
> To investigate locally, set `BOOKCLERK_LOG=bookclerk=debug`, or pass
> `-v` or `-vv` to the CLI.

Spawn stages and forwarded guest stderr are `bookclerk::spawn` tracing
events, so they follow that filter and stay inside the JSON daemon sink.

- A guest's or helper's stderr line is relayed at the level it names:
  a line whose first word (or second, after a timestamp) is `TRACE`,
  `DEBUG` or `INFO` keeps that level, with the prefix removed, and `WARN`
  or `ERROR` is relayed as WARN, so a guest can't raise an ERROR burst on
  the host. A line with no level word is relayed whole at INFO. Each line
  names its guest in `guest_ref`.
- Raw `bookclerk-spawn:` lines are written only when
  `BOOKCLERK_SPAWN_DIAG` is `1`, `true` or `stderr` (gateway CI sets
  this).
- Each spawn step is logged as its own text (`spawn begin plugin=…`).
  The process-wide stage ring numbers every step of every spawn, and
  only the raw `bookclerk-spawn: stage N:` lines carry that number.
- The process-wide stage ring and the per-session stderr tail are
  byte-capped. Plugin stderr isn't kept in the stage ring after the
  session exits.

## Redaction & privacy

Reports are redacted before they leave the machine:

- Secrets are redacted by exact value, and an upload is aborted if a
  registered secret remains. The Worker applies its own heuristics too.
- Titles, paths, account ids and secrets are fully redacted. A field whose
  name looks like a secret's (`password`, `*_token`, `*_key` and similar)
  is redacted by name, except `plugin_key`, which is a public plugin
  identity.
- Emails are **partially** masked, not fully `[REDACTED]`:
  `address@sub.domain.tld` becomes `a*****s@***.d****n.tld`. The local
  part and the registrable domain label keep their first and last
  characters, middle subdomain labels become `***`, and the TLD is
  unchanged.

A report includes `os`, `arch`, the Linux or macOS distribution string,
and the rustc channel and release.

The collector keeps no client IP address, and no hash of one. Each report
stores the uploader's `User-Agent` (at most 200 characters).

> [!NOTE]
> The hosted demo's copies send reports with sharing on, so the project
> hears about the demo's bugs. They carry the same redaction, and every
> demo copy shares one node id (see
> [`tools/demo/README.md`](https://github.com/Bookclerk/bookclerk/blob/main/tools/demo/README.md)).

Reports are kept for 180 days (the website's privacy page says 6 months).
See [Retention](#retention) for how that is enforced.

## Share usage

Separately from reports, an install can tell the project that it exists,
so the project has a rough idea of how widely Bookclerk is used. This is
its own switch, apart from `share_reports`, and it is off until you turn
it on:

```toml
[diagnostics]
share_usage = true
```

Or from the command line:

```sh
bookclerk config set diagnostics.share_usage true
```

The first-install setup asks about it too, next to report sharing, and
the operator can change it under **Settings → Server Settings →
Diagnostics**. A change from Settings applies without a restart: within
a minute of the switch turning on, the daemon loads (or makes) the
install ID and sends a check-in if one is due. A report sent before then
loads the ID itself. A report goes without one when the ID cannot be
loaded or made: the ID file cannot be read or written, or the report is
sent before the process knows its files directory (the CLI's earliest
start-up, for example).

With it on, the daemon and the desktop app send one check-in a week to
the collector's `/checkin` route. A check-in holds only:

- a random install ID, made the first time and kept in
  `$BOOKCLERK_FILES_DIR/install-id`;
- the Bookclerk version;
- the operating system and CPU architecture;
- whether it is the daemon or the desktop app.

The collector never stores the install ID. It keeps
`SHA-256(salt ‖ install ID)`, the **install key**, where the salt is 32
random bytes made for the current UTC month and deleted when the month
ends. So the project can count installs per week and per month, but cannot
follow one install from one month to the next. Check-in rows are deleted
after 35 days. Like reports, check-ins keep no IP address.

When usage sharing is on, reports carry the install ID too (from the
daemon, the desktop app and the `bookclerk` CLI alike), and the
collector swaps it for the same install key before the report is stored.
That lets the dashboard say how many installs a failure came from. With
usage sharing off, reports carry no install identity at all.

To start over with a new install ID:

```sh
bookclerk diagnostics reset-id
```

> [!NOTE]
> The hosted demo never checks in: its copies set `demo = true` in
> `[diagnostics]`, which turns check-ins off and marks their reports as
> demo reports, so the dashboard keeps them out of its install figures.

## Running the collector

The rest of this page is for the maintainers who run the collector
service.

```mermaid
flowchart TB
  Hosts["bookclerk / bookclerkd"] -- "POST /submit" --> Worker["Collector Worker<br/>(assigns report_id UUID)"]
  Hosts -- "POST /checkin (weekly, opt-in)" --> Worker
  Worker --> B2[("B2: diagnostics/&lt;version&gt;/&lt;report_id&gt;.json")]
  B2 --> Ingest["GitHub ingest"]
  Ingest -- "POST /triage" --> AI["Workers AI (no tools)"]
  AI --> Plan["validated plan"]
  Plan --> Issues["GitHub Issues"]
  Members["Organization members"] -- "GitHub sign-in" --> Dash["Member dashboard"]
  Dash --> Index[("D1 index of the B2 reports, issues and pull requests,<br/>synced from GitHub every 30 minutes")]
```

### GitHub Actions

| Workflow | Purpose |
| --- | --- |
| `diagnostics-collector-deploy` | Deploy the Worker on `diagnostics.<DOMAIN>`, or on workers.dev when `DOMAIN` is unset (and store that URL in `DIAGNOSTICS_COLLECTOR_BASE_URL`). Apply the dashboard D1 migrations, upload the dashboard's App secrets when set, then check `/health` |
| `diagnostics-ingest` | Daily `/report` pull, `/triage` on Workers AI, issues from the validated plan, regression issues, and one summary comment per issue kept in place |
| `ci` | Release builds with `BOOKCLERK_DIAGNOSTICS_COLLECTOR_URL` set to the collector URL |

Permissions are least privilege. The keys follow GitHub's parameterized
names (hyphenated in workflow YAML); see
[repository permissions](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#repository-permissions).

| Workflow | Permissions |
| --- | --- |
| Deploy | `contents: read`. In workers.dev mode an automation App token ("Variables: write") stores the URL |
| Ingest | `contents: read`, `issues: write` |

The secrets are `CLOUDFLARE_API_TOKEN`, `CLOUDFLARE_ACCOUNT_ID`,
`DIAGNOSTICS_REPORT_API_KEY` and `DIAGNOSTICS_B2_*`. The member dashboard
also needs `AUTOMATION_APP_CLIENT_ID`, `AUTOMATION_APP_CLIENT_SECRET` and
`AUTOMATION_APP_PRIVATE_KEY` (GitHub's PKCS#1 key file as downloaded).

> [!IMPORTANT]
> `REPORT_API_KEY` exists only as a Worker secret and the
> `DIAGNOSTICS_REPORT_API_KEY` Actions secret. Bookclerk builds carry only
> the collector URL, so a release can submit reports but never read or
> triage them.

### Retention

B2 enforces the 180-day retention with a lifecycle rule on the bucket. An
operator sets it once, because the Worker's B2 key can't change bucket
settings:

| Field | Value |
| --- | --- |
| File name prefix | `diagnostics/` |
| Days from uploading to hiding | `180` |
| Days from hiding to deleting | `1` |

To set it in the Backblaze web console:

1. Open **Buckets**, then the bucket.
2. Open **Lifecycle Settings** and choose **Use custom lifecycle rules**.
3. Add the rule above and save.

Or with the B2 command-line tool:

```sh
b2 bucket update --lifecycle-rule \
  '{"fileNamePrefix": "diagnostics/", "daysFromUploadingToHiding": 180, "daysFromHidingToDeleting": 1}' \
  <bucket-name>
```

"Days from hiding to deleting" also removes a replaced older version of a
file one day later, as the "Keep only the last version" preset does. To
keep that preset for the rest of the bucket, add a second rule with an
empty prefix, no hide days and `1` day from hiding to deleting.

B2 allows overlapping prefixes and applies, for each setting, the smallest
value any matching rule sets. So `diagnostics/` files still hide at 180
days (see Backblaze's
[lifecycle rules](https://www.backblaze.com/docs/cloud-storage-lifecycle-rules)).

> [!WARNING]
> Leave the catch-all rule's hide days empty. A number there would expire
> reports sooner too.

The dashboard's index row for a report (version, platform, trigger, level
counts and the error signature, no log text) stays after the report is
deleted, so counts and issue links keep working. Opening a deleted report
shows that it couldn't be loaded.

## Report IDs

Each `/submit` gets a UUID `report_id`. The B2 object is named
`diagnostics/<version>/<report_id>.json`.

### Access codes

When the dashboard database is configured, `/submit` also answers with an
`access_code`: 16 characters of Crockford's base32 alphabet (80 random
bits), shown in groups of four (`K7QM-3XRD-PW9T-H2CF`). Typed codes ignore
case, hyphens and spaces, and read `O` as `0` and `I` or `L` as `1`.

- The collector keeps only a SHA-256 hash of `<report_id>:<code>` in the
  `report_access` table, never the code, and never in the B2 object, so
  the ingest Action, triage and GitHub never see it. Hashes are compared
  in constant time.
- Codes have their own daily count in `index_days`, capped at
  `DASHBOARD_INDEX_DAILY_ROWS` like the index rows. A slot is reserved
  before the row is written, so overlapping uploads cannot pass the cap.
  Past it the report is stored without a code. The cron deletes rows
  older than 181 days, once a day.
- `POST /lookup` takes `{"report_id", "access_code"}` as JSON (what
  Bookclerk sends) or as a form (the page) in the body, never the URL. A
  wrong code and an unknown ID both get the same `404`.
- `POST /lookup/delete` deletes every B2 version of the object, then marks
  the report deleted in `report_access` and in the dashboard's index row.
  The index row, the issue links and the ledger stay. A form must tick the
  confirmation (`confirm=yes`).
- Attempts are limited to 10 a minute per report ID with the
  `LOOKUP_LIMIT` rate-limiting binding, so no IP address is involved.
- `GET /lookup` is the public form, with no script and the dashboard's
  security headers. The dashboard's sign-in page links to it, under the
  sign-in card, for a sender who lands there by mistake.

For deletion the Worker's B2 key needs `deleteFiles`, besides the
`listFiles` that `/report` already uses. Without it a delete answers `502`
and marks nothing, so the sender can try again.

Every diagnostics issue ends with the IDs it was filed with, in an HTML
comment GitHub doesn't render:

```text
<!-- bookclerk-diagnostics: report ids
<report_id>
-->
```

- The dashboard reads that block to link the reports to the issue.
- Operators can still copy an ID from the raw body to pull the full
  object from B2.
- People read the issue's summary comment instead (see
  [Summary comment](#summary-comment)). It shows what the issue tracks
  now, with dashboard links (for organization members only, by design).
  The filed list stays as it was filed, even after a member detaches or
  moves a report.

Issues filed before the block existed list their IDs under a visible
`### Report IDs` heading (`` - `<id>` · [dashboard](<url>) ``). The
dashboard reads that list only when a body has no block.

## Triage

`diagnostics-ingest` sends the sanitized batch to the collector's
`POST /triage` (with the same key as `/report`). The Worker asks a small
Workers AI model for an issue plan as JSON. The model is `TRIAGE_MODEL`,
by default `@cf/ibm-granite/granite-4.0-h-micro`, with a 131k-token
context.

The model has no tools, so a crafted report can at most shape that plan.
The workflow then validates the plan before anything is filed:

- at most five issues;
- fixed labels;
- only report IDs from the batch;
- bounded sizes;
- links removed, and mentions and issue references defanged.

An empty plan means the model decided nothing needs an issue, and files
nothing.

### The holding issue

Some reports go to one open **holding issue**,
`[diagnostics] untriaged reports`, instead:

- reports that don't fit in the batch's 120,000-byte limit;
- reports a non-empty plan leaves out of every issue;
- every report of a run whose reply held no valid plan, or that the
  budget or an outage kept from the model.

The run appends their IDs to the holding issue's report block
(`scripts/holding_issue.py` edits only that list), and comments once with
the reason. When the holding issue holds 500 IDs, or was filed with the
visible `### Report IDs` list instead, a new one is opened. So every
pulled report is tracked, and nothing is buried in a new issue per run.
Its summary comment, kept like every other issue's, is the list people
read.

### Attaching to open issues

When the dashboard database is bound, the collector puts the newest open
triaged issues (at most forty, number and title) in front of the batch.
The model may then answer with an `attach` list, instead of a new issue,
for a report that is the same failure as one of them.

The workflow validates that too:

- only issue numbers the collector offered (its answer's `open_issues`);
- only report UUIDs from the batch;
- each report under one issue;
- at most twenty issues;
- a report the model both attaches and files stays attached.

Nothing is written to the issue. The workflow sends the validated
attachments to the collector's `POST /issues/attach`. Its ledger records
each report under its issue as `triage` (an existing link, or a member's
exclusion for that pair, stands), and the next summary pass counts them on
the issue's comment.

Reports whose issue was closed or unknown by then, and every attached
report when the collector can't record them, go to the holding issue.
Without the dashboard, the model sees no open issues and plans new issues
only.

### Known failures skip the model

Ingest asks `/report` with `unfiled=1`. When the dashboard database is
bound, the collector checks each report against its
[report ledger](#report-ledger):

- **Already tracked.** It passes over reports some issue already tracks,
  and moves the cursor past them, so a re-pull from the start can't file
  them again.
- **Same failure as an open issue.** A report whose error signature an
  open, triaged issue already tracks, on the same `major.minor` version
  line, is linked to that issue (`source = signature`) and not returned.
  So a known failure grows its issue instead of reaching the model.
- **Same failure as a closed issue.** The report is compared with that
  issue's [fixed-in version](#fixed-in-version-and-regressions):
  - one that runs an older version on the same `major.minor` line is
    linked to the closed issue the same way, and not returned;
  - one that runs the fixed-in version or later is returned marked
    `regression_of`, with the closed issue, its fixed-in version, fix pull
    request and title (when several closed issues qualify, the one with
    the newest fix);
  - a closed issue on the same line with no fix recorded (closed as not a
    bug or won't fix, or an earlier regression issue a member closed)
    takes the report quietly instead, so a closed judgment isn't reopened
    by a new issue each time.

A `force_full` run turns the linking off and re-triages everything. The
pull log prints every count (`linked`, `regressions`).

### Regression issues

Ingest files the regressions without the model
(`scripts/regression_issue.py`):

- One `[diagnostics] Regression of #<n> in <major.minor>` issue per closed
  issue and version line, at most ten per run. The reports of further
  groups go to the holding issue.
- It is labeled `diagnostics,bug,regression`. Its body names the closed
  issue, the fix pull request and the fixed-in version as plain
  references, and lists the report IDs.
- Its first line is a marker naming the closed issue, which the dashboard
  reads so the issue shows as a regression of it.

The closed issue isn't reopened. While a regression issue is open, further
reports on its line join it through the signature link above. A report
from the same line that arrives before the issue exists is excluded from
the model's batch and from the holding issue.

### Summary comment

After the pull, every run (whether or not it pulled anything) keeps **one
comment per issue**, edited in place. Editing a comment notifies nobody.

1. The run asks `GET /issues/summaries` for the issues whose figures
   changed since their summary was last written (at most twenty issues
   per run).
2. It writes each comment (`scripts/update-summaries.sh` and the sans-IO
   renderer `scripts/summary_comment.py`), marked
   `<!-- bookclerk-diagnostics: summary -->` on its first line.
3. It echoes the summary's fingerprint to `POST /issues/summary` once the
   comment is written. The issue isn't handed out again until its figures
   change.

The comment shows:

- how many reports the issue tracks, and from how many installs. Only
  installs that [share usage](#share-usage) carry an install key, so when
  some of the reports carry none the count reads "at least N installs",
  and it is left out when none do. Install keys change every month, so an
  install that reported in two months counts twice;
- how they were linked: listed in the issue, matched by signature, added
  by triage, or attached by members;
- the first and last receipt;
- the versions and platforms seen (the top eight, the rest as "others");
- the newest ten reports, with dashboard links.

The figures come from the collector's ledger, with each report counted
under the issue that tracks it. So a member's attach or detach on the
dashboard reaches GitHub on the next run.

- The comment edited is the first one the workflow's own account wrote
  with the marker. A marker someone else posts is ignored.
- The report block in a triage or regression issue's body is never
  rewritten. Only the ingest run appends to the holding issue's block.
- Versions and platforms are untrusted report fields, and are validated or
  rewritten before they reach the comment. Report IDs link only to their
  own dashboard page.

### Triage budget

Spend is capped per UTC day by `TRIAGE_DAILY_NEURONS`. The default is
2,000, a fifth of the account's 10,000 free Workers AI neurons a day.
Values above 9,000 are clamped, so a typo can't reach paid usage. A call
costs at most a few hundred neurons.

1. Each call reserves its worst case (input bytes plus the 2,048-token
   output cap) in the `TRIAGE_BUDGET` KV namespace (one key per UTC day,
   expiring after eight days) before the model runs.
2. After the call it settles to the reported usage.
3. A call that would exceed the cap gets `429`, and ingest files one issue
   listing the report IDs instead.

The overview's **Triage budget** panel shows today's spend against the
cap and the total for the last 7 days. Triage runs once a day from the
ingest workflow and today's figure starts again at midnight UTC, so the
weekly total is what shows that it ran.

KV has no atomic increment, and its reads can be up to a minute stale. So
two calls close together (concurrent, or back-to-back runs such as a
scheduled run followed by a manual one) can each read an old total and
undercount. Each such lost update is one call's cost, far inside the gap
between the cap and the free allocation.

> [!NOTE]
> Wrangler creates the namespace on the first deploy (the binding has no
> id), so that deploy's Cloudflare API token needs Workers KV Storage
> edit. See
> [Cloudflare Worker deploys](/docs/contributing/automation/index.md#cloudflare-worker-deploys).

## Member dashboard

With the automation GitHub App's credentials set, the collector serves a
dashboard for members of the repository owner's organization. `/` signs
in with GitHub, and these pages are member-only:

| Page | Shows |
| --- | --- |
| `/dash` | The overview: reports per day, the [triage queue](#triage-queue-and-mutes), active installs from [usage check-ins](#share-usage), top signatures, platforms and versions (with errors and warnings per report) |
| `/reports` | A filterable list of reports (version, platform, trigger, filed, triage state), with the slowest request in each and checkboxes to mark a page of them at once |
| `/reports/<report_id>` | One report: its facts (route and slowest request when it has them, build commit, CPU and memory limits), triage state and mute, and its event log |
| `/issues` | Diagnostics issues, their pull requests, the version each fix shipped in, and reports that keep arriving from that version on |
| `/issues/<number>` | One issue's reports, with the [corrections](#corrections) below, a **Copy branch command** button, and for owners the [close buttons](#closing-issues) |
| `/demo` | The [hosted demo's runs](#hosted-demo-runs): runs open now, and runs, hours of use, run length and startup time per day over 30, 90 or 365 days |

The overview and the reports list show every report by default. A
**Source** switch narrows them to real installs or to the hosted demo, and
the choice is remembered between the two pages (in a `__Host-bc_source`
cookie holding only `all`, `installs` or `demo`). Demo reports carry a
**Demo** chip wherever they are listed.

A report page shows the app's version, build commit and Rust toolchain.
The index keeps the commit and toolchain, so they still show after the
sender deletes the stored copy.

The event log on a report page is built for reading a flood:

- level toggles, and crate toggles (the part of the target before `::`);
- runs of five or more debug and trace lines in a row from one crate fold
  into a line that names the crate and the count, and open on a click;
- a pause of more than 50 ms between two events is marked;
- **Jump to the first warning or error** links to it;
- **Copy shown** copies the lines on screen and **Copy all** every line,
  as plain text under a line naming the report; **Copy for issue** copies
  the report's facts and the shown lines as Markdown for a GitHub issue.

Links that leave the dashboard, such as issues and pull requests on
GitHub, open in a new tab without an opener or a referrer.

Without the credentials the routes stay off, and `/` answers like
`/health`.

> [!IMPORTANT]
> The deploy only uploads the credentials. Removing the Actions secrets
> later leaves the Worker's copies in place. To turn the dashboard off,
> delete the Worker secrets `GITHUB_APP_CLIENT_ID`,
> `GITHUB_APP_CLIENT_SECRET` and `GITHUB_APP_PRIVATE_KEY` (in the
> Cloudflare dashboard or with `wrangler secret delete`).

### Sign-in

1. The Worker sends the member to GitHub with a random state and a PKCE
   challenge, kept in a 10-minute `__Host-bc_oauth` cookie.
2. It trades the returned code for a user token, reads `GET /user`, and
   revokes that token at once.
3. It checks `GET /orgs/<owner>/memberships/<login>` with its own
   installation token. Only `state: active` passes, and organization
   owners get the owner tools.

When the repository belongs to a user instead of an organization, only
that user can sign in.

The page the member opened rides along as a `next` path in the same
cookie, so sign-in returns to it. That covers dashboard pages only:
`/dash`, `/reports`, `/issues` and `/reports/<id>`, with a filter query.

Coming back to a sign-in later doesn't strand the member:

- A callback that returns to a browser already signed in (the Back button,
  a restored tab, a reload) opens the dashboard, because its code is spent.
- A callback whose 10-minute cookie has gone says the link was used or
  took too long, with a **Sign in with GitHub again** button.
- An open dashboard page that finds its session ended, when its live
  socket closes or a refresh is sent to sign-in, moves to the sign-in
  page with itself as `next`. A page that was hidden for more than 15
  minutes refreshes when it is shown again.

### Links from other sites

The session cookie is SameSite=Strict, so a browser leaves it off a
navigation that starts on another site, such as a report link in a GitHub
issue, chat or email.

A dashboard page opened that way by a click, without a session cookie,
answers with a tiny page that reloads the same address. A click is
`Sec-Fetch-Site: cross-site`, `Sec-Fetch-Mode: navigate`,
`Sec-Fetch-Dest: document` and `Sec-Fetch-User: ?1`. That reload is
same-site, so it carries the cookie when there is one, and otherwise lands
on the sign-in page with the address kept as `next`.

- The reload isn't user-activated, so it can never repeat.
- Only clicked top-level GET page loads get this. Another site's scripts
  and frames still get the sign-in page.
- Owner actions stay POSTs with a same-origin `Origin` check, and `/live`
  is never reloaded this way.

### What the Worker holds

The App can write to the repository, so every installation token the
Worker mints is narrowed in the request to `read` on `metadata`, `issues`,
`pull_requests`, `members` and `contents` (for the release list and
compare reads below), on this one repository. The App's installation must
grant each of them, or the token can't be minted and sign-in fails.

The writes the Worker can ask GitHub for are an issue's summary comment,
from the owner's **Update GitHub now** button, closing an issue, from the
owner's [close buttons](#closing-issues), and filing a new issue a member
confirms after [**Triage now**](#triage-now):

- For each such call it mints a separate token narrowed to `issues: write`
  and nothing else, and keeps no copy.
- It still can't write pull requests, code or workflows.
- If the installation doesn't grant Issues: write, the button says so. A
  summary comment is then left to the scheduled run.

The client secret and private key are Worker secrets only. Neither, nor
any user token, is logged, stored or sent to a browser.
`scripts/tests/test_ci_contract.py` keeps them out of Bookclerk builds and
other workflows.

The App needs, in its settings:

- the organization permission **Members: read**, accepted on its
  installation;
- the repository permission **Issues: write**, for the **Update GitHub
  now**, close and **Open issue** buttons. The App's token is separate from
  the ingest Action's `GITHUB_TOKEN`, so the Action's permission doesn't
  cover it;
- the callback URL `https://diagnostics.<DOMAIN>/auth/callback`, or the
  workers.dev URL.

### Sessions

A random 32-byte `__Host-bc_session` cookie (HttpOnly, Secure,
SameSite=Strict) names a row in D1. The row keeps the cookie only as its
SHA-256, with the member, their role and the session's times.

- Sessions end 8 hours after sign-in, so a member removed from the
  organization loses access within 8 hours.
- Sign-out deletes the row, and owners can sign everyone out.
- State-changing requests are POSTs that also need a same-origin `Origin`
  header. The dashboard sends `Referrer-Policy: no-referrer`, under which
  browsers send `Origin: null` on its own forms (such as **Sign out**), so
  `null` is accepted only with `Sec-Fetch-Site: same-origin`, which pages
  cannot set.
- Pages carry a strict Content-Security-Policy (only this origin's
  styles, script, fonts and images; no inline script or style),
  `no-store`, and `frame-ancestors 'none'`.
- Every answer from the collector, dashboard or not, is closed to other
  origins: `Cross-Origin-Resource-Policy: same-origin`,
  `Cross-Origin-Opener-Policy: same-origin`,
  `Cross-Origin-Embedder-Policy: require-corp` and `X-Frame-Options: DENY`,
  and never an `Access-Control-Allow-Origin`. No browser needs it: the
  daemon posts reports from its own process, and the Actions call the
  key-protected routes directly.
- Report and issue text is escaped everywhere.

### Data

`DASHBOARD_DB` is a D1 database, created on the first deploy, with its
schema in `tools/diagnostics-collector/migrations/`. It holds one index
row per report, written by `/submit` after the B2 upload: version,
platform, trigger, level counts, a signature, the route and slowest
request when the report has them, whether the hosted demo sent it, its
install key when it has one, and its triage state.

The signature is a hash of the first error's target and its message, with
numbers, IDs, paths, URLs and quoted text replaced (warnings are used
when there is no error). When that event names an HTTP route (a `route`
field, or the request span's `http.request.route`), the route template is
part of the signature too, so slow responses on different routes group
apart. Raw messages stay in B2. The report page
downloads that one object with the Worker's B2 credentials, never
`REPORT_API_KEY`.

Because `/submit` is open, indexing stops at `DASHBOARD_INDEX_DAILY_ROWS`
reports a UTC day. The default is 10,000. Each report writes six D1 rows,
and its access code two more (the code and its own daily count, capped at
the same number), so that is at most 80,000 of D1's 100,000 free daily row
writes. A report past the cap is still stored.

`/checkin` is open too, so it stores at most `CHECKIN_DAILY_ROWS` check-ins
a UTC day (default 5,000) and answers 429 past that. Each writes four
rows (the check-in, its indexes and its day's counter), at most 20,000 a
day. The rest is left for sign-ins, the access log,
the sync and triage marks. The cron deletes check-in rows after 35 days
and every month's salt but the current one.

The hosted demo's runs take one row each (see
[Hosted demo runs](#hosted-demo-runs)), kept 400 days.

An owner's **Index older reports** button (also used once after the first
deploy) adds stored reports the index is missing, up to 40 per press.

The cron sync and the reindex stay within Workers Free's 50 D1 queries and
50 subrequests per invocation. Each issue page, pull request batch or
listing page is written in one or two batches that also save the cursor,
and work left over continues in the next run or press.

### Hosted demo runs

The [hosted demo](https://github.com/Bookclerk/bookclerk/blob/main/tools/demo/README.md) reports each visitor run to
the collector as it happens, over a service binding: a private
Worker-to-Worker link to the collector's `DemoRuns` entrypoint, which has
no public route.

1. When a run starts, the demo sends its start and hard stop.
2. When the visitor's copy first answers, it sends how long startup took.
3. When the run stops, it sends when and why (the visitor left, it sat
   idle, it reached its time limit, it failed, its machine stopped, the
   demo's sweep ended it, or the visitor started a new run in its place).

Each event names the run by a random run ID the demo makes for it,
separate from the visitor's session, and repeats the run's start and hard
stop, so any one event can create the row. No address, cookie or session
reaches the collector. A run whose stop never arrives is closed as
**missed** 15 minutes after its hard stop; it counts as a run but not
toward use or run length. A stop that arrives later replaces that. Runs
still open are the live use the `/demo` page shows, and an open `/demo`
page updates as each event arrives. The cron deletes runs
after 400 days.

Reporting never holds the demo up: without the binding nothing is sent,
and a refused event, a failure or an answer slower than two seconds is
logged and dropped. Events are parsed strictly
(`parseDemoRunEvent` in `demo-runs.ts`, fuzzed in `demo-runs.fuzz.test.ts`).

### Report ledger

`report_issue` records which issue each report is linked to, and how
(`source`):

| `source` | Meaning |
| --- | --- |
| `filed` | Listed in the issue body. The sync replaces these rows from GitHub |
| `signature` | Linked at ingest (see [Known failures skip the model](#known-failures-skip-the-model)) |
| `triage` | The model attached it to an open issue (see [Attaching to open issues](#attaching-to-open-issues)) |
| `manual` | A member attached it on the dashboard |
| `excluded` | A member detached it from that issue. The sync, signature matching and triage leave the pair alone from then on |

Every count on the pages reads this ledger, and a report counts once,
under the issue that tracks it. A report two issues list is a conflict:
the older issue keeps it, the report page says so, and the list shows
"+1" after the issue chip.

### Corrections

Members fix the ledger from the dashboard. The pages are plain forms that
post to `/issues/link` from the same origin. Every change is written to
the access log under the member's login, and open pages refresh.

The issue page lists:

- the reports the issue body filed;
- the ones linked later (by signature, triage or a member, each saying how
  and by whom);
- the ones detached from it;
- suggestions: indexed reports that share a signature with the issue's
  reports and are linked to no issue.

The actions are:

- **Detach** writes `excluded` for the pair. A filed report stays detached
  when the sync next reads the body.
- **Attach** writes `manual`, replacing an earlier exclusion since the
  member now asks for it. It also excludes the report from every other
  issue linking it, so the member's choice wins over the oldest-issue
  rule above. The other issues' bodies still list the report, but their
  summary comments stop counting it, and **Attach again** there reverses
  it.
- **Move** is an attach to the other issue, which takes the report from
  every issue the same way, plus a detach from this one, in one batch.
- **Merge into** attaches every report linked to the issue to the chosen
  open issue, takes them all from every other issue, and detaches them all
  from the first. The member closes the emptied issue on GitHub as a
  duplicate; the Worker doesn't.
- **Dismiss** on a suggestion writes the exclusion, so ingest doesn't link
  it by signature later.

The report page offers **Detach** under each issue that lists it (how a
conflict is resolved), and an **Attach to issue** picker when none does.
The picker is an issue-number field with suggestions, empty until the
member picks: issues that track the report's signature come first.

### Triage queue and mutes

A report **needs triage** while no issue lists it, no member has marked
it, and its signature isn't muted. The overview counts these and lists
the newest; `/reports?triage=needs` is the whole queue.

- **Mark seen** and **Mark noise** take a report out of the queue (on its
  page, or for the checked reports of a list page). **Back to the queue**
  clears the mark. Marks post to `/reports/triage`.
- **Mute** on a report page mutes its signature, with a reason and an end:
  a number of days, or a version. A version mute ends when a report with
  that signature arrives from that version or a later one, since the
  failure is then back where it was meant to be fixed. Mutes post to
  `/signatures/mute`.
- A muted signature's reports leave the queue, and `GET /report?unfiled=1`
  passes over them as it does filed reports, so ingest files no issue for
  them.

Each mark and mute is written to the access log under the member's login,
and open pages refresh.

### Triage now

A report no issue lists yet has a **Triage now** button on its page, so a
member need not wait for the daily ingest run:

1. The button posts to `/reports/ai-triage`. The Worker reads the stored
   report and asks the triage model about it alone, on Workers AI and
   under the same daily budget as the ingest run, with the open issues it
   may join.
2. The page shows what the model proposes: a new issue with its title,
   labels and body, or the open issue the report belongs to. When the model
   judges the report noise, the page says so and nothing is proposed.
3. A member presses **Open issue** or **Attach to #N** (posting to
   `/reports/ai-triage/file`). The Worker files the issue with the App's
   Issues: write token and lists it under **Tracked by** at once, or
   attaches the report to the offered issue, as the ingest run would.

The model's reply is read as the ingest run's plan validator reads it,
since the issue is public: only this report, a prefixed one-line title,
a bounded body with no links, mentions, cross-references or raw HTML, and
the report block written by the Worker. Attachments go only to the issues
the model was offered.

- A proposal lasts an hour; after that, press **Triage now** again. Nothing
  is filed until a member confirms it.
- A report waits 10 minutes between presses, and the dashboard runs
  triage at most 20 times a day. Each press, filing and attachment is
  written to the access log.
- A report whose stored copy was deleted has no button, since triage has
  nothing to read.
- When the budget is spent, triage is not configured, or the model's answer
  can't be used, the page says so and the daily run still triages the
  report. Without Issues: write on the App's installation, **Open issue**
  says so and the proposal stays. See
  [What the Worker holds](#what-the-worker-holds).

### Closing issues

Owners get two buttons on an open issue's page, posting to
`/admin/close-issue`:

- **Close as not planned** closes the issue on GitHub with that reason.
- **Close as fixed in** takes a version, removes any other `fixed-in:`
  label (a reopened issue can carry an earlier one), adds the
  `fixed-in:<version>` label (which the sync reads as the issue's
  [fixed-in version](#fixed-in-version-and-regressions)), then closes the
  issue as completed.

The index marks the issue closed at once, and the next sync reads it back
from GitHub.

### Sending corrections to GitHub

None of this reaches GitHub directly. The issue page counts the ledger
rows written since the ingest Action last recorded the issue's summary
comment (`issues.summary_sent_at`), and the next run renders them into
that comment.

Owners get **Update GitHub now** next to that count, which writes that
comment at once:

1. The Worker renders the issue's summary with the same renderer as the
   run. `dashboard/summary.ts` is a port of `scripts/summary_comment.py`
   that the collector's tests hold to identical output.
2. It edits the bot comment carrying the summary marker in place, or posts
   it.
3. It records the fingerprint written, so the run doesn't write it again.

A refusal is shown on the page. A locked or deleted issue is recorded as
the run would record it; anything else is left for the scheduled run.

The two writers take turns per issue. A summary handed to the run isn't
written by the button until the run acknowledges it (or ten minutes
pass), and an issue a press holds is left out of the run's next hand-out.
So neither writes over the other's comment and then records the other's
figures.

### Issue sync

Every 30 minutes a cron syncs the issues:

1. It lists issues labeled `diagnostics` changed since the last run.
2. It reads the report IDs from each issue's hidden report block (or, for
   an issue filed before it, the `### Report IDs` list).
3. It asks GraphQL for the pull requests that close or reference each
   issue (the newest 50 cross-references).

A run covers up to 100 issues: changed issues first, then the rest in
turn, least recently checked first, so every issue is rechecked. An issue
that lost the `diagnostics` label is dropped, and a failed GraphQL call
keeps the links already stored.

Two other things run the same sync:

- an owner's **Sync issues from GitHub now** button, at once;
- the ingest Action, which calls `POST /issues/notify` with
  `REPORT_API_KEY` and `{"issue": <number>}` after filing each issue. The
  Worker re-reads that issue from GitHub and links it immediately,
  reading again for a few seconds if GitHub doesn't have the new issue
  yet. Only the number is taken from the caller, and a transferred issue
  isn't followed.

Each sign-in and report view is written to an access log, kept 90 days
and shown to owners. The overview also shows today's triage spend from
`TRIAGE_BUDGET`.

### Fixed-in version and regressions

For each closed issue with a merged pull request, the same sync records
the version the fix shipped in (`issue_fix`, one row per change; the
newest wins).

1. It reads the repository's whole release list, prereleases included,
   at most five pages of a hundred. A longer list is left to `fixed-in`
   labels, because releases are listed by creation time and one published
   late could hide past any shorter read.
2. It asks GitHub's compare API whether the oldest release published
   after the merge contains the pull request's merge commit, then the
   next two.
3. The first that does is the fixed-in version, with a leading `v`
   dropped. A prerelease counts.

Until one does, the fix is `unreleased`, and the next sync continues with
the releases after the last one compared. So a fix that only a later
release carries is found over a few runs, and a new release is checked as
soon as it exists.

- At most five issues are resolved per sync, so a backlog clears over a
  few runs. A GitHub failure leaves the rows as they were.
- A `fixed-in:<version>` label on the issue (such as `fixed-in:v0.4.2`:
  `MAJOR.MINOR.PATCH`, with an optional prerelease and build) sets the
  version by hand and stops the automatic lookup for that issue. The
  label's value wins over an earlier automatic row.
- A release row is final. A later pull request that references the issue
  doesn't move it.

A report is a **regression** when it shares a signature with an issue's
tracked reports and runs the fixed-in version or a later one. Versions
compare in semver order. A build suffix or distribution repack of the
fixed version counts as it, and a version that doesn't parse never does.

- The issues list counts regressions per issue, and the overview counts
  issues with any.
- The report page marks such a report "Regression" and names the fixed-in
  version.
- While a fix is unreleased, the count is instead the reports that arrived
  after the merge.
- Reports from versions older than the fix are expected, and not flagged.

Nothing reopens an issue automatically. A regression that reaches ingest
gets a new issue instead (see [Regression issues](#regression-issues)),
and the issues list marks that issue "regression of #n" with a link to the
closed one.

The installation token the Worker mints for these reads carries
`contents: read` in addition to the issue, pull request, member and
metadata reads. It still can't write.

### Live updates

Open pages update on their own, without polling.

1. Each page opens a WebSocket to `/live` (same origin, and a live session
   required). It is held by one SQLite-backed Durable Object, `LiveHub`
   (binding `LIVE`, created by the deploy's `[[migrations]]` entry),
   through the hibernation API, so idle pages cost nothing.
2. When `/submit` indexes a report, `/issues/notify` links an issue, or a
   sync or reindex stores something, the demo reports a run event, or the
   cron closes a missed run, the Worker tells the hub. Syncs push only when
   the stored issues, links or pull requests actually changed.
3. The hub sends each page the bare text `changed`. The push carries no
   data, and the hub never reads what pages send.
4. The page fetches its own URL once with its session cookie, and swaps in
   the elements marked `data-live`. It fetches at most once every 3
   seconds, and not while the tab is hidden.

A page that connects within 15 seconds of a change refreshes once,
covering a change made between its own reads and its connect.

- A live refresh of a report page skips the stored report. It skips the
  access-log entry only when that member's view of the report is already
  logged in the last 8 hours.
- Sockets close at the first push after the session they were opened
  under expires, and at once when an owner signs everyone out. A page
  whose refresh is sent to sign-in closes its own. The hub keeps at most
  200.
- Without the `LIVE` binding, pages simply stay as loaded.

A Durable Object request is counted for each change. That stays well
inside Workers Free's daily allowance, because indexing is capped (above).

## Ingest cursor

The collector keeps `diagnostics-ingest`'s place in the stored reports in
its D1 database.

1. The run pulls with `GET /report?cursor=stored`.
2. Once every report in the batch is filed, it stores where the pull
   ended with `POST /report/cursor` (with the same key as `/report`). So a
   failed filing is retried by the next run.

A stored cursor only moves forward. A `force_full` run pulls from the
start (`since=0`) and leaves the stored cursor where it was.

## Version filter on ingest

`GET /report` resolves the latest **stable** GitHub release
(`releases/latest`) and downloads only matching objects:

| Match | Example (baseline `1.2.3`) |
| --- | --- |
| Equal | `1.2.3` |
| Newer | `1.2.4`, `2.0.0-rc.1` |
| Packaging derivative | `1.2.3+nix`, `1.2.3-1ubuntu2` |

- With **no GitHub releases** yet, every version is eligible. That is the
  current state of this repository.
- Same-version prereleases such as `1.2.3-rc.1` are excluded once `1.2.3`
  is stable.

The daily ingest workflow resolves that release itself with its job token,
and passes it as `?baseline_version=` (empty when there is no release). So
the Worker doesn't call GitHub on that path.

> [!NOTE]
> Unauthenticated GitHub API calls from Cloudflare's shared egress
> addresses are rate-limited and fail with `403`.

When a call to B2 or GitHub fails, the Worker answers `502` with
`{"error": "upstream failed"}` and logs the failing stage.

- Callers that send the report key (ingest) also get a `detail` naming the
  stage, the HTTP status and B2's short error code, for example
  `b2 list failed: HTTP 401: unauthorized`. It never includes credentials
  or response bodies.
- `/submit` callers get no detail.

## Next steps

- [ADR: Logging and tracing](/docs/adr/observability/index.md): why logs and
  reports are set separately.
- [Security checklist](/docs/server/security/index.md): redaction and the other defaults.
- [Running Bookclerk](/docs/server/operations/index.md#logging): where the daemon's logs go.
