Diagnostics & crash reports

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

This page explains Bookclerk's opt-in crash and error reports and its opt-in usage check-ins: how to turn them on, what they contain, and how logging verbosity works. The second half describes the collector service that receives them, for the people who run it.

Share reports

Reporting is off until you turn it on. The first-install setup asks, after you start a new library or join one as a branch (by invite or by signing in to the library), and the operator can change it any time under Settings → Server Settings → Diagnostics in the web app, or in config.toml:

[diagnostics]
share_reports = true

A change from Settings applies without a restart; an edit to config.toml applies on the daemon's next config reload or restart.

A build sends reports to the collector address baked in when it was built. Release builds in CI pick it up on their own. To build with it yourself:

BOOKCLERK_DIAGNOSTICS_COLLECTOR_URL="https://diagnostics.bookclerk.app" cargo build --release -p bookclerk-cli

To send reports somewhere else, set collector_url in [diagnostics], or BOOKCLERK_DIAGNOSTICS_COLLECTOR_URL at runtime.

Note

The collector is served on https://diagnostics.<DOMAIN> when the DOMAIN Actions variable is set (bookclerk.app upstream). A fork without a domain gets its Cloudflare account's workers.dev URL instead. The deploy stores that URL in the Actions variable DIAGNOSTICS_COLLECTOR_BASE_URL (organization first, then repository).

When a report is sent

With sharing on, a report is sent on a panic, an ERROR burst, a WARN burst, a burst of slow responses, a daemon or CLI job failure, or a CLI command failure. These keys tune it:

Key Default Meaning
upload_on_crash true Send the ring buffer from the panic hook
upload_on_error_burst true Send when ERROR events cross error_burst_threshold
error_burst_threshold 10 ERROR events inside the window that trigger a report
error_burst_window_secs 60 Sliding window for the error burst, in seconds
upload_on_warn_burst true Send when WARN events cross warn_burst_threshold, and when slow responses do (slow_burst)
warn_burst_threshold 20 WARN events (or slow responses) inside the window that trigger a report
warn_burst_window_secs 60 Sliding window for the warn burst, in seconds
ring_buffer_capacity 200 Most recent redacted events kept for a report (see the ring buffer)

Slow responses are kept apart from faults. The daemon logs a request that takes longer than a second (three seconds for static files and unmatched paths) as a WARN on the target bookclerk::slow, with its method, route template, status and latency_ms. Those warnings count toward their own slow_burst trigger, not warn_burst, so a starved machine and a real fault arrive as different kinds of report. A server error (5xx) is a WARN named "server error response" with the same fields, and counts toward warn_burst.

Your report history

Each report the collector accepts gets a report ID (see Report IDs) and an access code. The install keeps the last 200 reports it sent, with when and why each was sent and its code, in $BOOKCLERK_FILES_DIR/diagnostics/reports.json. The file is readable by its owner only, and the codes are registered with log redaction, so they never show in logs or in a later report.

To follow up on a report, search the project's GitHub issues for its ID: the issue the dashboard files for it, and its summary comment, name the IDs it tracks. Or give the ID to a maintainer, who can open the report itself.

The operator sees the history under Settings → Server Settings → Diagnostics, where each ID has a copy button and a link that searches the issues for it. From the command line:

bookclerk diagnostics reports

The API is GET /api/diagnostics/reports (operator only), newest first.

Reading a report back

The access code is the credential for one report. The ID alone opens nothing, so anyone who reads a GitHub issue can see the IDs it tracks but not pull the reports.

  • View in the history (or bookclerk diagnostics view <report-id>) reads the report back from the collector: the redacted log lines as they were sent, the GitHub issues tracking it, whether each is open, and the fix with the first release that has it, once one is recorded.
  • Besides the log, a report carries the install's version and platform, the commit it was built from (when the build knew it, from BOOKCLERK_BUILD_COMMIT or the checkout's git rev-parse HEAD), and on Linux the CPU and memory limits of its cgroup next to the machine's CPU count. The member dashboard shows both on the report page.
  • Delete (or bookclerk diagnostics delete <report-id>) removes the collector's stored copy. Its one-line summary (version, system and type of error, no log) and its issue links stay, so the report still counts toward the issue it joined. The history marks it deleted.
  • If neither the web app nor the CLI can start, open https://diagnostics.bookclerk.app/lookup (your collector's /lookup) and paste the ID and code from diagnostics/reports.json. The page shows the same report and has the same delete button.

The daemon makes these calls on the operator's behalf with GET /api/diagnostics/reports/{id} and DELETE /api/diagnostics/reports/{id} (operator only), so the web app's CSP and the collector's closed CORS stay as they are. The hosted demo lets a visitor view the reports their copy sent, but not delete them, because other people may be working on the demo's bug.

Reports sent before the collector issued codes, or past its daily index cap, have none: their rows show the ID only. A collector without the dashboard database issues no codes and answers lookups with 503.

Local verbosity vs ring buffer

What you see in the logs and what goes into a report are set separately. The design behind this split is in ADR: Logging and tracing.

  • stderr and the OS facility (journald, macOS os_log under the subsystem app.bookclerk, Windows Event Log) honor BOOKCLERK_LOG, then RUST_LOG, then the daemon default bookclerk=info,warn. The CLI is quiet by default (off). Use -v, -vv or -vvv, or BOOKCLERK_LOG.
  • The diagnostics ring buffer keeps Bookclerk's own events at TRACE and above, and every dependency's at INFO and above, so crash and burst reports include deep context even when the console is quiet, without a library's protocol frames (HTTP/2, TLS) pushing it all out.
  • Warnings and errors outlive the ring. The last 64 of them are kept beside it, so a report still carries the warnings that triggered it after newer events have filled the ring.
  • Events carry their span's fields. An event inside a request span carries that span's fields as span.field (for example http.request.method and http.request.route), except the concrete request path, which stays local.
  • Events don't repeat their span's fields. An event field with the same name and value as one of its span's fields is dropped from the report, so http.request.method appears once per line.
  • Records from the log crate arrive under their own target and module, not as log.
  • The WebSocket library's own log records reach neither. It traces every frame's payload in hex and the request head of each handshake, and a frame or a header can carry a bearer token in a form the redaction would not recognize. Bookclerk's own events about its sockets are logged as usual.

Tip

To investigate locally, set BOOKCLERK_LOG=bookclerk=debug, or pass -v or -vv to the CLI.

Spawn stages and forwarded guest stderr are bookclerk::spawn tracing events, so they follow that filter and stay inside the JSON daemon sink.

  • A guest's or helper's stderr line is relayed at the level it names: a line whose first word (or second, after a timestamp) is TRACE, DEBUG or INFO keeps that level, with the prefix removed, and WARN or ERROR is relayed as WARN, so a guest can't raise an ERROR burst on the host. A line with no level word is relayed whole at INFO. Each line names its guest in guest_ref.
  • Raw bookclerk-spawn: lines are written only when BOOKCLERK_SPAWN_DIAG is 1, true or stderr (gateway CI sets this).
  • Each spawn step is logged as its own text (spawn begin plugin=…). The process-wide stage ring numbers every step of every spawn, and only the raw bookclerk-spawn: stage N: lines carry that number.
  • The process-wide stage ring and the per-session stderr tail are byte-capped. Plugin stderr isn't kept in the stage ring after the session exits.

Redaction & privacy

Reports are redacted before they leave the machine:

  • Secrets are redacted by exact value, and an upload is aborted if a registered secret remains. The Worker applies its own heuristics too.
  • Titles, paths, account ids and secrets are fully redacted. A field whose name looks like a secret's (password, *_token, *_key and similar) is redacted by name, except plugin_key, which is a public plugin identity.
  • Emails are partially masked, not fully [REDACTED]: address@sub.domain.tld becomes a*****s@***.d****n.tld. The local part and the registrable domain label keep their first and last characters, middle subdomain labels become ***, and the TLD is unchanged.

A report includes os, arch, the Linux or macOS distribution string, and the rustc channel and release.

The collector keeps no client IP address, and no hash of one. Each report stores the uploader's User-Agent (at most 200 characters).

Note

The hosted demo's copies send reports with sharing on, so the project hears about the demo's bugs. They carry the same redaction, and every demo copy shares one node id (see tools/demo/README.md).

Reports are kept for 180 days (the website's privacy page says 6 months). See Retention for how that is enforced.

Share usage

Separately from reports, an install can tell the project that it exists, so the project has a rough idea of how widely Bookclerk is used. This is its own switch, apart from share_reports, and it is off until you turn it on:

[diagnostics]
share_usage = true

Or from the command line:

bookclerk config set diagnostics.share_usage true

The first-install setup asks about it too, next to report sharing, and the operator can change it under Settings → Server Settings → Diagnostics. A change from Settings applies without a restart: within a minute of the switch turning on, the daemon loads (or makes) the install ID and sends a check-in if one is due. A report sent before then loads the ID itself. A report goes without one when the ID cannot be loaded or made: the ID file cannot be read or written, or the report is sent before the process knows its files directory (the CLI's earliest start-up, for example).

With it on, the daemon and the desktop app send one check-in a week to the collector's /checkin route. A check-in holds only:

  • a random install ID, made the first time and kept in $BOOKCLERK_FILES_DIR/install-id;
  • the Bookclerk version;
  • the operating system and CPU architecture;
  • whether it is the daemon or the desktop app.

The collector never stores the install ID. It keeps SHA-256(salt ‖ install ID), the install key, where the salt is 32 random bytes made for the current UTC month and deleted when the month ends. So the project can count installs per week and per month, but cannot follow one install from one month to the next. Check-in rows are deleted after 35 days. Like reports, check-ins keep no IP address.

When usage sharing is on, reports carry the install ID too (from the daemon, the desktop app and the bookclerk CLI alike), and the collector swaps it for the same install key before the report is stored. That lets the dashboard say how many installs a failure came from. With usage sharing off, reports carry no install identity at all.

To start over with a new install ID:

bookclerk diagnostics reset-id

Note

The hosted demo never checks in: its copies set demo = true in [diagnostics], which turns check-ins off and marks their reports as demo reports, so the dashboard keeps them out of its install figures.

Running the collector

The rest of this page is for the maintainers who run the collector service.

POST /submit POST /checkin (weekly, opt-in) POST /triage GitHub sign-in bookclerk / bookclerkd Collector Worker(assigns report_id UUID) B2: diagnostics/<version>/<report_id>.json GitHub ingest Workers AI (no tools) validated plan GitHub Issues Organization members Member dashboard D1 index of the B2 reports, issues and pull requests,synced from GitHub every 30 minutes

GitHub Actions

Workflow Purpose
diagnostics-collector-deploy Deploy the Worker on diagnostics.<DOMAIN>, or on workers.dev when DOMAIN is unset (and store that URL in DIAGNOSTICS_COLLECTOR_BASE_URL). Apply the dashboard D1 migrations, upload the dashboard's App secrets when set, then check /health
diagnostics-ingest Daily /report pull, /triage on Workers AI, issues from the validated plan, regression issues, and one summary comment per issue kept in place
ci Release builds with BOOKCLERK_DIAGNOSTICS_COLLECTOR_URL set to the collector URL

Permissions are least privilege. The keys follow GitHub's parameterized names (hyphenated in workflow YAML); see repository permissions.

Workflow Permissions
Deploy contents: read. In workers.dev mode an automation App token ("Variables: write") stores the URL
Ingest contents: read, issues: write

The secrets are CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, DIAGNOSTICS_REPORT_API_KEY and DIAGNOSTICS_B2_*. The member dashboard also needs AUTOMATION_APP_CLIENT_ID, AUTOMATION_APP_CLIENT_SECRET and AUTOMATION_APP_PRIVATE_KEY (GitHub's PKCS#1 key file as downloaded).

Important

REPORT_API_KEY exists only as a Worker secret and the DIAGNOSTICS_REPORT_API_KEY Actions secret. Bookclerk builds carry only the collector URL, so a release can submit reports but never read or triage them.

Retention

B2 enforces the 180-day retention with a lifecycle rule on the bucket. An operator sets it once, because the Worker's B2 key can't change bucket settings:

Field Value
File name prefix diagnostics/
Days from uploading to hiding 180
Days from hiding to deleting 1

To set it in the Backblaze web console:

  1. Open Buckets, then the bucket.
  2. Open Lifecycle Settings and choose Use custom lifecycle rules.
  3. Add the rule above and save.

Or with the B2 command-line tool:

b2 bucket update --lifecycle-rule \
  '{"fileNamePrefix": "diagnostics/", "daysFromUploadingToHiding": 180, "daysFromHidingToDeleting": 1}' \
  <bucket-name>

"Days from hiding to deleting" also removes a replaced older version of a file one day later, as the "Keep only the last version" preset does. To keep that preset for the rest of the bucket, add a second rule with an empty prefix, no hide days and 1 day from hiding to deleting.

B2 allows overlapping prefixes and applies, for each setting, the smallest value any matching rule sets. So diagnostics/ files still hide at 180 days (see Backblaze's lifecycle rules).

Warning

Leave the catch-all rule's hide days empty. A number there would expire reports sooner too.

The dashboard's index row for a report (version, platform, trigger, level counts and the error signature, no log text) stays after the report is deleted, so counts and issue links keep working. Opening a deleted report shows that it couldn't be loaded.

Report IDs

Each /submit gets a UUID report_id. The B2 object is named diagnostics/<version>/<report_id>.json.

Access codes

When the dashboard database is configured, /submit also answers with an access_code: 16 characters of Crockford's base32 alphabet (80 random bits), shown in groups of four (K7QM-3XRD-PW9T-H2CF). Typed codes ignore case, hyphens and spaces, and read O as 0 and I or L as 1.

  • The collector keeps only a SHA-256 hash of <report_id>:<code> in the report_access table, never the code, and never in the B2 object, so the ingest Action, triage and GitHub never see it. Hashes are compared in constant time.
  • Codes have their own daily count in index_days, capped at DASHBOARD_INDEX_DAILY_ROWS like the index rows. A slot is reserved before the row is written, so overlapping uploads cannot pass the cap. Past it the report is stored without a code. The cron deletes rows older than 181 days, once a day.
  • POST /lookup takes {"report_id", "access_code"} as JSON (what Bookclerk sends) or as a form (the page) in the body, never the URL. A wrong code and an unknown ID both get the same 404.
  • POST /lookup/delete deletes every B2 version of the object, then marks the report deleted in report_access and in the dashboard's index row. The index row, the issue links and the ledger stay. A form must tick the confirmation (confirm=yes).
  • Attempts are limited to 10 a minute per report ID with the LOOKUP_LIMIT rate-limiting binding, so no IP address is involved.
  • GET /lookup is the public form, with no script and the dashboard's security headers. The dashboard's sign-in page links to it, under the sign-in card, for a sender who lands there by mistake.

For deletion the Worker's B2 key needs deleteFiles, besides the listFiles that /report already uses. Without it a delete answers 502 and marks nothing, so the sender can try again.

Every diagnostics issue ends with the IDs it was filed with, in an HTML comment GitHub doesn't render:

<!-- bookclerk-diagnostics: report ids
<report_id>
-->
  • The dashboard reads that block to link the reports to the issue.
  • Operators can still copy an ID from the raw body to pull the full object from B2.
  • People read the issue's summary comment instead (see Summary comment). It shows what the issue tracks now, with dashboard links (for organization members only, by design). The filed list stays as it was filed, even after a member detaches or moves a report.

Issues filed before the block existed list their IDs under a visible ### Report IDs heading (- `<id>` · [dashboard](<url>)). The dashboard reads that list only when a body has no block.

Triage

diagnostics-ingest sends the sanitized batch to the collector's POST /triage (with the same key as /report). The Worker asks a small Workers AI model for an issue plan as JSON. The model is TRIAGE_MODEL, by default @cf/ibm-granite/granite-4.0-h-micro, with a 131k-token context.

The model has no tools, so a crafted report can at most shape that plan. The workflow then validates the plan before anything is filed:

  • at most five issues;
  • fixed labels;
  • only report IDs from the batch;
  • bounded sizes;
  • links removed, and mentions and issue references defanged.

An empty plan means the model decided nothing needs an issue, and files nothing.

The holding issue

Some reports go to one open holding issue, [diagnostics] untriaged reports, instead:

  • reports that don't fit in the batch's 120,000-byte limit;
  • reports a non-empty plan leaves out of every issue;
  • every report of a run whose reply held no valid plan, or that the budget or an outage kept from the model.

The run appends their IDs to the holding issue's report block (scripts/holding_issue.py edits only that list), and comments once with the reason. When the holding issue holds 500 IDs, or was filed with the visible ### Report IDs list instead, a new one is opened. So every pulled report is tracked, and nothing is buried in a new issue per run. Its summary comment, kept like every other issue's, is the list people read.

Attaching to open issues

When the dashboard database is bound, the collector puts the newest open triaged issues (at most forty, number and title) in front of the batch. The model may then answer with an attach list, instead of a new issue, for a report that is the same failure as one of them.

The workflow validates that too:

  • only issue numbers the collector offered (its answer's open_issues);
  • only report UUIDs from the batch;
  • each report under one issue;
  • at most twenty issues;
  • a report the model both attaches and files stays attached.

Nothing is written to the issue. The workflow sends the validated attachments to the collector's POST /issues/attach. Its ledger records each report under its issue as triage (an existing link, or a member's exclusion for that pair, stands), and the next summary pass counts them on the issue's comment.

Reports whose issue was closed or unknown by then, and every attached report when the collector can't record them, go to the holding issue. Without the dashboard, the model sees no open issues and plans new issues only.

Known failures skip the model

Ingest asks /report with unfiled=1. When the dashboard database is bound, the collector checks each report against its report ledger:

  • Already tracked. It passes over reports some issue already tracks, and moves the cursor past them, so a re-pull from the start can't file them again.
  • Same failure as an open issue. A report whose error signature an open, triaged issue already tracks, on the same major.minor version line, is linked to that issue (source = signature) and not returned. So a known failure grows its issue instead of reaching the model.
  • Same failure as a closed issue. The report is compared with that issue's fixed-in version:
    • one that runs an older version on the same major.minor line is linked to the closed issue the same way, and not returned;
    • one that runs the fixed-in version or later is returned marked regression_of, with the closed issue, its fixed-in version, fix pull request and title (when several closed issues qualify, the one with the newest fix);
    • a closed issue on the same line with no fix recorded (closed as not a bug or won't fix, or an earlier regression issue a member closed) takes the report quietly instead, so a closed judgment isn't reopened by a new issue each time.

A force_full run turns the linking off and re-triages everything. The pull log prints every count (linked, regressions).

Regression issues

Ingest files the regressions without the model (scripts/regression_issue.py):

  • One [diagnostics] Regression of #<n> in <major.minor> issue per closed issue and version line, at most ten per run. The reports of further groups go to the holding issue.
  • It is labeled diagnostics,bug,regression. Its body names the closed issue, the fix pull request and the fixed-in version as plain references, and lists the report IDs.
  • Its first line is a marker naming the closed issue, which the dashboard reads so the issue shows as a regression of it.

The closed issue isn't reopened. While a regression issue is open, further reports on its line join it through the signature link above. A report from the same line that arrives before the issue exists is excluded from the model's batch and from the holding issue.

Summary comment

After the pull, every run (whether or not it pulled anything) keeps one comment per issue, edited in place. Editing a comment notifies nobody.

  1. The run asks GET /issues/summaries for the issues whose figures changed since their summary was last written (at most twenty issues per run).
  2. It writes each comment (scripts/update-summaries.sh and the sans-IO renderer scripts/summary_comment.py), marked <!-- bookclerk-diagnostics: summary --> on its first line.
  3. It echoes the summary's fingerprint to POST /issues/summary once the comment is written. The issue isn't handed out again until its figures change.

The comment shows:

  • how many reports the issue tracks, and from how many installs. Only installs that share usage carry an install key, so when some of the reports carry none the count reads "at least N installs", and it is left out when none do. Install keys change every month, so an install that reported in two months counts twice;
  • how they were linked: listed in the issue, matched by signature, added by triage, or attached by members;
  • the first and last receipt;
  • the versions and platforms seen (the top eight, the rest as "others");
  • the newest ten reports, with dashboard links.

The figures come from the collector's ledger, with each report counted under the issue that tracks it. So a member's attach or detach on the dashboard reaches GitHub on the next run.

  • The comment edited is the first one the workflow's own account wrote with the marker. A marker someone else posts is ignored.
  • The report block in a triage or regression issue's body is never rewritten. Only the ingest run appends to the holding issue's block.
  • Versions and platforms are untrusted report fields, and are validated or rewritten before they reach the comment. Report IDs link only to their own dashboard page.

Triage budget

Spend is capped per UTC day by TRIAGE_DAILY_NEURONS. The default is 2,000, a fifth of the account's 10,000 free Workers AI neurons a day. Values above 9,000 are clamped, so a typo can't reach paid usage. A call costs at most a few hundred neurons.

  1. Each call reserves its worst case (input bytes plus the 2,048-token output cap) in the TRIAGE_BUDGET KV namespace (one key per UTC day, expiring after eight days) before the model runs.
  2. After the call it settles to the reported usage.
  3. A call that would exceed the cap gets 429, and ingest files one issue listing the report IDs instead.

The overview's Triage budget panel shows today's spend against the cap and the total for the last 7 days. Triage runs once a day from the ingest workflow and today's figure starts again at midnight UTC, so the weekly total is what shows that it ran.

KV has no atomic increment, and its reads can be up to a minute stale. So two calls close together (concurrent, or back-to-back runs such as a scheduled run followed by a manual one) can each read an old total and undercount. Each such lost update is one call's cost, far inside the gap between the cap and the free allocation.

Note

Wrangler creates the namespace on the first deploy (the binding has no id), so that deploy's Cloudflare API token needs Workers KV Storage edit. See Cloudflare Worker deploys.

Member dashboard

With the automation GitHub App's credentials set, the collector serves a dashboard for members of the repository owner's organization. / signs in with GitHub, and these pages are member-only:

Page Shows
/dash The overview: reports per day, the triage queue, active installs from usage check-ins, top signatures, platforms and versions (with errors and warnings per report)
/reports A filterable list of reports (version, platform, trigger, filed, triage state), with the slowest request in each and checkboxes to mark a page of them at once
/reports/<report_id> One report: its facts (route and slowest request when it has them, build commit, CPU and memory limits), triage state and mute, and its event log
/issues Diagnostics issues, their pull requests, the version each fix shipped in, and reports that keep arriving from that version on
/issues/<number> One issue's reports, with the corrections below, a Copy branch command button, and for owners the close buttons
/demo The hosted demo's runs: runs open now, and runs, hours of use, run length and startup time per day over 30, 90 or 365 days

The overview and the reports list show every report by default. A Source switch narrows them to real installs or to the hosted demo, and the choice is remembered between the two pages (in a __Host-bc_source cookie holding only all, installs or demo). Demo reports carry a Demo chip wherever they are listed.

A report page shows the app's version, build commit and Rust toolchain. The index keeps the commit and toolchain, so they still show after the sender deletes the stored copy.

The event log on a report page is built for reading a flood:

  • level toggles, and crate toggles (the part of the target before ::);
  • runs of five or more debug and trace lines in a row from one crate fold into a line that names the crate and the count, and open on a click;
  • a pause of more than 50 ms between two events is marked;
  • Jump to the first warning or error links to it;
  • Copy shown copies the lines on screen and Copy all every line, as plain text under a line naming the report; Copy for issue copies the report's facts and the shown lines as Markdown for a GitHub issue.

Links that leave the dashboard, such as issues and pull requests on GitHub, open in a new tab without an opener or a referrer.

Without the credentials the routes stay off, and / answers like /health.

Important

The deploy only uploads the credentials. Removing the Actions secrets later leaves the Worker's copies in place. To turn the dashboard off, delete the Worker secrets GITHUB_APP_CLIENT_ID, GITHUB_APP_CLIENT_SECRET and GITHUB_APP_PRIVATE_KEY (in the Cloudflare dashboard or with wrangler secret delete).

Sign-in

  1. The Worker sends the member to GitHub with a random state and a PKCE challenge, kept in a 10-minute __Host-bc_oauth cookie.
  2. It trades the returned code for a user token, reads GET /user, and revokes that token at once.
  3. It checks GET /orgs/<owner>/memberships/<login> with its own installation token. Only state: active passes, and organization owners get the owner tools.

When the repository belongs to a user instead of an organization, only that user can sign in.

The page the member opened rides along as a next path in the same cookie, so sign-in returns to it. That covers dashboard pages only: /dash, /reports, /issues and /reports/<id>, with a filter query.

Coming back to a sign-in later doesn't strand the member:

  • A callback that returns to a browser already signed in (the Back button, a restored tab, a reload) opens the dashboard, because its code is spent.
  • A callback whose 10-minute cookie has gone says the link was used or took too long, with a Sign in with GitHub again button.
  • An open dashboard page that finds its session ended, when its live socket closes or a refresh is sent to sign-in, moves to the sign-in page with itself as next. A page that was hidden for more than 15 minutes refreshes when it is shown again.

The session cookie is SameSite=Strict, so a browser leaves it off a navigation that starts on another site, such as a report link in a GitHub issue, chat or email.

A dashboard page opened that way by a click, without a session cookie, answers with a tiny page that reloads the same address. A click is Sec-Fetch-Site: cross-site, Sec-Fetch-Mode: navigate, Sec-Fetch-Dest: document and Sec-Fetch-User: ?1. That reload is same-site, so it carries the cookie when there is one, and otherwise lands on the sign-in page with the address kept as next.

  • The reload isn't user-activated, so it can never repeat.
  • Only clicked top-level GET page loads get this. Another site's scripts and frames still get the sign-in page.
  • Owner actions stay POSTs with a same-origin Origin check, and /live is never reloaded this way.

What the Worker holds

The App can write to the repository, so every installation token the Worker mints is narrowed in the request to read on metadata, issues, pull_requests, members and contents (for the release list and compare reads below), on this one repository. The App's installation must grant each of them, or the token can't be minted and sign-in fails.

The writes the Worker can ask GitHub for are an issue's summary comment, from the owner's Update GitHub now button, closing an issue, from the owner's close buttons, and filing a new issue a member confirms after Triage now:

  • For each such call it mints a separate token narrowed to issues: write and nothing else, and keeps no copy.
  • It still can't write pull requests, code or workflows.
  • If the installation doesn't grant Issues: write, the button says so. A summary comment is then left to the scheduled run.

The client secret and private key are Worker secrets only. Neither, nor any user token, is logged, stored or sent to a browser. scripts/tests/test_ci_contract.py keeps them out of Bookclerk builds and other workflows.

The App needs, in its settings:

  • the organization permission Members: read, accepted on its installation;
  • the repository permission Issues: write, for the Update GitHub now, close and Open issue buttons. The App's token is separate from the ingest Action's GITHUB_TOKEN, so the Action's permission doesn't cover it;
  • the callback URL https://diagnostics.<DOMAIN>/auth/callback, or the workers.dev URL.

Sessions

A random 32-byte __Host-bc_session cookie (HttpOnly, Secure, SameSite=Strict) names a row in D1. The row keeps the cookie only as its SHA-256, with the member, their role and the session's times.

  • Sessions end 8 hours after sign-in, so a member removed from the organization loses access within 8 hours.
  • Sign-out deletes the row, and owners can sign everyone out.
  • State-changing requests are POSTs that also need a same-origin Origin header. The dashboard sends Referrer-Policy: no-referrer, under which browsers send Origin: null on its own forms (such as Sign out), so null is accepted only with Sec-Fetch-Site: same-origin, which pages cannot set.
  • Pages carry a strict Content-Security-Policy (only this origin's styles, script, fonts and images; no inline script or style), no-store, and frame-ancestors 'none'.
  • Every answer from the collector, dashboard or not, is closed to other origins: Cross-Origin-Resource-Policy: same-origin, Cross-Origin-Opener-Policy: same-origin, Cross-Origin-Embedder-Policy: require-corp and X-Frame-Options: DENY, and never an Access-Control-Allow-Origin. No browser needs it: the daemon posts reports from its own process, and the Actions call the key-protected routes directly.
  • Report and issue text is escaped everywhere.

Data

DASHBOARD_DB is a D1 database, created on the first deploy, with its schema in tools/diagnostics-collector/migrations/. It holds one index row per report, written by /submit after the B2 upload: version, platform, trigger, level counts, a signature, the route and slowest request when the report has them, whether the hosted demo sent it, its install key when it has one, and its triage state.

The signature is a hash of the first error's target and its message, with numbers, IDs, paths, URLs and quoted text replaced (warnings are used when there is no error). When that event names an HTTP route (a route field, or the request span's http.request.route), the route template is part of the signature too, so slow responses on different routes group apart. Raw messages stay in B2. The report page downloads that one object with the Worker's B2 credentials, never REPORT_API_KEY.

Because /submit is open, indexing stops at DASHBOARD_INDEX_DAILY_ROWS reports a UTC day. The default is 10,000. Each report writes six D1 rows, and its access code two more (the code and its own daily count, capped at the same number), so that is at most 80,000 of D1's 100,000 free daily row writes. A report past the cap is still stored.

/checkin is open too, so it stores at most CHECKIN_DAILY_ROWS check-ins a UTC day (default 5,000) and answers 429 past that. Each writes four rows (the check-in, its indexes and its day's counter), at most 20,000 a day. The rest is left for sign-ins, the access log, the sync and triage marks. The cron deletes check-in rows after 35 days and every month's salt but the current one.

The hosted demo's runs take one row each (see Hosted demo runs), kept 400 days.

An owner's Index older reports button (also used once after the first deploy) adds stored reports the index is missing, up to 40 per press.

The cron sync and the reindex stay within Workers Free's 50 D1 queries and 50 subrequests per invocation. Each issue page, pull request batch or listing page is written in one or two batches that also save the cursor, and work left over continues in the next run or press.

Hosted demo runs

The hosted demo reports each visitor run to the collector as it happens, over a service binding: a private Worker-to-Worker link to the collector's DemoRuns entrypoint, which has no public route.

  1. When a run starts, the demo sends its start and hard stop.
  2. When the visitor's copy first answers, it sends how long startup took.
  3. When the run stops, it sends when and why (the visitor left, it sat idle, it reached its time limit, it failed, its machine stopped, the demo's sweep ended it, or the visitor started a new run in its place).

Each event names the run by a random run ID the demo makes for it, separate from the visitor's session, and repeats the run's start and hard stop, so any one event can create the row. No address, cookie or session reaches the collector. A run whose stop never arrives is closed as missed 15 minutes after its hard stop; it counts as a run but not toward use or run length. A stop that arrives later replaces that. Runs still open are the live use the /demo page shows, and an open /demo page updates as each event arrives. The cron deletes runs after 400 days.

Reporting never holds the demo up: without the binding nothing is sent, and a refused event, a failure or an answer slower than two seconds is logged and dropped. Events are parsed strictly (parseDemoRunEvent in demo-runs.ts, fuzzed in demo-runs.fuzz.test.ts).

Report ledger

report_issue records which issue each report is linked to, and how (source):

source Meaning
filed Listed in the issue body. The sync replaces these rows from GitHub
signature Linked at ingest (see Known failures skip the model)
triage The model attached it to an open issue (see Attaching to open issues)
manual A member attached it on the dashboard
excluded A member detached it from that issue. The sync, signature matching and triage leave the pair alone from then on

Every count on the pages reads this ledger, and a report counts once, under the issue that tracks it. A report two issues list is a conflict: the older issue keeps it, the report page says so, and the list shows "+1" after the issue chip.

Corrections

Members fix the ledger from the dashboard. The pages are plain forms that post to /issues/link from the same origin. Every change is written to the access log under the member's login, and open pages refresh.

The issue page lists:

  • the reports the issue body filed;
  • the ones linked later (by signature, triage or a member, each saying how and by whom);
  • the ones detached from it;
  • suggestions: indexed reports that share a signature with the issue's reports and are linked to no issue.

The actions are:

  • Detach writes excluded for the pair. A filed report stays detached when the sync next reads the body.
  • Attach writes manual, replacing an earlier exclusion since the member now asks for it. It also excludes the report from every other issue linking it, so the member's choice wins over the oldest-issue rule above. The other issues' bodies still list the report, but their summary comments stop counting it, and Attach again there reverses it.
  • Move is an attach to the other issue, which takes the report from every issue the same way, plus a detach from this one, in one batch.
  • Merge into attaches every report linked to the issue to the chosen open issue, takes them all from every other issue, and detaches them all from the first. The member closes the emptied issue on GitHub as a duplicate; the Worker doesn't.
  • Dismiss on a suggestion writes the exclusion, so ingest doesn't link it by signature later.

The report page offers Detach under each issue that lists it (how a conflict is resolved), and an Attach to issue picker when none does. The picker is an issue-number field with suggestions, empty until the member picks: issues that track the report's signature come first.

Triage queue and mutes

A report needs triage while no issue lists it, no member has marked it, and its signature isn't muted. The overview counts these and lists the newest; /reports?triage=needs is the whole queue.

  • Mark seen and Mark noise take a report out of the queue (on its page, or for the checked reports of a list page). Back to the queue clears the mark. Marks post to /reports/triage.
  • Mute on a report page mutes its signature, with a reason and an end: a number of days, or a version. A version mute ends when a report with that signature arrives from that version or a later one, since the failure is then back where it was meant to be fixed. Mutes post to /signatures/mute.
  • A muted signature's reports leave the queue, and GET /report?unfiled=1 passes over them as it does filed reports, so ingest files no issue for them.

Each mark and mute is written to the access log under the member's login, and open pages refresh.

Triage now

A report no issue lists yet has a Triage now button on its page, so a member need not wait for the daily ingest run:

  1. The button posts to /reports/ai-triage. The Worker reads the stored report and asks the triage model about it alone, on Workers AI and under the same daily budget as the ingest run, with the open issues it may join.
  2. The page shows what the model proposes: a new issue with its title, labels and body, or the open issue the report belongs to. When the model judges the report noise, the page says so and nothing is proposed.
  3. A member presses Open issue or Attach to #N (posting to /reports/ai-triage/file). The Worker files the issue with the App's Issues: write token and lists it under Tracked by at once, or attaches the report to the offered issue, as the ingest run would.

The model's reply is read as the ingest run's plan validator reads it, since the issue is public: only this report, a prefixed one-line title, a bounded body with no links, mentions, cross-references or raw HTML, and the report block written by the Worker. Attachments go only to the issues the model was offered.

  • A proposal lasts an hour; after that, press Triage now again. Nothing is filed until a member confirms it.
  • A report waits 10 minutes between presses, and the dashboard runs triage at most 20 times a day. Each press, filing and attachment is written to the access log.
  • A report whose stored copy was deleted has no button, since triage has nothing to read.
  • When the budget is spent, triage is not configured, or the model's answer can't be used, the page says so and the daily run still triages the report. Without Issues: write on the App's installation, Open issue says so and the proposal stays. See What the Worker holds.

Closing issues

Owners get two buttons on an open issue's page, posting to /admin/close-issue:

  • Close as not planned closes the issue on GitHub with that reason.
  • Close as fixed in takes a version, removes any other fixed-in: label (a reopened issue can carry an earlier one), adds the fixed-in:<version> label (which the sync reads as the issue's fixed-in version), then closes the issue as completed.

The index marks the issue closed at once, and the next sync reads it back from GitHub.

Sending corrections to GitHub

None of this reaches GitHub directly. The issue page counts the ledger rows written since the ingest Action last recorded the issue's summary comment (issues.summary_sent_at), and the next run renders them into that comment.

Owners get Update GitHub now next to that count, which writes that comment at once:

  1. The Worker renders the issue's summary with the same renderer as the run. dashboard/summary.ts is a port of scripts/summary_comment.py that the collector's tests hold to identical output.
  2. It edits the bot comment carrying the summary marker in place, or posts it.
  3. It records the fingerprint written, so the run doesn't write it again.

A refusal is shown on the page. A locked or deleted issue is recorded as the run would record it; anything else is left for the scheduled run.

The two writers take turns per issue. A summary handed to the run isn't written by the button until the run acknowledges it (or ten minutes pass), and an issue a press holds is left out of the run's next hand-out. So neither writes over the other's comment and then records the other's figures.

Issue sync

Every 30 minutes a cron syncs the issues:

  1. It lists issues labeled diagnostics changed since the last run.
  2. It reads the report IDs from each issue's hidden report block (or, for an issue filed before it, the ### Report IDs list).
  3. It asks GraphQL for the pull requests that close or reference each issue (the newest 50 cross-references).

A run covers up to 100 issues: changed issues first, then the rest in turn, least recently checked first, so every issue is rechecked. An issue that lost the diagnostics label is dropped, and a failed GraphQL call keeps the links already stored.

Two other things run the same sync:

  • an owner's Sync issues from GitHub now button, at once;
  • the ingest Action, which calls POST /issues/notify with REPORT_API_KEY and {"issue": <number>} after filing each issue. The Worker re-reads that issue from GitHub and links it immediately, reading again for a few seconds if GitHub doesn't have the new issue yet. Only the number is taken from the caller, and a transferred issue isn't followed.

Each sign-in and report view is written to an access log, kept 90 days and shown to owners. The overview also shows today's triage spend from TRIAGE_BUDGET.

Fixed-in version and regressions

For each closed issue with a merged pull request, the same sync records the version the fix shipped in (issue_fix, one row per change; the newest wins).

  1. It reads the repository's whole release list, prereleases included, at most five pages of a hundred. A longer list is left to fixed-in labels, because releases are listed by creation time and one published late could hide past any shorter read.
  2. It asks GitHub's compare API whether the oldest release published after the merge contains the pull request's merge commit, then the next two.
  3. The first that does is the fixed-in version, with a leading v dropped. A prerelease counts.

Until one does, the fix is unreleased, and the next sync continues with the releases after the last one compared. So a fix that only a later release carries is found over a few runs, and a new release is checked as soon as it exists.

  • At most five issues are resolved per sync, so a backlog clears over a few runs. A GitHub failure leaves the rows as they were.
  • A fixed-in:<version> label on the issue (such as fixed-in:v0.4.2: MAJOR.MINOR.PATCH, with an optional prerelease and build) sets the version by hand and stops the automatic lookup for that issue. The label's value wins over an earlier automatic row.
  • A release row is final. A later pull request that references the issue doesn't move it.

A report is a regression when it shares a signature with an issue's tracked reports and runs the fixed-in version or a later one. Versions compare in semver order. A build suffix or distribution repack of the fixed version counts as it, and a version that doesn't parse never does.

  • The issues list counts regressions per issue, and the overview counts issues with any.
  • The report page marks such a report "Regression" and names the fixed-in version.
  • While a fix is unreleased, the count is instead the reports that arrived after the merge.
  • Reports from versions older than the fix are expected, and not flagged.

Nothing reopens an issue automatically. A regression that reaches ingest gets a new issue instead (see Regression issues), and the issues list marks that issue "regression of #n" with a link to the closed one.

The installation token the Worker mints for these reads carries contents: read in addition to the issue, pull request, member and metadata reads. It still can't write.

Live updates

Open pages update on their own, without polling.

  1. Each page opens a WebSocket to /live (same origin, and a live session required). It is held by one SQLite-backed Durable Object, LiveHub (binding LIVE, created by the deploy's [[migrations]] entry), through the hibernation API, so idle pages cost nothing.
  2. When /submit indexes a report, /issues/notify links an issue, or a sync or reindex stores something, the demo reports a run event, or the cron closes a missed run, the Worker tells the hub. Syncs push only when the stored issues, links or pull requests actually changed.
  3. The hub sends each page the bare text changed. The push carries no data, and the hub never reads what pages send.
  4. The page fetches its own URL once with its session cookie, and swaps in the elements marked data-live. It fetches at most once every 3 seconds, and not while the tab is hidden.

A page that connects within 15 seconds of a change refreshes once, covering a change made between its own reads and its connect.

  • A live refresh of a report page skips the stored report. It skips the access-log entry only when that member's view of the report is already logged in the last 8 hours.
  • Sockets close at the first push after the session they were opened under expires, and at once when an owner signs everyone out. A page whose refresh is sent to sign-in closes its own. The hub keeps at most 200.
  • Without the LIVE binding, pages simply stay as loaded.

A Durable Object request is counted for each change. That stays well inside Workers Free's daily allowance, because indexing is capped (above).

Ingest cursor

The collector keeps diagnostics-ingest's place in the stored reports in its D1 database.

  1. The run pulls with GET /report?cursor=stored.
  2. Once every report in the batch is filed, it stores where the pull ended with POST /report/cursor (with the same key as /report). So a failed filing is retried by the next run.

A stored cursor only moves forward. A force_full run pulls from the start (since=0) and leaves the stored cursor where it was.

Version filter on ingest

GET /report resolves the latest stable GitHub release (releases/latest) and downloads only matching objects:

Match Example (baseline 1.2.3)
Equal 1.2.3
Newer 1.2.4, 2.0.0-rc.1
Packaging derivative 1.2.3+nix, 1.2.3-1ubuntu2
  • With no GitHub releases yet, every version is eligible. That is the current state of this repository.
  • Same-version prereleases such as 1.2.3-rc.1 are excluded once 1.2.3 is stable.

The daily ingest workflow resolves that release itself with its job token, and passes it as ?baseline_version= (empty when there is no release). So the Worker doesn't call GitHub on that path.

Note

Unauthenticated GitHub API calls from Cloudflare's shared egress addresses are rate-limited and fail with 403.

When a call to B2 or GitHub fails, the Worker answers 502 with {"error": "upstream failed"} and logs the failing stage.

  • Callers that send the report key (ingest) also get a detail naming the stage, the HTTP status and B2's short error code, for example b2 list failed: HTTP 401: unauthorized. It never includes credentials or response bodies.
  • /submit callers get no detail.

Next steps

Source: docs/server/diagnostics.md at main · View as Markdown