Repository automation
This page explains the automation App and the workflows around the main CI run: dependency updates, security advisories, docs publishing, the website checks and the Cloudflare Worker deploys.
The website checks also run on pull requests that change the site. The
planner, its jobs and CI Gate are in Continuous integration.
.github/workflows/dependabot-auto-merge.yml runs
fastify/github-action-merge-dependabot
when a Dependabot pull request opens or updates. It approves the pull
request and turns on GitHub auto-merge (squash) for that pull request
only.
GitHub then merges it, or adds it to the merge queue when one is required,
once CI Gate and PR scan / osv-scan pass. Failing pull requests stay
open for a human.
The repository-level Allow auto-merge setting only makes the per-pull-request "Enable auto-merge" button and API available. It never merges a pull request on its own, so human pull requests merge exactly as before unless someone opts one in.
The action must run while checks are still pending, because GitHub rejects
enabling auto-merge on a pull request that is already mergeable. To retry a
Dependabot pull request (for example, after adding the secrets), comment
@dependabot rebase.
Version updates (.github/dependabot.yml, weekly, with a 7-day cooldown)
cover every lockfile of a shipped artifact or its build:
- Cargo at the workspace root;
- npm in
ui/,packages/plugin-sdk,tools/diagnostics-collector,tools/demoandtools/site; - uv in
packages/plugin-sdk-python; - GitHub Actions;
- the digest-pinned base images of
packaging/docker/Dockerfile.
Tool versions that several workflows, images and scripts share live in
.tool-versions instead. Dependabot leaves the uv
image and Node.js majors alone; the
tool versions bump moves uv, cargo-nextest and Rust, and a
Node.js major moves by hand. Dependabot still updates the node image's
digest within the pinned major.
Packages that must share one version are grouped so they move in one pull
request: capnp, capnp-rpc, capnpc and capnp-futures in Cargo, and
vitest with its @vitest/* plugins in each Worker under tools/
(@vitest/coverage-v8 accepts only the vitest release it shipped with, so
bumping either one alone fails npm ci).
examples/ lockfiles are left out, because they are reference plugins and
never packaged. The full OSV scan still reports them.
The Python SDK has no runtime dependencies. Its uv.lock pins the dev
and docs extras that the python_sdk check and
generate-api-docs.sh --python install, so OSV scans them, and a runtime
dependency added later is scanned from the start.
- The install is
uv sync --locked --no-install-project, which fails when the lockfile is out of date withpyproject.toml. - The SDK is imported from
src/rather than built, so its unlocked build backend never runs. - Docs publishing reads the SDK's source with the standard library and installs nothing.
After changing pyproject.toml dependencies, run this in
packages/plugin-sdk-python:
uv lock
The knobs live in the workflow's with: block:
target: anymerges every update type. Tighten it tominororpatchto leave larger bumps for manual review.exclude: 'crate-a,crate-b'skips auto-merge for pull requests touching those packages.
- In Settings → General → Pull Requests, enable Allow auto-merge.
- Create a GitHub App with repository permissions Contents: read and
write, Pull requests: read and write and Workflows: read and
write (GitHub refuses to let an App turn on auto-merge for a pull
request that edits
.github/workflows/), and no webhook. Install it on this repository only. Transferring the repository doesn't transfer the App: if the organization should own it, transfer the App separately. - Add two Dependabot secrets (Settings → Secrets and variables →
Dependabot), because Dependabot-triggered runs don't receive Actions
secrets:
DEPENDABOT_MERGE_APP_CLIENT_ID: the App's client ID.DEPENDABOT_MERGE_APP_KEY: a private key generated for the App.
The App token matters. Merges and queue entries attributed to
GITHUB_TOKEN don't start new workflow runs, so push CI on main, the
OSV SARIF upload and the merge_group checks would be skipped. The App can
also approve pull requests without the "Allow GitHub Actions to create and
approve pull requests" setting.
While the repository is still owned by a user, Protect Main keeps
"require branches to be up to date" on. Auto-merge then waits on a
Dependabot pull request that has fallen behind main until it is rebased.
Either:
- turn that option off in the
Protect Mainrequired-status-checks rule, so green Dependabot pull requests merge as soon as their checks pass. Dependabot rebases pull requests that actually conflict, andpushCI onmainis the safety net. This also relaxes the rule for human pull requests until the queue replaces it; or - leave it on, and comment
@dependabot rebaseon pull requests that fall behind.
Scheduled workflows that open pull requests or file issues use the same
GitHub App as Dependabot auto-merge, not
GITHUB_TOKEN:
- The organization doesn't allow GitHub Actions to create or approve pull
requests, so
GITHUB_TOKENcan't open them. - Pushes and pull requests made with
GITHUB_TOKENdon't start workflow runs, so their required checks would never report. App-token pushes start CI like any other contributor's.
| Workflow | Writes |
|---|---|
workerd-pin-bump.yml |
The deps/workerd-pin branch, pull request and auto-merge |
tool-versions-bump.yml |
The deps/tool-versions branch, pull request and auto-merge (tool versions) |
coverage-floors.yml |
The ci/coverage-floors branch, pull request and auto-merge (coverage) |
librofm-apk-probe.yml |
An api-drift issue; the api-drift/librofm branch and pull request |
graphicaudio-api-probe.yml |
An api-drift issue; the api-drift/graphicaudio branch and pull request |
chirp-api-probe.yml |
An api-drift issue; the api-drift/chirp branch and pull request |
osv-remediate.yml |
Reads OSV code-scanning alerts; the deps/osv-remediation branch, pull request and auto-merge |
cflite-batch.yml |
fuzz-crash issues for oracle failures; draft security advisories for crashes (triage) |
osv-advisories.yml |
Reads OSV code-scanning and Dependabot alerts; draft security advisories (verdicts) |
docs-publish.yml |
Commits and release tags on the bookclerk-docs repository (docs publishing) |
merge-queue-requeue.yml |
Auto-merge on a pull request the queue removed only because a job never got a runner (re-enqueue) |
cla.yml |
Signature commits on the cla-signatures branch, the only writer its ruleset allows; the check's pull request comment and Bookclerk CLA check run (CLA check) |
maintainer-review.yml |
The Maintainer review check run and the touches-ci-or-build label (maintainer review) |
signed-commits.yml |
The Signed commits check run and the pull request comment on how to sign (signed commits) |
visual-baselines.yml |
Pending sets and approved/* tags on the bookclerk-visual-baselines repository; the review comment and the approved lock on a pull request's branch; reads visual-reviewers membership (visual baselines) |
The probes share .github/actions/api-drift-report:
- One open issue per source, found by a hidden
<!-- api-drift:<source> -->marker and updated in place, with a comment per new detection. - When the probe's fixer produced verified changes, a draft pull request on a fixed branch that closes the issue. A maintainer marks it ready after CI and review.
Each workflow's own GITHUB_TOKEN stays read-only. Each job mints the App
token only when it has something to write, scoped to the permissions that
step needs. Commits are made through the API (sign-commits), so they are
signed and attributed to the App.
-
Give the App these repository permissions, then accept the new permissions on its installation (an installation keeps its old permissions until they are accepted):
- Contents: read and write, to push branches.
- Pull requests: read and write, to open, update and auto-merge pull requests.
- Workflows: read and write, only for
dependabot-auto-merge.yml. GitHub refuses to let an App turn on auto-merge for a pull request that edits.github/workflows/, and Dependabot's GitHub Actions updates bump the action pins there. No other workflow requests it. - Checks: read and write, for the
Bookclerk CLAcheck run (cla.yml), which GitHub lists under the App's name and the logo set in the App's Display information. - Commit statuses: read and write, so
cla.ymlcan update aBookclerk CLAcommit status left on a commit from before the check became a check run. - Issues: read and write, to file and update drift issues and add labels.
- Code scanning alerts: read, because
osv-remediate.ymlandosv-advisories.ymlread the OSV alerts onmain. - Dependabot alerts: read, for
osv-advisories.yml. - Repository security advisories: read and write, because
cflite-batch.ymlandosv-advisories.ymlfile draft advisories and deduplicate against them. - Metadata: read (always granted).
- Variables: read and write, as an organization or repository
permission, or both.
diagnostics-collector-deploy.ymlstores the workers.dev collector URL whenDOMAINis unset, organization first and repository as the fallback. That token can't be narrowed, because the token API has no Variables permission to request, so the job mints it only for the one write.
Note
Grant and accept a permission before merging a workflow change that requests it. A job that asks for a permission the installation lacks fails to mint its token, so
cla.yml, for one, would report no Bookclerk CLA check at all and block every pull request.Install the App on
bookclerk-docsas well, with the same Contents permission, fordocs-publish.yml. -
Add two Actions secrets (Settings → Secrets and variables → Actions). Scheduled runs don't receive the Dependabot copies.
AUTOMATION_APP_CLIENT_ID: the App's client ID.AUTOMATION_APP_PRIVATE_KEY: a private key generated for the App.
-
For
osv-remediate.yml, add one more Actions secret,OSV_REMEDIATION_BUNDLE_KEY: a random passphrase (for example, fromopenssl rand -base64 32). It only encrypts the bump while CI tests it, so rotating it affects nothing but a run already in progress. -
For
cflite-batch.yml, addFUZZ_CRASH_BUNDLE_KEY, generated the same way and kept by maintainers. It seals crash bundles that are read after the run. Without it, a run that finds a crash fails before uploading anything.
osv-advisories.yml seals its verdict report with FUZZ_CRASH_BUNDLE_KEY
too, and its intermediate evidence with OSV_REMEDIATION_BUNDLE_KEY.
Anyone can open a pull request, and a fork's pull request runs its own
copy of the CI workflows, both on the pull request and in the merge queue.
So a green CI Gate on an outside contribution only proves that its own
workflows passed. The Maintainer review check makes sure someone with
write access has read the change before it can merge:
-
A pull request passes when its author has write access to the repository, or is Dependabot or the automation App.
-
Any other author's pull request passes only once someone with write access, other than the author, approves its current head commit, and no such reviewer's latest review asks for changes. Every push needs a fresh approval.
-
For those pull requests, the check's page lists the changed files that run in CI or at build time, or have no text diff, and the pull request gets the
touches-ci-or-buildlabel. The list covers:- workflows and actions, scripts and Cargo configuration (
.github/,scripts/,.cargo/); - the Dev Container, packaging, the lint tools and fuzz corpora;
- build scripts, manifests and lockfiles anywhere (
build.rs,Cargo.toml,Cargo.lock,package.json,uv.lockand the like); - toolchain and container files (
.tool-versions,rust-toolchain.toml,Dockerfile); - binary files, and diffs too large to show.
Read those files in full before approving.
- workflows and actions, scripts and Cargo configuration (
-
The check belongs to the head commit, and passes while any open pull request on that commit passes: a maintainer who opened or approved one of them vouched for exactly that commit, so a second pull request on it can't block it.
-
Merge groups pass: every pull request in one already passed the check to enter the queue.
The check needs the approval to be on the head commit, so it never trusts an approval given before a later push. When it doesn't update after you approve, press Re-run on the check.
maintainer-review-trigger.yml runs on pull_request and
pull_request_review with no token permissions and no secrets.
maintainer-review.yml then runs from the default branch on
workflow_run, reads the pull request, its reviews and its changed files
through the API (scripts/maintainer_review.py), and posts the check as
the automation App. It never checks out or runs pull request code. A
first-time contributor's trigger run waits for a maintainer to approve
workflow runs, and until then no check arrives and the pull request stays
blocked.
- The App's Checks and Pull requests permissions (see Automation App) cover the check run and the label.
- Once
maintainer-review.ymlis onmain, addMaintainer reviewto the required status checks of the default branch's ruleset, expected from the automation App, likeBookclerk CLA. - Open pull requests whose head commit was pushed before the check
existed have no
Maintainer reviewcheck, so they wait for it forever and there is nothing to Re-run. Push to them, or submit any review (a comment review is enough): the review starts the trigger workflow without re-running CI.
The default branch's ruleset accepts only commits that GitHub shows as
Verified. A pull request with an unsigned commit still passes CI and
can be approved, but GitHub never lets it into the merge queue, and
nothing on the pull request says why. The Signed commits check says why:
- It fails while any of the pull request's commits isn't verified, whoever wrote it. The ruleset exempts nobody, so neither does the check. It judges only pull requests into the default branch, the one the ruleset protects.
- It keeps one comment on such a pull request, posted as the automation App and edited in place on every push. The comment lists each unverified commit with GitHub's reason (not signed, a key that isn't a signing key, an unverified email, and so on), then the steps to set up SSH signing, re-sign the commits without changing them (a rebase from where the branch left the default branch), and force-push.
- Once every commit is verified, the check passes and the comment says so. A pull request whose commits were always verified gets no comment.
- The check belongs to the head commit and passes while any open pull request on that commit passes; the ruleset still judges each pull request's own commits. Merge groups pass.
When it doesn't update after GitHub finishes verifying a signature, press Re-run on the check.
signed-commits.yml runs from the default branch on workflow_run when
the unprivileged Maintainer review trigger workflow completes for a pull
request event (opened, pushed to, reopened), so one trigger job serves both
checks. It reads the pull request's commits and comments through the API
(scripts/signed_commits.py) and never checks out or runs pull request
code. Like the maintainer review check, it waits for a maintainer to
approve a first-time contributor's workflow runs.
- The App's Checks and Pull requests permissions (see Automation App) cover the check run and the comment.
- Optionally, once
signed-commits.ymlis onmain, addSigned commitsto the required status checks of the default branch's ruleset, expected from the automation App. The ruleset's Require signed commits rule already blocks the merge, so requiring the check only makes the reason show among the failed checks.
osv-remediate.yml runs after every successful full OSV scan of main
(push, schedule or manual run). It also runs on demand from main,
without a preceding scan. It reads the open OSV-Scanner
alerts in Security → Code scanning and looks each advisory up on
api.osv.dev, with its aliases.
A request to api.osv.dev or the GitHub API that times out, drops its
connection or gets a 408, 429 or 5xx answer is tried up to four times,
2, 4 and 8 seconds apart, or longer when a Retry-After header asks for
up to a minute. A request that still fails, and any other error, stops
the run with an error message.
When OSV doesn't know an alert's rule id, the other advisory ids in the alert's rule and message text stand in for it. That happens, for example, with a new CVE that OSV-Scanner chose to name a group of advisories.
scripts/osv-remediate.py then tries lockfile-only updates until no locked
version of the package is inside the advisory's affected ranges:
- Cargo:
cargo update --package <name>@<locked> --precise <fix>, then the newest compatible release. - npm:
npm update <name> --package-lock-only --ignore-scripts. - uv:
uv lock --no-build --upgrade-package <name>==<fix>, then the newest allowed release.
<name> is the package, <locked> its locked version and <fix> the
smallest fixed version above it.
An attempt that doesn't clear the advisory is rolled back. Manifests are never edited, so a fix that needs a new major version, or a change upstream, stays an open alert for a maintainer.
A bump is published only once it is known to pass:
bumpcommits the bump locally and seals it, with the pull request body, usingOSV_REMEDIATION_BUNDLE_KEY(scripts/osv-remediation-bundle.sh). Anyone can download the artifacts of a public repository, so the bump never travels in the clear.verifycallsci.ymlwithprivate-head: true. Every CI job opens the sealed bundle right after checkout (.github/actions/osv-remediation-head) and tests the bump with the full suite. As in the merge queue, caches aren't saved, cargo output is quiet, and the first failed job cancels the run.publishruns only when thatCI Gatepassed. It pushesdeps/osv-remediationand opens the pull request ready for review, skipping the draft stage because the suite has already passed. It turns on auto-merge, so the merge queue lands it as soon as the pull request's required checks pass.
A bump that fails verification publishes nothing. The run fails, and the alerts stay in Security → Code scanning for a maintainer. A run with nothing to bump publishes nothing either.
While a remediation pull request is open, later runs stop at the pending
job and change nothing. Rebuilding deps/osv-remediation on every push to
main would restart the pull request's checks and drop it from the merge
queue, and the queue already tests it against the latest main.
- Once it merges or is closed, the next full scan of
mainstarts a fresh run. - To replace an open remediation pull request that failed its checks, close it and run the workflow manually.
Scan results aren't public, so the run keeps them private:
- Its log prints counts only. Command output is discarded, because a failed attempt would name a vulnerable package.
- The pull request lists only the bumps it applied, with their advisory ids.
That hides which advisories the project is acting on until the fix is
ready to land. It can't hide the advisories themselves: main's lockfiles
are public, so anyone can scan them.
Dependabot doesn't cover this. It raises alerts and security updates only for advisories GitHub has reviewed, and RustSec advisories often stay unreviewed. Its version updates wait out the cooldown and never touch transitive-only crates.
Note
Pull requests the App opens aren't authored by a person. Rulesets that require approval for such changes still need a maintainer's approval before the merge queue takes them. Auto-merge stays on while it waits.
osv-advisories.yml runs after every successful full OSV scan of main,
next to the remediation, and on demand from main. It answers a different question: does an open dependency
alert affect something Bookclerk ships?
A draft repository security advisory is filed only when it does (see the ADR). Drafts are private to maintainers, and are filed before the first release too, which proves the path. Nothing becomes public until a maintainer publishes.
shipped-artifacts.toml declares the
shipped artifacts, and which lockfile entries end up in each. Every
lockfile in the repository is either listed there or named as not shipped;
scripts/tests/test_osv_advisories.py fails on an unclassified one.
scripts/osv-advisories.py takes each open OSV-Scanner alert and
Dependabot alert on main, merged per lockfile when their ids or aliases
overlap. It gives every artifact of that lockfile a verdict:
| Lockfile | Artifact |
|---|---|
Cargo.lock |
Host binaries, platform and optional guests |
Cargo.lock |
The Rust plugin SDK |
ui/package-lock.json |
The web UI, served by bookclerkd |
packages/plugin-sdk/package-lock.json |
@bookclerk/plugin-sdk |
tools/diagnostics-collector/package-lock.json |
The diagnostics collector Worker |
tools/demo/package-lock.json |
The hosted demo |
packages/plugin-sdk-python/uv.lock |
bookclerk-plugin-sdk (PyPI) |
| Anything else | None |
An artifact is affected when:
- Host binaries and guests: the crate is in the shipped closure of the
shipped crates in
cargo metadata. When the upstream advisory names vulnerable functions, one of them must also be in the DWARF of the fat-LTO release build of the shipped binaries (the method of OSV-Scanner's Rust call analysis). - The Rust plugin SDK: the crate is in the SDK's shipped closure. It is a library, so there is no binary to analyze.
- The web UI: a production dependency has code in the production
bundle (
vite build --sourcemap, then the source maps). If the bundle can't be built, any production dependency counts. @bookclerk/plugin-sdk: a production dependency (notdevin the lockfile).- The diagnostics collector: a production dependency. It has none today.
- The Python SDK: never by a dependency, because the SDK has no runtime
dependencies. Both the test and the workflow fail if
[project].dependenciesor a non-dev extra gains one before its closure is resolved. - Anything else: never. Examples, fuzzing, scripts and dev dependencies stay an alert plus the remediation bump.
The shipped Rust closure follows normal edges on every target platform. It
excludes dev and build dependencies and proc macros, except *-src build
dependencies, which compile vendored C into the binary. Their own
dependencies, such as cc, are build tooling and are excluded.
A test keeps the UI from importing package files as ?url or ?raw
assets, or as wasm, which the source maps wouldn't show.
The call-analysis build takes hours. So it runs only when an advisory with
function data for a shipped crate has no advisory yet, or only an open
draft (which the analysis then refreshes). Even then, it runs only on the
weekly scheduled scan or a manual run; the call-analysis input forces
it.
- On other runs, such an advisory waits as "awaiting call analysis".
- Runs queue one at a time, and a newer run replaces one still waiting. So
a scheduled run queued behind another can give way to a push's run. Its
call analysis then waits a week, or a manual run with
call-analysisdoes it now. - An open draft whose advisory the analysis (or the web bundle) later shows unaffected is left as it is. It is reported as "open draft, now not affected" for a maintainer to close.
The web UI bundle is built only when an open advisory is on
ui/package-lock.json.
A build that fails, reports no executables, or leaves a binary with too
few functions or no main in its DWARF fails closed: the advisory is filed
on the closure alone. An advisory a maintainer judges unaffected is closed
as a draft (it isn't filed again) or ignored in osv-scanner.toml.
- Upstream ids, each as a hidden
<!-- osv-advisory: <id> <ecosystem>/<package> -->marker, so it is never filed twice and a draft a maintainer closed isn't filed again. - The locked and first fixed versions.
- Severity: a CVSS 3.1 vector from the GHSA record or the RustSec entry, else the highest GHSA severity. Failing both, low for an informational notice such as "unmaintained", and medium otherwise.
- CWE ids: CWE-1395 for every draft, the GHSA records' CWE ids, and CWEs mapped from RustSec categories and informational kinds, which RustSec records carry instead.
- The verdict and its evidence, saying so when reachability is unproven.
- A proposed range up to the highest release tag, for a maintainer to confirm. For the TypeScript and Python SDKs, it runs up to the version in their manifest.
The affected package's ecosystem is the one shipped-artifacts.toml
declares (other for packages on no public registry). The exception is an
artifact with a crate_manifest, which is filed as rust once that
crate's publish allows crates.io. Markers keep the declared ecosystem.
A later run that finds the same advisory never files a second draft. While the first is still a draft, the run:
- rewrites only the automated part of its description, between the
<!-- bookclerk-auto: begin … -->and<!-- bookclerk-auto: end -->comments (new evidence, fixed versions, aliases, the newest release); - adds markers for new aliases;
- fills an affected range that is still unset, once a release exists.
Text a maintainer adds outside that part is left alone, and so are the severity, CWE ids, summary and any range a maintainer set. Nothing is rewritten when only the "Last seen" line would change. Closed and published advisories are never touched.
One run files at most ten drafts and updates at most ten. A draft GitHub refuses is reported, and fails the run without holding back the others.
Before the first release, no draft proposes a range, because nothing is
published yet. The draft says so, and a maintainer either closes it once
the fix is on main, or sets the range when a release ships the code.
Three jobs keep secrets away from third-party build code:
fetchmints a read-only App token (code scanning, Dependabot and repository advisories) for one step. That step reads the alerts, their records on api.osv.dev, the newest release and the advisories on record, then revokes the token. The job then computes the closures from the lockfiles andcargo metadata(which runs no build script), and decides on call analysis. Its output leaves the job sealed withOSV_REMEDIATION_BUNDLE_KEY.analyzehas no secrets and no token. It builds the web UI (npm ci --ignore-scripts) and, when planned, the shipped binaries. Its output names no advisory (the bundled packages and a truncated SHA-256 of every function in each binary), so it travels unsealed.filebuilds nothing. It mints an advisory-write token, files the drafts and writes the verdict report.
The verdict report records every decision:
- one row per advisory and shipped artifact, with its verdict, packages and outcome (filed, already on record, capped, failed);
- the advisories on lockfiles that don't ship;
- the evidence behind each verdict.
It never leaves the job in the clear. It is sealed with
FUZZ_CRASH_BUNDLE_KEY (the passphrase maintainers keep) and uploaded as
the osv-advisories-report artifact of that workflow run, kept 30 days.
To read it:
-
Download the sealed file from the run page (Actions → OSV advisories → the run → Artifacts), or with the GitHub CLI:
gh run download <run-id> -n osv-advisories-reportReplace
<run-id>with the number in the run page's URL. -
Open it with the passphrase:
BUNDLE_KEY=<passphrase> scripts/osv-remediation-bundle.sh open sealed.bin <dir>Replace
<passphrase>with the bundle key maintainers keep, and<dir>with a new folder to open the report into.sealed.binis the file the download writes. -
Read
<dir>/report.md.
Logs print counts only. A new shipped artifact, or a new lockfile, goes
into shipped-artifacts.toml in the pull request that adds it. Cutting the
fix release automatically is a documented future goal, not built yet (see
the ADR).
docs-publish.yml keeps the separate public repository
Bookclerk/bookclerk-docs
in step with this one, as Markdown that the website and agents read.
- Its
mainmirrors this repository'smain. Every push tomainbuilds the tree and commits it there, with aSource-Commit: <sha>trailer naming the commit it was built from. A build identical to the docsmainadds no commit. - Bursts publish the newest commit only. Runs on
mainshare one concurrency group that cancels the older run. A run whose commit is no longer the tip ofmainwhen it reaches the publish step skips it. - A published release tags the matching docs commit (the one whose
trailer names the release's commit) with the same tag name. When no
maincommit matches because a burst skipped it, the release's build is committed besidemain, parented on its head without moving it, and that commit is tagged. An existing tag is never moved. - A release's commit must be on
main. The build refuses a release whose tag names a commit the default branch doesn't contain (see release tags).
The tree is docs/ as it is, with the generated references written into
the shelves their readers use. Only markdownlint's directive comments
(<!-- markdownlint-disable-line … -->,
<!-- markdownlint-disable-next-line … -->, <!-- markdownlint-disable … -->
and the rest) are left out of its pages' prose, since they matter to the
lint check alone; code spans and fenced code keep them.
Each reference has a folder, and generated.json at the root of the tree
lists them: the folder, its kind, its title, what it is generated from,
whether its landing page is hand-written, and every page it wrote. The
site reads the manifest rather than guessing from paths, so a reference
can move without a change to the Worker.
| Folder | Reference | Generated from | Landing page |
|---|---|---|---|
plugins/sdk/typescript/ |
TypeScript plugin SDK | packages/plugin-sdk |
hand-written guide |
plugins/sdk/python/ |
Python plugin SDK | packages/plugin-sdk-python |
hand-written guide |
plugins/sdk/rust/ |
Rust plugin SDK | crates/bookclerk-plugin-sdk |
hand-written guide |
server/api/ |
Web API | crates/bookclerkd/openapi.json |
generated index |
internals/crates/ |
Rust crates | every workspace library | generated index |
contributing/ui/ |
Web UI library | ui/src/lib |
hand-written guide |
A reference with a hand-written landing page keeps that page's guide and
appends the generated part under its Reference heading: the list of
module pages, then the package or crate root. The module pages sit beside
it (plugins/sdk/python/workerd.md, plugins/sdk/rust/net.md); a module
with children is a folder with its own README.md. The build fails when
the landing page is missing, or when a generated page would land on a
hand-written one.
- TypeScript: TypeDoc's Markdown theme for
@bookclerk/plugin-sdkandui/src/lib(typedoc-plugin-markdown, a dev dependency of both). One page per module, without source links or breadcrumbs. The SDK's root module is namedplugin-sdk(@moduleinsrc/index.ts) rather than TypeDoc's defaultindex; the build folds it into the landing page and writes the other modules (abi,workerd,sparse-workerd, …) as sibling pages, so no single page carries the whole SDK. - Python:
scripts/python_api_markdown.pyreads the SDK withast(it imports nothing), and renders each public module's__all__, with Google-style sections as lists and cross-references as links. Packages are folders, modules are pages. - Rust:
scripts/rustdoc_markdown.pyrenders rustdoc JSON for every workspace library from a pinned nightly, one page per public module. Items re-exported from private modules are documented where they are re-exported. Signatures are rebuilt from the JSON, doc headings sit under their item, and intra-doc links come from rustdoc's own link table.bookclerk-plugin-sdklands in the Rust SDK guide's folder; every other crate isinternals/crates/<crate>/README.mdwith its modules beside it, andinternals/crates/README.mdgroups them (plugin interface, plugins, host internals, development). - Web API:
scripts/openapi_markdown.pyrendersbookclerkd's committed OpenAPI document (crates/bookclerkd/openapi.json) intoserver/api/. It writes an index with a quick-start request, the credentials and every route, one page per route group (with each route's access rule, parameters, body, responses, an example request as acurlcommand, a PowerShellInvoke-RestMethodcall, Python (requests), JavaScript (fetch) and Rust (reqwest), and an example response for each status: the example the route states for that status (an error status shows the body that route really returns, the message the handler writes or the one the error middleware writes for the status), otherwise one built from the schemas, or a sentence saying why there is no JSON body; the request sends only required fields, since optional ones can be alternatives that must not be sent together), and a page of schemas. Parameters, body fields and responses are lists, one item per property (`name` (path) *type*, **required** — description), with an inline object's fields nested under it. A request body that names a schema lists that schema's fields in place, and a response whose body is an object, or an array of objects, lists its fields too, whatever its status or media type, so a route reads without a trip to the schemas page; when a status offers several media types, the fields and the example come from the JSON one.- The website lays the route group pages out like an OpenAPI viewer
(
tools/site/api-reference.ts): a method badge beside each path, the examples in a second column when the reading column is wide enough, and the usual "On this page" rail. The rail lists the routes nested by path, relative to the start every route of the page shares (/loginon the Auth page, whose headings keep the full path), with the shared start of each deeper path written once above the routes under it. Every method badge is the same width so the paths line up, and the rail is sized to fit the longest path on one line. - The property lists become property rows (
tools/site/api-reference.tslayoutProperties): name, location, type and a required mark on one line, the description under it, a nested object's fields folded with their count, and each response's body fields folded under its status mark. A row that folds is the fold's summary, so a tap anywhere on it opens the fields; every fold is closed on load and the page's script keeps at most one response's open at a time. Any other list, and the tables of pages generated before this format, pass through unchanged. - The example responses become one panel per status with a colored
status mark, each holding the example body (JSON, or a text body in
its own language; a binary body gets a sentence) and then, after a
Schema:line, the body's schema: its JSON Schema for a JSON body (component references expanded in place), or a sentence naming the media type and the shape for any other body whose schema is a string, such as "Atext/plainbody: a string." (a non-JSON body with another schema keeps the JSON Schema block); the page's script shows the statuses as tabs in one card with an Example/Schema switch between the two. The example request's samples become one panel per language the same way, and the page's script shows them as a language picker. Picking a language or a view picks it for every route on the page and is kept in the browser's local storage, a display preference the privacy page lists; a page restored from the back/forward cache reads the choice again. - Like every generated page, each one ends with a Source on GitHub link
to what it is generated from:
crates/bookclerkd/openapi.jsonhere. - Under Web API the sidebar lists every route group and the schemas page, not their routes; those are in the rail. The index, titled Web API like its sidebar entry and breadcrumb, puts its quick start beside the introduction and shows each route group as a card of badged routes.
- The Markdown itself stays plain, so the pages read the same on GitHub.
- The website lays the route group pages out like an OpenAPI viewer
(
Every relative link to a file in the tree must resolve, and links in a
generated page must stay inside the tree, or the build fails. The build
also refuses a page named index.md or in a folder named index (the
site serves a folder's README.md at the folder's URL and <page>/index.md
is the page's Markdown copy), and a page beside a folder of the same name
(make it that folder's README.md).
The Rust reference puts a module's docs at a different depth than its
source file, so a doc comment in a library crate must not link a file
relatively (a link target such as ../../docs/adr/x.md). Write the path in a code span or
link an absolute URL instead; intra-doc links to Rust items are fine. The
build checks this before it starts, and so do the plan job's
test_docs_markdown.py tests on every pull request, because the build
itself runs only on main.
last-updated.json at the root holds the instant each page of docs/
last changed, in UTC to the second (YYYY-MM-DDTHH:MM:SSZ). The website
shows it as "Last updated", in the reader's own time zone.
- It is the author date of the newest commit that changed the page's content. A pure move (a rename git sees as unchanged) is followed back to the old path rather than counted. Author dates survive history rewrites that reset commit dates.
- The workflow checks out the full history and passes
--require-history, so a shallow checkout fails the build. A local build from a shallow clone leaves the file out, with a warning. - The build dates only hand-written pages.
docs-publish.pyadds the pagesgenerated.jsonlists, since only it can compare them with the docs repository. A page whose content is unchanged there keeps its previous date, and a new or changed one gets the publish time. A generated page shows that date and names its source. - Files and entries published before instants were written hold a date
alone (
YYYY-MM-DD, UTC). The site and the next publish still read them, and the site shows such a date as it is.
buildhas no secrets and a read-only token. It runsscripts/docs-markdown.pyand uploads the tree as an artifact.publishchecks out onlyscripts/docs-publish.pyfrom the default branch. It mints the Automation App token forbookclerk-docsalone (contents: write), and writes one commit through the Git Data API, so GitHub signs it as the App. Pages over 32 KiB are uploaded as blobs first, so each create-tree call stays small. GitHub sometimes answers a create-tree call with HTTP 422 "Sorry, your request timed out" or a 5xx; the script retries those up to four times, waiting 5, 10 and 20 seconds.
A manual run publishes only from main. The publish script refuses
redirects, so the token is only sent to api.github.com.
docs-markdown.py clears its output before each build. So it refuses:
- a scratch root (
--scratch-root, defaulttarget/docs-markdown/) that isn't strictly inside the repository'starget/or the system temp folder; - an output folder that isn't strictly inside the scratch root;
- an existing output folder that lacks the marker file it writes;
- the filesystem root, the home folder, the repository root,
/mntand/tmp, as either the output or the scratch root.
The tree lands in target/docs-markdown/out/:
rustup toolchain install nightly-2025-11-15 --profile minimal
python3 scripts/docs-markdown.py
The nightly is pinned because rustdoc's JSON format changes between
nightlies, and rustdoc_markdown.py reads format 56.
- Bump
RUST_NIGHTLYandRUST_JSON_FORMATindocs-markdown.pyand the workflow together, and adjust the renderer for any change in the format. - The nightly may predate the workspace's
rust-version, so the build passes--ignore-rust-version. It only produces documentation.
site_audit (tools/site/audit/audit.ts) builds the site, serves it
through the real Worker under wrangler dev, and audits it with the
runner's Chrome.
The homepage's catalog sheets come from
tools/site/src/catalog-records.json, which the build and the Worker bundle
but never publish. Its series entries share names/universe context
and contain ordered books; unrelated novels live under standalones.
On each homepage request the Worker selects four distinct groups with equal
probability, then one uniformly selected book per group, and writes those
four sheets into the page. Visitors get no extra request, and neither the
full catalog nor its source links reach the page. Shared
universes, including the Cosmere, count as one group; unrelated standalones
count individually. Longer series cannot dominate a stack. The build-time
stack in index.html, used only if the Worker is bypassed, also uses four
different groups.
Metadata includes source editions, verification dates, complete narrator
credits, integer runtimes in minutes and optional series position/universe.
A card names a store (Audible, Libro.fm or Chirp) only when that store sells
the same edition; each non-Audible store a record names needs its product
page under listings, and the Audible page is sourceUrl.
Tests check series coverage, metadata integrity, store listings, escaping and distinct
group sampling independent of series size; the browser audit checks group
diversity and every authored card for text/stamp overlap.
The complete stack drops and stamps
once it fits below the header. Cards remain hidden until that entrance;
unsupported animation, reduced motion and very short views retain static
artwork. The page never captures or repositions ordinary scrolling.
After the drop, wheel input over an actual sheet optionally cycles the
front record behind the stack and stamps the next record, but only once a
mouse or pen has rested on the stack for 450 ms. Wheel events that arrive
sooner restart that delay, so scrolling past the stack stays ordinary
scrolling. A sideways touch
or mouse drag does the same on release. Native vertical panning and pinch
zoom remain available. Independent gestures can cycle every record;
residual wheel momentum cannot start another turn.
Site tests run the shipped controller with an independent bounds oracle
and cover the hover delay, local wheel targeting, repeated order changes, drag displacement,
taps, vertical swipes, multi-touch, lost capture, timeout, resize, navigation
and reduced motion. The browser audit also exercises real wheel events and
Chromium touch input, including native vertical scrolling. Cards keep a fixed
aspect ratio and centered, fitted artwork areas. The Worker-chosen records remain
visible without JavaScript. Page transitions and FAQ expansion use CSS
with ordinary navigation and native disclosures as fallbacks.
The pre-release notice measures its full contents and collapses to the badge and detail link when it would occupy more than two lines, regardless of viewport width. It rechecks after resizing and web-font loading; mobile CSS provides the compact fallback when scripts are unavailable. Marketing pages share the docs header's sticky behavior. Their section links account for the measured header height so headings remain visible when navigation wraps.
Docs pages are read from this checkout's docs/ through a local server
(tools/site/audit/serve.ts and entry.ts), not from the docs repository.
So the audit covers the docs as the pull request has them, and needs no
network. A docs key the local server fails renders the docs-unavailable
page. The visual checks
screenshot the site through the same server.
| Pages | Lighthouse (mobile and desktop) |
|---|---|
/, /download, /docs/guide/getting-started, /docs/contributing/ci |
Performance ≥ 90, accessibility 100, best practices ≥ 95, SEO 100, CLS ≤ 0.05 |
| 404 and docs-unavailable | Accessibility 100 |
The 404 and docs-unavailable pages' SEO and best-practice misses, such as
the status code and noindex, are intended.
- Retries. A run that misses only on performance or CLS is repeated, up
to three runs, and the run with the fewest misses kept. Lab scores vary on
shared runners, and Chrome sometimes records a load with no navigation
start (
NO_NAVSTART), more often on pages whoseCross-Origin-Opener-Policymoves the load to a new browsing context group; that run has no performance score or CLS. - Accessibility. axe-core then checks every page in light and dark color schemes, at 1350 and 390 px, against WCAG 2.0 to 2.2 A/AA and its best-practice rules. Any violation fails the check.
- Results. The results table goes to the job summary. A failed audit
uploads the Lighthouse reports as the
site-auditartifact. - Chrome starts. Chrome is started once more when its first start
never reaches the debugging port (busy runners). The tail of each failed
start's
chrome-err.loggoes to the job log, and a copy (chrome-err-<attempt>.log) sits beside the reports when the audit fails.
To run it locally:
- In
tools/site, runnpm run build. - In
tools/site/audit, runnpm ci && node audit.ts.--chrome <path>orCHROME_PATHpicks the browser.
Three workflows deploy Workers with wrangler deploy, sharing the
repository secrets CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID:
| Workflow | Worker | Custom domain |
|---|---|---|
site-deploy.yml |
bookclerk-site |
<DOMAIN> |
diagnostics-collector-deploy.yml |
bookclerk-diagnostics |
diagnostics.<DOMAIN> |
demo-deploy.yml |
bookclerk-demo |
demo.<DOMAIN> |
DOMAIN is the organization variable (default bookclerk.app for the
site). With DOMAIN unset, the diagnostics collector and the demo are
served on the account's workers.dev subdomain instead (see
Diagnostics and
tools/demo/README.md).
After each deploy, a step reads the startup time Cloudflare measured from wrangler's output and writes it to the run summary, with a warning past 500 ms (Cloudflare's limit is 1000 ms). See Worker startup time.
The demo image is the same one images-publish.yml publishes as
ghcr.io/<owner>/bookclerk-demo, so the deploy reuses it rather than
building it twice:
- When the published image for the deployed commit
(
bookclerk-demo:sha-<12 hex>) exists, the deploy checks its build provenance withgh attestation verify. The attestation must nameimages-publish.ymlin this repository, the deployed commit, and a GitHub-hosted runner. - A verified image is deployed by digest:
wrangler deploybuilds a one-line Dockerfile on it and pushes that to Cloudflare's container registry, which can't pull from GHCR itself. This takes a few minutes. - Without one (a push that changes the demo, a fork that publishes no
images), or when verification fails, the deploy builds the demo image
from the deployed commit with
packaging/demo/docker-bake.hcl, the buildimages-publish.ymlruns, and deploys it the same way. This takes about 15 minutes.
It runs when the demo changes on main, when a release is published, after
every Publish images run on main (the nightly publish, which is how crate
and plugin changes reach the demo), and on demand.
- A run after a publish or a release first asks the live demo's
/_demo/healthwhich commit it runs. It stops when it is the same one and an earlier run deployed that commit and passed its smoke check. Each run is namedDeploy demo <commit>after the commit it deploys, since a run started by a publish hasmain's newest commit as its own. - A run after a publish deploys the commit that publish built. It stops when the live demo already runs a newer commit, so the demo never goes back, and when the publish pushed no demo image. It fails when the registry can't be read, rather than skipping the deploy.
- The Worker reports its version before migrations and smoke checks, so a half-finished deploy is retried.
- A repository without the two Cloudflare secrets gets a notice instead of a failure.
The demo image carries a pre-seeded library whose store records and Discover shelves are cached for a day. A container that boots from an older template fetches them again, so its first pages are slow. The run after each nightly publish keeps the seed current without moving the demo to newer code:
- When the live demo already runs the published commit (the run would otherwise stop), it checks whether a run in the last 12 hours baked a seed: a push (which builds its image), a run after a publish (which deploys the image that publish just built) or another refresh. A release run doesn't count, since it can deploy an image built days earlier.
- If none did, it verifies that commit's published demo image as above, and stops when there is none. A refresh never builds the release binaries.
- It re-bakes only the seed on top of that image with
packaging/demo/Dockerfile.refreshand the commit's ownpackaging/demo/bake-seed.sh, then deploys the result with that commit's Worker, as any other deploy.
The refresh rides on the nightly publish rather than a schedule of its own,
because GitHub starts scheduled runs hours late: a separate schedule would
often fire before the nightly deploy and bake a seed that deploy replaces.
A manual run with the refresh input checked re-bakes the live commit's
seed at once, without the 12-hour check. Its run is named
Refresh demo seed, so it never counts as a deploy of a commit.
The API token is scoped to the Workers by name, so it needs these for each Worker the workflows deploy:
- Workers, with the Editor role, scoped to Individual Workers and
naming the Worker (
bookclerk-site,bookclerk-diagnostics,bookclerk-demo). This is the current Workers permission. The legacy Workers Scripts: Edit maps to Editor on all Workers, which is broader than the deploys need. - Zone → Workers Routes → Edit (Write in the API reference) on the
<DOMAIN>zone, for the custom domain.
bookclerk-diagnostics also needs:
- Account → Workers KV Storage → Edit, for the deploy that first
creates its
TRIAGE_BUDGETnamespace (the binding has no id, so wrangler creates it). Deploying bindings needs only Editor on the Worker, so this can be removed once the namespace exists. - Account → D1 → Edit. The first deploy creates the dashboard's
DASHBOARD_DBdatabase (no id in the binding), and every deploy applies its migrations withwrangler d1 migrations apply, so this one stays.
bookclerk-demo also needs:
- Account → D1 → Edit, for the same reasons (its
DBdatabase). - Account → Containers → Edit, since every deploy pushes the image and updates the container application.
- Account → Turnstile → Edit, since every deploy finds or creates the
bookclerk-demowidget, keeps its hostnames current and reads its keys (tools/demo/scripts/worker-secrets.ts).
Editor can't create a Worker; creating one takes the Admin role on the Workers product. So the first deploy of a new Worker needs one of these:
- Create the Worker in the Cloudflare dashboard first, under its exact name (an empty "Hello World" Worker is fine, because the first deploy replaces it). Then add it to the token's Individual Workers scope.
- Or temporarily give the token the Workers Admin role, run the first deploy, then remove the role again and add the new Worker to the Individual Workers scope.
See Workers roles and permissions.
After that, the two base permissions are enough, plus the per-Worker
permissions that every deploy uses: D1 for bookclerk-diagnostics and
bookclerk-demo, and Containers and Turnstile for the demo.
Warning
Renaming a Worker in wrangler.toml creates a new Worker. That repeats
this step and leaves the old one behind, so keep the names as they are.
indexnow.yml tells search engines which pages of the website are new or
changed, through IndexNow. Bing, Yandex,
Seznam, Naver and Yep share what it receives, so they need not wait to
recrawl the site.
It runs after each site-deploy and docs-publish run, daily, and on
demand. A run after docs-publish first waits ten minutes, until the site's
edge cache shows the published docs.
- It reads every URL that the live
/sitemap.xmlindex lists, and gives each one a fingerprint: its<lastmod>, or a SHA-256 of the page as Markdown for pages without one. - It compares them with the snapshot that the previous run kept as the
indexnow-snapshotartifact. URLs that are new, changed or no longer listed are submitted, each at most once per UTC day. A change made after that day's submission waits for the next day's run. A dated page submitted on the day it is dated is submitted once more on a later day, since its date cannot show a second change that day. - Only once IndexNow accepts the URLs does it save the new snapshot, so a failed run is retried in full.
With no snapshot (the first run, or 90 days without an accepted run) only
pages whose sitemap <lastmod> is today or one of the 7 days before are
submitted, and every listed URL becomes the snapshot the next run compares
against. Search engines learn the older and undated pages from the sitemaps.
Note
The snapshot is an artifact rather than an Actions cache entry. CI's build caches fill the repository's cache quota and push small entries out within minutes, so a cached snapshot was often missing and every URL was submitted again.
- Setup. Set the repository or organization variable
INDEXNOW_KEYto 8 to 128 letters, digits or-, for exampleuuidgen | tr -d -. The key is public by design:site-deploy.ymlserves it ashttps://<DOMAIN>/<key>.txt, and IndexNow fetches that file to check that a submission came from the site. Deploy the site once, then run the workflow by hand. - Forks. Without the variable the job is skipped, and the site has no key file.
- Skipped runs. A run warns and submits nothing when the site cannot
be reached, the key file is missing, a section sitemap cannot be read, or
/sitemap.xmlis the build's fallback (the docs dates could not be read). An undated page that cannot be fetched keeps its previous fingerprint, with a warning. A refusal from IndexNow fails the run. - Locally. In
tools/site, runINDEXNOW_KEY=<key> INDEXNOW_DRY_RUN=1 node indexnow-run.ts in.json out.jsonto list the URLs that would be submitted, with no key file needed.
Note
Cloudflare's Crawler Hints also sends IndexNow signals, but only for responses that miss the zone's cache. The docs pages are rendered by the Worker and skip that cache, so Crawler Hints covers the static files only.
- Continuous integration: the checks these workflows build on.
- Fuzzing: the crash triage that shares the App and the bundle keys.
- Release packaging: the workerd pin bump the App proposes.
Source: docs/contributing/automation.md at main · View as Markdown