Repository automation

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

This page explains the automation App and the workflows around the main CI run: dependency updates, security advisories, docs publishing, the website checks and the Cloudflare Worker deploys.

The website checks also run on pull requests that change the site. The planner, its jobs and CI Gate are in Continuous integration.

Dependabot auto-merge

.github/workflows/dependabot-auto-merge.yml runs fastify/github-action-merge-dependabot when a Dependabot pull request opens or updates. It approves the pull request and turns on GitHub auto-merge (squash) for that pull request only.

GitHub then merges it, or adds it to the merge queue when one is required, once CI Gate and PR scan / osv-scan pass. Failing pull requests stay open for a human.

The repository-level Allow auto-merge setting only makes the per-pull-request "Enable auto-merge" button and API available. It never merges a pull request on its own, so human pull requests merge exactly as before unless someone opts one in.

The action must run while checks are still pending, because GitHub rejects enabling auto-merge on a pull request that is already mergeable. To retry a Dependabot pull request (for example, after adding the secrets), comment @dependabot rebase.

What Dependabot updates

Version updates (.github/dependabot.yml, weekly, with a 7-day cooldown) cover every lockfile of a shipped artifact or its build:

  • Cargo at the workspace root;
  • npm in ui/, packages/plugin-sdk, tools/diagnostics-collector, tools/demo and tools/site;
  • uv in packages/plugin-sdk-python;
  • GitHub Actions;
  • the digest-pinned base images of packaging/docker/Dockerfile.

Tool versions that several workflows, images and scripts share live in .tool-versions instead. Dependabot leaves the uv image and Node.js majors alone; the tool versions bump moves uv, cargo-nextest and Rust, and a Node.js major moves by hand. Dependabot still updates the node image's digest within the pinned major.

Packages that must share one version are grouped so they move in one pull request: capnp, capnp-rpc, capnpc and capnp-futures in Cargo, and vitest with its @vitest/* plugins in each Worker under tools/ (@vitest/coverage-v8 accepts only the vitest release it shipped with, so bumping either one alone fails npm ci).

examples/ lockfiles are left out, because they are reference plugins and never packaged. The full OSV scan still reports them.

The Python SDK has no runtime dependencies. Its uv.lock pins the dev and docs extras that the python_sdk check and generate-api-docs.sh --python install, so OSV scans them, and a runtime dependency added later is scanned from the start.

  • The install is uv sync --locked --no-install-project, which fails when the lockfile is out of date with pyproject.toml.
  • The SDK is imported from src/ rather than built, so its unlocked build backend never runs.
  • Docs publishing reads the SDK's source with the standard library and installs nothing.

After changing pyproject.toml dependencies, run this in packages/plugin-sdk-python:

uv lock

Policy knobs

The knobs live in the workflow's with: block:

  • target: any merges every update type. Tighten it to minor or patch to leave larger bumps for manual review.
  • exclude: 'crate-a,crate-b' skips auto-merge for pull requests touching those packages.

Setup

  1. In Settings → General → Pull Requests, enable Allow auto-merge.
  2. Create a GitHub App with repository permissions Contents: read and write, Pull requests: read and write and Workflows: read and write (GitHub refuses to let an App turn on auto-merge for a pull request that edits .github/workflows/), and no webhook. Install it on this repository only. Transferring the repository doesn't transfer the App: if the organization should own it, transfer the App separately.
  3. Add two Dependabot secrets (Settings → Secrets and variables → Dependabot), because Dependabot-triggered runs don't receive Actions secrets:
    • DEPENDABOT_MERGE_APP_CLIENT_ID: the App's client ID.
    • DEPENDABOT_MERGE_APP_KEY: a private key generated for the App.

The App token matters. Merges and queue entries attributed to GITHUB_TOKEN don't start new workflow runs, so push CI on main, the OSV SARIF upload and the merge_group checks would be skipped. The App can also approve pull requests without the "Allow GitHub Actions to create and approve pull requests" setting.

Before the merge queue exists

While the repository is still owned by a user, Protect Main keeps "require branches to be up to date" on. Auto-merge then waits on a Dependabot pull request that has fallen behind main until it is rebased. Either:

  • turn that option off in the Protect Main required-status-checks rule, so green Dependabot pull requests merge as soon as their checks pass. Dependabot rebases pull requests that actually conflict, and push CI on main is the safety net. This also relaxes the rule for human pull requests until the queue replaces it; or
  • leave it on, and comment @dependabot rebase on pull requests that fall behind.

Automation App

Scheduled workflows that open pull requests or file issues use the same GitHub App as Dependabot auto-merge, not GITHUB_TOKEN:

  • The organization doesn't allow GitHub Actions to create or approve pull requests, so GITHUB_TOKEN can't open them.
  • Pushes and pull requests made with GITHUB_TOKEN don't start workflow runs, so their required checks would never report. App-token pushes start CI like any other contributor's.
Workflow Writes
workerd-pin-bump.yml The deps/workerd-pin branch, pull request and auto-merge
tool-versions-bump.yml The deps/tool-versions branch, pull request and auto-merge (tool versions)
coverage-floors.yml The ci/coverage-floors branch, pull request and auto-merge (coverage)
librofm-apk-probe.yml An api-drift issue; the api-drift/librofm branch and pull request
graphicaudio-api-probe.yml An api-drift issue; the api-drift/graphicaudio branch and pull request
chirp-api-probe.yml An api-drift issue; the api-drift/chirp branch and pull request
osv-remediate.yml Reads OSV code-scanning alerts; the deps/osv-remediation branch, pull request and auto-merge
cflite-batch.yml fuzz-crash issues for oracle failures; draft security advisories for crashes (triage)
osv-advisories.yml Reads OSV code-scanning and Dependabot alerts; draft security advisories (verdicts)
docs-publish.yml Commits and release tags on the bookclerk-docs repository (docs publishing)
merge-queue-requeue.yml Auto-merge on a pull request the queue removed only because a job never got a runner (re-enqueue)
cla.yml Signature commits on the cla-signatures branch, the only writer its ruleset allows; the check's pull request comment and Bookclerk CLA check run (CLA check)
maintainer-review.yml The Maintainer review check run and the touches-ci-or-build label (maintainer review)
signed-commits.yml The Signed commits check run and the pull request comment on how to sign (signed commits)
visual-baselines.yml Pending sets and approved/* tags on the bookclerk-visual-baselines repository; the review comment and the approved lock on a pull request's branch; reads visual-reviewers membership (visual baselines)

The probes share .github/actions/api-drift-report:

  • One open issue per source, found by a hidden <!-- api-drift:<source> --> marker and updated in place, with a comment per new detection.
  • When the probe's fixer produced verified changes, a draft pull request on a fixed branch that closes the issue. A maintainer marks it ready after CI and review.

Each workflow's own GITHUB_TOKEN stays read-only. Each job mints the App token only when it has something to write, scoped to the permissions that step needs. Commits are made through the API (sign-commits), so they are signed and attributed to the App.

Setup

  1. Give the App these repository permissions, then accept the new permissions on its installation (an installation keeps its old permissions until they are accepted):

    • Contents: read and write, to push branches.
    • Pull requests: read and write, to open, update and auto-merge pull requests.
    • Workflows: read and write, only for dependabot-auto-merge.yml. GitHub refuses to let an App turn on auto-merge for a pull request that edits .github/workflows/, and Dependabot's GitHub Actions updates bump the action pins there. No other workflow requests it.
    • Checks: read and write, for the Bookclerk CLA check run (cla.yml), which GitHub lists under the App's name and the logo set in the App's Display information.
    • Commit statuses: read and write, so cla.yml can update a Bookclerk CLA commit status left on a commit from before the check became a check run.
    • Issues: read and write, to file and update drift issues and add labels.
    • Code scanning alerts: read, because osv-remediate.yml and osv-advisories.yml read the OSV alerts on main.
    • Dependabot alerts: read, for osv-advisories.yml.
    • Repository security advisories: read and write, because cflite-batch.yml and osv-advisories.yml file draft advisories and deduplicate against them.
    • Metadata: read (always granted).
    • Variables: read and write, as an organization or repository permission, or both. diagnostics-collector-deploy.yml stores the workers.dev collector URL when DOMAIN is unset, organization first and repository as the fallback. That token can't be narrowed, because the token API has no Variables permission to request, so the job mints it only for the one write.

    Note

    Grant and accept a permission before merging a workflow change that requests it. A job that asks for a permission the installation lacks fails to mint its token, so cla.yml, for one, would report no Bookclerk CLA check at all and block every pull request.

    Install the App on bookclerk-docs as well, with the same Contents permission, for docs-publish.yml.

  2. Add two Actions secrets (Settings → Secrets and variables → Actions). Scheduled runs don't receive the Dependabot copies.

    • AUTOMATION_APP_CLIENT_ID: the App's client ID.
    • AUTOMATION_APP_PRIVATE_KEY: a private key generated for the App.
  3. For osv-remediate.yml, add one more Actions secret, OSV_REMEDIATION_BUNDLE_KEY: a random passphrase (for example, from openssl rand -base64 32). It only encrypts the bump while CI tests it, so rotating it affects nothing but a run already in progress.

  4. For cflite-batch.yml, add FUZZ_CRASH_BUNDLE_KEY, generated the same way and kept by maintainers. It seals crash bundles that are read after the run. Without it, a run that finds a crash fails before uploading anything.

osv-advisories.yml seals its verdict report with FUZZ_CRASH_BUNDLE_KEY too, and its intermediate evidence with OSV_REMEDIATION_BUNDLE_KEY.

Maintainer review

Anyone can open a pull request, and a fork's pull request runs its own copy of the CI workflows, both on the pull request and in the merge queue. So a green CI Gate on an outside contribution only proves that its own workflows passed. The Maintainer review check makes sure someone with write access has read the change before it can merge:

  • A pull request passes when its author has write access to the repository, or is Dependabot or the automation App.

  • Any other author's pull request passes only once someone with write access, other than the author, approves its current head commit, and no such reviewer's latest review asks for changes. Every push needs a fresh approval.

  • For those pull requests, the check's page lists the changed files that run in CI or at build time, or have no text diff, and the pull request gets the touches-ci-or-build label. The list covers:

    • workflows and actions, scripts and Cargo configuration (.github/, scripts/, .cargo/);
    • the Dev Container, packaging, the lint tools and fuzz corpora;
    • build scripts, manifests and lockfiles anywhere (build.rs, Cargo.toml, Cargo.lock, package.json, uv.lock and the like);
    • toolchain and container files (.tool-versions, rust-toolchain.toml, Dockerfile);
    • binary files, and diffs too large to show.

    Read those files in full before approving.

  • The check belongs to the head commit, and passes while any open pull request on that commit passes: a maintainer who opened or approved one of them vouched for exactly that commit, so a second pull request on it can't block it.

  • Merge groups pass: every pull request in one already passed the check to enter the queue.

The check needs the approval to be on the head commit, so it never trusts an approval given before a later push. When it doesn't update after you approve, press Re-run on the check.

maintainer-review-trigger.yml runs on pull_request and pull_request_review with no token permissions and no secrets. maintainer-review.yml then runs from the default branch on workflow_run, reads the pull request, its reviews and its changed files through the API (scripts/maintainer_review.py), and posts the check as the automation App. It never checks out or runs pull request code. A first-time contributor's trigger run waits for a maintainer to approve workflow runs, and until then no check arrives and the pull request stays blocked.

Setup

  1. The App's Checks and Pull requests permissions (see Automation App) cover the check run and the label.
  2. Once maintainer-review.yml is on main, add Maintainer review to the required status checks of the default branch's ruleset, expected from the automation App, like Bookclerk CLA.
  3. Open pull requests whose head commit was pushed before the check existed have no Maintainer review check, so they wait for it forever and there is nothing to Re-run. Push to them, or submit any review (a comment review is enough): the review starts the trigger workflow without re-running CI.

Signed commits

The default branch's ruleset accepts only commits that GitHub shows as Verified. A pull request with an unsigned commit still passes CI and can be approved, but GitHub never lets it into the merge queue, and nothing on the pull request says why. The Signed commits check says why:

  • It fails while any of the pull request's commits isn't verified, whoever wrote it. The ruleset exempts nobody, so neither does the check. It judges only pull requests into the default branch, the one the ruleset protects.
  • It keeps one comment on such a pull request, posted as the automation App and edited in place on every push. The comment lists each unverified commit with GitHub's reason (not signed, a key that isn't a signing key, an unverified email, and so on), then the steps to set up SSH signing, re-sign the commits without changing them (a rebase from where the branch left the default branch), and force-push.
  • Once every commit is verified, the check passes and the comment says so. A pull request whose commits were always verified gets no comment.
  • The check belongs to the head commit and passes while any open pull request on that commit passes; the ruleset still judges each pull request's own commits. Merge groups pass.

When it doesn't update after GitHub finishes verifying a signature, press Re-run on the check.

signed-commits.yml runs from the default branch on workflow_run when the unprivileged Maintainer review trigger workflow completes for a pull request event (opened, pushed to, reopened), so one trigger job serves both checks. It reads the pull request's commits and comments through the API (scripts/signed_commits.py) and never checks out or runs pull request code. Like the maintainer review check, it waits for a maintainer to approve a first-time contributor's workflow runs.

Setup

  1. The App's Checks and Pull requests permissions (see Automation App) cover the check run and the comment.
  2. Optionally, once signed-commits.yml is on main, add Signed commits to the required status checks of the default branch's ruleset, expected from the automation App. The ruleset's Require signed commits rule already blocks the merge, so requiring the check only makes the reason show among the failed checks.

OSV remediation

osv-remediate.yml runs after every successful full OSV scan of main (push, schedule or manual run). It also runs on demand from main, without a preceding scan. It reads the open OSV-Scanner alerts in Security → Code scanning and looks each advisory up on api.osv.dev, with its aliases.

A request to api.osv.dev or the GitHub API that times out, drops its connection or gets a 408, 429 or 5xx answer is tried up to four times, 2, 4 and 8 seconds apart, or longer when a Retry-After header asks for up to a minute. A request that still fails, and any other error, stops the run with an error message.

When OSV doesn't know an alert's rule id, the other advisory ids in the alert's rule and message text stand in for it. That happens, for example, with a new CVE that OSV-Scanner chose to name a group of advisories.

scripts/osv-remediate.py then tries lockfile-only updates until no locked version of the package is inside the advisory's affected ranges:

  • Cargo: cargo update --package <name>@<locked> --precise <fix>, then the newest compatible release.
  • npm: npm update <name> --package-lock-only --ignore-scripts.
  • uv: uv lock --no-build --upgrade-package <name>==<fix>, then the newest allowed release.

<name> is the package, <locked> its locked version and <fix> the smallest fixed version above it.

An attempt that doesn't clear the advisory is rolled back. Manifests are never edited, so a fix that needs a new major version, or a change upstream, stays an open alert for a maintainer.

A bump is published only once it is known to pass:

  1. bump commits the bump locally and seals it, with the pull request body, using OSV_REMEDIATION_BUNDLE_KEY (scripts/osv-remediation-bundle.sh). Anyone can download the artifacts of a public repository, so the bump never travels in the clear.
  2. verify calls ci.yml with private-head: true. Every CI job opens the sealed bundle right after checkout (.github/actions/osv-remediation-head) and tests the bump with the full suite. As in the merge queue, caches aren't saved, cargo output is quiet, and the first failed job cancels the run.
  3. publish runs only when that CI Gate passed. It pushes deps/osv-remediation and opens the pull request ready for review, skipping the draft stage because the suite has already passed. It turns on auto-merge, so the merge queue lands it as soon as the pull request's required checks pass.

A bump that fails verification publishes nothing. The run fails, and the alerts stay in Security → Code scanning for a maintainer. A run with nothing to bump publishes nothing either.

While a remediation pull request is open, later runs stop at the pending job and change nothing. Rebuilding deps/osv-remediation on every push to main would restart the pull request's checks and drop it from the merge queue, and the queue already tests it against the latest main.

  • Once it merges or is closed, the next full scan of main starts a fresh run.
  • To replace an open remediation pull request that failed its checks, close it and run the workflow manually.

Scan results aren't public, so the run keeps them private:

  • Its log prints counts only. Command output is discarded, because a failed attempt would name a vulnerable package.
  • The pull request lists only the bumps it applied, with their advisory ids.

That hides which advisories the project is acting on until the fix is ready to land. It can't hide the advisories themselves: main's lockfiles are public, so anyone can scan them.

Dependabot doesn't cover this. It raises alerts and security updates only for advisories GitHub has reviewed, and RustSec advisories often stay unreviewed. Its version updates wait out the cooldown and never touch transitive-only crates.

Note

Pull requests the App opens aren't authored by a person. Rulesets that require approval for such changes still need a maintainer's approval before the merge queue takes them. Auto-merge stays on while it waits.

OSV advisories

osv-advisories.yml runs after every successful full OSV scan of main, next to the remediation, and on demand from main. It answers a different question: does an open dependency alert affect something Bookclerk ships?

A draft repository security advisory is filed only when it does (see the ADR). Drafts are private to maintainers, and are filed before the first release too, which proves the path. Nothing becomes public until a maintainer publishes.

Verdicts

shipped-artifacts.toml declares the shipped artifacts, and which lockfile entries end up in each. Every lockfile in the repository is either listed there or named as not shipped; scripts/tests/test_osv_advisories.py fails on an unclassified one.

scripts/osv-advisories.py takes each open OSV-Scanner alert and Dependabot alert on main, merged per lockfile when their ids or aliases overlap. It gives every artifact of that lockfile a verdict:

Lockfile Artifact
Cargo.lock Host binaries, platform and optional guests
Cargo.lock The Rust plugin SDK
ui/package-lock.json The web UI, served by bookclerkd
packages/plugin-sdk/package-lock.json @bookclerk/plugin-sdk
tools/diagnostics-collector/package-lock.json The diagnostics collector Worker
tools/demo/package-lock.json The hosted demo
packages/plugin-sdk-python/uv.lock bookclerk-plugin-sdk (PyPI)
Anything else None

An artifact is affected when:

  • Host binaries and guests: the crate is in the shipped closure of the shipped crates in cargo metadata. When the upstream advisory names vulnerable functions, one of them must also be in the DWARF of the fat-LTO release build of the shipped binaries (the method of OSV-Scanner's Rust call analysis).
  • The Rust plugin SDK: the crate is in the SDK's shipped closure. It is a library, so there is no binary to analyze.
  • The web UI: a production dependency has code in the production bundle (vite build --sourcemap, then the source maps). If the bundle can't be built, any production dependency counts.
  • @bookclerk/plugin-sdk: a production dependency (not dev in the lockfile).
  • The diagnostics collector: a production dependency. It has none today.
  • The Python SDK: never by a dependency, because the SDK has no runtime dependencies. Both the test and the workflow fail if [project].dependencies or a non-dev extra gains one before its closure is resolved.
  • Anything else: never. Examples, fuzzing, scripts and dev dependencies stay an alert plus the remediation bump.

The shipped Rust closure follows normal edges on every target platform. It excludes dev and build dependencies and proc macros, except *-src build dependencies, which compile vendored C into the binary. Their own dependencies, such as cc, are build tooling and are excluded.

A test keeps the UI from importing package files as ?url or ?raw assets, or as wasm, which the source maps wouldn't show.

Call analysis

The call-analysis build takes hours. So it runs only when an advisory with function data for a shipped crate has no advisory yet, or only an open draft (which the analysis then refreshes). Even then, it runs only on the weekly scheduled scan or a manual run; the call-analysis input forces it.

  • On other runs, such an advisory waits as "awaiting call analysis".
  • Runs queue one at a time, and a newer run replaces one still waiting. So a scheduled run queued behind another can give way to a push's run. Its call analysis then waits a week, or a manual run with call-analysis does it now.
  • An open draft whose advisory the analysis (or the web bundle) later shows unaffected is left as it is. It is reported as "open draft, now not affected" for a maintainer to close.

The web UI bundle is built only when an open advisory is on ui/package-lock.json.

A build that fails, reports no executables, or leaves a binary with too few functions or no main in its DWARF fails closed: the advisory is filed on the closure alone. An advisory a maintainer judges unaffected is closed as a draft (it isn't filed again) or ignored in osv-scanner.toml.

What a draft carries

  • Upstream ids, each as a hidden <!-- osv-advisory: <id> <ecosystem>/<package> --> marker, so it is never filed twice and a draft a maintainer closed isn't filed again.
  • The locked and first fixed versions.
  • Severity: a CVSS 3.1 vector from the GHSA record or the RustSec entry, else the highest GHSA severity. Failing both, low for an informational notice such as "unmaintained", and medium otherwise.
  • CWE ids: CWE-1395 for every draft, the GHSA records' CWE ids, and CWEs mapped from RustSec categories and informational kinds, which RustSec records carry instead.
  • The verdict and its evidence, saying so when reachability is unproven.
  • A proposed range up to the highest release tag, for a maintainer to confirm. For the TypeScript and Python SDKs, it runs up to the version in their manifest.

The affected package's ecosystem is the one shipped-artifacts.toml declares (other for packages on no public registry). The exception is an artifact with a crate_manifest, which is filed as rust once that crate's publish allows crates.io. Markers keep the declared ecosystem.

Later runs

A later run that finds the same advisory never files a second draft. While the first is still a draft, the run:

  • rewrites only the automated part of its description, between the <!-- bookclerk-auto: begin … --> and <!-- bookclerk-auto: end --> comments (new evidence, fixed versions, aliases, the newest release);
  • adds markers for new aliases;
  • fills an affected range that is still unset, once a release exists.

Text a maintainer adds outside that part is left alone, and so are the severity, CWE ids, summary and any range a maintainer set. Nothing is rewritten when only the "Last seen" line would change. Closed and published advisories are never touched.

One run files at most ten drafts and updates at most ten. A draft GitHub refuses is reported, and fails the run without holding back the others.

Before the first release, no draft proposes a range, because nothing is published yet. The draft says so, and a maintainer either closes it once the fix is on main, or sets the range when a release ships the code.

Jobs and the verdict report

Three jobs keep secrets away from third-party build code:

  1. fetch mints a read-only App token (code scanning, Dependabot and repository advisories) for one step. That step reads the alerts, their records on api.osv.dev, the newest release and the advisories on record, then revokes the token. The job then computes the closures from the lockfiles and cargo metadata (which runs no build script), and decides on call analysis. Its output leaves the job sealed with OSV_REMEDIATION_BUNDLE_KEY.
  2. analyze has no secrets and no token. It builds the web UI (npm ci --ignore-scripts) and, when planned, the shipped binaries. Its output names no advisory (the bundled packages and a truncated SHA-256 of every function in each binary), so it travels unsealed.
  3. file builds nothing. It mints an advisory-write token, files the drafts and writes the verdict report.

The verdict report records every decision:

  • one row per advisory and shipped artifact, with its verdict, packages and outcome (filed, already on record, capped, failed);
  • the advisories on lockfiles that don't ship;
  • the evidence behind each verdict.

It never leaves the job in the clear. It is sealed with FUZZ_CRASH_BUNDLE_KEY (the passphrase maintainers keep) and uploaded as the osv-advisories-report artifact of that workflow run, kept 30 days.

To read it:

  1. Download the sealed file from the run page (Actions → OSV advisories → the run → Artifacts), or with the GitHub CLI:

    gh run download <run-id> -n osv-advisories-report
    

    Replace <run-id> with the number in the run page's URL.

  2. Open it with the passphrase:

    BUNDLE_KEY=<passphrase> scripts/osv-remediation-bundle.sh open sealed.bin <dir>
    

    Replace <passphrase> with the bundle key maintainers keep, and <dir> with a new folder to open the report into. sealed.bin is the file the download writes.

  3. Read <dir>/report.md.

Logs print counts only. A new shipped artifact, or a new lockfile, goes into shipped-artifacts.toml in the pull request that adds it. Cutting the fix release automatically is a documented future goal, not built yet (see the ADR).

Docs publishing

docs-publish.yml keeps the separate public repository Bookclerk/bookclerk-docs in step with this one, as Markdown that the website and agents read.

Main and releases

  • Its main mirrors this repository's main. Every push to main builds the tree and commits it there, with a Source-Commit: <sha> trailer naming the commit it was built from. A build identical to the docs main adds no commit.
  • Bursts publish the newest commit only. Runs on main share one concurrency group that cancels the older run. A run whose commit is no longer the tip of main when it reaches the publish step skips it.
  • A published release tags the matching docs commit (the one whose trailer names the release's commit) with the same tag name. When no main commit matches because a burst skipped it, the release's build is committed beside main, parented on its head without moving it, and that commit is tagged. An existing tag is never moved.
  • A release's commit must be on main. The build refuses a release whose tag names a commit the default branch doesn't contain (see release tags).

The tree

The tree is docs/ as it is, with the generated references written into the shelves their readers use. Only markdownlint's directive comments (<!-- markdownlint-disable-line … -->, <!-- markdownlint-disable-next-line … -->, <!-- markdownlint-disable … --> and the rest) are left out of its pages' prose, since they matter to the lint check alone; code spans and fenced code keep them.

Each reference has a folder, and generated.json at the root of the tree lists them: the folder, its kind, its title, what it is generated from, whether its landing page is hand-written, and every page it wrote. The site reads the manifest rather than guessing from paths, so a reference can move without a change to the Worker.

Folder Reference Generated from Landing page
plugins/sdk/typescript/ TypeScript plugin SDK packages/plugin-sdk hand-written guide
plugins/sdk/python/ Python plugin SDK packages/plugin-sdk-python hand-written guide
plugins/sdk/rust/ Rust plugin SDK crates/bookclerk-plugin-sdk hand-written guide
server/api/ Web API crates/bookclerkd/openapi.json generated index
internals/crates/ Rust crates every workspace library generated index
contributing/ui/ Web UI library ui/src/lib hand-written guide

A reference with a hand-written landing page keeps that page's guide and appends the generated part under its Reference heading: the list of module pages, then the package or crate root. The module pages sit beside it (plugins/sdk/python/workerd.md, plugins/sdk/rust/net.md); a module with children is a folder with its own README.md. The build fails when the landing page is missing, or when a generated page would land on a hand-written one.

  • TypeScript: TypeDoc's Markdown theme for @bookclerk/plugin-sdk and ui/src/lib (typedoc-plugin-markdown, a dev dependency of both). One page per module, without source links or breadcrumbs. The SDK's root module is named plugin-sdk (@module in src/index.ts) rather than TypeDoc's default index; the build folds it into the landing page and writes the other modules (abi, workerd, sparse-workerd, …) as sibling pages, so no single page carries the whole SDK.
  • Python: scripts/python_api_markdown.py reads the SDK with ast (it imports nothing), and renders each public module's __all__, with Google-style sections as lists and cross-references as links. Packages are folders, modules are pages.
  • Rust: scripts/rustdoc_markdown.py renders rustdoc JSON for every workspace library from a pinned nightly, one page per public module. Items re-exported from private modules are documented where they are re-exported. Signatures are rebuilt from the JSON, doc headings sit under their item, and intra-doc links come from rustdoc's own link table. bookclerk-plugin-sdk lands in the Rust SDK guide's folder; every other crate is internals/crates/<crate>/README.md with its modules beside it, and internals/crates/README.md groups them (plugin interface, plugins, host internals, development).
  • Web API: scripts/openapi_markdown.py renders bookclerkd's committed OpenAPI document (crates/bookclerkd/openapi.json) into server/api/. It writes an index with a quick-start request, the credentials and every route, one page per route group (with each route's access rule, parameters, body, responses, an example request as a curl command, a PowerShell Invoke-RestMethod call, Python (requests), JavaScript (fetch) and Rust (reqwest), and an example response for each status: the example the route states for that status (an error status shows the body that route really returns, the message the handler writes or the one the error middleware writes for the status), otherwise one built from the schemas, or a sentence saying why there is no JSON body; the request sends only required fields, since optional ones can be alternatives that must not be sent together), and a page of schemas. Parameters, body fields and responses are lists, one item per property (`name` (path) *type*, **required** — description), with an inline object's fields nested under it. A request body that names a schema lists that schema's fields in place, and a response whose body is an object, or an array of objects, lists its fields too, whatever its status or media type, so a route reads without a trip to the schemas page; when a status offers several media types, the fields and the example come from the JSON one.
    • The website lays the route group pages out like an OpenAPI viewer (tools/site/api-reference.ts): a method badge beside each path, the examples in a second column when the reading column is wide enough, and the usual "On this page" rail. The rail lists the routes nested by path, relative to the start every route of the page shares (/login on the Auth page, whose headings keep the full path), with the shared start of each deeper path written once above the routes under it. Every method badge is the same width so the paths line up, and the rail is sized to fit the longest path on one line.
    • The property lists become property rows (tools/site/api-reference.ts layoutProperties): name, location, type and a required mark on one line, the description under it, a nested object's fields folded with their count, and each response's body fields folded under its status mark. A row that folds is the fold's summary, so a tap anywhere on it opens the fields; every fold is closed on load and the page's script keeps at most one response's open at a time. Any other list, and the tables of pages generated before this format, pass through unchanged.
    • The example responses become one panel per status with a colored status mark, each holding the example body (JSON, or a text body in its own language; a binary body gets a sentence) and then, after a Schema: line, the body's schema: its JSON Schema for a JSON body (component references expanded in place), or a sentence naming the media type and the shape for any other body whose schema is a string, such as "A text/plain body: a string." (a non-JSON body with another schema keeps the JSON Schema block); the page's script shows the statuses as tabs in one card with an Example/Schema switch between the two. The example request's samples become one panel per language the same way, and the page's script shows them as a language picker. Picking a language or a view picks it for every route on the page and is kept in the browser's local storage, a display preference the privacy page lists; a page restored from the back/forward cache reads the choice again.
    • Like every generated page, each one ends with a Source on GitHub link to what it is generated from: crates/bookclerkd/openapi.json here.
    • Under Web API the sidebar lists every route group and the schemas page, not their routes; those are in the rail. The index, titled Web API like its sidebar entry and breadcrumb, puts its quick start beside the introduction and shows each route group as a card of badged routes.
    • The Markdown itself stays plain, so the pages read the same on GitHub.

Every relative link to a file in the tree must resolve, and links in a generated page must stay inside the tree, or the build fails. The build also refuses a page named index.md or in a folder named index (the site serves a folder's README.md at the folder's URL and <page>/index.md is the page's Markdown copy), and a page beside a folder of the same name (make it that folder's README.md).

The Rust reference puts a module's docs at a different depth than its source file, so a doc comment in a library crate must not link a file relatively (a link target such as ../../docs/adr/x.md). Write the path in a code span or link an absolute URL instead; intra-doc links to Rust items are fine. The build checks this before it starts, and so do the plan job's test_docs_markdown.py tests on every pull request, because the build itself runs only on main.

Last updated dates

last-updated.json at the root holds the instant each page of docs/ last changed, in UTC to the second (YYYY-MM-DDTHH:MM:SSZ). The website shows it as "Last updated", in the reader's own time zone.

  • It is the author date of the newest commit that changed the page's content. A pure move (a rename git sees as unchanged) is followed back to the old path rather than counted. Author dates survive history rewrites that reset commit dates.
  • The workflow checks out the full history and passes --require-history, so a shallow checkout fails the build. A local build from a shallow clone leaves the file out, with a warning.
  • The build dates only hand-written pages. docs-publish.py adds the pages generated.json lists, since only it can compare them with the docs repository. A page whose content is unchanged there keeps its previous date, and a new or changed one gets the publish time. A generated page shows that date and names its source.
  • Files and entries published before instants were written hold a date alone (YYYY-MM-DD, UTC). The site and the next publish still read them, and the site shows such a date as it is.

Jobs and safety

  • build has no secrets and a read-only token. It runs scripts/docs-markdown.py and uploads the tree as an artifact.
  • publish checks out only scripts/docs-publish.py from the default branch. It mints the Automation App token for bookclerk-docs alone (contents: write), and writes one commit through the Git Data API, so GitHub signs it as the App. Pages over 32 KiB are uploaded as blobs first, so each create-tree call stays small. GitHub sometimes answers a create-tree call with HTTP 422 "Sorry, your request timed out" or a 5xx; the script retries those up to four times, waiting 5, 10 and 20 seconds.

A manual run publishes only from main. The publish script refuses redirects, so the token is only sent to api.github.com.

docs-markdown.py clears its output before each build. So it refuses:

  • a scratch root (--scratch-root, default target/docs-markdown/) that isn't strictly inside the repository's target/ or the system temp folder;
  • an output folder that isn't strictly inside the scratch root;
  • an existing output folder that lacks the marker file it writes;
  • the filesystem root, the home folder, the repository root, /mnt and /tmp, as either the output or the scratch root.

Build the tree locally

The tree lands in target/docs-markdown/out/:

rustup toolchain install nightly-2025-11-15 --profile minimal
python3 scripts/docs-markdown.py

The nightly is pinned because rustdoc's JSON format changes between nightlies, and rustdoc_markdown.py reads format 56.

  • Bump RUST_NIGHTLY and RUST_JSON_FORMAT in docs-markdown.py and the workflow together, and adjust the renderer for any change in the format.
  • The nightly may predate the workspace's rust-version, so the build passes --ignore-rust-version. It only produces documentation.

Website checks

site_audit (tools/site/audit/audit.ts) builds the site, serves it through the real Worker under wrangler dev, and audits it with the runner's Chrome.

The homepage's catalog sheets come from tools/site/src/catalog-records.json, which the build and the Worker bundle but never publish. Its series entries share names/universe context and contain ordered books; unrelated novels live under standalones. On each homepage request the Worker selects four distinct groups with equal probability, then one uniformly selected book per group, and writes those four sheets into the page. Visitors get no extra request, and neither the full catalog nor its source links reach the page. Shared universes, including the Cosmere, count as one group; unrelated standalones count individually. Longer series cannot dominate a stack. The build-time stack in index.html, used only if the Worker is bypassed, also uses four different groups. Metadata includes source editions, verification dates, complete narrator credits, integer runtimes in minutes and optional series position/universe. A card names a store (Audible, Libro.fm or Chirp) only when that store sells the same edition; each non-Audible store a record names needs its product page under listings, and the Audible page is sourceUrl. Tests check series coverage, metadata integrity, store listings, escaping and distinct group sampling independent of series size; the browser audit checks group diversity and every authored card for text/stamp overlap. The complete stack drops and stamps once it fits below the header. Cards remain hidden until that entrance; unsupported animation, reduced motion and very short views retain static artwork. The page never captures or repositions ordinary scrolling. After the drop, wheel input over an actual sheet optionally cycles the front record behind the stack and stamps the next record, but only once a mouse or pen has rested on the stack for 450 ms. Wheel events that arrive sooner restart that delay, so scrolling past the stack stays ordinary scrolling. A sideways touch or mouse drag does the same on release. Native vertical panning and pinch zoom remain available. Independent gestures can cycle every record; residual wheel momentum cannot start another turn. Site tests run the shipped controller with an independent bounds oracle and cover the hover delay, local wheel targeting, repeated order changes, drag displacement, taps, vertical swipes, multi-touch, lost capture, timeout, resize, navigation and reduced motion. The browser audit also exercises real wheel events and Chromium touch input, including native vertical scrolling. Cards keep a fixed aspect ratio and centered, fitted artwork areas. The Worker-chosen records remain visible without JavaScript. Page transitions and FAQ expansion use CSS with ordinary navigation and native disclosures as fallbacks.

The pre-release notice measures its full contents and collapses to the badge and detail link when it would occupy more than two lines, regardless of viewport width. It rechecks after resizing and web-font loading; mobile CSS provides the compact fallback when scripts are unavailable. Marketing pages share the docs header's sticky behavior. Their section links account for the measured header height so headings remain visible when navigation wraps.

Docs pages are read from this checkout's docs/ through a local server (tools/site/audit/serve.ts and entry.ts), not from the docs repository. So the audit covers the docs as the pull request has them, and needs no network. A docs key the local server fails renders the docs-unavailable page. The visual checks screenshot the site through the same server.

Pages Lighthouse (mobile and desktop)
/, /download, /docs/guide/getting-started, /docs/contributing/ci Performance ≥ 90, accessibility 100, best practices ≥ 95, SEO 100, CLS ≤ 0.05
404 and docs-unavailable Accessibility 100

The 404 and docs-unavailable pages' SEO and best-practice misses, such as the status code and noindex, are intended.

  • Retries. A run that misses only on performance or CLS is repeated, up to three runs, and the run with the fewest misses kept. Lab scores vary on shared runners, and Chrome sometimes records a load with no navigation start (NO_NAVSTART), more often on pages whose Cross-Origin-Opener-Policy moves the load to a new browsing context group; that run has no performance score or CLS.
  • Accessibility. axe-core then checks every page in light and dark color schemes, at 1350 and 390 px, against WCAG 2.0 to 2.2 A/AA and its best-practice rules. Any violation fails the check.
  • Results. The results table goes to the job summary. A failed audit uploads the Lighthouse reports as the site-audit artifact.
  • Chrome starts. Chrome is started once more when its first start never reaches the debugging port (busy runners). The tail of each failed start's chrome-err.log goes to the job log, and a copy (chrome-err-<attempt>.log) sits beside the reports when the audit fails.

To run it locally:

  1. In tools/site, run npm run build.
  2. In tools/site/audit, run npm ci && node audit.ts. --chrome <path> or CHROME_PATH picks the browser.

Cloudflare Worker deploys

Three workflows deploy Workers with wrangler deploy, sharing the repository secrets CLOUDFLARE_API_TOKEN and CLOUDFLARE_ACCOUNT_ID:

Workflow Worker Custom domain
site-deploy.yml bookclerk-site <DOMAIN>
diagnostics-collector-deploy.yml bookclerk-diagnostics diagnostics.<DOMAIN>
demo-deploy.yml bookclerk-demo demo.<DOMAIN>

DOMAIN is the organization variable (default bookclerk.app for the site). With DOMAIN unset, the diagnostics collector and the demo are served on the account's workers.dev subdomain instead (see Diagnostics and tools/demo/README.md).

After each deploy, a step reads the startup time Cloudflare measured from wrangler's output and writes it to the run summary, with a warning past 500 ms (Cloudflare's limit is 1000 ms). See Worker startup time.

The demo deploy

The demo image is the same one images-publish.yml publishes as ghcr.io/<owner>/bookclerk-demo, so the deploy reuses it rather than building it twice:

  1. When the published image for the deployed commit (bookclerk-demo:sha-<12 hex>) exists, the deploy checks its build provenance with gh attestation verify. The attestation must name images-publish.yml in this repository, the deployed commit, and a GitHub-hosted runner.
  2. A verified image is deployed by digest: wrangler deploy builds a one-line Dockerfile on it and pushes that to Cloudflare's container registry, which can't pull from GHCR itself. This takes a few minutes.
  3. Without one (a push that changes the demo, a fork that publishes no images), or when verification fails, the deploy builds the demo image from the deployed commit with packaging/demo/docker-bake.hcl, the build images-publish.yml runs, and deploys it the same way. This takes about 15 minutes.

It runs when the demo changes on main, when a release is published, after every Publish images run on main (the nightly publish, which is how crate and plugin changes reach the demo), and on demand.

  • A run after a publish or a release first asks the live demo's /_demo/health which commit it runs. It stops when it is the same one and an earlier run deployed that commit and passed its smoke check. Each run is named Deploy demo <commit> after the commit it deploys, since a run started by a publish has main's newest commit as its own.
  • A run after a publish deploys the commit that publish built. It stops when the live demo already runs a newer commit, so the demo never goes back, and when the publish pushed no demo image. It fails when the registry can't be read, rather than skipping the deploy.
  • The Worker reports its version before migrations and smoke checks, so a half-finished deploy is retried.
  • A repository without the two Cloudflare secrets gets a notice instead of a failure.

The daily seed refresh

The demo image carries a pre-seeded library whose store records and Discover shelves are cached for a day. A container that boots from an older template fetches them again, so its first pages are slow. The run after each nightly publish keeps the seed current without moving the demo to newer code:

  1. When the live demo already runs the published commit (the run would otherwise stop), it checks whether a run in the last 12 hours baked a seed: a push (which builds its image), a run after a publish (which deploys the image that publish just built) or another refresh. A release run doesn't count, since it can deploy an image built days earlier.
  2. If none did, it verifies that commit's published demo image as above, and stops when there is none. A refresh never builds the release binaries.
  3. It re-bakes only the seed on top of that image with packaging/demo/Dockerfile.refresh and the commit's own packaging/demo/bake-seed.sh, then deploys the result with that commit's Worker, as any other deploy.

The refresh rides on the nightly publish rather than a schedule of its own, because GitHub starts scheduled runs hours late: a separate schedule would often fire before the nightly deploy and bake a seed that deploy replaces. A manual run with the refresh input checked re-bakes the live commit's seed at once, without the 12-hour check. Its run is named Refresh demo seed, so it never counts as a deploy of a commit.

API token permissions

The API token is scoped to the Workers by name, so it needs these for each Worker the workflows deploy:

  • Workers, with the Editor role, scoped to Individual Workers and naming the Worker (bookclerk-site, bookclerk-diagnostics, bookclerk-demo). This is the current Workers permission. The legacy Workers Scripts: Edit maps to Editor on all Workers, which is broader than the deploys need.
  • Zone → Workers Routes → Edit (Write in the API reference) on the <DOMAIN> zone, for the custom domain.

bookclerk-diagnostics also needs:

  • Account → Workers KV Storage → Edit, for the deploy that first creates its TRIAGE_BUDGET namespace (the binding has no id, so wrangler creates it). Deploying bindings needs only Editor on the Worker, so this can be removed once the namespace exists.
  • Account → D1 → Edit. The first deploy creates the dashboard's DASHBOARD_DB database (no id in the binding), and every deploy applies its migrations with wrangler d1 migrations apply, so this one stays.

bookclerk-demo also needs:

  • Account → D1 → Edit, for the same reasons (its DB database).
  • Account → Containers → Edit, since every deploy pushes the image and updates the container application.
  • Account → Turnstile → Edit, since every deploy finds or creates the bookclerk-demo widget, keeps its hostnames current and reads its keys (tools/demo/scripts/worker-secrets.ts).

First deploy of a new Worker

Editor can't create a Worker; creating one takes the Admin role on the Workers product. So the first deploy of a new Worker needs one of these:

  1. Create the Worker in the Cloudflare dashboard first, under its exact name (an empty "Hello World" Worker is fine, because the first deploy replaces it). Then add it to the token's Individual Workers scope.
  2. Or temporarily give the token the Workers Admin role, run the first deploy, then remove the role again and add the new Worker to the Individual Workers scope.

See Workers roles and permissions.

After that, the two base permissions are enough, plus the per-Worker permissions that every deploy uses: D1 for bookclerk-diagnostics and bookclerk-demo, and Containers and Turnstile for the demo.

Warning

Renaming a Worker in wrangler.toml creates a new Worker. That repeats this step and leaves the old one behind, so keep the names as they are.

IndexNow

indexnow.yml tells search engines which pages of the website are new or changed, through IndexNow. Bing, Yandex, Seznam, Naver and Yep share what it receives, so they need not wait to recrawl the site.

It runs after each site-deploy and docs-publish run, daily, and on demand. A run after docs-publish first waits ten minutes, until the site's edge cache shows the published docs.

  1. It reads every URL that the live /sitemap.xml index lists, and gives each one a fingerprint: its <lastmod>, or a SHA-256 of the page as Markdown for pages without one.
  2. It compares them with the snapshot that the previous run kept as the indexnow-snapshot artifact. URLs that are new, changed or no longer listed are submitted, each at most once per UTC day. A change made after that day's submission waits for the next day's run. A dated page submitted on the day it is dated is submitted once more on a later day, since its date cannot show a second change that day.
  3. Only once IndexNow accepts the URLs does it save the new snapshot, so a failed run is retried in full.

With no snapshot (the first run, or 90 days without an accepted run) only pages whose sitemap <lastmod> is today or one of the 7 days before are submitted, and every listed URL becomes the snapshot the next run compares against. Search engines learn the older and undated pages from the sitemaps.

Note

The snapshot is an artifact rather than an Actions cache entry. CI's build caches fill the repository's cache quota and push small entries out within minutes, so a cached snapshot was often missing and every URL was submitted again.

  • Setup. Set the repository or organization variable INDEXNOW_KEY to 8 to 128 letters, digits or -, for example uuidgen | tr -d -. The key is public by design: site-deploy.yml serves it as https://<DOMAIN>/<key>.txt, and IndexNow fetches that file to check that a submission came from the site. Deploy the site once, then run the workflow by hand.
  • Forks. Without the variable the job is skipped, and the site has no key file.
  • Skipped runs. A run warns and submits nothing when the site cannot be reached, the key file is missing, a section sitemap cannot be read, or /sitemap.xml is the build's fallback (the docs dates could not be read). An undated page that cannot be fetched keeps its previous fingerprint, with a warning. A refusal from IndexNow fails the run.
  • Locally. In tools/site, run INDEXNOW_KEY=<key> INDEXNOW_DRY_RUN=1 node indexnow-run.ts in.json out.json to list the URLs that would be submitted, with no key file needed.

Note

Cloudflare's Crawler Hints also sends IndexNow signals, but only for responses that miss the zone's cache. The docs pages are rendered by the Worker and skip that cache, so Crawler Hints covers the static files only.

Next steps

Source: docs/contributing/automation.md at main · View as Markdown