Continuous integration

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

This page explains how Bookclerk's CI decides what to run: the execution modes, the planner, each job, and the branch protection that ties them together.

The workflow is .github/workflows/ci.yml. A dependency-aware planner lets pull requests skip unrelated work, a draft pull request runs only the fast jobs, pull requests test on Linux only, and merge_group always runs the full suite on every platform. A push to main, which only advances through the merge queue, compiles the full suite to refresh the dependency caches and runs no tests.

Fuzzing has its own page, Fuzzing. The workflows outside the pull-request checks (dependency updates, advisories, docs publishing and Worker deploys) are in Repository automation.

Selective CI

SELECTIVE_CI in .github/workflows/ci.yml selects the execution mode:

Mode When What runs
full A main push, merge_group, or any fail-closed trigger Every job and check at full scope (build only on a main push)
shadow SELECTIVE_CI != "1" Every job at full scope, with the selective prediction in the plan summary
selective SELECTIVE_CI == "1" on a pull request Only the predicted checks, each with its prerequisites

Full-suite merge_group runs audit the selective coverage. A check that fails there but was skipped on the pull request is a planner bug, and it keeps the pull request out of main.

Draft pull requests

A draft pull request gets fast feedback without holding the shared runners (the organization's plan caps how many Actions jobs run at once). Its plan resolves with --draft (draft: true in the artifact):

  • Only the check job (formatting, Clippy, docs and lint) and the test job (Rust unit and integration tests, with coverage) run, at the scope the planner predicted (DRAFT_JOBS in scripts/ci_plan/execute.py).
  • Every other check the planner selected is listed under deferred in the artifact and reads "deferred: draft pull request" in the plan summary. CI Gate expects the deferred jobs to be skipped, so it reports on the draft instead of waiting.
  • CodeQL, ClusterFuzzLite, the Docker, Dev Container and image publishing builds and ABS Integration skip their jobs on a draft too.
  • OSV scanning and the CLA check still run, so their required checks report.

Nothing is dropped, only deferred. Marking the pull request ready for review (ready_for_review) starts the full selective run of every one of these workflows, and each later push repeats it. GitHub keeps drafts out of the merge queue, and the queue always runs the full suite, so no deferred check can be skipped before a merge.

Note

On a draft, the CodeQL results merge rule waits for analyses that arrive once the pull request is ready. Coverage covers only the areas the test and check jobs measure; Worker coverage arrives with the ready run.

Platform legs in the merge queue

Every pull request run, draft or ready, also resolves with --linux-only (linux_only: true in the artifact). The Windows and macOS runs are the slowest jobs and rarely fail where Linux passes, so they wait for the merge queue:

  • The confinement, native-gateway and tray matrices take their os legs from the plan's os_matrix output: ubuntu-latest on a pull request, ubuntu-latest and windows-latest in the merge queue and on a main push (which saves the Windows cache).
  • The macOS job runs only where os_legs is true, which is never on a pull request.
  • The Windows-only checks (QUEUE_ONLY_CHECKS in scripts/ci_plan/execute.py, today windows_cluster) are listed under deferred with the reason "Windows, runs in the merge queue".
  • The release build already links thin LTO on pull requests and fat LTO in the queue (see Release build).

Every Linux job, coverage, CodeQL, OSV and fuzzing still run on the pull request, so reviewers and Codecov see their results before it is queued. CI Gate in the merge group requires every platform, so a Windows or macOS failure ejects the pull request from the queue and keeps it out of main.

Full suite on Windows and macOS

The jobs above run chosen parts of the suite on Windows and macOS, and in November 2026 host bugs on both systems reached main through the tests they leave out (#1228, #1229). The test-os job runs the whole rust_test suite there too, in the merge queue and on the build-only main push:

  • Its legs come from the plan's os_test_matrix output (OS_TEST_PARTS in scripts/ci_plan/execute.py): three parts on windows-latest and two on macos-latest, split with --partition as the Linux test parts are. Windows takes about twice Linux's time and macOS about 1.3 times, so each part takes about 30 minutes. A Linux-only pull request run has no legs; a build-only main push has one per OS, which only compiles and saves the job's own cache (see Caches).
  • The job sets CI_OS_SUITE=1, so its rust_test run uses the ci-os nextest profile (.config/nextest.toml) instead of the merge group's retry and fail-fast: every test runs, a failing one gets one retry, and a test still running after ten minutes is stopped and named, so a hang fails one test instead of holding the job until its timeout. A failing test command doesn't stop the part either: the custom harnesses and feature runs after it still run, and the job fails once all of them have reported. Setup and prerequisite failures still stop it. Running rust_test on Windows or macOS without it keeps the usual behavior.
  • CI Gate needs it. The job is listed in OS_LEG_JOBS (scripts/ci_plan/plan.py) with test as its owning job, so the gate expects it wherever test runs outside a Linux-only pull request run, and a failing leg keeps the group out of main. The merge-queue fail-fast watcher cancels the run when a leg fails, as it does for any other job. It ran report-only (REPORT_JOBS, "report only" in its name) from 20 November 2026 until each OS had three green runs in a row, on 22 November 2026; REPORT_JOBS stays for the next new leg. The curated Windows and macOS jobs above still run beside it.

Pull request lanes

GitHub gives a merge_group run no priority over pull request runs. Every job waits in one queue for the organization's 60 runners (GitHub Team), so a burst of ready pull requests can hold most of them and leave the merge queue waiting long enough to hit its status check timeout. To leave room for the queue without leaving runners idle, heavy pull request jobs wait in shared lanes:

  • A lane is a job-level concurrency group named ci-pr-lane-<n> with queue: max: one job runs in it at a time, and up to 100 wait behind it in the order they arrived. The default would keep only one waiting job and cancel the others. A job pushed out of a full lane is cancelled, which fails CI Gate visibly rather than skipping a check.
  • The lane jobs are check, test, e2e, release, confinement, native-gateway, tray, every postgres version and site. At the end of the plan job, scripts/ci_lanes.py picks a lane for each leg the run will start and hands the choice to the jobs as the lanes output. Each job's name ends in its lane (unit + integration tests · lane 3).
  • Each leg goes to the open lane with the fewest jobs running or waiting in it. The script counts them from the names of the jobs of every active pull request CI run, through the jobs API (the plan job reads it with actions: read). A base lane wins a tie, so lent lanes fill only when every base lane is busy.
  • Base lanes 0 to 29 are always open to pull requests. With CodeQL's four slots (codeql-pr-slot-<rust|other>-<even|odd>) and ClusterFuzzLite's two (cflite-pr-slot-<even|odd>), which work the same way, pull requests may always hold 36 runners, so the merge queue keeps at least 24: enough for two groups' fan-outs at build concurrency 2. Actions expressions have no modulo, so a run picks its CodeQL and ClusterFuzzLite slot from whether its run id ends in an even digit.
  • Lent lanes 30 to 53 are open only while no merge_group CI run has a gating job running or waiting (a run whose jobs are listed as "report only" no longer counts once only those are left; the script reads each active merge group's jobs to tell), and only when nothing runs or waits in them, so each normally holds one job. When the queue is idle, pull requests can use all 60 runners. When a group arrives, no new job enters a lent lane, and those lanes are back to the queue within one pull request job. There is no lock, so two plans that read the jobs API within the same few seconds, before either run's jobs exist, can both use the same empty lane; that lane then empties within two jobs. The queue itself never waits in a lane, so it uses every runner pull requests don't hold.
  • When the API can't be read, or a job's lane is missing, the jobs use the base lanes only (lane 0 for a missing one), which is the most pull requests could ever hold before lanes were lent.
  • A waiting job holds no runner. When a newer push supersedes a run, the workflow-level group cancels the old run, and its waiting jobs leave their lanes with it.
  • Any other event (merge_group, a main push, a schedule) gets a group of its own from the run id and matrix leg, so it never waits.

scripts/tests/test_ci_contract.py fails if a new job runs on pull requests without a lane or slot, if the base lanes and slots grow past 36, or if every lane and slot together pass 60. scripts/tests/test_ci_lanes.py tests the lane choice and fuzzes the API readers (see Fuzzing).

Nothing is skipped: a pull request still runs every job it ran before, only later when the lanes are busy.

The organization moved from GitHub Free (20 runners: 9 base lanes, 8 lent, 2 CodeQL slots, 1 ClusterFuzzLite slot, so the queue kept 8) to GitHub Team (60 runners) on 20 November 2026. macOS stays capped at 5 runners at once on both plans.

Lanes replaced one fixed slot per job (ci-pr-slot-<job>) in November 2026. Measured from 13:00 to 23:40 UTC on 11 November, only 5.7 of the 20 runners were busy on average: every pull request's unit + integration tests waited behind one test slot, up to 3 hours, while the other slots sat empty and runners idled for 1,686 runner-minutes with pull request jobs waiting. In a replay of that day's jobs, the mean wait for a pull request job drops from about 29 minutes to about 4.

Build-only main pushes

main only advances through the merge queue, and the queue's run already tested that exact commit. So the push run resolves with --build-only (build_only: true in the artifact):

  • Only the jobs that save caches run: check, test, native-gateway and release (BUILD_ONLY_JOBS in scripts/ci_plan/execute.py), plus the macOS job, which runs the native-gateway leg. The gate expects every other job to be skipped. The Windows native-gateway leg also compiles the Windows cluster tests, whose own job saves no cache. The release job runs a fat and a thin leg so both of its caches stay warm. Each leg also compiles the envelope seed (and runs nothing), because the seed's dependencies are built with their own profile (see Small-VPS envelope).
  • Builds, Clippy and installs run as usual. cargo test and the nextest run become cargo test … --no-run. Formatting, scripts, staging and runtime assertions are dropped.
  • Doctests and the API docs (generate-api-docs.sh) run whole. Both build their dependencies with --all-features, a feature set no other command uses, and Cargo can't only compile them. Without them in the lint cache, the check job compiled about 1,500 dependencies for them on every queue run (about 10 of its 18 minutes).

Caches are saved only on main, because merge-group refs can't share theirs. So this run is what keeps pull-request and queue runs warm.

A dependency cache key covers Cargo.lock, the manifests' dependencies, the toolchain and the CI environment, not the repository's own sources. It doesn't cover the manifests' [profile] tables, so the CI environment carries their digest (see Caches). An existing key is never saved again, so most merges leave every key unchanged and their build would save nothing.

To skip that wasted build, each build-only job first looks up its own key (lookup-only, run only when CI_BUILD_ONLY is true):

  • On a hit it sets CI_SAVED_CACHE_CURRENT=true, skips restoring the cache, and ci-exec.py run issues no commands.
  • The Linux native-gateway child always skips, because the test job saves the Linux debug cache.
  • A changed lockfile or toolchain, or an evicted cache, misses the lookup and builds as before.

A miss on Windows compiles the gateway smoke and the cluster tests from scratch, which takes about 45 minutes, so on a main push the native-gateway job's time limit is 75 minutes instead of 45.

ci-exec.py ignores the variable outside build-only runs, so it can never skip a test.

Test runner and retries

rust_test runs under cargo-nextest, installed pinned in the test job. Tests from every binary share one parallel pool, rather than cargo test running binaries one after another, and each test runs in its own process.

  • Merge groups retry once. A failed test is retried once (MERGE_GROUP_TEST_RETRIES). nextest marks a pass on retry FLAKY in its summary, and a test that fails twice fails the job. File an issue for every FLAKY line.
  • Pull requests never retry, so a flaky test still shows up red where it can be fixed.

A whole-workspace run splits into three parts (TEST_PARTS in scripts/ci_plan/execute.py), one test job leg each:

  • The plan's test_parts output lists the parts, and each leg gets CI_TEST_PART=<part>/<parts>. Every part restores the debug cache, compiles every test binary and runs its share of the tests with nextest's --partition count:<part>/<parts>. Only the first part also runs the custom harnesses, the feature runs and the bench and example builds.
  • Each part writes its coverage report, JUnit results and Web API contract files under part-<n>/ (rust/part-2/lcov.info), in an artifact of its own (coverage-rust-<n>). The coverage gate reads the parts together: a line counts as hit when any part ran it.
  • A selective run of a few packages and the build-only main push stay in one leg: their tests are too few to repay two more compiles.

On 12 November 2026 the nextest run took 21 of the job's 32 to 36 minutes, with 4,148 tests on the runner's four cores. Split three ways, each part compiles in about 4.5 minutes and runs about 7, so the job takes about 14 minutes for about 8 more runner-minutes. The slowest single test takes about 2.5 minutes, so the parts stay even.

A merge group only needs to know whether it can land, so it fails fast. nextest stops at the first test that fails its retry. The merge-queue-fail-fast job cancels the rest of the run as soon as any job fails, so CI Gate reports at once and the queue ejects the group without waiting for the slowest job. It also cancels the run of a group the queue has dropped (see Superseded merge groups). Its 310-minute timeout outlasts the merge queue's 300-minute status check timeout, so the job keeps watching for as long as the queue waits for a group.

That job runs no repository code and is the only one with actions: write. Pull requests run every selected test and job to the end (nextest's --no-fail-fast, and no cancelling job), so one round shows every failure among them.

A full-suite coverage build, plus the stale dependencies a partial cache restore brings, once ran the job out of space on runners with about 20 GiB free. So when the runner has less than 40 GiB free (CI_MIN_FREE_GIB), the job first removes the preinstalled SDKs it never uses (Android, .NET, GHC and the CodeQL bundle). Current runner images have about 90 GiB free, so it normally skips that: deleting those hundreds of thousands of files took 2 to 5 minutes of every test part.

Test targets with harness = false whose main doesn't answer libtest's --list protocol would list zero tests under nextest. So they are excluded from the nextest run and run with cargo test --test (CUSTOM_HARNESS_TESTS). scripts/tests/test_ci_contract.py fails when a new one isn't listed.

Locally, cargo test --workspace still runs everything. To reproduce ci-exec.py run rust_test, install nextest:

cargo install cargo-nextest --locked

Caches

Rust dependency caches use three shared keys, plus one for the full suite on Windows and macOS. Each is saved by one job on main and restored by the others:

  • lint: saved by check.
  • debug: saved by test on Linux (its first part), by native-gateway on Windows and by the macOS job on macOS. Restored by e2e, confinement, tray, desktop, postgres and windows-cluster.
  • release: saved by release (the job that also runs the envelope), once per LTO mode (see Release build).
  • debug-tests: saved by the first leg of test-os on Windows and on macOS (see Full suite on Windows and macOS). Every test binary of the workspace is far more than the other jobs on those systems compile, and saving it into their key would make every one of them restore it. It is keyed by CI_BUILD_PLAN_DEBUG, since it compiles a debug check.

Fewer entries keep the repository under the Actions cache limit, so jobs stop missing their cache. GitHub Team lets the repository raise that limit above the included 10 GB (Settings > Actions > General > Cache settings), billed for the storage above it; the limit is about 25 GB since 20 November 2026, so the debug-tests caches fit beside the others.

rust-cache also hashes every installed Rust toolchain into the key, and runner images preinstall their own (a Windows image update in November 2026 dropped one). So every job that caches runs tool_versions.py prune-rust right after installing the pinned toolchain. It uninstalls every other toolchain, so a job saves and looks up the same key whichever image version it lands on.

rust-cache hashes the manifests' dependencies but not their [profile] tables. A profile change would otherwise keep the old key: every job would restore dependencies built under the old profile, rebuild all of them, and never save the result, because the key already exists. The workflow's CI_CARGO_PROFILES variable holds a digest of the workspace Cargo.toml profile tables, and every cache step lists it in env-vars, so a profile change starts a new key that the next main push saves. The workflow contract test fails until the digest matches Cargo.toml and prints the new value.

The same gap opens when a change makes a build-only job compile something new without touching Cargo.lock or a manifest: a new command, a new feature set, or a profile override on one command. The build-only push would find its key saved and skip the build, so queue runs would compile the new dependencies from scratch until the next lockfile change. So each cache also carries a digest of its build-only builds: CI_BUILD_PLAN_LINT, CI_BUILD_PLAN_DEBUG and CI_BUILD_PLAN_RELEASE (build_plan_digests in scripts/ci_plan/execute.py). Each covers every OS's build-only commands, with their Cargo and Rust environment, in the jobs that save that cache, plus the contents of generate-api-docs.sh. Each cache step lists only its own, so a change re-keys only the caches whose builds changed. The contract test fails until the values match and prints the new ones. Until the next main push saves the new key, runs build that cache cold.

The digest doesn't see inside a cargo alias (cargo build-app and cargo install-platform run bookclerk-dev). A change to what those compile should also change a digest, for example through the command line the planner passes them.

The test job instruments its build for coverage and still uses the debug key: the instrumentation applies only to workspace crates, which no cache holds, so the cached dependencies match. The build-only main push that saves the key runs uninstrumented.

Build profiles

The workspace Cargo.toml builds every dependency at opt-level = 1 in the dev and test profiles, for local builds and CI alike. Workspace crates stay at opt-level = 0, so they rebuild quickly, debug fully and carry the coverage instrumentation. fdk-aac-sys is the one exception: GCC refuses libfdk-aac's forced inlines at -O1, so it stays at opt-level = 0. Measured on the bookclerk-library unit tests (4 cores), the suite ran in 432 s instead of 573 s, for a cold dependency build 67 s longer; opt-level = 2 gained only 8 s more for another 100 s of cold build. The cost is paid once per cache key, since the caches hold the built dependencies; changing a profile updates CI_CARGO_PROFILES (see Caches).

CI builds the dev and test profiles with CARGO_PROFILE_DEV_DEBUG=line-tables-only. That keeps file and line in panics while cutting link time and cache size. A failing test prints its panic backtrace (RUST_BACKTRACE=1), and RUST_LIB_BACKTRACE=0 keeps error libraries such as anyhow from capturing one for every error they build.

Plan → execution contract

git diff --no-renames stage 1: affected checks cargo metadata --no-deps scripts/ci_plan/relations.toml stage 2: prerequisites ci-exec resolve: mode ci-plan.json artifact jobs: validate, then ci-exec run CHECK CI Gate: expected execution vs results
  1. The plan job runs scripts/ci-exec.py resolve. It writes ci-plan.json (the schema version, run id, a fresh execution_id, the checked-out commit, the prediction and the resolved execution) and the job outputs that if: gates use, both from the same JSON.
  2. Every downstream job downloads that artifact and runs ci-exec.py validate first. A missing, stale or foreign artifact fails the job.
  3. Each check is one workflow step running ci-exec.py run <check>. The executor runs the check's prerequisites once per execution, in order (builds, pinned workerd, platform install, guest staging), then its commands.
  4. CI Gate compares every job's result with execution.jobs. Expected jobs must succeed and unexpected jobs must be skipped; failures, cancellations, unexpected skips and unexpected runs all fail the gate. Because the gate uses the execution, not the prediction, shadow mode's intentional full runs pass.

Local reproduction uses the same entry points:

python3 scripts/ci-plan.py --paths crates/bookclerk-cli/src/main.rs --format summary
python3 scripts/ci-exec.py resolve --selective 1 --paths crates/bookclerk-cli/src/main.rs
python3 scripts/ci-exec.py show rust_test --run-id local   # print commands
python3 scripts/ci-exec.py run rust_test --run-id local    # run them
  • Resolving again mints a new execution_id and starts a fresh execution. Prerequisite completion from the previous artifact isn't reused, even in the same temporary directory with the same commit and run id.
  • Checks that share one artifact still run each prerequisite only once.
  • show prints commands and doesn't record completion.
  • prereqs <check> runs only the check's prerequisites. With --github-env it also exports the environment the check's commands run with (PATH with the built helpers first, BOOKCLERK_WORKERD_BIN), for a job that runs the tests its own way, such as mutation testing.

scripts/tests/test_ci_contract.py (PyYAML, hash-pinned in scripts/tests/requirements.txt) fails when the workflow and the planner could diverge:

  • unknown or unused outputs;
  • a check with zero or two steps;
  • a job the gate doesn't see, or one that doesn't validate first;
  • unpinned actions, or continue-on-error.

Planner

python3 scripts/ci-plan.py --base <sha> --head <sha> --format summary
python3 scripts/tests/test_ci_plan.py        # planner + executor scenarios
python3 scripts/tests/test_ci_relations.py   # relations.toml guardrails
python3 scripts/tests/test_ci_contract.py    # workflow contract (needs PyYAML)

Stage 1: affected checks

Each changed path is classified (a rename contributes its old and new paths):

Input Effect
Package source (src/, build.rs, Cargo.toml, other package files) The package is compiled-affected (see below)
tests/**, benches/**, examples/** in a package The owner only, by target type
A dev edge to a compiled package The consumer's tests, bench and example builds and doctests
A declared embed (such as the TS or Python SDK runtime in bookclerk-workerd) The package is compiled-affected
A declared test_input The consumer's tests
A declared fixture The checks declared for that fixture
A declared check_input (a workspace executable a non-Cargo check launches, such as the bookclerk-db-testkit kit the SDK test wrappers spawn) The checks declared for it, on top of the owner's own selection
A declared test_guests executable The launching package's tests re-run, with no propagation
ui/, packages/plugin-sdk*/, ABI scripts The UI, SDK ABI and API-doc checks
A first-party workerd guest without a Cargo package (crates/bookclerk-plugins/<tier>/<name>/) Its type-check, build and tests (workerd_guests) and its staged E2E. packages/plugin-sdk/ also selects workerd_guests, because the guests type-check against it, and so does crates/bookclerk-db-testkit/, because a guest with a plugin database tests it under every profile through the kit
A reference example that declares [[databases]] and has a test script in package.json, a pyproject.toml or a Cargo.toml (examples/plugins-<name>/) Its checks and suite under every database profile (plugin_examples) and its staged E2E. crates/bookclerk-db-testkit/ and crates/bookclerk-plugin-tools/ also select plugin_examples, as do packages/plugin-sdk/ (with a TypeScript example), packages/plugin-sdk-python/ (with a Python one) and anything that affects a Rust example's tests, such as the Rust SDK
docs/** The website tests (site_test)
tools/site/**, assets/brand/** The website tests and audit (site_test, site_audit)
tools/diagnostics-collector/** The collector's tests (collector_test)
tools/demo/** and the files the demo copies The hosted demo Worker's tests (demo_test)
Markdown, TypeScript, JavaScript, JSON, YAML or CSS anywhere (not lockfiles) Formatting and Markdown lint (lint), on top of whatever else the file selects
.prettierrc.json, .prettierignore, .markdownlint-cli2.jsonc, tools/lint/** Only lint
Root Cargo.toml or Cargo.lock, the toolchain, .cargo/, workflows, the planner, fuzz/Cargo.*, unknown paths, planner errors Full suite
fuzz/corpus/<target>/**, fuzz/oracles/<target>.rs The owning crate's tests
fuzz/fuzz_targets/**, fuzz/dictionaries/** Nothing here
packaging/docker/**, .dockerignore Nothing here
.devcontainer/**, scripts/dev-env/** Full suite (unclassified)
packaging/demo/** Nothing here

Some rows need more detail:

  • Compiled-affected packages propagate over normal and build edges to their compiled consumers. Optional edges count, because there is no feature resolution.
  • Owner-only paths run --tests, cargo bench --benches --no-run or cargo build --examples, by target type.
  • Dev edges are not propagated to the consumer's production consumers.
  • Fixtures skip the owner's compile propagation.
  • docs/** runs the site tests because their link check covers the docs the site publishes.
  • The demo copies are crates/bookclerkd/openapi.json, the website's footer and fonts, and the app's mark and icons in ui/public/ (bookclerk-mark.svg, favicon.ico, favicon-32x32.png, apple-touch-icon.png). The demo's route table must cover every operation, and the copies must match.
  • Fuzz corpora and oracles are declared test_input. An undeclared corpus or oracle is unclassified.
  • "Nothing here" paths are covered elsewhere. The ClusterFuzzLite workflows and the Docker image workflow select themselves by path, and the demo image is built by the demo deploy.
  • Dev Container paths also select the Dev Container image workflow by path.

A planner error still uses the normal full plan when metadata, relations and the guest inventory are available; an unavailable diff is the usual case. If those inputs are missing, ci-exec.py resolve fails instead of publishing a plan with empty prerequisites.

Cargo.lock stays a full-suite trigger for now. Precision for dependency updates can follow.

Stage 2: prerequisites

Every selected check lists what a clean runner needs, with a reason:

  • rust_test: cargo build -p for executables its packages' tests launch (test_guests), plus pinned workerd when they spawn through it. Tests run with BOOKCLERK_REQUIRE_TEST_GUESTS=1, so a missing guest fails instead of skipping.
  • e2e: the jail, the launcher, the native guests in scope, pinned workerd, cargo install-platform --skip-build, and cargo stage-plugins --plugin <id>… (or --optional --examples for the full installation). It runs bookclerk-plugin-e2e (--lib --test staged_plugins --test installed_plugin_path --test cold_start --test workerd_isolate_egress; never native_gateway) with BOOKCLERK_REQUIRE_STAGED_PLUGINS=1. The cold_start measurement is ignored, so the check only keeps it compiling.
  • postgres rpc_like: the postgres guest, the launcher, the jail and pinned workerd.
  • native_gateway: cargo build -p bookclerk-jail -p bookclerk-workerd and pinned workerd only, with no platform install, staging, database services or app build. The fixture guest is a bin target of bookclerk-plugin-e2e, so cargo test --test native_gateway builds it.
  • windows_cluster: cargo build -p bookclerk-plugin-database-sqlite -p bookclerk-jail -p bookclerk-workerd and pinned workerd, then the existing identity filters and cargo test -p bookclerk-cli --test library_secret_startup. The CLI target still sets plugins.isolation = "required" and keeps its assertions, and BOOKCLERK_REQUIRE_TEST_GUESTS=1 is set for it.

Prerequisites never select suites. Building the sqlite guest for host tests doesn't select sqlite's own tests or E2E.

Staged E2E scope

bookclerk-plugin-e2e owns the staged-installation suite. The ordinary test step excludes it (--exclude bookclerk-plugin-e2e on the full suite).

  • Full. A direct source change in a staged_e2e.full_packages package (dev, manifest, abi, workerd, jail, sandbox, the e2e crate), an inventory manifest (crates/bookclerk-plugins/*/*/plugin.toml, examples/plugins-*/plugin.toml), host discovery, registry or install preflight, or the full suite.
  • Subsets. Each compiled-affected guest package adds its id. Changed non-Cargo guests (examples and first-party workerd guests) add theirs. The workerd SDK runtime embed adds every workerd-runtime guest. Shared host runtime changes add the runtime_smoke guests (native Rust, workerd TS, workerd Python).
  • Combination. Full wins over subsets, subsets are unioned, and prerequisites add nothing. Platform guests (sqlite, local, embedder) are always installed and checked.

An embedded SDK change marks bookclerk-workerd affected through the embed, so it selects the workerd-runtime subset, not the full installation. A change under crates/bookclerk-workerd/src/ is a direct source change and selects full.

Declarations (scripts/ci_plan/relations.toml)

This file is the only handwritten graph. It holds the relationships cargo metadata can't describe, and doesn't restate the workspace dependency graph:

  • embedded assets;
  • cross-package test inputs and fixtures;
  • executables that tests launch;
  • feature-specific test runs and postgres step owners;
  • E2E scope rules and platform-job membership;
  • release packaging inputs.

scripts/tests/test_ci_relations.py is a guardrail, not a completeness guarantee. It recognizes literal include_str! / include_bytes! paths, Rust string literals that resolve to another package or repository path, and workspace binary names in tests/. Each must be declared, related through Cargo, under a full-suite path, or listed in [[reviewed]] with a reason.

Warning

Computed paths, macro-generated includes and dynamically discovered fixtures are not detected. Declare them, or rely on the conservative fallbacks.

Postgres tests only run in the postgres check, through the name filters of POSTGRES_STEPS in scripts/ci_plan/execute.py. scripts/tests/test_ci_contract.py finds every Postgres-gated test: an #[ignore] reason naming Postgres, or a body that reads BOOKCLERK_TEST_POSTGRES_URL directly or through a helper in the same package. It fails when no step filter matches the test's function name, or when a step matches no test. Name such tests postgres_*.

Jobs

Job Runs when Checks
plan Always Planner, executor, contract and guardrail tests; DB isolation and docs lints; resolve; the ci-plan artifact
fmt / clippy / docs Any of its checks is selected lint, ui, plugin_sdk_abi, python_sdk, workerd_guests, plugin_examples, author_surface, fmt, clippy, clippy_publish, api_docs, doctest, store_free
unit + integration tests rust_test is selected nextest over the affected packages (the workspace on the full suite), custom-harness targets, feature test runs, bench and example builds
staged plugin installation (e2e) e2e is selected Builds, installs and stages the guests in scope, then the bookclerk-plugin-e2e staged targets
release build + small-VPS envelope (<lto> LTO) A shipped binary (hosts, helpers, platform guests) is compiled-affected (release), or an [envelope].paths input changed, an [envelope].packages member is compiled-affected, or the full suite (envelope) Each selected check as its own step; see Release build and Small-VPS envelope
sandbox + jailed tiers (Linux; Windows in the queue) sandbox, jail, media or media-worker is affected Clippy and enforcement tests (Windows --test-threads=1)
native-behind-workerd gateway (Linux; Windows in the queue) See the gateway job The four native_gateway targets, plus per-OS extras
tray (Linux; Windows in the queue) bookclerk-tray is affected Clippy and tests
desktop window (Linux) A [desktop].paths input changed, or the full suite Clippy on bookclerk-desktop --features window; see Desktop job
macOS (sandbox, gateway, tray) Any of the three jobs above runs, outside a pull request The macOS run of each of them, one step each; see macOS job
postgres 16/17/18 An owning package's unit tests are affected Only the owners' steps, on every supported major
windows cluster startup + identity A [windows_cluster].paths input changed, or the full suite; never on a pull request See Windows cluster
website + collector Workers See Workers site_test, site_audit, collector_test, demo_test; no Rust toolchain
UI visual and accessibility A file under ui/, tools/site/, tools/diagnostics-collector/ or assets/brand/ changed, SECURITY.md changed, or the full suite ui_visual in the pinned Playwright image; see UI visual job
CI Gate Always The stable required check (see the contract)

The postgres owners are the library, the db guest, the postgres guest and the plugin host's RPC LIKE.

Tool versions

The root .tool-versions file is the one pin for every tool version that crosses a boundary between workflows, images and scripts:

Pin Read by Bumped by
nodejs (a major) Every actions/setup-node step (node-version-file); the node images' tags Hand, with the node image tags
uv Every astral-sh/setup-uv step (version-file); the Dev Container's ghcr.io/astral-sh/uv image tool-versions-bump.yml
cargo-nextest The unit + integration tests job (through scripts/tool_versions.py github-output); scripts/dev-env/provision.sh tool-versions-bump.yml
rust rust-toolchain.toml (channel, which repeats the pin because rustup can't read .tool-versions); every dtolnay/rust-toolchain step (through a Read tool pins step before it, installing the toolchain file's rustfmt and clippy so rustup doesn't add them mid-job); scripts/dev-env/provision.sh tool-versions-bump.yml

scripts/tests/test_dev_env.py (in the plan job) fails when a workflow sets one of these versions itself, when a Dockerfile's node or uv image disagrees with the file, or when rust-toolchain.toml names another Rust release. A change to .tool-versions runs the full suite and rebuilds the Dev Container image.

scripts/tool_versions.py parses the file strictly: exactly the pins above, in that order, one <tool> <version> per line, no comments. actions/setup-node reads the first line that looks like a version, so nodejs stays first.

The daily tool-versions-bump.yml workflow takes the newest release of uv, cargo-nextest and Rust (the rust-lang/rust GitHub releases) that is at least 7 days old (the same cooldown as Dependabot and the workerd pin). For uv it also resolves the image digest and rewrites every Dockerfile that builds from the image. For Rust it also rewrites rust-toolchain.toml. It opens one pull request for every tool due that day, on the deps/tool-versions branch, with the automation App, and turns on auto-merge. Dependabot ignores the uv image and Node.js majors, so the two never race. Its ignore names the uv image astral-sh/uv, without the ghcr.io host, because that is the name Dependabot matches ignores against.

A new Rust release can bring new Clippy lints and Cargo behavior, so it arrives as that pull request, and CI shows anything it breaks there, instead of reaching every pull request and merge group the day it is released. Pinned nightlies (docs-publish.yml, ClusterFuzzLite) keep their own pins.

Note

The workerd binary keeps its own pin in crates/bookclerk-workerd/workerd-pin.json. Base image digests shared by several Dockerfiles stay with Dependabot's grouped updates; test_dev_env.py checks they match.

Formatting and Markdown lint

The lint check needs no Rust toolchain. It installs tools/lint (pinned Prettier and markdownlint-cli2) and runs both from the repository root:

  • Prettier checks TypeScript, JavaScript, JSON, YAML and CSS against .prettierrc.json. .prettierignore leaves out Markdown, HTML (page templates whose whitespace reaches the output), lockfiles, generated files, byte-identical mirrors, fuzz or test corpora, and the build output that nested .gitignore files hide.
  • markdownlint checks every Markdown file against .markdownlint-cli2.jsonc, which relaxes the rules that fight GitHub-first docs (see Writing docs).

Each language's own linter runs in its own check: oxlint in ui, the TypeScript SDK and the three Workers, and ruff check plus ruff format --check in python_sdk.

The plugin_sdk_abi and python_sdk checks build the bookclerk-db-testkit executable first (cargo build -p bookclerk-db-testkit) and name it to the SDK tests as BOOKCLERK_DB_TESTKIT: the SDKs' createTestDatabase wrappers spawn it (see Working within database limits), so both checks need the Rust toolchain.

The plugin_examples check builds the kit and the bookclerk-plugin author CLI, then runs bookclerk-plugin test <dir> on each database example, which runs the example's own suite once per profile (first-party, d1, minimum; see The suite under every profile). Before that, a TypeScript example is type-checked against the built TypeScript SDK and its modules/ rebuilt (a committed tree that differs is drift, as in workerd_guests). A Python example is checked with ruff check and ruff format --check, then tested with pytest in the Python SDK's locked environment (the one python_sdk uses) with the SDK imported from its sources. A Rust example is a workspace package that fmt and clippy already cover, so it only runs its cargo test suite.

To fix formatting locally:

$ npm ci --prefix tools/lint
$ node tools/lint/node_modules/prettier/bin/prettier.cjs --write .
$ node tools/lint/node_modules/markdownlint-cli2/markdownlint-cli2-bin.mjs --fix

Clippy and Cargo warnings

The workflow sets RUSTFLAGS="-D warnings", which turns every rustc and Clippy warning into an error. It does not reach the warnings Cargo itself emits about the manifests, such as:

  • unused_dependencies: a crate lists a dependency its code never uses.
  • unused_workspace_dependencies: an entry in [workspace.dependencies] that no member inherits.
  • unused_workspace_package_fields: a [workspace.package] field no member inherits.

The clippy check therefore also sets CARGO_BUILD_WARNINGS=deny (Cargo's build.warnings), so any Cargo warning fails it too. To see the same result locally:

$ CARGO_BUILD_WARNINGS=deny cargo clippy --workspace --all-targets -- -D warnings

The clippy check runs on Linux. Cargo builds a dependency declared under [target.'cfg(windows)'.dependencies] (or another platform's table) only for that platform, so the check cannot tell whether such an entry is used; review platform-only entries by hand. Fix a reported dependency by removing it from the manifest.

Release build

The release job runs at one of two scopes:

  • affected: cargo build --release -p <affected shipped>.
  • full: build-app --release --platform plus helper layout assertions. It runs when a packaging input changes (bookclerk-dev, the workerd pins, the platform manifests) or on the full suite.

The full scope builds the media worker with its libav feature, as release archives and the Docker image do, so it also compiles the pinned FFmpeg decoders (bookclerk-libav; about half a minute on 4 cores). The affected scope leaves the feature off. Test builds turn it on through bookclerk-media's dev-dependencies, so the test jobs cover the FFmpeg path on every OS. Windows jobs that compile bookclerk-media install MSYS2's make first (FFmpeg's configure runs under the runner's MSYS2 with MSVC; see media.md).

The release profile ships fat LTO with one codegen unit, which links on a single core. On pull requests the job sets CARGO_PROFILE_RELEASE_LTO=thin, which links in parallel and still checks that the release build compiles. The merge queue and OSV remediation runs keep fat LTO, so the build and the envelope that gate main use the binaries we ship.

Cargo recompiles dependencies when the LTO mode changes, and the cache key covers every CARGO* variable, so each mode has its own release cache. The job is a matrix over the mode: one thin leg on a pull request, one fat leg in the merge queue, and both on the build-only main push, which seeds the two caches. Each leg skips its build there when its cache is already current. On a 4-core machine a warm thin build of bookclerkd took about 10 minutes against 17 for fat; a thin build without its own cache gains nothing.

Native-behind-workerd gateway job

The job runs when any of these hold:

  • a [native_gateway].packages member (plugin-host, workerd, plugin-sdk, sandbox, jail, the e2e crate) is in the Cargo-compiled closure;
  • a [native_gateway].paths smoke input changed;
  • it is the full suite.

On every OS (on macOS, as a step of the macOS job) it runs cargo test -p bookclerk-plugin-e2e --test native_gateway --test native_gateway_isolation --test native_gateway_lifecycle --test native_gateway_authenticated_endpoint (see the smoke below).

  • Windows first adds Clippy on workerd, the SDK http feature, the host --lib, the workerd lib tests (inherited handle: SOCKET_PROXY), and the lifecycle churn test alone.
  • macOS adds the workerd lib tests.

Windows cluster startup and identity

On windows-latest only, the job runs identity create, restart and binding, and different-library rejection. Then it runs the full library_secret_startup target with the sqlite guest, the jail, bookclerk-workerd and pinned workerd.

Its job saves no cache. Its bookclerk-cli and bookclerk-library test builds unify dependency features differently from every other Windows command, so on a build-only main push the Windows native-gateway leg also compiles them (BUILD_ONLY_HOSTED_CHECKS in scripts/ci_plan/execute.py), and the debug cache it saves holds their dependencies. Before that, the job compiled about 250 dependencies, about 13 of its 17 minutes, on every queue run.

Desktop job

The desktop window's Tauri code (bookclerk-desktop's window feature) only builds with the WebKitGTK development files, so every other job builds the crate without it. This job, on ubuntu-latest, installs those files and capnproto, then runs Clippy on bookclerk-desktop --features window --all-targets. Clippy type-checks the window code without linking it.

A pull request runs it when a [desktop].paths input in scripts/ci_plan/relations.toml changes: the window crate, its request policy (bookclerk-window-policy), or the embedded host it drives (bookclerkd's embedded.rs, lib.rs and Cargo.toml). It doesn't select on every package the window links, which is most of the host. The full suite, in the merge queue and on main, always runs it, which catches a host change that breaks the window. Windows and macOS don't run it yet.

UI visual job

The web app's Storybook stories and the pages of the website and the diagnostics collector, each checked with axe-core and compared with a baseline screenshot at two viewports in light and dark. The job runs inside the Playwright image pinned by digest, so every run renders with the same browser and fonts, and needs no Rust toolchain (the image already has the Python the plan scripts need). It builds Storybook and the website, runs the checks and uploads the ui-visual artifact.

A change under ui/, tools/site/, tools/diagnostics-collector/ or assets/brand/, or to SECURITY.md, selects it; a change to the host that serves ui/dist or to docs/ doesn't. Baselines live in a separate repository, named by ui/visual/baselines.lock; the job records a set every run and compares with the locked one when there is one. See Visual and accessibility testing.

Small-VPS envelope

The check makes sure Bookclerk fits a small VPS. Its paths are the harness, the seed, the search rebuilds, the works page index and the works HTTP handlers. It runs in the release job, after the release build step when both are selected, so the hosts are compiled once per run.

  1. On the ubuntu-latest VM, install-platform --release builds what the release step did not (all of it when only the envelope is selected), and the seed is compiled. The seed is a load generator, not a shipped binary, so it's compiled without LTO and with 16 codegen units (ENVELOPE_SEED_PROFILE in scripts/ci_plan/execute.py): about 98 s instead of 173 s after a source change, measured locally. Its dependencies are therefore separate artifacts from the release build's, and only stay cheap because the build-only main push compiles them into both release caches. Without that, they compile from scratch and the job gets slower, not faster.
  2. docker run --memory=1g --memory-swap=1g --cpus=1 runs the seed (10,000 titles written 200 at a time, the way a scan page writes them, then the works page index built over them), a bookclerk library search --rebuild-index that builds the works search index, the daemon and the load in an Ubuntu container of the runner's own release (ubuntu:<VERSION_ID> from /etc/os-release, or ubuntu:24.04 on a non-Ubuntu host), because the hosts link the runner's glibc and libssl.
  3. On failure, it uploads bookclerkd-envelope.log with the metrics.

small-vps.sh reads memory.max and cpu.max, and refuses to record numbers when they are max or looser than 1 GiB and one core. It drops the build and object file cache, and writes envelope-metrics.json even when a later step fails.

What is gated:

  • The works routes must return 200 with at least one work each: the first page by title and by author, the status and account filters, a page 8,000 works in (by following next_cursor), and q=Title. The indexes are built before the daemon starts, as a library that has been running has them, so the daemon queues no rebuild of its own; before the routes are timed, the harness still waits up to 900 s for any index rebuild job to finish and for both indexes to answer.
  • Daemon anonymous memory stays under 256 MiB after idle and during the API windows. That is RssAnon when /proc/<pid>/status reports it, otherwise memory.stat anon, or memory.current when anon is absent.

What is only recorded:

  • memory.peak. The container can't reset it, so the lifetime peak includes the seed.
  • Latency.
  • available_parallelism (the daemon's logical_cpus), separately from the affinity mask.

One cpu.max core means one media worker.

Website and collector Workers

The job's checks are selected by path, with no Rust toolchain:

  • site_test: npm run typecheck (strict tsc --noEmit), npm run lint (oxlint) and npm test in tools/site, covering the generator, the Worker and the docs link check. Selected by tools/site/**, assets/brand/** or docs/**.
  • site_audit: tsc on the audit, then Lighthouse and axe-core on the built site (see Website checks). Selected by tools/site/** or assets/brand/**.
  • collector_test: npm run typecheck (strict tsc --noEmit), npm run lint (oxlint) and npm test in tools/diagnostics-collector. Selected by its folder.
  • demo_test: npm run typecheck (strict tsc --noEmit), npm run lint (oxlint) and npm test in tools/demo. Selected by its folder.

The full suite runs all four.

npm test is Vitest in all three Workers, configured in each one's vitest.config.ts with two projects:

  • workers runs the Worker's tests inside the Workers runtime through @cloudflare/vitest-plugin, on the compatibility date in wrangler.toml. The collector and the demo test against real D1 databases with their migrations/ applied. The site's Worker tests serve a real build, which a global setup makes in Node first.
  • node runs what needs the host: the site's build and docs-tree tests, the demo's page-script tests and the collector's Python-oracle fuzz test.

npm run coverage adds V8 coverage from both projects (text-summary and lcov under coverage/). CI runs it in place of npm test to feed the coverage report.

Other workflows

These run outside CI Gate:

  • CodeQL (see CodeQL below) feeds the repository's code scanning merge rule.
  • OSV scanning is a separate workflow and gate. A new push cancels a pull request's scan still in progress. A green OSV job that applies osv-scanner.toml IgnoredVulns means zero unignored, actionable findings, not that ignored advisories have left the dependency graph. Each ignore must record reachability evidence and an ignoreUntil review date. The scanner reads the config beside each lockfile, so fuzz/osv-scanner.toml repeats, with the same expiry, the root exceptions that the fuzz lockfile shares.
  • SQL-v1 property (.github/workflows/sql-v1-proptest.yml) runs a longer proptest suite (PROPTEST_CASES=256) on bookclerk-plugin-abi and bookclerk-db-exec: grammar-aware sqlite admission, plus a PostgreSQL 16 differential (normalized DbValue / DbErrorClass). It runs on workflow_dispatch and weekly. Pull-request CI keeps the modest in-crate case counts, deterministic corpus replay, and the postgres matrix (binding UTF8 readiness and the sqlite/postgres differential).
  • Docker image (.github/workflows/docker-image.yml) builds and smoke-tests the release image (see below).
  • Desktop packages (.github/workflows/desktop-packages.yml) builds the desktop app's unsigned packages on Linux, Windows and macOS with cargo package-desktop. The images workflow calls it nightly when main has moved, by hand and for a release (see Desktop packages). A pull request runs it only when it changes the packaging, and skips drafts.
  • Dev Container image (.github/workflows/devcontainer-image.yml) builds and checks the contributor image (see below).
  • ABS Integration (.github/workflows/abs-integration.yml) tests the audiobookshelf plugin against a real Audiobookshelf server (see below).

CodeQL

.github/workflows/codeql.yml runs CodeQL's default query suite on GitHub Actions workflows, JavaScript/TypeScript, Python and Rust, each in build mode none. The repository's "CodeQL results" merge rule reads its analyses. It skips a draft pull request and runs when the pull request is marked ready for review. GitHub's default setup for code scanning must be off, or the uploads are refused.

Rust takes about 20 minutes on a hosted runner, and CodeQL has no incremental analysis for Rust yet (tracked in #585). Rust can't be left out of a pull request either: the "CodeQL results" merge rule reports a pull request without /language:rust as neutral, "1 configuration not found", and that blocks the merge. So a pull request that changes no Rust input reuses main's Rust results instead:

  1. The Choose languages job diffs the merge commit against its base (git diff -z --no-renames --name-only).
  2. scripts/codeql_languages.py treats a path as a Rust input when it is a .rs file, a .capnp or .proto schema that a build script compiles into Rust, a .c, .cc or .h file (C or C++ a build script compiles into a crate: the FFmpeg shim, the SentencePiece tokenizer), a Cargo.toml, Cargo.lock, rust-toolchain.toml, anything under a .cargo/ directory at any depth, the workflow, or the scripts that choose and reuse. A name that isn't UTF-8 also counts. A test fails when a build.rs reads a file the chooser doesn't count, because CodeQL runs build scripts even in build mode none.
  3. Before allowing reuse, it scans the merge commit's tracked .rs files. A path also counts when a Rust source embeds it with include_str!, include_bytes! or include! (a plugin.toml, the brand logo, the SDK files bookclerk-workerd injects, even ci.yml), or when it sits in a crate with a build.rs. An include whose path it can't resolve turns reuse off for every pull request until the scanner learns that form. With no Rust input, it sets the reuse-rust output.
  4. Analyze (<language>) runs once per language, always all four.
  5. With reuse-rust set, Analyze (rust) runs scripts/codeql_reuse.py. It looks for a codeql.yml run on main for exactly the merge commit's base parent, started by a push, the schedule or a manual run, whose Analyze (rust) job succeeded. It downloads that run's codeql-rust-sarif artifact and checks that it is CodeQL Rust SARIF. The job then uploads it with category /language:rust.
  6. If any of that fails, the job analyzes Rust as usual. That happens, for example, when main's run for the base commit is still going.

The reused results match main's analysis of the base commit, so code scanning reports no new and no fixed Rust alerts. A real analysis would report the same, because nothing it reads changed. The job matrix stays the same either way, because code scanning records the matrix values with each analysis and a pull request has to match main's configuration.

Pushes to main, the Monday schedule and manual runs analyze every language, and the Rust job keeps its SARIF as the codeql-rust-sarif artifact for 30 days. The Rust extractor's own cargo output (build scripts and proc macros) is cached under codeql-rust-target-<rustc version>-<Cargo.lock hash>: main saves it and pull requests only restore it.

The file names are the pull request author's, so the chooser is an untrusted-input boundary, fuzzed in scripts/tests/test_codeql_languages.py (see Fuzzing).

The Docker image job builds packaging/docker/Dockerfile and runs packaging/docker/smoke.sh, which boots the image twice on one fresh /config volume:

  1. The first boot must install the bundled sqlite, local and embedder plugins and load the database guest under the default isolation.
  2. The second must reinstall nothing.
  3. The image must not override BOOKCLERK_PLUGIN_ISOLATION or BOOKCLERK_MEDIA_ISOLATION.

It is a full release build, so it runs weekly, on demand, and on pull requests that touch the image or its startup install path: packaging/docker/**, .dockerignore, the workflow, bookclerk-dev, the platform plugin sources, the bundled-install, trust and ledger modules, the database loader and the workerd pin.

The ABS Integration job (.github/workflows/abs-integration.yml) runs a real Audiobookshelf server beside the production image with the audiobookshelf plugin added (tests/abs-integration/). The plugin is built from the checkout, so it does not get the host-implied grant to its base_url that the crates.io package gets. The test sets it up the way an operator would:

  1. It approves the plugin through the consent API with a TCP grant to the server.
  2. It sets base_url, library_id and the sealed api_key with bookclerk plugins config set, and deploys the plugin.
  3. It checks integrations test and integrations status.
  4. It redeems a claim ticket on the portal and signs in to the portal with Audiobookshelf credentials.
  5. It starts a remote library scan.

Both containers share one network namespace, because the plugin accepts plain http only for a loopback host. The job runs by hand, on a published release, and on pull requests that change the test or the plugin. To run it locally, build the images as its workflow does, then run:

bash tests/abs-integration/smoke.sh

The Dev Container image job builds .devcontainer/Dockerfile and runs the contributor toolchain inside it as the dev user (cargo, rustfmt, Clippy, cargo-nextest, capnp, Node.js, npm, uv, PyYAML, git, ssh). It runs weekly, on demand, and on pull requests that touch .devcontainer/**, scripts/dev-env/**, scripts/tests/requirements.txt, rust-toolchain.toml, .tool-versions, .dockerignore or the workflow.

scripts/tests/test_dev_env.py (in the plan job) checks on every run that the image's tool pins match CI and .tool-versions, and that every Dockerfile's base images are tag and digest pinned, aligned, and covered by Dependabot.

Native-behind-workerd gateway smoke

The smoke tests are crates/bookclerk-plugin-e2e/tests/native_gateway.rs, plus native_gateway_isolation.rs, native_gateway_lifecycle.rs and native_gateway_authenticated_endpoint.rs. They:

  1. Install the test-only native_gateway_probe guest (an SDK-only bin of the e2e crate, never staged or packaged) into a temporary files folder.
  2. Grant exactly one ephemeral loopback TCP port.
  3. Spawn it through PluginSession::spawn_with with Isolation::Required.

The session goes through host discovery and launch planning, then two sibling bookclerk-jail processes:

  • bookclerk-workerd and pinned workerd in the gateway jail;
  • the native backend in a deny-network sibling, joined by inherited duplex links. On Windows, that is two host-created AppContainers plus a session Job, with no named-pipe SOCKET_PROXY.

Against harness-held listeners, the tests assert that:

  • describe, open and a real CLI entrypoint RPC answer through the front door;
  • a unique payload round-trips through bookclerk_plugin_sdk::net::connect to the granted port;
  • a second, live listener on an ungranted port is refused by policy (403) and is never dialed;
  • a direct std::net connect from the guest to the live granted listener is blocked;
  • both gateway_pid() and guest_pid() exit after drop, the host session folder is gone, and the guest can't read gateway state or the WORKERD_GRANT_* / BOOKCLERK_JAIL_* env;
  • two concurrent sessions can't exercise each other's grants or gateway state (see Concurrent sessions);
  • a guest exit, a missing workerd, or a mid-session kill fails closed.

Concurrent sessions

After both sessions authenticate and complete a connect, channel_ident through cli_invoke reports each configured tag. Those sessions don't publish candidate-endpoint metadata, so extra_status=not-run. That status isn't proof that the other session's endpoint is absent.

So a paired fixture keeps one pre-authenticated endpoint B, with its own challenge (A's challenge isn't replayed onto B). It publishes the same candidate metadata in both cases, through the same cli_invoke path:

  • The normal launcher path doesn't preserve the test-only extra fd and doesn't list B's handles. The RPC reports extra_status=absent, because that slot is inaccessible.
  • A live object at the slot that isn't the candidate is ident-failed (a numeric collision), not absence.
  • When the fixture also transfers B through the guest jail handoff, the same RPC reports B's tag while A still answers.
  • A transfer without the candidate metadata stays not-run, even though the endpoint was handed off.

On Windows the challenge is written with an event, so that guest end isn't bound to the test process's I/O completion port. Binding it makes the guest report ident-failed instead of the tag. Production spawn doesn't add the slot, and a numeric fd or handle isn't identity.

Nothing skips

A missing helper, runtime or confinement backend, a startup failure or a timeout fails the test.

  • On Windows, sequential_and_concurrent_spawn_cycles_do_not_leak runs by itself before the suite. The suite runs it again in the original order, with windows_job_extras_deny_the_next_direct_ping.
  • Smoke-only inputs (the four targets, tests/native_gateway/ and the fixture guest source) select native_gateway but not the staged E2E suite.

Coverage

CI Gate checks how much of a change its tests run. It fails when the change's patch coverage is below 80% (patch_target), an area's changed lines miss that area's own patch target, or an area falls below its floor, and writes a summary either way. Setting mode = "report" in scripts/coverage_areas.toml turns those failures back into warnings.

Coverage is measured in the jobs that already run the tests, so it costs no extra test run:

Area Measured by Tool
rust test (rust_test) cargo-llvm-cov, source-based LLVM coverage
site, collector, demo Workers (site_test, collector_test, demo_test) Vitest V8 coverage (npm run coverage)
python-sdk fmt / clippy / docs (python_sdk) pytest-cov
workerd-guests fmt / clippy / docs (workerd_guests) Node's test runner coverage (node --test --experimental-test-coverage)

How a run gets there:

  1. The workflow sets CI_COVERAGE_DIR on those steps. With it set, scripts/ci-exec.py runs each check with coverage and writes <dir>/<area>/lcov.info, or <dir>/<area>/part-<n>/lcov.info for one part of the split test job (see Test runner and retries). Unset (locally, by default), the checks run as before.
  2. Each job uploads its reports as a coverage-* artifact. The plan job uploads the hunk headers of git diff --unified=0 base...head as ci-patch, because CI Gate checks out a single commit.
  3. After comparing the job results, CI Gate downloads both and runs scripts/coverage_gate.py. It writes the summary to the job's step summary.

The summary has two figures:

  • Patch coverage: of the executable lines the change adds, the share the tests ran. The whole change must reach 80%. Lines in an area this run did not measure, such as Rust on a docs-only change, are listed apart rather than counted.
  • Area patch targets: each area's floor rounded down to a multiple of five, between 80% and 95%. An area whose target is above 80% must also reach it on its own changed lines, so well-tested code keeps new code to the same standard. A floor of 98% gives a target of 95%, and one of 93% gives 90%. Targets rise with the floors, so nobody edits them. The goal is 100%, but the gate stops at 95% because it has no reviewed way yet to mark lines no test can run.
  • Area floors: each area's total line coverage against its floor in scripts/coverage_areas.toml. A selective Rust run tests only the affected packages, so the rust floor is checked only when rust_test ran the whole workspace.

Floors ratchet. After every successful merge group run, which measures every area in full, coverage-floors.yml reads that run's coverage and runs scripts/coverage_gate.py --ratchet on main once main contains the commit that run measured. It waits up to five minutes for that commit to land and skips the run if it does not. Only code already on main runs in the job, because the job also holds the automation App token:

  1. Each floor rises to the area's measured coverage rounded down, less one point, so noise does not fail a run.
  2. A floor rises by at most five points per run and never above 99.
  3. A floor never falls. Lowering one stays a reviewed, hand-made change; never lower a floor to make a change pass.

When a floor rises, the workflow opens a ci/coverage-floors pull request with the automation App and turns on auto-merge, so the merge queue lands it once its checks pass. Turning on auto-merge gets four attempts, because GitHub's API sometimes fails a single call. If all four fail, the run fails, and the next merge to main tries again. While that pull request is open, later rises update it rather than opening another. Once it is in the merge queue, GitHub refuses updates to its branch, so a later rise leaves it alone and says so in the run's summary; a later merge raises the floors further only if its own coverage allows. Floors move in whole points, so most merges change nothing and the run ends after a few seconds. To run it by hand, use Run workflow on the Actions tab; it then reads the latest successful merge group run, whose coverage artifacts are kept for seven days.

Generated files are left out of both figures. They are listed under generated in the areas file, because tests exercise the generator rather than each line it writes. A file that an area owns and that starts with a generated-file header (GENERATED FILE - do not edit) must be on that list, or a test fails. Add a new generated file there when its generator lands.

Note

An unpublished OSV remediation run uploads no coverage and CI Gate skips the report. The diff and reports would show the bumped versions.

To check coverage before pushing, run a Worker's npm run coverage, or pytest with --cov=packages/plugin-sdk-python/src. For Rust, cargo llvm-cov nextest -p <crate> covers one crate; the whole workspace needs a lot of disk.

Web API contract

The daemon's test build layers a check over its router (crates/bookclerkd/src/openapi_contract.rs): every response the tests produce is compared with the committed OpenAPI document. The route the router matched, the method, the status and the media type must be declared, and a JSON body must validate against the declared schema, references followed and formats asserted. Each response is one JSON line in target/contract-report/<pid>.jsonl (one file per test process, since cargo-nextest runs each test in its own); each part of the test job keeps its files in contract/part-<n>/ beside its coverage report, and CI Gate runs scripts/contract_coverage.py over them.

CI Gate writes the only Web API contract report of a run. The plan job's tests of the script run it too, but neither they nor any other script test in that job write to the job's step summary.

The step summary shows how many of the declared responses (route, status and media type) the tests observed, in all and (in a fold) per tag, and lists every disagreement: a status or media type the route does not declare, a body that does not validate, a body that is not JSON, and a matched route the document does not describe (an HTML page, such as the invite page or the SPA, is served on purpose without one). A declared response nobody observed is not a finding yet; it marks a route the tests do not reach.

The step is enforced: a disagreement fails CI Gate, and so does an observed share below the floor [contract] floor in scripts/coverage_areas.toml states (a percentage of the declared responses; raise it when the share rises, keeping it a little below the measured value, and never lower it to make a change pass). When no report file exists because the daemon's tests did not run (a build-only main push, a selective run that did not test the daemon's package, or a test job that failed or was cancelled before writing one), the floor is not checked and the step prints one line instead of adding an empty report to the summary; a failed or cancelled test job fails CI Gate on its own. When the plan tested the daemon's package, the test job succeeded and no report exists all the same, or a report line is not a whole hit (a partial write), the gate fails too. The layer never fails a test, so a disagreement shows up in CI Gate's summary, not in a test's output.

A disagreement means the document and the daemon differ for one response. For a route's own statuses, fix the #[utoipa::path] declaration or the handler that is wrong: a status the handler answers on purpose is declared (with a realistic example when the body is JSON), and a handler that answers something its route should not is corrected. The access layer's refusals (401, 403, the second-factor enrollment 403, the session_check_unavailable 503), the CSRF 403, the 504 timeout and the JSON extractor's rejections are stated by the document builder (crates/bookclerkd/src/openapi.rs), not by each route. Where a route also answers a 503 of its own, the builder states both bodies and says only the session-check refusal carries Retry-After. A public route whose handler checks the session itself declares a bare 503 that names session_check_unavailable, and the builder adds the body and the header.

To see the report locally, start it empty (each test process appends a file of its own, so an old run's hits would otherwise stay in it), run the daemon's tests and the script:

rm -rf target/contract-report
cargo nextest run -p bookclerkd
python3 scripts/contract_coverage.py --reports target/contract-report --mode enforce

BOOKCLERK_CONTRACT_REPORT_DIR moves the report files elsewhere.

Codecov

The same reports also go to Codecov, which comments on pull requests and posts codecov/project and codecov/patch statuses. Both statuses are informational: CI Gate is the coverage check that blocks a merge, and it reports on every pull request.

The upload runs in its own workflow, coverage-upload.yml, after each CI run of a pull request or merge group finishes:

  1. It runs from the default branch through workflow_run. CI keeps its read-only token, and fork pull requests upload too.
  2. It downloads the run's coverage-* artifacts and plan, and runs scripts/coverage_gate.py --export. That rewrites each report through the gate's parser, keeping only repository-relative paths, line numbers and hit counts.
  3. It uploads each area under its own flag, signing in to Codecov with OIDC, so there is no Codecov token secret. The commit, branch and pull request come from the CI run GitHub recorded, never from an artifact.

A newer CI run on the same pull request branch cancels an upload still waiting or running; each merge group's upload runs to the end.

The upload never checks out the pull request. Its checkout is the default branch, so it sends Codecov no file list and no line fixes from it. The reports' paths are already repository-relative. A head branch name outside a conservative character set skips the upload, because Codecov must never file it under the default branch.

The settings live in .github/codecov.yml. Each area is a flag with carryforward, so a selective run that skips an area keeps that area's last figure. A selective run that tests only some Rust packages uploads under rust-partial instead of rust. That flag is never carried forward, so the whole-workspace rust figure stays as it was while the change's own lines are still measured. The patch target matches CI Gate's, and Codecov ignores the same generated files. A test checks that the flags, their paths, the target and the ignored files agree with scripts/coverage_areas.toml.

Components give parts of the code that cut across flags their own figure on the dashboard and in the comment: the security surface, the daemon, the CLI, the plugins, the plugin SDKs and the Workers.

Merge groups upload too. Their commit is the one that lands on main, so it gives later pull requests their base.

Important

Uploads need the Codecov GitHub App installed on the repository. Without it, the upload steps warn and the workflow still passes.

To make codecov/patch required later:

  1. Remove its informational: true line in .github/codecov.yml.
  2. Add codecov/patch to the default branch's ruleset, with the Codecov App as its source.

Test results

The test jobs that measure coverage also write a JUnit report beside each LCOV report. The upload workflow sends them to Codecov Test Analytics, under the same flags. Codecov then lists a pull request's failed tests in its comment and, from merge group runs, finds tests that are flaky on main.

  • Rust: the main nextest run uses the ci profile in .config/nextest.toml, which writes target/nextest/ci/junit.xml. A workflow step copies it into the coverage directory, even when tests fail. Feature-test runs and custom-harness tests write no report.
  • The Workers: Vitest's junit reporter, next to the coverage run.
  • The Python SDK: pytest's --junitxml.
  • The first-party workerd guests: Node's junit test reporter.

The test jobs upload their artifacts even when tests fail, so failures reach Codecov. Like the coverage reports, the JUnit reports come from the pull request's own run. coverage_gate.py --export reads each one with a fuzzed parser and writes back only test names, durations, outcomes and failure text.

Bundle sizes

Codecov Bundle Analysis tracks the size of the web UI and of each Cloudflare Worker script, and comments when a pull request changes one by at least 1 KiB. That gives early notice before a Worker nears Cloudflare's script size limit.

  1. The ui check keeps ui/dist after its build. The Worker checks (site_audit, collector_test, demo_test) bundle each Worker with wrangler deploy --dry-run --outdir, the same bundling a deploy does, without deploying or needing an account. Each lands in $CI_BUNDLE_DIR/<bundle>/, uploaded as a bundle-* artifact.
  2. The upload workflow downloads them. coverage_gate.py --export --bundles copies only regular files with safe names, known asset types and bounded sizes (select_bundle_files, a fuzzed boundary). Source maps are left out.
  3. The pinned @codecov/bundle-analyzer in tools/codecov, installed without install scripts, sends each bundle's file names and sizes to Codecov, signed in with OIDC like the coverage uploads.

The bundle status is informational, like the coverage statuses; its settings are under bundle_analysis in .github/codecov.yml.

Mutation testing

Coverage shows which lines the tests run, not whether a test would notice them doing the wrong thing. Mutation testing checks that: it makes a small change to the code, such as flipping a comparison or making a function return a default value, and runs the tests. If a test fails, the mutant is caught. If every test still passes, the mutant is missed: nothing checks that behavior, so a real bug there would slip through too.

Mutation testing is slow, so it does not run on pull requests. Three workflows run it once a day on main instead, each on the code merged since its own last successful run:

Each run works like this:

  1. scripts/mutation_base.py finds the commit the workflow's last successful scheduled run tested, through the GitHub API. A day with nothing merged, or a run that failed, is covered by the next run. If the API cannot be reached, the run fails rather than guess. With no earlier successful run, it falls back to the last commit from more than 25 hours ago, then to the parent of HEAD. Whatever it picks must be a full SHA and HEAD or one of its ancestors. When nothing merged since the last run, the range is empty and nothing is mutated.
  2. The run diffs main against that commit and mutates only what changed, as described below for each language. The run number seeds every sample, so the choice changes from run to run.
  3. The job summary lists what was tested and what was missed. A run that did not test its range fails, so the next run covers the same changes: the baseline failed, the tool exited with an error, or it left no results. A run stopped at its time limit, or early because the runner ran low on disk or memory, still succeeds with what it tested.
  4. When a mutant was missed, the reusable mutation-issue.yml workflow files the summary as an issue titled Missed mutants: <area> (rust, python-sdk, site, demo or collector), labelled mutation-testing. A later run with missed mutants in the same area comments on the open issue instead of opening another. Close the issue once the tests catch them.

The issue job holds the only write token in these workflows. It checks out nothing and runs no repository code; it reads only the summaries the mutants jobs uploaded.

To test a range by hand, run the workflow from the Actions tab and set base to the commit to start from. Running it from a branch tests that branch against base, which is how to try a change to the workflows themselves. Only runs on main file issues, and a manual run never moves the starting point of the next scheduled one.

Rust

cargo-mutants tests the Rust code:

  1. It mutates only the functions changed since the last run (--in-diff) in any workspace member (--workspace; without it cargo-mutants looks only at the default members), so the cost follows what merged. With no Rust file changed, the job builds nothing. Test code, the generated files and the staged-plugin harness (bookclerk-plugin-e2e, whose tests need the plugins CI's e2e check stages) are never mutated.
  2. Mutants live only in function bodies, so a change to data that code reads (an attribute, a constant, a type definition) gives no mutants from the diff. When a changed Rust file gets none, the job mutates the functions in that file instead, so the tests of the code beside the changed data are still checked. The summary names those files. Up to 60 of their mutants are all tested; past that, at most 60 spread across the files are (counting every mutant a chosen line brings in, such as its function's body replacements), and the choice changes with each run. Each file gets at least its smallest line while that fits. If every line alone brings in more than 60, the one that brings in the fewest is tested, so such a run can exceed 60. Each mutant runs its package's whole test suite, two minutes for some crates, so a large file would otherwise hold a runner for the whole time limit. Code in other files that reads the data is not covered.
  3. Before the first mutant, the job builds what CI's rust_test check builds first (the jail, the workerd front door, the pinned workerd and the platform guests) with ci-exec.py prereqs rust_test, and turns on the same confinement enforcement as the test job. Tests that launch a helper then run instead of skipping.
  4. Each mutant runs the tests of its own package under cargo-nextest.
  5. The run stops after 4 hours and reports what it tested so far. It also stops early, the same way, when less than 3 GiB of disk is free or less than 768 MiB of memory stays available for two checks 15 seconds apart. A runner that runs out of either loses contact with GitHub, and the job then fails with no log and no results, so the next run would retry an even larger range. The job frees the preinstalled SDKs it never uses before it builds, logs the disk and memory left every 10 minutes, and the summary says why a run stopped early.
  6. Those checks are too slow for a mutant that allocates without bound, so cargo-mutants also runs in a cgroup capped at all but 1.5 GiB of the runner's memory. The kernel then kills the largest process in it, normally that mutant's test, instead of starving the runner, and the mutant counts as caught. The log notes the cap and how many processes it stopped. When the cgroup cannot be set up, a warning says so and the run goes on uncapped. A kill that lands on cargo itself leaves the mutant unclassified; each one gets its own warning, and the job summary says the results are incomplete, because no list holds it. A kill that lands on cargo-mutants itself, before the time limit and without the watchdog, fails the run, so the next run retries the same range.

The job summary shows the share of mutants the tests caught and lists the missed ones. The full results, with a log per mutant, are in the mutants-rust artifact. A mutant that does not build (unviable) says nothing about the tests and is left out of the score; one whose tests time out counts as caught.

To fix a missed mutant, add a test that fails with the change it names, or tighten an assertion that should already have caught it. To reproduce one locally:

cargo install cargo-mutants --locked
git diff origin/main... > /tmp/change.diff
RUSTFLAGS=--cap-lints=warn cargo mutants --workspace --in-diff /tmp/change.diff

For a file the summary says was mutated whole, mutate all of it with --file:

RUSTFLAGS=--cap-lints=warn cargo mutants --workspace --file crates/bookclerkd/src/api.rs

Capping lints keeps a mutant that only leaves a parameter unused from failing to build under the crates' deny lints; the workflow caps them for every build in the job. Settings live in .cargo/mutants.toml: nextest as the test runner, the custom-harness test binaries CI runs with cargo test left out of nextest, and the excluded files, whose generated entries must match generated in scripts/coverage_areas.toml. Tests check both lists.

Workers

The mutation-workers.yml workflow does the same for the Cloudflare Workers (tools/site, tools/demo and tools/diagnostics-collector) with Stryker, one job per Worker:

  1. It finds the lines changed since the last run in the Worker's TypeScript sources: the lines added, and the line before each run of lines it deletes, so removing a guard counts as changing the code around it. That line can be a blank line or a closing brace with no mutant; when none of a file's changed lines has one, step 3 samples the file. Tests, test support, type declarations and configuration files are left out. A Worker with no changed source line skips Stryker, and its summary says so.
  2. It passes those lines to Stryker as --mutate path:start-end. Stryker also mutates literals (strings, arrays, objects), so a change to data gets mutants too.
  3. Some changes give no mutant at all: a type, an interface, an import or an export list. When a changed file's lines give none, a second pass mutates a sample of the file's other lines, so the tests of the code beside the change are still checked, as the Rust job does for data changes. The sample skips lines Stryker never mutates (types, imports, bare punctuation). At most 30 lines are sampled across those files, one per file first while the budget lasts, and the choice changes with each run. Stryker takes about five seconds a mutant here, so a whole file would not fit the time limit. The summary names the files and how many lines it sampled in each, or says when too little time was left for the second pass.
  4. Each mutant runs the Vitest tests that cover it, including the ones that run inside the Workers runtime.
  5. Both passes together stop after 40 minutes. The summary then counts the mutants tested before the limit and says the run was stopped.

The job summary and the mutants-<worker> artifact have the same shape as the Rust job's, and missed mutants become a site, demo or collector issue. Stryker is installed for the run only, at the version pinned in the workflow, so the Workers' lockfiles do not change. Settings live in scripts/stryker.config.json:

  • inPlace, because some tests read repository files by relative path. Stryker then rewrites the sources it mutates on disk while it runs, so the step sets MUTATION_TESTING=1 and each Worker's vitest.config.ts leaves out the tests that read its own source text (tools/site/worker-source.test.ts, tools/diagnostics-collector/cache-control-source.test.ts). Those tests would fail on the instrumented code. A mutant never shows in the source text, because Stryker switches mutants on at run time, and the one function they call, the collector's assetUrlsModule, has its own test in cache-control.test.ts. Every other run keeps them.
  • A tsconfigFile that does not exist, because TypeScript 7 has no JavaScript API for Stryker's tsconfig rewriting. Type checking stays with each Worker's npm run typecheck.
  • disableTypeChecks off, so Stryker does not add // @ts-nocheck to every file. That would change the collector's hashed dashboard assets, and Vitest does not check types anyway.
  • vitest.related off, because the Workers runtime loads modules outside Vitest's module graph, so Vitest cannot tell which tests import a file.
  • incremental, writing to reports/mutation/mutation.json, because only incremental mode saves the mutants tested so far when the run is stopped. The second pass also mutates the first pass's lines, so their results carry over from that file without being tested again. The report step reads it.

To reproduce a Worker's run locally, from its directory:

npm install --no-save @stryker-mutator/core@10.1.1 @stryker-mutator/vitest-runner@10.1.1
MUTATION_TESTING=1 npx stryker run ../../scripts/stryker.config.json --mutate "src/page.ts:10-20"

The report lands in reports/mutation/mutation.json. Delete that folder afterwards: incremental mode reuses the results in it on the next run.

Python SDK

The mutation-python.yml workflow does the same for the Python plugin SDK (packages/plugin-sdk-python) with mutmut. Before mutmut starts, it installs capnp and builds the bookclerk-db-testkit binary that tests/test_testkit.py drives, as CI's python_sdk check does, so no test run builds it:

  1. It finds the lines changed since the last run under the SDK's src/: the lines added, placed in the new file, and the lines deleted, placed in the starting commit's copy of the file. Removing a guard, even a function's last lines, counts as changing that function, and removing a constant counts as a change outside every function. Removing a whole function selects nothing in that function. Tests and the generated ABI modules are left out. With no changed source line, mutmut does not run and the summary says so.
  2. mutmut names each mutant after its function, so scripts/mutmut_report.py turns each function holding a changed line into a name pattern, such as bookclerk_plugin_sdk.tools.x_check_plugin__mutmut_*. A method's name is xǁ<Class>ǁ<method>, with mutmut's ǁ separator.
  3. mutmut only mutates function bodies. A changed line outside every function, such as a constant, an import or a class attribute, selects the whole module instead, so the tests of the code beside the changed data are checked, apart from the functions mutmut never mutates (step 4). The summary names those modules. Blank lines, comments and docstrings outside functions select nothing. The SDK's tests run in about a second, so most modules fit in the time limit.
  4. mutmut never mutates a decorated function (other than one with a single @staticmethod or @classmethod) or an enum's methods. The summary names the changed functions it skipped for that reason.
  5. Each mutant runs the tests that cover its function. A mutant no test runs counts as missed. mutmut picks those tests by node id, and pytest cannot find a node id again once it has escaped a backslash or a non-ASCII character in it, so tests/conftest.py refuses such test ids. Give a parametrized case an ASCII id instead of repr of its input. mutmut also runs the tests from a copy one folder deeper, under mutants/, so a test finds the repository root by a folder it holds, never by a fixed parents[N]; tests/test_conventions.py checks that.
  6. The run stops after 60 minutes. mutmut records each result as it lands, so the summary then counts the mutants tested before the limit and says the run was stopped. If mutmut fails or stops before testing any selected mutant, the summary says that instead of reporting no mutants.

The job summary has the same shape as the other jobs', with each missed mutant's line and change, and missed mutants become a python-sdk issue. Mutants mutmut ran that ended with a result it does not count (a crash, a skip or a suspicious result) are left out of the score, and the summary says how many there were. The mutants-python-sdk artifact lists every mutant, one file per outcome. mutmut is installed for the run only, at the version pinned in the workflow, so the SDK's uv.lock does not change. Settings live in the SDK's pyproject.toml, under [tool.mutmut]: the package to mutate, the generated modules to leave alone (the same list ruff excludes, which a test checks) and the test folder.

To reproduce the run locally, build the testkit first (it needs Rust and capnp, as the workflow installs), then run mutmut from packages/plugin-sdk-python:

cargo build -p bookclerk-db-testkit --bin bookclerk-db-testkit
export BOOKCLERK_DB_TESTKIT="$(git rev-parse --show-toplevel)/target/debug/bookclerk-db-testkit"
python3 -m venv /tmp/mutmut && /tmp/mutmut/bin/pip install pytest "mutmut==3.8.1"
/tmp/mutmut/bin/mutmut run "bookclerk_plugin_sdk.tools.x_check_plugin__mutmut_*"
/tmp/mutmut/bin/mutmut show bookclerk_plugin_sdk.tools.x_check_plugin__mutmut_3

mutmut works in a mutants/ copy of the SDK and reuses the results in it on the next run. Delete that folder afterwards.

Note

None of the three workflows is a required check, and none runs on pull requests. A missed mutant never blocks a merge; it becomes an issue.

Worker startup time

Cloudflare rejects a deploy whose Worker takes more than one second to run its global scope. CI tracks how close each Worker (site, demo, collector) is, with scripts/worker_startup.py and the budgets in scripts/worker_startup.toml.

  • Pull requests: each Worker check (site_audit, collector_test, demo_test) bundles its Worker and runs wrangler check startup on it seven times. That profiles startup locally in workerd and needs no Cloudflare account, so forks get it too. The step summary shows each run's active CPU time and the median, and a median over the Worker's limit_ms budget warns.
  • Deploys: wrangler deploy prints the startup time Cloudflare measured. Each deploy workflow writes it to its run summary and warns past deploy_warn_ms (500 ms).

Note

Local figures are a trend, not Cloudflare's numbers: the runner's CPU differs, the profiler samples roughly every 10 ms (a small Worker reads as 0 to 15 ms), and the first run is often a cold start. The median of seven runs smooths that out; budgets sit well above today's medians so only a real jump, such as a large library imported at the top level, warns.

Both start in report mode. Set mode = "enforce" in the budgets file to make a Worker over budget fail its check.

Branch protection and the merge queue

Configure repository rulesets or branch protection so the required CI check is CI Gate (plus the OSV check), not every matrix child. The Bookclerk CLA and Maintainer review checks are required too, expected from the automation App (see the CLA check and maintainer review). The ruleset also requires signed commits; the Signed commits check explains a pull request that its unsigned commits keep out of the queue (see signed commits).

Warning

Skipped release, confinement, native-gateway, tray and macOS jobs report skipped. If those job names are required individually, merges stay pending forever.

Also enable a merge queue that consumes merge_group checks, so the full suite runs on the synthetic merge commit before landing. Merge queues are only available on organization-owned repositories (free for public repositories).

Queue settings:

  • squash;
  • "only merge non-failing pull requests";
  • build concurrency 2;
  • merge limits: minimum 1, maximum 5, wait time 5 minutes;
  • status check timeout 300 minutes.

With a queue required, "require branches to be up to date" no longer matters. The queue tests each entry against the current main, stacked on the entries ahead of it.

Sizing the queue to the runner pool

The organization runs on GitHub Team, which allows 60 hosted jobs at once across every workflow, 5 of them on macOS, and hosted runners serve queued jobs first come, first served. A job from a pull request gets no priority over one from a merge group, or the other way round; the pull request lanes keep at least 24 runners for the queue.

The merge queue starts one merge_group run per entry it builds, each holding the entries ahead of it. Merge limits don't change that: GitHub documents that they "do not combine merge_group builds" and only decide how many passing entries land at once. Build concurrency is the only setting that caps how many groups test at the same time.

One merge group asks for about 28 jobs: the CI jobs (most of them start together right after plan), the five test-os legs, plus OSV-Scanner, the CLA check and the coverage upload. Most finish within 10 minutes. The test-os legs (about 30 minutes each), the fat-LTO release build (about 26), the three Linux test parts and fmt / clippy / docs (about 15 each), the long Windows jobs and merge-queue fail-fast, which holds a runner until the run ends, keep going. A group takes about 350 runner-minutes; with its whole fan-out running at once it takes about 35 to 40 minutes.

Build concurrency Merge-group runners (peak, typical) macOS runners Left for pull requests
1 about 25, then about 10 3 about 35 to 50
2 about 50 when fan-outs overlap, about 20 typical 6 at overlaps (4 once the macOS job folds into test-os) 36 guaranteed, about 40 typical
3 or more the whole pool 9 or more, over the 5 allowed 36, the lane floor

Build concurrency 2 lands about 3 pull requests an hour, against about 1.7 at 1. It is the highest that fits: the organization may run only 5 macOS jobs at once. Each group runs the macOS job and two macOS test-os legs, so one of six macOS jobs can wait for a runner where two groups overlap. Once test-os replaces the macOS job, two groups need 4; a third group would always wait.

On GitHub Free (20 runners, until 20 November 2026) these docs advised build concurrency 1, though the setting was back at 2 by then. Two groups then took most of the pool: on the busy night of 9 to 10 November 2026, with two groups and leftover runs building (see Superseded merge groups), merge-group jobs waited 10 to 25 minutes for a runner, runs took about 65 minutes, and the queue landed about 1.3 an hour while pull request checks waited behind it.

A failure costs more with two groups building: a failure (including a flaky test) ejects the head entry and rebuilds the group behind it. "Jump to the front" likewise restarts the groups in progress.

The status check timeout counts from the moment a group requests its checks, including the time its jobs wait for runners. On 10 November 2026, with pull request jobs holding most of the pool, a green group ran past the earlier 120-minute timeout and was ejected, so the timeout is now 300 minutes.

Raise build concurrency again only when the macOS limit grows, for example an Enterprise plan (50 macOS jobs) or a self-hosted macOS runner reserved for merge_group.

Superseded merge groups

The queue rebuilds its groups when an entry ahead fails or leaves, when a pull request jumps to the front, or when one is dequeued. It deletes each dropped group's gh-readonly-queue/main/pr-<number>-<sha> branch, but GitHub doesn't cancel that group's run, and the workflow's concurrency never cancels a merge_group run. The run would keep asking for its 22 jobs although nothing can merge from it.

merge-queue-fail-fast therefore checks the group's branch (github.event.merge_group.head_ref) with git ls-remote --exit-code on every pass and cancels the run once the branch is gone. It starts with the run, so it gets a runner before the jobs that fan out after plan. The queue also deletes a group's branch after merging it, so the check comes after the job's CI Gate test, and the job looks again for a started CI Gate just before it cancels. A transport error counts as the branch being present. A cancelled run doesn't trigger re-enqueue, because its jobs carry no not-acquired annotation.

macOS job

The macOS runs of confinement, native-gateway and tray share one job, macOS (sandbox, gateway, tray), on macos-latest. Since 2 November 2026, GitHub has often failed to give macOS jobs a runner, whichever macOS label they ask for. In the runs checked on 4 November, no more than two of this repository's macOS jobs ran at the same time. One job per run asks for one macOS runner instead of three.

The job runs when any of the three jobs is planned. It runs each planned check as its own step once setup succeeded, and a check still runs after an earlier check failed, so one round shows every failure. It saves the macOS debug cache on main, as native-gateway does on Windows. In OS_LEG_JOBS in scripts/ci_plan/plan.py, CI Gate expects the job whenever it expects any of the three, except on a pull request, where it waits for the merge queue (see Platform legs in the merge queue).

GitHub sometimes never gives the job a runner. It cancels the job with the not-acquired annotation (see re-enqueue below), and CI Gate fails, as for any job that doesn't succeed. After one retry, the macOS checks are best effort:

  1. A pull request run is re-run once (attempt 2). On a merge group, the pull request goes back in the queue once.
  2. If the macOS job gets no runner again, CI Gate passes without it. A step in CI Gate runs scripts/merge_queue_requeue.py --stranded-job to check that the job was cancelled with no runner and no step and carries the not-acquired annotation, and that the same happened on the try before: the previous attempt of a pull request run, or the pull request's previous merge group, when it has entered the queue at least twice on its current head. That merge group must have started after the pull request first entered the queue on its current head, so a run of an older head never counts. A first miss after a retry for another reason still fails.
  3. The gate prints a warning and a step-summary note that the macOS checks didn't run on that commit.

A build-only main push needs no retry. There the macOS job only compiles and saves the macOS debug cache, and the merge queue already ran the macOS checks on that commit. If GitHub never gives it a runner, the same step vouches for it on the first try, so main doesn't show a failed run for a cache save. The next main push that gets a runner saves the cache.

CI Gate sees such a job as abandoned in needs (the jobs API reports it as cancelled), so it accepts either result for a job the step vouched for.

A macOS job that ran and failed, or was cancelled after it started, always fails the gate. So does a stranded job on an OSV remediation run or a push to any branch other than main. The gate job reads the GitHub API with a read-only token (actions, checks and issues).

Merge-queue re-enqueue

Hosted runners, most often macOS arm64, sometimes never pick up a merge-group job. GitHub cancels the job with the annotation "The job was not acquired by Runner of type hosted even after multiple attempts", CI Gate fails, and the queue removes the pull request although every job that ran passed.

merge-queue-requeue.yml puts such a pull request back in the queue. It runs when a merge_group CI run fails, and scripts/merge_queue_requeue.py re-enables auto-merge with the automation App only when all of these hold:

  1. Every job that didn't succeed or skip is CI Gate, merge-queue fail-fast, or a job that was cancelled before it got a runner or ran a step.
  2. Each of those never-started jobs carries GitHub's not-acquired annotation. A job cancelled because someone removed the pull request from the queue has none, so a hand dequeue stays dequeued.
  3. The pull request is open, isn't a draft, comes from this repository and doesn't already have auto-merge on.
  4. Its current head is the one that was queued, and it has entered the queue fewer than three times, so one commit is retried at most twice. A commit pushed after the failure waits for its own enqueue, and the enqueue names the queued head (--match-head-commit), so a push in between makes it fail.

Every check stays required: the new merge group runs the full suite again. A run whose CI Gate passed (the macOS job let through) is never re-enqueued or re-run. The workflow runs from the default branch, reads only the GitHub API and runs no pull request code. The step summary says why it did or didn't enqueue.

A pull request's own CI run can fail the same way. When a pull_request CI run fails, the same workflow re-runs its failed jobs (the stranded jobs and CI Gate) on the same commit when:

  1. Every job that didn't succeed or skip passes the same test as steps 1 and 2 above. A run cancelled by a newer push has no not-acquired annotation, so it is never re-run.
  2. The run is on its first or second attempt, so one run gets at most two re-runs.
  3. No newer pull_request CI run exists for the same head branch and repository. CI cancels a pull request's older runs when a new one starts, so re-running a run that a push has superseded would cancel CI for the current commit. This is checked last, just before the re-run.

The re-run uses the workflow's own token (actions: write), not the automation App, and keeps the original run's context, so a fork's pull request is re-run with the same read-only permissions it had.

Expected pull-request feedback

With SELECTIVE_CI=1, a pull request should see these jobs after plan:

Change Jobs after plan Notes
docs/** only Workers (site_test only), fmt / clippy / docs (lint only) The site's link check covers the docs it publishes
tools/site/** only Workers (site_test, site_audit), fmt / clippy / docs (lint only) No Rust toolchain
tools/diagnostics-collector/** only Workers (collector_test), fmt / clippy / docs (lint only) No Rust toolchain
tools/demo/** only Workers (demo_test), fmt / clippy / docs (lint only) No Rust toolchain; packaging/demo/** selects nothing
README.md, AGENTS.md, other top-level prose, config/config.example.toml, scripts/tests/** fmt / clippy / docs (lint only) for Markdown and JSON The plan job runs the planner and lint tests on every event
ui/** only fmt / clippy / docs (UI, UI API docs and lint) No Rust toolchain checks
crates/bookclerk-cli/src/** check, test (-p bookclerk-cli), release (bookclerk-cli) No postgres, no E2E
crates/bookclerk-plugin-host/src/** check, test (host and dependents), e2e (runtime smoke), release (hosts), native gateway (Linux; Windows and macOS in the queue), postgres (rpc_like only) No sandbox matrix
One optional guest (such as Libro) check, test (guest tests), e2e (libro only) Other storefronts are neither built nor staged
packages/plugin-sdk/embed/** check (SDK ABI), test (workerd tests), e2e (workerd-runtime subset), release (launcher), native gateway (Linux; Windows and macOS in the queue) Not the full installation

A draft pull request runs only the fmt / clippy / docs and test jobs from this table; the rest wait until it is ready for review (see Draft pull requests). For merge_group, --force-full runs every job and check on every platform. A main push adds --build-only.

Next steps

Source: docs/contributing/ci.md at main · View as Markdown