Continuous integration
This page explains how Bookclerk's CI decides what to run: the execution modes, the planner, each job, and the branch protection that ties them together.
The workflow is .github/workflows/ci.yml. A dependency-aware planner
lets pull requests skip unrelated work, a draft pull request runs only the
fast jobs, pull requests test on Linux only, and merge_group always runs
the full suite on every platform. A push to main, which only advances through the merge queue,
compiles the full suite to refresh the dependency caches and runs no tests.
Fuzzing has its own page, Fuzzing. The workflows outside the pull-request checks (dependency updates, advisories, docs publishing and Worker deploys) are in Repository automation.
SELECTIVE_CI in .github/workflows/ci.yml selects the execution mode:
| Mode | When | What runs |
|---|---|---|
full |
A main push, merge_group, or any fail-closed trigger |
Every job and check at full scope (build only on a main push) |
shadow |
SELECTIVE_CI != "1" |
Every job at full scope, with the selective prediction in the plan summary |
selective |
SELECTIVE_CI == "1" on a pull request |
Only the predicted checks, each with its prerequisites |
Full-suite merge_group runs audit the selective coverage. A check that
fails there but was skipped on the pull request is a planner bug, and it
keeps the pull request out of main.
A draft pull request gets fast feedback without holding the shared runners
(the organization's plan caps how many Actions jobs run at once). Its plan
resolves with --draft (draft: true in the artifact):
- Only the
checkjob (formatting, Clippy, docs and lint) and thetestjob (Rust unit and integration tests, with coverage) run, at the scope the planner predicted (DRAFT_JOBSinscripts/ci_plan/execute.py). - Every other check the planner selected is listed under
deferredin the artifact and reads "deferred: draft pull request" in the plan summary.CI Gateexpects the deferred jobs to be skipped, so it reports on the draft instead of waiting. - CodeQL, ClusterFuzzLite, the Docker, Dev Container and image publishing builds and ABS Integration skip their jobs on a draft too.
- OSV scanning and the CLA check still run, so their required checks report.
Nothing is dropped, only deferred. Marking the pull request ready for
review (ready_for_review) starts the full selective run of every one of
these workflows, and each later push repeats it. GitHub keeps drafts out of
the merge queue, and the queue always runs the full suite, so no deferred
check can be skipped before a merge.
Note
On a draft, the CodeQL results merge rule waits for analyses that arrive
once the pull request is ready. Coverage covers only the areas the test
and check jobs measure; Worker coverage arrives with the ready run.
Every pull request run, draft or ready, also resolves with --linux-only
(linux_only: true in the artifact). The Windows and macOS runs are the
slowest jobs and rarely fail where Linux passes, so they wait for the merge
queue:
- The
confinement,native-gatewayandtraymatrices take theiroslegs from the plan'sos_matrixoutput:ubuntu-lateston a pull request,ubuntu-latestandwindows-latestin the merge queue and on amainpush (which saves the Windows cache). - The macOS job runs only where
os_legsis true, which is never on a pull request. - The Windows-only checks (
QUEUE_ONLY_CHECKSinscripts/ci_plan/execute.py, todaywindows_cluster) are listed underdeferredwith the reason "Windows, runs in the merge queue". - The release build already links thin LTO on pull requests and fat LTO in the queue (see Release build).
Every Linux job, coverage, CodeQL, OSV and fuzzing still run on the pull
request, so reviewers and Codecov see their results before it is queued.
CI Gate in the merge group requires every platform, so a Windows or macOS
failure ejects the pull request from the queue and keeps it out of main.
The jobs above run chosen parts of the suite on Windows and macOS, and in
November 2026 host bugs on both systems reached main through the tests
they leave out (#1228, #1229). The test-os job runs the whole
rust_test suite there too, in the merge queue and on the build-only
main push:
- Its legs come from the plan's
os_test_matrixoutput (OS_TEST_PARTSinscripts/ci_plan/execute.py): three parts onwindows-latestand two onmacos-latest, split with--partitionas the Linux test parts are. Windows takes about twice Linux's time and macOS about 1.3 times, so each part takes about 30 minutes. A Linux-only pull request run has no legs; a build-onlymainpush has one per OS, which only compiles and saves the job's own cache (see Caches). - The job sets
CI_OS_SUITE=1, so itsrust_testrun uses theci-osnextest profile (.config/nextest.toml) instead of the merge group's retry and fail-fast: every test runs, a failing one gets one retry, and a test still running after ten minutes is stopped and named, so a hang fails one test instead of holding the job until its timeout. A failing test command doesn't stop the part either: the custom harnesses and feature runs after it still run, and the job fails once all of them have reported. Setup and prerequisite failures still stop it. Runningrust_teston Windows or macOS without it keeps the usual behavior. CI Gateneeds it. The job is listed inOS_LEG_JOBS(scripts/ci_plan/plan.py) withtestas its owning job, so the gate expects it wherevertestruns outside a Linux-only pull request run, and a failing leg keeps the group out ofmain. The merge-queue fail-fast watcher cancels the run when a leg fails, as it does for any other job. It ran report-only (REPORT_JOBS, "report only" in its name) from 20 November 2026 until each OS had three green runs in a row, on 22 November 2026;REPORT_JOBSstays for the next new leg. The curated Windows and macOS jobs above still run beside it.
GitHub gives a merge_group run no priority over pull request runs. Every
job waits in one queue for the organization's 60 runners (GitHub Team), so a burst of
ready pull requests can hold most of them and leave the merge queue waiting
long enough to hit its status check timeout. To leave room for the queue
without leaving runners idle, heavy pull request jobs wait in shared
lanes:
- A lane is a job-level
concurrencygroup namedci-pr-lane-<n>withqueue: max: one job runs in it at a time, and up to 100 wait behind it in the order they arrived. The default would keep only one waiting job and cancel the others. A job pushed out of a full lane is cancelled, which failsCI Gatevisibly rather than skipping a check. - The lane jobs are
check,test,e2e,release,confinement,native-gateway,tray, everypostgresversion andsite. At the end of theplanjob,scripts/ci_lanes.pypicks a lane for each leg the run will start and hands the choice to the jobs as thelanesoutput. Each job's name ends in its lane (unit + integration tests · lane 3). - Each leg goes to the open lane with the fewest jobs running or waiting
in it. The script counts them from the names of the jobs of every
active pull request CI run, through the jobs API (the plan job reads it
with
actions: read). A base lane wins a tie, so lent lanes fill only when every base lane is busy. - Base lanes 0 to 29 are always open to pull requests. With CodeQL's
four slots (
codeql-pr-slot-<rust|other>-<even|odd>) and ClusterFuzzLite's two (cflite-pr-slot-<even|odd>), which work the same way, pull requests may always hold 36 runners, so the merge queue keeps at least 24: enough for two groups' fan-outs at build concurrency 2. Actions expressions have no modulo, so a run picks its CodeQL and ClusterFuzzLite slot from whether its run id ends in an even digit. - Lent lanes 30 to 53 are open only while no
merge_groupCI run has a gating job running or waiting (a run whose jobs are listed as "report only" no longer counts once only those are left; the script reads each active merge group's jobs to tell), and only when nothing runs or waits in them, so each normally holds one job. When the queue is idle, pull requests can use all 60 runners. When a group arrives, no new job enters a lent lane, and those lanes are back to the queue within one pull request job. There is no lock, so two plans that read the jobs API within the same few seconds, before either run's jobs exist, can both use the same empty lane; that lane then empties within two jobs. The queue itself never waits in a lane, so it uses every runner pull requests don't hold. - When the API can't be read, or a job's lane is missing, the jobs use the base lanes only (lane 0 for a missing one), which is the most pull requests could ever hold before lanes were lent.
- A waiting job holds no runner. When a newer push supersedes a run, the workflow-level group cancels the old run, and its waiting jobs leave their lanes with it.
- Any other event (
merge_group, amainpush, a schedule) gets a group of its own from the run id and matrix leg, so it never waits.
scripts/tests/test_ci_contract.py fails if a new job runs on pull
requests without a lane or slot, if the base lanes and slots grow past 36,
or if every lane and slot together pass 60. scripts/tests/test_ci_lanes.py
tests the lane choice and fuzzes the API readers (see
Fuzzing).
Nothing is skipped: a pull request still runs every job it ran before, only later when the lanes are busy.
The organization moved from GitHub Free (20 runners: 9 base lanes, 8 lent, 2 CodeQL slots, 1 ClusterFuzzLite slot, so the queue kept 8) to GitHub Team (60 runners) on 20 November 2026. macOS stays capped at 5 runners at once on both plans.
Lanes replaced one fixed slot per job (ci-pr-slot-<job>) in
November 2026. Measured from 13:00 to 23:40 UTC on 11 November, only
5.7 of the 20 runners were busy on average: every pull request's
unit + integration tests waited behind one test slot, up to 3 hours,
while the other slots sat empty and runners idled for 1,686 runner-minutes
with pull request jobs waiting. In a replay of that day's jobs, the mean wait
for a pull request job drops from about 29 minutes to about 4.
main only advances through the merge queue, and the queue's run already
tested that exact commit. So the push run resolves with --build-only
(build_only: true in the artifact):
- Only the jobs that save caches run:
check,test,native-gatewayandrelease(BUILD_ONLY_JOBSinscripts/ci_plan/execute.py), plus the macOS job, which runs thenative-gatewayleg. The gate expects every other job to be skipped. The Windowsnative-gatewayleg also compiles the Windows cluster tests, whose own job saves no cache. The release job runs a fat and a thin leg so both of its caches stay warm. Each leg also compiles the envelope seed (and runs nothing), because the seed's dependencies are built with their own profile (see Small-VPS envelope). - Builds, Clippy and installs run as usual.
cargo testand the nextest run becomecargo test … --no-run. Formatting, scripts, staging and runtime assertions are dropped. - Doctests and the API docs (
generate-api-docs.sh) run whole. Both build their dependencies with--all-features, a feature set no other command uses, and Cargo can't only compile them. Without them in thelintcache, thecheckjob compiled about 1,500 dependencies for them on every queue run (about 10 of its 18 minutes).
Caches are saved only on main, because merge-group refs can't share
theirs. So this run is what keeps pull-request and queue runs warm.
A dependency cache key covers Cargo.lock, the manifests' dependencies,
the toolchain and the CI environment, not the repository's own sources.
It doesn't cover the manifests' [profile] tables, so the CI environment
carries their digest (see Caches). An existing key
is never saved again, so most merges leave every key unchanged and their
build would save nothing.
To skip that wasted build, each build-only job first looks up its own key
(lookup-only, run only when CI_BUILD_ONLY is true):
- On a hit it sets
CI_SAVED_CACHE_CURRENT=true, skips restoring the cache, andci-exec.py runissues no commands. - The Linux
native-gatewaychild always skips, because thetestjob saves the Linuxdebugcache. - A changed lockfile or toolchain, or an evicted cache, misses the lookup and builds as before.
A miss on Windows compiles the gateway smoke and the cluster tests from
scratch, which takes about 45 minutes, so on a main push the
native-gateway job's time limit is 75 minutes instead of 45.
ci-exec.py ignores the variable outside build-only runs, so it can never
skip a test.
rust_test runs under cargo-nextest, installed pinned
in the test job. Tests from every binary share one parallel pool, rather
than cargo test running binaries one after another, and each test runs in
its own process.
- Merge groups retry once. A failed test is retried once
(
MERGE_GROUP_TEST_RETRIES). nextest marks a pass on retryFLAKYin its summary, and a test that fails twice fails the job. File an issue for everyFLAKYline. - Pull requests never retry, so a flaky test still shows up red where it can be fixed.
A whole-workspace run splits into three parts (TEST_PARTS in
scripts/ci_plan/execute.py), one test job leg each:
- The plan's
test_partsoutput lists the parts, and each leg getsCI_TEST_PART=<part>/<parts>. Every part restores thedebugcache, compiles every test binary and runs its share of the tests with nextest's--partition count:<part>/<parts>. Only the first part also runs the custom harnesses, the feature runs and the bench and example builds. - Each part writes its coverage report, JUnit results and Web API
contract files under
part-<n>/(rust/part-2/lcov.info), in an artifact of its own (coverage-rust-<n>). The coverage gate reads the parts together: a line counts as hit when any part ran it. - A selective run of a few packages and the build-only
mainpush stay in one leg: their tests are too few to repay two more compiles.
On 12 November 2026 the nextest run took 21 of the job's 32 to 36 minutes, with 4,148 tests on the runner's four cores. Split three ways, each part compiles in about 4.5 minutes and runs about 7, so the job takes about 14 minutes for about 8 more runner-minutes. The slowest single test takes about 2.5 minutes, so the parts stay even.
A merge group only needs to know whether it can land, so it fails fast.
nextest stops at the first test that fails its retry. The
merge-queue-fail-fast job cancels the rest of the run as soon as any job
fails, so CI Gate reports at once and the queue ejects the group without
waiting for the slowest job. It also cancels the run of a group the queue
has dropped (see Superseded merge groups). Its
310-minute timeout outlasts the merge queue's 300-minute status check
timeout, so the job keeps watching for as long as the queue waits for a
group.
That job runs no repository code and is the only one with
actions: write. Pull requests run every selected test and job to the
end (nextest's --no-fail-fast, and no cancelling job), so one round shows
every failure among them.
A full-suite coverage build, plus the stale dependencies a partial cache
restore brings, once ran the job out of space on runners with about 20 GiB
free. So when the runner has less than 40 GiB free (CI_MIN_FREE_GIB), the
job first removes the preinstalled SDKs it never uses (Android, .NET, GHC
and the CodeQL bundle). Current runner images have about 90 GiB free, so it
normally skips that: deleting those hundreds of thousands of files took 2 to
5 minutes of every test part.
Test targets with harness = false whose main doesn't answer libtest's
--list protocol would list zero tests under nextest. So they are excluded
from the nextest run and run with cargo test --test
(CUSTOM_HARNESS_TESTS). scripts/tests/test_ci_contract.py fails when a
new one isn't listed.
Locally, cargo test --workspace still runs everything. To reproduce
ci-exec.py run rust_test, install nextest:
cargo install cargo-nextest --locked
Rust dependency caches use three shared keys, plus one for the full
suite on Windows and macOS. Each is saved by one job on main and restored by the others:
lint: saved bycheck.debug: saved byteston Linux (its first part), bynative-gatewayon Windows and by the macOS job on macOS. Restored bye2e,confinement,tray,desktop,postgresandwindows-cluster.release: saved byrelease(the job that also runs the envelope), once per LTO mode (see Release build).debug-tests: saved by the first leg oftest-oson Windows and on macOS (see Full suite on Windows and macOS). Every test binary of the workspace is far more than the other jobs on those systems compile, and saving it into their key would make every one of them restore it. It is keyed byCI_BUILD_PLAN_DEBUG, since it compiles adebugcheck.
Fewer entries keep the repository under the Actions cache limit, so jobs
stop missing their cache. GitHub Team lets the repository raise that limit
above the included 10 GB (Settings > Actions > General > Cache settings),
billed for the storage above it; the limit is about 25 GB since
20 November 2026, so the debug-tests caches fit beside the others.
rust-cache also hashes every installed Rust toolchain into the key, and
runner images preinstall their own (a Windows image update in November 2026
dropped one). So every job that caches runs tool_versions.py prune-rust
right after installing the pinned toolchain. It uninstalls every other
toolchain, so a job saves and looks up the same key whichever image version
it lands on.
rust-cache hashes the manifests' dependencies but not their [profile]
tables. A profile change would otherwise keep the old key: every job would
restore dependencies built under the old profile, rebuild all of them, and
never save the result, because the key already exists. The workflow's
CI_CARGO_PROFILES variable holds a digest of the workspace Cargo.toml
profile tables, and every cache step lists it in env-vars, so a profile
change starts a new key that the next main push saves. The workflow
contract test fails until the digest matches Cargo.toml and prints the new
value.
The same gap opens when a change makes a build-only job compile something
new without touching Cargo.lock or a manifest: a new command, a new
feature set, or a profile override on one command. The build-only push would
find its key saved and skip the build, so queue runs would compile the new
dependencies from scratch until the next lockfile change. So each cache
also carries a digest of its build-only builds: CI_BUILD_PLAN_LINT,
CI_BUILD_PLAN_DEBUG and CI_BUILD_PLAN_RELEASE
(build_plan_digests in scripts/ci_plan/execute.py). Each covers every
OS's build-only commands, with their Cargo and Rust environment, in the jobs
that save that cache, plus the contents of generate-api-docs.sh. Each cache
step lists only its own, so a change re-keys only the caches whose builds
changed. The contract test fails until the values match and prints the new
ones. Until the next main push saves the new key, runs build that cache
cold.
The digest doesn't see inside a cargo alias (cargo build-app and
cargo install-platform run bookclerk-dev). A change to what those
compile should also change a digest, for example through the command line
the planner passes them.
The test job instruments its build for coverage and still
uses the debug key: the instrumentation applies only to workspace crates,
which no cache holds, so the cached dependencies match. The build-only
main push that saves the key runs uninstrumented.
The workspace Cargo.toml builds every dependency at opt-level = 1 in the
dev and test profiles, for local builds and CI alike. Workspace crates stay
at opt-level = 0, so they rebuild quickly, debug fully and carry the
coverage instrumentation. fdk-aac-sys is the one exception: GCC refuses
libfdk-aac's forced inlines at -O1, so it stays at opt-level = 0. Measured on the bookclerk-library unit tests
(4 cores), the suite ran in 432 s instead of 573 s, for a cold dependency
build 67 s longer; opt-level = 2 gained only 8 s more for another 100 s of
cold build. The cost is paid once per cache key, since the caches hold the
built dependencies; changing a profile updates CI_CARGO_PROFILES (see
Caches).
CI builds the dev and test profiles with
CARGO_PROFILE_DEV_DEBUG=line-tables-only. That keeps file and line in
panics while cutting link time and cache size. A failing test prints its
panic backtrace (RUST_BACKTRACE=1), and RUST_LIB_BACKTRACE=0 keeps
error libraries such as anyhow from capturing one for every error they
build.
- The
planjob runsscripts/ci-exec.py resolve. It writesci-plan.json(the schema version, run id, a freshexecution_id, the checked-out commit, the prediction and the resolvedexecution) and the job outputs thatif:gates use, both from the same JSON. - Every downstream job downloads that artifact and runs
ci-exec.py validatefirst. A missing, stale or foreign artifact fails the job. - Each check is one workflow step running
ci-exec.py run <check>. The executor runs the check's prerequisites once per execution, in order (builds, pinnedworkerd, platform install, guest staging), then its commands. CI Gatecompares every job's result withexecution.jobs. Expected jobs must succeed and unexpected jobs must be skipped; failures, cancellations, unexpected skips and unexpected runs all fail the gate. Because the gate uses the execution, not the prediction, shadow mode's intentional full runs pass.
Local reproduction uses the same entry points:
python3 scripts/ci-plan.py --paths crates/bookclerk-cli/src/main.rs --format summary
python3 scripts/ci-exec.py resolve --selective 1 --paths crates/bookclerk-cli/src/main.rs
python3 scripts/ci-exec.py show rust_test --run-id local # print commands
python3 scripts/ci-exec.py run rust_test --run-id local # run them
- Resolving again mints a new
execution_idand starts a fresh execution. Prerequisite completion from the previous artifact isn't reused, even in the same temporary directory with the same commit and run id. - Checks that share one artifact still run each prerequisite only once.
showprints commands and doesn't record completion.prereqs <check>runs only the check's prerequisites. With--github-envit also exports the environment the check's commands run with (PATHwith the built helpers first,BOOKCLERK_WORKERD_BIN), for a job that runs the tests its own way, such as mutation testing.
scripts/tests/test_ci_contract.py (PyYAML, hash-pinned in
scripts/tests/requirements.txt) fails when the workflow and the planner
could diverge:
- unknown or unused outputs;
- a check with zero or two steps;
- a job the gate doesn't see, or one that doesn't validate first;
- unpinned actions, or
continue-on-error.
python3 scripts/ci-plan.py --base <sha> --head <sha> --format summary
python3 scripts/tests/test_ci_plan.py # planner + executor scenarios
python3 scripts/tests/test_ci_relations.py # relations.toml guardrails
python3 scripts/tests/test_ci_contract.py # workflow contract (needs PyYAML)
Each changed path is classified (a rename contributes its old and new paths):
| Input | Effect |
|---|---|
Package source (src/, build.rs, Cargo.toml, other package files) |
The package is compiled-affected (see below) |
tests/**, benches/**, examples/** in a package |
The owner only, by target type |
| A dev edge to a compiled package | The consumer's tests, bench and example builds and doctests |
A declared embed (such as the TS or Python SDK runtime in bookclerk-workerd) |
The package is compiled-affected |
A declared test_input |
The consumer's tests |
A declared fixture |
The checks declared for that fixture |
A declared check_input (a workspace executable a non-Cargo check launches, such as the bookclerk-db-testkit kit the SDK test wrappers spawn) |
The checks declared for it, on top of the owner's own selection |
A declared test_guests executable |
The launching package's tests re-run, with no propagation |
ui/, packages/plugin-sdk*/, ABI scripts |
The UI, SDK ABI and API-doc checks |
A first-party workerd guest without a Cargo package (crates/bookclerk-plugins/<tier>/<name>/) |
Its type-check, build and tests (workerd_guests) and its staged E2E. packages/plugin-sdk/ also selects workerd_guests, because the guests type-check against it, and so does crates/bookclerk-db-testkit/, because a guest with a plugin database tests it under every profile through the kit |
A reference example that declares [[databases]] and has a test script in package.json, a pyproject.toml or a Cargo.toml (examples/plugins-<name>/) |
Its checks and suite under every database profile (plugin_examples) and its staged E2E. crates/bookclerk-db-testkit/ and crates/bookclerk-plugin-tools/ also select plugin_examples, as do packages/plugin-sdk/ (with a TypeScript example), packages/plugin-sdk-python/ (with a Python one) and anything that affects a Rust example's tests, such as the Rust SDK |
docs/** |
The website tests (site_test) |
tools/site/**, assets/brand/** |
The website tests and audit (site_test, site_audit) |
tools/diagnostics-collector/** |
The collector's tests (collector_test) |
tools/demo/** and the files the demo copies |
The hosted demo Worker's tests (demo_test) |
| Markdown, TypeScript, JavaScript, JSON, YAML or CSS anywhere (not lockfiles) | Formatting and Markdown lint (lint), on top of whatever else the file selects |
.prettierrc.json, .prettierignore, .markdownlint-cli2.jsonc, tools/lint/** |
Only lint |
Root Cargo.toml or Cargo.lock, the toolchain, .cargo/, workflows, the planner, fuzz/Cargo.*, unknown paths, planner errors |
Full suite |
fuzz/corpus/<target>/**, fuzz/oracles/<target>.rs |
The owning crate's tests |
fuzz/fuzz_targets/**, fuzz/dictionaries/** |
Nothing here |
packaging/docker/**, .dockerignore |
Nothing here |
.devcontainer/**, scripts/dev-env/** |
Full suite (unclassified) |
packaging/demo/** |
Nothing here |
Some rows need more detail:
- Compiled-affected packages propagate over normal and build edges to their compiled consumers. Optional edges count, because there is no feature resolution.
- Owner-only paths run
--tests,cargo bench --benches --no-runorcargo build --examples, by target type. - Dev edges are not propagated to the consumer's production consumers.
- Fixtures skip the owner's compile propagation.
docs/**runs the site tests because their link check covers the docs the site publishes.- The demo copies are
crates/bookclerkd/openapi.json, the website's footer and fonts, and the app's mark and icons inui/public/(bookclerk-mark.svg,favicon.ico,favicon-32x32.png,apple-touch-icon.png). The demo's route table must cover every operation, and the copies must match. - Fuzz corpora and oracles are declared
test_input. An undeclared corpus or oracle is unclassified. - "Nothing here" paths are covered elsewhere. The ClusterFuzzLite workflows and the Docker image workflow select themselves by path, and the demo image is built by the demo deploy.
- Dev Container paths also select the Dev Container image workflow by path.
A planner error still uses the normal full plan when metadata, relations
and the guest inventory are available; an unavailable diff is the usual
case. If those inputs are missing, ci-exec.py resolve fails instead of
publishing a plan with empty prerequisites.
Cargo.lock stays a full-suite trigger for now. Precision for dependency
updates can follow.
Every selected check lists what a clean runner needs, with a reason:
rust_test:cargo build -pfor executables its packages' tests launch (test_guests), plus pinnedworkerdwhen they spawn through it. Tests run withBOOKCLERK_REQUIRE_TEST_GUESTS=1, so a missing guest fails instead of skipping.e2e: the jail, the launcher, the native guests in scope, pinnedworkerd,cargo install-platform --skip-build, andcargo stage-plugins --plugin <id>…(or--optional --examplesfor the full installation). It runsbookclerk-plugin-e2e(--lib --test staged_plugins --test installed_plugin_path --test cold_start --test workerd_isolate_egress; nevernative_gateway) withBOOKCLERK_REQUIRE_STAGED_PLUGINS=1. Thecold_startmeasurement is ignored, so the check only keeps it compiling.postgresrpc_like: the postgres guest, the launcher, the jail and pinnedworkerd.native_gateway:cargo build -p bookclerk-jail -p bookclerk-workerdand pinnedworkerdonly, with no platform install, staging, database services or app build. The fixture guest is a bin target ofbookclerk-plugin-e2e, socargo test --test native_gatewaybuilds it.windows_cluster:cargo build -p bookclerk-plugin-database-sqlite -p bookclerk-jail -p bookclerk-workerdand pinnedworkerd, then the existing identity filters andcargo test -p bookclerk-cli --test library_secret_startup. The CLI target still setsplugins.isolation = "required"and keeps its assertions, andBOOKCLERK_REQUIRE_TEST_GUESTS=1is set for it.
Prerequisites never select suites. Building the sqlite guest for host tests doesn't select sqlite's own tests or E2E.
bookclerk-plugin-e2e owns the staged-installation suite. The ordinary
test step excludes it (--exclude bookclerk-plugin-e2e on the full suite).
- Full. A direct source change in a
staged_e2e.full_packagespackage (dev, manifest, abi, workerd, jail, sandbox, the e2e crate), an inventory manifest (crates/bookclerk-plugins/*/*/plugin.toml,examples/plugins-*/plugin.toml), host discovery, registry or install preflight, or the full suite. - Subsets. Each compiled-affected guest package adds its id. Changed
non-Cargo guests (examples and first-party workerd guests) add theirs. The workerd SDK runtime embed adds
every workerd-runtime guest. Shared host runtime changes add the
runtime_smokeguests (native Rust, workerd TS, workerd Python). - Combination. Full wins over subsets, subsets are unioned, and
prerequisites add nothing. Platform guests (
sqlite,local,embedder) are always installed and checked.
An embedded SDK change marks bookclerk-workerd affected through the
embed, so it selects the workerd-runtime subset, not the full
installation. A change under crates/bookclerk-workerd/src/ is a direct
source change and selects full.
This file is the only handwritten graph. It holds the relationships
cargo metadata can't describe, and doesn't restate the workspace
dependency graph:
- embedded assets;
- cross-package test inputs and fixtures;
- executables that tests launch;
- feature-specific test runs and postgres step owners;
- E2E scope rules and platform-job membership;
- release packaging inputs.
scripts/tests/test_ci_relations.py is a guardrail, not a completeness
guarantee. It recognizes literal include_str! / include_bytes! paths,
Rust string literals that resolve to another package or repository path,
and workspace binary names in tests/. Each must be declared, related
through Cargo, under a full-suite path, or listed in [[reviewed]] with a
reason.
Warning
Computed paths, macro-generated includes and dynamically discovered fixtures are not detected. Declare them, or rely on the conservative fallbacks.
Postgres tests only run in the postgres check, through the name filters
of POSTGRES_STEPS in scripts/ci_plan/execute.py.
scripts/tests/test_ci_contract.py finds every Postgres-gated test: an
#[ignore] reason naming Postgres, or a body that reads
BOOKCLERK_TEST_POSTGRES_URL directly or through a helper in the same
package. It fails when no step filter matches the test's function name, or
when a step matches no test. Name such tests postgres_*.
| Job | Runs when | Checks |
|---|---|---|
plan |
Always | Planner, executor, contract and guardrail tests; DB isolation and docs lints; resolve; the ci-plan artifact |
fmt / clippy / docs |
Any of its checks is selected | lint, ui, plugin_sdk_abi, python_sdk, workerd_guests, plugin_examples, author_surface, fmt, clippy, clippy_publish, api_docs, doctest, store_free |
unit + integration tests |
rust_test is selected |
nextest over the affected packages (the workspace on the full suite), custom-harness targets, feature test runs, bench and example builds |
staged plugin installation (e2e) |
e2e is selected |
Builds, installs and stages the guests in scope, then the bookclerk-plugin-e2e staged targets |
release build + small-VPS envelope (<lto> LTO) |
A shipped binary (hosts, helpers, platform guests) is compiled-affected (release), or an [envelope].paths input changed, an [envelope].packages member is compiled-affected, or the full suite (envelope) |
Each selected check as its own step; see Release build and Small-VPS envelope |
sandbox + jailed tiers (Linux; Windows in the queue) |
sandbox, jail, media or media-worker is affected | Clippy and enforcement tests (Windows --test-threads=1) |
native-behind-workerd gateway (Linux; Windows in the queue) |
See the gateway job | The four native_gateway targets, plus per-OS extras |
tray (Linux; Windows in the queue) |
bookclerk-tray is affected |
Clippy and tests |
desktop window (Linux) |
A [desktop].paths input changed, or the full suite |
Clippy on bookclerk-desktop --features window; see Desktop job |
macOS (sandbox, gateway, tray) |
Any of the three jobs above runs, outside a pull request | The macOS run of each of them, one step each; see macOS job |
postgres 16/17/18 |
An owning package's unit tests are affected | Only the owners' steps, on every supported major |
windows cluster startup + identity |
A [windows_cluster].paths input changed, or the full suite; never on a pull request |
See Windows cluster |
website + collector Workers |
See Workers | site_test, site_audit, collector_test, demo_test; no Rust toolchain |
UI visual and accessibility |
A file under ui/, tools/site/, tools/diagnostics-collector/ or assets/brand/ changed, SECURITY.md changed, or the full suite |
ui_visual in the pinned Playwright image; see UI visual job |
CI Gate |
Always | The stable required check (see the contract) |
The postgres owners are the library, the db guest, the postgres guest and
the plugin host's RPC LIKE.
The root .tool-versions file is the one pin for
every tool version that crosses a boundary between workflows, images and
scripts:
| Pin | Read by | Bumped by |
|---|---|---|
nodejs (a major) |
Every actions/setup-node step (node-version-file); the node images' tags |
Hand, with the node image tags |
uv |
Every astral-sh/setup-uv step (version-file); the Dev Container's ghcr.io/astral-sh/uv image |
tool-versions-bump.yml |
cargo-nextest |
The unit + integration tests job (through scripts/tool_versions.py github-output); scripts/dev-env/provision.sh |
tool-versions-bump.yml |
rust |
rust-toolchain.toml (channel, which repeats the pin because rustup can't read .tool-versions); every dtolnay/rust-toolchain step (through a Read tool pins step before it, installing the toolchain file's rustfmt and clippy so rustup doesn't add them mid-job); scripts/dev-env/provision.sh |
tool-versions-bump.yml |
scripts/tests/test_dev_env.py (in the plan job) fails when a workflow
sets one of these versions itself, when a Dockerfile's node or uv image
disagrees with the file, or when rust-toolchain.toml names another Rust
release. A change to .tool-versions runs the full suite and
rebuilds the Dev Container image.
scripts/tool_versions.py parses the file strictly: exactly the pins above,
in that order, one <tool> <version> per line, no comments.
actions/setup-node reads the first line that looks like a version, so
nodejs stays first.
The daily tool-versions-bump.yml workflow takes the newest release of uv,
cargo-nextest and Rust (the rust-lang/rust GitHub releases) that is at
least 7 days old (the same cooldown as Dependabot and the workerd pin). For
uv it also resolves the image digest and rewrites every Dockerfile that
builds from the image. For Rust it also rewrites rust-toolchain.toml. It
opens one pull request for every tool due that day, on the
deps/tool-versions branch, with the
automation App, and turns on auto-merge. Dependabot ignores the uv image and Node.js majors, so
the two never race. Its ignore names the uv image astral-sh/uv, without the
ghcr.io host, because that is the name Dependabot matches ignores against.
A new Rust release can bring new Clippy lints and Cargo behavior, so it
arrives as that pull request, and CI shows anything it breaks there, instead
of reaching every pull request and merge group the day it is released.
Pinned nightlies (docs-publish.yml, ClusterFuzzLite) keep their own pins.
Note
The workerd binary keeps its own pin in
crates/bookclerk-workerd/workerd-pin.json. Base image
digests shared by several Dockerfiles stay with Dependabot's grouped
updates; test_dev_env.py checks they match.
The lint check needs no Rust toolchain. It installs tools/lint (pinned
Prettier and markdownlint-cli2) and runs both from the repository root:
- Prettier checks TypeScript, JavaScript, JSON, YAML and CSS against
.prettierrc.json..prettierignoreleaves out Markdown, HTML (page templates whose whitespace reaches the output), lockfiles, generated files, byte-identical mirrors, fuzz or test corpora, and the build output that nested.gitignorefiles hide. - markdownlint checks every Markdown file against
.markdownlint-cli2.jsonc, which relaxes the rules that fight GitHub-first docs (see Writing docs).
Each language's own linter runs in its own check: oxlint in ui, the
TypeScript SDK and the three Workers, and ruff check plus
ruff format --check in python_sdk.
The plugin_sdk_abi and python_sdk checks build the bookclerk-db-testkit
executable first (cargo build -p bookclerk-db-testkit) and name it to the
SDK tests as BOOKCLERK_DB_TESTKIT: the SDKs' createTestDatabase wrappers
spawn it (see Working within database limits),
so both checks need the Rust toolchain.
The plugin_examples check builds the kit and the bookclerk-plugin
author CLI, then runs bookclerk-plugin test <dir> on each database
example, which runs the example's own suite once per profile
(first-party, d1, minimum; see
The suite under every profile).
Before that, a TypeScript example is type-checked against the built
TypeScript SDK and its modules/ rebuilt (a committed tree that differs is
drift, as in workerd_guests). A Python example is checked with ruff check and ruff format --check, then tested with pytest in the Python
SDK's locked environment (the one python_sdk uses) with the SDK imported
from its sources. A Rust example is a workspace package that fmt and
clippy already cover, so it only runs its cargo test suite.
To fix formatting locally:
$ npm ci --prefix tools/lint
$ node tools/lint/node_modules/prettier/bin/prettier.cjs --write .
$ node tools/lint/node_modules/markdownlint-cli2/markdownlint-cli2-bin.mjs --fix
The workflow sets RUSTFLAGS="-D warnings", which turns every rustc and
Clippy warning into an error. It does not reach the warnings Cargo itself
emits about the manifests, such as:
unused_dependencies: a crate lists a dependency its code never uses.unused_workspace_dependencies: an entry in[workspace.dependencies]that no member inherits.unused_workspace_package_fields: a[workspace.package]field no member inherits.
The clippy check therefore also sets CARGO_BUILD_WARNINGS=deny
(Cargo's build.warnings), so any Cargo warning fails it too. To see the
same result locally:
$ CARGO_BUILD_WARNINGS=deny cargo clippy --workspace --all-targets -- -D warnings
The clippy check runs on Linux. Cargo builds a dependency declared under
[target.'cfg(windows)'.dependencies] (or another platform's table) only
for that platform, so the check cannot tell whether such an entry is used;
review platform-only entries by hand. Fix a reported dependency by removing
it from the manifest.
The release job runs at one of two scopes:
affected:cargo build --release -p <affected shipped>.full:build-app --release --platformplus helper layout assertions. It runs when a packaging input changes (bookclerk-dev, the workerd pins, the platform manifests) or on the full suite.
The full scope builds the media worker with its libav feature, as
release archives and the Docker image do, so it also compiles the pinned
FFmpeg decoders (bookclerk-libav; about half a minute on 4 cores). The
affected scope leaves the feature off. Test builds turn it on through
bookclerk-media's dev-dependencies, so the test jobs cover the FFmpeg
path on every OS. Windows jobs that compile bookclerk-media install
MSYS2's make first (FFmpeg's configure runs under the runner's MSYS2
with MSVC; see media.md).
The release profile ships fat LTO with one codegen unit, which links on a
single core. On pull requests the job sets
CARGO_PROFILE_RELEASE_LTO=thin, which links in parallel and still checks
that the release build compiles. The merge queue and OSV remediation runs
keep fat LTO, so the build and the envelope that gate main use the
binaries we ship.
Cargo recompiles dependencies when the LTO mode changes, and the cache key
covers every CARGO* variable, so each mode has its own release cache.
The job is a matrix over the mode: one thin leg on a pull request, one
fat leg in the merge queue, and both on the build-only main push, which
seeds the two caches. Each leg skips its build there when its cache is
already current. On a 4-core machine a warm thin build of bookclerkd took
about 10 minutes against 17 for fat; a thin build without its own cache
gains nothing.
The job runs when any of these hold:
- a
[native_gateway].packagesmember (plugin-host, workerd, plugin-sdk, sandbox, jail, the e2e crate) is in the Cargo-compiled closure; - a
[native_gateway].pathssmoke input changed; - it is the full suite.
On every OS (on macOS, as a step of the macOS job) it runs
cargo test -p bookclerk-plugin-e2e --test native_gateway --test native_gateway_isolation --test native_gateway_lifecycle --test native_gateway_authenticated_endpoint
(see the smoke below).
- Windows first adds Clippy on workerd, the SDK
httpfeature, the host--lib, the workerd lib tests (inheritedhandle:SOCKET_PROXY), and the lifecycle churn test alone. - macOS adds the workerd lib tests.
On windows-latest only, the job runs identity create, restart and
binding, and different-library rejection. Then it runs the full
library_secret_startup target with the sqlite guest, the jail,
bookclerk-workerd and pinned workerd.
Its job saves no cache. Its bookclerk-cli and bookclerk-library test
builds unify dependency features differently from every other Windows
command, so on a build-only main push the Windows native-gateway leg
also compiles them (BUILD_ONLY_HOSTED_CHECKS in
scripts/ci_plan/execute.py), and the debug cache it saves holds their
dependencies. Before that, the job compiled about 250 dependencies, about
13 of its 17 minutes, on every queue run.
The desktop window's Tauri code (bookclerk-desktop's window feature)
only builds with the WebKitGTK development files, so every other job builds
the crate without it. This job, on ubuntu-latest, installs those files
and capnproto, then runs Clippy on bookclerk-desktop --features window --all-targets. Clippy type-checks the window code without linking it.
A pull request runs it when a [desktop].paths input in
scripts/ci_plan/relations.toml changes: the window crate, its request
policy (bookclerk-window-policy), or the embedded host it drives
(bookclerkd's embedded.rs, lib.rs and Cargo.toml). It doesn't select
on every package the window links, which is most of the host. The full
suite, in the merge queue and on main, always runs it, which catches a
host change that breaks the window. Windows and macOS don't run it yet.
The web app's Storybook stories and the pages of the website and the
diagnostics collector, each checked with axe-core and compared with a
baseline screenshot at two viewports in light and dark. The job runs inside
the Playwright image pinned by digest, so every run renders with the same
browser and fonts, and needs no Rust toolchain (the image already has the
Python the plan scripts need). It builds Storybook and the website, runs the
checks and uploads the ui-visual artifact.
A change under ui/, tools/site/, tools/diagnostics-collector/ or
assets/brand/, or to SECURITY.md, selects it; a change to the host that
serves ui/dist or to docs/ doesn't. Baselines live in a separate repository, named by
ui/visual/baselines.lock; the job records a set every run and compares
with the locked one when there is one. See
Visual and accessibility testing.
The check makes sure Bookclerk fits a small VPS. Its paths are the harness, the seed, the search rebuilds, the works page index and the works HTTP handlers. It runs in the release job, after the release build step when both are selected, so the hosts are compiled once per run.
- On the
ubuntu-latestVM,install-platform --releasebuilds what the release step did not (all of it when only the envelope is selected), and the seed is compiled. The seed is a load generator, not a shipped binary, so it's compiled without LTO and with 16 codegen units (ENVELOPE_SEED_PROFILEinscripts/ci_plan/execute.py): about 98 s instead of 173 s after a source change, measured locally. Its dependencies are therefore separate artifacts from the release build's, and only stay cheap because the build-onlymainpush compiles them into both release caches. Without that, they compile from scratch and the job gets slower, not faster. docker run --memory=1g --memory-swap=1g --cpus=1runs the seed (10,000 titles written 200 at a time, the way a scan page writes them, then the works page index built over them), abookclerk library search --rebuild-indexthat builds the works search index, the daemon and the load in an Ubuntu container of the runner's own release (ubuntu:<VERSION_ID>from/etc/os-release, orubuntu:24.04on a non-Ubuntu host), because the hosts link the runner's glibc and libssl.- On failure, it uploads
bookclerkd-envelope.logwith the metrics.
small-vps.sh reads memory.max and cpu.max, and refuses to record
numbers when they are max or looser than 1 GiB and one core. It drops the
build and object file cache, and writes envelope-metrics.json even when a
later step fails.
What is gated:
- The works routes must return 200 with at least one work each: the first
page by title and by author, the
statusandaccountfilters, a page 8,000 works in (by followingnext_cursor), andq=Title. The indexes are built before the daemon starts, as a library that has been running has them, so the daemon queues no rebuild of its own; before the routes are timed, the harness still waits up to 900 s for any index rebuild job to finish and for both indexes to answer. - Daemon anonymous memory stays under 256 MiB after idle and during the API
windows. That is
RssAnonwhen/proc/<pid>/statusreports it, otherwisememory.statanon, ormemory.currentwhen anon is absent.
What is only recorded:
memory.peak. The container can't reset it, so the lifetime peak includes the seed.- Latency.
available_parallelism(the daemon'slogical_cpus), separately from the affinity mask.
One cpu.max core means one media worker.
The job's checks are selected by path, with no Rust toolchain:
site_test:npm run typecheck(stricttsc --noEmit),npm run lint(oxlint) andnpm testintools/site, covering the generator, the Worker and the docs link check. Selected bytools/site/**,assets/brand/**ordocs/**.site_audit:tscon the audit, then Lighthouse and axe-core on the built site (see Website checks). Selected bytools/site/**orassets/brand/**.collector_test:npm run typecheck(stricttsc --noEmit),npm run lint(oxlint) andnpm testintools/diagnostics-collector. Selected by its folder.demo_test:npm run typecheck(stricttsc --noEmit),npm run lint(oxlint) andnpm testintools/demo. Selected by its folder.
The full suite runs all four.
npm test is Vitest in all three Workers, configured in each one's
vitest.config.ts with two projects:
workersruns the Worker's tests inside the Workers runtime through@cloudflare/vitest-plugin, on the compatibility date inwrangler.toml. The collector and the demo test against real D1 databases with theirmigrations/applied. The site's Worker tests serve a real build, which a global setup makes in Node first.noderuns what needs the host: the site's build and docs-tree tests, the demo's page-script tests and the collector's Python-oracle fuzz test.
npm run coverage adds V8 coverage from both projects (text-summary and
lcov under coverage/). CI runs it in place of npm test to feed the
coverage report.
These run outside CI Gate:
- CodeQL (see CodeQL below) feeds the repository's code scanning merge rule.
- OSV scanning is a separate workflow and gate. A new push cancels a
pull request's scan still in progress. A green OSV job that
applies
osv-scanner.tomlIgnoredVulnsmeans zero unignored, actionable findings, not that ignored advisories have left the dependency graph. Each ignore must record reachability evidence and anignoreUntilreview date. The scanner reads the config beside each lockfile, sofuzz/osv-scanner.tomlrepeats, with the same expiry, the root exceptions that the fuzz lockfile shares. - SQL-v1 property (
.github/workflows/sql-v1-proptest.yml) runs a longerproptestsuite (PROPTEST_CASES=256) onbookclerk-plugin-abiandbookclerk-db-exec: grammar-aware sqlite admission, plus a PostgreSQL 16 differential (normalizedDbValue/DbErrorClass). It runs onworkflow_dispatchand weekly. Pull-request CI keeps the modest in-crate case counts, deterministic corpus replay, and the postgres matrix (binding UTF8 readiness and the sqlite/postgres differential). - Docker image (
.github/workflows/docker-image.yml) builds and smoke-tests the release image (see below). - Desktop packages (
.github/workflows/desktop-packages.yml) builds the desktop app's unsigned packages on Linux, Windows and macOS withcargo package-desktop. The images workflow calls it nightly whenmainhas moved, by hand and for a release (see Desktop packages). A pull request runs it only when it changes the packaging, and skips drafts. - Dev Container image (
.github/workflows/devcontainer-image.yml) builds and checks the contributor image (see below). - ABS Integration (
.github/workflows/abs-integration.yml) tests theaudiobookshelfplugin against a real Audiobookshelf server (see below).
.github/workflows/codeql.yml runs
CodeQL's default query suite on GitHub Actions workflows,
JavaScript/TypeScript, Python and Rust, each in build mode none. The
repository's "CodeQL results" merge rule reads its analyses. It skips a
draft pull request and runs when the pull request is
marked ready for review. GitHub's
default setup for code scanning must be off, or the uploads are refused.
Rust takes about 20 minutes on a hosted runner, and CodeQL has no
incremental analysis for Rust yet (tracked in
#585). Rust can't be
left out of a pull request either: the "CodeQL results" merge rule reports
a pull request without /language:rust as neutral, "1 configuration not
found", and that blocks the merge. So a pull request that changes no Rust
input reuses main's Rust results instead:
- The
Choose languagesjob diffs the merge commit against its base (git diff -z --no-renames --name-only). scripts/codeql_languages.pytreats a path as a Rust input when it is a.rsfile, a.capnpor.protoschema that a build script compiles into Rust, a.c,.ccor.hfile (C or C++ a build script compiles into a crate: the FFmpeg shim, the SentencePiece tokenizer), aCargo.toml,Cargo.lock,rust-toolchain.toml, anything under a.cargo/directory at any depth, the workflow, or the scripts that choose and reuse. A name that isn't UTF-8 also counts. A test fails when abuild.rsreads a file the chooser doesn't count, because CodeQL runs build scripts even in build modenone.- Before allowing reuse, it scans the merge commit's tracked
.rsfiles. A path also counts when a Rust source embeds it withinclude_str!,include_bytes!orinclude!(aplugin.toml, the brand logo, the SDK filesbookclerk-workerdinjects, evenci.yml), or when it sits in a crate with abuild.rs. An include whose path it can't resolve turns reuse off for every pull request until the scanner learns that form. With no Rust input, it sets thereuse-rustoutput. Analyze (<language>)runs once per language, always all four.- With
reuse-rustset,Analyze (rust)runsscripts/codeql_reuse.py. It looks for acodeql.ymlrun onmainfor exactly the merge commit's base parent, started by a push, the schedule or a manual run, whoseAnalyze (rust)job succeeded. It downloads that run'scodeql-rust-sarifartifact and checks that it is CodeQL Rust SARIF. The job then uploads it with category/language:rust. - If any of that fails, the job analyzes Rust as usual. That happens, for
example, when
main's run for the base commit is still going.
The reused results match main's analysis of the base commit, so code
scanning reports no new and no fixed Rust alerts. A real analysis would
report the same, because nothing it reads changed. The job matrix stays
the same either way, because code scanning records the matrix values with
each analysis and a pull request has to match main's configuration.
Pushes to main, the Monday schedule and manual runs analyze every
language, and the Rust job keeps its SARIF as the codeql-rust-sarif
artifact for 30 days. The Rust extractor's own cargo output (build
scripts and proc macros) is cached under
codeql-rust-target-<rustc version>-<Cargo.lock hash>: main saves it
and pull requests only restore it.
The file names are the pull request author's, so the chooser is an
untrusted-input boundary, fuzzed in
scripts/tests/test_codeql_languages.py (see
Fuzzing).
The Docker image job builds packaging/docker/Dockerfile and runs
packaging/docker/smoke.sh, which boots the image twice on one fresh
/config volume:
- The first boot must install the bundled
sqlite,localandembedderplugins and load the database guest under the default isolation. - The second must reinstall nothing.
- The image must not override
BOOKCLERK_PLUGIN_ISOLATIONorBOOKCLERK_MEDIA_ISOLATION.
It is a full release build, so it runs weekly, on demand, and on pull
requests that touch the image or its startup install path:
packaging/docker/**, .dockerignore, the workflow, bookclerk-dev, the
platform plugin sources, the bundled-install, trust and ledger modules, the
database loader and the workerd pin.
The ABS Integration job (.github/workflows/abs-integration.yml) runs a
real Audiobookshelf server beside the production image with the
audiobookshelf plugin added
(tests/abs-integration/). The plugin is
built from the checkout, so it does not get the host-implied grant to its
base_url that the crates.io package gets. The test sets it up the way an
operator would:
- It approves the plugin through the consent API with a TCP grant to the server.
- It sets
base_url,library_idand the sealedapi_keywithbookclerk plugins config set, and deploys the plugin. - It checks
integrations testandintegrations status. - It redeems a claim ticket on the portal and signs in to the portal with Audiobookshelf credentials.
- It starts a remote library scan.
Both containers share one network namespace, because the plugin accepts
plain http only for a loopback host. The job runs by hand, on a published
release, and on pull requests that change the test or the plugin. To run it
locally, build the images as its workflow does, then run:
bash tests/abs-integration/smoke.sh
The Dev Container image job builds .devcontainer/Dockerfile and runs the
contributor toolchain inside it as the dev user (cargo, rustfmt, Clippy,
cargo-nextest, capnp, Node.js, npm, uv, PyYAML, git, ssh). It runs
weekly, on demand, and on pull requests that touch .devcontainer/**,
scripts/dev-env/**, scripts/tests/requirements.txt,
rust-toolchain.toml, .tool-versions, .dockerignore or the workflow.
scripts/tests/test_dev_env.py (in the plan job) checks on every run that
the image's tool pins match CI and .tool-versions, and that every Dockerfile's base images are
tag and digest pinned, aligned, and covered by Dependabot.
The smoke tests are crates/bookclerk-plugin-e2e/tests/native_gateway.rs,
plus native_gateway_isolation.rs, native_gateway_lifecycle.rs and
native_gateway_authenticated_endpoint.rs. They:
- Install the test-only
native_gateway_probeguest (an SDK-only bin of the e2e crate, never staged or packaged) into a temporary files folder. - Grant exactly one ephemeral loopback TCP port.
- Spawn it through
PluginSession::spawn_withwithIsolation::Required.
The session goes through host discovery and launch planning, then two
sibling bookclerk-jail processes:
bookclerk-workerdand pinnedworkerdin the gateway jail;- the native backend in a deny-network sibling, joined by inherited duplex
links. On Windows, that is two host-created AppContainers plus a session
Job, with no named-pipe
SOCKET_PROXY.
Against harness-held listeners, the tests assert that:
- describe, open and a real CLI entrypoint RPC answer through the front door;
- a unique payload round-trips through
bookclerk_plugin_sdk::net::connectto the granted port; - a second, live listener on an ungranted port is refused by policy
(
403) and is never dialed; - a direct
std::netconnect from the guest to the live granted listener is blocked; - both
gateway_pid()andguest_pid()exit after drop, the host session folder is gone, and the guest can't read gateway state or theWORKERD_GRANT_*/BOOKCLERK_JAIL_*env; - two concurrent sessions can't exercise each other's grants or gateway state (see Concurrent sessions);
- a guest exit, a missing
workerd, or a mid-session kill fails closed.
After both sessions authenticate and complete a connect, channel_ident
through cli_invoke reports each configured tag. Those sessions don't
publish candidate-endpoint metadata, so extra_status=not-run. That status
isn't proof that the other session's endpoint is absent.
So a paired fixture keeps one pre-authenticated endpoint B, with its own
challenge (A's challenge isn't replayed onto B). It publishes the same
candidate metadata in both cases, through the same cli_invoke path:
- The normal launcher path doesn't preserve the test-only extra fd and
doesn't list B's handles. The RPC reports
extra_status=absent, because that slot is inaccessible. - A live object at the slot that isn't the candidate is
ident-failed(a numeric collision), not absence. - When the fixture also transfers B through the guest jail handoff, the same RPC reports B's tag while A still answers.
- A transfer without the candidate metadata stays
not-run, even though the endpoint was handed off.
On Windows the challenge is written with an event, so that guest end isn't
bound to the test process's I/O completion port. Binding it makes the
guest report ident-failed instead of the tag. Production spawn doesn't
add the slot, and a numeric fd or handle isn't identity.
A missing helper, runtime or confinement backend, a startup failure or a timeout fails the test.
- On Windows,
sequential_and_concurrent_spawn_cycles_do_not_leakruns by itself before the suite. The suite runs it again in the original order, withwindows_job_extras_deny_the_next_direct_ping. - Smoke-only inputs (the four targets,
tests/native_gateway/and the fixture guest source) selectnative_gatewaybut not the staged E2E suite.
CI Gate checks how much of a change its tests run. It fails when the
change's patch coverage is below 80% (patch_target), an area's changed
lines miss that area's own patch target, or an area falls below its floor,
and writes a summary either way. Setting mode = "report" in
scripts/coverage_areas.toml turns those failures back into warnings.
Coverage is measured in the jobs that already run the tests, so it costs no extra test run:
| Area | Measured by | Tool |
|---|---|---|
rust |
test (rust_test) |
cargo-llvm-cov, source-based LLVM coverage |
site, collector, demo |
Workers (site_test, collector_test, demo_test) |
Vitest V8 coverage (npm run coverage) |
python-sdk |
fmt / clippy / docs (python_sdk) |
pytest-cov |
workerd-guests |
fmt / clippy / docs (workerd_guests) |
Node's test runner coverage (node --test --experimental-test-coverage) |
How a run gets there:
- The workflow sets
CI_COVERAGE_DIRon those steps. With it set,scripts/ci-exec.pyruns each check with coverage and writes<dir>/<area>/lcov.info, or<dir>/<area>/part-<n>/lcov.infofor one part of the splittestjob (see Test runner and retries). Unset (locally, by default), the checks run as before. - Each job uploads its reports as a
coverage-*artifact. The plan job uploads the hunk headers ofgit diff --unified=0 base...headasci-patch, because CI Gate checks out a single commit. - After comparing the job results, CI Gate downloads both and runs
scripts/coverage_gate.py. It writes the summary to the job's step summary.
The summary has two figures:
- Patch coverage: of the executable lines the change adds, the share the tests ran. The whole change must reach 80%. Lines in an area this run did not measure, such as Rust on a docs-only change, are listed apart rather than counted.
- Area patch targets: each area's floor rounded down to a multiple of five, between 80% and 95%. An area whose target is above 80% must also reach it on its own changed lines, so well-tested code keeps new code to the same standard. A floor of 98% gives a target of 95%, and one of 93% gives 90%. Targets rise with the floors, so nobody edits them. The goal is 100%, but the gate stops at 95% because it has no reviewed way yet to mark lines no test can run.
- Area floors: each area's total line coverage against its floor in
scripts/coverage_areas.toml. A selective Rust run tests only the affected packages, so therustfloor is checked only whenrust_testran the whole workspace.
Floors ratchet. After every successful merge group run, which measures
every area in full,
coverage-floors.yml reads
that run's coverage and runs scripts/coverage_gate.py --ratchet on main
once main contains the commit that run measured. It waits up to five minutes
for that commit to land and skips the run if it does not. Only code already on
main runs in the job, because the job also holds the automation App token:
- Each floor rises to the area's measured coverage rounded down, less one point, so noise does not fail a run.
- A floor rises by at most five points per run and never above 99.
- A floor never falls. Lowering one stays a reviewed, hand-made change; never lower a floor to make a change pass.
When a floor rises, the workflow opens a ci/coverage-floors pull request
with the automation App and turns on auto-merge, so the merge queue lands
it once its checks pass. Turning on auto-merge gets four attempts, because
GitHub's API sometimes fails a single call. If all four fail, the run fails,
and the next merge to main tries again. While that pull request is open,
later rises update it rather than opening another. Once it is in the merge
queue, GitHub refuses updates to its branch, so a later rise leaves it alone
and says so in the run's summary; a later merge raises the floors further
only if its own coverage allows. Floors move in whole points, so most
merges change nothing and the run ends after a few seconds. To run it by
hand, use Run workflow on the Actions tab; it then reads the latest
successful merge group run, whose coverage artifacts are kept for seven
days.
Generated files are left out of both figures. They are listed under
generated in the areas file, because tests exercise the generator rather
than each line it writes. A file that an area owns and that starts with a
generated-file header (GENERATED FILE - do not edit) must be on that
list, or a test fails. Add a new generated file there when its generator
lands.
Note
An unpublished OSV remediation run uploads no coverage and CI Gate skips the report. The diff and reports would show the bumped versions.
To check coverage before pushing, run a Worker's npm run coverage, or
pytest with --cov=packages/plugin-sdk-python/src. For Rust,
cargo llvm-cov nextest -p <crate> covers one crate; the whole workspace
needs a lot of disk.
The daemon's test build layers a check over its router
(crates/bookclerkd/src/openapi_contract.rs): every response the tests
produce is compared with the committed OpenAPI document. The route the
router matched, the method, the status and the media type must be
declared, and a JSON body must validate against the declared schema,
references followed and formats asserted. Each response is one JSON line
in target/contract-report/<pid>.jsonl (one file per test process, since
cargo-nextest runs each test in its own); each part of the test job keeps
its files in contract/part-<n>/ beside its coverage report, and CI Gate runs
scripts/contract_coverage.py over
them.
CI Gate writes the only Web API contract report of a run. The plan job's tests of the script run it too, but neither they nor any other script test in that job write to the job's step summary.
The step summary shows how many of the declared responses (route, status and media type) the tests observed, in all and (in a fold) per tag, and lists every disagreement: a status or media type the route does not declare, a body that does not validate, a body that is not JSON, and a matched route the document does not describe (an HTML page, such as the invite page or the SPA, is served on purpose without one). A declared response nobody observed is not a finding yet; it marks a route the tests do not reach.
The step is enforced: a disagreement fails CI Gate, and so does an
observed share below the floor [contract] floor in
scripts/coverage_areas.toml states
(a percentage of the declared responses; raise it when the share rises,
keeping it a little below the measured value, and never lower it to make
a change pass). When no report file exists because the daemon's tests
did not run (a build-only main push, a selective run that did not test
the daemon's package, or a test job that failed or was cancelled before
writing one), the floor is not checked and the step prints one line
instead of adding an empty report to the summary; a failed or cancelled
test job fails CI Gate on its own. When the plan tested the daemon's
package, the test job succeeded and no report exists all the same, or a
report line is not a whole hit (a partial write), the gate fails too. The layer never fails a
test, so a disagreement shows up in CI Gate's summary, not in a test's
output.
A disagreement means the document and the daemon differ for one
response. For a route's own statuses, fix the #[utoipa::path]
declaration or the handler that is wrong: a status the handler answers
on purpose is declared (with a realistic example when the body is JSON),
and a handler that answers something its route should not is corrected.
The access layer's refusals (401, 403, the second-factor enrollment
403, the session_check_unavailable 503), the CSRF 403, the 504
timeout and the JSON extractor's rejections are stated by the document
builder (crates/bookclerkd/src/openapi.rs), not by each route. Where a
route also answers a 503 of its own, the builder states both bodies and
says only the session-check refusal carries Retry-After. A public route
whose handler checks the session itself declares a bare 503 that names
session_check_unavailable, and the builder adds the body and the header.
To see the report locally, start it empty (each test process appends a file of its own, so an old run's hits would otherwise stay in it), run the daemon's tests and the script:
rm -rf target/contract-report
cargo nextest run -p bookclerkd
python3 scripts/contract_coverage.py --reports target/contract-report --mode enforce
BOOKCLERK_CONTRACT_REPORT_DIR moves the report files elsewhere.
The same reports also go to Codecov, which
comments on pull requests and posts codecov/project and codecov/patch
statuses. Both statuses are informational: CI Gate is the coverage check
that blocks a merge, and it reports on every pull request.
The upload runs in its own workflow,
coverage-upload.yml, after
each CI run of a pull request or merge group finishes:
- It runs from the default branch through
workflow_run. CI keeps its read-only token, and fork pull requests upload too. - It downloads the run's
coverage-*artifacts and plan, and runsscripts/coverage_gate.py --export. That rewrites each report through the gate's parser, keeping only repository-relative paths, line numbers and hit counts. - It uploads each area under its own flag, signing in to Codecov with OIDC, so there is no Codecov token secret. The commit, branch and pull request come from the CI run GitHub recorded, never from an artifact.
A newer CI run on the same pull request branch cancels an upload still waiting or running; each merge group's upload runs to the end.
The upload never checks out the pull request. Its checkout is the default branch, so it sends Codecov no file list and no line fixes from it. The reports' paths are already repository-relative. A head branch name outside a conservative character set skips the upload, because Codecov must never file it under the default branch.
The settings live in .github/codecov.yml.
Each area is a flag with carryforward, so a selective run that skips an
area keeps that area's last figure. A selective run that tests only some
Rust packages uploads under rust-partial instead of rust. That flag is
never carried forward, so the whole-workspace rust figure stays as it
was while the change's own lines are still measured. The patch target
matches CI Gate's, and Codecov ignores the same generated files. A test
checks that the flags, their paths, the target and the ignored files agree
with scripts/coverage_areas.toml.
Components give parts of the code that cut across flags their own figure on the dashboard and in the comment: the security surface, the daemon, the CLI, the plugins, the plugin SDKs and the Workers.
Merge groups upload too. Their commit is the one that lands on main, so it
gives later pull requests their base.
Important
Uploads need the Codecov GitHub App installed on the repository. Without it, the upload steps warn and the workflow still passes.
To make codecov/patch required later:
- Remove its
informational: trueline in.github/codecov.yml. - Add
codecov/patchto the default branch's ruleset, with the Codecov App as its source.
The test jobs that measure coverage also write a JUnit report beside each
LCOV report. The upload workflow sends them to Codecov Test Analytics, under
the same flags. Codecov then lists a pull request's failed tests in its
comment and, from merge group runs, finds tests that are flaky on main.
- Rust: the main nextest run uses the
ciprofile in.config/nextest.toml, which writestarget/nextest/ci/junit.xml. A workflow step copies it into the coverage directory, even when tests fail. Feature-test runs and custom-harness tests write no report. - The Workers: Vitest's
junitreporter, next to the coverage run. - The Python SDK: pytest's
--junitxml. - The first-party workerd guests: Node's
junittest reporter.
The test jobs upload their artifacts even when tests fail, so failures reach
Codecov. Like the coverage reports, the JUnit reports come from the pull
request's own run. coverage_gate.py --export reads each one with a fuzzed
parser and writes back only test names, durations, outcomes and failure
text.
Codecov Bundle Analysis tracks the size of the web UI and of each Cloudflare Worker script, and comments when a pull request changes one by at least 1 KiB. That gives early notice before a Worker nears Cloudflare's script size limit.
- The
uicheck keepsui/distafter its build. The Worker checks (site_audit,collector_test,demo_test) bundle each Worker withwrangler deploy --dry-run --outdir, the same bundling a deploy does, without deploying or needing an account. Each lands in$CI_BUNDLE_DIR/<bundle>/, uploaded as abundle-*artifact. - The upload workflow downloads them.
coverage_gate.py --export --bundlescopies only regular files with safe names, known asset types and bounded sizes (select_bundle_files, a fuzzed boundary). Source maps are left out. - The pinned
@codecov/bundle-analyzerintools/codecov, installed without install scripts, sends each bundle's file names and sizes to Codecov, signed in with OIDC like the coverage uploads.
The bundle status is informational, like the coverage statuses; its
settings are under bundle_analysis in .github/codecov.yml.
Coverage shows which lines the tests run, not whether a test would notice them doing the wrong thing. Mutation testing checks that: it makes a small change to the code, such as flipping a comparison or making a function return a default value, and runs the tests. If a test fails, the mutant is caught. If every test still passes, the mutant is missed: nothing checks that behavior, so a real bug there would slip through too.
Mutation testing is slow, so it does not run on pull requests. Three
workflows run it once a day on main instead, each on the code merged
since its own last successful run:
mutation.ymlfor Rust, at 02:17 UTC.mutation-python.ymlfor the Python SDK, at 02:47 UTC.mutation-workers.ymlfor the Workers, at 03:17 UTC.
Each run works like this:
scripts/mutation_base.pyfinds the commit the workflow's last successful scheduled run tested, through the GitHub API. A day with nothing merged, or a run that failed, is covered by the next run. If the API cannot be reached, the run fails rather than guess. With no earlier successful run, it falls back to the last commit from more than 25 hours ago, then to the parent ofHEAD. Whatever it picks must be a full SHA andHEADor one of its ancestors. When nothing merged since the last run, the range is empty and nothing is mutated.- The run diffs
mainagainst that commit and mutates only what changed, as described below for each language. The run number seeds every sample, so the choice changes from run to run. - The job summary lists what was tested and what was missed. A run that did not test its range fails, so the next run covers the same changes: the baseline failed, the tool exited with an error, or it left no results. A run stopped at its time limit, or early because the runner ran low on disk or memory, still succeeds with what it tested.
- When a mutant was missed, the reusable
mutation-issue.ymlworkflow files the summary as an issue titledMissed mutants: <area>(rust,python-sdk,site,demoorcollector), labelledmutation-testing. A later run with missed mutants in the same area comments on the open issue instead of opening another. Close the issue once the tests catch them.
The issue job holds the only write token in these workflows. It checks out nothing and runs no repository code; it reads only the summaries the mutants jobs uploaded.
To test a range by hand, run the workflow from the Actions tab and set
base to the commit to start from. Running it from a branch tests that
branch against base, which is how to try a change to the workflows
themselves. Only runs on main file issues, and a manual run never moves
the starting point of the next scheduled one.
cargo-mutants tests the Rust code:
- It mutates only the functions changed since the last run (
--in-diff) in any workspace member (--workspace; without it cargo-mutants looks only at the default members), so the cost follows what merged. With no Rust file changed, the job builds nothing. Test code, the generated files and the staged-plugin harness (bookclerk-plugin-e2e, whose tests need the plugins CI'se2echeck stages) are never mutated. - Mutants live only in function bodies, so a change to data that code reads (an attribute, a constant, a type definition) gives no mutants from the diff. When a changed Rust file gets none, the job mutates the functions in that file instead, so the tests of the code beside the changed data are still checked. The summary names those files. Up to 60 of their mutants are all tested; past that, at most 60 spread across the files are (counting every mutant a chosen line brings in, such as its function's body replacements), and the choice changes with each run. Each file gets at least its smallest line while that fits. If every line alone brings in more than 60, the one that brings in the fewest is tested, so such a run can exceed 60. Each mutant runs its package's whole test suite, two minutes for some crates, so a large file would otherwise hold a runner for the whole time limit. Code in other files that reads the data is not covered.
- Before the first mutant, the job builds what CI's
rust_testcheck builds first (the jail, the workerd front door, the pinned workerd and the platform guests) withci-exec.py prereqs rust_test, and turns on the same confinement enforcement as the test job. Tests that launch a helper then run instead of skipping. - Each mutant runs the tests of its own package under cargo-nextest.
- The run stops after 4 hours and reports what it tested so far. It also stops early, the same way, when less than 3 GiB of disk is free or less than 768 MiB of memory stays available for two checks 15 seconds apart. A runner that runs out of either loses contact with GitHub, and the job then fails with no log and no results, so the next run would retry an even larger range. The job frees the preinstalled SDKs it never uses before it builds, logs the disk and memory left every 10 minutes, and the summary says why a run stopped early.
- Those checks are too slow for a mutant that allocates without bound, so cargo-mutants also runs in a cgroup capped at all but 1.5 GiB of the runner's memory. The kernel then kills the largest process in it, normally that mutant's test, instead of starving the runner, and the mutant counts as caught. The log notes the cap and how many processes it stopped. When the cgroup cannot be set up, a warning says so and the run goes on uncapped. A kill that lands on cargo itself leaves the mutant unclassified; each one gets its own warning, and the job summary says the results are incomplete, because no list holds it. A kill that lands on cargo-mutants itself, before the time limit and without the watchdog, fails the run, so the next run retries the same range.
The job summary shows the share of mutants the tests caught and lists the
missed ones. The full results, with a log per mutant, are in the
mutants-rust artifact. A mutant that does not build (unviable) says
nothing about the tests and is left out of the score; one whose tests time
out counts as caught.
To fix a missed mutant, add a test that fails with the change it names, or tighten an assertion that should already have caught it. To reproduce one locally:
cargo install cargo-mutants --locked
git diff origin/main... > /tmp/change.diff
RUSTFLAGS=--cap-lints=warn cargo mutants --workspace --in-diff /tmp/change.diff
For a file the summary says was mutated whole, mutate all of it with
--file:
RUSTFLAGS=--cap-lints=warn cargo mutants --workspace --file crates/bookclerkd/src/api.rs
Capping lints keeps a mutant that only leaves a parameter unused from
failing to build under the crates' deny lints; the workflow caps them for
every build in the job. Settings live in
.cargo/mutants.toml: nextest as the test
runner, the custom-harness test binaries CI runs with cargo test left
out of nextest, and the excluded files, whose generated entries must match
generated in scripts/coverage_areas.toml. Tests check both lists.
The mutation-workers.yml
workflow does the same for the Cloudflare Workers (tools/site,
tools/demo and tools/diagnostics-collector) with
Stryker, one job per Worker:
- It finds the lines changed since the last run in the Worker's TypeScript sources: the lines added, and the line before each run of lines it deletes, so removing a guard counts as changing the code around it. That line can be a blank line or a closing brace with no mutant; when none of a file's changed lines has one, step 3 samples the file. Tests, test support, type declarations and configuration files are left out. A Worker with no changed source line skips Stryker, and its summary says so.
- It passes those lines to Stryker as
--mutate path:start-end. Stryker also mutates literals (strings, arrays, objects), so a change to data gets mutants too. - Some changes give no mutant at all: a type, an interface, an import or an export list. When a changed file's lines give none, a second pass mutates a sample of the file's other lines, so the tests of the code beside the change are still checked, as the Rust job does for data changes. The sample skips lines Stryker never mutates (types, imports, bare punctuation). At most 30 lines are sampled across those files, one per file first while the budget lasts, and the choice changes with each run. Stryker takes about five seconds a mutant here, so a whole file would not fit the time limit. The summary names the files and how many lines it sampled in each, or says when too little time was left for the second pass.
- Each mutant runs the Vitest tests that cover it, including the ones that run inside the Workers runtime.
- Both passes together stop after 40 minutes. The summary then counts the mutants tested before the limit and says the run was stopped.
The job summary and the mutants-<worker> artifact have the same shape as
the Rust job's, and missed mutants become a site, demo or collector
issue. Stryker
is installed for the run only, at the version pinned in the workflow, so
the Workers' lockfiles do not change. Settings live in
scripts/stryker.config.json:
inPlace, because some tests read repository files by relative path. Stryker then rewrites the sources it mutates on disk while it runs, so the step setsMUTATION_TESTING=1and each Worker'svitest.config.tsleaves out the tests that read its own source text (tools/site/worker-source.test.ts,tools/diagnostics-collector/cache-control-source.test.ts). Those tests would fail on the instrumented code. A mutant never shows in the source text, because Stryker switches mutants on at run time, and the one function they call, the collector'sassetUrlsModule, has its own test incache-control.test.ts. Every other run keeps them.- A
tsconfigFilethat does not exist, because TypeScript 7 has no JavaScript API for Stryker's tsconfig rewriting. Type checking stays with each Worker'snpm run typecheck. disableTypeChecksoff, so Stryker does not add// @ts-nocheckto every file. That would change the collector's hashed dashboard assets, and Vitest does not check types anyway.vitest.relatedoff, because the Workers runtime loads modules outside Vitest's module graph, so Vitest cannot tell which tests import a file.incremental, writing toreports/mutation/mutation.json, because only incremental mode saves the mutants tested so far when the run is stopped. The second pass also mutates the first pass's lines, so their results carry over from that file without being tested again. The report step reads it.
To reproduce a Worker's run locally, from its directory:
npm install --no-save @stryker-mutator/core@10.1.1 @stryker-mutator/vitest-runner@10.1.1
MUTATION_TESTING=1 npx stryker run ../../scripts/stryker.config.json --mutate "src/page.ts:10-20"
The report lands in reports/mutation/mutation.json. Delete that folder
afterwards: incremental mode reuses the results in it on the next run.
The mutation-python.yml
workflow does the same for the Python plugin SDK
(packages/plugin-sdk-python) with mutmut.
Before mutmut starts, it installs capnp and builds the
bookclerk-db-testkit binary that tests/test_testkit.py drives, as CI's
python_sdk check does, so no test run builds it:
- It finds the lines changed since the last run under the SDK's
src/: the lines added, placed in the new file, and the lines deleted, placed in the starting commit's copy of the file. Removing a guard, even a function's last lines, counts as changing that function, and removing a constant counts as a change outside every function. Removing a whole function selects nothing in that function. Tests and the generated ABI modules are left out. With no changed source line, mutmut does not run and the summary says so. - mutmut names each mutant after its function, so
scripts/mutmut_report.pyturns each function holding a changed line into a name pattern, such asbookclerk_plugin_sdk.tools.x_check_plugin__mutmut_*. A method's name isxǁ<Class>ǁ<method>, with mutmut'sǁseparator. - mutmut only mutates function bodies. A changed line outside every function, such as a constant, an import or a class attribute, selects the whole module instead, so the tests of the code beside the changed data are checked, apart from the functions mutmut never mutates (step 4). The summary names those modules. Blank lines, comments and docstrings outside functions select nothing. The SDK's tests run in about a second, so most modules fit in the time limit.
- mutmut never mutates a decorated function (other than one with a single
@staticmethodor@classmethod) or an enum's methods. The summary names the changed functions it skipped for that reason. - Each mutant runs the tests that cover its function. A mutant no test
runs counts as missed. mutmut picks those tests by node id, and pytest
cannot find a node id again once it has escaped a backslash or a
non-ASCII character in it, so
tests/conftest.pyrefuses such test ids. Give a parametrized case an ASCII id instead ofreprof its input. mutmut also runs the tests from a copy one folder deeper, undermutants/, so a test finds the repository root by a folder it holds, never by a fixedparents[N];tests/test_conventions.pychecks that. - The run stops after 60 minutes. mutmut records each result as it lands, so the summary then counts the mutants tested before the limit and says the run was stopped. If mutmut fails or stops before testing any selected mutant, the summary says that instead of reporting no mutants.
The job summary has the same shape as the other jobs', with each missed
mutant's line and change, and missed mutants become a python-sdk issue.
Mutants mutmut ran that ended with a
result it does not count (a crash, a skip or a suspicious result) are
left out of the score, and the summary says how many there were. The mutants-python-sdk artifact lists every mutant, one file per
outcome. mutmut is installed for the run only, at the version pinned in
the workflow, so the SDK's uv.lock does not change. Settings live in the
SDK's pyproject.toml, under [tool.mutmut]: the package to mutate, the
generated modules to leave alone (the same list ruff excludes, which a
test checks) and the test folder.
To reproduce the run locally, build the testkit first (it needs Rust and
capnp, as the workflow installs), then run mutmut from
packages/plugin-sdk-python:
cargo build -p bookclerk-db-testkit --bin bookclerk-db-testkit
export BOOKCLERK_DB_TESTKIT="$(git rev-parse --show-toplevel)/target/debug/bookclerk-db-testkit"
python3 -m venv /tmp/mutmut && /tmp/mutmut/bin/pip install pytest "mutmut==3.8.1"
/tmp/mutmut/bin/mutmut run "bookclerk_plugin_sdk.tools.x_check_plugin__mutmut_*"
/tmp/mutmut/bin/mutmut show bookclerk_plugin_sdk.tools.x_check_plugin__mutmut_3
mutmut works in a mutants/ copy of the SDK and reuses the results in it
on the next run. Delete that folder afterwards.
Note
None of the three workflows is a required check, and none runs on pull requests. A missed mutant never blocks a merge; it becomes an issue.
Cloudflare rejects a deploy whose Worker takes more than one second to run
its global scope. CI tracks how close each Worker (site, demo, collector)
is, with scripts/worker_startup.py and
the budgets in
scripts/worker_startup.toml.
- Pull requests: each Worker check (
site_audit,collector_test,demo_test) bundles its Worker and runswrangler check startupon it seven times. That profiles startup locally in workerd and needs no Cloudflare account, so forks get it too. The step summary shows each run's active CPU time and the median, and a median over the Worker'slimit_msbudget warns. - Deploys:
wrangler deployprints the startup time Cloudflare measured. Each deploy workflow writes it to its run summary and warns pastdeploy_warn_ms(500 ms).
Note
Local figures are a trend, not Cloudflare's numbers: the runner's CPU differs, the profiler samples roughly every 10 ms (a small Worker reads as 0 to 15 ms), and the first run is often a cold start. The median of seven runs smooths that out; budgets sit well above today's medians so only a real jump, such as a large library imported at the top level, warns.
Both start in report mode. Set mode = "enforce" in the budgets file
to make a Worker over budget fail its check.
Configure repository rulesets or branch protection so the required CI check is
CI Gate (plus the OSV check), not every matrix child. The
Bookclerk CLA and Maintainer review checks are required too, expected
from the automation App (see the CLA check and
maintainer review). The ruleset also
requires signed commits; the Signed commits check explains a pull request
that its unsigned commits keep out of the queue (see
signed commits).
Warning
Skipped release, confinement, native-gateway, tray and macOS jobs
report skipped. If those job names are required individually, merges stay
pending forever.
Also enable a merge queue that consumes merge_group checks, so the full
suite runs on the synthetic merge commit before landing. Merge queues are
only available on organization-owned repositories (free for public
repositories).
Queue settings:
- squash;
- "only merge non-failing pull requests";
- build concurrency 2;
- merge limits: minimum 1, maximum 5, wait time 5 minutes;
- status check timeout 300 minutes.
With a queue required, "require branches to be up to date" no longer
matters. The queue tests each entry against the current main, stacked on
the entries ahead of it.
The organization runs on GitHub Team, which allows 60 hosted jobs at once across every workflow, 5 of them on macOS, and hosted runners serve queued jobs first come, first served. A job from a pull request gets no priority over one from a merge group, or the other way round; the pull request lanes keep at least 24 runners for the queue.
The merge queue starts one merge_group run per entry it builds, each
holding the entries ahead of it. Merge limits don't change that: GitHub
documents that they "do not combine merge_group builds" and only decide
how many passing entries land at once. Build concurrency is the only
setting that caps how many groups test at the same time.
One merge group asks for about 28 jobs: the CI jobs (most of them start
together right after plan), the five
test-os legs, plus OSV-Scanner, the
CLA check and the coverage upload. Most finish within 10 minutes. The
test-os legs (about 30 minutes each), the fat-LTO release build (about
26), the three Linux test parts and fmt / clippy / docs (about 15 each),
the long Windows jobs and merge-queue fail-fast, which holds a runner
until the run ends, keep going. A group takes about 350 runner-minutes;
with its whole fan-out running at once it takes about 35 to 40 minutes.
| Build concurrency | Merge-group runners (peak, typical) | macOS runners | Left for pull requests |
|---|---|---|---|
| 1 | about 25, then about 10 | 3 | about 35 to 50 |
| 2 | about 50 when fan-outs overlap, about 20 typical | 6 at overlaps (4 once the macOS job folds into test-os) |
36 guaranteed, about 40 typical |
| 3 or more | the whole pool | 9 or more, over the 5 allowed | 36, the lane floor |
Build concurrency 2 lands about 3 pull requests an hour, against about 1.7
at 1. It is the highest that fits: the organization may run only 5 macOS jobs
at once. Each group runs the macOS job and two macOS test-os legs, so
one of six macOS jobs can wait for a runner where two groups overlap. Once test-os replaces the macOS job, two groups
need 4; a third group would always wait.
On GitHub Free (20 runners, until 20 November 2026) these docs advised build concurrency 1, though the setting was back at 2 by then. Two groups then took most of the pool: on the busy night of 9 to 10 November 2026, with two groups and leftover runs building (see Superseded merge groups), merge-group jobs waited 10 to 25 minutes for a runner, runs took about 65 minutes, and the queue landed about 1.3 an hour while pull request checks waited behind it.
A failure costs more with two groups building: a failure (including a flaky test) ejects the head entry and rebuilds the group behind it. "Jump to the front" likewise restarts the groups in progress.
The status check timeout counts from the moment a group requests its checks, including the time its jobs wait for runners. On 10 November 2026, with pull request jobs holding most of the pool, a green group ran past the earlier 120-minute timeout and was ejected, so the timeout is now 300 minutes.
Raise build concurrency again only when the macOS limit grows, for
example an Enterprise plan (50 macOS jobs) or a self-hosted macOS runner
reserved for merge_group.
The queue rebuilds its groups when an entry ahead fails or leaves, when a
pull request jumps to the front, or when one is dequeued. It deletes each
dropped group's gh-readonly-queue/main/pr-<number>-<sha> branch, but
GitHub doesn't cancel that group's run, and the workflow's concurrency
never cancels a merge_group run. The run would keep asking for its
22 jobs although nothing can merge from it.
merge-queue-fail-fast therefore checks the group's branch
(github.event.merge_group.head_ref) with git ls-remote --exit-code on
every pass and cancels the run once the branch is gone. It starts with the
run, so it gets a runner before the jobs that fan out after plan. The
queue also deletes a group's branch after merging it, so the check comes
after the job's CI Gate test, and the job looks again for a started
CI Gate just before it cancels. A transport error counts as the branch
being present. A cancelled run doesn't trigger
re-enqueue, because its jobs carry no
not-acquired annotation.
The macOS runs of confinement, native-gateway and tray share one
job, macOS (sandbox, gateway, tray), on macos-latest. Since 2 November
2026, GitHub has often failed to give macOS jobs a runner, whichever macOS
label they ask for. In the runs checked on 4 November, no more than two
of this repository's macOS jobs ran at the same time. One job per run asks
for one macOS runner instead of three.
The job runs when any of the three jobs is planned. It runs each planned
check as its own step once setup succeeded, and a check still runs after
an earlier check failed, so one round shows every failure. It saves the macOS debug cache on
main, as native-gateway does on Windows. In OS_LEG_JOBS in
scripts/ci_plan/plan.py, CI Gate
expects the job whenever it expects any of the three, except on a pull
request, where it waits for the merge queue (see
Platform legs in the merge queue).
GitHub sometimes never gives the job a runner. It cancels the job with the
not-acquired annotation (see re-enqueue below),
and CI Gate fails, as for any job that doesn't succeed. After one retry,
the macOS checks are best effort:
- A pull request run is re-run once (attempt 2). On a merge group, the pull request goes back in the queue once.
- If the macOS job gets no runner again,
CI Gatepasses without it. A step inCI Gaterunsscripts/merge_queue_requeue.py --stranded-jobto check that the job was cancelled with no runner and no step and carries the not-acquired annotation, and that the same happened on the try before: the previous attempt of a pull request run, or the pull request's previous merge group, when it has entered the queue at least twice on its current head. That merge group must have started after the pull request first entered the queue on its current head, so a run of an older head never counts. A first miss after a retry for another reason still fails. - The gate prints a warning and a step-summary note that the macOS checks didn't run on that commit.
A build-only main push needs no retry. There
the macOS job only compiles and saves the macOS debug cache, and the
merge queue already ran the macOS checks on that commit. If GitHub never
gives it a runner, the same step vouches for it on the first try, so
main doesn't show a failed run for a cache save. The next main push
that gets a runner saves the cache.
CI Gate sees such a job as abandoned in needs (the jobs API reports
it as cancelled), so it accepts either result for a job the step
vouched for.
A macOS job that ran and failed, or was cancelled after it started, always
fails the gate. So does a stranded job on an OSV remediation run or a push
to any branch other than main. The gate job reads the GitHub API with a read-only token
(actions, checks and issues).
Hosted runners, most often macOS arm64, sometimes never pick up a
merge-group job. GitHub cancels the job with the annotation "The job was
not acquired by Runner of type hosted even after multiple attempts",
CI Gate fails, and the queue removes the pull request although every job
that ran passed.
merge-queue-requeue.yml puts such a pull request back in the queue. It
runs when a merge_group CI run fails, and
scripts/merge_queue_requeue.py
re-enables auto-merge with the automation App
only when all of these hold:
- Every job that didn't succeed or skip is
CI Gate,merge-queue fail-fast, or a job that was cancelled before it got a runner or ran a step. - Each of those never-started jobs carries GitHub's not-acquired annotation. A job cancelled because someone removed the pull request from the queue has none, so a hand dequeue stays dequeued.
- The pull request is open, isn't a draft, comes from this repository and doesn't already have auto-merge on.
- Its current head is the one that was queued, and it has entered the
queue fewer than three times, so one commit is retried at most twice. A
commit pushed after the failure waits for its own enqueue, and the
enqueue names the queued head (
--match-head-commit), so a push in between makes it fail.
Every check stays required: the new merge group runs the full suite again.
A run whose CI Gate passed (the macOS job let through) is
never re-enqueued or re-run.
The workflow runs from the default branch, reads only the GitHub API and
runs no pull request code. The step summary says why it did or didn't
enqueue.
A pull request's own CI run can fail the same way. When a pull_request
CI run fails, the same workflow re-runs its failed jobs (the stranded jobs
and CI Gate) on the same commit when:
- Every job that didn't succeed or skip passes the same test as steps 1 and 2 above. A run cancelled by a newer push has no not-acquired annotation, so it is never re-run.
- The run is on its first or second attempt, so one run gets at most two re-runs.
- No newer
pull_requestCI run exists for the same head branch and repository. CI cancels a pull request's older runs when a new one starts, so re-running a run that a push has superseded would cancel CI for the current commit. This is checked last, just before the re-run.
The re-run uses the workflow's own token (actions: write), not the
automation App, and keeps the original run's context, so a fork's pull
request is re-run with the same read-only permissions it had.
With SELECTIVE_CI=1, a pull request should see these jobs after plan:
| Change | Jobs after plan |
Notes |
|---|---|---|
docs/** only |
Workers (site_test only), fmt / clippy / docs (lint only) |
The site's link check covers the docs it publishes |
tools/site/** only |
Workers (site_test, site_audit), fmt / clippy / docs (lint only) |
No Rust toolchain |
tools/diagnostics-collector/** only |
Workers (collector_test), fmt / clippy / docs (lint only) |
No Rust toolchain |
tools/demo/** only |
Workers (demo_test), fmt / clippy / docs (lint only) |
No Rust toolchain; packaging/demo/** selects nothing |
README.md, AGENTS.md, other top-level prose, config/config.example.toml, scripts/tests/** |
fmt / clippy / docs (lint only) for Markdown and JSON |
The plan job runs the planner and lint tests on every event |
ui/** only |
fmt / clippy / docs (UI, UI API docs and lint) |
No Rust toolchain checks |
crates/bookclerk-cli/src/** |
check, test (-p bookclerk-cli), release (bookclerk-cli) |
No postgres, no E2E |
crates/bookclerk-plugin-host/src/** |
check, test (host and dependents), e2e (runtime smoke), release (hosts), native gateway (Linux; Windows and macOS in the queue), postgres (rpc_like only) |
No sandbox matrix |
| One optional guest (such as Libro) | check, test (guest tests), e2e (libro only) |
Other storefronts are neither built nor staged |
packages/plugin-sdk/embed/** |
check (SDK ABI), test (workerd tests), e2e (workerd-runtime subset), release (launcher), native gateway (Linux; Windows and macOS in the queue) | Not the full installation |
A draft pull request runs only the fmt / clippy / docs and test jobs from
this table; the rest wait until it is ready for review (see
Draft pull requests). For merge_group,
--force-full runs every job and check on every platform. A main push
adds --build-only.
- Fuzzing: the fuzz targets and their own workflows.
- Repository automation: the workflows outside pull-request checks.
- Dev Container: the environment that mirrors these checks locally.
Source: docs/contributing/ci.md at main · View as Markdown