Release packaging

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

This page explains how Bookclerk is packaged: what ships in a platform release, how the pinned Cloudflare workerd binary is kept current, the plugin author tools, and how to build the archives yourself.

Platform installs, crates.io plugin installs and CI all use the same artifact layout (see Plugin registry).

What pulls from crates.io?

Artifact crates.io role Installed automatically?
Host (bookclerk, bookclerkd) Not on crates.io today No: an OS installer or GitHub Release
Platform plugins (sqlite, local, embedder) May publish metadata crates later Bundled in the platform install (cargo package-platform)
Storefront plugins (Audible, Libro, …) A discovery index, when published Opt-in, through bookclerk plugins install (which records a library install)

crates.io is a catalog, not a runtime dependency resolver. Nothing is compiled from crates.io on the operator's machine.

  • bookclerk plugins search works today.
  • bookclerk plugins install resolves a prebuilt HTTPS archive from the package's artifact URLs, verifies it, and records it as an installed version. Each host then fetches the archive for its own target, verifies its payload digest, and materializes it (see Installing plugins).

What a platform release contains

A platform release ships, so a fresh install works without storefronts or a Rust toolchain:

  • the hosts;
  • bookclerk-jail, bookclerk-workerd and bookclerk-media-worker;
  • the pinned Cloudflare workerd binary;
  • the platform plugins (sqlite, local and embedder) under plugins/<plugin-key-fs-id>/, recorded in install-ledger.json.

On startup the host installs those bundled plugins from the plugins/ folder beside its executable into $BOOKCLERK_FILES_DIR/plugins/, and stamps its own ledger. So any files folder works with a release, including a container volume (see Running Bookclerk → Docker).

cargo build-app --platform and cargo ensure-workerd fetch the pinned workerd into target/<profile>/. The download is tried up to four times, waiting 2, 4 and 8 seconds between tries, when the connection drops, a try takes longer than five minutes, or the server answers 5xx, 408 or 429. Any other answer, a redirect loop, or a digest that does not match the pin fails at once. Reference Echo examples are never packaged.

Cloudflare workerd pin

The release tag and per-architecture sha256 digests live in crates/bookclerk-workerd/workerd-pin.json. The Rust pin.rs and the SDK stub copies are generated from it by scripts/sync-workerd-pin.py.

Important

Don't float "latest" on every cargo dev. Hosts and packages must share one audited binary.

A daily GitHub Actions job (.github/workflows/workerd-pin-bump.yml) proposes bumps:

  1. scripts/bump-workerd-pin.py picks the newest Cloudflare workerd release that is at least 7 days old. That is the same supply-chain cooldown as Dependabot's cooldown.default-days: 7 in .github/dependabot.yml. It writes workerd-pin.json, then regenerates pin.rs and the SDK stubs.
  2. The job installs capnproto (needed to compile schema/plugin.capnp), rebuilds bookclerk-workerd, runs cargo ensure-workerd, then runs scripts/test-workerd-echo.sh against the Echo workerd example guests.
  3. It opens deps/workerd-pin with the automation App token and turns on auto-merge, so the merge queue lands it once its required checks pass (see Automation App).

To check by hand, without writing anything:

python3 scripts/bump-workerd-pin.py --check --cooldown-days 7
python3 scripts/sync-workerd-pin.py --check

The Pyodide bundle

The embedder plugin loads the ONNX Runtime shared library from beside its own executable. The library is the upstream build, pinned by version and SHA-256 for each platform in crates/bookclerk-onnxruntime/onnxruntime-pin.json:

  • cargo ensure-onnxruntime downloads it into target/onnxruntime-<version>/ and checks the digest. Staging, cargo install-platform and cargo package-platform do the same and copy it into the plugin's folder, so platform packages carry it.
  • Upstream publishes no build for Intel macOS. There the plugin is still packaged, reports itself unhealthy and serves no model.
  • The library and the plugin need the C++ runtime (libstdc++ on Linux), which the Docker image and every supported distribution already have.

See Embedding model packs.

Python isolates load the Pyodide runtime bundle when they start. workerd would download it on its own connection, but the jail gives workerd no network, so the bundle is pinned too: pyodide_bundle in workerd-pin.json names the file and its sha256.

  • cargo ensure-workerd (and every caller of ensure_workerd) downloads it from pyodide-capnp-bin.edgeworker.net into pyodide-bundles/ beside the binary and checks the digest. Platform packages carry the same directory.
  • The launcher passes that directory to workerd with --pyodide-bundle-disk-cache-dir, and the gateway jail may only read it.
  • A Python plugin refuses to start, naming the missing path, when the bundle is not there.

A pin bump keeps the current bundle. When a new workerd release wants another one, the staged Python plugin checks fail and the plugin's error names the bundle URL workerd asked for. Download that file, put its name and sha256 under pyodide_bundle, and run python3 scripts/sync-workerd-pin.py --write.

Plugin author tools (check / fmt / sync-embed / package / smoke)

Each language SDK bakes in the same authoring CLI, so there is no multi-architecture tool binary to ship. Commands and flags match across languages, and each SDK is self-contained for its workerd guest language (embedding and runtime settings):

Launcher Example
Rust cargo plugin -- check . / cargo plugin -- smoke . / cargo plugin -- test . / cargo plugin -- package --out dist .
TypeScript npx bookclerk-plugin check . / npx bookclerk-plugin smoke .
Python python -m bookclerk_plugin_sdk check . / python -m bookclerk_plugin_sdk smoke .

smoke:

  1. Downloads the pinned Cloudflare workerd binary. TypeScript and Python don't need the Rust bookclerk-workerd launcher.
  2. Materializes Cap'n Proto and the bridge, including a per-isolate BRIDGE_TOKEN.
  3. POSTs describe (plus the role's health route) with Authorization: Bearer ….

test (Rust CLI only) runs the plugin's own suite once per database capability profile; see Database limits.

The Rust CLI lives in the bookclerk-plugin-tools crate (cargo plugin runs its bookclerk-plugin binary), and uses the bookclerk-workerd library for smoke. Guest plugins depend only on bookclerk-plugin-sdk, so they never pull in workerd download dependencies. scripts/sync-workerd-pin.py keeps the pin and bridge copies in sync (--check in CI).

Workerd packaging. Authors import the language package (@bookclerk/plugin-sdk/workerd, bookclerk_plugin_sdk.workerd), and bookclerk-workerd injects those modules into the isolate.

  • The Python SDK also requires, and writes, compatibility_flags = ["python_workers", "disable_python_external_sdk"] for .py mains.
  • The optional sync-embed vendors SDK sources for offline trees.

The shared contracts are plugin.capnp, plugin-toml.json, and the fixtures under crates/bookclerk-plugin-abi/fixtures/tools/.

Local packaging (current OS/arch)

Host bundles need ui/dist. cargo package-hosts rebuilds it when the SPA sources are stale, or run cd ui && npm ci && npm run build.

cargo package-plugins      # product plugin archives (no examples) → target/dist/plugins/
cargo package-hosts        # hosts + jail + media-worker + bookclerk-workerd + workerd → target/dist/
cargo package-platform     # hosts + helpers + pinned workerd + sqlite + local → target/dist/

Archive names follow the Plugin registry:

bookclerk-plugin-source-audible-0.1.0-linux-x64-gnu.tar.gz
bookclerk-platform-0.1.0-linux-x64-gnu.tar.gz

Each output folder includes SHA256SUMS* files for verification.

Container images

.github/workflows/images-publish.yml publishes three images to ghcr.io/<owner>/, where <owner> is the repository's owner. A fork publishes to its own namespace with its own GITHUB_TOKEN.

Image Built from Architectures
bookclerk packaging/docker/Dockerfile amd64, arm64
bookclerk-dev .devcontainer/Dockerfile amd64, arm64
bookclerk-demo packaging/demo/Dockerfile amd64

The bookclerk-dev image carries inline cache metadata, so the Dev Container's build.cacheFrom reuses its layers. The demo image builds on the production builder and runtime, all three in one build from packaging/demo/docker-bake.hcl. That build links with thin LTO and parallel codegen units. Shipped images keep Cargo.toml's fat LTO: packaging/docker/Dockerfile takes the settings as build arguments that default to it. The demo's sample-data seed links the host's library crates, so it builds with exactly the package set cargo package-platform built (cargo build-app --platform --print), where those crates are already compiled. The storefronts build in a separate Cargo invocation, because building them beside the seed would change those crates' features and recompile them. The demo image's last stage seeds the demo library at build time (packaging/demo/bake-seed.sh): it runs the seed and bookclerkd with network access to the storefronts' public catalogs, downloads the covers a new session shows first into the cover cache, then strips every per-container secret, so building the demo image needs outbound network (see tools/demo/README.md). The hosted demo's deploy pulls the published demo image without signing in, so the package must be public. When the image for the deployed commit exists and its attestation verifies, the deploy uses it instead of building it again. Otherwise a push or manual deploy builds the image itself, and a deploy started by a publish that pushed no demo image stops (see The demo deploy).

Each architecture builds on a native runner and is checked before it is pushed, then the architectures are joined into one index:

  • The production image boots twice through packaging/docker/smoke.sh.
  • The Dev Container's tools run as the dev user.

Tags

  • sha-<12 hex>: built from that main commit, and never moved. The per-architecture sha-<12 hex>-<arch> tags are the index's inputs.
  • edge: the newest build from main.
  • <version>, <major>.<minor>, <major>, latest: added when a v<major>.<minor>.<patch> release is published, by retagging that commit's images (built then if missing). Leading zeros and suffixes are refused by scripts/image_release_tags.py.

When it runs

The workflow runs:

  • nightly when main has moved (an image whose sha- tag exists is not rebuilt);
  • on a published release whose commit is on main (see release tags);
  • on main pushes that change the Dev Container's inputs;
  • by hand (a run on any branch but main is a dry run).

A pull request that changes the workflow, .github/actions/image-build or scripts/image_release_tags.py builds and checks every image with a read-only token, and pushes nothing. Both paths run the same build steps from that action.

Provenance and permissions

Every published index gets a GitHub build-provenance attestation. It is signed through Sigstore with the workflow's OIDC identity and stored in the registry. To check one, replace <owner> with the repository's owner:

gh attestation verify oci://ghcr.io/<owner>/bookclerk:edge --owner <owner>

If the attest job fails, re-running the workflow attests the published index without rebuilding it.

Only the push jobs hold packages: write. Only the attest job holds id-token: write and attestations: write.

Making the packages public

The first publish creates each package as private, so docker pull without signing in is refused. An organization owner makes each package public once. A package appears only after its first publish: pushes to main publish only bookclerk-dev, so bookclerk and bookclerk-demo first appear after the nightly run or a manual run (see When it runs).

  1. Open https://github.com/orgs/<owner>/packages/container/<image>/settings (for example https://github.com/orgs/bookclerk/packages/container/bookclerk/settings).
  2. Under Danger Zone, choose Change visibility, then Public.
  3. Type the package name to confirm.
  4. Repeat for bookclerk-dev and bookclerk-demo.

Note

The organization setting that lets members create public packages (Settings > Packages > Package creation) only decides which visibilities new packages may get. It does not change packages that already exist.

Warning

GitHub does not let a public package go back to private. Later publishes keep the package's visibility, so this step is needed only once.

Desktop packages

The desktop app's unsigned packages (see Packages) are built by .github/workflows/desktop-packages.yml: a .deb and an AppImage on Linux, an NSIS installer on Windows, and a .dmg on macOS. Each is a full release build on its own runner.

The images workflow calls it, so it runs on the images' schedule (see When it runs):

  • nightly, and by hand, when main has moved since the last publish;
  • for a published release, whose packages it attaches to the release.

A pull request runs it only when it changes the workflow or the packaging configuration (tauri.packaging*.json, crates/bookclerk-dev/src/package.rs). CI's desktop job covers the window's code on every pull request. A change that breaks how the packaged app finds its runtime shows up in the next nightly build.

Every build uploads the packages as workflow artifacts, kept for 14 days. For a release, the publish job attaches them to the GitHub release with a build-provenance attestation each. To check a downloaded package, replace <owner> with the repository's owner:

gh attestation verify <downloaded package> --owner <owner>

Only that job holds contents: write, id-token: write and attestations: write.

The package builds keep no Rust cache. Three release caches would not fit beside CI's caches in the repository's 10 GB, and a nightly build has no one waiting on it.

Release tags

Publishing a release starts the image publish, the docs publish, the website deploy and the Audiobookshelf integration test, each for the commit the release's v* tag names. Those runs use that commit's own copy of each workflow, so whoever can create or move a v* tag decides what gets built, signed and published as a release. Three layers keep that to reviewed code:

  1. Rulesets. Only repository administrators may create a v* tag, and nobody may move or delete one. The automation App can write contents, but it can't tag a release. To set them up, run this once as an administrator, with your own GitHub CLI login:

    bash scripts/setup-release-rulesets.sh Bookclerk/bookclerk
    

    It creates, or updates in place, the Release tags: creators and Release tags: history rulesets. An administrator still publishes releases from the web UI as before, since that creates the tag.

  2. Immutable releases. Turn on Settings → General → Releases → Enable release immutability, so a published release's tag and assets can't change afterwards.

  3. The commit must be on main. The image and docs publishes start with .github/actions/release-on-main, which fails unless the default branch contains the release's commit. The website deploy builds its code from the default branch and takes only the release's docs/ pages from the tag.

Note

The check in step 3 runs from the tagged commit's own workflow, so it catches a tag put on the wrong commit by mistake, not a forged one: a commit written to drop the check would skip it. The rulesets are the control that stops that.

GitHub Actions (signing sketch — not a contract)

Note

The workflow below is an illustrative sketch for publisher-side packaging. Digest pinning, notarization and Authenticode requirements are set by follow-up install and trust work, not frozen by this page.

Per-OS matrix builds should run the same cargo aliases, then optionally sign or notarize:

# .github/workflows/release.yml (sketch — subject to change)
jobs:
  package:
    strategy:
      matrix:
        include:
          - os: ubuntu-latest
            target: x86_64-unknown-linux-gnu
          - os: macos-latest
            target: aarch64-apple-darwin
          - os: windows-latest
            target: x86_64-pc-windows-msvc
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v7
      - uses: dtolnay/rust-toolchain@stable
      - uses: actions/setup-node@v7
        with:
          node-version-file: .tool-versions
      - run: cd ui && npm ci && npm run build
      - run: cargo package-platform --out dist
      - run: cargo package-plugins --out dist/plugins
      # Signing (examples — pick per OS):
      # - Linux: cosign sign-blob / minisign
      # - macOS: codesign + notarytool (after package-platform)
      # - Windows: Authenticode (signtool)
      - uses: actions/upload-artifact@v4
        with:
          name: bookclerk-${{ matrix.target }}
          path: dist/

Publish the release assets to GitHub Releases (or S3 or R2), and set artifact_base_url in each plugin crate's [package.metadata.bookclerk] before cargo publish on crates.io.

Next steps

Source: docs/contributing/packaging.md at main · View as Markdown