Durable job queue

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

bookclerkd admits background work as durable rows in the host library database (jobs + job_temp_paths on the same schema as holdings and accounts). HTTP and the interval scheduler are producers; leased workers claim jobs. There is no external broker and no dedicated jobs database.

The queue is host-owned. Guests never receive SQL against jobs (or any other library table). A plugin that needs its own durable store declares a named database binding; that unit is physically separate and holds only plugin-owned DDL. Job lifecycle (admit, claim, heartbeat, complete) stays host SeaORM / typed host plans on the library.

This is not a general pub/sub bus. Domain events such as book_acquired / plugin event(batch) stay off JobKind. They use a durable outbox (domain_events + event_deliveries + event_subscriber_nodes) with the same fenced-lease pattern as jobs. Acquire success publishes book_acquired with producer source on the envelope. Each host heartbeats the instances its deployments run (present or placed, whether or not the guest is up) and its loaded integrations into a per-node catalog keyed by (node_id, plugin instance id), or (node_id, PluginKey) for a plugin without an instance (plugin_id stores the canonical PluginKey, not the display alias) and does not delete other nodes’ rows. The catalog key is the node id from node-identity.json, the same id the node registers with the library and claims its node.<node_id> queue under. Dispatch uses the live union (any enabled node whose heartbeat is within 60s). Optional payload-object filters and resource_class (currently network only) are matched host-side. Each dispatcher tick dispatches at most 32 undispatched events, then always runs a wake slice (so a backlog cannot starve sleepers or the catalog heartbeat) and continues when either undispatched remain or wake still_pending. Late-join is a missing (event_id, plugin_id) anti-join paged 200, restricted to the retention window — an unchanged catalog with no missing pairs does a bounded empty SELECT and zero dispatch writes. The subscriber snapshot an event's first dispatch freezes (so its pages resume unchanged after a crash, and a retry that sends the same list replays their receipts) covers only that dispatch; a late-join adds deliveries for the subscribers it finds now. D1 dispatch receipts are per pair (dispatch-{event_id}-{plugin_id} / reconcile-{event_id}-{plugin_id}). Catalog rows follow deployments, not processes: an instance deployed on a node keeps its enabled row while its guest is not running, so its deliveries are written at publish time and wait as pending. An event fans out to every subscribed instance, and each copy is leased to one node; only jobs are claimed. Each VPS leases only deliveries for instances loaded on that process and only events its own node catalog matches (type, schema version, filter). [events] retention and concurrency are stored in the database after the first import (core.events; see configuration and ADR: Control plane). [events.concurrency] is both the local worker count and the library-wide max running deliveries per (plugin_id, resource_class) (serialized with a portable bookclerk_slots row). EventResult::suspended may set wakeOnEventType / wakeOnFilterJson; the host derives wake grants from declared subscriptions (schema versions plus the intersection of sub.filter and the requested filter — requested keys only add constraints) and wakes matching parked rows in the same account when a later event is published. Publish commits domain_events.wake_pending = 1 and returns; the dispatcher claims bounded wake slices with a unique UUID fence token (wake_lease_* + delivery cursor, at most 32 events and one page each sized from negotiated maxBinds) so producer latency does not track sleeper count. Cursor release, finish, and the sleeper UPDATE require that token in the same statement; a lost fence does not clobber another owner or a later wake registration. Accepting a wake clears the registration so a later matching event does not re-wake a retry. Duplicate retries leave the flag set until a claimed slice finishes. wakeOnEventType must be a declared subscription (empty stays timestamp-only); an empty requested filter keeps the subscription filter and cannot broaden it. Late-join skip cache is invalidated on dispatch error and reconciles at least every 60s as a backstop. GET /api/status includes event queue counts plus durable retry/suspend totals and average dispatch/handler latency. A pending delivery that was never leased expires (state expired) once it has been due for longer than its consumer's message_retention_period (default one day, capped at [events].retention_days; host and node consumers never expire), so a deployed instance that is not running does not accumulate an unbounded backlog (see Retention of undelivered events). Retention is independent of jobs ([events].retention_days for acked/rejected/expired + empty parent events after that cutoff, dead_letter_retention_days for dead letters). Prune is a library singleton, not a dispatcher tick: see Library maintenance below. Operator APIs: GET /api/events, GET /api/events/deliveries?state=dead_letter, POST /api/events/deliveries/{id}/retry, POST /api/events/deliveries/{id}/acknowledge, POST /api/events/deliveries/{id}/cancel, POST /api/events/deliveries/{id}/resume. CLI: bookclerk events list|dead-letters|retry|ack|cancel|resume. Plugin-owned queues ([queues] in plugin.toml) ride the same outbox tables under their own queue_name; GET /api/queues, GET /api/queues/{name} and bookclerk queues list|inspect <name> show per-queue counters, live consumers, recent messages and deliveries (see abi.md#queues).

See architecture.md, plugins.md, and operations.md.

Command envelope

Each row stores a versioned JSON command (JobPayload.v, currently 1). Unknown kinds, unsupported versions, and malformed JSON are rewritten to kind=invalid / state=failed / error_kind=invalid_job and never run a handler.

Dispatch goes through JobTransport (InProcessJobTransport today; a workerd adapter can replace it later). Handlers receive a JobExecCtx with the lease fence and a cooperative cancel flag.

State machine

retry with backoff terminal pending running succeeded failed cancelled
  • Admission is one atomic backend operation (a guest atomic batch in production, a local BEGIN in tests). A partial unique index on dedup_key for pending/running rows enforces active-key uniqueness. Both paths take the key from JobKind::dedup_key, so a new kind needs no second rule, and a duplicate returns the id of the job it folded into.
  • A worker claims with a conditional pending → running update and a unique per-attempt lease_generation. Lost RPCs retry the same operation_id so a replay cannot claim a second row. That replay is bounded (2s); if the database stays Unavailable, the worker returns the error, drops the job_runtime read permit so a plugin swap can proceed, and lets an ambiguously claimed row expire for reclaim. The next loop iteration uses a new operation id.
  • Heartbeat, progress, complete, fail, and reclaim are fenced on (job_id, lease_owner, lease_generation). Zero rows (Ok(false)) means the fence is lost; the worker cancels the handler and ignores the unfenced result. A heartbeat error (transient DB/RPC failure) is retried only until the last confirmed lease expiry; past that deadline the worker cancels locally and ignores the result. It must not finalize the row as cancelled unless cancel_requested is set.
  • Destination publish is not atomic with that fence. plugin_copy stream-copy is at-least-once: a lost lease can still make object bytes visible after the last heartbeat. Duplicates are absorbed by retry-stable commit_tokens. A missing stage counts as success only when the published object's commit-token metadata is that token (see storage-bounds.md). A pre-commit heartbeat does not make publication exactly-once.
  • Storefront scan jobs keep their place outside the job row, in account_scan_state (one row per store account): the guest's next-page cursor after each written page, and the sync token and time of the last finished pass. A retried or later job resumes a cursor younger than 24 hours and otherwise starts a new pass, full when no complete pass ended in seven days (see Paged scans). The same row keeps the daily quality-upgrade check's place (quality_cursor, the last holding id asked about, empty after a full walk) and time (quality_checked_at); see Upgrading to better files. The row goes away with its account.
  • Storage scans checkpoint storage_scan JSON (instance id, generation, phase, cursor) with checkpoint_running_job while the row stays running. The generation is a UUID bound in storage_scan_generations to that storage instance and the job. Local-backed scans, including fan-out and plugin wrappers, also store the node placement (the node id from node-identity.json). A resume on another node, or a backend whose placement cannot be proved, starts a new scan instead of adopting the old rows. Object rows live in storage_scan_rows, not in the 64 KiB checkpoint. Page inserts happen before the cursor advances; replaying a page is idempotent. run_acquire performs one fenced scan and matches against that index. An apply checkpoint is adopted only when its generation row still exists and is marked complete; a missing or incomplete inventory is scanned again. A successful acquire, including one with no targets, deletes the generation before the handler returns. Inventory rows and the generation row are deleted in one database transaction. Startup reclaim removes abandoned generations and does not remove one owned by a pending or running job. While the daemon is up, a bounded sweep and each confirmed terminal transition delete inventories for terminal jobs. A failed attempt that will be retried, a lost fence, and suspended work leave the generation in place. Local list indexes are node-local scratch and are not a portable checkpoint.
  • A convert job is an acquire job whose payload carries the copy to add (convert, written format:quality). Its dedup key ends in :convert=<copy>, so a convert and an acquire of the same title are separate jobs, and it is routed only to a node with Local trust. It streams the stored copy into the acquire scratch directory ({cache}/acquire/<title>/convert-source), under the same temp quota as an acquire, makes the copy there through the copy pipeline and removes the directory when it ends. Older versions converted in {cache}/convert, which the startup sweep still cleans.
  • Reclaim also requires lease_expires_at to still be null or <= now, so a heartbeat that extends the same generation cannot be stolen.
  • POST /api/jobs/{id}/cancel cancels pending immediately and flags running for a cooperative stop between acquire titles, scan sources, and listen-sync providers. Cancel retries if a worker claims the row between the pending read and the pending-only CAS, so a cancel that returns never leaves a running job unflagged. Integration scan_library RPCs are not interruptible; cancel is checked before the call and a repeat remote scan is idempotent.

Job kinds

Kind Dedup key Resource class Handler
scan scan:account={id|all}, then :instance={id} when pinned network run_scan
acquire acquire:title={id|all}:account={id|all}, then :force=1, :pdf=1, :paced=1 and :instance={id} when set network run_acquire
listen_sync listen_sync network run_listen_sync
integration_scan integration_scan:id={id}:force={0|1} network run_integration_scan
plugin_copy plugin_copy:plugin={id}:from={key}:to={key} network ABI JobRunner stream-copy (run_plugin_copy)
plugin_scheduled plugin_scheduled:instance={id}:cron={schedule} network scheduled(controller) on the instance (run_plugin_scheduled)
discover_candidates discover_candidates:subject={key} network run_discover_candidates, then admits the subject's discover_feed
discover_feed discover_feed:subject={key} network run_discover_feed (no network; ranks stored candidates)
discover_request_queue discover_request_queue network run_discover_request_queue (no network; ranks the shared request queue)
cache_sweep cache_sweep:node={id} network run_cache_sweep (no network; one node's image caches and job scratch)
search_index_rebuild search_index_rebuild:node={id} network run_search_index_rebuild (no network; rebuilds one node's search index)
library_index_rebuild library_index_rebuild network run_library_index_rebuild (no network; rewrites every row of the library's page index)
branch_request branch_request:node={id}:work={id} network none: the branch it is pinned to takes it over the node socket (requests.claim / requests.finish)
notify_send notify_send:delivery={id} network run_notify_send on a node running the notifier instance (requires it); parks itself between retries with defer_job
ingest_batch ingest_batch:batch={id} network run_ingest_batch on the node that holds the upload's bytes (pinned by payload.node_id; never handed to a branch)

plugin_scheduled jobs aren't enqueued through the API. Each daemon's cron ticker claims a due fire with a compare-and-swap on the instance's plugin_cron_cursors row and inserts the job in the same batch, with requires = [instance] so it routes to a node that runs the instance. The dedup key holds one fire per schedule, so a fire that comes due while the previous one is still active is skipped. A failed fire retries up to 3 attempts with the same scheduled_time_ms. A pinned fire can only be claimed by a node trusted with every job (a shared node, or a branch node whose owner is an Owner or Administrator). When no such node runs the instance past the requirements sweep's grace period, the sweep fails the fire as unroutable. That covers an instance deployed nowhere and one deployed only on branch nodes owned by members. See Scheduled triggers.

The two Discover kinds carry a subject key (operator, user:{id} or portal:{id}) and build that person's stored shelves. Reads, the daily scheduler and Discover's host event consumer (host:discover, which consumes wishlist_changed, listening_synced, book_acquired, holding_added and holding_removed and admits jobs with a 30 s run_after) queue them. See Precomputed shelves and Taste events.

ingest_batch stores one upload (or one drop-folder intake batch): it checks each file against the manifest, expands archives in the media worker, groups files into books, resolves each through a patron-held core.import listing and writes it to every destination. The upload routes queue it on commit with payload.ingest_batch, and a storage intake queues it with uploads::queue_intake_batch, which copies each object from the named storage instance first. Its end publishes ingest_batch_finished in the same write as the batch's final state; see Uploads.

branch_request is an interlibrary request: a user of this library asks a branch for a book it listed. The job is pinned to the branch's node (payload.node_id) and names the listed edition and the work's strong identifiers. No claim loop runs it: ordinary claims, local or remote, never hand it out, and only that branch takes it with requests.claim. Its answer completes the job (accepted), fails it without a retry (not_listed), or fails it with the normal retry (branch_failed). See Requesting a book from a branch.

discover_request_queue carries no subject: one ranking of the shared request queue serves everyone. Discover's event consumer queues it on every one of those five events (a holding event too, since a new holding can hide an entry), and a read of GET /api/request-queue queues it when the stored ranking is missing an entry or older than a day. See Request queue.

A cache_sweep job belongs to one node: JobPayload.node_id names it, and only that node's routed claim takes the job (the unrouted claim never does). Each daemon admits one for itself from its heartbeat when a sweep is due (see Cache retention), at priority 100 so a backlog of store work does not hold it back. The dedup key folds a sweep that comes due while another is queued into that one; one that comes due while a sweep runs is asked for again on the next heartbeat, so it runs under the limits in force by then. The job evicts from the node's image caches, then deletes scratch directories under {cache}/acquire, {cache}/acquire-pdf and {cache}/convert that no active job has registered and in which nothing has changed for an hour (a job may create its directory just before registering it, so a file still being written keeps its directory). The heartbeat queues it only after its own beat is recorded, so the requirements sweep never finds the job before the node's heartbeat. When the queue is full, the admission fails, or the queued job has waited five minutes for a worker, the heartbeat sweeps the image caches in the background instead.

A search_index_rebuild job is pinned to one node the same way. See Search index.

A library_index_rebuild job is not pinned: any node claims it. See Library index.

Reserved classes (no worker in this release): media, transcription, indexing.

Search index

Searches read a Tantivy works index on the node's own disk, so each daemon keeps its own copy current. GET /api/library/works?q= and bookclerk library search read it, under search_index_works/, with one document per held top-level work (its titles, authors, narrators, series and store identifiers). It is kept current two ways:

  • From events. Each node registers its own consumer, host:search-index:{node id}, on the core.events outbox for edition_linked, edition_updated, holding_added and holding_removed, all host event types no plugin may produce. The id names the node, so every node gets every event (Discover's host:discover is one consumer for the library). An edition event refreshes the document of the edition's work, and a holding event the document of its listing's work. A refresh deletes the work's document and adds it back while the work is held, so a work nobody holds any more leaves the index. A holding event whose listing is already gone removes the marker below, since its work can no longer be found, so the node rebuilds.
  • Rebuild job. A finished rebuild leaves a marker for the current document layout in the index folder. When it is missing (a new node, a deleted folder, or an index of an older layout, which is discarded), the heartbeat queues a search_index_rebuild job pinned to the node, at priority 50, after its beat is recorded. It queues at most one every ten minutes while the index stays incomplete, so a failing rebuild is not re-queued on every beat. The job reads every top-level work in pages and replaces the index in one commit.

One writer runs at a time. A delivery that finds a rebuild (or the CLI) writing is retried after 10 s, for up to an hour. A CLI rebuild (library search --rebuild-index, or the one after library scan) that finds a daemon writing waits for it, pausing up to 2 s between tries, and gives up after five minutes. Searches read the last commit, so they keep answering from the old index while a rebuild runs. A search on a node whose works index was never built (or is of an older layout) answers 503 (the CLI says to run library search --rebuild-index) until the job finishes; a search never creates or replaces an index.

Note

Earlier builds also kept a per-title index under search_index/. Nothing reads or writes it any more, so that folder can be deleted.

A works search answers in relevance order (title matches weigh most, then authors) and ignores sort; the CLI prints each match's work id and title, best first. author:, narrator:, series:, title: and identifiers: (also spelled asin:, isbn: or productid:) narrow a term to one field; identifiers match whole and ignore case. Each request reads a window of at most 1,000 matches from the cursor's position, and only those documents, then reads the matched works' rows from the library index by id in chunks, applying the other filters there. A page can therefore come back with fewer rows than limit and still carry a next_cursor. The cursor holds the query, the filters and how many matches earlier pages used, and is refused under another query or other filters. Relevance is computed again for each page, so a work indexed or dropped between two requests can make the next page repeat or skip a match.

Library index

GET /api/library/works pages one row per held work from the library_work_rows table instead of joining works, expressions, editions, listings and holdings on every request. A podcast is one row, its show, counting its episodes' editions and holdings. The table lives in the library database, so the whole library shares one index:

  • From events. Every daemon registers the same consumer, host:library-index, for edition_linked, edition_updated, edition_acquire_changed, holding_added, holding_removed and book_acquired. The catalog folds the registrations into one subscriber, so each event is handled once in the library. An edition event refreshes its edition's work, a holding event the work of its listing, and book_acquired the work of its edition (or of its title, when the payload names no edition). A refresh reads the work as it is now and writes its whole row, so a row's authors and series are as current as the last of these events: a credit or series change alone does not refresh it. A work whose last holding is gone keeps its row with a holding count of 0, which pages skip, and the row is deleted only when the work itself is. A failed refresh is retried up to ten times, then dead-lettered.
  • Rebuild job. A finished rebuild records the index layout in library_index_builds. When the current layout has no row (a new library, a database copied to another backend, or a binary that changed how rows are derived, or a change to the library's collections, which removes the row and queues the job at once), the heartbeat queues a library_index_rebuild job at priority 50. It queues at most one every ten minutes while the index stays unbuilt. The job refreshes every top-level work in pages of 100, reporting progress and stopping when cancelled, then records the build, unless the collections changed during the walk, in which case it leaves the build unrecorded so the next job starts over. A page is read in a fixed number of chunked lookups (works, editions, listings, holdings, acquire rows, credits and series for all 100 works at once) and its rows and filter values are written in shared atomic batches, so a page costs about as many round trips as one work. The works search rebuild reads its documents the same way. The ignored library_index_timing test in bookclerk-library times a rebuild of 10,000 works.

Two refreshes of one work can run at once, from two deliveries or a delivery and the rebuild. Each stamps its row with the time it started reading, and a write only replaces a row stamped no later, so a slow refresh never overwrites a newer one. Keeping an unheld work's row (rather than deleting it) keeps that stamp, so a slow refresh that read the work while it was still held cannot add it back.

Each row names a cover edition: a held edition, preferring one with a cover of its own or a store's cover on one of its listings, then an acquired one, then audio. A held work always names one, even before any cover is known, because store text and a title's enrichment are written after the link whose event refreshed the row. The route reads the cover when it serves the page: the edition's own (the first store cover a listing of it reported), else the first store cover among its listings, else its work's (or, for an episode, its show's), so a cover stored later shows without another refresh. The work detail route gives each edition its cover the same way.

Each row carries the sort keys of every order the page offers: title, first author, newest holding, series then position (works in no series last), newest reported purchase and longest held runtime (works with no purchase time or runtime last). Pages use a keyset cursor (the last row's sort values, sent back as an opaque cursor), so every page costs the same and a row added between two requests never shifts the next page. A cursor issued for one sort is refused under another.

The same refresh writes the work's filter values to library_work_facets: its authors and narrators (from credits and listings), series, genres (the work's and listings' categories, split on commas, semicolons and bars), publishers, the storefronts it is held through, its library tags (work_tags) and the collections whose storage prefixes hold one of its editions' files (core.collections), and explicit when a store marks one of its listings explicit, each under its lower-cased key. The row write, the removal of the old values and as many new values as fit share one atomic batch; the rest follow in further batches, each checking that the row still carries this refresh's stamp, so a later refresh's values are never mixed with an older one's. facet and value on GET /api/library/works filter by one value, and GET /api/library/works/facets/{facet} pages a facet's values with how many held works have each. Tags and progress are per person, so they are read from the caller's own user_work_tags and user_title_state rows instead.

Admission and backpressure

POST /api/library/scan, /acquire, /api/discover/sync-listening, and /integrations/{id}/scan (and the scheduler) call LibraryStore::enqueue_job:

Outcome HTTP Meaning
Created 200 New row; worker is notified
Duplicate 409 Same dedup key already pending or running; job_id is the existing row
QueueFull 429 pending + running >= jobs.max_pending
Unroutable 409 error: "unroutable": the job's required instances share no node (see job routing below); no row is written

A scan goes through the scan planner first (control_plane::plan_scan). An operator, an Owner, an Administrator, or the scheduler scanning every account gets one scan job per scan-enabled account id, so the response also carries job_ids (200 when any was accepted, 409 when all were already queued, 429 when the queue was full, 422 nothing_to_scan for an empty plan); with no scan-enabled account it stays one scan:account=all job. The scan endpoint takes the operator, Owners and Administrators only: any other portal user is 401, and an operator impersonating one is 403. The planner also scopes any other user to the accounts linked to them (naming someone else's account is forbidden), but no HTTP caller reaches that path yet.

Admission, claim, and handler execution share an RwLock (job_runtime). A database plugin swap takes the write permit so it cannot observe an idle worker that is about to claim. Admission waits up to 15s for a read permit and returns 503 if a swap holds the lock too long.

Admission, claim, and scratch-quota updates take a write lock on a bookclerk_slots row (job-queue) so COUNT then INSERT cannot exceed max_pending under READ COMMITTED on every required backend. The required postgres job queue CI job runs those concurrency tests against a disposable multi-connection database (BOOKCLERK_TEST_POSTGRES_URL + BOOKCLERK_REQUIRE_POSTGRES_TESTS=1). They are #[ignore] in the default workspace suite so a missing Postgres cannot false-pass. TOTP enroll/disable atomic conformance (postgres_totp_*) is not ignored: the same job provisions Postgres and runs those tests automatically (BOOKCLERK_REQUIRE_POSTGRES_TESTS=1).

Claim (native and D1) marks pending rows with malformed JSON, an unknown kind, an unknown resource_class, or an unsupported envelope version as invalid_job before class-specific selection, so a bad row cannot occupy max_pending forever or abort a D1 batch.

Planners and the CLI

Every scan and acquire is admitted through a planner, whoever asks:

  • Scan (control_plane::plan_scan): operator authority, an Owner or an Administrator scanning every account gets one job per scan-enabled account; anyone else only the accounts they own (see the control-plane ADR, Per-item trust and the scan planner).
  • Acquire (control_plane::plan_acquire): each named title id (titles, UUID, ASIN, ISBN or product id) becomes its own job, so each can go to any node trusted with it. Operator authority, an Owner or an Administrator acquiring everything gets one unsplit job. Anyone else must own the account they name, when they name one, and every account that holds a title a named id matches (a listing's product id, an ASIN or ISBN of its edition, or a holding id; 403 otherwise). An account is named by its id or label, ignoring ASCII case, and a job matches it the same way when it runs. With neither an account nor a title, they get one job per account they wholly own. A node is trusted with an acquire the same way: by the accounts holding the titles its filters match. POST /api/library/acquire takes {account, plugin_instance_id, titles, editions, force, pdf_only} and answers a multi-job plan like a scan (job_ids; 200, 409, 429, or 422 nothing_to_acquire for an empty plan).
  • editions names editions (the ids a work's detail lists). Each becomes one title: of the library titles that sell it, the first in an account the caller may acquire for (and in account when that is set). The edition's titles share its files, so one is enough. An edition with no title (or none in account) is 400; one whose titles are all in other people's accounts is 403. The title is then planned like any other.
  • force (re-download acquired titles) and pdf_only (companion PDFs only) are part of the acquire's dedup key and run only on shared nodes; a host is never handed such a job. A pinned instance is part of the key too, so a pinned acquire never stands in for an unpinned one or another pin.

The CLI bookclerk library scan and library acquire do not run guests. They plan as operator authority (the CLI holds the database), record each job's routing requirements, insert the rows, and then wait for them, printing state changes to stderr; the first Ctrl-C asks unfinished jobs to cancel, a second stops waiting. Waiting needs a live bookclerkd node of the library (the CLI fails before queueing otherwise); --no-wait only queues and prints the job ids. library scan --account <id> is still exactly one job for that account (bypassing scan_enabled), and library acquire --asin <ASIN> exactly one job for that title. --match-storage and a rebuild of the CLI machine's search index still run in the CLI after the scans succeed; daemons keep their own index current from events (see Search index). library acquire --dry-run plans keys locally without queueing. library acquire has no --license or -o key=value flags (both were removed, never implemented); a queued acquire runs with the settings of the node that claims it. A branch node's operator asks its library for a scan with bookclerk libraries scan (Libraries and branch nodes).

Leases and crash recovery

  • Default lease is 60s (jobs.lease_seconds); the worker heartbeats at lease/3. Ok(false) (fence lost) sets the handler cancel flag. A heartbeat Err is logged and retried only until the last confirmed Ok(true) (or the original claim) would have expired; after that the worker treats the result as unfenced and ignores it. Finalization calls fail_job(..., "cancelled") only when cancel_requested is set. A local cancel without that flag is treated as fence loss and ignored.
  • Event delivery workers use the same 60s lease and lease/3 heartbeat during event(batch). Fence loss cancels the guest RPC and ignores the result. Leases are restricted to instances loaded on this process; releasing an unexecuted lease does not consume attempt_count. Expired-lease reclaim restores resume_pending when a checkpoint exists so a crash during resume does not burn an attempt.
  • Startup and each worker tick call reclaim_expired_leases.
  • Books left queued / downloading with no running acquire job are set to error (orphaned_after_restart). The next acquire job retries them.
  • Scratch dirs under {cache}/acquire and {cache}/acquire-pdf are reserved against jobs.temp_quota_bytes (default 2 GiB) and unregistered only after that path is deleted (or already gone). Startup sweeps unregistered orphans, and each node's cache_sweep job sweeps those in which nothing has changed for an hour. Fetch, remux/transcode, and companion-PDF downloads (PDF-only and sidecars discovered during audio acquire) are bounded by the remaining quota: sources can call FetchOptions::enforce_cache_budget, the pipeline watchdog cancels a stage that crosses the budget, and PDF bodies are streamed with an explicit remaining-byte cap.

Scan, listen-sync, and integration-scan retries are idempotent (upserts / remote rescan). Acquire retries skip titles already acquired, except when download_pdf is on and the companion PDF is not stored (pdf_status is not acquired, or pdf_storage_key is missing/empty). Those rows stay in the job target list so a later acquire can resume the PDF-only side effect. A companion PDF that exceeds the remaining scratch budget is soft-failed (pdf_status=error) so the audio acquire can succeed; a store that answers the resumed PDF with rateLimited fails the title instead, so the job records the account's cool-down (see Download pacing). Dedicated acquire --pdf still fails the job when the PDF cannot be stored; that path always unregisters its acquire-pdf scratch reservation, including on fetch/quota errors.

Download pacing

Stores throttle an account that downloads too much, usually without publishing the limit, so the host paces the downloads it starts itself (bookclerk_library::pacing, run_acquire). The pace is per store account, keyed (plugin_key, account_id) like accounts themselves, so it holds across plugin instances and every node of the cluster.

  • Limits. A storefront declares a daily budget, counted over any rolling 24 hours, and a minimum gap between downloads in its manifest ([acquire]). The person an account is linked to may override them on the Accounts page (POST /api/portal/sources/{source}/connections/{account_id}/pace, whose body is at most 4096 bytes), and an operator with bookclerk library set-pace: the store's pace, a custom budget and gap, or no limit. The override is stored in account_pace.
  • Cool-down. When fetchTitle answers rateLimited, the host records a cool-down on the account until the store's retryAfterMs, or for the storefront's cooldown_seconds (one hour when it declares none), capped at a week. Automatic downloads on the account wait for it, even with no limit set, until it ends or someone clears it. The Audible plugin answers rateLimited when Audible refuses a license because it is throttling the account, and declares a 24-hour cool-down.
  • Who waits. An acquire job whose payload has paced set waits for the pace: scheduled auto-acquire and the acquire a branch queues for a request it accepted from a library (which is how "Acquire all of this branch's members' books" reaches it). A paced title that fails and is retried waits for the pace again before the retry asks the store, and is set aside like any other when the wait is long. A paced run tries audio not yet downloaded first, then missing companion PDFs of acquired titles, then titles that failed before, so a title that keeps failing or has no PDF does not spend the budget of the audio behind it. An acquire a person starts in the web app or CLI leaves paced unset: its downloads count against the budget but are never held, and when the store answers rateLimited the job records the cool-down and fails for good with the store's answer (it does not go back to the queue for another attempt) instead of retrying into the throttle. paced is part of the dedup key, so a hand-started acquire is never folded into a waiting automatic one.
  • Ledger. Every title Bookclerk may have fetched from the store is a row in account_downloads (start time, job, paced or not), once per attempt that asked the store, at the time it started, so a retried title counts each try and its retry moves the gap on. Each row's id is made once per download, so a write retried after a lost answer adds no second row. That includes a title that failed after its fetch (converting or storing it), since the store download already happened, and a store download that a better copy from a joined library then replaced, and a missing companion PDF fetched for a title whose audio was already stored (AcquireResult::store_fetches). A library copy from another store that replaced the download and then failed packaging sends the acquire back to the store, and that second download is a second row. A copy matched in storage or taken straight from a joined library is not counted, nor is a title no destination takes or one the store refused with rateLimited. A title a Host downloaded for a hand-started acquire is recorded in the same batch that commits its upload on the owner (control_plane::commit_upload), so the two never part, and once per upload however often the commit is repeated. A ledger write that fails is retried, and a paced run that still cannot record a download holds that account rather than download past a budget it cannot count; the job fails if rows are still unwritten at its end. Backend migration copies the ledger with account_pace, so a migrated library keeps its rolling window. Rows older than two days are pruned when the account records its next download.
  • Where it runs. A paced acquire is claimed only by a node that holds the library database (control_plane::claimant_may_run), like a forced or PDF-only one: a Host runs handed acquires without the library, so it could not check the pace.
  • Changing the pace. Setting an account's pace or clearing its pause wakes every parked paced acquire that names that account or no account (LibraryStore::wake_paced_acquires), so it checks the new pace at once instead of sleeping out the old deadline. A pace body with no mode and clear_cooldown: true only clears the pause and leaves the stored override as it is. A run that parks checks each held account once more after parking and wakes itself if a change made while it ran lets the account download sooner.

A paced run checks the account's pace before each title, while it holds the account's cluster semaphore, so no other node downloads on the account between the check and the fetch. A wait of up to two minutes (a short gap) is slept in place, checking for cancel each second. A longer one sets that account's titles aside, and the run goes on with the other accounts' titles. At the end, a run that set titles aside is parked (LibraryStore::defer_job): it goes back to pending with run_after at the moment the first held account may download again and its progress saying why ("3 title(s) waiting for the daily download budget until …"). Parking does not spend an attempt and drops the storage-scan checkpoint, so the next claim starts fresh and skips titles already acquired. An auto-acquire of a large library this way drains over several days, and the next scheduled auto-acquire is folded into the parked job by its dedup key.

Routing to a compatible node

Every daemon sharing the database runs workers, so a job must say what the node that runs it needs. At admission the daemon records JobPayload.requires: the plugin instance ids the claiming node must run (control_plane::resolve_job_requirements).

  • A job pinned to an instance (plugin_instance_id) requires it, when some node has a deployment of it that should run. A pin to an instance deployed nowhere is not recorded, so the job is claimed anywhere and fails there as before instead of waiting forever.
  • An acquire requires every deployed instance of a node-bound output (today the verified first-party local destination), because it writes to every deployed destination and a local output is the disk of the nodes it is deployed to. If plugin discovery fails on the admitting node (say an unrelated plugin has a duplicate alias), every instance whose plugin key names the first-party local package is required instead, unverified, so the acquire is never left claimable by a node without those outputs.
  • Everything else (unpinned scans, listen sync, integration scans, plugin_copy) has no requirements and is claimable on any node.

The required instances must share a node where all of them are deployed to run (control_plane::common_deployment_nodes); otherwise admission refuses the job with a conflict (HTTP 409 unroutable) instead of leaving a row no node can ever claim. Required instances are never dropped to make a node fit.

The daemon's worker claims through control_plane::claim_routed_job: it pages through the ready pending jobs of its class, best first (16 per page, keyset-ordered by priority, created_at, id, until it claims one or the rows run out, so jobs this node cannot run never hide runnable ones behind them), and skips one unless every required instance has a deployment on this node whose desired state runs it (present or placed) and whose observation from this node says the guest is up (running or healthy) — the same placement and observation rows delivery routing reads. Then it takes the job's library permits (below) and claims that row with claim_job_by_id, one CAS on state = 'pending' and the lease generation it read, so concurrent claimers on any number of connections admit one. A lost CAS gives the permits back and tries the next candidate. A job whose account or instance semaphore is full is skipped too; a full class semaphore ends the attempt. Requirements recorded at admission are kept current by the requirements sweep below.

Requirements sweep

Deployments keep moving after a job is queued: a local output is deployed or retired, or the scheduler moves a replica. Each daemon runs a sweep loop (bookclerkd requirements_sweep, rules in control_plane::sweep_job_requirements), but only the holder of the jobs:requirements-sweep library lease (90 s, renewed every pass) sweeps. A pass runs every 30 s, on a deployment or install wakeup on this node, and after a scheduler pass that moved replicas. A wakeup waits until wakeups have been quiet for 0.5 s (at most 3 s), so a burst of deployment changes, such as enabling several plugins at once, is swept once. Discovery runs on a blocking thread. It re-resolves the requirements of every pending acquire, pinned job, or job that already has requirements, with the node-bound output keys the daemon discovers plus every instance keyed as the first-party local destination (so a sweeping node without that plugin, or whose discovery failed, never strips them; a pass that cannot read the instance list is skipped), and:

  • applies at once a set that is unchanged or only gains instances when some node is deployed to run all of them (a newly deployed local output is added to queued acquires), clearing any unsettled mark;
  • otherwise (unsettled: no node runs all of them, or a recorded instance is no longer deployed anywhere) stamps JobPayload.requires_unsettled_since_ms on the first sweep and drops no recorded instance for a 10-minute grace period, so a placement move that retires the old replica before the new one is up never fails queued work. Instances the job gained are still recorded at once (the recorded set becomes the union of old and new), so a job that waits cannot be claimed on its old set and skip a newly deployed local output;
  • when the grace period has passed, writes the new set if some node can run it (an instance was only dropped, which is what admission would record now), or fails the job with error_kind = "unroutable" and a message naming the instances. A failed job frees its max_pending slot and dedup key.

A job pinned to a node (JobPayload.node_id) is failed as unroutable when that node has never sent a heartbeat, or its last one is older than the grace period, so a removed or long-stopped node does not keep its queued work. While the node is live, any requirements the job has are reconciled like any other job's.

The grace period is a wall-clock stamp rather than a count of unsettled sweeps, because sweeps also run on wakeups and a count would expire faster on a busy library. Every write is a compare-and-swap on state = 'pending' and the lease generation, which it bumps: a job claimed meanwhile is left alone, and a claimer that read the row before the sweep wrote it loses its own CAS (a routed claim matches the generation of the record whose requirements it checked) and moves on to re-read it on a later pass. A wakeup on a node that does not hold the lease does nothing; the holder's timer picks the change up within 30 s.

Before requirements and permits, the routed claim filters candidates through the trust predicate (control_plane::trust). The daemon's own worker holds the database and is trusted with every job. A Branch node (H-4) owned by a user who is not an Owner or Administrator may claim only a scan or acquire of a named account linked to that user; a scan-all, any pinned or instance-bound job, listen sync, integration scans, and plugin copies are refused to it. control_plane::handoff_routed_job repeats the check before a job's secrets would be handed over and, if the node lost that trust, returns the job to pending without spending an attempt. It also hands nothing over unless the claim still owns the job's running, unexpired lease (same lease owner and generation), so a late handoff after a reclaim, completion or expiry unseals nothing. See Per-item trust and the scan planner in docs/adr/control-plane.md.

The unrouted claim_next_job (and its guest atomic plan) never claims a job with requirements; it remains for tests and tools that run one process. It pages past routed rows the same way (the atomic plan filters them out before its LIMIT 1).

Library semaphores

Process-local limits add up across nodes, so the limits that protect shared things are counted library-wide. A semaphore name with limit N is the library_leases rows sem:<name>#0 … sem:<name>#<N-1>; a permit is one of those slot leases (holder, fencing token, expiry), taken with the same fenced compare-and-swap as the scheduler lease (control_plane::semaphore). Only slots below the limit are taken, so no more than N permits are live at once. A routed claim takes, all or none, one permit of:

Semaphore Limit Counts
jobs:<class> [jobs.concurrency].network Running jobs of the class, library-wide
plugin-instance:<id> [jobs.concurrency].plugin_instance (default 1) Running jobs that require that instance
account:<plugin_key>:<account_id> 1 Scans and acquires touching that store account: the payload's account, else every account (of the pinned plugin, when pinned)

Permits last as long as the job lease, are renewed with each heartbeat, and are released after the row is finalized. A routed claim's holder is <lease owner>/<job id>/g<generation>/<random>, with the lease generation the claim runs at (the CAS only succeeds one past the generation it read), so a node that finds its permits again by holder prefix never mistakes a leftover permit of an earlier claim of the same job for the current one. A heartbeat whose permit renewal matches no row (another holder took the slot) is a lost fence: the handler is cancelled and its result ignored. When a node dies, its permits and job lease expire; the next claimer reclaims the row, takes the slot with a larger token, and the old holder can neither renew the permit nor finalize the job (the job's lease generation moved on too). Lowering a limit does not revoke slots above it; live permits on those slots still count, so no new permit is granted until the live count is under the new limit. (That count is read before the slot CAS, so racing claimers can briefly overshoot the new limit, never the old one, until the high slots drain.) Each node reads the limits from its own config.toml ([jobs] is not a database document yet), so nodes should agree on them.

Resource-class concurrency

[jobs.concurrency].network is the number of leased network workers on each node (default 1) and also the library-wide jobs:network permit count above, so adding nodes does not multiply concurrent store traffic. The historical work_lock still wraps run_* so the shared StorageIndex stays single-writer within one process. Codec work stays in the [media].workers pool.

Adding a job kind

  1. Add a JobKind variant, envelope fields, and dedup-key rule in crates/bookclerk-library/src/models.rs.
  2. Add a JobCommand arm on InProcessJobTransport.
  3. Enqueue from the API and/or scheduler with the same EnqueueJobSpec.
  4. Reserve scratch paths via LibraryStore::reserve_job_temp_path when the job creates temp files; unregister one path after it is gone.
  5. Add concurrent admission/claim, fence-loss, and restart tests.

Configuration

See configuration.md ([jobs] / BOOKCLERK_JOBS_* and [events] / BOOKCLERK_EVENTS_*).

Library maintenance

Library-wide cleanup runs on one daemon at a time. Each duty holds its own library_leases row; every daemon polls them every 60 s, starting 5 minutes after it starts so the first pass doesn't compete with startup. The holder renews its leases on each poll and runs a duty when it takes the lease and then hourly. A daemon that stops polling loses its leases after 180 s and the next daemon to poll takes them over with a larger fencing token. Before each delete the holder renews the lease with a compare-and-swap on its holder and token (the fence), so a daemon that stalled past its expiry is refused before it deletes anything.

Lease Duty
maintenance.jobs Terminal jobs older than jobs.retention_days
maintenance.events Expires never-leased deliveries past their retention period, then deletes terminal deliveries, empty parent events, stale subscriber catalog rows ([events] retention)
maintenance.kv Expired plugin KV rows (already read as missing)
maintenance.sessions Expired operator sessions, OIDC RP states, WebAuthn challenges
maintenance.records Expired credentials and spent tokens, finished uploads, abandoned scan inventories, and old audit rows (windows below)

The maintenance.records windows are fixed in crates/bookclerk-library/src/store/retention.rs, not configured. Each is longer than the row can still be used; the grace periods only keep recent history visible:

Rows Deleted when
OIDC authorization codes, portal sessions Past expires_at
OIDC refresh tokens Past expires_at, or revoked
Claim tickets, user invites, node enrollment links 30 days past expires_at (expired and redeemed ones stay listed until then)
Committed or aborted host uploads 1 day past expires_at; open uploads are kept, since an abort still reads them, and so is a branch's catalog upload while a file record names it as writer, since that is how its files are known to be the branch's
Storage-scan inventories Job terminal or gone; a jobless (CLI) scan silent for 1 day
security_audit_events, configuration_audit 365 days old
configuration_changes 30 days old (the document and its audit row keep the content)
notifications with their delivery log and inbox rows 30 days old

The prunes are idempotent, so the fence bounds wasted work rather than guarding correctness: the window between a fence check and its delete is one statement. Node-local cleanup (scratch directories, expired job and delivery leases) still runs on every node. GET /api/library-status lists every lease with its holder, token, and whether it is held.

Source: docs/internals/jobs.md at main · View as Markdown