Durable job queue
bookclerkd admits background work as durable rows in the host library
database (jobs + job_temp_paths on the same schema as holdings and
accounts). HTTP and the interval scheduler are producers; leased workers
claim jobs. There is no external broker and no dedicated jobs database.
The queue is host-owned. Guests never receive SQL against jobs (or any other
library table). A plugin that needs its own durable store declares a named
database binding; that unit is physically separate and holds only plugin-owned
DDL. Job lifecycle (admit, claim, heartbeat, complete) stays host SeaORM /
typed host plans on the library.
This is not a general pub/sub bus. Domain events such as book_acquired /
plugin event(batch) stay off JobKind. They use a durable outbox
(domain_events + event_deliveries + event_subscriber_nodes) with
the same fenced-lease pattern as jobs. Acquire success publishes
book_acquired with producer source on the envelope. Each host heartbeats the instances its deployments
run (present or placed, whether or not the guest is up) and its loaded
integrations into a per-node catalog keyed by
(node_id, plugin instance id), or (node_id, PluginKey) for a plugin
without an instance (plugin_id stores the canonical PluginKey, not the
display alias) and does not delete other nodes’ rows. The catalog key is
the node id from node-identity.json, the same id the node registers with
the library and claims its node.<node_id> queue under. Dispatch uses
the live union (any enabled node whose heartbeat is within 60s). Optional
payload-object filters and resource_class (currently network only) are
matched host-side. Each dispatcher tick dispatches at most 32 undispatched
events, then always runs a wake slice (so a backlog cannot starve sleepers or
the catalog heartbeat) and continues when either undispatched remain or
wake still_pending. Late-join is a missing (event_id, plugin_id) anti-join
paged 200, restricted to the retention window — an unchanged catalog with no
missing pairs does a bounded empty SELECT and zero dispatch writes. The
subscriber snapshot an event's first dispatch freezes (so its pages resume
unchanged after a crash, and a retry that sends the same list replays their
receipts) covers only that dispatch; a late-join adds deliveries for the
subscribers it finds now. D1
dispatch receipts are per pair (dispatch-{event_id}-{plugin_id} /
reconcile-{event_id}-{plugin_id}). Catalog rows
follow deployments, not processes: an instance deployed on a node keeps
its enabled row while its guest is not running, so its deliveries are
written at publish time and wait as pending. An event fans out to every
subscribed instance, and each copy is leased to one node; only jobs are
claimed. Each VPS leases only deliveries for instances loaded on that
process and only events its own node catalog matches (type, schema
version, filter). [events] retention and concurrency are stored in the database after the
first import (core.events; see configuration and
ADR: Control plane). [events.concurrency] is both the
local worker count and the
library-wide max running deliveries per (plugin_id, resource_class)
(serialized with a portable bookclerk_slots row).
EventResult::suspended may set wakeOnEventType / wakeOnFilterJson; the
host derives wake grants from declared subscriptions (schema versions plus the
intersection of sub.filter and the requested filter — requested keys only
add constraints) and wakes matching parked rows in the same account when a
later event is published. Publish commits domain_events.wake_pending = 1 and
returns; the dispatcher claims bounded wake slices with a unique UUID fence
token (wake_lease_* + delivery cursor, at most 32 events and one page
each sized from negotiated maxBinds)
so producer latency does not track sleeper count. Cursor release, finish,
and the sleeper UPDATE require that token in the same statement; a lost
fence does not clobber another owner or a later wake registration.
Accepting a wake clears the registration so a later matching event does not
re-wake a retry. Duplicate retries leave the flag set until a claimed
slice finishes. wakeOnEventType must be a declared subscription (empty stays
timestamp-only); an empty requested filter keeps the subscription filter and
cannot broaden it. Late-join skip cache is invalidated on dispatch error and
reconciles at least every 60s as a backstop.
GET /api/status includes event queue counts plus durable retry/suspend totals
and average dispatch/handler latency. A pending delivery that was never
leased expires (state expired) once it has been due for longer than its
consumer's message_retention_period (default one day, capped at
[events].retention_days; host and node consumers never expire), so a
deployed instance that is not running does not accumulate an unbounded
backlog (see
Retention of undelivered events).
Retention is independent of jobs
([events].retention_days for acked/rejected/expired + empty parent events after that
cutoff, dead_letter_retention_days for dead letters). Prune is a
library singleton, not a dispatcher tick: see Library maintenance below.
Operator APIs: GET /api/events,
GET /api/events/deliveries?state=dead_letter,
POST /api/events/deliveries/{id}/retry,
POST /api/events/deliveries/{id}/acknowledge,
POST /api/events/deliveries/{id}/cancel,
POST /api/events/deliveries/{id}/resume. CLI:
bookclerk events list|dead-letters|retry|ack|cancel|resume. Plugin-owned
queues ([queues] in plugin.toml) ride the same outbox tables under their
own queue_name; GET /api/queues, GET /api/queues/{name} and
bookclerk queues list|inspect <name> show per-queue counters, live
consumers, recent messages and deliveries (see
abi.md#queues).
See architecture.md, plugins.md, and operations.md.
Each row stores a versioned JSON command (JobPayload.v, currently 1).
Unknown kinds, unsupported versions, and malformed JSON are rewritten to
kind=invalid / state=failed / error_kind=invalid_job and never run a
handler.
Dispatch goes through JobTransport (InProcessJobTransport today; a workerd
adapter can replace it later). Handlers receive a JobExecCtx with the lease
fence and a cooperative cancel flag.
- Admission is one atomic backend operation (a guest atomic batch in
production, a local
BEGINin tests). A partial unique index ondedup_keyforpending/runningrows enforces active-key uniqueness. Both paths take the key fromJobKind::dedup_key, so a new kind needs no second rule, and a duplicate returns the id of the job it folded into. - A worker claims with a conditional
pending→runningupdate and a unique per-attemptlease_generation. Lost RPCs retry the sameoperation_idso a replay cannot claim a second row. That replay is bounded (2s); if the database staysUnavailable, the worker returns the error, drops thejob_runtimeread permit so a plugin swap can proceed, and lets an ambiguously claimed row expire for reclaim. The next loop iteration uses a new operation id. - Heartbeat, progress, complete, fail, and reclaim are fenced on
(job_id, lease_owner, lease_generation). Zero rows (Ok(false)) means the fence is lost; the worker cancels the handler and ignores the unfenced result. A heartbeat error (transient DB/RPC failure) is retried only until the last confirmed lease expiry; past that deadline the worker cancels locally and ignores the result. It must not finalize the row ascancelledunlesscancel_requestedis set. - Destination publish is not atomic with that fence.
plugin_copystream-copy is at-least-once: a lost lease can still make object bytes visible after the last heartbeat. Duplicates are absorbed by retry-stablecommit_tokens. A missing stage counts as success only when the published object'scommit-tokenmetadata is that token (see storage-bounds.md). A pre-commit heartbeat does not make publication exactly-once. - Storefront
scanjobs keep their place outside the job row, inaccount_scan_state(one row per store account): the guest's next-page cursor after each written page, and the sync token and time of the last finished pass. A retried or later job resumes a cursor younger than 24 hours and otherwise starts a new pass, full when no complete pass ended in seven days (see Paged scans). The same row keeps the daily quality-upgrade check's place (quality_cursor, the last holding id asked about, empty after a full walk) and time (quality_checked_at); see Upgrading to better files. The row goes away with its account. - Storage scans checkpoint
storage_scanJSON (instance id, generation, phase, cursor) withcheckpoint_running_jobwhile the row staysrunning. The generation is a UUID bound instorage_scan_generationsto that storage instance and the job. Local-backed scans, including fan-out and plugin wrappers, also store the node placement (the node id fromnode-identity.json). A resume on another node, or a backend whose placement cannot be proved, starts a new scan instead of adopting the old rows. Object rows live instorage_scan_rows, not in the 64 KiB checkpoint. Page inserts happen before the cursor advances; replaying a page is idempotent.run_acquireperforms one fenced scan and matches against that index. Anapplycheckpoint is adopted only when its generation row still exists and is marked complete; a missing or incomplete inventory is scanned again. A successful acquire, including one with no targets, deletes the generation before the handler returns. Inventory rows and the generation row are deleted in one database transaction. Startup reclaim removes abandoned generations and does not remove one owned by a pending or running job. While the daemon is up, a bounded sweep and each confirmed terminal transition delete inventories for terminal jobs. A failed attempt that will be retried, a lost fence, and suspended work leave the generation in place. Local list indexes are node-local scratch and are not a portable checkpoint. - A convert job is an
acquirejob whose payload carries the copy to add (convert, writtenformat:quality). Its dedup key ends in:convert=<copy>, so a convert and an acquire of the same title are separate jobs, and it is routed only to a node with Local trust. It streams the stored copy into the acquire scratch directory ({cache}/acquire/<title>/convert-source), under the same temp quota as an acquire, makes the copy there through the copy pipeline and removes the directory when it ends. Older versions converted in{cache}/convert, which the startup sweep still cleans. - Reclaim also requires
lease_expires_atto still be null or<= now, so a heartbeat that extends the same generation cannot be stolen. POST /api/jobs/{id}/cancelcancelspendingimmediately and flagsrunningfor a cooperative stop between acquire titles, scan sources, and listen-sync providers. Cancel retries if a worker claims the row between the pending read and the pending-only CAS, so a cancel that returns never leaves a running job unflagged. Integrationscan_libraryRPCs are not interruptible; cancel is checked before the call and a repeat remote scan is idempotent.
| Kind | Dedup key | Resource class | Handler |
|---|---|---|---|
scan |
scan:account={id|all}, then :instance={id} when pinned |
network |
run_scan |
acquire |
acquire:title={id|all}:account={id|all}, then :force=1, :pdf=1, :paced=1 and :instance={id} when set |
network |
run_acquire |
listen_sync |
listen_sync |
network |
run_listen_sync |
integration_scan |
integration_scan:id={id}:force={0|1} |
network |
run_integration_scan |
plugin_copy |
plugin_copy:plugin={id}:from={key}:to={key} |
network |
ABI JobRunner stream-copy (run_plugin_copy) |
plugin_scheduled |
plugin_scheduled:instance={id}:cron={schedule} |
network |
scheduled(controller) on the instance (run_plugin_scheduled) |
discover_candidates |
discover_candidates:subject={key} |
network |
run_discover_candidates, then admits the subject's discover_feed |
discover_feed |
discover_feed:subject={key} |
network |
run_discover_feed (no network; ranks stored candidates) |
discover_request_queue |
discover_request_queue |
network |
run_discover_request_queue (no network; ranks the shared request queue) |
cache_sweep |
cache_sweep:node={id} |
network |
run_cache_sweep (no network; one node's image caches and job scratch) |
search_index_rebuild |
search_index_rebuild:node={id} |
network |
run_search_index_rebuild (no network; rebuilds one node's search index) |
library_index_rebuild |
library_index_rebuild |
network |
run_library_index_rebuild (no network; rewrites every row of the library's page index) |
branch_request |
branch_request:node={id}:work={id} |
network |
none: the branch it is pinned to takes it over the node socket (requests.claim / requests.finish) |
notify_send |
notify_send:delivery={id} |
network |
run_notify_send on a node running the notifier instance (requires it); parks itself between retries with defer_job |
ingest_batch |
ingest_batch:batch={id} |
network |
run_ingest_batch on the node that holds the upload's bytes (pinned by payload.node_id; never handed to a branch) |
plugin_scheduled jobs aren't enqueued through the API. Each daemon's cron
ticker claims a due fire with a compare-and-swap on the instance's
plugin_cron_cursors row and inserts the job in the same batch, with
requires = [instance] so it routes to a node that runs the instance. The
dedup key holds one fire per schedule, so a fire that comes due while the
previous one is still active is skipped. A failed fire retries up to 3
attempts with the same scheduled_time_ms. A pinned fire can only be
claimed by a node trusted with every job (a shared node, or a branch node
whose owner is an Owner or Administrator). When no such node runs the instance
past the requirements sweep's grace period, the sweep fails the fire as
unroutable. That covers an instance deployed nowhere and one deployed only
on branch nodes owned by members. See
Scheduled triggers.
The two Discover kinds carry a subject key (operator, user:{id} or
portal:{id}) and build that person's stored shelves. Reads, the daily
scheduler and Discover's host event consumer (host:discover, which
consumes wishlist_changed, listening_synced, book_acquired,
holding_added and holding_removed and admits jobs with a 30 s run_after) queue them. See
Precomputed shelves and
Taste events.
ingest_batch stores one upload (or one drop-folder intake batch): it
checks each file against the manifest, expands archives in the media
worker, groups files into books, resolves each through a patron-held
core.import listing and writes it to every destination. The upload
routes queue it on commit with payload.ingest_batch, and a storage
intake queues it with uploads::queue_intake_batch, which copies each
object from the named storage instance first. Its end publishes
ingest_batch_finished in the same write as the batch's final state; see
Uploads.
branch_request is an interlibrary request: a user of this library asks
a branch for a book it listed. The job is pinned to the branch's node
(payload.node_id) and names the listed edition and the work's strong
identifiers. No claim loop runs it: ordinary claims, local or remote,
never hand it out, and only that branch takes it with requests.claim.
Its answer completes the job (accepted), fails it without a retry
(not_listed), or fails it with the normal retry (branch_failed). See
Requesting a book from a branch.
discover_request_queue carries no subject: one ranking of the shared
request queue serves everyone. Discover's event consumer queues it on
every one of those five events (a holding event too, since a new holding
can hide an entry), and a read of GET /api/request-queue queues it when
the stored ranking is missing an entry or older than a day. See
Request queue.
A cache_sweep job belongs to one node: JobPayload.node_id names it,
and only that node's routed claim takes the job (the unrouted claim never
does). Each daemon admits one for itself from its heartbeat when a sweep is
due (see Cache retention), at
priority 100 so a backlog of store work does not hold it back. The dedup key
folds a sweep that comes due while another is queued into that one; one
that comes due while a sweep runs is asked for again on the next heartbeat,
so it runs under the limits in force by then. The job evicts from the node's image caches, then deletes scratch
directories under {cache}/acquire, {cache}/acquire-pdf and
{cache}/convert that no active job has registered and in which nothing
has changed for an hour (a job may create its directory just before
registering it, so a file still being written keeps its directory). The heartbeat queues it only after its own beat is
recorded, so the requirements sweep never finds the job before the node's
heartbeat. When the queue is full, the admission fails, or the queued job
has waited five minutes for a worker, the heartbeat sweeps the image caches
in the background instead.
A search_index_rebuild job is pinned to one node the same way. See
Search index.
A library_index_rebuild job is not pinned: any node claims it. See
Library index.
Reserved classes (no worker in this release): media, transcription,
indexing.
Searches read a Tantivy works index on the node's own disk, so each
daemon keeps its own copy current. GET /api/library/works?q= and
bookclerk library search read it, under search_index_works/, with one
document per held top-level work (its titles, authors, narrators, series
and store identifiers). It is kept current two ways:
- From events. Each node registers its own consumer,
host:search-index:{node id}, on thecore.eventsoutbox foredition_linked,edition_updated,holding_addedandholding_removed, all host event types no plugin may produce. The id names the node, so every node gets every event (Discover'shost:discoveris one consumer for the library). An edition event refreshes the document of the edition's work, and a holding event the document of its listing's work. A refresh deletes the work's document and adds it back while the work is held, so a work nobody holds any more leaves the index. A holding event whose listing is already gone removes the marker below, since its work can no longer be found, so the node rebuilds. - Rebuild job. A finished rebuild leaves a marker for the current
document layout in the index folder. When it is missing (a new node, a
deleted folder, or an index of an older layout, which is discarded), the
heartbeat queues a
search_index_rebuildjob pinned to the node, at priority 50, after its beat is recorded. It queues at most one every ten minutes while the index stays incomplete, so a failing rebuild is not re-queued on every beat. The job reads every top-level work in pages and replaces the index in one commit.
One writer runs at a time. A delivery that finds a rebuild (or the CLI)
writing is retried after 10 s, for up to an hour. A CLI rebuild
(library search --rebuild-index, or the one after library scan) that
finds a daemon writing waits for it, pausing up to 2 s between tries, and
gives up after five minutes. Searches read the last commit, so they keep
answering from the old index while a rebuild runs. A search on a node
whose works index was never built (or is of an older layout) answers 503
(the CLI says to run library search --rebuild-index) until the job
finishes; a search never creates or replaces an index.
Note
Earlier builds also kept a per-title index under search_index/.
Nothing reads or writes it any more, so that folder can be deleted.
A works search answers in relevance order (title matches weigh most, then
authors) and ignores sort; the CLI prints each match's work id and
title, best first. author:, narrator:, series:, title:
and identifiers: (also spelled asin:, isbn: or productid:) narrow
a term to one field; identifiers match whole and ignore case. Each request
reads a window of at most 1,000 matches from the cursor's position, and
only those documents, then reads the matched works' rows from the library
index by id in chunks, applying the other filters there. A page can
therefore come back with fewer rows than limit and still carry a
next_cursor. The cursor holds the query, the filters and how many
matches earlier pages used, and is refused under another query or other
filters. Relevance is computed again for each page, so a work indexed or
dropped between two requests can make the next page repeat or skip a
match.
GET /api/library/works pages one row per held work from the
library_work_rows table instead of joining works, expressions, editions,
listings and holdings on every request. A podcast is one row, its show,
counting its episodes' editions and holdings. The table lives in the
library database, so the whole library shares one index:
- From events. Every daemon registers the same consumer,
host:library-index, foredition_linked,edition_updated,edition_acquire_changed,holding_added,holding_removedandbook_acquired. The catalog folds the registrations into one subscriber, so each event is handled once in the library. An edition event refreshes its edition's work, a holding event the work of its listing, andbook_acquiredthe work of its edition (or of its title, when the payload names no edition). A refresh reads the work as it is now and writes its whole row, so a row's authors and series are as current as the last of these events: a credit or series change alone does not refresh it. A work whose last holding is gone keeps its row with a holding count of 0, which pages skip, and the row is deleted only when the work itself is. A failed refresh is retried up to ten times, then dead-lettered. - Rebuild job. A finished rebuild records the index layout in
library_index_builds. When the current layout has no row (a new library, a database copied to another backend, or a binary that changed how rows are derived, or a change to the library's collections, which removes the row and queues the job at once), the heartbeat queues alibrary_index_rebuildjob at priority 50. It queues at most one every ten minutes while the index stays unbuilt. The job refreshes every top-level work in pages of 100, reporting progress and stopping when cancelled, then records the build, unless the collections changed during the walk, in which case it leaves the build unrecorded so the next job starts over. A page is read in a fixed number of chunked lookups (works, editions, listings, holdings, acquire rows, credits and series for all 100 works at once) and its rows and filter values are written in shared atomic batches, so a page costs about as many round trips as one work. The works search rebuild reads its documents the same way. The ignoredlibrary_index_timingtest inbookclerk-librarytimes a rebuild of 10,000 works.
Two refreshes of one work can run at once, from two deliveries or a delivery and the rebuild. Each stamps its row with the time it started reading, and a write only replaces a row stamped no later, so a slow refresh never overwrites a newer one. Keeping an unheld work's row (rather than deleting it) keeps that stamp, so a slow refresh that read the work while it was still held cannot add it back.
Each row names a cover edition: a held edition, preferring one with a cover of its own or a store's cover on one of its listings, then an acquired one, then audio. A held work always names one, even before any cover is known, because store text and a title's enrichment are written after the link whose event refreshed the row. The route reads the cover when it serves the page: the edition's own (the first store cover a listing of it reported), else the first store cover among its listings, else its work's (or, for an episode, its show's), so a cover stored later shows without another refresh. The work detail route gives each edition its cover the same way.
Each row carries the sort keys of every order the page offers: title,
first author, newest holding, series then position (works in no series
last), newest reported purchase and longest held runtime (works with no
purchase time or runtime last). Pages use a keyset cursor (the last row's
sort values, sent back as an opaque cursor), so every page costs the
same and a row added between two requests never shifts the next page. A
cursor issued for one sort is refused under another.
The same refresh writes the work's filter values to
library_work_facets: its authors and narrators (from credits and
listings), series, genres (the work's and listings' categories, split on
commas, semicolons and bars), publishers, the storefronts it is held
through, its library tags (work_tags) and the collections whose
storage prefixes hold one of its editions' files (core.collections),
and explicit when a store marks one of its listings explicit, each under its lower-cased key. The row write, the removal of
the old values and as many new values as fit share one atomic batch; the
rest follow in further batches, each checking that the row still carries
this refresh's stamp, so a later refresh's values are never mixed with an
older one's. facet and value on GET /api/library/works filter by one
value, and GET /api/library/works/facets/{facet} pages a facet's values
with how many held works have each. Tags and progress are per person, so
they are read from the caller's own user_work_tags and
user_title_state rows instead.
POST /api/library/scan, /acquire, /api/discover/sync-listening, and
/integrations/{id}/scan (and the scheduler) call
LibraryStore::enqueue_job:
| Outcome | HTTP | Meaning |
|---|---|---|
| Created | 200 | New row; worker is notified |
| Duplicate | 409 | Same dedup key already pending or running; job_id is the existing row |
| QueueFull | 429 | pending + running >= jobs.max_pending |
| Unroutable | 409 | error: "unroutable": the job's required instances share no node (see job routing below); no row is written |
A scan goes through the scan planner first (control_plane::plan_scan).
An operator, an Owner, an Administrator, or the scheduler scanning every
account gets one scan job per scan-enabled account id, so the response
also carries job_ids (200 when any was accepted, 409 when all were
already queued, 429 when the queue was full, 422 nothing_to_scan for an
empty plan); with no scan-enabled account it stays one scan:account=all
job. The scan endpoint takes the operator, Owners and Administrators only:
any other portal user is 401, and an operator impersonating one is 403.
The planner also scopes any other user to the accounts linked to them
(naming someone else's account is forbidden), but no HTTP caller reaches
that path yet.
Admission, claim, and handler execution share an RwLock (job_runtime).
A database plugin swap takes the write permit so it cannot observe an idle
worker that is about to claim. Admission waits up to 15s for a read permit
and returns 503 if a swap holds the lock too long.
Admission, claim, and scratch-quota updates take a write lock on a
bookclerk_slots row (job-queue) so COUNT then INSERT cannot
exceed max_pending under READ COMMITTED on every required backend. The required postgres job queue CI job runs those
concurrency tests against a disposable multi-connection database
(BOOKCLERK_TEST_POSTGRES_URL + BOOKCLERK_REQUIRE_POSTGRES_TESTS=1).
They are #[ignore] in the default workspace suite so a missing Postgres
cannot false-pass. TOTP enroll/disable atomic conformance
(postgres_totp_*) is not ignored: the same job provisions Postgres and
runs those tests automatically (BOOKCLERK_REQUIRE_POSTGRES_TESTS=1).
Claim (native and D1) marks pending rows with malformed JSON, an unknown kind,
an unknown resource_class, or an unsupported envelope version as
invalid_job before class-specific selection, so a bad row cannot occupy
max_pending forever or abort a D1 batch.
Every scan and acquire is admitted through a planner, whoever asks:
- Scan (
control_plane::plan_scan): operator authority, an Owner or an Administrator scanning every account gets one job per scan-enabled account; anyone else only the accounts they own (see the control-plane ADR, Per-item trust and the scan planner). - Acquire (
control_plane::plan_acquire): each named title id (titles, UUID, ASIN, ISBN or product id) becomes its own job, so each can go to any node trusted with it. Operator authority, an Owner or an Administrator acquiring everything gets one unsplit job. Anyone else must own the account they name, when they name one, and every account that holds a title a named id matches (a listing's product id, an ASIN or ISBN of its edition, or a holding id; 403 otherwise). An account is named by its id or label, ignoring ASCII case, and a job matches it the same way when it runs. With neither an account nor a title, they get one job per account they wholly own. A node is trusted with an acquire the same way: by the accounts holding the titles its filters match.POST /api/library/acquiretakes{account, plugin_instance_id, titles, editions, force, pdf_only}and answers a multi-job plan like a scan (job_ids; 200, 409, 429, or 422nothing_to_acquirefor an empty plan). editionsnames editions (the ids a work's detail lists). Each becomes one title: of the library titles that sell it, the first in an account the caller may acquire for (and inaccountwhen that is set). The edition's titles share its files, so one is enough. An edition with no title (or none inaccount) is 400; one whose titles are all in other people's accounts is 403. The title is then planned like any other.force(re-download acquired titles) andpdf_only(companion PDFs only) are part of the acquire's dedup key and run only on shared nodes; a host is never handed such a job. A pinned instance is part of the key too, so a pinned acquire never stands in for an unpinned one or another pin.
The CLI bookclerk library scan and library acquire do not run guests.
They plan as operator authority (the CLI holds the database), record each
job's routing requirements, insert the rows, and then wait for them,
printing state changes to stderr; the first Ctrl-C asks unfinished jobs to
cancel, a second stops waiting. Waiting needs a live bookclerkd node of
the library (the CLI fails before queueing otherwise); --no-wait only
queues and prints the job ids. library scan --account <id> is still
exactly one job for that account (bypassing scan_enabled), and
library acquire --asin <ASIN> exactly one job for that title.
--match-storage and a rebuild of the CLI machine's search index still run
in the CLI after the scans succeed; daemons keep their own index current
from events (see Search index). library acquire --dry-run plans keys locally without queueing. library acquire has no --license or -o key=value flags
(both were removed, never implemented); a queued acquire runs with the
settings of the node that claims it.
A branch node's operator asks its library for a scan with bookclerk libraries scan (Libraries and branch nodes).
- Default lease is 60s (
jobs.lease_seconds); the worker heartbeats atlease/3.Ok(false)(fence lost) sets the handler cancel flag. A heartbeatErris logged and retried only until the last confirmedOk(true)(or the original claim) would have expired; after that the worker treats the result as unfenced and ignores it. Finalization callsfail_job(..., "cancelled")only whencancel_requestedis set. A local cancel without that flag is treated as fence loss and ignored. - Event delivery workers use the same 60s lease and
lease/3heartbeat duringevent(batch). Fence loss cancels the guest RPC and ignores the result. Leases are restricted to instances loaded on this process; releasing an unexecuted lease does not consumeattempt_count. Expired-lease reclaim restoresresume_pendingwhen a checkpoint exists so a crash during resume does not burn an attempt. - Startup and each worker tick call
reclaim_expired_leases. - Books left
queued/downloadingwith no running acquire job are set toerror(orphaned_after_restart). The next acquire job retries them. - Scratch dirs under
{cache}/acquireand{cache}/acquire-pdfare reserved againstjobs.temp_quota_bytes(default 2 GiB) and unregistered only after that path is deleted (or already gone). Startup sweeps unregistered orphans, and each node'scache_sweepjob sweeps those in which nothing has changed for an hour. Fetch, remux/transcode, and companion-PDF downloads (PDF-only and sidecars discovered during audio acquire) are bounded by the remaining quota: sources can callFetchOptions::enforce_cache_budget, the pipeline watchdog cancels a stage that crosses the budget, and PDF bodies are streamed with an explicit remaining-byte cap.
Scan, listen-sync, and integration-scan retries are idempotent (upserts /
remote rescan). Acquire retries skip titles already acquired, except when
download_pdf is on and the companion PDF is not stored (pdf_status is
not acquired, or pdf_storage_key is missing/empty). Those rows stay in
the job target list so a later acquire can resume the PDF-only side effect.
A companion PDF that exceeds the remaining scratch budget is soft-failed
(pdf_status=error) so the audio acquire can succeed; a store that
answers the resumed PDF with rateLimited fails the title instead, so the
job records the account's cool-down (see Download pacing).
Dedicated
acquire --pdf still fails the job when the PDF cannot be stored; that
path always unregisters its acquire-pdf scratch reservation, including
on fetch/quota errors.
Stores throttle an account that downloads too much, usually without
publishing the limit, so the host paces the downloads it starts itself
(bookclerk_library::pacing, run_acquire). The pace is per store
account, keyed (plugin_key, account_id) like accounts themselves, so it
holds across plugin instances and every node of the cluster.
- Limits. A storefront declares a daily budget, counted over any
rolling 24 hours, and a minimum gap between downloads in its manifest
(
[acquire]). The person an account is linked to may override them on the Accounts page (POST /api/portal/sources/{source}/connections/{account_id}/pace, whose body is at most 4096 bytes), and an operator withbookclerk library set-pace: the store's pace, a custom budget and gap, or no limit. The override is stored inaccount_pace. - Cool-down. When
fetchTitleanswersrateLimited, the host records a cool-down on the account until the store'sretryAfterMs, or for the storefront'scooldown_seconds(one hour when it declares none), capped at a week. Automatic downloads on the account wait for it, even with no limit set, until it ends or someone clears it. The Audible plugin answersrateLimitedwhen Audible refuses a license because it is throttling the account, and declares a 24-hour cool-down. - Who waits. An acquire job whose payload has
pacedset waits for the pace: scheduled auto-acquire and the acquire a branch queues for a request it accepted from a library (which is how "Acquire all of this branch's members' books" reaches it). A paced title that fails and is retried waits for the pace again before the retry asks the store, and is set aside like any other when the wait is long. A paced run tries audio not yet downloaded first, then missing companion PDFs of acquired titles, then titles that failed before, so a title that keeps failing or has no PDF does not spend the budget of the audio behind it. An acquire a person starts in the web app or CLI leavespacedunset: its downloads count against the budget but are never held, and when the store answersrateLimitedthe job records the cool-down and fails for good with the store's answer (it does not go back to the queue for another attempt) instead of retrying into the throttle.pacedis part of the dedup key, so a hand-started acquire is never folded into a waiting automatic one. - Ledger. Every title Bookclerk may have fetched from the store is a
row in
account_downloads(start time, job, paced or not), once per attempt that asked the store, at the time it started, so a retried title counts each try and its retry moves the gap on. Each row's id is made once per download, so a write retried after a lost answer adds no second row. That includes a title that failed after its fetch (converting or storing it), since the store download already happened, and a store download that a better copy from a joined library then replaced, and a missing companion PDF fetched for a title whose audio was already stored (AcquireResult::store_fetches). A library copy from another store that replaced the download and then failed packaging sends the acquire back to the store, and that second download is a second row. A copy matched in storage or taken straight from a joined library is not counted, nor is a title no destination takes or one the store refused withrateLimited. A title a Host downloaded for a hand-started acquire is recorded in the same batch that commits its upload on the owner (control_plane::commit_upload), so the two never part, and once per upload however often the commit is repeated. A ledger write that fails is retried, and a paced run that still cannot record a download holds that account rather than download past a budget it cannot count; the job fails if rows are still unwritten at its end. Backend migration copies the ledger withaccount_pace, so a migrated library keeps its rolling window. Rows older than two days are pruned when the account records its next download. - Where it runs. A paced acquire is claimed only by a node that holds
the library database (
control_plane::claimant_may_run), like a forced or PDF-only one: a Host runs handed acquires without the library, so it could not check the pace. - Changing the pace. Setting an account's pace or clearing its pause
wakes every parked paced acquire that names that account or no account
(
LibraryStore::wake_paced_acquires), so it checks the new pace at once instead of sleeping out the old deadline. A pace body with nomodeandclear_cooldown: trueonly clears the pause and leaves the stored override as it is. A run that parks checks each held account once more after parking and wakes itself if a change made while it ran lets the account download sooner.
A paced run checks the account's pace before each title, while it holds
the account's cluster semaphore, so no other node downloads on the account
between the check and the fetch. A wait of up to two minutes (a short gap)
is slept in place, checking for cancel each second. A longer one sets that
account's titles aside, and the run goes on with the other accounts'
titles. At the end, a run that set titles aside is parked
(LibraryStore::defer_job): it goes back to pending with run_after at
the moment the first held account may download again and its progress
saying why ("3 title(s) waiting for the daily download budget until
…"). Parking does not spend an attempt and drops the storage-scan
checkpoint, so the next claim starts fresh and skips titles already
acquired. An auto-acquire of a large library this way drains over several
days, and the next scheduled auto-acquire is folded into the parked job by
its dedup key.
Every daemon sharing the database runs workers, so a job must say what the
node that runs it needs. At admission the daemon records
JobPayload.requires: the plugin instance ids the claiming node must run
(control_plane::resolve_job_requirements).
- A job pinned to an instance (
plugin_instance_id) requires it, when some node has a deployment of it that should run. A pin to an instance deployed nowhere is not recorded, so the job is claimed anywhere and fails there as before instead of waiting forever. - An acquire requires every deployed instance of a node-bound output (today
the verified first-party
localdestination), because it writes to every deployed destination and alocaloutput is the disk of the nodes it is deployed to. If plugin discovery fails on the admitting node (say an unrelated plugin has a duplicate alias), every instance whose plugin key names the first-partylocalpackage is required instead, unverified, so the acquire is never left claimable by a node without those outputs. - Everything else (unpinned scans, listen sync, integration scans,
plugin_copy) has no requirements and is claimable on any node.
The required instances must share a node where all of them are deployed to
run (control_plane::common_deployment_nodes); otherwise admission refuses
the job with a conflict (HTTP 409 unroutable) instead of leaving a row no
node can ever claim.
Required instances are never dropped to make a node fit.
The daemon's worker claims through control_plane::claim_routed_job: it
pages through the ready pending jobs of its class, best first (16 per page,
keyset-ordered by priority, created_at, id, until it claims one or the rows
run out, so jobs this node cannot run never hide runnable ones behind them),
and skips one unless every required instance has a deployment on this node whose desired
state runs it (present or placed) and whose observation from this node
says the guest is up (running or healthy) — the same placement and
observation rows delivery routing reads. Then it takes the job's library
permits (below) and claims that row with claim_job_by_id, one CAS on
state = 'pending' and the lease generation it read, so concurrent
claimers on any number of connections admit one. A lost CAS gives the
permits back and tries the next candidate. A job whose account or instance
semaphore is full is skipped too; a full class semaphore ends the attempt.
Requirements recorded at admission are kept current by the requirements
sweep below.
Deployments keep moving after a job is queued: a local output is deployed
or retired, or the scheduler moves a replica. Each daemon runs a sweep loop
(bookclerkd requirements_sweep, rules in
control_plane::sweep_job_requirements), but only the holder of the
jobs:requirements-sweep library lease (90 s, renewed every pass) sweeps.
A pass runs every 30 s, on a deployment or install wakeup on this node, and
after a scheduler pass that moved replicas. A wakeup waits until wakeups have
been quiet for 0.5 s (at most 3 s), so a burst of deployment changes, such
as enabling several plugins at once, is swept once. Discovery runs on a
blocking thread. It re-resolves the requirements
of every pending acquire, pinned job, or job that already has requirements,
with the node-bound output keys the daemon discovers plus every instance
keyed as the first-party local destination (so a sweeping node without
that plugin, or whose discovery failed, never strips them; a pass that cannot
read the instance list is skipped), and:
- applies at once a set that is unchanged or only gains instances when
some node is deployed to run all of them (a newly deployed
localoutput is added to queued acquires), clearing any unsettled mark; - otherwise (unsettled: no node runs all of them, or a recorded instance
is no longer deployed anywhere) stamps
JobPayload.requires_unsettled_since_mson the first sweep and drops no recorded instance for a 10-minute grace period, so a placement move that retires the old replica before the new one is up never fails queued work. Instances the job gained are still recorded at once (the recorded set becomes the union of old and new), so a job that waits cannot be claimed on its old set and skip a newly deployedlocaloutput; - when the grace period has passed, writes the new set if some node can run
it (an instance was only dropped, which is what admission would record
now), or fails the job with
error_kind = "unroutable"and a message naming the instances. A failed job frees itsmax_pendingslot and dedup key.
A job pinned to a node (JobPayload.node_id) is failed as unroutable
when that node has never sent a heartbeat, or its last one is older than
the grace period, so a removed or long-stopped node does not keep its
queued work. While the node is live, any requirements the job has are
reconciled like any other job's.
The grace period is a wall-clock stamp rather than a count of unsettled
sweeps, because sweeps also run on wakeups and a count would expire faster
on a busy library. Every write is a compare-and-swap on state = 'pending'
and the lease generation, which it bumps: a job claimed meanwhile is left
alone, and a claimer that read the row before the sweep wrote it loses its
own CAS (a routed claim matches the generation of the record whose
requirements it checked) and moves on to re-read it on a later pass. A wakeup on a node that does not hold the lease does
nothing; the holder's timer picks the change up within 30 s.
Before requirements and permits, the routed claim filters candidates
through the trust predicate (control_plane::trust). The daemon's own
worker holds the database and is trusted with every job. A Branch node
(H-4) owned by a user who is not an Owner or Administrator may claim only
a scan or acquire of a named account linked to that user; a scan-all, any
pinned or instance-bound job, listen sync, integration scans, and plugin
copies are refused to it. control_plane::handoff_routed_job repeats the
check before a job's secrets would be handed over and, if the node lost
that trust, returns the job to pending without spending an attempt. It
also hands nothing over unless the claim still owns the job's running,
unexpired lease (same lease owner and generation), so a late handoff
after a reclaim, completion or expiry unseals nothing. See
Per-item trust and the scan planner in docs/adr/control-plane.md.
The unrouted claim_next_job (and its guest atomic plan) never claims a job
with requirements; it remains for tests and tools that run one process. It
pages past routed rows the same way (the atomic plan filters them out before
its LIMIT 1).
Process-local limits add up across nodes, so the limits that protect shared
things are counted library-wide. A semaphore name with limit N is the
library_leases rows sem:<name>#0 … sem:<name>#<N-1>; a permit is one
of those slot leases (holder, fencing token, expiry), taken with the same
fenced compare-and-swap as the scheduler lease
(control_plane::semaphore). Only slots below the limit are taken, so no
more than N permits are live at once. A routed claim takes, all or none,
one permit of:
| Semaphore | Limit | Counts |
|---|---|---|
jobs:<class> |
[jobs.concurrency].network |
Running jobs of the class, library-wide |
plugin-instance:<id> |
[jobs.concurrency].plugin_instance (default 1) |
Running jobs that require that instance |
account:<plugin_key>:<account_id> |
1 | Scans and acquires touching that store account: the payload's account, else every account (of the pinned plugin, when pinned) |
Permits last as long as the job lease, are renewed with each heartbeat, and
are released after the row is finalized. A routed claim's holder is
<lease owner>/<job id>/g<generation>/<random>, with the lease generation
the claim runs at (the CAS only succeeds one past the generation it read),
so a node that finds its permits again by holder prefix never mistakes a
leftover permit of an earlier claim of the same job for the current one. A heartbeat whose permit renewal
matches no row (another holder took the slot) is a lost fence: the handler
is cancelled and its result ignored. When a node dies, its permits and job
lease expire; the next claimer reclaims the row, takes the slot with a
larger token, and the old holder can neither renew the permit nor finalize
the job (the job's lease generation moved on too). Lowering a limit does
not revoke slots above it; live permits on those slots still count, so no
new permit is granted until the live count is under the new limit. (That
count is read before the slot CAS, so racing claimers can briefly overshoot
the new limit, never the old one, until the high slots drain.) Each node reads the limits from its own
config.toml ([jobs] is not a database document yet), so nodes should
agree on them.
[jobs.concurrency].network is the number of leased network workers on each
node (default 1) and also the library-wide jobs:network permit count
above, so adding nodes does not multiply concurrent store traffic. The
historical work_lock still wraps run_* so the shared StorageIndex stays
single-writer within one process. Codec work stays in the [media].workers
pool.
- Add a
JobKindvariant, envelope fields, and dedup-key rule incrates/bookclerk-library/src/models.rs. - Add a
JobCommandarm onInProcessJobTransport. - Enqueue from the API and/or scheduler with the same
EnqueueJobSpec. - Reserve scratch paths via
LibraryStore::reserve_job_temp_pathwhen the job creates temp files; unregister one path after it is gone. - Add concurrent admission/claim, fence-loss, and restart tests.
See configuration.md ([jobs] / BOOKCLERK_JOBS_* and
[events] / BOOKCLERK_EVENTS_*).
Library-wide cleanup runs on one daemon at a time. Each duty holds its own
library_leases row; every daemon polls them every 60 s, starting 5 minutes
after it starts so the first pass doesn't compete with startup. The holder renews
its leases on each poll and runs a duty when it takes the lease and then
hourly. A daemon that stops polling loses its leases after 180 s and the
next daemon to poll takes them over with a larger fencing token. Before
each delete the holder renews the lease with a compare-and-swap on its
holder and token (the fence), so a daemon that stalled past its expiry is
refused before it deletes anything.
| Lease | Duty |
|---|---|
maintenance.jobs |
Terminal jobs older than jobs.retention_days |
maintenance.events |
Expires never-leased deliveries past their retention period, then deletes terminal deliveries, empty parent events, stale subscriber catalog rows ([events] retention) |
maintenance.kv |
Expired plugin KV rows (already read as missing) |
maintenance.sessions |
Expired operator sessions, OIDC RP states, WebAuthn challenges |
maintenance.records |
Expired credentials and spent tokens, finished uploads, abandoned scan inventories, and old audit rows (windows below) |
The maintenance.records windows are fixed in
crates/bookclerk-library/src/store/retention.rs, not configured. Each is longer than
the row can still be used; the grace periods only keep recent history
visible:
| Rows | Deleted when |
|---|---|
| OIDC authorization codes, portal sessions | Past expires_at |
| OIDC refresh tokens | Past expires_at, or revoked |
| Claim tickets, user invites, node enrollment links | 30 days past expires_at (expired and redeemed ones stay listed until then) |
| Committed or aborted host uploads | 1 day past expires_at; open uploads are kept, since an abort still reads them, and so is a branch's catalog upload while a file record names it as writer, since that is how its files are known to be the branch's |
| Storage-scan inventories | Job terminal or gone; a jobless (CLI) scan silent for 1 day |
security_audit_events, configuration_audit |
365 days old |
configuration_changes |
30 days old (the document and its audit row keep the content) |
notifications with their delivery log and inbox rows |
30 days old |
The prunes are idempotent, so the fence bounds wasted work rather than
guarding correctness: the window between a fence check and its delete is
one statement. Node-local cleanup (scratch directories, expired job and
delivery leases) still runs on every node. GET /api/library-status
lists every lease with its holder, token, and whether it is held.
Source: docs/internals/jobs.md at main · View as Markdown