# Durable job queue

`bookclerkd` admits background work as **durable rows** in the host library
database (`jobs` + `job_temp_paths` on the same schema as `holdings` and
`accounts`). HTTP and the interval scheduler are producers; leased workers
claim jobs. There is no external broker and **no dedicated jobs database**.

The queue is host-owned. Guests never receive SQL against `jobs` (or any other
library table). A plugin that needs its own durable store declares a named
database binding; that unit is physically separate and holds only plugin-owned
DDL. Job lifecycle (admit, claim, heartbeat, complete) stays host SeaORM /
typed host plans on the library.

This is not a general pub/sub bus. Domain events such as `book_acquired` /
plugin `event(batch)` stay **off** `JobKind`. They use a durable outbox
(`domain_events` + `event_deliveries` + `event_subscriber_nodes`) with
the same fenced-lease pattern as jobs. Acquire success publishes
`book_acquired` with producer `source` on the envelope. Each host heartbeats the instances its deployments
run (`present` or `placed`, whether or not the guest is up) and its loaded
integrations into a **per-node** catalog keyed by
`(node_id, plugin instance id)`, or `(node_id, PluginKey)` for a plugin
without an instance (`plugin_id` stores the canonical PluginKey, not the
display alias) and does **not** delete other nodes’ rows. The catalog key is
the node id from `node-identity.json`, the same id the node registers with
the library and claims its `node.<node_id>` queue under. Dispatch uses
the live union (any enabled node whose heartbeat is within 60s). Optional
payload-object filters and `resource_class` (currently `network` only) are
matched host-side. Each dispatcher tick dispatches at most 32 undispatched
events, then always runs a wake slice (so a backlog cannot starve sleepers or
the catalog heartbeat) and continues when either undispatched remain or
`wake still_pending`. Late-join is a **missing `(event_id, plugin_id)` anti-join**
paged 200, restricted to the retention window — an unchanged catalog with no
missing pairs does a bounded empty `SELECT` and zero dispatch writes. The
subscriber snapshot an event's first dispatch freezes (so its pages resume
unchanged after a crash, and a retry that sends the same list replays their
receipts) covers only that dispatch; a late-join adds deliveries for the
subscribers it finds now. D1
dispatch receipts are per pair (`dispatch-{event_id}-{plugin_id}` /
`reconcile-{event_id}-{plugin_id}`). Catalog rows
follow deployments, not processes: an instance deployed on a node keeps
its enabled row while its guest is not running, so its deliveries are
written at publish time and wait as `pending`. An event fans out to every
subscribed instance, and each copy is *leased* to one node; only jobs are
*claimed*. Each VPS leases only deliveries for instances loaded on that
process **and** only events its own node catalog matches (type, schema
version, filter). `[events]` retention and concurrency are stored in the database after the
first import (`core.events`; see [configuration](/docs/server/configuration/index.md) and
[ADR: Control plane](/docs/adr/control-plane/index.md)). `[events.concurrency]` is both the
local worker count **and** the
library-wide max `running` deliveries per `(plugin_id, resource_class)`
(serialized with a portable `bookclerk_slots` row).
`EventResult::suspended` may set `wakeOnEventType` / `wakeOnFilterJson`; the
host derives wake grants from declared subscriptions (schema versions plus the
intersection of `sub.filter` and the requested filter — requested keys only
add constraints) and wakes matching parked rows in the same account when a
later event is published. Publish commits `domain_events.wake_pending = 1` and
returns; the dispatcher claims bounded wake slices with a unique UUID fence
token (`wake_lease_*` + delivery cursor, at most 32 events and one page
each sized from negotiated `maxBinds`)
so producer latency does not track sleeper count. Cursor release, finish,
and the sleeper UPDATE require that token in the same statement; a lost
fence does not clobber another owner or a later wake registration.
Accepting a wake clears the registration so a later matching event does not
re-wake a retry. Duplicate retries leave the flag set until a claimed
slice finishes. `wakeOnEventType` must be a declared subscription (empty stays
timestamp-only); an empty requested filter keeps the subscription filter and
cannot broaden it. Late-join skip cache is invalidated on dispatch error and
reconciles at least every 60s as a backstop.
`GET /api/status` includes event queue counts plus durable retry/suspend totals
and average dispatch/handler latency. A `pending` delivery that was never
leased expires (state `expired`) once it has been due for longer than its
consumer's `message_retention_period` (default one day, capped at
`[events].retention_days`; host and node consumers never expire), so a
deployed instance that is not running does not accumulate an unbounded
backlog (see
[Retention of undelivered events](/docs/plugins/abi/index.md#retention-of-undelivered-events)).
Retention is independent of jobs
(`[events].retention_days` for acked/rejected/expired + empty parent events after that
cutoff, `dead_letter_retention_days` for dead letters). Prune is a
library singleton, not a dispatcher tick: see *Library maintenance* below.
Operator APIs: `GET /api/events`,
`GET /api/events/deliveries?state=dead_letter`,
`POST /api/events/deliveries/{id}/retry`,
`POST /api/events/deliveries/{id}/acknowledge`,
`POST /api/events/deliveries/{id}/cancel`,
`POST /api/events/deliveries/{id}/resume`. CLI:
`bookclerk events list|dead-letters|retry|ack|cancel|resume`. Plugin-owned
queues (`[queues]` in `plugin.toml`) ride the same outbox tables under their
own `queue_name`; `GET /api/queues`, `GET /api/queues/{name}` and
`bookclerk queues list|inspect <name>` show per-queue counters, live
consumers, recent messages and deliveries (see
[abi.md#queues](/docs/plugins/abi/index.md#queues)).

See [architecture.md](/docs/internals/architecture/index.md), [plugins.md](/docs/plugins/index.md), and
[operations.md](/docs/server/operations/index.md).

## Command envelope

Each row stores a versioned JSON command (`JobPayload.v`, currently `1`).
Unknown kinds, unsupported versions, and malformed JSON are rewritten to
`kind=invalid` / `state=failed` / `error_kind=invalid_job` and never run a
handler.

Dispatch goes through `JobTransport` (`InProcessJobTransport` today; a workerd
adapter can replace it later). Handlers receive a `JobExecCtx` with the lease
fence and a cooperative cancel flag.

## State machine

```mermaid
stateDiagram-v2
  direction LR
  [*] --> pending
  pending --> running
  running --> succeeded
  running --> failed
  running --> cancelled
  failed --> pending: retry with backoff
  succeeded --> [*]
  failed --> [*]: terminal
  cancelled --> [*]
```

- **Admission** is one atomic backend operation (a guest atomic batch in
  production, a local `BEGIN` in tests). A partial unique index on `dedup_key` for
  `pending`/`running` rows enforces active-key uniqueness. Both paths take the
  key from `JobKind::dedup_key`, so a new kind needs no second rule, and a
  duplicate returns the id of the job it folded into.
- A **worker** claims with a conditional `pending` → `running` update and a
  unique per-attempt `lease_generation`. Lost RPCs retry the same
  `operation_id` so a replay cannot claim a second row. That replay is
  bounded (2s); if the database stays `Unavailable`, the worker returns the
  error, drops the `job_runtime` read permit so a plugin swap can proceed,
  and lets an ambiguously claimed row expire for reclaim. The next loop
  iteration uses a new operation id.
- Heartbeat, progress, complete, fail, and reclaim are fenced on
  `(job_id, lease_owner, lease_generation)`. Zero rows (`Ok(false)`) means the
  fence is lost; the worker cancels the handler and ignores the unfenced
  result. A heartbeat **error** (transient DB/RPC failure) is retried only
  until the last confirmed lease expiry; past that deadline the worker
  cancels locally and ignores the result. It must not finalize the row as
  `cancelled` unless `cancel_requested` is set.
- Destination **publish** is not atomic with that fence. `plugin_copy`
  stream-copy is at-least-once: a lost lease can still make object bytes
  visible after the last heartbeat. Duplicates are absorbed by retry-stable
  `commit_token`s. A missing stage counts as success only when the published
  object's `commit-token` metadata is that token (see
  [storage-bounds.md](/docs/internals/storage-bounds/index.md)). A pre-commit heartbeat does not
  make publication exactly-once.
- Storefront `scan` jobs keep their place outside the job row, in
  `account_scan_state` (one row per store account): the guest's next-page
  cursor after each written page, and the sync token and time of the last
  finished pass. A retried or later job resumes a cursor younger than 24
  hours and otherwise starts a new pass, full when no complete pass ended in
  seven days (see [Paged scans](/docs/plugins/abi/index.md#paged-scans)). The same
  row keeps the daily quality-upgrade check's place (`quality_cursor`, the
  last holding id asked about, empty after a full walk) and time
  (`quality_checked_at`); see
  [Upgrading to better files](/docs/server/output/index.md#upgrading-to-better-files).
  The row goes away with its account.
- Storage scans checkpoint `storage_scan` JSON (instance id, generation,
  phase, cursor) with `checkpoint_running_job` while the row stays `running`.
  The generation is a UUID bound in `storage_scan_generations` to that storage
  instance and the job. Local-backed scans, including fan-out and plugin
  wrappers, also store the node placement (the node id from
  `node-identity.json`). A resume on another node, or a backend whose placement cannot
  be proved, starts a new scan instead of adopting the old rows. Object rows
  live in `storage_scan_rows`, not in the 64 KiB checkpoint. Page inserts happen before the cursor advances; replaying
  a page is idempotent.   `run_acquire` performs one fenced scan and matches
  against that index. An `apply` checkpoint is adopted only when its
  generation row still exists and is marked complete; a missing or incomplete
  inventory is scanned again.   A successful acquire, including one with no
  targets, deletes the generation before the handler returns. Inventory rows
  and the generation row are deleted in one database transaction. Startup
  reclaim removes abandoned generations and does not remove one owned by a
  pending or running job. While the daemon is up, a bounded sweep and each
  confirmed terminal transition delete inventories for terminal jobs. A
  failed attempt that will be retried, a lost fence, and suspended work leave
  the generation in place. Local list indexes are node-local scratch and are
  not a portable checkpoint.
- A convert job is an `acquire` job whose payload carries the copy to add
  (`convert`, written `format:quality`). Its dedup key ends in
  `:convert=<copy>`, so a convert and an acquire of the same title are
  separate jobs, and it is routed only to a node with Local trust. It
  streams the stored copy into the acquire scratch directory
  (`{cache}/acquire/<title>/convert-source`), under the same temp quota as
  an acquire, makes the copy there through the copy pipeline and removes
  the directory when it ends. Older versions converted in
  `{cache}/convert`, which the startup sweep still cleans.
- Reclaim also requires `lease_expires_at` to still be null or `<= now`, so a
  heartbeat that extends the same generation cannot be stolen.
- `POST /api/jobs/{id}/cancel` cancels `pending` immediately and flags
  `running` for a cooperative stop between acquire titles, scan sources, and
  listen-sync providers. Cancel retries if a worker claims the row between the
  pending read and the pending-only CAS, so a cancel that returns never leaves
  a running job unflagged. Integration `scan_library` RPCs are not interruptible;
  cancel is checked before the call and a repeat remote scan is idempotent.

## Job kinds

| Kind | Dedup key | Resource class | Handler |
| --- | --- | --- | --- |
| `scan` | `scan:account={id\|all}`, then `:instance={id}` when pinned | `network` | `run_scan` |
| `acquire` | `acquire:title={id\|all}:account={id\|all}`, then `:force=1`, `:pdf=1`, `:paced=1` and `:instance={id}` when set | `network` | `run_acquire` |
| `listen_sync` | `listen_sync` | `network` | `run_listen_sync` |
| `integration_scan` | `integration_scan:id={id}:force={0\|1}` | `network` | `run_integration_scan` |
| `plugin_copy` | `plugin_copy:plugin={id}:from={key}:to={key}` | `network` | ABI `JobRunner` stream-copy (`run_plugin_copy`) |
| `plugin_scheduled` | `plugin_scheduled:instance={id}:cron={schedule}` | `network` | `scheduled(controller)` on the instance (`run_plugin_scheduled`) |
| `discover_candidates` | `discover_candidates:subject={key}` | `network` | `run_discover_candidates`, then admits the subject's `discover_feed` |
| `discover_feed` | `discover_feed:subject={key}` | `network` | `run_discover_feed` (no network; ranks stored candidates) |
| `discover_request_queue` | `discover_request_queue` | `network` | `run_discover_request_queue` (no network; ranks the shared request queue) |
| `cache_sweep` | `cache_sweep:node={id}` | `network` | `run_cache_sweep` (no network; one node's image caches and job scratch) |
| `search_index_rebuild` | `search_index_rebuild:node={id}` | `network` | `run_search_index_rebuild` (no network; rebuilds one node's search index) |
| `library_index_rebuild` | `library_index_rebuild` | `network` | `run_library_index_rebuild` (no network; rewrites every row of the library's page index) |
| `branch_request` | `branch_request:node={id}:work={id}` | `network` | none: the branch it is pinned to takes it over the node socket (`requests.claim` / `requests.finish`) |
| `notify_send` | `notify_send:delivery={id}` | `network` | `run_notify_send` on a node running the notifier instance (requires it); parks itself between retries with `defer_job` |
| `ingest_batch` | `ingest_batch:batch={id}` | `network` | `run_ingest_batch` on the node that holds the upload's bytes (pinned by `payload.node_id`; never handed to a branch) |

`plugin_scheduled` jobs aren't enqueued through the API. Each daemon's cron
ticker claims a due fire with a compare-and-swap on the instance's
`plugin_cron_cursors` row and inserts the job in the same batch, with
`requires = [instance]` so it routes to a node that runs the instance. The
dedup key holds one fire per schedule, so a fire that comes due while the
previous one is still active is skipped. A failed fire retries up to 3
attempts with the same `scheduled_time_ms`. A pinned fire can only be
claimed by a node trusted with every job (a shared node, or a branch node
whose owner is an Owner or Administrator). When no such node runs the instance
past the requirements sweep's grace period, the sweep fails the fire as
`unroutable`. That covers an instance deployed nowhere and one deployed only
on branch nodes owned by members. See
[Scheduled triggers](/docs/plugins/abi/index.md#scheduled-triggers).

The two Discover kinds carry a subject key (`operator`, `user:{id}` or
`portal:{id}`) and build that person's stored shelves. Reads, the daily
scheduler and Discover's host event consumer (`host:discover`, which
consumes `wishlist_changed`, `listening_synced`, `book_acquired`,
`holding_added` and `holding_removed` and admits jobs with a 30 s `run_after`) queue them. See
[Precomputed shelves](/docs/internals/discovery/index.md#precomputed-shelves) and
[Taste events](/docs/internals/discovery/index.md#taste-events).

`ingest_batch` stores one upload (or one drop-folder intake batch): it
checks each file against the manifest, expands archives in the media
worker, groups files into books, resolves each through a patron-held
`core.import` listing and writes it to every destination. The upload
routes queue it on commit with `payload.ingest_batch`, and a storage
intake queues it with `uploads::queue_intake_batch`, which copies each
object from the named storage instance first. Its end publishes
`ingest_batch_finished` in the same write as the batch's final state; see
[Uploads](/docs/server/uploads/index.md).

`branch_request` is an interlibrary request: a user of this library asks
a branch for a book it listed. The job is pinned to the branch's node
(`payload.node_id`) and names the listed edition and the work's strong
identifiers. No claim loop runs it: ordinary claims, local or remote,
never hand it out, and only that branch takes it with `requests.claim`.
Its answer completes the job (accepted), fails it without a retry
(`not_listed`), or fails it with the normal retry (`branch_failed`). See
[Requesting a book from a branch](/docs/server/branches/index.md#requesting-a-book-from-a-branch).

`discover_request_queue` carries no subject: one ranking of the shared
request queue serves everyone. Discover's event consumer queues it on
every one of those five events (a holding event too, since a new holding
can hide an entry), and a read of `GET /api/request-queue` queues it when
the stored ranking is missing an entry or older than a day. See
[Request queue](/docs/internals/discovery/index.md#request-queue).

A `cache_sweep` job belongs to one node: `JobPayload.node_id` names it,
and only that node's routed claim takes the job (the unrouted claim never
does). Each daemon admits one for itself from its heartbeat when a sweep is
due (see [Cache retention](/docs/server/integrations/index.md#cache-retention)), at
priority 100 so a backlog of store work does not hold it back. The dedup key
folds a sweep that comes due while another is queued into that one; one
that comes due while a sweep runs is asked for again on the next heartbeat,
so it runs under the limits in force by then. The job evicts from the node's image caches, then deletes scratch
directories under `{cache}/acquire`, `{cache}/acquire-pdf` and
`{cache}/convert` that no active job has registered and in which nothing
has changed for an hour (a job may create its directory just before
registering it, so a file still being written keeps its directory). The heartbeat queues it only after its own beat is
recorded, so the requirements sweep never finds the job before the node's
heartbeat. When the queue is full, the admission fails, or the queued job
has waited five minutes for a worker, the heartbeat sweeps the image caches
in the background instead.

A `search_index_rebuild` job is pinned to one node the same way. See
[Search index](#search-index).

A `library_index_rebuild` job is not pinned: any node claims it. See
[Library index](#library-index).

Reserved classes (no worker in this release): `media`, `transcription`,
`indexing`.

## Search index

Searches read a Tantivy works index on the node's own disk, so each
daemon keeps its own copy current. `GET /api/library/works?q=` and
`bookclerk library search` read it, under `search_index_works/`, with one
document per held top-level work (its titles, authors, narrators, series
and store identifiers). It is kept current two ways:

- **From events.** Each node registers its own consumer,
  `host:search-index:{node id}`, on the `core.events` outbox for
  `edition_linked`, `edition_updated`, `holding_added` and
  `holding_removed`, all host event types no plugin may produce. The id
  names the node, so every node gets every event (Discover's
  `host:discover` is one consumer for the library). An edition event
  refreshes the document of the edition's work, and a holding event the
  document of its listing's work. A refresh deletes the work's document
  and adds it back while the work is held, so a work nobody holds any
  more leaves the index. A holding event whose listing is already gone
  removes the marker below, since its work can no longer be found, so the
  node rebuilds.
- **Rebuild job.** A finished rebuild leaves a marker for the current
  document layout in the index folder. When it is missing (a new node, a
  deleted folder, or an index of an older layout, which is discarded), the
  heartbeat queues a `search_index_rebuild` job pinned to the node, at
  priority 50, after its beat is recorded. It queues at most one every ten
  minutes while the index stays incomplete, so a failing rebuild is not
  re-queued on every beat. The job reads every top-level work in pages and
  replaces the index in one commit.

One writer runs at a time. A delivery that finds a rebuild (or the CLI)
writing is retried after 10 s, for up to an hour. A CLI rebuild
(`library search --rebuild-index`, or the one after `library scan`) that
finds a daemon writing waits for it, pausing up to 2 s between tries, and
gives up after five minutes. Searches read the last commit, so they keep
answering from the old index while a rebuild runs. A search on a node
whose works index was never built (or is of an older layout) answers 503
(the CLI says to run `library search --rebuild-index`) until the job
finishes; a search never creates or replaces an index.

> [!NOTE]
> Earlier builds also kept a per-title index under `search_index/`.
> Nothing reads or writes it any more, so that folder can be deleted.

A works search answers in relevance order (title matches weigh most, then
authors) and ignores `sort`; the CLI prints each match's work id and
title, best first. `author:`, `narrator:`, `series:`, `title:`
and `identifiers:` (also spelled `asin:`, `isbn:` or `productid:`) narrow
a term to one field; identifiers match whole and ignore case. Each request
reads a window of at most 1,000 matches from the cursor's position, and
only those documents, then reads the matched works' rows from the library
index by id in chunks, applying the other filters there. A page can
therefore come back with fewer rows than `limit` and still carry a
`next_cursor`. The cursor holds the query, the filters and how many
matches earlier pages used, and is refused under another query or other
filters. Relevance is computed again for each page, so a work indexed or
dropped between two requests can make the next page repeat or skip a
match.

## Library index

`GET /api/library/works` pages one row per held work from the
`library_work_rows` table instead of joining works, expressions, editions,
listings and holdings on every request. A podcast is one row, its show,
counting its episodes' editions and holdings. The table lives in the
library database, so the whole library shares one index:

- **From events.** Every daemon registers the same consumer,
  `host:library-index`, for `edition_linked`, `edition_updated`,
  `edition_acquire_changed`, `holding_added`, `holding_removed` and
  `book_acquired`. The catalog folds the registrations into one
  subscriber, so each event is handled once in the library. An edition
  event refreshes its edition's work, a holding event the work of its
  listing, and `book_acquired` the work of its edition (or of its title,
  when the payload names no edition). A refresh reads the work as it is
  now and writes its whole row, so a row's authors and series are as
  current as the last of these events: a credit or series change alone
  does not refresh it. A work whose last holding is gone keeps its row
  with a holding count of 0, which pages skip, and the row is deleted
  only when the work itself is. A failed refresh is retried up to ten
  times, then dead-lettered.
- **Rebuild job.** A finished rebuild records the index layout in
  `library_index_builds`. When the current layout has no row (a new
  library, a database copied to another backend, or a binary that changed
  how rows are derived, or a change to the library's collections, which
  removes the row and queues the job at once), the heartbeat queues a `library_index_rebuild`
  job at priority 50. It queues at most one every ten minutes while the
  index stays unbuilt. The job refreshes every top-level work in pages of
  100, reporting progress and stopping when cancelled, then records the
  build, unless the collections changed during the walk, in which case it
  leaves the build unrecorded so the next job starts over. A page is read in a fixed number of chunked lookups (works,
  editions, listings, holdings, acquire rows, credits and series for all
  100 works at once) and its rows and filter values are written in shared
  atomic batches, so a page costs about as many round trips as one work.
  The works search rebuild reads its documents the same way. The ignored
  `library_index_timing` test in `bookclerk-library` times a rebuild of
  10,000 works.

Two refreshes of one work can run at once, from two deliveries or a
delivery and the rebuild. Each stamps its row with the time it started
reading, and a write only replaces a row stamped no later, so a slow
refresh never overwrites a newer one. Keeping an unheld work's row (rather
than deleting it) keeps that stamp, so a slow refresh that read the work
while it was still held cannot add it back.

Each row names a cover edition: a held edition, preferring one with a
cover of its own or a store's cover on one of its listings, then an
acquired one, then audio. A held work always names one, even before any
cover is known, because store text and a title's enrichment are written
after the link whose event refreshed the row. The route reads the cover
when it serves the page: the edition's own (the first store cover a
listing of it reported), else the first store cover among its listings, else its work's (or, for an episode, its show's), so
a cover stored later shows without another refresh. The work detail route
gives each edition its cover the same way.

Each row carries the sort keys of every order the page offers: title,
first author, newest holding, series then position (works in no series
last), newest reported purchase and longest held runtime (works with no
purchase time or runtime last). Pages use a keyset cursor (the last row's
sort values, sent back as an opaque `cursor`), so every page costs the
same and a row added between two requests never shifts the next page. A
cursor issued for one `sort` is refused under another.

The same refresh writes the work's filter values to
`library_work_facets`: its authors and narrators (from credits and
listings), series, genres (the work's and listings' categories, split on
commas, semicolons and bars), publishers, the storefronts it is held
through, its library tags (`work_tags`) and the collections whose
storage prefixes hold one of its editions' files (`core.collections`),
and `explicit` when a store marks one of its listings explicit, each under its lower-cased key. The row write, the removal of
the old values and as many new values as fit share one atomic batch; the
rest follow in further batches, each checking that the row still carries
this refresh's stamp, so a later refresh's values are never mixed with an
older one's. `facet` and `value` on `GET /api/library/works` filter by one
value, and `GET /api/library/works/facets/{facet}` pages a facet's values
with how many held works have each. Tags and progress are per person, so
they are read from the caller's own `user_work_tags` and
`user_title_state` rows instead.

## Admission and backpressure

`POST /api/library/scan`, `/acquire`, `/api/discover/sync-listening`, and
`/integrations/{id}/scan` (and the scheduler) call
`LibraryStore::enqueue_job`:

| Outcome | HTTP | Meaning |
| --- | --- | --- |
| Created | 200 | New row; worker is notified |
| Duplicate | 409 | Same dedup key already `pending` or `running`; `job_id` is the existing row |
| QueueFull | 429 | `pending + running >= jobs.max_pending` |
| Unroutable | 409 | `error: "unroutable"`: the job's required instances share no node (see job routing below); no row is written |

A scan goes through the scan planner first (`control_plane::plan_scan`).
An operator, an Owner, an Administrator, or the scheduler scanning every
account gets one `scan` job per scan-enabled account id, so the response
also carries `job_ids` (200 when any was accepted, 409 when all were
already queued, 429 when the queue was full, 422 `nothing_to_scan` for an
empty plan); with no scan-enabled account it stays one `scan:account=all`
job. The scan endpoint takes the operator, Owners and Administrators only:
any other portal user is 401, and an operator impersonating one is 403.
The planner also scopes any other user to the accounts linked to them
(naming someone else's account is forbidden), but no HTTP caller reaches
that path yet.

Admission, claim, and handler execution share an `RwLock` (`job_runtime`).
A database plugin swap takes the write permit so it cannot observe an idle
worker that is about to claim. Admission waits up to 15s for a read permit
and returns `503` if a swap holds the lock too long.

Admission, claim, and scratch-quota updates take a write lock on a
`bookclerk_slots` row (`job-queue`) so `COUNT` then `INSERT` cannot
exceed `max_pending` under `READ COMMITTED` on every required backend. The required `postgres job queue` CI job runs those
concurrency tests against a disposable multi-connection database
(`BOOKCLERK_TEST_POSTGRES_URL` + `BOOKCLERK_REQUIRE_POSTGRES_TESTS=1`).
They are `#[ignore]` in the default workspace suite so a missing Postgres
cannot false-pass. TOTP enroll/disable atomic conformance
(`postgres_totp_*`) is not ignored: the same job provisions Postgres and
runs those tests automatically (`BOOKCLERK_REQUIRE_POSTGRES_TESTS=1`).

Claim (native and D1) marks pending rows with malformed JSON, an unknown kind,
an unknown `resource_class`, or an unsupported envelope version as
`invalid_job` before class-specific selection, so a bad row cannot occupy
`max_pending` forever or abort a D1 batch.

### Planners and the CLI

Every scan and acquire is admitted through a planner, whoever asks:

- **Scan** (`control_plane::plan_scan`): operator authority, an Owner or an
  Administrator scanning every account gets one job per scan-enabled
  account; anyone else only the accounts they own (see the control-plane
  ADR, *Per-item trust and the scan planner*).
- **Acquire** (`control_plane::plan_acquire`): each named title id
  (`titles`, UUID, ASIN, ISBN or product id) becomes its own job, so each
  can go to any node trusted with it. Operator authority, an Owner or an
  Administrator acquiring everything gets one unsplit job. Anyone else must
  own the account they name, when they name one, and every account that
  holds a title a named id matches (a listing's product id, an ASIN or ISBN
  of its edition, or a holding id; 403 otherwise). An account is named by
  its id or label, ignoring ASCII case, and a job matches it the same way
  when it runs. With neither an account nor a title, they get one job per
  account they wholly own. A node is trusted with an acquire the same way: by the
  accounts holding the titles its filters match.
  `POST /api/library/acquire` takes `{account, plugin_instance_id, titles,
  editions, force, pdf_only}` and answers a multi-job plan like a scan
  (`job_ids`; 200, 409, 429, or 422 `nothing_to_acquire` for an empty
  plan).
- `editions` names editions (the ids a work's detail lists). Each becomes
  one title: of the library titles that sell it, the first in an account
  the caller may acquire for (and in `account` when that is set). The
  edition's titles share its files, so one is enough. An edition with no
  title (or none in `account`) is 400; one whose titles are all in other
  people's accounts is 403. The title is then planned like any other.
- `force` (re-download acquired titles) and `pdf_only` (companion PDFs only)
  are part of the acquire's dedup key and run only on shared nodes; a host
  is never handed such a job. A pinned instance is part of the key too, so
  a pinned acquire never stands in for an unpinned one or another pin.

The CLI `bookclerk library scan` and `library acquire` do not run guests.
They plan as operator authority (the CLI holds the database), record each
job's routing requirements, insert the rows, and then wait for them,
printing state changes to stderr; the first Ctrl-C asks unfinished jobs to
cancel, a second stops waiting. Waiting needs a live `bookclerkd` node of
the library (the CLI fails before queueing otherwise); `--no-wait` only
queues and prints the job ids. `library scan --account <id>` is still
exactly one job for that account (bypassing `scan_enabled`), and
`library acquire --asin <ASIN>` exactly one job for that title.
`--match-storage` and a rebuild of the CLI machine's search index still run
in the CLI after the scans succeed; daemons keep their own index current
from events (see [Search index](#search-index)). `library acquire
--dry-run` plans keys locally without queueing. `library acquire` has no `--license` or `-o key=value` flags
(both were removed, never implemented); a queued acquire runs with the
settings of the node that claims it.
A branch node's operator asks its library for a scan with `bookclerk
libraries scan` ([Libraries and branch nodes](/docs/server/branches/index.md)).

## Leases and crash recovery

- Default lease is 60s (`jobs.lease_seconds`); the worker heartbeats at
  `lease/3`. `Ok(false)` (fence lost) sets the handler cancel flag. A
  heartbeat `Err` is logged and retried only until the last confirmed
  `Ok(true)` (or the original claim) would have expired; after that the
  worker treats the result as unfenced and ignores it. Finalization calls
  `fail_job(..., "cancelled")` only when `cancel_requested` is set. A local
  cancel without that flag is treated as fence loss and ignored.
- Event delivery workers use the same 60s lease and `lease/3` heartbeat
  during `event(batch)`. Fence loss cancels the guest RPC and ignores the
  result. Leases are restricted to instances loaded on this process;
  releasing an unexecuted lease does not consume `attempt_count`.
  Expired-lease reclaim restores `resume_pending` when a checkpoint exists
  so a crash during resume does not burn an attempt.
- Startup and each worker tick call `reclaim_expired_leases`.
- Books left `queued` / `downloading` with **no** running acquire job are set
  to `error` (`orphaned_after_restart`). The next acquire job retries them.
- Scratch dirs under `{cache}/acquire` and `{cache}/acquire-pdf` are reserved
  against `jobs.temp_quota_bytes` (default 2 GiB) and unregistered only after
  that path is deleted (or already gone). Startup sweeps unregistered orphans,
  and each node's `cache_sweep` job sweeps those in which nothing has changed
  for an hour.
  Fetch, remux/transcode, and companion-PDF downloads (PDF-only and sidecars
  discovered during audio acquire) are bounded by the remaining quota: sources
  can call `FetchOptions::enforce_cache_budget`, the pipeline watchdog cancels
  a stage that crosses the budget, and PDF bodies are streamed with an
  explicit remaining-byte cap.

Scan, listen-sync, and integration-scan retries are idempotent (upserts /
remote rescan). Acquire retries skip titles already `acquired`, except when
`download_pdf` is on and the companion PDF is not stored (`pdf_status` is
not `acquired`, or `pdf_storage_key` is missing/empty). Those rows stay in
the job target list so a later acquire can resume the PDF-only side effect.
A companion PDF that exceeds the remaining scratch budget is soft-failed
(`pdf_status=error`) so the audio acquire can succeed; a store that
answers the resumed PDF with `rateLimited` fails the title instead, so the
job records the account's cool-down (see [Download pacing](#download-pacing)).
Dedicated
`acquire --pdf` still fails the job when the PDF cannot be stored; that
path always unregisters its `acquire-pdf` scratch reservation, including
on fetch/quota errors.

## Download pacing

Stores throttle an account that downloads too much, usually without
publishing the limit, so the host paces the downloads it starts itself
(`bookclerk_library::pacing`, `run_acquire`). The pace is per store
account, keyed `(plugin_key, account_id)` like accounts themselves, so it
holds across plugin instances and every node of the cluster.

- **Limits.** A storefront declares a daily budget, counted over any
  rolling 24 hours, and a minimum gap between downloads in its manifest
  ([`[acquire]`](/docs/plugins/manifest/index.md#acquire-download-pace)). The
  person an account is linked to may override them on the **Accounts**
  page (`POST /api/portal/sources/{source}/connections/{account_id}/pace`,
  whose body is at most 4096 bytes), and an operator with `bookclerk library set-pace`: the store's pace, a
  custom budget and gap, or no limit. The override is stored in
  `account_pace`.
- **Cool-down.** When `fetchTitle` answers `rateLimited`, the host records
  a cool-down on the account until the store's `retryAfterMs`, or for the
  storefront's `cooldown_seconds` (one hour when it declares none), capped
  at a week. Automatic downloads on the account wait for it, even with no
  limit set, until it ends or someone clears it. The Audible plugin answers
  `rateLimited` when Audible refuses a license because it is throttling the
  account, and declares a 24-hour cool-down.
- **Who waits.** An acquire job whose payload has `paced` set waits for the
  pace: scheduled auto-acquire and the acquire a branch queues for a
  request it accepted from a library (which is how "Acquire all of this
  branch's members' books" reaches it). A paced title that fails and is
  retried waits for the pace again before the retry asks the store, and is
  set aside like any other when the wait is long. A paced run tries audio not yet downloaded
  first, then missing companion PDFs of acquired titles, then titles that
  failed before, so a title that keeps failing or has no PDF does not spend
  the budget of the audio behind it. An acquire a person starts in the
  web app or CLI leaves `paced` unset: its downloads count against the
  budget but are never held, and when the store answers `rateLimited` the
  job records the cool-down and fails for good with the store's answer
  (it does not go back to the queue for another attempt) instead of
  retrying into the throttle. `paced` is part of the dedup key, so a
  hand-started acquire is never folded into a waiting automatic one.
- **Ledger.** Every title Bookclerk may have fetched from the store is a
  row in `account_downloads` (start time, job, paced or not), once per
  attempt that asked the store, at the time it started, so a retried
  title counts each try and its retry moves the gap on. Each row's id is made once per
  download, so a write retried after a lost answer adds no second row. That includes a title that failed
  after its fetch (converting or storing it), since the store download
  already happened, and a store download that a better copy from a
  joined library then replaced, and a missing companion PDF fetched for a
  title whose audio was already stored (`AcquireResult::store_fetches`). A
  library copy from another store that replaced the download and then
  failed packaging sends the acquire back to the store, and that second
  download is a second row. A
  copy matched in storage or taken straight from a joined library is not
  counted, nor is a title no destination takes or one the
  store refused with `rateLimited`. A title a Host downloaded for a
  hand-started acquire is recorded in the same batch that commits its
  upload on the owner (`control_plane::commit_upload`), so the two never
  part, and once per upload however often the commit is repeated. A ledger write that fails is retried, and a paced
  run that still cannot record a download holds that account rather than
  download past a budget it cannot count; the job fails if rows are still
  unwritten at its end. Backend migration copies the ledger with
  `account_pace`, so a migrated library keeps its rolling window. Rows
  older than two days are pruned when the account records its next
  download.
- **Where it runs.** A paced acquire is claimed only by a node that holds
  the library database (`control_plane::claimant_may_run`), like a forced
  or PDF-only one: a Host runs handed acquires without the library, so it
  could not check the pace.
- **Changing the pace.** Setting an account's pace or clearing its pause
  wakes every parked paced acquire that names that account or no account
  (`LibraryStore::wake_paced_acquires`), so it checks the new pace at once
  instead of sleeping out the old deadline. A pace body with no `mode` and
  `clear_cooldown: true` only clears the pause and leaves the stored
  override as it is. A run that parks checks each held account once more
  after parking and wakes itself if a change made while it ran lets the
  account download sooner.

A paced run checks the account's pace before each title, while it holds
the account's cluster semaphore, so no other node downloads on the account
between the check and the fetch. A wait of up to two minutes (a short gap)
is slept in place, checking for cancel each second. A longer one sets that
account's titles aside, and the run goes on with the other accounts'
titles. At the end, a run that set titles aside is parked
(`LibraryStore::defer_job`): it goes back to `pending` with `run_after` at
the moment the first held account may download again and its progress
saying why ("3 title(s) waiting for the daily download budget until
…"). Parking does not spend an attempt and drops the storage-scan
checkpoint, so the next claim starts fresh and skips titles already
acquired. An auto-acquire of a large library this way drains over several
days, and the next scheduled auto-acquire is folded into the parked job by
its dedup key.

## Routing to a compatible node

Every daemon sharing the database runs workers, so a job must say what the
node that runs it needs. At admission the daemon records
`JobPayload.requires`: the plugin instance ids the claiming node must run
(`control_plane::resolve_job_requirements`).

- A job pinned to an instance (`plugin_instance_id`) requires it, when some
  node has a deployment of it that should run. A pin to an instance deployed
  nowhere is not recorded, so the job is claimed anywhere and fails there as
  before instead of waiting forever.
- An acquire requires every deployed instance of a node-bound output (today
  the verified first-party `local` destination), because it writes to every
  deployed destination and a `local` output is the disk of the nodes it is
  deployed to. If plugin discovery fails on the admitting node (say an
  unrelated plugin has a duplicate alias), every instance whose plugin key
  names the first-party `local` package is required instead, unverified, so
  the acquire is never left claimable by a node without those outputs.
- Everything else (unpinned scans, listen sync, integration scans,
  `plugin_copy`) has no requirements and is claimable on any node.

The required instances must share a node where all of them are deployed to
run (`control_plane::common_deployment_nodes`); otherwise admission refuses
the job with a conflict (HTTP 409 `unroutable`) instead of leaving a row no
node can ever claim.
Required instances are never dropped to make a node fit.

The daemon's worker claims through `control_plane::claim_routed_job`: it
pages through the ready pending jobs of its class, best first (16 per page,
keyset-ordered by priority, `created_at`, id, until it claims one or the rows
run out, so jobs this node cannot run never hide runnable ones behind them),
and skips one unless every required instance has a deployment on this node whose desired
state runs it (`present` or `placed`) and whose observation from this node
says the guest is up (`running` or `healthy`) — the same placement and
observation rows delivery routing reads. Then it takes the job's library
permits (below) and claims that row with `claim_job_by_id`, one CAS on
`state = 'pending'` and the lease generation it read, so concurrent
claimers on any number of connections admit one. A lost CAS gives the
permits back and tries the next candidate. A job whose account or instance
semaphore is full is skipped too; a full class semaphore ends the attempt.
Requirements recorded at admission are kept current by the requirements
sweep below.

### Requirements sweep

Deployments keep moving after a job is queued: a `local` output is deployed
or retired, or the scheduler moves a replica. Each daemon runs a sweep loop
(`bookclerkd` `requirements_sweep`, rules in
`control_plane::sweep_job_requirements`), but only the holder of the
`jobs:requirements-sweep` library lease (90 s, renewed every pass) sweeps.
A pass runs every 30 s, on a deployment or install wakeup on this node, and
after a scheduler pass that moved replicas. A wakeup waits until wakeups have
been quiet for 0.5 s (at most 3 s), so a burst of deployment changes, such
as enabling several plugins at once, is swept once. Discovery runs on a
blocking thread. It re-resolves the requirements
of every pending acquire, pinned job, or job that already has requirements,
with the node-bound output keys the daemon discovers plus every instance
keyed as the first-party `local` destination (so a sweeping node without
that plugin, or whose discovery failed, never strips them; a pass that cannot
read the instance list is skipped), and:

- **applies at once** a set that is unchanged or only gains instances when
  some node is deployed to run all of them (a newly deployed `local` output
  is added to queued acquires), clearing any unsettled mark;
- otherwise (**unsettled**: no node runs all of them, or a recorded instance
  is no longer deployed anywhere) stamps
  `JobPayload.requires_unsettled_since_ms` on the first sweep and drops no
  recorded instance for a **10-minute grace period**, so a placement move that
  retires the old replica before the new one is up never fails queued work.
  Instances the job gained are still recorded at once (the recorded set
  becomes the union of old and new), so a job that waits cannot be claimed on
  its old set and skip a newly deployed `local` output;
- when the grace period has passed, writes the new set if some node can run
  it (an instance was only dropped, which is what admission would record
  now), or fails the job with `error_kind = "unroutable"` and a message naming
  the instances. A failed job frees its `max_pending` slot and dedup key.

A job pinned to a node (`JobPayload.node_id`) is failed as `unroutable`
when that node has never sent a heartbeat, or its last one is older than
the grace period, so a removed or long-stopped node does not keep its
queued work. While the node is live, any requirements the job has are
reconciled like any other job's.

The grace period is a wall-clock stamp rather than a count of unsettled
sweeps, because sweeps also run on wakeups and a count would expire faster
on a busy library. Every write is a compare-and-swap on `state = 'pending'`
and the lease generation, which it bumps: a job claimed meanwhile is left
alone, and a claimer that read the row before the sweep wrote it loses its
own CAS (a routed claim matches the generation of the record whose
requirements it checked) and moves on to re-read it on a later pass. A wakeup on a node that does not hold the lease does
nothing; the holder's timer picks the change up within 30 s.

Before requirements and permits, the routed claim filters candidates
through the trust predicate (`control_plane::trust`). The daemon's own
worker holds the database and is trusted with every job. A Branch node
(H-4) owned by a user who is not an Owner or Administrator may claim only
a scan or acquire of a named account linked to that user; a scan-all, any
pinned or instance-bound job, listen sync, integration scans, and plugin
copies are refused to it. `control_plane::handoff_routed_job` repeats the
check before a job's secrets would be handed over and, if the node lost
that trust, returns the job to `pending` without spending an attempt. It
also hands nothing over unless the claim still owns the job's running,
unexpired lease (same lease owner and generation), so a late handoff
after a reclaim, completion or expiry unseals nothing. See
*Per-item trust and the scan planner* in `docs/adr/control-plane.md`.

The unrouted `claim_next_job` (and its guest atomic plan) never claims a job
with requirements; it remains for tests and tools that run one process. It
pages past routed rows the same way (the atomic plan filters them out before
its `LIMIT 1`).

## Library semaphores

Process-local limits add up across nodes, so the limits that protect shared
things are counted library-wide. A semaphore `name` with limit `N` is the
`library_leases` rows `sem:<name>#0` … `sem:<name>#<N-1>`; a permit is one
of those slot leases (holder, fencing token, expiry), taken with the same
fenced compare-and-swap as the scheduler lease
(`control_plane::semaphore`). Only slots below the limit are taken, so no
more than `N` permits are live at once. A routed claim takes, all or none,
one permit of:

| Semaphore | Limit | Counts |
| --- | --- | --- |
| `jobs:<class>` | `[jobs.concurrency].network` | Running jobs of the class, library-wide |
| `plugin-instance:<id>` | `[jobs.concurrency].plugin_instance` (default 1) | Running jobs that require that instance |
| `account:<plugin_key>:<account_id>` | 1 | Scans and acquires touching that store account: the payload's account, else every account (of the pinned plugin, when pinned) |

Permits last as long as the job lease, are renewed with each heartbeat, and
are released after the row is finalized. A routed claim's holder is
`<lease owner>/<job id>/g<generation>/<random>`, with the lease generation
the claim runs at (the CAS only succeeds one past the generation it read),
so a node that finds its permits again by holder prefix never mistakes a
leftover permit of an earlier claim of the same job for the current one. A heartbeat whose permit renewal
matches no row (another holder took the slot) is a lost fence: the handler
is cancelled and its result ignored. When a node dies, its permits and job
lease expire; the next claimer reclaims the row, takes the slot with a
larger token, and the old holder can neither renew the permit nor finalize
the job (the job's lease generation moved on too). Lowering a limit does
not revoke slots above it; live permits on those slots still count, so no
new permit is granted until the live count is under the new limit. (That
count is read before the slot CAS, so racing claimers can briefly overshoot
the new limit, never the old one, until the high slots drain.) Each node reads the limits from its own
`config.toml` (`[jobs]` is not a database document yet), so nodes should
agree on them.

## Resource-class concurrency

`[jobs.concurrency].network` is the number of leased network workers on each
node (default **1**) and also the library-wide `jobs:network` permit count
above, so adding nodes does not multiply concurrent store traffic. The
historical `work_lock` still wraps `run_*` so the shared `StorageIndex` stays
single-writer within one process. Codec work stays in the `[media].workers`
pool.

## Adding a job kind

1. Add a `JobKind` variant, envelope fields, and dedup-key rule in
   [`crates/bookclerk-library/src/models.rs`](https://github.com/Bookclerk/bookclerk/blob/main/crates/bookclerk-library/src/models.rs).
2. Add a `JobCommand` arm on [`InProcessJobTransport`](https://github.com/Bookclerk/bookclerk/blob/main/crates/bookclerkd/src/job_handler.rs).
3. Enqueue from the API and/or scheduler with the same `EnqueueJobSpec`.
4. Reserve scratch paths via `LibraryStore::reserve_job_temp_path` when the
   job creates temp files; unregister one path after it is gone.
5. Add concurrent admission/claim, fence-loss, and restart tests.

## Configuration

See [configuration.md](/docs/server/configuration/index.md) (`[jobs]` / `BOOKCLERK_JOBS_*` and
`[events]` / `BOOKCLERK_EVENTS_*`).

## Library maintenance

Library-wide cleanup runs on one daemon at a time. Each duty holds its own
`library_leases` row; every daemon polls them every 60 s, starting 5 minutes
after it starts so the first pass doesn't compete with startup. The holder renews
its leases on each poll and runs a duty when it takes the lease and then
hourly. A daemon that stops polling loses its leases after 180 s and the
next daemon to poll takes them over with a larger fencing token. Before
each delete the holder renews the lease with a compare-and-swap on its
holder and token (the fence), so a daemon that stalled past its expiry is
refused before it deletes anything.

| Lease | Duty |
| --- | --- |
| `maintenance.jobs` | Terminal jobs older than `jobs.retention_days` |
| `maintenance.events` | Expires never-leased deliveries past their retention period, then deletes terminal deliveries, empty parent events, stale subscriber catalog rows (`[events]` retention) |
| `maintenance.kv` | Expired plugin KV rows (already read as missing) |
| `maintenance.sessions` | Expired operator sessions, OIDC RP states, WebAuthn challenges |
| `maintenance.records` | Expired credentials and spent tokens, finished uploads, abandoned scan inventories, and old audit rows (windows below) |

The `maintenance.records` windows are fixed in
`crates/bookclerk-library/src/store/retention.rs`, not configured. Each is longer than
the row can still be used; the grace periods only keep recent history
visible:

| Rows | Deleted when |
| --- | --- |
| OIDC authorization codes, portal sessions | Past `expires_at` |
| OIDC refresh tokens | Past `expires_at`, or revoked |
| Claim tickets, user invites, node enrollment links | 30 days past `expires_at` (expired and redeemed ones stay listed until then) |
| Committed or aborted host uploads | 1 day past `expires_at`; open uploads are kept, since an abort still reads them, and so is a branch's catalog upload while a file record names it as writer, since that is how its files are known to be the branch's |
| Storage-scan inventories | Job terminal or gone; a jobless (CLI) scan silent for 1 day |
| `security_audit_events`, `configuration_audit` | 365 days old |
| `configuration_changes` | 30 days old (the document and its audit row keep the content) |
| `notifications` with their delivery log and inbox rows | 30 days old |

The prunes are idempotent, so the fence bounds wasted work rather than
guarding correctness: the window between a fence check and its delete is
one statement. Node-local cleanup (scratch directories, expired job and
delivery leases) still runs on every node. `GET /api/library-status`
lists every lease with its holder, token, and whether it is held.
