Node API and the live socket

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

The wire protocols between the hosts of one library, and between the web app and its daemon. Setting up branches (including shared branches) is in Libraries and branches; the decision record is the control-plane ADR.

The node API

All of it is under /api/node/v1/, outside the browser session and operator-auth layers, and every request first passes the transport policy.

Route Purpose
GET info The library: {library_id, library_name, origin, transport, enrollment, spki_fingerprint?, software_version, ancestry}
POST enroll Redeem an invite: {link_id, secret, sign_pk, box_pk, software_version} → {library_id, node_id}
POST enroll/oauth Redeem an approved sign-in: {code, code_verifier, redirect_uri, sign_pk, box_pk, branch_name, software_version, connection?} → {library_id, node_id}; for connection: "shared_branch" also {connection, bootstrap} (sealed to box_pk); for a branch whose user chose to bring their store connections, also accounts, the user's shareable store accounts and secrets sealed to box_pk
GET connect The node socket (WebSocket, subprotocol bookclerk-node.v1)
GET artifacts/{sha256} An owner plugin archive, for a request signed Authorization: Bookclerk-Node <node_id> <issued_at> <sig>
PUT uploads/{id}/files/{n}, POST uploads/{id}/commit Uploads (see Uploads)

One route sits beside the node API, under /api/copies/v1/, with the same transport policy and its own bearer token:

Route Purpose
GET /api/copies/v1/{copy_id}/files/{n} One file of a copy the library granted, for Authorization: Bookclerk-Copy <token>. A branch gets the token from copy.grant on the node socket, a clerk from copy.grant on the clerk socket (see Taking a copy from the library)

Handshake. hello {node_id, incarnation, software_version, resume} → challenge {nonce, library_id} → auth {sig} (Ed25519 over the library, node, incarnation, nonce and the owner's canonical origin, so a signature relayed through another server fails) → welcome. An unknown node or a bad signature closes with 4401, a revoked node with 4403, a newer connection for the same node supersedes the old one with 4409, and a binary or malformed frame closes with 4400.

Frames are JSON text, {"v":1, "id", "type", ...}, at most 1 MiB; a request is answered by a frame with the same id. Both sides ping every 20 s and close after 60 s without a frame.

Host → owner Owner answers
heartbeat ok (capacity and use land in node_status)
claim {filter: {kinds, plugins, scope?}} work (job, lease, fence, sealed secrets, for_node_owner on the job when all its accounts are the node owner's, and for a scan scan_tokens: the owner's sync token per handed account whose last complete pass is under 7 days old) or none {retry_after_secs}
renew renewed, or lost when the fence is gone
complete, fail, release ok, or lost
deployments.list deployments (what targets this node, with versions and the requested grant)
instance.config CONFIG, and SECRETS sealed to the node only for a node trusted with every job
observation ok (materialized or error for its own deployments)
secret.update ok (a refreshed store token, sealed to the owner, only for a ref handed with a live job), or error refused kept_by_branch for a single-keeper account a branch keeps and has pushed a sign-in for
accounts.update {sealed, revision} accounts.updated {outcome}: applied, stale (not newer than the last one applied) or not_shared (not an account this branch keeps there); see One refresh keeper
scan.results scan.accepted {batch, upserted}
scan.removed {account_id, batch, product_ids} (at most 500 ids, after the job's rows) scan.accepted {batch, upserted} (holdings removed), lost, or error refused
scan.pass_end {account_id, sync_token, complete, full} (one per account, last) scan.accepted {batch: 0, upserted} (holdings a complete full pass pruned), lost, or error refused
acquire.plan acquire.targets {targets, destinations}: each target is {holding_id, plugin_key, account_id, title, call_number?, parent?, parent_call_number?}, where title and parent are store listings {product_id, source, marketplace, title, content_kind, is_abridged, asin?, isbn?, subtitle?, authors?, narrators?, series?, series_index?, series_product_id?, publisher?, categories?, length_minutes?, purchased_at?, published_at?} (dates in RFC 3339)
upload.grant (each file of the manifest may carry probe {bitrate_kbps, lossless, source_store}, as may catalog.upload's) upload.granted {upload_id, sealed_token, expires_in_secs}
catalog.begin {share_books, include_branches} catalog.session {fence, cursor, ancestry, destinations, wanted, members_books}; members_books (absent reads as false) is the library's "Acquire all of this branch's members' books": the branch then also sends the members' books it lists (see Acquiring all of a branch's members' books)
catalog.sync {fence, batch, cursor, done, titles} catalog.accepted {batch, upserted, skipped, wanted}, or lost
catalog.upload {fence, store, product_id, files} upload.granted (then the upload routes above, job id catalog)
catalog.questions catalog.ask {works}: up to 100 questions {work, identifiers} waiting for this branch, or error refused
catalog.found {answers, missing} ok, or error invalid / refused (see Shared catalog)
requests.claim requests.claimed {requests}: up to 16 interlibrary requests {job, generation, work, store, marketplace, product_id} leased to this branch for 5 minutes, or error refused
requests.finish {job, generation, outcome} ok, lost when the lease moved on, or error invalid for a malformed job id or generation; outcome is accepted, not_listed or failed (see Requesting a book from a branch)
window.books {after} window.page {works, next?}: up to 100 works {work, identifiers, editions} of the library's books, or error refused / invalid (see Books in other libraries). Each edition is {edition, yours?, copy?}; copy {source_store?, content_type, bitrate_kbps?, lossless?, version} describes the audio file the library holds of it (a lowercase type/subtype media type of at most 64 bytes, 1 to 20000 kbit/s, and a version of 1 to 64 letters, digits and - that changes when the file does) and is absent when it holds none. after and next are opaque cursor strings: empty or absent for the first page, otherwise up to 64 letters, digits and -. Each next sorts after the after it answers.
copy.grant {store, marketplace?, product_id} copy.granted {copy_id, sealed_token, expires_in_secs, copy, files}: the copy the library holds of that listing's edition, for the node's owner. copy is the same object a window.page edition carries. files lists 1 to 1000 files {name, role, size, content_type?, probe?} with distinct names, the primary audio (role audio) first and exactly once; the other roles are part, chapter, pdf, cover, sidecar, transcript and epub. A file's place in the list is its {n} in the download route. The token is sealed to the node's box key and lives expires_in_secs (one hour). Otherwise error refused (not a branch this library trusts), not_found (the person may not take the edition, or the library holds no copy of it; one code for both), invalid, or unavailable (the library can't read its storage right now).
node.leaving accepted, then close 4403 (the node is revoked)
scan.request {account, plugin_instance_id} scan.planned {created, duplicates, queue_full}, or error forbidden / invalid
job.status {job_ids} (at most 64) job.statuses {jobs}: only jobs this node may claim

A claim's scope is own (the default, left off the wire) or all_members. Under own the owner hands a branch trusted with every job (an Owner's or Administrator's) only the jobs whose accounts all belong to the user who owns the node; all_members takes every job the node is trusted with. A scope never widens trust. A branch always sends a filter.

The owner also sends wake frames (deployment, install, placement, configuration and revocation changes, catalog.asked when catalog questions wait for a branch, and requests.queued when interlibrary requests do), which the host acks. Secrets are never frame text: job secrets, instance secrets, upload tokens and copy tokens are sealed to the node's box key and bound to the node and the job or instance they belong to; a write-back is sealed to the owner's box key. A host job's storefront guest that renews an account's tokens mid-job hands them back (Refreshed credentials), and the host writes them back to the owner this way, as the account row the job was handed. A single-keeper account the branch keeps for that library is the exception: the branch saves the refresh in its own library, and its link pushes it up as accounts.update.

The live socket (/api/live)

The web UI holds one WebSocket to GET /api/live (subprotocol bookclerk-live.v1) and re-reads a panel's ordinary routes when the daemon says its topic changed, instead of polling on timers. The socket carries topic names only (library, jobs, deliveries, nodes, config, consents, libraries, users, discover, playback, notifications), never rows, ids or secrets:

{"v":1,"type":"hello","topics":["jobs","nodes"]}
{"v":1,"type":"changed","topics":["jobs"]}
  • Origin must match exactly: the configured integrations.public_origin or, without one, http://<Host> for a loopback Host with no forwarding headers. Anything else (or no Origin) is 403 origin_mismatch, also with auth.enabled = false, so another site's page cannot open it.
  • Session. The operator, or a signed-in user. The operator hears every topic; an Owner or Administrator hears library, jobs, deliveries, nodes, users, discover, playback and notifications; anyone else hears library, nodes, discover, playback and notifications. discover says only that someone's stored shelves changed, playback only that someone's progress or bookmarks did, and notifications only that someone's inbox did; each page re-reads its own. The session is re-checked every 30 s, and a signed-out user's socket closes with 4401.
  • Changes are found by reading revisions from the shared database every 2 s (so a change made on any shared node, or by bookclerk libraries join|leave, is seen) and by in-process notices for immediate updates. Each topic has one counter row in live_topic_revisions, and every write to a topic's tables adds one to it in the same atomic batch as the write, so a poll reads nine small rows rather than counting tables. A sweep that changes nothing (no expired claim, nothing to prune) leaves the counter alone. New code that writes one of these tables must bump its topic the same way (exec_bumped, or bump / bump_if_stamped / bump_if_any in a host batch or named plan). The first read is taken before hello, so nothing written after the client hears hello is missed. The client sends nothing; a binary frame closes the socket.
  • While the socket is down, the UI polls as it used to and reconnects with backoff (1 s up to 30 s, jittered). Behind a reverse proxy, forward the WebSocket upgrade (see Operations → Reverse proxy).

Source: docs/internals/node-api.md at main · View as Markdown