Discovery design
How Discover finds and ranks books. For using Discover and the wishlist, see Discover and wishlist.
Bookclerk’s discovery stack turns local taste (owned library + listening) into unowned storefront candidates, then ranks those with embeddings and heuristics. Operator-facing surfaces ship first; the schema is ready for Connect-portal identities later.
- Discover unowned titles from storefront catalogs (Libro related books, Audible public search / series ASIN, Chirp GraphQL related/series/author/ deals, GraphicAudio Magento series + related), not only from the owned library.
- Evaluate those candidates against local ownership, listening, ratings, and embeddings.
- Personal wishlist (store-agnostic) with a global request queue ranked by how many users wishlisted the same work — including one-click from Discover cards and catalog search.
- Suggest where to buy once a candidate is known.
- Stay fit for a cheap VPS (≈1 vCPU, 1–2 GB RAM): no separate vector DB; embeddings score remote candidates locally.
Owned library rows are seeds, not the candidate pool. Open wishlist items merge into the personalized Discover feed (and the shared global queue).
Chirp and GraphicAudio expansion prefer seeds already owned on those sources
(Chirp product id → relatedAudiobooks; Magento product id → related block +
series page). Series / author signals from any seed can still query Chirp or
Magento when the catalog has a match. GraphicAudio series-set SKUs are
included by default; set exclude_graphicaudio_series_sets = true to drop
them. Chirp top/free deals are pulled once per recommend run (unowned only).
Listening taste is optional and provider-agnostic. Integrations that
support listening sync (Audiobookshelf today; third-party plugins via capability
sync_listening) write into the shared listening_progress table, one row
per listener and work (see Listening sync).
Discover reads each row's authors and library title through its work. Ranking never
calls those adapters — if no listening data is present, or you pass
--no-listening to the CLI, discovery still runs on owned-library taste
alone. The CLI scopes with external_user_id and/or --listening-provider for
multi-user or multi-integration libraries; the daemon scopes each stored feed
to its person (see Precomputed shelves).
The daemon never builds shelves while a page waits. Each person (the
operator, a household user, or a portal identity) has a subject key
(operator, user:{id}, portal:{id}) and two stored layers:
-
Candidates (
discover_candidates): adiscover_candidatesjob seeds from that person's own taste (books on the store accounts linked to their identities, household books they have listened to, their listening and their open wishlist, with book and wishlist seeds alternating so each store's call budget reaches both) and asks every deployed storefront, not only the stores they have accounts with. While a person has fewer than five books and listens of their own, the rest of the household's books blend in at half weight, behind their own, with the household's listening to those books at the same weight: a first wish or book adds to the shelves they already had instead of replacing them, and the household drops out once their own taste can stand alone. A person with no signal of their own yet (and the operator) seeds from the household as it stands. With fewer than five seeds in all, each store is also asked for a fixed set of starter genres (Science Fiction, Fantasy, Mystery & Thriller, Romance, History & Biography, Self-help) through its catalog search, as far as its call budget reaches, so a new library still gets shelves. Every title a store returns goes into the shared listing pool (see Shared listing pool), and the person'sdiscover_candidatesrows are references to those listings plus what is personal to them: the seed that found the title, how the store found it and the starter genre.storefront_max_remote_callsis split across the stores, so all stores together never make more calls than it allows (deals count as one call, and a store whose share is zero is not asked). Only calls the store actually answers count: an answer read from the 24-hour store-call cache is free, so a new wish is asked about on top of the books whose answers are cached instead of pushing their titles off the shelves. Stores run concurrently, each call has a 10 s timeout, a store stops starting new calls 20 s into a fetch and saves what it has, and each store's rows are saved as that store finishes, so one slow store never discards the others. A save upserts the store's new rows, then deletes its rows that fetch did not return. It sends them as a few atomic host batches, each as large as the database plugin'smaxStatementsandmaxRequestBytesallow, not one statement per row and not an interactive transaction, which the external database plugins refuse. The delete rides in the last batch, so a reader can briefly see part of a store's new rows beside its old ones. A store whose calls all fail or come back empty keeps its previous rows, since a plugin adapter can report a guest failure as an empty answer. A store with an answer that had a title the pool could not place this time keeps its previous rows beside the new ones, so that title is not dropped until a complete fetch replaces them. The finished fetch is recorded indiscover_fetches, so a fetch that found nothing new still counts as fresh for a day. -
Feed (
discover_feeds): when the fetch finishes it admits adiscover_feedjob, which ranks the stored candidates into shelves with no network calls and stores the result. It uses only candidates fetched for the person's current seeds (their own, or the household's), and none whendiscovery.storefront_candidatesis off.The fetch reads the library once: the taste and the set of titles the library already holds (by ASIN, ISBN and store product id, used to keep owned titles out of the candidates) come from the same read. The read is one title per work the library holds, built from holdings, listings, editions and works (a book held on two accounts or from two stores counts once). Finished flags, ratings and tags come from each user's own state on the work (
user_title_state): a person's own books carry theirs, and household books carry everyone's merged (finished when anyone finished it, each rating the mean of those who gave one). A work a person finished, rated or tagged counts as one of their own books, so a user with no portal identity or linked store account still gets a taste of their own once they have any. A portal visitor with no user has no stored state.The fetch then stores what the ranking needs from that taste in
discover_tastes, one row per person: the languages they listen in, the authors, narrators, categories and series they like with their weights, the works whose embeddings make up the similarity centroid, and the shelf titles' context (wish titles and the stores they have an account with included). The ranking reads that row instead of the library, so its cost follows the person's candidates, not the size of the library. It checks the candidates it ranks against the library by their ASINs, ISBNs and product ids only. Like a stored feed, the row is one cell, written as one bind of one statement: a taste whose JSON would pass 128 KiB, or what the database's statements carry less 1 KiB for the rest of the upsert (about 2.6 KiB on D1), keeps only its heaviest entries, which move a score the most, and its first wish titles. A ranking with no stored taste, or one more than a day old, loads the taste from the library and stores it again. Every taste event queues a fetch, which stores a fresh taste (also withdiscovery.storefront_candidatesoff, when the fetch calls no store), so the day only matters for finishes, ratings and tags, which send no event.Both steps read through the database plugin's streamed result path (SQL contract), so a page of candidates or a stored feed larger than one inline 256 KiB result still arrives whole. Against a database plugin without streamed results, the ranking halves a page of candidates when its result would exceed the inline limit (
maxResultBytes). A stored feed is one cell, and the plugin counts a row as JSON, where each quote or backslash in the feed takes two bytes. So the stored feed keeps card references, not whole cards. Each card keeps what is personal to the person (its shelf, order, score, reasons, shelf tags, prices and wishers) and its identity (title, authors, series, store editions). The catalog metadata that the person's candidates' listings hold in the pool (description, cover, subtitle, publisher, length, date, genres, language and narrators) is left out and joined back from those listings when the page reads the feed, in chunks that stay under the inline result limit. A field is left out only when that join gives it back unchanged: metadata merged in from another store edition is left out when that edition's listing gives the same value, and stays on the card when none does. A card with no stored candidate, such as a wish, keeps its metadata. A fetch that changes candidates before the page reads the feed changes what the join fills in: an updated row shows its new metadata, a removed row lets another edition's row fill the field, and a field no row holds stays empty until the ranking that fetch queues replaces the feed.GET /api/discover/statusneeds only whether a feed exists and when it was ranked, so it reads the feed without the join.The stored feed carries a format number. A feed in another format, such as the whole cards an older release stored, reads as missing and is ranked again. The ranking job logs the stored feed's size and card count (
stored discover feed). As a backstop the stored feed is kept under 192 KiB in that escaped form: if it is ever larger, long descriptions left on cards are shortened to 300 characters, then each card keeps three reasons and dropsseed_categories, then every shelf is cut to its best items (down to four), and last the final shelves are dropped. Writing a feed replaces the old row without reading it, so an old feed that is too large to read cannot block its replacement.
Note
A candidate fetch asks the storefronts deployed on the node that claims it. In a library that splits storefronts across nodes, stores deployed only on other nodes are not asked yet.
Storefront answers are cached for 24 hours in discover_store_calls, keyed by
a hash of the store and the request, so people whose seeds overlap share one
call. The cache stores the parsed answer, not the store's raw reply: the
listings it returned, in order, in discover_store_call_hits, each with how
the store found it and its rank. Failed and empty answers, and answers none of whose titles could be stored,
are not cached. The fetch time is written after the hits, so an answer cut
off part way reads as expired. Each write claims the cached call with its
own token, and its hits and fetch time apply only while the call still
carries it, so two members refreshing one call at once never leave a mix of
both answers.
A fetch reads the cached answers of every call a store's budget can reach, and the listings they name, in a few batched reads before it makes any call. An answer it makes and caches is reused by a later call in the same fetch with the same key.
Catalog data has one home: the library model's store_listings
(ADR). When a store answers a Discover call,
each title is resolved into the library the way a scanned title is (shared
identifiers first, then the work's identifiers, then the work-level match on
title, authors and kind), so it gets a work, an expression, an edition and a
listing row. The listing has no holding, so the library does not own
it, and a later scan of the same product finds the same listing and only
adds its holding. Work identity always comes from that resolution, never
from comparing display text. Two stores' listings share an expression only
when kind, language, abridgement and narrators are known and equal on both,
so the resolver compares languages by their ISO 639-1 code (one store's
"English" and another's eng or en-US are one language), and the Audible
catalog reports abridgement from each product's format type.
The store's own text for the listing is stored on the row as parsed
columns: subtitle, authors and narrators as the store prints them, series,
the whole synopsis (never shortened), publisher, date, length, language,
cover, product URL, ratings and abridgement. What it costs is stored in
listing_offers, one row per listing and audience: non_member (what
someone without an account at the store pays) and member, each with the
price, the store's label, the currency and when it was quoted. An older
quote never overwrites a newer one.
A listing a store reported in the last day keeps its resolution and text,
and a new answer only refreshes its offers, so members whose tastes overlap
cost one resolution a day per title. A title that other writers kept
changing on every attempt is skipped and stays stale, so the next answer
resolves it again, and the answer that skipped it is not cached; any other
read or write failure fails the whole call. A discover_candidates job also
deletes listings that no store has reported for a week and that nothing
refers to: no holding, candidate, cached call, wish or resolution override (at
most once an hour on each daemon). The edition and work they made stay.
A new catalog listing emits no library event; a scan that later holds it
announces it with holding_added. A catalog answer that fills or relinks a
stored edition announces that change as a scan would.
The discover_candidates and discover_feed jobs are deduplicated per
subject, so a burst of triggers folds into
the job already waiting.
GET /api/discover/recommendations reads the stored feed and returns at once,
filtering the caller's hidden shelves. Its state says what the page is
seeing:
state |
Meaning |
|---|---|
ready |
The stored feed is current. |
refreshing |
The stored feed is shown while a newer one is built: a read found it stale, or a fetch or ranking for the caller is waiting or running (queued by a taste event or the Refresh button). The page shows an "Updating shelves" notice. |
preparing |
No feed yet (or the stored one does not decode); shelves are empty and the page shows placeholders. |
When a ranking job stores someone's shelves, the live socket's discover
topic changes and an open Discover page reads its feed again in place.
While the socket is down and the state is preparing or refreshing, the
page reads the feed every 4 seconds instead, up to 15 times. A preparing
page then keeps reading every 30 seconds until shelves arrive. With the
socket open, a preparing or refreshing page also reads every 30 seconds,
in case a write was missed or a live re-read failed. A live re-read that
fails is also retried 30 seconds later whatever the feed shows, so a ready
page does not keep stale shelves until the next change.
A read queues what is stale. It queues a ranking when the last fetch finished after the feed was ranked. It queues a fetch when none has finished or the last one is older than 24 hours.
The scheduler queues the daily fetch for every subject with a stored feed. Deleting a user removes their Discover rows. A fetch or ranking still queued or running for them deletes its rows instead of writing them back.
GET /api/discover/status reports whether the caller's shelves are stored
yet (and, as updating, whether a fetch or ranking for them is waiting or
running), and queues their refresh as a read would. Callers that must wait
on shelves, such as the hosted demo, poll it. Any signed-in caller can use
it, since it only reports the caller's own shelves.
POST /api/discover/refresh queues a storefront fetch (and the ranking
that follows it) for the caller now and answers with the same status. The
Discover page's Refresh button calls it: unchanged store calls answer
from the day's cache, so a refresh is cheap, and a fetch already waiting
absorbs the request. When the job queue is full the route answers 429
and queues nothing, and the page shows the error.
Shelves also refresh when someone's taste changes, without waiting for a
read or the daily fetch. The daemon registers one host consumer,
host:discover, on the durable event outbox for five host
events:
| Event | Published when | Discover queues |
|---|---|---|
wishlist_changed |
A wishlist entry is added or changes status, or an open one is deleted with its user. | A fetch for the person who wished (their seeds changed), and a ranking for everyone else with shelves (wish counts changed). |
listening_synced |
A listening sync changed, for some stored row, whose it is, which owned book or work it matched, its authors, whether they started it, which quarter they are in, or whether they finished it. | A fetch for everyone with shelves. |
book_acquired |
The household acquires a book. | A fetch for everyone with shelves. |
holding_added |
An account or a branch comes to hold a listing. | A fetch for everyone with shelves. |
holding_removed |
An account or a branch stops holding a listing. | A fetch for everyone with shelves. |
Every one of the five events also queues one ranking of the shared request queue.
Each job it queues may only run 30 seconds later, so the changes of one sync or one browsing session fold into a single job per person. The one exception is the fetch for the person who wished: they are looking at the page, so it may run after 2 seconds. The fetches are cheap when little changed, because unchanged store calls come from the 24-hour cache. Apart from the person who wished, someone without stored shelves is left to their first read.
A job that has already started may have read the data before the change, so a change only folds into a job that is still pending. When the job the consumer needs is running or has just finished (or the queue is full), the delivery comes back 30 seconds later and queues again, up to 20 times. A wish of the person looking at the page comes back after 5 seconds instead, so a wish clicked while their fetch runs gets the next one soon after. A retry skips each person who already has a job of that kind admitted after the event (waiting, running or finished), so with more people than the queue holds, finished refreshes are not queued again ahead of the people still waiting for room. Every node registers the same consumer and the outbox delivers each event once in the library.
A listening sync compares each row it writes with the row stored before it, so a sync that reports only some people, or repeats what is stored, publishes nothing, while a book finished, unfinished and finished again is three changes. When one sync reports a row twice, its first stored state is compared with its last.
A ranking job checks the newest candidate fetch again when it finishes. A fetch that finished while it ranked had its follow-up ranking folded into the running job, so the job ranks again, up to three passes in all.
Ratings have no event. Discover weighs a book's own rating, but ratings change only when a Libation library is imported, and the next daily refresh picks that import up.
discover enrich-metadata (and discover recommend) fill low-volume
metadata gaps on held editions (subjects / description / cover /
language) from the deployed metadata-provider plugins, such as
openlibrary. It walks editions that some account or branch holds, a page
at a time by edition id, and writes what it finds to the edition's work
(description and subjects), its expression (language) and the edition
itself (cover). It is never a bulk catalog backend.
Per Open Library API guidelines:
- The
openlibraryplugin identifies itself with itscontact_emailsetting (integrations) - The host spaces editions about 400 ms apart and stops after 25 per run. It reads the library a page at a time and stops reading once the run's requests are spent
- An edition with an ISBN is looked up by ISBN; one without is searched by title and author, and only a result with the same title (ignoring case and punctuation) is used. The first record that fills a gap wins, and the work only takes the fields it lacks
- Editions already enriched are skipped (the edition's
enrich_sourcemetadata:<provider>, or the legacyopenlibrary) — provenance cache. The work, expression and cover are written first, each taking only the fields it lacks, and the edition is marked last. A failure before the mark keeps whatever was already written and leaves the edition unmarked, so a later run asks again while any of its gaps is still open - For bulk data, use monthly dumps, not the API
WorldCat remains reserved (API key + ToS).
The host loads no model. Models run in plugins and are called through one router (see the model plugins ADR and Model plugins).
Until Discover is moved onto that router, its shelves rank with the
local-hash embedder (local-hash-v1, 384 numbers): a hash of the
words of a text. It needs no download and no extra runtime, and it
carries no meaning, so two books score alike only when their texts share
words. Hash vectors are stored under their own model id and are never
compared with a model's vectors.
The embedding model benchmarks record what a real model adds and which models the first-party plugin will ship.
Embedding model benchmarks compares MiniLM with newer and multilingual models on taste quality, memory and speed.
embed_dirty_works keeps one vector per work and model in embeddings,
stamped with a hash of the text it was computed from. It first deletes the
vectors of works that no longer exist, then walks works in id order, 128
at a time. Each page costs one read of the works, one read of their stored
hashes, and one embedder call for the works whose text changed, so a large
library is neither held in memory at once nor read one hash per work.
lofty, the tag library bookclerk-media uses, pulls crates.io paste
1.0.15 as a compile-time proc-macro, so it is in Cargo.lock, which
OSV scans. That crate is unmaintained
(RUSTSEC-2024-0436:
INFO, no CVE, no patched release). There is no crates.io drop-in under the same
name; renaming to maintained pastey would
still leave OSV matching the lockfile package name paste. Bookclerk therefore
keeps the registry dependency and records a narrow, dated ignore in
osv-scanner.toml (ignoreUntil). That ignore filters
the INFO finding from the actionable scan result; the paste package remains
in the lockfile. Revisit when upstream depends on pastey (or drops paste)
directly, or when the ignore expires.
Candidates are unowned storefront hits (plus open wishlist items), scored by:
- Storefront origin (related / series / author / catalog search)
- Series completion — own some of a series → boost unowned siblings; prefer
the next index; stronger when the series is actively listened to, and
stronger still when multiple books in that series have listening activity.
Listening weight is driven by absolute hours heard (soft-saturated), with
percent complete only a secondary bonus — so 50% of a 30‑hour book outweighs
50% of a 3‑hour book. Finished titles still get a completion bump on top of
hours listened.
A candidate counts toward a series only when its storefront lists it in
that series. Audible's series keyword search also returns loose matches,
so the Audible plugin keeps only results whose own series has the same
name. Names are compared without a leading article or trailing generic
words such as "Series" or "Sequence", so "The Dune Sequence" matches
"Dune" but "Midnight World" does not match "Midnight".
The Audible plugin asks, in order, for the seed's series, more by its
author, more by its narrator, and then the titles Audible lists as
similar to the seed (
/1.0/catalog/products/{asin}/sims, its "listeners also enjoyed"), and keeps that order when it cuts the list to the requested size. It never keyword-searches the seed's title: that matched words, not books (The Great Hunt found Turkey Hunting for Kids). A narrator's other titles are kept only when one shares a top-level genre (the first rung of an Audible genre ladder) with the seed, looked up by the seed's ASIN; a seed with no ASIN or no genres gets none. Narrators also record children's books, so the narrator of The Silent Patient brought Panda Panic. - Overlap with liked authors
- Genres the person likes, read from the candidate's own genres
(
genre_matchinbookclerk-discover). The genres of the seed the store was asked about count only for a candidate whose store gave none, and then at a third of the weight: a store finds candidates from a seed, but that does not make them the seed's genre. Each liked genre counts once per candidate, however often the store's genre ladders repeat it. - Embedding similarity to finished / highly rated / recently listened works
- Wishlist boost (
wish_count × 40on the shared queue / Discover merge) - Purchase hints resolved at view time (see below)
Before scoring, a candidate in a language the reader does not listen in is
dropped (retain_reader_languages in bookclerk-discover). The reader's
languages are the normalized languages of the taste books (the subject's own,
with the household's blended in while those are few), or the default
preferred language (en) when no taste book has one. A candidate whose store
gave no language, or one normalize_language does not recognize, stays, as
in catalog search, so a store's related titles should carry their language.
Libro.fm's search and related lists leave it out (they often hold a seed's
translations), so its plugin reads each related title from
explore/audiobook_details, which gives the language, the list price and
whether the title is for sale, and leaves out one that is not. Editions in two different languages are different
expressions, so they are never merged into one candidate, and dropping one
language never drops the other edition with it.
Whether a wished title is already owned is decided by the library's
resolver (LibraryStore::owned_work_for), the same way a scanned listing
finds its work: shared identifiers, then the work's identifiers, then the
work-level title-and-author match. A wish whose work no library title
resolves to stays on the queue, so Dune Messiah is not hidden by an owned
Dune. Storefront candidates are dropped by owned ASIN, ISBN and product
id, and then by work: a candidate read from the shared listing pool uses
the work its listing resolved into, a live store hit is resolved the same
way (LibraryStore::work_for), and one whose work has a library title is
dropped when the feed is ranked. Another store's edition of an owned book
(a different ASIN, ISBN or narrator) therefore never reaches a shelf, such
as Because you wishlisted, while a sequel stays.
Shelves then partition the ranked list so that every title lands on one shelf. Shelves claim titles in page order: the wishlist first, then the series shelves, the person's own wishes, authors, narrators, genres, starter genres, similar taste, Chirp deals, then one Picked for you on {Store} shelf per store for whatever is left, each title under the store where it costs the person least (member price on stores they have an account with, list price elsewhere, its own store when no price is known on the card). A wished title shows on the wishlist shelf alone (every wish shows, however many share a series; wishes past the shelf's size are claimed too, and shown nowhere). A shelf that is not about series keeps at most two titles of one series, so a long series cannot crowd the others out; the two series shelves and the wishlist instead rotate between series, so their first cards show several series rather than one in a row.
There is no triage / approve / reject path. An item stays in the global queue while at least one user has it open on their personal wishlist and the title is not in the household library.
- Personal wishlist:
GET/POST/api/wishlist,DELETE /api/wishlist/{uuid}(un-wishlist = cancel own row only) - CLI:
bookclerk discover wishlist add|list|remove - Global queue:
GET /api/request-queue— shared order using overall / operator taste (not per-portal personalization) plus a heavy wish-count weight. Each row includeswishers(tiny profile pictures: upload / Gravatar / SSO / monogram) for the people who have the title open. Discover'srequestsshelf (the first row) carries the samewishers/wish_countfields. The route only reads a stored ranking; see Request queue. - Identity merge: a wish is resolved to a library work when it is created
and stores the
work_id. A wish that names a store edition the server could look up is for the work that edition's pooled listing resolved to (see Per-source snapshots); otherwise its own title, authors and identifiers find the work (LibraryStore::work_for_wish, the same resolution a scanned listing gets); wishes with the same work are one queue entry, and Discover treats such an entry as owned when a library title resolves to that work. The entry (GET /api/request-queue) carries thatwork_id, and the Wishlist and Discover pages match the viewer's own wish on it before the bibliographic key, since the entry keeps only one wisher's key. Wishes the library could not place fall back to canonical ISBN-10↔13 when present, else ASIN, else soft title+author; ASIN-keyed and ISBN-keyed rows for the same work are merged (ISBN is not universal across Chirp / GA / Audible public search) - Per-source snapshots:
POST /api/wishlistresolves live prices and further stores (server-side purchase hints) while it asks each store about every edition the wish names (catalog_detail, at most eight editions at once), so a slow price lookup never costs the wish its editions. Editions that only the price lookup found are asked about with the time left. It records the store's answer in the shared listing pool, with any member price a purchase hint quoted. It then persists the editions, HTML description and purchase hints intotitle_request_sources(one row persource+product_id), each naming its pooled listing inlisting_idwhen the store answered. The pool holds only what stores reported: the text and prices the browser sends stay on the wish's own rows, so one person's wish never changes what anyone else's Discover shows. The hints and the edition lookups share one 5 s deadline, well inside the route's 8 s budget, so the wish is always saved; an edition whose store is not deployed, fails or does not answer in time is saved without a listing.GET /api/wishlist/ request-queue merge those rows for the richest blurb and multi-store Where-to-buy links (legacy thin rows still fall back to provider title-meta). Title detail loads Audible reviews via paginatedPOST /api/discover/title-reviews(sort_by=MostHelpful|MostRecent, infinite scroll seeds helpful then continues recent); title-meta carries ratings/counts only. When the provider record is missing or sparse, title-meta gap-fills from storefrontcatalog_detail(narrator, runtime, description, publisher, genres, abridged): the request carries the title'sstore_editions(at most 16), and the host asks each listing store that declarescatalog.detailByProductId(Libro.fm, GraphicAudio) for its own product (two at most), then the stores that declarecatalog.detailByIsbn(Libro.fm) for the ISBN, three lookups and 3.5 s at most. No plugin id is named.POST /api/discover/title-meta/batchanswers every title within 6 s: a title still being looked up then answers with what it has and is flagged in the response'sincompletelist, so the page shows it but does not cache it, and the log names the lookup it was waiting on. After multi-store search ranks a page, Discover asks the same stores to fill lean rows, by ISBN or by their own product id (bounded timeout), so list cards get genres/runtime without N+1 product fetches inside each store’ssearch_catalog. See Catalog search capabilities.
Multi-region storefronts are deferred (US default for now).
The shared queue is ranked by a job, not inside the page request. Ranking reads the whole library and listening history, so doing it per request held the Discover gate and slowed every Wishlist load as libraries grew.
- A
discover_request_queuejob ranks every open entry against household taste. It stores one row per entry indiscover_request_queue, keyed by the work every wish names: the taste score, up to six taste reasons and whether the library already holds the work (an account or a branch holds a listing of one of its editions). It copies no title, cover or price: those stay with the wishes. - Discover's consumer (see Taste events) queues the job
on every taste event (
wishlist_changed,listening_synced,book_acquired,holding_addedandholding_removed), after the same 30-second debounce. The queue is not personalized, so one job serves everyone, and its dedup key folds a burst of events into one pass. GET /api/request-queuereads the open wishes, joins each entry with its stored row, and adds the wish-count boost and the wish reasons from the live counts. It hides entries the last pass found in the library. An entry wished since that pass shows straight away with no taste score, so its wish count alone places it.- When an entry has no stored row, or the ranking is older than a day, the read queues a pass and still answers with what it has. The day limit covers taste changes no event announces, such as a removed book.
Note
Store titles and prices for the queue live in the shared store listing pool, not in this table.
[discovery]
storefront_candidates = true
storefront_seed_limit = 8
storefront_max_remote_calls = 32
# exclude_graphicaudio_series_sets = true # opt-in; default keeps Magento series sets
listen_sync_interval_minutes = 60 # 0 disables daemon listening sync
recommend_limit = 20
Listening sync fans out through IntegrationRegistry (every integration with
supports_listening_sync). Daemon honors listen_sync_interval_minutes; CLI
discover sync-listening and POST /api/discover/sync-listening do the same.
Shelf visibility is per-user (SQLite user_preferences, not TOML). Use
the Preferences dialog in the GUI header, or GET / PATCH /api/preferences with
{ "disabled_shelves": ["chirp_deals", "genre"] }. Empty list = all shelves on.
CLI discover recommend applies the operator prefs row.
Discover shelves stay taste-ranked (library, listening, embeddings).
Catalog search (GET /api/discover/search) is a non-personalized multi-store
browser: one page per connected storefront in parallel, merge the hits of one
recording into one row, then server-side include/exclude filters and sort. By default results are hard-filtered
to the client language (lang / language=). Chirp catalog hits carry Chirp’s
language field (English / Spanish / …) plus promotedTags as genres, so
non-matching languages are dropped; unknown/missing language metadata (e.g.
sparse GraphicAudio rows) still passes. All languages (all_languages=1)
disables the hard filter
(soft prefer remains a weak tie-break).
A row is one recording, never one title. Two stores' hits share a row when
they share an ISBN or ASIN, or when the shared listing pool resolved both
store products into the same expression (LibraryStore::listing_expressions).
Stores rarely share a record number, so hits from two different stores that
share none of those still share a row when all of these hold
(same_recording_by_runtime in bookclerk-discover):
- Both report a runtime, and the runtimes differ by at most 1 minute (stores round a recording's length differently, by less than a minute).
- They name the same work: the resolver's work key matches. It reads the
cleaned title without a trailing edition marker (
First Family - AbridgedandFirst Family (Abridged)both key asFirst Family) and the authors. - They credit the same, non-empty set of narrators.
- Nothing tells them apart: no two different ISBNs or ASINs, no abridged against unabridged, no two different languages, and no two different expressions in the pool.
Two products of one store never merge this way, and title and authors alone never merge hits. A narration and a dramatisation of one book, or an unabridged and an abridged recording, stay separate rows with their own cover, narrators and price. Each merged row keeps every store's edition, so every store's price is still offered, and a field one store left out (a runtime, a series) is filled from another store's hit. Each store listing names one publisher, and stores disagree (Audible lists First Family under Grand Central Publishing, Chirp and Libro.fm under Hachette Audio), so a merged row shows the publisher most of its listings name, ignoring case; on a tie the first one stays.
Note
David Baldacci's First Family, read by Ron McLarty, is the reference case: Audible, Chirp and Libro.fm each list an unabridged (871 minutes) and an abridged (370 minutes) recording, and search shows two rows.
Stores give ISBNs where they can: Audible's catalog isbn attribute and
Libro.fm's product ids are ISBNs, and Chirp, which lists none, names a
publisher's cover image after the edition's ISBN-13, which the Chirp plugin
reads when the file name is a valid ISBN-13. Chirp's typeahead rows, which
lead page one of a Chirp search, ask for the runtime, publisher and series
too.
Series names are cleaned when a hit arrives: a trailing edition marker and
the word "Series" are removed (King and Maxwell (abridged) reads
King and Maxwell, King & Maxwell Series reads King & Maxwell). The
series filter and the web app's series choices treat & and and, case
and accents as the same, so one series is one choice whichever store spelled
it.
| Param | Notes |
|---|---|
q |
Query (min 2 chars) |
page_size |
Default 24, clamp 1–48 (limit still accepted) |
cursor |
Opaque hex cursor (per-source page counters + query fingerprint) |
sort |
relevance (default) / popularity / rating / title / author / price / length — storefront rank (rankHint, declared by Audible today) for relevance/popularity; host re-ranks price/length/rating/title/author. Rating sort uses a Bayesian average of rating_overall with rating_count (prior mean 4.0, strength 100) so tiny sample 5.0s do not outrank high-volume 4.8s |
sort_dir |
asc / desc (defaults: desc for relevance/popularity/rating/price; asc for title/author/length). Relevance ignores direction for ranking |
field |
Optional scoped search: author / narrator / series / genre |
lang |
Default language (BCP-47); hard-include when language omitted |
language |
Comma-separated language codes to include |
all_languages |
1 / true — no hard language filter |
author, narrator, series, genre, source |
Comma-separated includes (OR within a kind, AND across kinds) |
exclude_source |
Comma-separated store ids to hide (empty = all sources, including future) |
exclude_narrator |
Comma-separated substrings; GUI Hide Virtual Voice sends Virtual Voice |
min_rating |
Keep hits with rating_overall >= min or missing rating |
min_length_minutes / max_length_minutes |
Runtime bounds; missing length still passes |
Response envelope: { items, page_size, has_more, next_cursor?, sort, sort_dir }.
Hits may include catalog price_cents when the storefront provided it (no
purchase-hints round-trip). The UI infinite-scrolls with next_cursor and
dedupes rare cross-page work_keys. Facet option lists grow from loaded pages
(not full upstream facet counts). Typeahead still uses a small first page
(page_size=10).
Per-user Discover defaults live on GET/PATCH /api/preferences:
discover_sort, discover_sort_dir, discover_language (null = browser
language), discover_excluded_sources (empty = all stores including future).
Audible Discover browse uses public GET /1.0/catalog/search (website index).
The older GET /1.0/catalog/products?keywords= filter often omits titles that
still resolve by ASIN (e.g. English A Game of Thrones B002UZZ93G).
| Surface | Commands / routes |
|---|---|
| CLI | bookclerk discover recommend (prints shelves), embed, sync-listening, wishlist … |
| Daemon | GET /api/discover/recommendations (stored feed), GET /api/discover/status, POST /api/discover/refresh, GET /api/discover/search?q= (paged multi-store catalog browse + typeahead), POST /api/discover/purchase-hints (+ /batch), GET/POST /api/wishlist, DELETE /api/wishlist/{uuid}, GET /api/request-queue, GET/PATCH /api/preferences |
| GUI | Discover — personalized shelves + catalog search (filter rail, sort + direction, language, Hide Virtual Voice); Wishlist — personal list with global queue sidebar |
A patron's scope (ADR) filters the feed and
catalog search on each answer, after the stored feed is read or the
stores have answered. The page index can't judge titles the library
doesn't hold, so each card or hit is checked by its own metadata
(WorkScope::admits_unheld): authors, narrators, series, genres and the
store's explicit flag (is_explicit, merged across stores so that one
explicit mark wins). Shelves left empty are dropped. A search page keeps
its cursor, so it can come back short while more pages follow. When the
scope can't be read, the answer is refused, never sent unfiltered.
Wishlists are store-agnostic. Every wish names the library work it is
for: adding one resolves its ASIN, ISBN, title and authors the way a scan
resolves a listing. A book the library does not hold gets a catalog work and
edition (no listing or holding), with ids derived from what it was resolved
from, so two people wishing for the same ASIN share one work. Wishes for one
work are one queue entry, and a person's second wish for a work they already
wish for returns the first. The wish's work_key is derived on read from its
identifiers, for display matching only. The global queue ranks with
overall / operator taste plus wish_count × 40. Every viewer sees the same
queue order; Discover personalizes the feed (including wishlist items).
| Shelf | Kind id | Signal |
|---|---|---|
| On the wishlist | requests |
Open shared wishlist items, first on the page and in the viewer's own taste order (tiny wisher avatars on each card) |
| Finish these series | finish_series |
Incomplete series + next index; series take turns |
| Pick up where you left off | keep_listening |
Active / multi-book listening in a series; series take turns |
| Because you wishlisted {Title} | wish (wish:dune, …) |
Titles the stores found from one of the person's own open wishes (newest three wishes) |
| More from {Author} | author |
Liked-author overlap (top authors) |
| If you like {Author} | because |
Related/similar titles not by that author |
| Narrated by {Narrator} | narrator |
Liked-narrator overlap |
| Because you like {Genre} | genre |
Category/subject overlap from finished titles |
| Similar to books you finish | similar_taste |
Embedding similarity |
| Popular in {Genre} | starter_genre (starter_genre:fantasy, …) |
Starter-genre browse while the taste is thin |
| Chirp deals right now | chirp_deals |
Chirp top + free deals |
| Picked for you on {Store} | from_store (from_audible, …) |
Titles no personal shelf took, ranked by taste, under the store where they cost the person least |
| Top picks for you | top_picks |
Fallback when every other shelf is empty |
Shelf titles stay dynamic (More from {Author}, …). Prefs filter on stable
category tags carried on each recommendation (finish_series, author,
genre, …), not on human-readable reason strings.
All kinds are offered by default. Each user hides shelves via
disabled_shelves in /api/preferences (Discover settings in the GUI). The
recommendations endpoint applies the caller's prefs server-side; the API still
returns shelf_kinds so the prefs UI knows what can be toggled.
Kind matching: author hides every author:… row; from_store hides all
from_* rows; exact ids like finish_series or from_chirp also work.
Discover shows one card per work, and every card shows one recording of it. Which cards belong together comes from the library's data model, never from comparing titles and authors:
- Candidates read from the shared listing pool carry the work and the
expression (
expressions.id: one recording or text) their listing resolved into; a live store hit carries the work the library's resolver places it in (LibraryStore::work_for), when there is one. Each wish carries its work, and the queue entry carries the edition the newest wish names and that edition's expression. - Listings of one recording (the same expression, or a shared ISBN or ASIN)
are one card, with
store_editionsand a price for each store that sells it. - Listings of one work in different recordings (a narration and a full-cast dramatisation, or two narrators) are one card too, but it keeps the owning recording's cover, text and prices; the other recording adds only its taste signals (score, reasons, shelf tags). A wish always owns its work's card and keeps the edition it names, so On the wishlist shows the wished edition. Between store cards, the higher-scoring one owns the card.
- Cards with no work and no shared identifier stay apart, even with the same title.
Live prices use public storefront endpoints (no auth). The Discover feed
loads with no_purchase_hints=true; the GUI:
- Progressively reveals shelves and cards while scrolling. A shelf adds eight cards at a time and keeps at least a screen's width of cards past the visible ones, adding the next eight once less than that is left.
- Prices a shelf at a time: once a shelf is within half a screen of the viewport, every card it has added looks up its prices and details together, and each new group of eight does the same as it is added.
- Batches those lookups into
POST /api/discover/purchase-hints/batchandPOST /api/discover/title-meta/batch(24 titles at most per request). Lookups asked within 60 ms share a batch, at most two batches of each kind are in flight at a time (lookups asked meanwhile go out together in the next one, and a group larger than one batch sends its parts at once), and a title already waiting is not asked twice. - Lets covers go first: each batch waits until the covers on screen have loaded, for at most 2 seconds, so on a small machine covers never queue behind store lookups. Only covers that are loading and on screen count; one off screen or clipped by a shelf does not.
- Keeps each finished answer per title for the tab (prices 10
minutes, details 6 hours) in memory and
sessionStorage, and puts only the titles it lacks into a batch, so a reload or a return to Discover asks nothing it already has. The batch routes arePOSTs and API JSON isno-store, so the browser's HTTP cache never holds them. Signing out clears this cache, since prices depend on the viewer's linked stores, and an answer to a lookup still running then is neither kept nor shared with the next viewer.
The daemon still searches all stores, then highlights best using the
caller’s associated accounts (portal account links, or all operator accounts)
as a membership signal: linked stores are compared at member price; other
stores at list / non-member price; the shelf shows the global cheapest of those
effective prices. Client-supplied preferred_sources are ignored.
Each store is asked about a card the strongest way it supports
(store_query_opts in bookclerk-discover):
- By the product the card already names at that store (
lookup = product). - By the card's exact ISBN, when the store declares
catalog.priceByIsbn. - By the card's exact ASIN, when the store declares
catalog.priceByAsin. - Otherwise by title and authors (
lookup = title).
Of the first-party storefronts, Audible declares ASIN lookups and Libro.fm ISBN lookups. Chirp and GraphicAudio, whose catalogs know neither, are asked by title, as are Audible and Libro.fm for a card without the identifier they take.
A title search can match another recording of the book (a full-cast
dramatisation for a narration), so a store answers it with records:
every priced listing the search found, each with its full catalog record
(see Price lookups). The host stores the
records in the shared listing pool, which resolves
each one into a recording, and keeps a record's price only when the record
carries the card's ISBN or ASIN or the pool placed it in the same
expression as the card's own store editions. Titles are never compared, and
the store's own pick is not used. Placing the records shares the batch's
deadline; when it fails or runs out, only records that carry the card's
ISBN or ASIN price it, and the answer is marked incomplete and not
cached, so the card asks again. Placement keeps running after the
deadline (at most two batches' placements at once), so the card's next
ask finds the records in the pool. The batch logs one line per store
counting the records it returned, kept and dropped, and why each dropped
record was dropped (card not pooled, record not pooled, another
recording). An exact ISBN or ASIN answer is the card's edition
and is kept as is. A store that answers a title search with one bare hint
and no records (a guest from before records) is kept only when the hint's
title and authors match the card's, as before. The single-title route
prices its card through the same batch path.
A bare hint can still name another recording than the card's. Before
answering, the purchase-hint routes and the wishlist's own price
lookup drop every price whose product the
shared listing pool places only in another
expression (recording) than the card's own store editions
(retain_card_recording in bookclerk-discover, reading
LibraryStore::listing_expressions). A card that names no store edition
but an ASIN is checked as that Audible product. A price for a product the
pool never listed stays, and so does every price when the pool knows none
of the card's editions.
The batch route checks all of its cards in one pool read with its own
1.5 second budget after the batch answers, so a batch that used its whole
10 seconds is still checked; the wishlist's check shares its enrichment
budget. When the pool read fails or runs out of time,
the prices stay unchecked: a Discover answer is marked incomplete, so the
page shows it but asks again later, and a new wish is saved with the
resolved prices.
A page of cards is priced with one call per store: the batch route
sends every visible card's query to each deployed store's purchaseHints
entrypoint together, each asked the strongest way the store supports (see
above), so Audible prices up to fifty titles in one catalog request and
Chirp prices a page in one aliased GraphQL query; Libro.fm and GraphicAudio
have no bulk endpoint, so the SDK's default purchaseHints looks their
titles up four at a time inside the guest. Audible and Chirp resolve
the titles whose product id they already know first, then look the rest up
by title for at most five seconds, so a slow search never costs a known
title its price; Audible prices each query in its own marketplace. The batch answers within
10 seconds with what has resolved: a store still working then counts as
timed out. A query no store answered with a hint carries only its trusted
store links, unpriced, with incomplete: true: a store's None is no
proof it lacks the title, since a guest out of its own time budget answers
None for titles it never tried. Such an answer is not cached, the page
keeps no copy, and the card asks again after 15 seconds (twice at most)
without dropping prices it already shows, so one slow store never turns a
page of cards into a failed request or a card left unpriced. Queries a
store answered with a hint are cached for 10 minutes.
Cards show a price before any lookup: a candidate read from the pool
carries its listing's non_member offer when it was quoted in the last day
(by anyone's fetch), and at the end of each store's fetch the
discover_candidates job prices up to 120 distinct titles of the rest in
one purchase_hints call, so a refresh after a wish asks the store only
about titles it has not priced. The answers are stored as the listings'
offers: the list price as non_member, a member price as member. The
card shows the non-member price as the store's list price and the live
lookup refreshes it on screen when the card scrolls into view (a live
lookup does not write the pool). Member prices are
applied only for stores the viewer has an account with, decided server-side
as before. A store paused for rate limiting is not asked, a 429 here pauses
its live lookups too, and a failure or timeout leaves the rest unpriced until
a card asks.
Stores are paused in two ways. A store that answers 429 returns
rate_limited with its Retry-After, and its price lookups are left alone
for that long (at most fifteen minutes), or for a minute that doubles on
each repeat when it did not say; a store that answers again closes the
pause. A store that times out three lookups in a row is paused for five
minutes (the breaker logs price lookups paused after repeated timeouts),
then tried again. The single-title route honors the same pauses.
- Chirp personalized endpoints that require a logged-in session
(
currentUserRelatedAudiobooks, …) - Multi-region storefront merchandising (US only for now)
- Bulk Open Library harvest (use dumps)
- Paid WorldCat / Goodreads / Hardcover providers
- A long-running vector DB sidecar
- Audible Spatial/Atmos (L1) merchandising
Source: docs/internals/discovery.md at main · View as Markdown