ADR: Library data model (works, expressions, editions, holdings)
- Status: Accepted; steps 1 to 4 and step 6 implemented (step 6 dropped the flat
bookstable on 2026-11-21), and step 5's holdings and branch titles (of file quality, moved to step 5, the probe of each stored file is recorded and the comparison is still to come); the naming discriminator replaced by a required<call number>tag (see Status of the work). Amended 2026-11-13: the kind enums were reshaped for mixed media and every kind became required (decisions 1 and 5). A later record, Metadata claims and file sync, decides to replace the work-level matcher of decision 4 with weighed evidence and to add acatalog_idslot; that work has not started - Date: 2026-11-04
- Related: Schema versioning, SQL database contract, Plugin capabilities v3, Untrusted-input fuzzing, Control plane
The library stores one books row per store product per account, keyed
(plugin_key, account_id, product_id). That row mixes four things:
- bibliographic metadata (title, authors, narrators, series, publisher);
- the store listing (marketplace, categories, store ratings);
- the listener's state (
is_finished,tags); - acquisition (
acquire_status,storage_key,pdf_storage_key).
Authors and narrators are comma-separated text and content_kind is a free
string. A works table and a work_editions join exist, but they are
rebuilt from ASIN or ISBN alone and works still carries narrators.
This shape causes four problems:
- Mixed media. Audible and OpenLibrary return podcasts, print and
ebook editions.
ScanBook.contentKindis optional text andCatalogHithas no kind at all, so everything looks like an audiobook. - Record clashes in a library. Two accounts that own the same edition
are two rows that acquire to the same naming-template path. Branch titles
live in
branch_titles, keyed(provenance, store, product_id), and only match a library row with the same store and product id, so an Audible copy and a Libro.fm copy of one edition never meet.title_filesthen refuses the upload (#300, #314, #321). - Future media. Transcripts, generated EPUBs and read-along (#121, #125) need files with roles, not two storage-key columns.
- Many under one book. A work can have several narrations, translations, abridgements, publishers and store releases. None of that has a place.
The host schema is still the unreleased development pack
(schema versioning), and no plugin has been built
against a released plugin.capnp. Both get more expensive to change after the
first release, when the schema freezes and the ABI becomes append-only, so the
model changes now.
The model follows the IFLA Functional Requirements for Bibliographic Records (FRBR) entity model, and its successor the IFLA Library Reference Model, which keeps the same four Group 1 entities. Using the established library model, rather than one of our own, keeps Bookclerk's vocabulary recognisable to librarians and cataloguers, lines its data up with sources that already speak FRBR (OpenLibrary's works and editions), and leaves room for features that model already describes.
| FRBR entity | Bookclerk | Notes |
|---|---|---|
| Work | works |
|
| Expression | expressions |
|
| Manifestation | editions |
"Edition" is the word people use for a manifestation; the docs and UI say edition |
| Item | holdings and edition_files |
A holding is an exemplar someone owns; its files are the copy at a destination |
| Person, corporate body (Group 2) | contributors |
|
| Responsibility relationships | credits with a role |
Created by (work), realised by (expression), produced by (manifestation) |
| Level | One row is | Carries |
|---|---|---|
| Work | A creative work, independent of any store or format | Title, kind, original language, first published, parent_work_id, author credits, series entries |
| Expression | One realisation of a work: the text, a narration, a translation, a dramatisation, an abridgement | Work, kind, language, abridgement, narrator, translator and cast credits |
| Edition | One release of an expression | Expression, media_kind, publisher credits, published date, duration or page count, cover, identifiers |
Each level answers one question, and no value repeats across levels:
| Enum | Question | Values |
|---|---|---|
works.kind |
What family of creative work is it? | book, podcast |
expressions.kind |
How was the work realised? | text, narration, dramatisation, graphic, original_audio |
editions.media_kind |
What carrier do you get? | audio, ebook, print |
Podcasts use the parent link: the show is a podcast work with no parent,
and each episode is a podcast work whose parent_work_id is the show,
with its own expression and edition. Whether a podcast is a show or an
episode is that link, not a separate kind. An episode whose show is not
resolved yet is stored with an empty match key, so a show can never match
it by text; it gains its key when it is linked to its show. Naming's
podcast and podcastparent conditions read the parent link.
dramatisation covers full-cast productions, whether they adapt a book
(GraphicAudio) or are original audio dramas: both are book works.
graphic is sequential art (comics, graphic novels, manga), so a digital
comic is book + graphic + ebook. original_audio is audio that is
itself the first realisation rather than a reading of a text: podcast
episodes, lectures, single-voice originals. A translation is not a kind:
the expression's language against the work's original language, plus
translator credits, say it. Abridgement (unknown, unabridged,
abridged) lives on the expression, because an abridged text is a
different realisation of the work.
ebook covers any downloadable document (EPUB, PDF, CBZ). Print and ebook
editions are stored, not dropped: they show in discover and their ISBNs
help tie audio editions to a work. Whether an edition is acquired depends
on a storefront that declares it can fetch that carrier (see
Plugin ABI: declaring media) and on
what the host supports; today the host acquires audio only.
Each enum is a Rust enum matched exhaustively, a capnp enum on the wire, and a
CHECK constraint in SQL. A new variant is appended in all three at once.
| Table | One row is | Unique key |
|---|---|---|
work_identifiers |
One external id of a work (an OpenLibrary work id, for example), shared by all its expressions and editions | (scheme, value) |
edition_identifiers |
One external id of an edition: scheme, value, marketplace (empty for global schemes) and strength (see below) |
(scheme, value, marketplace) for strong schemes only |
contributors |
A person or organisation, with a sort name | Surrogate id |
credits |
A contributor's role on a work, expression or edition, with a position |
(target_kind, target_id, contributor_id, role) |
series, series_entries |
A series and a work's place in it (position text plus a sortable decimal) | (series_id, work_id) |
store_listings |
What a store sells: plugin_key, marketplace, product_id, the edition, the store's ratings and its own text for the listing as parsed columns (synopsis, cover, credits as printed, series and the store's series or show product id, URL), observed_at |
(plugin_key, marketplace, product_id) |
listing_offers |
What a listing costs one audience (non_member or member): price, label, currency, observed_at |
(listing_id, audience) |
holdings |
Ownership of a listing by an account (plugin_key, account_id), by a branch (node_id, node:<node_id> provenance), or by a patron who uploaded the book (user_id, patron:<user_id>; see Uploads and ingest batches), as typed holder columns with cascading foreign keys, with entitlement (purchase, plus, loan, free, or own for an upload) and purchased_at |
Surrogate id; unique (listing_id, holder) |
edition_files |
A file of an edition at a destination, with role (audio, part, chapter, pdf, cover, sidecar, transcript, epub), digest, quality, the library copy it belongs to (copy_format, copy_quality) and writer |
(destination, storage_key) |
edition_copies |
An edition's copy at one destination: state (stored, pending, failed, missing), the primary audio storage_key there, failed attempts, next_attempt_at and the last error (storage destinations, decision 3) |
(edition_id, destination) |
destination_objects |
The last state seen of one file at a destination: size, change signature, modified_at_ms, the listing that last saw it (listed_seq), the change sequence when it last changed (changed_seq) and removed (a tombstone kept 30 days) (storage destinations, decision 5) |
(destination, object_key) |
destination_feeds |
A destination's change sequence, feed state (off, healthy, gap, failing) and cursor, listing in progress and schedule, and file count |
destination |
edition_acquisitions |
An edition's acquire state (status, primary storage_key, last error) and the account holding whose title's write set it (holding_id), plus its companion PDF's state (pdf_status, pdf_storage_key, pdf_holding_id); only the holding that acquired the audio or the PDF changes it |
edition_id |
user_title_state |
One user's finished flag, ratings and tags for a work (listening progress stays per provider in listening_progress, keyed by work) |
(user_id, work_id) |
resolution_overrides |
An Administrator's manual merge or split, which the resolver obeys | (subject_kind, subject_id) |
metadata_claims |
One source's value for one field of an edition (chapters only so far), naming a long value by digest, with the runtime it was measured on, a per-edition revision and an Administrator's pin (edition chapters) |
(subject_kind, subject_id, field, source_kind, source_id, locale) |
chapter_lists, chapter_list_entries |
A chapter list stored once by its SHA-256 digest: a head row written last, and one row per chapter (position, start_ms, depth, title) |
digest; (digest, position) |
edition_chapters |
The chapter list an edition plays, as the resolver chose it from the claims, with its source and runtime | edition_id |
Identifiers have a strength, fixed per scheme:
- Strong schemes name exactly one edition: a store's own product id in its marketplace, an Audible ASIN in its marketplace, an OpenLibrary edition id. They are unique.
- Weak schemes are shared in practice and only suggest a match: ISBN-10
and ISBN-13 (Audible's abridged and unabridged products can carry one ISBN,
as
bookclerk-enrich'sISBN_MATCH_GAP_CLOSEnotes). They are indexed but not unique, and one weak value can sit on several editions.
Credit roles are a closed enum: author, narrator, translator, editor,
illustrator, publisher, host, cast, adaptor.
These tables replace books, works, work_editions, title_files and
branch_titles. listening_progress, title_requests, ignored_titles and
embeddings point at works or editions instead of book_uuid.
Acquisition is per edition, not per holding: two holdings of one edition
acquire once. edition_files decides what an acquire or a host upload may
write at a key that is already recorded:
| Existing key holds | Incoming file | Outcome |
|---|---|---|
| Same edition, same role | Better quality (higher bitrate or a lossless source) | Replace |
| Same edition, same role | Same or worse quality | Keep; the upload is refused |
| Same edition, another role | A sidecar, PDF or transcript | Add at its own key |
| Another edition | Anything | Never written there: naming templates must write the edition's <call number>, so its path is its own |
This is the rule #300 asks for. Keys a title will write are derived from its edition and the download options before an upload is granted.
Storefronts and metadata providers describe what they see; the host decides identity. A sans-IO resolver maps each incoming listing to an edition, an expression and a work:
- Edition by a shared strong identifier. A weak identifier (an ISBN) proposes a candidate edition, which is accepted only when its expression is compatible (same kind, language, abridgement and narrator credits) and its media kind matches. Two strong identifiers that point at different editions are never merged automatically; the conflict is recorded for an Administrator.
- Expression within the work: same kind, language, abridgement and narrator or translator credits. Any of these that the source left unspecified never matches a specified value, so missing data makes a new expression rather than a wrong merge.
- Work by a shared
work_identifiersrow, else by a work-level matcher, else a new work. The work-level matcher compares normalised title and author credits only, reusingbookclerk-enrich's text normalisation but notcalculate_match_confidence: that score includes recording signals (duration, narrator) and treats any ten-character alphanumeric title as an ASIN. A matcher hit joins the work when the kind is the same and, for a podcast episode, the parent show is the same, so two shows by one host never share an episode called "Trailer". A book or a show only matches works without a parent, so a show never joins one of its episodes. In the rare case that two existing works already match (an Administrator split them), the listing joins neither and the choice goes to the Administrator. A wrong automatic merge is undone by an override that splits the work. resolution_overrideswin over every automatic step.
Resolution runs on guest-supplied identifiers and credits, so it is an
untrusted-input boundary. It ships with a cargo-fuzz target, a seed corpus
and a replay test, with an oracle that checks: one strong identifier never
maps to two editions, a weak identifier alone never merges two editions with
incompatible expressions or media kinds, editions with conflicting strong
identifiers are never merged, works of different kinds are never merged,
a book or show never matches an episode by text, no work becomes its own
parent, and resolving the same input twice changes
nothing
(fuzzing ADR).
Each link is written as one atomic batch, never as a read-modify-write across statements, because D1 has no interactive transactions (#320). All DDL and queries go through SQL-v1.
ScanBook and CatalogHit gain:
workKind :WorkKind,expressionKind :ExpressionKindandmediaKind :MediaKind;identifiers :List(Identifier)(scheme, value, marketplace). TheIdentifierSchemeenum fixes whether an id names the work or the edition, and so which identifier table it lands in, so the record has no separate level that could contradict it;credits :List(Credit)(name, role, level) in billing order, where the list position is the credit's position andleveldefaults to the role's usual level;seriesEntries :List(SeriesEntry)(name, position, the store's own series id);parentProductIdfor podcast episodes;- on
ScanBookonly,languageandabridgement :Abridgement, whichCatalogHitalready has, pluspublishedAt,purchasedAtandcategories, which the scan path used to drop.
Cap'n Proto reads an absent enum field as ordinal 0, so ordinal 0 is always the value a guest gets when it says nothing. Two rules decide what ordinal 0 means:
- For the three kinds, ordinal 0 is
unspecified, and the host refuses a record that leaves any kind on it. Every record names all three, so a guest that forgets a field fails loudly instead of describing an audiobook by accident. The same holds for the media declaration indescribe()andplugin.toml, which has no default either. - Elsewhere ordinal 0 is the common case:
isbnforIdentifierScheme,authorfor the credit role androleDefaultfor the credit level.unknown @0is used only where a source can honestly not know the answer, and the resolver treats unknown differently from any known value. Today that isAbridgement.
After the v1 freeze the same rules decide any enum field added later:
either ordinal 0 is the right answer for a guest built before the field
existed, or it is unspecified and the field is required from the start.
The flat text fields they replace (authors, narrators, asin, isbn,
contentKind, series, seriesIndex) are removed from both records, and
the host has no fallback to them. Nothing has been released against them,
so this is the one cheap moment to remove them; once v1 freezes, removed
fields would stay on the wire for good. The pull request that changes the
records also moves every first-party storefront to the structured fields
and regenerates the TS and Python SDKs. plugin.capnp is append-only from
the v1 freeze.
A new metadata entrypoint returns the same edition records for a lookup by
identifier or by title and author. OpenLibrary and Audnexus, which live in
bookclerk-discover and bookclerk-enrich today, move into plugins behind
it.
The resolver publishes these on the core.events outbox:
edition.linkedwhen a listing joins an edition, new or existing;edition.updatedwhen a rescan or a metadata lookup changes an existing edition's or its work's metadata (title, cover, credits, series, media kind) without linking anything new;holding.addedandholding.removed;edition_files.written.
Integrations (Audiobookshelf), embeddings, discover and the search index
subscribe to them, so a metadata-only change refreshes them as a new link
does. The batch rebuild_works_from_library() pass is removed.
Bookclerk has no release, so no library holds anything that has to be
carried across. The schema step changes the unreleased checksum like any
other schema change. A development library then fails closed as
schema versioning requires, and is recreated with
cargo reset --yes. No in-place migration, state export or partial rebuild
is written.
The Libation importer in bookclerk-migrate writes the new tables directly.
/api/library/books and /api/library/books/{uuid} are removed, with no
compatibility view. They are replaced by /api/library/works,
/api/library/editions and /api/library/holdings, with typed OpenAPI
schemas and examples, and media-kind filters. Errors name the entity that was
not found (#559).
- Two stores or two accounts selling one edition meet in one place, so the library can deduplicate acquires and branch deliveries by edition.
- Mixed media is typed end to end, and print and ebook editions can show in discover without ever being acquired.
- Per-user state means members sharing a library no longer overwrite each other's finished flags, ratings or tags.
- Identity depends on identifier quality. A store that reuses or omits identifiers produces new editions rather than wrong merges; Administrators fix the rest with overrides.
- Three levels add a join to most title queries. The list views read a denormalised page index refreshed by the events above, not the full join.
- Every development library is reset once. Backups taken before the change can't be restored afterwards.
- Every storefront plugin changes in the same pull request as the records. A third-party plugin built against the development schema has to be rebuilt, which is acceptable before the first release.
Planned as separate pull requests, in this order:
- This ADR.
- ABI: the new capnp fields and enums replace the flat ones, both SDKs,
and the four first-party storefronts filling them. Until step 3 the host
maps the structured fields onto the current
booksrow. - Schema and resolver, with its fuzz target.
- Acquire and files: per-edition acquisition,
edition_files, and the required<call number>naming tag. - Library: branch titles become holdings; node frames carry identifiers and media kind.
- Web API and UI.
- The
metadataentrypoint, with OpenLibrary and Audnexus moved into plugins.
The dotted names in decision 6 name the events; the published event types
are spelled in snake case, like book_acquired. Step 3 publishes
edition_linked, edition_updated and holding_added, so subscribe to
those spellings. It derives a listing from every stored books row, so the
new tables fill as titles are scanned or imported. Step 5 publishes
holding_removed when a scan, a revoke or a remap removes a holding, and
makes a title a branch shares a node holding (holder node:<node id>),
resolved into the library like a scanned listing: the catalog.sync
frame carries the listing's marketplace beside the ASIN, ISBN and content
kind, the holding carries the share and delivery state, delivered files
are the edition's (an edition already in the outputs is not delivered
again), and a branch that stops sharing, withdraws a title or leaves
loses its holdings with holding_removed;
edition_files_written comes with the step that writes files. The old tables are dropped by
the step that moves their last reader off them: title_files with step 4,
branch_titles with step 5 and books with step 6.
Step 4 records every file in edition_files with its edition and role,
and acquires an edition once for all its account rows from one store.
Instead of a discriminator appended on a clash, naming templates must
write the edition's short <call number> (in the folder template, or in
both file templates), so two editions never plan one path; the built-in
profiles carry it, saving templates without it is refused, and an acquire
whose name another edition still holds is refused rather than overwrite
it. The store product id does not qualify, since one edition sold by two
stores has two, and podcast feeds, some storefronts and imported files
have no ISBN or ASIN. An edition's id is a version 5 UUID over the store
listing that first named it (plugin key, marketplace, product id), so
libraries that see the same listing first agree on its id and its
<call number>, which is the first 40 bits of that UUID in Crockford
base32. If that id is already taken (the listing left the library while
its first edition stayed, holding other listings), the new edition gets a
random id instead.
Step 5 takes the quality comparison, because node frames do not carry a
file's quality yet: storefronts may declare a file's format and quality,
and the media worker probes each file and records it. The probe is in
place: each audio file's row holds its media type, bitrate, whether it is
lossless and the store it came from, written by the acquire that stored
it or carried by the upload that delivered it. The comparison follows
one rule for where a copy comes from, and an install's own acquire
applies it (a library does not yet replace its own copy with a better
one a host uploads). When a library already
holds a copy of the edition from the same store, another install of that
library takes the library's copy whatever its format, and the media
worker converts it to the format that install prefers. When the
library's copy came from a different store, the install downloads its
own from its store, probes it, and keeps whichever of the two is better.
The same edition from another
store is then acquired anyway, the better audio kept and the sidecars
merged, and both copies marked acquired. edition_files_written comes
with step 5 too.
Step 6 keeps each edition's acquire state in edition_acquisitions,
written in the same batch as the title's own status. A title of an edition
another title already acquired takes that copy instead of fetching it again,
and a sibling title that is queued, downloading or failed never hides files
the edition already has. Only the title that acquired the edition, or any
title once that one has left the library, can change its state. An
acquire job picks its titles from account holdings and their editions'
state, not from books, so a title whose edition another account's title
already acquired is not visited again. book_acquired
names the edition (editionId), and a wishlist item records the edition
that fulfilled it (resolved_edition_id). Metadata gap-fill and the
Audible catalog match walk held editions, not books: what they find
fills the work, expression and edition, a matched ASIN becomes an edition
identifier, and the edition records who enriched it (enrich_source).
Step 6 also adds the page index the Consequences call for:
library_work_rows, one row per held top-level work, refreshed by a
library-wide host:library-index event consumer and rebuilt by a
library_index_rebuild job when the index has no finished build of the
current layout. GET /api/library/works pages it by keyset cursor, and
GET /api/library/works/{id} reads one work with its editions, listings,
holdings and the caller's own state. With q, the list route searches a
per-node works index kept beside the title search index, fed by the same
events and rebuilt by the same job. GET /api/library/editions/{id}/cover
serves the cover an acquire wrote beside an edition's file, and
POST /api/library/acquire takes editions, each queued as one of its
titles until books is retired. The web app lists works, and
/api/library/books (with /{uuid} and /{uuid}/cover) is removed.
PUT /api/library/works/{id}/state edits the caller's own state, and an
Administrator pins a listing to an edition with
PUT /api/library/listings/{id}/pin (DELETE removes the pin) and reads
the recorded conflicts from GET /api/library/conflicts. A pin moves the
listing in the batch that records it, and its edition_linked dedup key
carries the pin's revision, so moving a listing back publishes again.
POST /api/library/works/{id}/split moves some of a work's expressions to
a new work (copying its title, credits, series and each person's state)
and isolates the new work, so the matcher keeps joining the original.
User-facing pages (the glossary, Where books go, Branches, the plugin ABI reference) change with the step that changes their behaviour.
Source: docs/adr/library-data-model.md at main · View as Markdown