ADR: Library data model (works, expressions, editions, holdings)

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

  • Status: Accepted; steps 1 to 4 and step 6 implemented (step 6 dropped the flat books table on 2026-11-21), and step 5's holdings and branch titles (of file quality, moved to step 5, the probe of each stored file is recorded and the comparison is still to come); the naming discriminator replaced by a required <call number> tag (see Status of the work). Amended 2026-11-13: the kind enums were reshaped for mixed media and every kind became required (decisions 1 and 5). A later record, Metadata claims and file sync, decides to replace the work-level matcher of decision 4 with weighed evidence and to add a catalog_id slot; that work has not started
  • Date: 2026-11-04
  • Related: Schema versioning, SQL database contract, Plugin capabilities v3, Untrusted-input fuzzing, Control plane

Context

The library stores one books row per store product per account, keyed (plugin_key, account_id, product_id). That row mixes four things:

  • bibliographic metadata (title, authors, narrators, series, publisher);
  • the store listing (marketplace, categories, store ratings);
  • the listener's state (is_finished, tags);
  • acquisition (acquire_status, storage_key, pdf_storage_key).

Authors and narrators are comma-separated text and content_kind is a free string. A works table and a work_editions join exist, but they are rebuilt from ASIN or ISBN alone and works still carries narrators.

This shape causes four problems:

  1. Mixed media. Audible and OpenLibrary return podcasts, print and ebook editions. ScanBook.contentKind is optional text and CatalogHit has no kind at all, so everything looks like an audiobook.
  2. Record clashes in a library. Two accounts that own the same edition are two rows that acquire to the same naming-template path. Branch titles live in branch_titles, keyed (provenance, store, product_id), and only match a library row with the same store and product id, so an Audible copy and a Libro.fm copy of one edition never meet. title_files then refuses the upload (#300, #314, #321).
  3. Future media. Transcripts, generated EPUBs and read-along (#121, #125) need files with roles, not two storage-key columns.
  4. Many under one book. A work can have several narrations, translations, abridgements, publishers and store releases. None of that has a place.

The host schema is still the unreleased development pack (schema versioning), and no plugin has been built against a released plugin.capnp. Both get more expensive to change after the first release, when the schema freezes and the ABI becomes append-only, so the model changes now.

Decision

1. Three bibliographic levels, following FRBR

The model follows the IFLA Functional Requirements for Bibliographic Records (FRBR) entity model, and its successor the IFLA Library Reference Model, which keeps the same four Group 1 entities. Using the established library model, rather than one of our own, keeps Bookclerk's vocabulary recognisable to librarians and cataloguers, lines its data up with sources that already speak FRBR (OpenLibrary's works and editions), and leaves room for features that model already describes.

FRBR entity Bookclerk Notes
Work works
Expression expressions
Manifestation editions "Edition" is the word people use for a manifestation; the docs and UI say edition
Item holdings and edition_files A holding is an exemplar someone owns; its files are the copy at a destination
Person, corporate body (Group 2) contributors
Responsibility relationships credits with a role Created by (work), realised by (expression), produced by (manifestation)
Level One row is Carries
Work A creative work, independent of any store or format Title, kind, original language, first published, parent_work_id, author credits, series entries
Expression One realisation of a work: the text, a narration, a translation, a dramatisation, an abridgement Work, kind, language, abridgement, narrator, translator and cast credits
Edition One release of an expression Expression, media_kind, publisher credits, published date, duration or page count, cover, identifiers

Each level answers one question, and no value repeats across levels:

Enum Question Values
works.kind What family of creative work is it? book, podcast
expressions.kind How was the work realised? text, narration, dramatisation, graphic, original_audio
editions.media_kind What carrier do you get? audio, ebook, print

Podcasts use the parent link: the show is a podcast work with no parent, and each episode is a podcast work whose parent_work_id is the show, with its own expression and edition. Whether a podcast is a show or an episode is that link, not a separate kind. An episode whose show is not resolved yet is stored with an empty match key, so a show can never match it by text; it gains its key when it is linked to its show. Naming's podcast and podcastparent conditions read the parent link.

dramatisation covers full-cast productions, whether they adapt a book (GraphicAudio) or are original audio dramas: both are book works. graphic is sequential art (comics, graphic novels, manga), so a digital comic is book + graphic + ebook. original_audio is audio that is itself the first realisation rather than a reading of a text: podcast episodes, lectures, single-voice originals. A translation is not a kind: the expression's language against the work's original language, plus translator credits, say it. Abridgement (unknown, unabridged, abridged) lives on the expression, because an abridged text is a different realisation of the work.

ebook covers any downloadable document (EPUB, PDF, CBZ). Print and ebook editions are stored, not dropped: they show in discover and their ISBNs help tie audio editions to a work. Whether an edition is acquired depends on a storefront that declares it can fetch that carrier (see Plugin ABI: declaring media) and on what the host supports; today the host acquires audio only.

Each enum is a Rust enum matched exhaustively, a capnp enum on the wire, and a CHECK constraint in SQL. A new variant is appended in all three at once.

2. Supporting tables

Table One row is Unique key
work_identifiers One external id of a work (an OpenLibrary work id, for example), shared by all its expressions and editions (scheme, value)
edition_identifiers One external id of an edition: scheme, value, marketplace (empty for global schemes) and strength (see below) (scheme, value, marketplace) for strong schemes only
contributors A person or organisation, with a sort name Surrogate id
credits A contributor's role on a work, expression or edition, with a position (target_kind, target_id, contributor_id, role)
series, series_entries A series and a work's place in it (position text plus a sortable decimal) (series_id, work_id)
store_listings What a store sells: plugin_key, marketplace, product_id, the edition, the store's ratings and its own text for the listing as parsed columns (synopsis, cover, credits as printed, series and the store's series or show product id, URL), observed_at (plugin_key, marketplace, product_id)
listing_offers What a listing costs one audience (non_member or member): price, label, currency, observed_at (listing_id, audience)
holdings Ownership of a listing by an account (plugin_key, account_id), by a branch (node_id, node:<node_id> provenance), or by a patron who uploaded the book (user_id, patron:<user_id>; see Uploads and ingest batches), as typed holder columns with cascading foreign keys, with entitlement (purchase, plus, loan, free, or own for an upload) and purchased_at Surrogate id; unique (listing_id, holder)
edition_files A file of an edition at a destination, with role (audio, part, chapter, pdf, cover, sidecar, transcript, epub), digest, quality, the library copy it belongs to (copy_format, copy_quality) and writer (destination, storage_key)
edition_copies An edition's copy at one destination: state (stored, pending, failed, missing), the primary audio storage_key there, failed attempts, next_attempt_at and the last error (storage destinations, decision 3) (edition_id, destination)
destination_objects The last state seen of one file at a destination: size, change signature, modified_at_ms, the listing that last saw it (listed_seq), the change sequence when it last changed (changed_seq) and removed (a tombstone kept 30 days) (storage destinations, decision 5) (destination, object_key)
destination_feeds A destination's change sequence, feed state (off, healthy, gap, failing) and cursor, listing in progress and schedule, and file count destination
edition_acquisitions An edition's acquire state (status, primary storage_key, last error) and the account holding whose title's write set it (holding_id), plus its companion PDF's state (pdf_status, pdf_storage_key, pdf_holding_id); only the holding that acquired the audio or the PDF changes it edition_id
user_title_state One user's finished flag, ratings and tags for a work (listening progress stays per provider in listening_progress, keyed by work) (user_id, work_id)
resolution_overrides An Administrator's manual merge or split, which the resolver obeys (subject_kind, subject_id)
metadata_claims One source's value for one field of an edition (chapters only so far), naming a long value by digest, with the runtime it was measured on, a per-edition revision and an Administrator's pin (edition chapters) (subject_kind, subject_id, field, source_kind, source_id, locale)
chapter_lists, chapter_list_entries A chapter list stored once by its SHA-256 digest: a head row written last, and one row per chapter (position, start_ms, depth, title) digest; (digest, position)
edition_chapters The chapter list an edition plays, as the resolver chose it from the claims, with its source and runtime edition_id

Identifiers have a strength, fixed per scheme:

  • Strong schemes name exactly one edition: a store's own product id in its marketplace, an Audible ASIN in its marketplace, an OpenLibrary edition id. They are unique.
  • Weak schemes are shared in practice and only suggest a match: ISBN-10 and ISBN-13 (Audible's abridged and unabridged products can carry one ISBN, as bookclerk-enrich's ISBN_MATCH_GAP_CLOSE notes). They are indexed but not unique, and one weak value can sit on several editions.

Credit roles are a closed enum: author, narrator, translator, editor, illustrator, publisher, host, cast, adaptor.

These tables replace books, works, work_editions, title_files and branch_titles. listening_progress, title_requests, ignored_titles and embeddings point at works or editions instead of book_uuid.

3. Files belong to editions

Acquisition is per edition, not per holding: two holdings of one edition acquire once. edition_files decides what an acquire or a host upload may write at a key that is already recorded:

Existing key holds Incoming file Outcome
Same edition, same role Better quality (higher bitrate or a lossless source) Replace
Same edition, same role Same or worse quality Keep; the upload is refused
Same edition, another role A sidecar, PDF or transcript Add at its own key
Another edition Anything Never written there: naming templates must write the edition's <call number>, so its path is its own

This is the rule #300 asks for. Keys a title will write are derived from its edition and the download options before an upload is granted.

4. One resolver, in the host, fed by plugins

Storefronts and metadata providers describe what they see; the host decides identity. A sans-IO resolver maps each incoming listing to an edition, an expression and a work:

  1. Edition by a shared strong identifier. A weak identifier (an ISBN) proposes a candidate edition, which is accepted only when its expression is compatible (same kind, language, abridgement and narrator credits) and its media kind matches. Two strong identifiers that point at different editions are never merged automatically; the conflict is recorded for an Administrator.
  2. Expression within the work: same kind, language, abridgement and narrator or translator credits. Any of these that the source left unspecified never matches a specified value, so missing data makes a new expression rather than a wrong merge.
  3. Work by a shared work_identifiers row, else by a work-level matcher, else a new work. The work-level matcher compares normalised title and author credits only, reusing bookclerk-enrich's text normalisation but not calculate_match_confidence: that score includes recording signals (duration, narrator) and treats any ten-character alphanumeric title as an ASIN. A matcher hit joins the work when the kind is the same and, for a podcast episode, the parent show is the same, so two shows by one host never share an episode called "Trailer". A book or a show only matches works without a parent, so a show never joins one of its episodes. In the rare case that two existing works already match (an Administrator split them), the listing joins neither and the choice goes to the Administrator. A wrong automatic merge is undone by an override that splits the work.
  4. resolution_overrides win over every automatic step.

Resolution runs on guest-supplied identifiers and credits, so it is an untrusted-input boundary. It ships with a cargo-fuzz target, a seed corpus and a replay test, with an oracle that checks: one strong identifier never maps to two editions, a weak identifier alone never merges two editions with incompatible expressions or media kinds, editions with conflicting strong identifiers are never merged, works of different kinds are never merged, a book or show never matches an episode by text, no work becomes its own parent, and resolving the same input twice changes nothing (fuzzing ADR).

Each link is written as one atomic batch, never as a read-modify-write across statements, because D1 has no interactive transactions (#320). All DDL and queries go through SQL-v1.

5. The plugin ABI carries structured records

ScanBook and CatalogHit gain:

  • workKind :WorkKind, expressionKind :ExpressionKind and mediaKind :MediaKind;
  • identifiers :List(Identifier) (scheme, value, marketplace). The IdentifierScheme enum fixes whether an id names the work or the edition, and so which identifier table it lands in, so the record has no separate level that could contradict it;
  • credits :List(Credit) (name, role, level) in billing order, where the list position is the credit's position and level defaults to the role's usual level;
  • seriesEntries :List(SeriesEntry) (name, position, the store's own series id);
  • parentProductId for podcast episodes;
  • on ScanBook only, language and abridgement :Abridgement, which CatalogHit already has, plus publishedAt, purchasedAt and categories, which the scan path used to drop.

Cap'n Proto reads an absent enum field as ordinal 0, so ordinal 0 is always the value a guest gets when it says nothing. Two rules decide what ordinal 0 means:

  • For the three kinds, ordinal 0 is unspecified, and the host refuses a record that leaves any kind on it. Every record names all three, so a guest that forgets a field fails loudly instead of describing an audiobook by accident. The same holds for the media declaration in describe() and plugin.toml, which has no default either.
  • Elsewhere ordinal 0 is the common case: isbn for IdentifierScheme, author for the credit role and roleDefault for the credit level. unknown @0 is used only where a source can honestly not know the answer, and the resolver treats unknown differently from any known value. Today that is Abridgement.

After the v1 freeze the same rules decide any enum field added later: either ordinal 0 is the right answer for a guest built before the field existed, or it is unspecified and the field is required from the start.

The flat text fields they replace (authors, narrators, asin, isbn, contentKind, series, seriesIndex) are removed from both records, and the host has no fallback to them. Nothing has been released against them, so this is the one cheap moment to remove them; once v1 freezes, removed fields would stay on the wire for good. The pull request that changes the records also moves every first-party storefront to the structured fields and regenerates the TS and Python SDKs. plugin.capnp is append-only from the v1 freeze.

A new metadata entrypoint returns the same edition records for a lookup by identifier or by title and author. OpenLibrary and Audnexus, which live in bookclerk-discover and bookclerk-enrich today, move into plugins behind it.

6. Events instead of rebuild passes

The resolver publishes these on the core.events outbox:

  • edition.linked when a listing joins an edition, new or existing;
  • edition.updated when a rescan or a metadata lookup changes an existing edition's or its work's metadata (title, cover, credits, series, media kind) without linking anything new;
  • holding.added and holding.removed;
  • edition_files.written.

Integrations (Audiobookshelf), embeddings, discover and the search index subscribe to them, so a metadata-only change refreshes them as a new link does. The batch rebuild_works_from_library() pass is removed.

7. Existing development libraries: reset

Bookclerk has no release, so no library holds anything that has to be carried across. The schema step changes the unreleased checksum like any other schema change. A development library then fails closed as schema versioning requires, and is recreated with cargo reset --yes. No in-place migration, state export or partial rebuild is written.

The Libation importer in bookclerk-migrate writes the new tables directly.

8. Web API

/api/library/books and /api/library/books/{uuid} are removed, with no compatibility view. They are replaced by /api/library/works, /api/library/editions and /api/library/holdings, with typed OpenAPI schemas and examples, and media-kind filters. Errors name the entity that was not found (#559).

Consequences

  • Two stores or two accounts selling one edition meet in one place, so the library can deduplicate acquires and branch deliveries by edition.
  • Mixed media is typed end to end, and print and ebook editions can show in discover without ever being acquired.
  • Per-user state means members sharing a library no longer overwrite each other's finished flags, ratings or tags.
  • Identity depends on identifier quality. A store that reuses or omits identifiers produces new editions rather than wrong merges; Administrators fix the rest with overrides.
  • Three levels add a join to most title queries. The list views read a denormalised page index refreshed by the events above, not the full join.
  • Every development library is reset once. Backups taken before the change can't be restored afterwards.
  • Every storefront plugin changes in the same pull request as the records. A third-party plugin built against the development schema has to be rebuilt, which is acceptable before the first release.

Status of the work

Planned as separate pull requests, in this order:

  1. This ADR.
  2. ABI: the new capnp fields and enums replace the flat ones, both SDKs, and the four first-party storefronts filling them. Until step 3 the host maps the structured fields onto the current books row.
  3. Schema and resolver, with its fuzz target.
  4. Acquire and files: per-edition acquisition, edition_files, and the required <call number> naming tag.
  5. Library: branch titles become holdings; node frames carry identifiers and media kind.
  6. Web API and UI.
  7. The metadata entrypoint, with OpenLibrary and Audnexus moved into plugins.

The dotted names in decision 6 name the events; the published event types are spelled in snake case, like book_acquired. Step 3 publishes edition_linked, edition_updated and holding_added, so subscribe to those spellings. It derives a listing from every stored books row, so the new tables fill as titles are scanned or imported. Step 5 publishes holding_removed when a scan, a revoke or a remap removes a holding, and makes a title a branch shares a node holding (holder node:<node id>), resolved into the library like a scanned listing: the catalog.sync frame carries the listing's marketplace beside the ASIN, ISBN and content kind, the holding carries the share and delivery state, delivered files are the edition's (an edition already in the outputs is not delivered again), and a branch that stops sharing, withdraws a title or leaves loses its holdings with holding_removed; edition_files_written comes with the step that writes files. The old tables are dropped by the step that moves their last reader off them: title_files with step 4, branch_titles with step 5 and books with step 6.

Step 4 records every file in edition_files with its edition and role, and acquires an edition once for all its account rows from one store. Instead of a discriminator appended on a clash, naming templates must write the edition's short <call number> (in the folder template, or in both file templates), so two editions never plan one path; the built-in profiles carry it, saving templates without it is refused, and an acquire whose name another edition still holds is refused rather than overwrite it. The store product id does not qualify, since one edition sold by two stores has two, and podcast feeds, some storefronts and imported files have no ISBN or ASIN. An edition's id is a version 5 UUID over the store listing that first named it (plugin key, marketplace, product id), so libraries that see the same listing first agree on its id and its <call number>, which is the first 40 bits of that UUID in Crockford base32. If that id is already taken (the listing left the library while its first edition stayed, holding other listings), the new edition gets a random id instead.

Step 5 takes the quality comparison, because node frames do not carry a file's quality yet: storefronts may declare a file's format and quality, and the media worker probes each file and records it. The probe is in place: each audio file's row holds its media type, bitrate, whether it is lossless and the store it came from, written by the acquire that stored it or carried by the upload that delivered it. The comparison follows one rule for where a copy comes from, and an install's own acquire applies it (a library does not yet replace its own copy with a better one a host uploads). When a library already holds a copy of the edition from the same store, another install of that library takes the library's copy whatever its format, and the media worker converts it to the format that install prefers. When the library's copy came from a different store, the install downloads its own from its store, probes it, and keeps whichever of the two is better. The same edition from another store is then acquired anyway, the better audio kept and the sidecars merged, and both copies marked acquired. edition_files_written comes with step 5 too.

Step 6 keeps each edition's acquire state in edition_acquisitions, written in the same batch as the title's own status. A title of an edition another title already acquired takes that copy instead of fetching it again, and a sibling title that is queued, downloading or failed never hides files the edition already has. Only the title that acquired the edition, or any title once that one has left the library, can change its state. An acquire job picks its titles from account holdings and their editions' state, not from books, so a title whose edition another account's title already acquired is not visited again. book_acquired names the edition (editionId), and a wishlist item records the edition that fulfilled it (resolved_edition_id). Metadata gap-fill and the Audible catalog match walk held editions, not books: what they find fills the work, expression and edition, a matched ASIN becomes an edition identifier, and the edition records who enriched it (enrich_source).

Step 6 also adds the page index the Consequences call for: library_work_rows, one row per held top-level work, refreshed by a library-wide host:library-index event consumer and rebuilt by a library_index_rebuild job when the index has no finished build of the current layout. GET /api/library/works pages it by keyset cursor, and GET /api/library/works/{id} reads one work with its editions, listings, holdings and the caller's own state. With q, the list route searches a per-node works index kept beside the title search index, fed by the same events and rebuilt by the same job. GET /api/library/editions/{id}/cover serves the cover an acquire wrote beside an edition's file, and POST /api/library/acquire takes editions, each queued as one of its titles until books is retired. The web app lists works, and /api/library/books (with /{uuid} and /{uuid}/cover) is removed. PUT /api/library/works/{id}/state edits the caller's own state, and an Administrator pins a listing to an edition with PUT /api/library/listings/{id}/pin (DELETE removes the pin) and reads the recorded conflicts from GET /api/library/conflicts. A pin moves the listing in the batch that records it, and its edition_linked dedup key carries the pin's revision, so moving a listing back publishes again. POST /api/library/works/{id}/split moves some of a work's expressions to a new work (copying its title, credits, series and each person's state) and isolates the new work, so the matcher keeps joining the original.

User-facing pages (the glossary, Where books go, Branches, the plugin ABI reference) change with the step that changes their behaviour.

Source: docs/adr/library-data-model.md at main · View as Markdown