bookclerk_acquire

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

Acquire orchestration: license → download → decrypt → metadata → storage.

Host-facing pipeline for fetching a storefront title, packaging audio, and writing it to one or more output destinations. DRM decrypt happens inside content-source plugins; this crate never sees license keys. See docs/server/media.md for encode/remux details.

Item Kind Summary
LibraryCopies trait Where an acquire asks for the copy a library holds of a title.
LibraryCopy trait A copy of one edition a library offered this install.
CoverImages trait Turns a cover URL a metadata provider named into an image the acquire can embed, without the acquire itself requesting the URL.
AcquireDestination struct One configured output destination for an acquire run.
AcquireDestinations struct Ordered set of acquire destinations built from the deployed destination plugins.
DestinationKeys struct The destination an acquire request names files for, which translates its host keys.
LibraryCopyFile struct One file of a library's copy, as the library listed it.
LibraryCopyListing struct The title an acquire wants, as its store lists it.
LibraryCopySource struct The libraries an acquire asks for a copy before it fetches from the store. The default asks none, so every title comes from its store.
MatchStorageOptions struct Options for match_storage_to_library.
MatchStorageSummary struct Summary of a storage→library match run.
NamingContext struct Metadata available to naming templates.
AcquireMetadata struct The deployed metadata providers an acquire asks for the plain-audio overlay (catalog tags, cover and chapter layout of an enriched ASIN), and what turns the overlay's cover URL into an image.
AcquireRequest struct Request to acquire a single title.
AcquireResult struct Result after a successful acquire.
CopyOutcome struct What one destination ended up with in one acquire.
PlannedTitleNames struct The names an acquire of one title writes on one destination, as the owner plans them, to bound a host's upload of that title.
ReconcileOptions struct Options for reconcile_library.
ReconcileSummary struct Summary of a reconcile run.
StorageIndex struct Index of storage object keys, keyed by ASIN found in the path.
ScanCheckpoint struct Versioned scan progress stored in the job payload checkpoint.
AcquireError enum Failures from the acquire / convert / reconcile pipeline.
CopyResult enum Whether a destination holds an acquire's audio.
from_store function True when the library downloaded its copy from store, the store this install's title belongs to. A copy whose store the library does not know is never taken, and one from another store only replaces the install's own download when it is the better copy.
match_storage_to_library function List audio in storage, probe metadata (no body download), match to library.
match_storage_using_index function Match library rows against an existing scan index.
audio_basename function Basename of the acquired audio file (for .cue FILE lines).
chapter_storage_key function Storage key for a split chapter file.
chapter_storage_key_with_folder function Chapter storage key with a separate folder naming context (podcast parent folder).
default_storage_key function Build a relative storage key for a acquired title using the default NamingProfile (Audiobookshelf).
resolve_templates function Resolve optional template overrides against a naming profile.
sidecar_key function Replace the audio file extension on a storage key with a sidecar suffix.
storage_key function Build a storage key from optional classic Libation folder/file templates.
storage_key_with_contexts function Build a storage key using separate folder and file naming contexts.
storage_key_with_rules function Like storage_key but applies classic ReplacementCharacters.
swap_audio_extension function Replace the audio file extension on a storage key (e.g. .m4b → .mp3).
acquire_book function Run the acquire pipeline for one book.
acquire_book_indexed function Acquire with an optional pre-built StorageIndex (avoids re-listing storage when liberating many titles). On success, newly written keys are inserted into the index so later books in the same batch can match them.
acquire_pdf_only function Download and store companion PDF only (classic acquire --pdf).
call_number_of function The short Bookclerk id of the edition account holding holding_id (holdings.id) sells, for <call number> (bookclerk_library::call_number).
convert_copy function Adds copy to the edition of the title req names, made from a copy the library already stores, on every destination whose filter takes that copy.
planned_storage_key function Compute the storage key that would be used (for dry-run / set-status).
planned_storage_key_for function Planned audio storage key for ext, honoring folder/file templates and save_podcasts_to_parent_folder: the host key the templates build, translated by the request's destination (AcquireRequest::destination_keys).
planned_storage_key_with_rules function The host key planned_storage_key_for translates, built with an explicit replacement-rule set and not translated (used by reconcile to probe wildcard patterns across sanitization profiles).
planned_title_folders function The folders an acquire of req writes the title's files under on a destination with req.options: the folder of its planned audio key, which its sidecars, chapter files and parts share. When podcast episodes are saved to the parent's folder, the episode's own folder is included too, since a library without the parent's row names it so. An empty string is the destination's root (a template with no folder).
planned_title_names function The title's planned names on the destination req.options describe: PlannedTitleNames.
share_acquired_sibling function Marks book acquired at the files of its edition, when another book of the edition acquired it (edition_acquisitions) and every destination holds the copy, so the edition is not fetched again. The edition's companion PDF is shared too when that book has one.
extract_asins_from_key function Extract Audible-like ASINs and ISBN-13 tokens from a storage key / file path.
find_existing_for_request function Same as find_existing_for_title but for a acquire request before DB status is Acquired.
find_existing_for_title function Find an existing acquired file for a title (planned path or ASIN-in-path).
reconcile_library function Scan storage and update library acquire status / storage_key for matches.
scan_storage function Builds or resumes a durable identity/object index.
Result type alias Result alias for AcquireError.
LibraryCopyError type alias Why a library's copy could not be asked for, downloaded or used. The acquire logs it and fetches from the store instead.
NAME_WILDCARD constant In a PlannedTitleNames pattern: one or more characters of a file name (no /), where the source's part or chapter title goes; the marker the owner's upload scope matches (bookclerk_library::control_plane::TitleNames).
NUMBER_WILDCARD constant In a PlannedTitleNames pattern: one or more ASCII digits, where a part or chapter number goes (the owner's upload scope matches it too).

Traits

LibraryCopies

pub trait LibraryCopies: Send + Sync {
    fn copy_of<'life0, 'life1, 'async_trait>(
        &'life0 self,
        listing: LibraryCopyListing<'life1>,
    ) -> Pin<Box<dyn Future<Output = Option<Box<dyn LibraryCopy>>> + Send + 'async_trait>>
    where
        Self: 'async_trait,
        'life0: 'async_trait,
        'life1: 'async_trait,;
}

Where an acquire asks for the copy a library holds of a title.

LibraryCopies::copy_of

fn copy_of<'life0, 'life1, 'async_trait>(
    &'life0 self,
    listing: LibraryCopyListing<'life1>,
) -> Pin<Box<dyn Future<Output = Option<Box<dyn LibraryCopy>>> + Send + 'async_trait>>
where
    Self: 'async_trait,
    'life0: 'async_trait,
    'life1: 'async_trait,

The copy a library this install joined offers of listing; None when no library holds one this install's person may take, when this install does not keep copies, or when no library answers.

A copy from the listing's own store (from_store) is taken instead of the store download; a copy from another store only replaces the store download when it is the better one. An implementation that asks several libraries therefore keeps asking past a copy from another store and returns it only when no library has one from the listing's own, and never returns a copy whose store the library does not know.

LibraryCopy

pub trait LibraryCopy: Send + Sync {
    fn source_store(&self) -> Option<&str>;
    fn probe(&self) -> FileProbe;
    fn files(&self) -> &[LibraryCopyFile];
    fn fetch_file<'life0, 'life1, 'async_trait>(
        &'life0 self,
        index: usize,
        dest: &'life1 Path,
    ) -> Pin<Box<dyn Future<Output = Result<(), LibraryCopyError>> + Send + 'async_trait>>
    where
        Self: 'async_trait,
        'life0: 'async_trait,
        'life1: 'async_trait,;
    fn taken<'life0, 'async_trait>(
        &'life0 self,
    ) -> Pin<Box<dyn Future<Output = Result<(), LibraryCopyError>> + Send + 'async_trait>>
    where
        Self: 'async_trait,
        'life0: 'async_trait,;
}

A copy of one edition a library offered this install.

LibraryCopy::source_store

fn source_store(&self) -> Option<&str>

The store the library downloaded the copy from, when it knows.

LibraryCopy::probe

fn probe(&self) -> FileProbe

What the library's probe found in the copy's primary audio: its media type, average bitrate and whether it is lossless. The acquire compares it with its own store download when the copy came from another store (bookclerk_library::FileProbe::is_better_than), and probes the copy itself before it believes the answer.

LibraryCopy::files

fn files(&self) -> &[LibraryCopyFile]

The copy's files, the primary audio first; a file's place is its download index.

LibraryCopy::fetch_file

fn fetch_file<'life0, 'life1, 'async_trait>(
    &'life0 self,
    index: usize,
    dest: &'life1 Path,
) -> Pin<Box<dyn Future<Output = Result<(), LibraryCopyError>> + Send + 'async_trait>>
where
    Self: 'async_trait,
    'life0: 'async_trait,
    'life1: 'async_trait,

Downloads file index to dest, which must end up holding exactly the file's listed size. dest is a path nothing else writes to, so an implementation may resume what it already holds.

Errors

Returns an error when the library refuses or the transfer fails.

LibraryCopy::taken

fn taken<'life0, 'async_trait>(
    &'life0 self,
) -> Pin<Box<dyn Future<Output = Result<(), LibraryCopyError>> + Send + 'async_trait>>
where
    Self: 'async_trait,
    'life0: 'async_trait,

Called once the acquire holds the copy's files and before it stores any of them, so the implementation can record where the book comes from (a branch never shares a book back to the library it took it from). Recording first means audio a failed store step leaves on a destination is already known as the library's. It may run again for the same title, so it must be safe to repeat.

Errors

Returns an error when the record cannot be written; the acquire then does not use the copy.

CoverImages

pub trait CoverImages: Send + Sync {
    fn full_cover<'life0, 'life1, 'async_trait>(
        &'life0 self,
        url: &'life1 str,
    ) -> Pin<Box<dyn Future<Output = Option<Vec<u8>>> + Send + 'async_trait>>
    where
        Self: 'async_trait,
        'life0: 'async_trait,
        'life1: 'async_trait,;
}

Turns a cover URL a metadata provider named into an image the acquire can embed, without the acquire itself requesting the URL.

The daemon implements this with its cover service: a covers plugin downloads the image and the confined media worker re-encodes it, so what reaches the tagger is a JPEG the host produced.

CoverImages::full_cover

fn full_cover<'life0, 'life1, 'async_trait>(
    &'life0 self,
    url: &'life1 str,
) -> Pin<Box<dyn Future<Output = Option<Vec<u8>>> + Send + 'async_trait>>
where
    Self: 'async_trait,
    'life0: 'async_trait,
    'life1: 'async_trait,

The full-size JPEG for url, or None when it cannot be fetched or is not an image.

Structs

AcquireDestination

pub struct AcquireDestination {
    pub id: DestinationId,
    pub backend: Box<dyn StorageBackend>,
    pub options: DownloadOptions,
    pub filter: DestinationFilter,
}

One configured output destination for an acquire run.

Fields

  • id: The destination's id: its storage plugin instance id.
  • backend: Constructed storage backend for this destination.
  • options: Source download options (quality, chapter prefs, …).
  • filter: Which books the destination takes; no rules take every book.

AcquireDestinations

pub struct AcquireDestinations {
    pub items: Vec<AcquireDestination>,
    pub primary: DestinationId,
    pub multi_destination: MultiDestinationMode,
}

Ordered set of acquire destinations built from the deployed destination plugins.

Fields

  • items: Configured acquire destinations in evaluation order.
  • primary: The primary destination, whose keys the library records as the title's storage key.
  • multi_destination: Whether multiple destinations are enabled for this run.

AcquireDestinations::len

pub fn len(&self) -> usize

Number of destinations.

AcquireDestinations::is_empty

pub fn is_empty(&self) -> bool

Returns true when no destinations are deployed.

AcquireDestinations::primary_destination

pub fn primary_destination(&self) -> &AcquireDestination

Returns the primary destination used for library storage keys.

AcquireDestinations::destination

pub fn destination(&self, id: &DestinationId) -> Option<&AcquireDestination>

Returns the destination id, if configured.

AcquireDestinations::set_filters

pub fn set_filters(&mut self, filters: &BTreeMap<String, DestinationFilter>)

Replaces every destination's filter with the one stored for it now.

A long acquire run calls this before each book's target set is chosen, so a filter edited mid-run applies to the books that start afterwards. A destination absent from filters takes every book.

Arguments
  • filters - Stored filters by plugin instance id.

AcquireDestinations::for_book

pub fn for_book(&self, facts: &PlacementFacts) -> Result<AcquireDestinations>

The destinations that take the book facts describes: the target set an acquire of it writes to, fixed before the acquire starts.

The primary stays the primary when it takes the book; otherwise the first destination that does becomes it.

Errors

Returns AcquireError::NoDestinationAccepts when no destination's filter takes the book.

AcquireDestinations::reachable

pub async fn reachable(
    &self,
) -> Result<(AcquireDestinations, Vec<(DestinationId, String)>)>

Splits the destinations into the ones that answer a listing now and the ones that do not.

An acquire run lists every destination before it starts. One that is offline is left out of the run rather than failing it: the titles go to the destinations that answer, and the missing copies are written by a later run once it is back. The primary stays the primary when it answers; otherwise the first destination that does becomes it.

Returns

The destinations that answered, and each one that did not with its listing error.

Errors

Returns AcquireError::NoDestinationReachable when none answers.

AcquireDestinations::copy_plan

pub fn copy_plan(&self) -> Vec<(OutputCopy, AcquireDestinations)>

The copies of each book this acquire makes and the destinations that receive each, in order: the main output's copy first, then output.copies (see bookclerk_source::DownloadOptions::copies_to_make, read from the primary destination's options). A destination receives a copy when its filter's copy rules take it (DestinationFilter::accepts_copy). A copy no destination takes is left out.

Each destination's options package the copy (bookclerk_source::DownloadOptions::for_copy). The main output's copy keeps the plain names; every other copy goes under a folder of its own at the top, named after the copy's label, then the destination's usual folder template (mp3 64k-mono/Title [ID]). Two copies so never share a folder or a key, on one destination or across several, and a file at the plain names is always the main copy's. The label is a short folder of its own, so path-length limits shorten the book's folders before it. Each copy's primary destination is the set's primary when it receives the copy, else the first that does.

AcquireDestinations::added_copy_plan

pub fn added_copy_plan(&self, copy: &OutputCopy) -> Option<AcquireDestinations>

The destinations that receive copy when it is added to a book already stored (a convert, see crate::convert_copy): each whose filter's copy rules take it, with options that package it under a folder of its own named after the copy, as every copy but the main output's is. None when no destination takes the copy.

The options carry copy as the library's only extra copy, so an MP3 or Opus copy is always encoded at its own quality.

AcquireDestinations::listing_backend

pub fn listing_backend(&self) -> Result<Box<dyn StorageBackend>>

Clone destination backends into a listing/match storage handle.

Reuses the backends acquire already built instead of spawning the destination guests a second time.

Errors

Returns an error when the operation fails.

AcquireDestinations::into_listing_backend

pub fn into_listing_backend(self) -> Result<Box<dyn StorageBackend>>

Consumes this set and returns a listing backend over the primary destination.

Errors

Returns an error when the operation fails.

DestinationKeys

pub struct DestinationKeys {
    /* private fields */
}

The destination an acquire request names files for, which translates its host keys.

The default stands for no destination in particular: host keys are kept as written. crate::AcquireRequests made for one destination carry that destination's backend.

DestinationKeys::of

pub fn of(backend: &dyn StorageBackend) -> Self

Keys translated by backend.

DestinationKeys::translate

pub async fn translate(&self, host_key: String) -> String

The key the destination stores host_key under.

A key that is not a valid host key (a reconcile pattern, an empty name) is returned as given. When the destination cannot translate it (its guest is down), the host key is returned and a warning logged: the write that follows then either stores it as written or fails and is retried like any other failed copy.

Implements: Clone, Debug, Default

LibraryCopyFile

pub struct LibraryCopyFile {
    pub name: String,
    pub role: FileRole,
    pub size: u64,
}

One file of a library's copy, as the library listed it.

Fields

  • name: The file's path below the title's folder on the library. Only its extension is read, to tell what kind of file it is; nothing is ever written to this path.
  • role: What the file is.
  • size: Exact size in bytes.

Implements: Clone, Debug, Eq, PartialEq

LibraryCopyListing

pub struct LibraryCopyListing<'a> {
    pub store: &'a str,
    pub plugin_key: &'a str,
    pub account_id: &'a str,
    pub marketplace: &'a str,
    pub product_id: &'a str,
}

The title an acquire wants, as its store lists it.

Fields

  • store: Store id as this install's books name it (the store's alias, such as audible).
  • plugin_key: Key of the plugin the title belongs to on this install.
  • account_id: Store account on this install that owns the title.
  • marketplace: Lowercase marketplace; empty for a store without marketplaces.
  • product_id: The store's product id.

Implements: Clone, Copy, Debug, Eq, PartialEq

LibraryCopySource

pub struct LibraryCopySource {
    /* private fields */
}

The libraries an acquire asks for a copy before it fetches from the store. The default asks none, so every title comes from its store.

LibraryCopySource::new

pub fn new(copies: Arc<dyn LibraryCopies>) -> Self

Asks copies for each title before its store.

LibraryCopySource::read_by

pub fn read_by(self, media: Arc<MediaPool>) -> Self

The same source, reading downloaded files with media instead of the process pool.

Implements: Clone, Debug, Default

MatchStorageOptions

pub struct MatchStorageOptions {
    pub account: Option<String>,
    pub clear_missing: bool,
    pub asins: Vec<String>,
    pub only_mark_found: bool,
    pub only_clear_missing: bool,
    pub fix_layout: bool,
    pub download: DownloadOptions,
}

Options for match_storage_to_library.

Fields

  • account: Optional account id filter; None means all accounts.
  • clear_missing: Clear Acquired when no matching audio is found.
  • asins: Limit to these ASINs / ids (empty = all).
  • only_mark_found: When true, only mark found files as Acquired (do not clear missing).
  • only_clear_missing: When true, only clear Acquired rows with no matching file.
  • fix_layout: Relocate matched audio (and sidecars) onto the configured template layout.
  • download: When true, treat unmatched titles as download candidates.

Implements: Clone, Debug, Default

MatchStorageSummary

pub struct MatchStorageSummary {
    pub matched: u32,
    pub relocated: u32,
    pub cleared: u32,
    pub unchanged: u32,
    pub unmatched_files: u32,
}

Summary of a storage→library match run.

Fields

  • matched: Titles matched to existing storage objects.
  • relocated: Titles whose storage_key was updated to a newly matched object.
  • cleared: Titles cleared from Acquired because no matching file was found.
  • unchanged: Titles already consistent with storage.
  • unmatched_files: Storage keys that could not be matched to a library row.

Implements: Clone, Debug, Default

NamingContext

pub struct NamingContext {
    pub asin: String,
    pub title: String,
    pub subtitle: Option<String>,
    pub authors: Option<String>,
    pub narrators: Option<String>,
    pub series: Option<String>,
    pub series_index: Option<String>,
    pub series_asin: Option<String>,
    pub year_published: Option<i32>,
    pub account_id: Option<String>,
    pub account_nickname: Option<String>,
    pub locale: Option<String>,
    pub language: Option<String>,
    pub publisher: Option<String>,
    pub categories: Option<String>,
    pub length_minutes: Option<i64>,
    pub bitrate: Option<i64>,
    pub samplerate: Option<i64>,
    pub channels: Option<i64>,
    pub codec: Option<String>,
    pub is_abridged: bool,
    pub content_kind: Option<String>,
    pub call_number: Option<String>,
    pub chapter_number: Option<u32>,
    pub chapter_count: Option<u32>,
    pub chapter_title: Option<String>,
}

Metadata available to naming templates.

Fields

  • asin: Primary product id (Audible ASIN, ISBN, …) for this job.
  • title: Display title used for naming and metadata.
  • subtitle: Optional subtitle substituted into naming templates.
  • authors: Author list for naming templates and tags.
  • narrators: Narrator list for naming templates and tags.
  • series: Series name for naming templates, when present.
  • series_index: Series position string for naming templates, when present.
  • series_asin: Podcast parent / series ASIN for SavePodcastsToParentFolder.
  • year_published: Publication year for <year> (from published_at when known).
  • account_id: Store account id that owns this title.
  • account_nickname: Operator-facing account label for <account nickname> templates.
  • locale: Marketplace / locale code for naming templates.
  • language: Content language code for naming templates.
  • publisher: Publisher name for naming templates, when known.
  • categories: Category / genre path list for naming templates.
  • length_minutes: Runtime in minutes for naming templates, when known.
  • bitrate: Audio bitrate label substituted into naming templates, when known.
  • samplerate: Audio sample rate in Hz for naming templates, when known.
  • channels: Channel count for naming templates, when known.
  • codec: Audio codec label for naming templates, when known.
  • is_abridged: Whether the edition is abridged (template boolean).
  • content_kind: Title kind (book, episode, …) for naming templates.
  • call_number: The edition's short Bookclerk id for <call number>; None for a title with no edition, which then names by its product id.
  • chapter_number: Chapter-specific: 1-based index.
  • chapter_count: Total chapter count when naming a split chapter file.
  • chapter_title: Chapter title when naming a split chapter file.

Implements: Clone, Debug, Default

AcquireMetadata

pub struct AcquireMetadata {
    /* private fields */
}

The deployed metadata providers an acquire asks for the plain-audio overlay (catalog tags, cover and chapter layout of an enriched ASIN), and what turns the overlay's cover URL into an image.

AcquireMetadata::new

pub fn new(providers: Arc<dyn MetadataProviders>) -> Self

Wraps the providers an acquire should ask.

Arguments
  • providers - Usually the daemon's or CLI's integration registry.

AcquireMetadata::with_covers

pub fn with_covers(self, covers: Arc<dyn CoverImages>) -> Self

The same providers, with covers fetching the overlay's cover.

AcquireMetadata::providers

pub fn providers(&self) -> &dyn MetadataProviders

The wrapped providers.

Implements: Clone, Debug, Default

AcquireRequest

pub struct AcquireRequest {
    pub asin: String,
    pub holding_id: Option<String>,
    pub source: String,
    pub account_id: String,
    pub title: String,
    pub authors: Option<String>,
    pub narrators: Option<String>,
    pub series: Option<String>,
    pub series_index: Option<String>,
    pub options: DownloadOptions,
    pub files_dir: PathBuf,
    pub cache_dir: PathBuf,
    pub force: bool,
    pub write_destinations: Option<Vec<DestinationId>>,
    pub placement: Option<PlacementFacts>,
    pub job_id: Option<String>,
    pub temp_quota_bytes: Option<u64>,
    pub call_number: Option<String>,
    pub parent_call_number: Option<String>,
    pub metadata: AcquireMetadata,
    pub library_copies: LibraryCopySource,
    pub destination_keys: DestinationKeys,
}

Request to acquire a single title.

Fields

  • asin: Download product id (Audible ASIN / Libro ISBN).
  • holding_id: The account holding (holdings.id) the title is acquired for, when known: acquire state is written under it. Without one the state is written to the title asin names on account_id.
  • source: Which store owns this title (plugin id: audible, libro, …).
  • account_id: Store account id that owns this title.
  • title: Display title used for naming and metadata.
  • authors: Author list for naming templates and tags.
  • narrators: Narrator list for naming templates and tags.
  • series: Series name for naming templates, when present.
  • series_index: Series position string for naming templates, when present.
  • options: Source download options (quality, chapter prefs, …).
  • files_dir: Bookclerk files directory root (BOOKCLERK_FILES_DIR).
  • cache_dir: Scratch directory for encrypted + decrypted temps.
  • force: When true, download even if matching media already exists in storage.
  • write_destinations: When set, only write prepared audio to these destinations (output.multi_destination = refetch_missing).
  • placement: What the destination filters know about the book. When set, the acquire writes only to the destinations whose filter takes it (AcquireDestinations::for_book) and fails before fetching when none does. None writes to every destination given: a host acquiring for an owner gets destinations the owner already chose.
  • job_id: Durable daemon job id that owns this acquire's scratch directory.
  • temp_quota_bytes: Refuse a new scratch dir when registered temps already exceed this.
  • call_number: The <call number> to name the title by instead of its edition in library: a host acquiring for a shared node names the title as the owner does, since its scratch library's edition ids are its own.
  • parent_call_number: Likewise for the title's podcast parent, whose folder an episode may be saved in.
  • metadata: Metadata providers asked for the plain-audio catalog and chapter overlay. The default has none, so plain audio keeps its store tags.
  • library_copies: The libraries asked for the copy they already hold of the title before it is fetched from its store. The default asks none.
  • destination_keys: The destination this request names files for, which translates the host keys its naming templates build. The default keeps host keys; the per-destination requests set it for each destination.

Implements: Clone, Debug

AcquireResult

pub struct AcquireResult {
    pub asin: String,
    pub storage_key: String,
    pub written_keys: Vec<String>,
    pub matched_existing: bool,
    pub from_library: bool,
    pub store_fetches: u32,
    pub copies: Vec<CopyOutcome>,
}

Result after a successful acquire.

Fields

  • asin: Primary product id (Audible ASIN, ISBN, …) for this job.
  • storage_key: Object-storage key written for the primary audio artifact.
  • written_keys: All object keys written during this acquire (audio + sidecars).
  • matched_existing: True when an existing file was matched and no download ran.
  • from_library: True when the audio was taken from the copy a library this install joined already held, instead of fetched from the store.
  • store_fetches: How many times this acquire downloaded the title (or its companion PDF) from the store: 0 for a copy matched in storage or taken from a joined library, 1 for a store download (even one a better copy from a joined library replaced afterwards), 2 when that library copy then failed packaging and the store was asked again. Download pacing counts each against the account (bookclerk_library::pacing).
  • copies: Each destination this acquire wrote or found the audio on, and the ones whose writes failed; recorded as the edition's copies (edition_copies) once the acquire returns. Empty when nothing was written or checked per destination (an edition already stored everywhere, or a companion PDF).

Implements: Clone, Debug, Deserialize<'de>, Serialize

CopyOutcome

pub struct CopyOutcome {
    pub destination: DestinationId,
    pub result: CopyResult,
}

What one destination ended up with in one acquire.

Fields

  • destination: The destination's id.
  • result: Whether the destination now holds the audio.

Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize

PlannedTitleNames

pub struct PlannedTitleNames {
    pub folders: Vec<String>,
    pub own_folder: bool,
    pub audio_stems: Vec<String>,
    pub part_patterns: Vec<String>,
}

The names an acquire of one title writes on one destination, as the owner plans them, to bound a host's upload of that title.

Fields

  • folders: The folders the title's files go under (planned_title_folders).
  • own_folder: True when the folder template has a component of the title's own (its id, title or subtitle), so no other title's files go there and any name under it is the title's. False for a folder several titles share (<author>), the destination's root, or a podcast parent's folder: there only the names below are the title's.
  • audio_stems: The planned audio key of a single-file acquire, without its extension (one per folder variant): the title's audio is this stem with any audio extension, and its sidecars this stem, a . and a suffix.
  • part_patterns: Part and chapter keys without their extension, with NAME_WILDCARD and NUMBER_WILDCARD where the source's part or chapter title and number go. Those titles are not known before the acquire runs, so they are matched rather than listed; a template with no part or chapter field yields a literal name.

Implements: Clone, Debug, Eq, PartialEq

ReconcileOptions

pub struct ReconcileOptions {
    pub account: Option<String>,
    pub clear_missing: bool,
    pub asins: Vec<String>,
    pub only_mark_found: bool,
    pub only_clear_missing: bool,
    pub download: DownloadOptions,
    pub fence: Option<JobFence>,
    pub checkpoint: Option<JobCheckpoint>,
}

Options for reconcile_library.

Fields

  • account: Optional account id filter; None means all accounts.
  • clear_missing: When true, books marked Acquired whose file is missing become NotAcquired.
  • asins: Limit to these ASINs (empty = all).
  • only_mark_found: When true, only mark found files as Acquired (do not clear missing).
  • only_clear_missing: When true, only clear Acquired rows with no matching file.
  • download: Naming prefs (templates, podcast parent folder) for planned-path matching.
  • fence: Lease fence when this reconcile is a phase of a durable job.
  • checkpoint: Checkpoint to resume, when fence is set.

Implements: Clone, Debug, Default

ReconcileSummary

pub struct ReconcileSummary {
    pub matched: u32,
    pub cleared: u32,
    pub unchanged: u32,
}

Summary of a reconcile run.

Fields

  • matched: Titles matched to existing storage objects.
  • cleared: Titles cleared from Acquired because no matching file was found.
  • unchanged: Titles already consistent with storage.

Implements: Clone, Debug, Default

StorageIndex

pub struct StorageIndex {
    /* private fields */
}

Index of storage object keys, keyed by ASIN found in the path.

StorageIndex::from_storage

pub async fn from_storage(storage: &dyn StorageBackend) -> Result<Self>

Empty overlay.

This does not list storage. Call crate::scan_storage when identity matching needs a durable index. Exact keys are resolved with StorageBackend::exists at decision time.

Errors

Always returns Ok. The storage argument is accepted so older call sites keep compiling while they move to crate::scan_storage.

StorageIndex::with_scan

pub fn with_scan(scan_id: impl Into<String>) -> Self

Overlay plus a durable scan id.

StorageIndex::scan_id

pub fn scan_id(&self) -> Option<&str>

Durable scan generation, when present.

StorageIndex::insert_key

pub fn insert_key(&mut self, key: String)

Insert a storage key into the index (test helper / incremental).

StorageIndex::contains_key

pub fn contains_key(&self, key: &str) -> bool

Returns true when key is present in the index.

StorageIndex::keys_matching_pattern

pub fn keys_matching_pattern(&self, pattern: &str) -> Vec<String>

Overlay keys matching a reconcile wildcard pattern, best media rank first.

StorageIndex::find_key_matching_pattern

pub fn find_key_matching_pattern(&self, pattern: &str) -> Option<&str>

First key matching a reconcile wildcard pattern, preferring better media types.

StorageIndex::best_key_for_asin

pub fn best_key_for_asin(&self, asin: &str) -> Option<&str>

Best matching storage key for an ASIN / ISBN token, if any.

Implements: Clone, Debug, Default

ScanCheckpoint

pub struct ScanCheckpoint {
    pub v: u32,
    pub op: String,
    pub instance_id: String,
    pub namespace: String,
    pub generation: String,
    pub phase: String,
    pub cursor: Option<String>,
    pub node_local: bool,
    pub placement: Option<String>,
}

Versioned scan progress stored in the job payload checkpoint.

Fields

  • v: Schema version. Unknown versions fail closed.
  • op: Always storage_scan.
  • instance_id: StorageBackend::instance_id this cursor belongs to.
  • namespace: List prefix (empty scans the namespace).
  • generation: storage_scan_rows.scan_id.
  • phase: list while paging, apply after the scan committed.
  • cursor: Backend list cursor. None starts at the first page.
  • node_local: True when the list index or scratch is node-local.
  • placement: Host placement for node-local bytes. Absent means the scan was recorded as portable. A mismatch restarts instead of adopting another node's rows.

Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize

Enums

AcquireError

pub enum AcquireError {
    Source(SourceError),
    Storage(StorageError),
    Media(MediaError),
    Library(LibraryError),
    Io(Error),
    NoDestinationAccepts,
    NoDestinationReachable { errors: String },
    Other(Error),
}

Failures from the acquire / convert / reconcile pipeline.

Variants

  • Source: Failure originating in a content-source plugin.
  • Storage: Failure originating in an object-storage backend.
  • Media: Failure originating in media encode/remux/fixup.
  • Library: Failure originating in the library database layer.
  • Io: Filesystem or other I/O failure.
  • NoDestinationAccepts: No deployed destination's filter takes the book, so the acquire stops before it fetches anything.
  • NoDestinationReachable: No deployed destination answered a listing, so an acquire run has nowhere to write.
    • errors: Each destination with its listing error, ; -separated.
  • Other: Catch-all for otherwise unclassified failures.

Implements: Debug, Display, Error, From<Error>, From<LibraryError>, From<MediaError>, From<SourceError>, From<StorageError>

CopyResult

pub enum CopyResult {
    Stored { key: String },
    Failed { error: String },
}

Whether a destination holds an acquire's audio.

Variants

  • Stored: The destination holds the complete files.
    • key: The edition's primary audio key there.
  • Failed: The destination's writes failed; the copy is retried later.
    • error: The last write's error.

Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize

Functions

from_store

pub fn from_store(copy_store: Option<&str>, store: &str) -> bool

True when the library downloaded its copy from store, the store this install's title belongs to. A copy whose store the library does not know is never taken, and one from another store only replaces the install's own download when it is the better copy.

An implementation of LibraryCopies that asks several libraries uses this to pass over a copy from another store and ask the next library.

match_storage_to_library

pub async fn match_storage_to_library(
    library: &LibraryStore,
    storage: &dyn StorageBackend,
    options: MatchStorageOptions,
) -> Result<MatchStorageSummary>

List audio in storage, probe metadata (no body download), match to library.

Errors

Returns an error when the operation fails.

match_storage_using_index

pub async fn match_storage_using_index(
    library: &LibraryStore,
    storage: &dyn StorageBackend,
    options: MatchStorageOptions,
    index: &StorageIndex,
) -> Result<MatchStorageSummary>

Match library rows against an existing scan index.

Does not delete the generation. The caller finishes it when the job completes.

Errors

Returns an error when the operation fails.

audio_basename

pub fn audio_basename(storage_key: &str) -> String

Basename of the acquired audio file (for .cue FILE lines).

chapter_storage_key

pub fn chapter_storage_key(
    ctx: &NamingContext,
    folder_template: Option<&str>,
    chapter_file_template: Option<&str>,
    replacement_rules: &[ReplacementRule],
    chapter_number: usize,
    chapter_title: &str,
    ext: &str,
) -> String

Storage key for a split chapter file.

chapter_storage_key_with_folder

pub fn chapter_storage_key_with_folder(
    folder_ctx: &NamingContext,
    file_ctx: &NamingContext,
    folder_template: Option<&str>,
    chapter_file_template: Option<&str>,
    replacement_rules: &[ReplacementRule],
    chapter_number: usize,
    chapter_title: &str,
    ext: &str,
    path_limits: PathLimits,
) -> String

Chapter storage key with a separate folder naming context (podcast parent folder).

default_storage_key

pub fn default_storage_key(
    authors: Option<&str>,
    title: &str,
    asin: &str,
    ext: &str,
) -> String

Build a relative storage key for a acquired title using the default NamingProfile (Audiobookshelf).

Uses POSIX separator rules for creation. Reconcile probes this layout with wildcard sanitization so historical Windows/S3 keys still match.

resolve_templates

pub fn resolve_templates(
    profile: NamingProfile,
    folder_template: Option<&str>,
    file_template: Option<&str>,
    chapter_file_template: Option<&str>,
) -> ResolvedNamingTemplates

Resolve optional template overrides against a naming profile.

sidecar_key

pub fn sidecar_key(audio_storage_key: &str, suffix: &str) -> String

Replace the audio file extension on a storage key with a sidecar suffix.

storage_key

pub fn storage_key(
    ctx: &NamingContext,
    folder_template: Option<&str>,
    file_template: Option<&str>,
    ext: &str,
) -> String

Build a storage key from optional classic Libation folder/file templates.

storage_key_with_contexts

pub fn storage_key_with_contexts(
    folder_ctx: &NamingContext,
    file_ctx: &NamingContext,
    folder_template: Option<&str>,
    file_template: Option<&str>,
    ext: &str,
    replacement_rules: &[ReplacementRule],
    path_limits: PathLimits,
) -> String

Build a storage key using separate folder and file naming contexts.

Used when SavePodcastsToParentFolder evaluates the folder template against the podcast parent while the file template still uses the episode.

storage_key_with_rules

pub fn storage_key_with_rules(
    ctx: &NamingContext,
    folder_template: Option<&str>,
    file_template: Option<&str>,
    ext: &str,
    replacement_rules: &[ReplacementRule],
) -> String

Like storage_key but applies classic ReplacementCharacters.

swap_audio_extension

pub fn swap_audio_extension(key: &str, new_ext: &str) -> String

Replace the audio file extension on a storage key (e.g. .m4b → .mp3).

acquire_book

pub async fn acquire_book(
    library: &LibraryStore,
    destinations: &AcquireDestinations,
    req: AcquireRequest,
    source: &dyn ContentSource,
) -> Result<AcquireResult>

Run the acquire pipeline for one book.

Arguments

  • library - Library store for status updates.
  • source - Content-source plugin used to fetch the title.
  • destinations - Output backends to write packaged audio into.
  • req - Title identity, account, and download options.

Returns

AcquireResult describing written keys and match status.

Errors

Returns AcquireError when fetch, package, storage, or library updates fail.

acquire_book_indexed

pub async fn acquire_book_indexed(
    library: &LibraryStore,
    destinations: &AcquireDestinations,
    req: AcquireRequest,
    index: Option<&mut StorageIndex>,
    source: &dyn ContentSource,
) -> Result<AcquireResult>

Acquire with an optional pre-built StorageIndex (avoids re-listing storage when liberating many titles). On success, newly written keys are inserted into the index so later books in the same batch can match them.

Fetch always goes through ContentSource::fetch_title (Plain only; Audible decrypts Adrm/CENC inside the plugin before returning).

Every key the run writes on a destination (audio, chapter or part files, sidecars, the companion PDF) is recorded for the book's edition in the library's edition_files, with its role, just before it is written, so a host upload of another edition is never granted it.

A book whose edition another book has already acquired (the same edition bought on two accounts, say) is marked acquired at that book's key without fetching it again.

Errors

Returns an error when the operation fails.

acquire_pdf_only

pub async fn acquire_pdf_only(
    library: &LibraryStore,
    destinations: &AcquireDestinations,
    req: &AcquireRequest,
    source: &dyn ContentSource,
) -> Result<AcquireResult>

Download and store companion PDF only (classic acquire --pdf).

Fetches via ContentSource::fetch_title; requires PlainFetch.pdf_url, then HTTP GETs that URL. Each PDF key written is recorded for the book's edition in edition_files.

Errors

Returns an error when the operation fails.

call_number_of

pub async fn call_number_of(
    library: &LibraryStore,
    holding_id: &str,
) -> Result<Option<String>>

The short Bookclerk id of the edition account holding holding_id (holdings.id) sells, for <call number> (bookclerk_library::call_number).

Returns

None when no account holding has that id.

Errors

Returns an error when the holding cannot be read.

convert_copy

pub async fn convert_copy(
    library: &LibraryStore,
    destinations: &AcquireDestinations,
    req: AcquireRequest,
    copy: &OutputCopy,
) -> Result<AcquireResult>

Adds copy to the edition of the title req names, made from a copy the library already stores, on every destination whose filter takes that copy.

The source is the best stored copy the destinations hold: one kept as delivered or as an M4B before an encoded one, then the main output's copy, then the primary destination's. Its files are downloaded into scratch and packaged like a store download, so tags, chapters, cover and naming follow the copy's options. The new files are recorded as copy's on the edition (edition_files); the title's storage key and its main copy are left as they are. No store is asked.

Arguments

  • library - Library store holding the edition's recorded files.
  • destinations - Every reachable destination, with its current filter.
  • req - The title, as an acquire of it would name it; force makes the copy again when the edition already has it.
  • copy - The copy to add.

Returns

What was written: matched_existing is true when the edition already had the copy and force was off.

Errors

Returns AcquireError::Other for a copy that is not a new encoding (original), the library's main output, a title that has no library row or is not stored, or an edition with no stored audio on a reachable destination; AcquireError::NoDestinationAccepts when no destination takes the book or the copy; and the download, packaging or storage error otherwise.

planned_storage_key

pub async fn planned_storage_key(library: &LibraryStore, req: &AcquireRequest) -> String

Compute the storage key that would be used (for dry-run / set-status).

Uses the library row (when present) so podcast episodes honor save_podcasts_to_parent_folder the same way as a real acquire.

planned_storage_key_for

pub async fn planned_storage_key_for(
    library: &LibraryStore,
    req: &AcquireRequest,
    ext: &str,
) -> String

Planned audio storage key for ext, honoring folder/file templates and save_podcasts_to_parent_folder: the host key the templates build, translated by the request's destination (AcquireRequest::destination_keys).

planned_storage_key_with_rules

pub async fn planned_storage_key_with_rules(
    library: &LibraryStore,
    req: &AcquireRequest,
    ext: &str,
    replacement_rules: &[ReplacementRule],
) -> String

The host key planned_storage_key_for translates, built with an explicit replacement-rule set and not translated (used by reconcile to probe wildcard patterns across sanitization profiles).

planned_title_folders

pub async fn planned_title_folders(
    library: &LibraryStore,
    req: &AcquireRequest,
) -> Vec<String>

The folders an acquire of req writes the title's files under on a destination with req.options: the folder of its planned audio key, which its sidecars, chapter files and parts share. When podcast episodes are saved to the parent's folder, the episode's own folder is included too, since a library without the parent's row names it so. An empty string is the destination's root (a template with no folder).

planned_title_names

pub async fn planned_title_names(
    library: &LibraryStore,
    req: &AcquireRequest,
) -> PlannedTitleNames

The title's planned names on the destination req.options describe: PlannedTitleNames.

share_acquired_sibling

pub async fn share_acquired_sibling(
    library: &LibraryStore,
    destinations: &[(&str, &dyn StorageBackend)],
    title: &AcquireTitle,
) -> Result<Option<String>>

Marks book acquired at the files of its edition, when another book of the edition acquired it (edition_acquisitions) and every destination holds the copy, so the edition is not fetched again. The edition's companion PDF is shared too when that book has one.

A destination holds the copy when the sibling's primary key there is the edition's own (unrecorded, as storage matching adopts it, or recorded for this edition, never another's: a listing that moved editions leaves its old files with the old one) and every audio key recorded for the edition there (parts, chapters) is present.

Arguments

  • destinations - Each destination's id and backend.
  • book - The book to mark acquired.

Returns

The shared primary key; None when the book has no edition yet, or no other book of its edition is acquired with a copy every destination holds (a destination deployed since, another node's layout, or a file removed since). The acquire then plans as usual and syncs or fetches what is missing.

Errors

Returns an error when a library lookup or write, or a backend probe, fails.

extract_asins_from_key

pub fn extract_asins_from_key(key: &str) -> Vec<String>

Extract Audible-like ASINs and ISBN-13 tokens from a storage key / file path.

Matches:

  • filename stem B00EXAMPLE.m4b
  • bracket form Title [B00EXAMPLE].m4b (classic Libation)
  • path segment containing a standalone ASIN token
  • ISBN-13 shaped tokens (13 digits, optional hyphens)

find_existing_for_request

pub async fn find_existing_for_request(
    index: &StorageIndex,
    storage: &dyn StorageBackend,
    library: &LibraryStore,
    req: &AcquireRequest,
) -> Result<Option<String>>

Same as find_existing_for_title but for a acquire request before DB status is Acquired.

Matching strategy:

  1. Exact planned path under the creation replacement rules
  2. Same templates with sanitizable characters as wildcards (cross OS/backend)
  3. Template path without podcast-parent rewrite (exact, then wildcard)
  4. ASIN token found anywhere in a storage key

Errors

Returns a storage or database error from the lookup. A completed miss is Ok(None).

find_existing_for_title

pub async fn find_existing_for_title(
    index: &StorageIndex,
    storage: &dyn StorageBackend,
    library: &LibraryStore,
    title: &AcquireTitle,
    download: &DownloadOptions,
) -> Result<Option<String>>

Find an existing acquired file for a title (planned path or ASIN-in-path).

Honors configured folder/file templates and save_podcasts_to_parent_folder via the same path planner as acquire.

Errors

Returns a storage or database error from the lookup. A completed miss is Ok(None) and is not an error.

reconcile_library

pub async fn reconcile_library(
    library: &LibraryStore,
    storage: &dyn StorageBackend,
    options: ReconcileOptions,
) -> Result<ReconcileSummary>

Scan storage and update library acquire status / storage_key for matches.

Errors

Returns an error when the operation fails.

scan_storage

pub async fn scan_storage(
    library: &LibraryStore,
    storage: &dyn StorageBackend,
    fence: Option<&JobFence>,
    prior: Option<&JobCheckpoint>,
    probe_metadata: bool,
) -> Result<StorageIndex>

Builds or resumes a durable identity/object index.

Errors

Returns AcquireError when listing, the database, or a checkpoint write fails, or when prior names a different storage instance.

Type aliases

Result

pub type Result<T> = Result<T, AcquireError>;

Result alias for AcquireError.

LibraryCopyError

pub type LibraryCopyError = Box<dyn Error + Send + Sync>;

Why a library's copy could not be asked for, downloaded or used. The acquire logs it and fetches from the store instead.

Constants

NAME_WILDCARD

pub const NAME_WILDCARD: char = bookclerk_library::control_plane::UPLOAD_NAME_WILDCARD;

In a PlannedTitleNames pattern: one or more characters of a file name (no /), where the source's part or chapter title goes; the marker the owner's upload scope matches (bookclerk_library::control_plane::TitleNames).

NUMBER_WILDCARD

pub const NUMBER_WILDCARD: char = bookclerk_library::control_plane::UPLOAD_NUMBER_WILDCARD;

In a PlannedTitleNames pattern: one or more ASCII digits, where a part or chapter number goes (the owner's upload scope matches it too).

Generated from crates/bookclerk-acquire at main · View as Markdown