bookclerk_storage
Storage backends for acquired audiobooks, covers, and PDFs.
Host acquire / library code that writes through StorageBackend. Guest
plugins do not depend on this crate.
Product narrative: destination sections in docs/server/configuration.md. Style:
docs/contributing/code-documentation.md.
| Item | Kind | Summary |
|---|---|---|
StorageBackend |
trait | Pluggable storage for acquired audio and sidecar files. |
B2ApplicationKey |
struct | An application key's secret half. Its Debug output never shows it. |
B2Backend |
struct | A bucket (and prefix) on Backblaze B2. |
B2Settings |
struct | How to reach one bucket. |
FanoutBackend |
struct | Multiplexes one logical storage key across multiple backends. |
LocalFsBackend |
struct | Stores objects under a root directory; keys map to relative paths. |
CloudflareAccountId |
struct | A Cloudflare account id: 32 lowercase hex digits. |
CloudflareApiToken |
struct | A Cloudflare API token with Queues read and write on the account. |
CloudflareQueueId |
struct | A Cloudflare Queue id: 32 lowercase hex digits (the queue's id, not its name). |
S3Backend |
struct | S3-compatible object storage. |
S3Credentials |
struct | Static AWS-style credentials for the S3 destination. |
S3Settings |
struct | Connection settings for one S3 / MinIO destination. |
QueueUrl |
struct | An Amazon SQS queue URL, checked at the boundary. |
ByteRange |
struct | Inclusive byte range for a streamed read. |
ListPage |
struct | One page of ObjectInfo from StorageBackend::list_page. |
ObjectInfo |
struct | Listing entry returned by StorageBackend::list. |
ObjectMeta |
struct | Metadata attached to a stored object. |
ObjectProbe |
struct | Cheap object probe (S3 HeadObject / local sidecar meta) — never downloads object bodies. |
PutStreamResult |
struct | Result of StorageBackend::put_stream. |
TransferOptions |
struct | Knobs for transfer_object. |
StorageError |
enum | Errors from local / S3 / fan-out storage backends. |
TransferOutcome |
enum | How transfer_object moved the object. |
ensure_scalar_len |
function | Rejects a scalar put whose buffer already exceeds limit. |
normalize_range |
function | Normalizes an internal range. |
parse_sha256_hex |
function | Parses a hex SHA-256 (64 hex chars) into raw bytes. |
read_scalar_body |
function | Reads at most limit + 1 bytes. |
reject_scalar_hint |
function | Early-reject when a HEAD/stat hint already exceeds limit. |
sha256_field_from_raw |
function | Validates a raw SHA-256 field (ABI Data, 32 bytes) into hex. |
node_placement_id |
function | This node's id, or a process-lifetime id when the node identity cannot be read. |
bookclerk_meta_sidecar_key |
function | Sidecar key for probe metadata of this exact object ({key}.bookclerk-meta.json). |
is_audio_key |
function | True when key ends with a known acquired audio extension. |
transfer_object |
function | Copies source_key onto dest_key. |
Result |
type alias | Result alias for storage backend operations in this crate. |
B2_AUTHORIZE_URL |
constant | Where b2_authorize_account lives; every other URL comes from its reply. |
MAX_LIST_PAGE |
constant | ABI list page ceiling. |
MAX_SCALAR_OBJECT_BYTES |
constant | ABI scalar ceiling. Audiobook bodies must not use crate::StorageBackend::get / crate::StorageBackend::put. |
MAX_SUPPORTED_OBJECT_BYTES |
constant | Largest object this process will upload or server-copy. |
AUDIO_EXTENSIONS |
constant | Audio extensions considered acquired media for storage matching. |
intent::ChangeFeedPageintent::ConditionalPutResultintent::HintedListPageintent::KeyResultintent::ListRequestintent::MoveOptionsintent::MovePrefixPageintent::MoveResultintent::ObjectChangeintent::ObjectChangeKindintent::StorageCapabilitiesintent::WriteCondition
b2_api: Backblaze B2 native API replies, read into checked types.fs_kind: What kind of file system a local path lives on, so the local backend only leans on hard links, inode numbers and atomic renames where the file system keeps their promises.guest: Serving a backend's intent operations from a storage guest. Serves aStorageBackend's intent operations through a storage guest'sDestinationcapability.intent: Intent operations (moves, batches, conditional writes, change feeds) and their fallbacks. Intent-based storage operations and the host's fallback for each.key_syntax: The host's key syntax, and how a storage backend turns a host key into the path or object key it stores.r2_events: Cloudflare R2 event notifications and Queues pull replies, parsed into change-feed entries.s3_events: Object-store event notifications, parsed into change-feed entries.s3_guest: A storage guest'sDestinationcapability over an object store.
pub trait StorageBackend: Send + Sync {
fn name(&self) -> &'static str;
fn instance_id(&self) -> String;
fn scan_placement(&self) -> Option<String>;
fn clone_box(&self) -> Box<dyn StorageBackend>;
fn put<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
data: Bytes,
meta: ObjectMeta,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn put_file<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
key: &'life1 str,
path: &'life2 Path,
meta: ObjectMeta,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,;
fn get<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Bytes>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn exists<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<bool>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn list<'life0, 'life1, 'async_trait>(
&'life0 self,
prefix: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Vec<ObjectInfo>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn list_audio<'life0, 'life1, 'async_trait>(
&'life0 self,
prefix: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Vec<ObjectInfo>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn probe<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<ObjectProbe>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn copy<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
from: &'life1 str,
to: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,;
fn copy_sized<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
from: &'life1 str,
to: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<u64>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,;
fn rename<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
from: &'life1 str,
to: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,;
fn capabilities(&self) -> StorageCapabilities;
fn move_object<'life0, 'life1, 'life2, 'life3, 'async_trait>(
&'life0 self,
from: &'life1 str,
to: &'life2 str,
options: &'life3 MoveOptions,
) -> Pin<Box<dyn Future<Output = Result<MoveResult>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,;
fn move_many<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
pairs: &'life1 [(String, String)],
options: &'life2 MoveOptions,
) -> Pin<Box<dyn Future<Output = Result<Vec<KeyResult>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,;
fn move_prefix<'life0, 'life1, 'life2, 'life3, 'life4, 'async_trait>(
&'life0 self,
from_prefix: &'life1 str,
to_prefix: &'life2 str,
cursor: Option<&'life3 str>,
limit: u32,
options: &'life4 MoveOptions,
) -> Pin<Box<dyn Future<Output = Result<MovePrefixPage>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,
'life4: 'async_trait,;
fn delete_many<'life0, 'life1, 'async_trait>(
&'life0 self,
keys: &'life1 [String],
) -> Pin<Box<dyn Future<Output = Result<Vec<KeyResult>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn translate_keys<'life0, 'life1, 'async_trait>(
&'life0 self,
keys: &'life1 [HostKey],
) -> Pin<Box<dyn Future<Output = Result<Vec<String>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn head_many<'life0, 'life1, 'async_trait>(
&'life0 self,
keys: &'life1 [String],
) -> Pin<Box<dyn Future<Output = Result<Vec<Option<ObjectProbe>>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn put_stream_conditional<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
key: &'life1 str,
body: Pin<Box<dyn AsyncRead + Send>>,
meta: ObjectMeta,
condition: &'life2 WriteCondition,
) -> Pin<Box<dyn Future<Output = Result<ConditionalPutResult>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,;
fn list_hinted<'life0, 'life1, 'async_trait>(
&'life0 self,
request: &'life1 ListRequest,
) -> Pin<Box<dyn Future<Output = Result<HintedListPage>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn changes<'life0, 'life1, 'async_trait>(
&'life0 self,
cursor: Option<&'life1 str>,
limit: u32,
) -> Pin<Box<dyn Future<Output = Result<Option<ChangeFeedPage>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn delete<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn head<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Option<ObjectProbe>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn list_page<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
prefix: &'life1 str,
cursor: Option<&'life2 str>,
limit: u32,
) -> Pin<Box<dyn Future<Output = Result<ListPage>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,;
fn get_stream<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
range: Option<ByteRange>,
) -> Pin<Box<dyn Future<Output = Result<(ObjectProbe, Pin<Box<dyn AsyncRead + Send>>)>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn put_stream<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
body: Pin<Box<dyn AsyncRead + Send>>,
meta: ObjectMeta,
) -> Pin<Box<dyn Future<Output = Result<PutStreamResult>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
fn supports_server_copy(&self) -> bool;
fn touch_file<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
created: Option<SystemTime>,
modified: Option<SystemTime>,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,;
}
Pluggable storage for acquired audio and sidecar files.
fn name(&self) -> &'static str
Backend name for logs (local, s3).
fn instance_id(&self) -> String
Stable identity of this authorized namespace (root, bucket, prefix, session).
Self::supports_server_copy plus equal ids is the only signal that
Self::copy stays inside one backend. Comparing Self::name is not
enough.
fn scan_placement(&self) -> Option<String>
Host placement when this backend's bytes are not portable across nodes.
None means a scan may resume on any host with the same
Self::instance_id (object storage). Some is a stable host id.
Wrappers and fan-out must forward a child's placement; they must not
guess locality from an instance_id prefix. When placement cannot be
proved, return a host id so the scan restarts on another node instead
of adopting the wrong inventory.
fn clone_box(&self) -> Box<dyn StorageBackend>
Clone into a new boxed backend (same client / root).
fn put<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
data: Bytes,
meta: ObjectMeta,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Write bytes under key.
fn put_file<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
key: &'life1 str,
path: &'life2 Path,
meta: ObjectMeta,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
Stream a local file into storage.
The default opens the file and calls Self::put_stream. It does not
read the body into a Bytes buffer. Backends may override with a
native copy that still avoids a userspace buffer of the whole object.
Returns I/O errors from the file or the backend write.
fn get<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Bytes>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Download the full object body into memory.
Returns crate::StorageError::NotFound when missing, and I/O or S3
failures otherwise.
fn exists<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<bool>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Return whether key exists without downloading the body.
Propagates backend probe failures (not merely absence).
fn list<'life0, 'life1, 'async_trait>(
&'life0 self,
prefix: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Vec<ObjectInfo>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Compatibility collector for a namespace that fits in one page.
Returns StorageError::PayloadTooLarge when another page exists so a
product caller cannot rebuild a full inventory by accident. Scans use
Self::list_page.
Returns StorageError::PayloadTooLarge when the prefix holds more
than one page, plus listing errors from Self::list_page.
fn list_audio<'life0, 'life1, 'async_trait>(
&'life0 self,
prefix: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Vec<ObjectInfo>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Audio objects under prefix, capped at one page of matches.
See AUDIO_EXTENSIONS. Pages are discarded as they are scanned. More
than crate::bounded::MAX_LIST_PAGE audio objects is an error.
Returns StorageError::PayloadTooLarge when the audio match set
exceeds one page, plus listing errors.
fn probe<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<ObjectProbe>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Probe object metadata without downloading the body.
S3 uses HeadObject (user metadata). Local reads an optional
.bookclerk-meta.json sidecar written on put.
fn copy<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
from: &'life1 str,
to: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
Copy from → to within the same backend (S3 server-side copy / local
file copy). Preserves object metadata when the backend supports it.
fn copy_sized<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
from: &'life1 str,
to: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<u64>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
Self::copy, returning the bytes copied. The default copies and
then probes the target.
Returns StorageError::NotFound when from is missing, and
backend errors otherwise.
fn rename<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
from: &'life1 str,
to: &'life2 str,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
Move from → to, replacing any object at to.
Forwards to Self::move_object, so a backend that renames natively
does it in one step.
fn capabilities(&self) -> StorageCapabilities
Which intent operations this backend does natively. The default is none: every operation runs its fallback from the primitives.
fn move_object<'life0, 'life1, 'life2, 'life3, 'async_trait>(
&'life0 self,
from: &'life1 str,
to: &'life2 str,
options: &'life3 MoveOptions,
) -> Pin<Box<dyn Future<Output = Result<MoveResult>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,
Move one object. The default is crate::intent::fallback_move
(probe, copy, delete).
Returns StorageError::NotFound when neither key exists,
StorageError::Conflict when a precondition fails, and backend
errors otherwise.
fn move_many<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
pairs: &'life1 [(String, String)],
options: &'life2 MoveOptions,
) -> Pin<Box<dyn Future<Output = Result<Vec<KeyResult>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
Move each (from, to) pair, one outcome per pair in order. The
default loops Self::move_object.
Returns an error only when the whole call fails; per-pair failures are in the outcomes.
fn move_prefix<'life0, 'life1, 'life2, 'life3, 'life4, 'async_trait>(
&'life0 self,
from_prefix: &'life1 str,
to_prefix: &'life2 str,
cursor: Option<&'life3 str>,
limit: u32,
options: &'life4 MoveOptions,
) -> Pin<Box<dyn Future<Output = Result<MovePrefixPage>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
'life3: 'async_trait,
'life4: 'async_trait,
Move one page of the objects under from_prefix to to_prefix.
The default is crate::intent::fallback_move_prefix.
Returns StorageError::InvalidKey for overlapping prefixes,
StorageError::InvalidCursor for a cursor the backend no longer
accepts (start over with None), and listing errors.
fn delete_many<'life0, 'life1, 'async_trait>(
&'life0 self,
keys: &'life1 [String],
) -> Pin<Box<dyn Future<Output = Result<Vec<KeyResult>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Delete each key, one outcome per key in order. The default loops
Self::delete.
Returns an error only when the whole call fails.
fn translate_keys<'life0, 'life1, 'async_trait>(
&'life0 self,
keys: &'life1 [HostKey],
) -> Pin<Box<dyn Future<Output = Result<Vec<String>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Translates each host key into the key this backend stores it
under, one per key in order (crate::key_syntax).
The default stores a host key as written. The same host key must always translate to the same key, and two different host keys should translate to different keys.
Returns an error only when the whole call fails.
fn head_many<'life0, 'life1, 'async_trait>(
&'life0 self,
keys: &'life1 [String],
) -> Pin<Box<dyn Future<Output = Result<Vec<Option<ObjectProbe>>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Probe each key, one entry per key in order. The default loops
Self::head.
Returns the first probe failure other than absence.
fn put_stream_conditional<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
key: &'life1 str,
body: Pin<Box<dyn AsyncRead + Send>>,
meta: ObjectMeta,
condition: &'life2 WriteCondition,
) -> Pin<Box<dyn Future<Output = Result<ConditionalPutResult>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
Write body under key only when condition holds. The default
checks first and then writes, and reports that it was not atomic.
Returns StorageError::Conflict when the condition fails, and
backend errors otherwise.
fn list_hinted<'life0, 'life1, 'async_trait>(
&'life0 self,
request: &'life1 ListRequest,
) -> Pin<Box<dyn Future<Output = Result<HintedListPage>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
One listing page with hints. The default ignores the hints and says so in the reply.
Returns listing errors.
fn changes<'life0, 'life1, 'async_trait>(
&'life0 self,
cursor: Option<&'life1 str>,
limit: u32,
) -> Pin<Box<dyn Future<Output = Result<Option<ChangeFeedPage>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Changes since cursor from the backend's change feed. Ok(None)
means the backend has no feed and the caller relies on listings,
which is the default.
Returns StorageError::InvalidCursor for an expired cursor (start
over with a full listing) and feed errors.
fn delete<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Delete an object (no-op if missing).
fn head<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
) -> Pin<Box<dyn Future<Output = Result<Option<ObjectProbe>>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Probe metadata without downloading the body.
Default maps Self::probe absence onto Ok(None).
Propagates backend probe failures other than not-found.
fn list_page<'life0, 'life1, 'life2, 'async_trait>(
&'life0 self,
prefix: &'life1 str,
cursor: Option<&'life2 str>,
limit: u32,
) -> Pin<Box<dyn Future<Output = Result<ListPage>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
'life2: 'async_trait,
One page of keys under prefix.
cursor is opaque and scoped to this backend's ordering. A missing,
stale, or non-advancing cursor returns StorageError::InvalidCursor
and does not restart at the first page. limit is clamped to
crate::bounded::MAX_LIST_PAGE. Ordering and mutation behavior are
backend-specific; see docs/internals/storage-bounds.md.
Returns StorageError::InvalidCursor or a backend listing error.
fn get_stream<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
range: Option<ByteRange>,
) -> Pin<Box<dyn Future<Output = Result<(ObjectProbe, Pin<Box<dyn AsyncRead + Send>>)>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Streamed read. Never reassembles the object into host Bytes.
ObjectProbe::size is the whole object size, not the returned slice.
range.length == Some(0) is rejected. None means through EOF, matching
the wire encoding where length 0 is "to end".
Returns StorageError::NotFound when missing, StorageError::InvalidKey
for a bad range, and I/O or S3 failures otherwise.
fn put_stream<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
body: Pin<Box<dyn AsyncRead + Send>>,
meta: ObjectMeta,
) -> Pin<Box<dyn Future<Output = Result<PutStreamResult>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Streamed write. body ownership is transferred to the backend.
Returns I/O or S3 failures from the sink.
fn supports_server_copy(&self) -> bool
True when Self::copy stays inside this namespace without downloading.
The default is false. A backend opts in only for copies that are valid for its own bucket, root, and prefix.
fn touch_file<'life0, 'life1, 'async_trait>(
&'life0 self,
key: &'life1 str,
created: Option<SystemTime>,
modified: Option<SystemTime>,
) -> Pin<Box<dyn Future<Output = Result<()>> + Send + 'async_trait>>
where
Self: 'async_trait,
'life0: 'async_trait,
'life1: 'async_trait,
Set filesystem timestamps (local) or best-effort logical timestamp tags (S3).
Local backends update mtime/ctime. S3 backends must not CopyObject to
rewrite user-metadata (creates a second full-size version on versioned
buckets). System Last-Modified cannot be set on AWS S3; logical times
belong in PutObject x-amz-meta-* (S3 backends set them at upload only).
Implemented by: B2Backend, FanoutBackend, LocalFsBackend, S3Backend
pub struct B2ApplicationKey(/* private */);
An application key's secret half. Its Debug output never shows it.
pub fn new(secret: impl Into<String>) -> Self
Wraps the secret.
Implements: Clone, Debug
pub struct B2Backend {
/* private fields */
}
A bucket (and prefix) on Backblaze B2.
pub fn new(settings: B2Settings) -> Result<Self>
A backend for a bucket. Nothing is sent until the first call, which
signs in, finds the bucket, and checks the key may reach the prefix
(a key restricted to another bucket or prefix fails that call with
StorageError::InvalidKey).
Returns StorageError::InvalidKey for an empty key id or bucket,
or a prefix B2 cannot store, and StorageError::Remote when the
HTTP client cannot be built.
Implements: Clone, StorageBackend
pub struct B2Settings {
pub key_id: String,
pub application_key: B2ApplicationKey,
pub bucket: String,
pub prefix: String,
pub authorize_url: String,
}
How to reach one bucket.
Fields
key_id: The application key's id.application_key: The application key.bucket: The bucket's name.prefix: Prefix under which every key is stored (may be empty).authorize_url: Base URL ofb2_authorize_account(B2_AUTHORIZE_URL).
pub fn new(
key_id: impl Into<String>,
application_key: B2ApplicationKey,
bucket: impl Into<String>,
prefix: impl Into<String>,
) -> Self
Settings for bucket under prefix, signing in at
B2_AUTHORIZE_URL.
Implements: Clone, Debug
pub struct FanoutBackend {
/* private fields */
}
Multiplexes one logical storage key across multiple backends.
Mutations (put*, copy, rename, delete, touch_file) run on every
child. Reads try children in order and succeed on the first hit.
pub fn new(backends: Vec<Box<dyn StorageBackend>>) -> Result<Self>
Build a fan-out over backends (must be non-empty).
Returns an error when the operation fails.
Implements: StorageBackend
pub struct LocalFsBackend {
/* private fields */
}
Stores objects under a root directory; keys map to relative paths.
An optional key prefix is prepended to every key (same model as S3),
so library storage_key values stay relative to the prefix.
pub const CAPABILITIES: StorageCapabilities = _
What every local backend does natively: moves are renames, batches
run in one call, and conditional writes are checked here. A backend
built with Self::with_watch (feature watch) also has a change
feed.
pub fn new(root: PathBuf) -> Result<Self>
Create a backend rooted at root with no key prefix.
Returns an error when the operation fails.
pub fn with_prefix(root: PathBuf, prefix: &str) -> Result<Self>
Create a backend rooted at root with an optional key prefix
(e.g. library/). The prefix directory is created when needed.
Returns an error when the operation fails.
pub fn with_watch(self) -> Self
Starts a change feed for this backend's root (below its prefix),
served by StorageBackend::changes for as long as the backend or
a clone of it lives.
The watcher runs only on a local disk; on a network file system or a
FUSE mount (see crate::fs_kind), or when the operating system
refuses it, every page says the feed may have missed changes, so the
caller lists instead. Call it after Self::with_fs_kind for the
override to apply.
pub fn keeping_host_keys(self) -> Self
Stores host keys as written instead of translating them for this file system, and says it does not translate keys.
For a branch's outbox: the files it holds are uploaded to the owner under host keys, and each of the owner's destinations translates them there.
pub fn with_fs_kind(self, kind: FsKind) -> Self
Treats every path as living on a file system of kind, instead of
classifying it.
Tests use it to drive the network-mount paths on a local disk. Production backends leave it unset.
pub fn with_scan_placement(self, placement: impl Into<String>) -> Self
Overrides the host placement reported for scan adoption.
Production backends leave this unset and use crate::node_placement_id.
Implements: Clone, Debug, StorageBackend
pub struct CloudflareAccountId(/* private */);
A Cloudflare account id: 32 lowercase hex digits.
pub fn parse(id: &str) -> Result<Self>
Checks an account id.
Returns StorageError::S3 when id is not 32 hex digits.
pub fn as_str(&self) -> &str
The id as text.
Implements: Clone, Debug, Eq, PartialEq
pub struct CloudflareApiToken(/* private */);
A Cloudflare API token with Queues read and write on the account.
pub fn new(token: String) -> Result<Self>
Wraps a token.
Returns StorageError::S3 for an empty token or one holding
characters a header cannot carry.
Implements: Clone, Debug
pub struct CloudflareQueueId(/* private */);
A Cloudflare Queue id: 32 lowercase hex digits (the queue's id, not its name).
pub fn parse(id: &str) -> Result<Self>
Checks a queue id.
Returns StorageError::S3 when id is not 32 hex digits.
pub fn as_str(&self) -> &str
The id as text.
Implements: Clone, Debug, Eq, PartialEq
pub struct S3Backend {
/* private fields */
}
S3-compatible object storage.
pub async fn from_parts_with_journal(
cfg: &S3Settings,
prefix: &str,
creds: Option<&S3Credentials>,
journal_dir: &Path,
) -> Result<Self>
Build with explicit credentials and a granted multipart journal directory.
The directory must already be a location the process is allowed to
persist. For the destination guest that is its HOME, not the host
files directory. A journal that cannot be created or read fails
construction instead of claiming later recovery.
Returns an error when the client cannot be built, the journal directory cannot be created, or reading existing records fails.
pub fn with_part_size(self, part_size: usize) -> Self
Overrides the multipart part size. Values below 1 are ignored.
pub async fn from_parts(
cfg: &S3Settings,
prefix: &str,
creds: Option<&S3Credentials>,
) -> Result<Self>
Build with explicit settings and credentials.
Without Self::from_parts_with_journal, multipart uploads are not
recoverable after process death.
Returns an error when the operation fails.
pub fn with_notification_queue(self, queue: QueueUrl, state_dir: &Path) -> Result<Self>
Serves changes from the bucket's notification queue.
The bucket must send its object-created and object-removed events to
queue (directly, through SNS, or through an EventBridge rule), and
nothing else may drain the queue. The feed keeps the receipts of the
messages it has handed out in state_dir (the guest's HOME) until
the host's next call says it has read them.
Returns an error when the queue client cannot be built or
state_dir cannot be created.
pub fn with_cloudflare_queue(
self,
account: &CloudflareAccountId,
queue: &CloudflareQueueId,
token: CloudflareApiToken,
state_dir: &Path,
) -> Result<Self>
Serves changes from an R2 bucket's event notifications, pulled
from a Cloudflare Queue.
The bucket's event notification rule must send object-create and
object-delete events to queue, HTTP pull must be on for it, and
nothing else may consume it. Like the SQS feed, the feed keeps the
lease ids it has handed out in state_dir (the guest's HOME) until
the host's next call says it has read them.
account- the Cloudflare account the bucket and queue belong to.queue- the queue's id.token- an API token with Queues read and write on the account.state_dir- where the feed keeps its state.
Returns an error when the queue client cannot be built or
state_dir cannot be created.
pub fn has_notification_queue(&self) -> bool
True when changes is served from a notification queue.
pub const CAPABILITIES: StorageCapabilities = _
What every S3 backend can do natively once the store has shown it
enforces conditions; each operation falls back to the host's
algorithm on a store that does not (see Self::conditional_support).
pub async fn conditional_support(&self) -> ConditionalSupport
Which conditional requests this store enforces, probed on first use and remembered for the backend's lifetime.
Each request kind is probed on its own, because a store can enforce a
condition on one and ignore it on another. Under .bookclerk-probe/
the probe writes a scratch object with If-None-Match: * twice
(the second must be refused) and overwrites it with a wrong
If-Match (refused); completes a one-part multipart upload onto it
with each condition (both refused) and aborts the upload; copies it
onto a second existing object with a wrong
x-amz-copy-source-if-match (refused) and with If-None-Match: *
(refused); copies it as an upload part with a wrong
x-amz-copy-source-if-match (refused) and aborts that upload; and
deletes it with a wrong If-Match (refused). It then removes both
objects. Any other answer, including an error, counts as "not
enforced", so a store is never trusted by default.
pub async fn retry_orphans(&self) -> Result<usize>
Aborts multipart uploads recorded for this endpoint, bucket, and prefix whose owner lock is not held.
Live uploads are left alone. Failed aborts stay on disk and stop being retried after 8 failed attempts.
Returns I/O errors reading the journal directory.
Implements: Clone, Debug, StorageBackend
pub struct S3Credentials {
pub access_key_id: String,
pub secret_access_key: String,
pub session_token: Option<String>,
}
Static AWS-style credentials for the S3 destination.
These come from the instance's sealed secrets; without them the SDK default provider chain applies inside the guest.
Fields
access_key_id: AWS access key id.secret_access_key: AWS secret access key (redacted inDebug; never logged).session_token: Optional STS session token when using temporary credentials.
Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct S3Settings {
pub bucket: String,
pub region: String,
pub endpoint: Option<String>,
pub force_path_style: bool,
}
Connection settings for one S3 / MinIO destination.
The s3 destination plugin declares these as [[settings]]; the host
resolves them from the plugin instance document and hands them to the
guest as its open context.
Fields
bucket: Destination bucket name (required).region: AWS region id (also used by MinIO path-style clients).endpoint: Optional custom endpoint (MinIO, LocalStack, …).force_path_style: Force path-style addressing (typical for MinIO).
Implements: Clone, Debug, Default, Eq, PartialEq
pub struct QueueUrl {
/* private fields */
}
An Amazon SQS queue URL, checked at the boundary.
Accepted: https://sqs.<region>.amazonaws.com/<account>/<name> (also
.amazonaws.com.cn) and the legacy
https://<region>.queue.amazonaws.com/<account>/<name>, where the
account is 12 digits and the name 1 to 80 letters, digits, - or _,
counting an optional .fifo suffix.
pub fn parse(url: &str) -> Result<Self>
Parses and checks a queue URL.
Returns StorageError::S3 naming what is wrong with the URL.
pub fn as_str(&self) -> &str
The URL as given.
pub fn region(&self) -> &str
The AWS region the queue lives in.
Implements: Clone, Debug, Eq, PartialEq
pub struct ByteRange {
pub offset: u64,
pub length: Option<u64>,
}
Inclusive byte range for a streamed read.
Fields
offset: Starting offset.length: Number of bytes;Nonemeans to end of object.
Implements: Clone, Copy, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct ListPage {
pub objects: Vec<ObjectInfo>,
pub next_cursor: Option<String>,
}
One page of ObjectInfo from StorageBackend::list_page.
Fields
objects: Objects in this page.next_cursor: Continuation token;Nonewhen this is the last page.
Implements: Clone, Debug, Default, Deserialize<'de>, Serialize
pub struct ObjectInfo {
pub key: String,
pub size: u64,
pub etag: Option<String>,
pub modified_at_ms: Option<u64>,
pub version_id: Option<String>,
pub signature: Option<String>,
}
Listing entry returned by StorageBackend::list.
The signature fields are filled only by a listing asked for them
(crate::ListRequest::with_signatures) on a backend that has them.
Fields
key: Relative storage key (prefix + path; no leading slash).size: Object size in bytes.etag: Backend entity tag. Opaque; not a content checksum.modified_at_ms: Last modification time in UTC unix milliseconds, when known.version_id: Backend object version id (versioned buckets), when there is one.signature: Opaque change signature: an unchanged value means the object did not change. Compare only values from the same backend.
Implements: Clone, Debug, Default, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct ObjectMeta {
pub content_type: Option<String>,
pub content_length: Option<u64>,
pub asin: Option<String>,
pub title: Option<String>,
pub creation_time: Option<String>,
pub last_write_time: Option<String>,
pub sha256_hex: Option<String>,
pub commit_token: Option<String>,
}
Metadata attached to a stored object.
Fields
-
content_type: MIME type stored with the object (e.g.audio/mp4). -
content_length: Object size in bytes when known at put time. -
asin: Free-form ASIN / title tags for S3 object metadata. -
title: Display title stored as object user-metadata when supported. -
creation_time: Creation timestamp as RFC 3339 (S3 metadatacreation-time). -
last_write_time: Last-write timestamp as RFC 3339 (S3 metadatalast-write-time). -
sha256_hex: Lowercase hex SHA-256 (64 chars) of the object body when known.This is a content digest. It is never derived from an S3 ETag.
-
commit_token: Retry-stable publication token stored with the object, when staged.
Implements: Clone, Debug, Default, Deserialize<'de>, Serialize
pub struct ObjectProbe {
pub key: String,
pub size: u64,
pub content_type: Option<String>,
pub meta: ObjectMeta,
pub etag: Option<String>,
pub modified_at_ms: Option<u64>,
pub version_id: Option<String>,
pub signature: Option<String>,
}
Cheap object probe (S3 HeadObject / local sidecar meta) — never downloads
object bodies.
Fields
key: Relative storage key that was probed.size: Object size in bytes from HeadObject / local metadata.content_type: MIME type when the backend exposes it.meta: User-metadata / sidecar fields (ASIN, title, timestamps, …).etag: Backend entity tag. Opaque; not a content checksum.modified_at_ms: Last modification time in UTC unix milliseconds, when known.version_id: Backend object version id (versioned buckets), when there is one.signature: Opaque change signature, asObjectInfo::signature.
Implements: Clone, Debug, Default, Deserialize<'de>, Serialize
pub struct PutStreamResult {
pub bytes_written: u64,
pub etag: Option<String>,
pub sha256_hex: Option<String>,
}
Result of StorageBackend::put_stream.
Fields
bytes_written: Bytes accepted from the body stream.etag: Backend etag when available. Opaque; not a content checksum.sha256_hex: Hex SHA-256 of the bytes accepted, when the backend computed one.
Implements: Clone, Debug, Default, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct TransferOptions {
pub max_attempts: u32,
pub backoff: Duration,
pub attempt_timeout: Option<Duration>,
pub stage_journal_dir: Option<PathBuf>,
}
Knobs for transfer_object.
Fields
max_attempts: Total attempts including the first. Zero is treated as one.backoff: Base backoff between attempts. Attemptnwaitsbackoff * n.attempt_timeout: Optional wall-clock budget for a single attempt.stage_journal_dir: Directory that records attempt-owned stages so a restarted process can delete them.Nonestill deletes on drop while this process is alive.
Implements: Clone, Debug, Default
pub enum StorageError {
NotFound(String),
InvalidKey(String),
Io(Error),
S3(String),
Remote(String),
PayloadTooLarge(String),
Integrity(String),
Conflict(String),
InvalidCursor(String),
Other(Error),
}
Errors from local / S3 / fan-out storage backends.
Variants
NotFound: Object key was not found on get/probe/copy source.InvalidKey: Key failed validation (path escape, empty bucket, bad destination config).Io: Local filesystem I/O failure.S3: S3 SDK or HTTP failure (operator-facing detail).Remote: An object store's API refused or failed a request (operator-facing detail, never credentials).PayloadTooLarge: Scalar get/put exceeded the ABI small-object cap.Integrity: Declared length or SHA-256 did not match the bytes consumed.Conflict: A precondition failed: the target exists, or the object changed since it was read.InvalidCursor: List cursor is missing, stale, or not from this backend.Other: Catch-all for wrapped backend failures.
Implements: Debug, Display, Error, From<Error>
pub enum TransferOutcome {
ServerCopy,
Streamed(PutStreamResult),
AlreadyPublished,
}
How transfer_object moved the object.
Variants
ServerCopy: Destinationcopyinside one storage instance.Streamed: Streamed body.result.sha256_hexis the digest of bytes written.AlreadyPublished: Destination probe already matched the source size and digest.
Implements: Clone, Debug, Eq, PartialEq
pub fn ensure_scalar_len(len: usize, limit: u64) -> Result<()>
Rejects a scalar put whose buffer already exceeds limit.
Returns StorageError::PayloadTooLarge when len is above limit.
pub fn normalize_range(
range: Option<ByteRange>,
object_size: Option<u64>,
) -> Result<Option<ReadSpan>>
Normalizes an internal range.
None reads the whole object. ByteRange::length None reads through
EOF (the Cap'n Proto wire uses integer 0 for that case and decodes it to
None before it reaches a backend). Some(0) is rejected so local and S3
cannot disagree about an empty window versus "to end".
object_size, when known, is the whole object. Offsets past that size, a
window that would extend past it, and offset + length overflow are errors.
Returns StorageError::InvalidKey for an ambiguous, overflowing, or
unsatisfiable range.
pub fn parse_sha256_hex(value: &str) -> Result<[u8; 32]>
Parses a hex SHA-256 (64 hex chars) into raw bytes.
Returns StorageError::Integrity when the shape is wrong.
pub async fn read_scalar_body(
reader: Pin<Box<dyn AsyncRead + Send>>,
limit: u64,
) -> Result<Bytes>
Reads at most limit + 1 bytes.
A body longer than limit returns StorageError::PayloadTooLarge and
stops. The allocation is capped at limit + 1 regardless of any advertised
content length.
Returns StorageError::PayloadTooLarge or StorageError::Io.
pub fn reject_scalar_hint(hint: u64, limit: u64) -> Result<()>
Early-reject when a HEAD/stat hint already exceeds limit.
This does not allocate hint bytes. The body must still be counted.
Returns StorageError::PayloadTooLarge when hint is above limit.
pub fn sha256_field_from_raw(bytes: Option<&[u8]>) -> Result<Option<String>>
Validates a raw SHA-256 field (ABI Data, 32 bytes) into hex.
Empty means "not provided". Any other length is an integrity error.
Returns StorageError::Integrity when the digest is not 32 bytes.
pub fn node_placement_id() -> String
This node's id, or a process-lifetime id when the node identity cannot be read.
BOOKCLERK_PLACEMENT_ID overrides it for tests. Otherwise the value is
the node_id in $BOOKCLERK_FILES_DIR/node-identity.json, which the
control plane writes when the node first opens the library. When that is
not available the id lasts for this process only, so a scan is not
adopted by another process that cannot prove it is the same node.
pub fn bookclerk_meta_sidecar_key(object_key: &str) -> String
Sidecar key for probe metadata of this exact object ({key}.bookclerk-meta.json).
The record is not shared with another object that only shares a title stem.
book.m4b and book.jpg keep separate integrity and commit metadata.
pub fn is_audio_key(key: &str) -> bool
True when key ends with a known acquired audio extension.
pub async fn transfer_object(
source: &dyn StorageBackend,
source_key: &str,
dest: &dyn StorageBackend,
dest_key: &str,
meta: ObjectMeta,
options: &TransferOptions,
) -> Result<TransferOutcome>
Copies source_key onto dest_key.
meta supplies content type and any caller-provided digest. A digest on
the source probe is required to match bytes actually written. When the
source probe has no digest, the destination records the digest computed
while streaming. An S3 ETag is an identity hint for "the object changed",
not a content checksum, and is never copied into sha256_hex.
Returns StorageError::Integrity when the source changes between
attempts or a digest/length check fails, StorageError::NotFound when
the source is missing, and the destination error after retries are exhausted.
pub type Result<T> = Result<T, StorageError>;
Result alias for storage backend operations in this crate.
pub const B2_AUTHORIZE_URL: &str = "https://api.backblazeb2.com";
Where b2_authorize_account lives; every other URL comes from its
reply.
pub const MAX_LIST_PAGE: u32 = bookclerk_plugin_sdk::MAX_LIST_PAGE;
ABI list page ceiling.
pub const MAX_SCALAR_OBJECT_BYTES: u64 = _;
ABI scalar ceiling. Audiobook bodies must not use crate::StorageBackend::get /
crate::StorageBackend::put.
pub const MAX_SUPPORTED_OBJECT_BYTES: u64 = _;
Largest object this process will upload or server-copy.
S3 allows 10,000 parts. Each part is 8 MiB, so the application ceiling is 80 GiB. AWS's own 5 TiB object maximum is wider; requests above this ceiling fail before any part buffer is filled.
pub const AUDIO_EXTENSIONS: &[&str] = _;
Audio extensions considered acquired media for storage matching.
Includes default remux/encode outputs (m4b / mp3 / m4a / opus) plus plain
passthrough containers Chirp / GraphicAudio may store under noop/as_is
output (flac / aac / ogg / oga). Keep aligned with
bookclerk_source::media sniffing and GraphicAudio ZIP audio filters.
Generated from crates/bookclerk-storage at main · View as Markdown