bookclerk_intake
Drop folder items and when they have settled, without IO.
A drop folder is a storage plugin instance with the intake role (see
docs/server/drop-folders.md). The host lists it page by page and feeds
every listed object to an ItemGrouper, which:
- drops objects that are never part of an item: names starting with
., partial downloads and temporaries, the host's own_importedand_failedfolders, and keys that are not a clean relative path; - groups the rest into items, one per top-level entry (one level
deeper in
subfoldermode, where the top level names the patron); - gives each item a
ItemSignatureover its members' keys, sizes and change signatures, independent of listing order.
step then compares each item with what the host stored for it last
time and says what to store now, and whether the item is ready to
ingest: an item is ready once its signature has stayed the same for the
folder's quiet period. No portable way exists to see whether another
program still has a file open, so an unchanged signature over time is
the only test of a finished copy.
Keys come from the folder, which anyone with access to it can fill, so
this crate is an untrusted-input boundary: the intake_items fuzz
target checks it against an independent reading.
| Item | Kind | Summary |
|---|---|---|
ListedObject |
struct | One object as a drop folder listing reports it. |
ItemMember |
struct | A member file of an item. |
ItemSignature |
struct | A digest of an item's members, independent of listing order. |
ObservedItem |
struct | One item as a whole listing shows it. |
ItemGrouper |
struct | Groups a drop folder's listing into items. |
TrackedItem |
struct | One item's stored record. |
Placement |
enum | Where the grouper put one listed object. |
ItemLimit |
enum | Why an item can never become ready as listed. |
TrackedState |
enum | What the host stored about one item. |
Step |
enum | What to do with one item's record after a listing. |
signature_of |
function | The order-independent digest of sorted members: each member's key, size and signature, length-prefixed so no two lists collide by concatenation, then a marker when the item is over a limit. An item within every limit hashes exactly as before the marker existed. |
step |
function | Compares one item's stored record with the latest full listing. |
MAX_KEY_BYTES |
constant | Longest object key the grouper accepts, in UTF-8 bytes (the largest object-store key). |
MAX_ITEM_MEMBERS |
constant | Most member files one item may hold. An item with more is reported as ItemLimit::TooManyMembers and never becomes ready. |
pub struct ListedObject<'a> {
pub key: &'a str,
pub size: u64,
pub signature: &'a str,
}
One object as a drop folder listing reports it.
Fields
key: Object key relative to the drop folder's root,/-separated.size: Size in bytes.signature: The backend's change signature (ObjectInfo.signature, else its ETag or modification time); empty when the backend has none, in which case only the size can show a change.
Implements: Clone, Copy, Debug, Eq, PartialEq
pub struct ItemMember {
pub key: String,
pub size: u64,
pub signature: String,
}
A member file of an item.
Fields
key: Object key relative to the drop folder's root.size: Size in bytes.signature: The backend's change signature, possibly empty.
Implements: Clone, Debug, Eq, Ord, PartialEq, PartialOrd
pub struct ItemSignature(/* private */);
A digest of an item's members, independent of listing order.
pub fn to_hex(self) -> String
The signature as 64 lowercase hex characters, as the host stores it.
pub fn from_hex(text: &str) -> Option<Self>
Reads a stored signature; None for anything but 64 hex characters.
Implements: Clone, Copy, Debug, Eq, Hash, Ord, PartialEq, PartialOrd
pub struct ObservedItem {
pub key: String,
pub patron_folder: Option<String>,
pub members: Vec<ItemMember>,
pub total_bytes: u64,
pub signature: ItemSignature,
pub limit: Option<ItemLimit>,
}
One item as a whole listing shows it.
Fields
key: The item's key: its top-level name, or<patron folder>/<name>insubfoldermode.patron_folder: The patron folder's name insubfoldermode;Noneinownermode.members: Member files sorted by key, at mostMAX_ITEM_MEMBERS.total_bytes: Sum of the members' sizes, saturating.signature: Digest of every member's key, size and signature, and of whether the item went pastMAX_ITEM_MEMBERS(so a member added past the cap still changes it).limit: Set when the item can never become ready as listed.
Implements: Clone, Debug, Eq, PartialEq
pub struct ItemGrouper<'s> {
/* private fields */
}
Groups a drop folder's listing into items.
pub fn new(settings: &'s IntakeSettings) -> Self
An empty grouper for one listing of a folder with settings.
pub fn add(&mut self, object: ListedObject<'_>) -> Placement
Adds one listed object and says where it went.
A key listed twice counts once, with the size and signature of the last listing.
pub fn finish(self) -> Vec<ObservedItem>
Every item seen, sorted by key.
Implements: Debug
pub struct TrackedItem {
pub state: TrackedState,
pub signature: ItemSignature,
pub quiet_since_ms: u64,
}
One item's stored record.
Fields
state: The item's state.signature: The signature the state applies to.quiet_since_ms: When the signature was first seen unchanged, in unix milliseconds.
Implements: Clone, Copy, Debug, Eq, PartialEq
pub enum Placement {
Member(String),
Ignored,
Stray,
}
Where the grouper put one listed object.
Variants
Member: A member of the item with this key.Ignored: Never part of an item: an ignored name, a host folder, or a key that is not a clean relative path.Stray: A file directly under the root insubfoldermode, which belongs to no patron folder; it is left alone.
Implements: Clone, Debug, Eq, PartialEq
pub enum ItemLimit {
TooManyMembers,
}
Why an item can never become ready as listed.
Variants
TooManyMembers: The item holds more thanMAX_ITEM_MEMBERSfiles.
Implements: Clone, Copy, Debug, Eq, PartialEq
pub enum TrackedState {
Settling,
Queued,
Done,
Failed,
}
What the host stored about one item.
Variants
Settling: Seen, not yet quiet for long enough.Queued: Handed to an ingest job that has not finished.Done: Ingested and left in place (after_ingest = leave); ingested again only if it changes.Failed: Ingest failed; tried again only if it changes.
Implements: Clone, Copy, Debug, Eq, PartialEq
pub enum Step {
Keep,
Store(TrackedItem),
Ready(TrackedItem),
Forget,
}
What to do with one item's record after a listing.
Variants
Keep: Nothing changed.Store: Store this record (new or changed).Ready: The item has settled: store thisQueuedrecord and queue one ingest job for it.Forget: The item is gone: delete its record.
Implements: Clone, Copy, Debug, Eq, PartialEq
pub fn signature_of(members: &[ItemMember], limit: Option<ItemLimit>) -> ItemSignature
The order-independent digest of sorted members: each member's key,
size and signature, length-prefixed so no two lists collide by
concatenation, then a marker when the item is over a limit. An item
within every limit hashes exactly as before the marker existed.
members- The members, sorted by key.limit- The limit the item went past, if any.
pub fn step(
tracked: Option<&TrackedItem>,
observed: Option<&ObservedItem>,
now_ms: u64,
settle_ms: u64,
) -> Step
Compares one item's stored record with the latest full listing.
tracked- The stored record, if any.observed- The item as the listing shows it;Nonewhen the listing no longer has it.now_ms- The current time, unix milliseconds.settle_ms- The folder's quiet period, milliseconds.
What to store. A queued item keeps its record until the ingest job finishes, even if its files changed or vanished: the job compares the signature before it reads and reports the item failed if it moved.
pub const MAX_KEY_BYTES: usize = 1024;
Longest object key the grouper accepts, in UTF-8 bytes (the largest object-store key).
pub const MAX_ITEM_MEMBERS: usize = 4096;
Most member files one item may hold. An item with more is reported as
ItemLimit::TooManyMembers and never becomes ready.
Generated from crates/bookclerk-intake at main · View as Markdown