ADR: Drop folders are a host role on storage instances
- Status: Accepted (decisions made with the maintainer on 2026-11-20)
- Date: 2026-11-20
- Related: Storage destinations, Metadata claims and file sync, Event delivery model, Untrusted input fuzzing
Patrons want to copy books into a folder and have them show up in the library, without the web app. Uploading a book in the web app is being built at the same time, with one ingest job that matches, tags and stores what it is given.
Storage plugins already list, read, move and delete files, and some report outside changes (the local plugin's watcher, the S3 plugin's queue feed). Every deployed storage instance is a destination (Storage destinations), so a watched folder must not also receive acquired books, or Bookclerk would ingest its own writes.
A storage instance is a destination unless its core.intake document, a
host document on the instance's scope, holds intake settings. Then it is a
drop folder. No new plugin, entrypoint or ABI field is added: any storage
plugin, first-party or not, can back a drop folder.
The role is read when the instance's guest starts, and a drop folder is never handed to an acquire. So the role changes only while the instance is deployed nowhere; the host refuses the change otherwise. A drop folder's settings may change at any time.
The daemon on each node that runs a drop folder lists it, groups the
listing into items (each top-level file or folder), and records each
item's state in intake_items. An item whose files keep the same sizes and
signatures for settle_seconds is handed to the ingest job as one batch of
storage references into the folder. The job copies each file and fails
one that no longer has the size that settled, or whose resumed read comes
from another version; the runner moves or deletes the item afterwards only
if its signature has not changed. The runner stores the batch id on the
item before it creates the batch, so after a crash it finds the batch, or
knows it was never created and lets the item settle again: an item is
never handed to two batches.
Grouping and settling are sans-IO code in bookclerk-intake, fuzzed
against an independent oracle, because the listing is untrusted input.
A full listing every scan_interval_seconds works for every plugin,
including one that sleeps and a share that sends no events. A change feed
only starts a pass sooner. Claiming the work is the ingest job's, never an
event handler's (Event delivery model).
By default an added item moves to _imported/<date>/ and a failed one to
_failed/ with a note saying why. delete and leave are settings; with
leave the stored signature keeps an item from being added twice. Moves
never replace an existing file.
Each dropped book is held by a patron: one named patron for the folder, or one per top-level subfolder named after the patron's username. A book whose files carry no strong identifier is still added and playable and waits for confirmation; it is never matched by title alone.
- An operator turns a destination into a drop folder in three steps (undeploy, set, deploy), and the CLI and web API refuse the role change while it runs.
- A slow copy that pauses longer than
settle_secondscan be picked up early; the job's signature check then fails it and it settles again, and operators can raisesettle_secondsfor slow shares. - Drop folders and uploads share one ingest path, so matching, tagging and storage rules apply to both without special cases.
bookclerk-intakeand the settings reader are fuzzed boundaries listed indocs/contributing/fuzzing.md.
Source: docs/adr/drop-folders.md at main · View as Markdown