Drop folders
A drop folder is a folder Bookclerk watches. Copy a book into it, and once the files stop changing, Bookclerk adds the book to the library and moves the files out of the way. It is the hands-off twin of uploading a book in the web app: both go through the same ingest job, so a dropped book is matched, tagged and stored exactly like an uploaded one.
Any storage plugin instance can be a drop folder: a local folder (local),
a mounted share, or an S3 prefix (s3). A drop folder is never a
destination: acquired books are not written into it.
- You copy files into the folder. Each top-level entry is one item: a
single file (
Dune.m4b) or a folder of files (Dune/01.mp3,Dune/02.mp3,Dune/cover.jpg). - Bookclerk lists the folder every
scan_interval_seconds(5 minutes by default), and sooner when the storage plugin reports a change (the local plugin's watcher, or the S3 plugin's queue feed). - An item is ready once none of its files has changed for
settle_seconds(one minute by default). A file still being copied keeps changing size or modification time, so it keeps waiting. Partial downloads (*.part,*.crdownload,*.tmpand similar) are ignored. - Bookclerk hands the ready item to the ingest job as one batch, the
same job an upload runs. The job copies the files from the
folder, checks that they still have the size that settled, and adds the
book. A read that breaks partway, as a network share or S3 prefix can,
picks up where it stopped, up to three times, before that file fails.
A resumed read must come from the same version of the file (its version
id or ETag); a file replaced in between fails as
intake_changed, and on storage that reports no version the read starts over instead. Files it does not keep, such as a stray.exe, are refused on their own without failing the book. - When the batch ends, the item is moved to
_imported/<date>/(the default, with the date in UTC), deleted, or left where it is. An item that could not be added, or with a file that failed to store, goes to_failed/, with a<item>.bookclerk-error.txtnote beside it saying why (for exampleintake_changedwhen a file was still growing). A note never replaces a file already there; the next free name (<item>.bookclerk-error-2.txt, …) is used. An item whose files changed while it was being added stays where it is and settles again.
Moves never replace a file already at the target, and where the storage reports an ETag, a move or delete touches only the version that was listed. If one file of an item cannot be moved or deleted, the files already moved stay moved, the rest stay in place, and Bookclerk records why; the files left behind are not added again unless they change.
If the job could not be queued (for example, the queue is full) or Bookclerk stopped before the batch was created, the item's files are left alone and it settles again, then is handed off once more. An item is never handed to two batches at once.
Note
Bookclerk never matches a dropped book by its title alone. A book whose
files carry no ASIN, ISBN or other strong identifier is still added and
playable, as needs_review, and waits for an Administrator to confirm
which edition it is (see
Review uploaded books).
Files and folders whose names start with a dot, and the _imported and
_failed folders at the top of the drop folder, are always skipped.
Every dropped book gets a patron holding, as an uploaded book does. The
folder's attribute_to setting says whose:
owner(the default): one patron, named by user id with--patron, holds everything the folder adds.subfolder: each top-level folder is a patron's username, and holds that patron's books (alice/Dune.m4b,bob/Emma/01.mp3). An item under a name that matches nobody waits until an Administrator creates that patron or moves the files.
A storage instance's role is read when its plugin starts, so change it while the instance is deployed nowhere.
-
Create a storage instance for the folder, for example a second
localinstance:bookclerk plugins instances create local --name "Drop folder" bookclerk plugins config set <instance-id> root /srv/audiobooks/drop -
Make it a drop folder.
--patrontakes the user id of the patron who holds what it adds:bookclerk plugins instances intake set <instance-id> --patron 1The patron must be an active user.
Or, for one subfolder per patron:
bookclerk plugins instances intake set <instance-id> --subfolder -
Deploy it on the node that can see the folder:
bookclerk plugins deploy <instance-id> --node <node-id>
To check the settings, run bookclerk plugins instances intake get <instance-id>. To turn the drop folder back into a destination, undeploy
it, then run bookclerk plugins instances intake clear <instance-id>.
The web API has the same three operations on
/api/config/plugin-instances/{id}/intake (GET, PUT with the settings,
DELETE). They are operator routes: the operator token, or an Owner who
has elevated, can call them (see
Owners and elevation).
The settings of a drop folder that is already one can change at any time,
deployed or not; Bookclerk reads them on every pass. Options left out of
intake set keep their current value. intake set applies only if nobody
changed the settings since it read them; otherwise it stops and asks you
to run it again.
| Setting | CLI option | Default | Meaning |
|---|---|---|---|
settle_seconds |
--settle-seconds |
60 |
How long an item must stay unchanged before it is added (5 to 86400). Raise it for slow network copies. |
scan_interval_seconds |
--scan-interval-seconds |
300 |
Time between two full listings (30 to 86400). Change events only make a pass come sooner. |
after_ingest |
--after-ingest |
move |
move to _imported/<date>/, delete, or leave. With leave, Bookclerk remembers the item and adds it again only if its files change. |
attribute_to |
--patron / --subfolder |
owner |
Who holds the books; see Who holds a dropped book. |
patron |
--patron |
none | The holding patron's user id. Required for owner, refused for subfolder. |
ignore |
--ignore (repeatable), --clear-ignore |
none | Extra file name patterns to skip, with * and ?, matched without regard to case (up to 64). --clear-ignore removes them all. Written out, the settings may take at most 12 KiB, which only patterns full of " come near. |
prune_imported_days |
--prune-imported-days, --no-prune |
off | With move, delete _imported/ day folders older than this many days (1 to 3650). Checked at most once an hour. --no-prune turns it off, and can go with --after-ingest leave in the same command. |
Note
delete removes the files from the drop folder only after the book is
stored in every destination. Keep move if the drop folder is your only
copy until you have checked the result.
- Output and destinations reference: where added books are stored.
- Managing plugins: create, configure and deploy plugin instances.
Source: docs/server/drop-folders.md at main · View as Markdown