Drop folders

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

A drop folder is a folder Bookclerk watches. Copy a book into it, and once the files stop changing, Bookclerk adds the book to the library and moves the files out of the way. It is the hands-off twin of uploading a book in the web app: both go through the same ingest job, so a dropped book is matched, tagged and stored exactly like an uploaded one.

Any storage plugin instance can be a drop folder: a local folder (local), a mounted share, or an S3 prefix (s3). A drop folder is never a destination: acquired books are not written into it.

How it works

  1. You copy files into the folder. Each top-level entry is one item: a single file (Dune.m4b) or a folder of files (Dune/01.mp3, Dune/02.mp3, Dune/cover.jpg).
  2. Bookclerk lists the folder every scan_interval_seconds (5 minutes by default), and sooner when the storage plugin reports a change (the local plugin's watcher, or the S3 plugin's queue feed).
  3. An item is ready once none of its files has changed for settle_seconds (one minute by default). A file still being copied keeps changing size or modification time, so it keeps waiting. Partial downloads (*.part, *.crdownload, *.tmp and similar) are ignored.
  4. Bookclerk hands the ready item to the ingest job as one batch, the same job an upload runs. The job copies the files from the folder, checks that they still have the size that settled, and adds the book. A read that breaks partway, as a network share or S3 prefix can, picks up where it stopped, up to three times, before that file fails. A resumed read must come from the same version of the file (its version id or ETag); a file replaced in between fails as intake_changed, and on storage that reports no version the read starts over instead. Files it does not keep, such as a stray .exe, are refused on their own without failing the book.
  5. When the batch ends, the item is moved to _imported/<date>/ (the default, with the date in UTC), deleted, or left where it is. An item that could not be added, or with a file that failed to store, goes to _failed/, with a <item>.bookclerk-error.txt note beside it saying why (for example intake_changed when a file was still growing). A note never replaces a file already there; the next free name (<item>.bookclerk-error-2.txt, …) is used. An item whose files changed while it was being added stays where it is and settles again.

Moves never replace a file already at the target, and where the storage reports an ETag, a move or delete touches only the version that was listed. If one file of an item cannot be moved or deleted, the files already moved stay moved, the rest stay in place, and Bookclerk records why; the files left behind are not added again unless they change.

If the job could not be queued (for example, the queue is full) or Bookclerk stopped before the batch was created, the item's files are left alone and it settles again, then is handed off once more. An item is never handed to two batches at once.

Note

Bookclerk never matches a dropped book by its title alone. A book whose files carry no ASIN, ISBN or other strong identifier is still added and playable, as needs_review, and waits for an Administrator to confirm which edition it is (see Review uploaded books).

Files and folders whose names start with a dot, and the _imported and _failed folders at the top of the drop folder, are always skipped.

Who holds a dropped book

Every dropped book gets a patron holding, as an uploaded book does. The folder's attribute_to setting says whose:

  • owner (the default): one patron, named by user id with --patron, holds everything the folder adds.
  • subfolder: each top-level folder is a patron's username, and holds that patron's books (alice/Dune.m4b, bob/Emma/01.mp3). An item under a name that matches nobody waits until an Administrator creates that patron or moves the files.

Set up a drop folder

A storage instance's role is read when its plugin starts, so change it while the instance is deployed nowhere.

  1. Create a storage instance for the folder, for example a second local instance:

    bookclerk plugins instances create local --name "Drop folder"
    bookclerk plugins config set <instance-id> root /srv/audiobooks/drop
    
  2. Make it a drop folder. --patron takes the user id of the patron who holds what it adds:

    bookclerk plugins instances intake set <instance-id> --patron 1
    

    The patron must be an active user.

    Or, for one subfolder per patron:

    bookclerk plugins instances intake set <instance-id> --subfolder
    
  3. Deploy it on the node that can see the folder:

    bookclerk plugins deploy <instance-id> --node <node-id>
    

To check the settings, run bookclerk plugins instances intake get <instance-id>. To turn the drop folder back into a destination, undeploy it, then run bookclerk plugins instances intake clear <instance-id>.

The web API has the same three operations on /api/config/plugin-instances/{id}/intake (GET, PUT with the settings, DELETE). They are operator routes: the operator token, or an Owner who has elevated, can call them (see Owners and elevation).

Settings

The settings of a drop folder that is already one can change at any time, deployed or not; Bookclerk reads them on every pass. Options left out of intake set keep their current value. intake set applies only if nobody changed the settings since it read them; otherwise it stops and asks you to run it again.

Setting CLI option Default Meaning
settle_seconds --settle-seconds 60 How long an item must stay unchanged before it is added (5 to 86400). Raise it for slow network copies.
scan_interval_seconds --scan-interval-seconds 300 Time between two full listings (30 to 86400). Change events only make a pass come sooner.
after_ingest --after-ingest move move to _imported/<date>/, delete, or leave. With leave, Bookclerk remembers the item and adds it again only if its files change.
attribute_to --patron / --subfolder owner Who holds the books; see Who holds a dropped book.
patron --patron none The holding patron's user id. Required for owner, refused for subfolder.
ignore --ignore (repeatable), --clear-ignore none Extra file name patterns to skip, with * and ?, matched without regard to case (up to 64). --clear-ignore removes them all. Written out, the settings may take at most 12 KiB, which only patterns full of " come near.
prune_imported_days --prune-imported-days, --no-prune off With move, delete _imported/ day folders older than this many days (1 to 3650). Checked at most once an hour. --no-prune turns it off, and can go with --after-ingest leave in the same command.

Note

delete removes the files from the drop folder only after the book is stored in every destination. Keep move if the drop folder is your only copy until you have checked the result.

Next steps

Source: docs/server/drop-folders.md at main · View as Markdown