# Bounded storage

Audiobook bytes and storage inventories must not determine process memory.
Scalar get/put are for objects at or below the ABI scalar cap (256 KiB,
`MAX_SCALAR_BYTES`, or a lower negotiated limit). Larger bodies use
`get_stream` / `put_stream`, a same-namespace server copy, or an explicitly
quota-managed scratch file.

This is the contract for #120. It does not change playback, and it does not
make destination publication exactly-once. See [jobs.md](/docs/internals/jobs/index.md).

## Limits

| Boundary | Cap | Notes |
| --- | --- | --- |
| Scalar get/put | 256 KiB (`MAX_SCALAR_BYTES`) | HEAD/content-length is an early hint. The body is counted. `limit+1` is `PayloadTooLarge`, not a short success. The advertised size is not allocated. |
| Stream window | 1 MiB | ABI `ByteSource` pull. Unchanged. |
| List page | 256 objects | `limit` is clamped. Oversized pages are errors. |
| Checkpoint JSON | 64 KiB | Cursor and identity only. Per-object rows live in `storage_scan_rows`. |
| Application object size | 80 GiB | `8 MiB * 10,000` S3 parts. Above this, copy and upload fail before filling another part buffer. AWS allows larger objects; this process does not. |
| Single `CopyObject` | 5 GiB inclusive | Larger copies use `UploadPartCopy`. This matches AWS's documented `CopyObject` ceiling. It was not re-verified against live AWS in this change. |

`list` and `list_audio` are compatibility collectors. They return at most one page and error when the namespace is larger. Product scans call `list_page`.

## Identity and copy

`supports_server_copy` defaults to false. Local and S3 opt in. A copy runs only when both ends have the same `instance_id` (canonical root + prefix, or endpoint + region + bucket + prefix, or plugin session). Equal backend names are not identity. Two buckets, prefixes, or roots stream.

`copy` stays inside one backend. Staged publication (`commit`) uses that copy and is size-aware, so a multipart upload is not followed by a single `CopyObject` above 5 GiB.

An S3 ETag is an opaque identity hint. It is never treated as SHA-256, including multipart ETags, which are composite part tags. When a digest is supplied it is checked while the body is written. A caller-supplied digest that disagrees with the source probe is rejected before any byte is published. Size alone does not skip publication: `AlreadyPublished` requires the stored whole-object SHA-256 as well.

A local file's ETag is its change signature: size and modification time, plus inode and status-change time on a local Unix disk, and the file index on a local Windows disk. Windows stamps file times from a clock that can tick only every few milliseconds, so without the index two same-size writes in one tick would share an ETag; each write publishes a new file by rename, which gets a new index. An ETag a caller kept from an earlier release no longer matches a local Windows file, so its first `ifMatch` or `ifSourceMatch` reports a conflict once. On network and FUSE file systems (classified per directory by `statfs` on Linux, `MNT_LOCAL` on macOS and FreeBSD, `GetDriveTypeW` on Windows; unknown counts as remote) it is size and modification time only, because inode numbers and change times can shift after a remount. There the backend also never publishes or moves through hard links, settles a retried rename that answers "not found" after it happened, and reports `atomic: false`; a move whose source and target are on different devices goes straight to the host fallback, which checks the copy's size before deleting the source. Reading the file leaves it as it is; writing, replacing, renaming or touching the file changes it. Probes and listings with `withSignatures` report it, and `ifMatch` and `ifSourceMatch` compare against it. It is not a digest.

An S3 object's change signature is its ETag, from `HeadObject` or `ListObjectsV2`. A store can enforce `If-None-Match` / `If-Match` on one request kind and ignore it on another, so the backend probes each kind once, under `.bookclerk-probe/`, and trusts only the kinds it saw refuse a condition. `writes`: `PutObject` refuses a second `If-None-Match: *` write and a wrong `If-Match`. `completes`: `CompleteMultipartUpload` onto the existing probe object refuses both conditions (the upload is then aborted). `copies`: `CopyObject` onto a second existing object refuses `If-None-Match: *`. `copy_pins` and `part_copy_pins`: `CopyObject` and `UploadPartCopy` each refuse a wrong `x-amz-copy-source-if-match` (the part-copy upload is aborted). `deletes`: `DeleteObject` refuses a wrong `If-Match`. A conditional write goes to S3 only when both `writes` and `completes` hold, because the object's size picks the single-request or multipart path. A native move needs `deletes`, `copy_pins` and `part_copy_pins`, because it pins every copy request to the source ETag and deletes the source conditionally; a `noOverwrite` move also needs `copies` and `completes`. Anything else, including a probe request that errors, runs the host fallback and reports `atomic: false`. A move pins the copy with `x-amz-copy-source-if-match`, sends `If-None-Match: *` on the copy (or on `CompleteMultipartUpload` for a multipart copy) for `noOverwrite`, and deletes the source with `If-Match` set to the copied ETag. S3 answers a refused copy with 412 without saying whether the source or the target condition failed; either is reported as `Conflict`.

Local integrity lives in `{key}.bookclerk-meta.json` for that exact object. `book.m4b` and `book.jpg` do not share a stem sidecar. `probe` loads `sha256_hex` and `commit_token` from that record. Descriptive fields (ASIN, title, timestamps) stay descriptive; they are not integrity. A missing stage plus an existing destination counts as commit success only when that object's `commit_token` equals the retry-stable token and the stored digest is the one commit returns.

S3 user metadata is fixed at `CreateMultipartUpload`, before the body exists, so a digest computed while streaming cannot ride in that metadata. After `CompleteMultipartUpload` returns an ETag, the backend writes `{key}.bookclerk-integrity/{hex(etag)}.json` (a few hundred bytes, not a second copy of the audiobook). `probe` reads that object only when its stored ETag equals the current HEAD ETag, and stops after 64 KiB even when `Content-Length` is missing or understated. A larger hint is rejected before the body is read. A record for any other ETag is ignored, including one left by a failed or interleaved replacement. A missing record means the digest is unknown, not that another version's digest applies. `CopyObject` copies inline user metadata only, so a small stage-to-final copy writes a new record for the ETag returned by that `CopyObject`, not for a later HEAD. The digest survives commit replay, and a replacement that lands before a follow-up HEAD does not inherit it. `delete` removes the object, every record under that key's integrity prefix, and a legacy `{key}.bookclerk-meta.json` if one is still present, so a later object at the same key cannot inherit it. An integrity write that fails after the body is complete leaves the new bytes without a digest; it does not keep the previous version's digest. Multipart ETags are not SHA-256.

## Ranges

`ObjectProbe.size` is the whole object, including for a ranged read. `ByteRange.length == None` reads through EOF. The Cap'n Proto field uses `0` for that case and decodes it to `None` before a backend sees it. `Some(0)` is rejected so local and S3 cannot disagree. Offsets past the object and `offset + length` overflow are errors. HEAD does not download the body.

## Pagination

Local `list_page` builds an on-disk external sort under `{root}/.bookclerk-list-index` when `cursor` is absent, in chunks of 2048 keys. Each build first creates and locks `.preparing/<uuid>.lock`, then moves that lock file to `builds/<uuid>.lock` and only then creates its scratch directory `builds/<uuid>/`, so a sweeper that only visits `builds/` cannot delete or lock a build that is not yet owned. The lock stays on the open file across the move. The lock sits beside the directory, not inside it, because Windows refuses to rename a directory that holds an open file. A scratch directory with neither that lock nor an older in-directory `active.lock` held is abandoned. A build also holds `builds/<uuid>/active.lock` while it runs, so a sweeper from an earlier release sharing the root leaves it alone. A generation file keeps a slower build from replacing a newer one, and two prefixes do not share `run-0.bin`. A build that loses the race reads the published index. Readers that already opened the previous file keep that inode across the rename. A missing or deleted cursor key is `InvalidCursor` and does not restart at page one. Record lengths above 64 KiB are rejected instead of allocated. The index directory must be a real directory inside the storage root; a symlink is rejected. Abandoned `builds/` directories and lock files whose lock is not held are removed on the next build. The sweeper holds that lock only long enough to decide the directory is abandoned, then removes it. Active builds are left alone. The index is node-local scratch, not a portable checkpoint. Concurrent creates/deletes during a scan are weakly consistent with the snapshot taken at the fresh scan. Insertion after the snapshot can be missed until the next `cursor == None` rebuild. The job layer may restart explicitly after `InvalidCursor`.

S3 cursors are the service continuation token. A repeated token or a truncated page without a token is `InvalidCursor`. Empty pages are followed at most 8 times.

Fan-out walks children in configuration order. Its `instance_id` is a length-prefixed encoding of those child ids in that order, not a sorted join. The cursor is `bcf2` plus a hash of that identity, the child index, and that child's cursor. A cursor from a different order is `InvalidCursor`. Children are not sort-merged: opaque S3/plugin cursors are not lexicographic keys. Duplicate keys keep the earlier child (source precedence) and are skipped with `exists` on previous children. That is one HEAD per key per earlier child. It is not an integrity hole. Replacing it needs a bounded on-disk seen-set with its own owner, cursor lifetime, and cleanup; an in-memory set of every key is not acceptable, and plugin cursors are not a global sort. One page of objects is retained. A follow-up tracks a bounded dedup design (#233).

Internal keys (`.bookclerk-stage`, `.bookclerk-list-index`, `.bookclerk-tmp`) are not listed as completed media.

## Transfers and cancellation

`transfer_object` streams source to destination unless a same-instance server copy applies. Each attempt reopens the source. A partially consumed reader is not retried. The streamed path writes an attempt-owned key under `.bookclerk-stage/{uuid}/`, checks the source size, ETag, and digest, then publishes with `copy` when the destination supports it (otherwise a second bounded stream from the stage). The stage is armed in a lease before `put_stream`. Explicit failure, drop, attempt timeout, and caller cancellation all delete that stage. Same-instance copy probes the source first and rejects anything above `MAX_SUPPORTED_OBJECT_BYTES` without writing the destination. Direct local `copy` applies the same cap before `fs::copy`. When `stage_journal_dir` is set (acquire sync uses `$BOOKCLERK_FILES_DIR/stage-journal`), record creation must succeed before `put_stream`. The JSON is written and synced through the writable handle of a `.json.partial` file, that handle is closed, then the file is renamed into the scanned name. The owner lock is removed while it is still held, after the record is gone. Opening that lock and acquiring it share `journal.lock` with the reaper, so a sweep cannot unlink the file in the gap before `flock`. `journal.lock` is not held across remote delete or upload. Recovery and debris cleanup walk filenames in order and retain at most 8 records or owner ids at a time, so a failed delete does not block later records and the pass does not keep the whole inventory. A failed remote delete keeps the record and the lock. The reaper removes abandoned partials and owner files that no remaining record names, and it will not unlink a lock another live attempt holds. A read-only reopen is not used: Windows `FlushFileBuffers` requires write access. The reaper ignores partial files. A failed delete leaves the record. The next transfer reaps records whose owner lock is not held and whose storage instance matches, and skips a live attempt. An existing destination, including one a concurrent writer replaced, is not deleted. An existing destination, including one a concurrent writer replaced, is not deleted. A same-instance S3 copy pins every `CopyObject` and `UploadPartCopy` with the probed ETag (`x-amz-copy-source-if-match`). A changed source aborts the copy and does not complete a mixed object. The 5 GiB `CopyObject` ceiling is unchanged; tests may shrink the part size to exercise both sides without a multi-gigabyte body. That fixture is not live AWS conformance. Completed destinations whose stored digest and size already match are skipped. Interrupted objects restart from byte 0; a range read is not a resumable write. Publication is at-least-once relative to database leases, not exactly-once. Multi-destination sync is not a distributed transaction: a failed destination does not roll back one that already published.

Fan-out `put_stream` keeps the cancel guard armed until every child `put_stream` has been joined. Source EOF is not child completion. A drop during the join, or one child's finalization error, aborts and reaps the remaining children. Dropping a `JoinHandle` is not the cleanup path.

Local temp files are removed by a `Drop` guard. S3 multipart uploads are aborted on the error path and, if the future is dropped, from a spawned abort. The destination guest records upload ids under `$HOME/multipart-journal` (the jail-granted plugin data directory), not `$BOOKCLERK_FILES_DIR`. The record stores endpoint, bucket, prefix, key, upload id, and an owner id. The owner holds a lock for the life of the backend. A second backend aborts a record only when the endpoint, bucket, and prefix match and the owner lock is not held. A failed journal write aborts the upload instead of continuing as if recovery existed. In-process backends without a journal do not claim crash recovery. `NoSuchUpload` clears the record. Other abort failures increment a counter and stop automatic retries after 8 attempts; the record stays. Corrupt records are renamed aside. Credentials are not written to the journal.

## Scans and jobs

`storage_scan_rows` holds object and identity rows for one generation. The generation id is a random UUID (`scan-{uuid}`), not a PID. `storage_scan_generations` binds that id to the storage `instance_id` and, when the scan is fenced, the job id. Local filesystem backends, fan-out that includes one, and plugin storage wrappers also record node placement (the node id from `node-identity.json` when `BOOKCLERK_FILES_DIR` is set). A checkpoint is adopted only when that placement matches. Object stores leave placement unset and stay portable. Inventory rows and the generation row are removed in one transaction. A confirmed terminal job drops its inventories while the daemon is up; a pending retry does not. A reused id cannot adopt another instance. The checkpoint (`schema_version` 1, op `storage_scan`) stores instance id, namespace, generation, phase (`list` or `apply`), and cursor. Rows for a page are inserted, then the cursor advances. A crash replays the page (inserts are idempotent) and does not skip it. A stale cursor deletes that generation and restarts once. Cancel and a lost fence stop the scan between pages. `run_acquire` scans once under the job fence and passes that index to storage matching. Adopting a checkpoint requires the generation row to still exist for that instance. An `apply` checkpoint is adopted only when that row is marked complete; a missing or incomplete generation starts a new scan instead of returning an empty index. A `list` checkpoint resumes only when its row exists. The generation is deleted on success, including when the target set is empty, before the job handler returns. A crash after that delete and before the job is terminal therefore rescans. Startup reclaim removes generations whose job is terminal or missing, and jobless generations whose heartbeat is older than the lease. A generation owned by a pending or running job is kept even if the heartbeat is old. Clearing Acquired status happens only after phase `apply`, and only when a fresh `exists` says the stored key is gone. A listing error does not clear. A checkpoint for a different `instance_id` fails closed.

`checkpoint_running_job` writes that JSON under the lease fence without suspending the row. Reclaim leaves the payload in place.

Destination publish remains at-least-once relative to the database lease. A lost fence can still make bytes visible. Retry-stable commit tokens and digest checks make replay idempotent. This is not exactly-once publication.

## What was measured

Ignored tests (not the default `cargo test` suite):

```text
cargo test -p bookclerk-storage --lib list_page_one_hundred_thousand_objects -- --ignored --nocapture
cargo test -p bookclerk-storage --test resource_bounds -- --ignored --nocapture --test-threads=1
```

Measured on this Linux cloud agent (debug build, `/proc/self/status` `VmHWM`; cgroup memory files were not mounted):

| Workload | Result |
| --- | --- |
| Transfer 32 MiB sparse object | 759 ms, VmHWM 5,885,952 bytes |
| Transfer 256 MiB sparse object | 4,890 ms, VmHWM 5,951,488 bytes |
| RSS delta | 65,536 bytes (cap 1 GiB, allowed delta 32 MiB) |
| Local list of 100,001 objects | create 1,232 ms; 391 pages in 621 ms; 1 index build; sort chunk 2,048 |
| Fan-out of that tree (two views, duplicate keys collapsed) | 3,227 ms, 100,001 unique keys |

The 256 MiB object is larger than the 32 MiB allowed delta and was fully published (`metadata.len` matched). Parent RSS did not track object size. That run is an in-process diagnostic. It does not impose a memory limit and it is not #120 acceptance for the host/workerd/guest tree. A missing `VmHWM` reading fails the test instead of being treated as zero.

Partial external measurement (not acceptance). The workerd conformance test moves only the gateway and guest into the cgroup and starts the guest with `Command`, not `bookclerk-jail`. Its `memory.peak` includes reclaimable file cache. It stays labeled partial.

```text
cargo test -p bookclerk-workerd --test conformance external_native_behind_workerd_budget -- --ignored --nocapture --test-threads=1
```

Previous partial numbers on this class of Linux agent (debug build, `memory.max` = 402,653,184): 32 MiB peak 105,603,072; 416 MiB object peak 402,653,184; 100,001 keys in 391 pages. That peak sat on the cap and is not evidence that anonymous memory tracked the object.

Production-path acceptance (host re-exec'd into the cgroup before its runtime, then `PluginSession` with required isolation). Passed on this Linux agent:

```text
BOOKCLERK_RESOURCE_ARTIFACT_DIR=/opt/cursor/artifacts cargo test -p bookclerk-plugin-host --test external_budget -- --ignored --nocapture --test-threads=1
```

| | |
| --- | --- |
| Processes | host in `…/bookclerk-budget-*/bookclerk-host`; gateway and guest in `…/bookclerk-session-…` under that same budget. Guest `Seccomp` = 2 |
| `memory.max` after startup | 285,949,952 bytes |
| 8 MiB object | sampled anon 33,480,704 |
| 353,058,816 byte object (larger than `memory.max`) | sampled anon 34,660,352; sampled file cache 244,822,016; on-disk SHA-256 matched a digest computed from the source bytes |
| 100,001 keys, exact ordered list | sampled anon 37,847,040; sampled file cache 253,419,520 |
| Lifetime `memory.peak` | 566,173,696. This is from cgroup creation, including startup under a 2 GiB ceiling, and is not the transfer bound |

Anonymous RSS stayed near 35 MiB while the object was 353 MiB. The cgroup total during the transfer was mostly file cache. `s3_journal_restart` killed a jailed `destination-s3` guest (`Seccomp` 2) after a multipart part was accepted and the restarted guest aborted that upload (6.6 s). That fixture is local HTTP, not live AWS. SQLite ran the scan-adoption tests. Postgres was not available here (`BOOKCLERK_TEST_POSTGRES_URL` unset); `storage_scan_generations.completed` is in the unreleased migration the existing PostgreSQL jobs apply.
