bookclerk_archive

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

Bounded extraction of the archives people upload.

An uploaded ZIP or tar file is untrusted input: its entry names may try to escape the output folder, its sizes may lie, and its contents may expand without end. This crate decides what an archive may write and stops it when it overreaches. It never opens a file itself:

  • EntryPath::admit turns a raw entry name into a relative path that cannot leave the output folder, or refuses it.
  • extract walks an archive read from any Read + Seek source and writes each admitted entry through a Sink, counting the bytes it actually writes against Limits rather than trusting the archive's headers.
  • ExtractReport::validate re-checks a report that crossed a process boundary, because the code that produced it parsed untrusted bytes.

The same code runs in the confined media worker's extract job, in the archive_extract fuzz target (over an in-memory sink) and in the unit tests. Nested archives are never expanded, links and devices are never created, and encrypted entries are refused.

Item Kind Summary
Sink trait Where an extraction writes its files.
Limits struct The caps one extraction runs under.
EntryPath struct A relative path that stays inside the folder it is joined to.
FoldedNames struct The files written so far, by folded name, and the folders they imply.
ExtractReport struct What one extraction did.
RefusedEntry struct One entry the extraction did not write, and why.
WrittenEntry struct One file the extraction wrote.
DirSink struct Writes files under one folder on disk, which should start empty.
MemorySink struct Keeps files in memory, for tests and the fuzz target.
ArchiveFormat enum The archive formats Bookclerk expands.
NameClash enum Why a path cannot join a FoldedNames set.
PathRefusal enum Why a name was not admitted as an EntryPath.
RefusalReason enum Why one entry was not written.
ReportError enum A report that breaks the limits it was produced under.
StopReason enum Why an extraction stopped before the end of the archive.
ExtractError enum Why an archive could not be walked at all.
extract function Expands an archive through sink under limits.
SNIFF_BYTES constant Bytes ArchiveFormat::sniff needs to recognise every format: the tar magic sits at offset 257.
MAX_DEPTH constant Most components an admitted path may have.
MAX_PATH_BYTES constant Longest admitted path, in bytes of UTF-8.
MAX_REFUSED_NAME_CHARS constant Longest name kept for a refused entry, in characters. Longer names are cut; the name is only shown to people, never used as a path.

Modules

  • fixtures: Archive builders for tests and fuzz seeds, including archives a careful tool would refuse to write.

Traits

Sink

pub trait Sink {
    type Writer: Write;
    fn create(&mut self, path: &EntryPath) -> Result<Self::Writer>;
    fn discard(&mut self, path: &EntryPath);
}

Where an extraction writes its files.

crate::extract only ever names admitted EntryPaths, checked against earlier names, so a sink never sees two paths that clash.

Sink::Writer

type Writer: Write

The writer for one file.

Sink::create

fn create(&mut self, path: &EntryPath) -> Result<Self::Writer>

Creates the file at path, and any folders it sits in.

Arguments
  • path - The file to create.
Errors

The I/O error, when the file cannot be created; the entry is then refused as corrupt and the walk goes on.

Sink::discard

fn discard(&mut self, path: &EntryPath)

Removes a file created for an entry that was then refused.

Arguments
  • path - A file this sink created.

Implemented by: DirSink, MemorySink

Structs

Limits

pub struct Limits {
    pub max_entries: u32,
    pub max_total_bytes: u64,
    pub max_entry_bytes: u64,
    pub max_ratio: u64,
    pub ratio_floor_bytes: u64,
}

The caps one extraction runs under.

Every cap counts bytes as they are written, never sizes an archive declares. The host chooses them per job (the total is never more than the uploader's remaining quota) and sends them to the worker with the job.

Fields

  • max_entries: Most entries the archive may list, counting refused ones. An archive that lists more is stopped before anything is written.
  • max_total_bytes: Most bytes the whole extraction may write. Reaching it stops the extraction.
  • max_entry_bytes: Most bytes one entry may write. An entry that needs more is refused and its partial file discarded.
  • max_ratio: Largest ratio of written bytes to compressed bytes an entry may reach once it is past Limits::ratio_floor_bytes. Audio barely compresses, so an honest audiobook sits near 1:1.
  • ratio_floor_bytes: Bytes an entry (or a whole gzip stream) may write before the ratio applies, so small, very compressible text files are not refused.

Limits::UPLOAD_DEFAULTS

pub const UPLOAD_DEFAULTS: Self = _

The defaults for uploads: 10,000 entries, 64 GiB in all, 20 GiB per entry, 100:1 past 1 MiB.

Limits::with_total_at_most

pub fn with_total_at_most(self, total: u64) -> Self

The same caps with the total lowered to total when that is smaller, for an uploader with less quota left than the default total.

Arguments
  • total - The most bytes this extraction may write.

Limits::ratio_allowance

pub fn ratio_allowance(&self, compressed: u64) -> u64

Most bytes something compressed to compressed bytes may expand to.

Arguments
  • compressed - The entry's (or stream's) compressed size in bytes.

Implements: Clone, Copy, Debug, Default, Deserialize<'de>, Eq, PartialEq, Serialize

EntryPath

pub struct EntryPath(/* private */);

A relative path that stays inside the folder it is joined to.

Components are separated by /, none is empty, . or .., and none holds a control character, a : or a \. The path is valid UTF-8, at most MAX_PATH_BYTES long and at most MAX_DEPTH deep, and no component is a Windows device name or ends in a dot or a space, so the same path is safe on every filesystem a destination may use.

EntryPath::admit

pub fn admit(raw: &str) -> Result<Self, PathRefusal>

Admits a name as a relative path, or says why it cannot be one.

\ is read as a separator (ZIP tools on Windows write it), empty and . components are dropped, and everything else in the policy above is refused rather than repaired.

Arguments
  • raw - The name as the archive or manifest spells it.
Errors

The PathRefusal for the first rule the name breaks.

EntryPath::admit_bytes

pub fn admit_bytes(raw: &[u8]) -> Result<Self, PathRefusal>

Admits a name given as bytes, which must be UTF-8.

Arguments
  • raw - The name's bytes, as a tar header carries them.
Errors

PathRefusal::NotUtf8, or what EntryPath::admit refuses.

EntryPath::as_str

pub fn as_str(&self) -> &str

The path, components separated by /.

EntryPath::components

pub fn components(&self) -> impl Iterator<Item = &str>

The path's components, outermost first.

EntryPath::file_name

pub fn file_name(&self) -> &str

The last component.

EntryPath::fold_key

pub fn fold_key(&self) -> String

The key two paths clash on: NFC-normalised and lower-cased, so names a case-insensitive or normalising filesystem would merge share it.

EntryPath::is_system_file

pub fn is_system_file(&self) -> bool

Whether this is an operating system's bookkeeping file rather than content: macOS resource forks (__MACOSX/, ._name), .DS_Store, and Windows' Thumbs.db and desktop.ini.

Implements: Clone, Debug, Deserialize<'de>, Display, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize

FoldedNames

pub struct FoldedNames {
    /* private fields */
}

The files written so far, by folded name, and the folders they imply.

FoldedNames::insert refuses a path that would overwrite an earlier file on any filesystem, or needs a folder where a file is.

FoldedNames::new

pub fn new() -> Self

An empty set.

FoldedNames::check

pub fn check(&self, path: &EntryPath) -> Result<(), NameClash>

Checks a file path against the set without adding it.

Arguments
  • path - The file about to be written.
Errors

The clash, when the path cannot be written beside the others.

FoldedNames::insert

pub fn insert(&mut self, path: &EntryPath) -> Result<(), NameClash>

Adds a file path, after FoldedNames::check.

Arguments
  • path - The file being written.
Errors

The clash, when the path cannot be written beside the others; the set is then unchanged.

FoldedNames::len

pub fn len(&self) -> usize

How many files the set holds.

FoldedNames::is_empty

pub fn is_empty(&self) -> bool

Whether the set holds no file.

Implements: Clone, Debug, Default

ExtractReport

pub struct ExtractReport {
    pub format: ArchiveFormat,
    pub written: Vec<WrittenEntry>,
    pub refused: Vec<RefusedEntry>,
    pub stopped: Option<StopReason>,
    pub total_bytes: u64,
}

What one extraction did.

Fields

  • format: The format the archive was walked as.
  • written: The files written, in archive order.
  • refused: The entries refused, in archive order. Folders are not listed.
  • stopped: Why the walk stopped early, if it did.
  • total_bytes: Bytes written in all; the sum of WrittenEntry::bytes.

ExtractReport::new

pub fn new(format: ArchiveFormat) -> Self

An empty report for an archive of format.

Arguments
  • format - The format being walked.

ExtractReport::refuse

pub fn refuse(&mut self, raw_name: &[u8], reason: RefusalReason)

Records a refused entry, cleaning its name for display.

Arguments
  • raw_name - The entry's name bytes.
  • reason - Why it was refused.

ExtractReport::validate

pub fn validate(&self, limits: &Limits) -> Result<(), ReportError>

Checks a report produced by another process against the limits the job ran under.

Paths were already admitted when the report was deserialised; this adds the checks that span entries.

Arguments
  • limits - The limits sent with the job.
Errors

The first ReportError found.

Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize

RefusedEntry

pub struct RefusedEntry {
    pub name: String,
    pub reason: RefusalReason,
}

One entry the extraction did not write, and why.

Fields

  • name: The entry's name as the archive spells it, for people to read: invalid UTF-8 and control characters replaced, cut to MAX_REFUSED_NAME_CHARS.
  • reason: Why it was refused.

Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize

WrittenEntry

pub struct WrittenEntry {
    pub path: EntryPath,
    pub bytes: u64,
}

One file the extraction wrote.

Fields

  • path: Where it was written, relative to the output folder.
  • bytes: How many bytes were written.

Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize

DirSink

pub struct DirSink {
    /* private fields */
}

Writes files under one folder on disk, which should start empty.

Files are created with create_new, so nothing that already exists is opened and no link is followed. Folders are created one component at a time and must be real folders.

DirSink::new

pub fn new(root: impl Into<PathBuf>) -> Self

A sink writing under root.

Arguments
  • root - An existing, empty folder.

DirSink::root

pub fn root(&self) -> &Path

The output folder.

Implements: Clone, Debug, Sink

MemorySink

pub struct MemorySink {
    /* private fields */
}

Keeps files in memory, for tests and the fuzz target.

MemorySink::new

pub fn new() -> Self

An empty sink.

MemorySink::files

pub fn files(&self) -> BTreeMap<EntryPath, Vec<u8>>

The files written so far, by path.

Implements: Clone, Debug, Default, Sink

Enums

ArchiveFormat

pub enum ArchiveFormat {
    Zip,
    Tar,
    TarGz,
}

The archive formats Bookclerk expands.

The format is always decided from the bytes (ArchiveFormat::sniff), never from the file name, so a renamed file cannot pick a decoder.

Variants

  • Zip: A ZIP archive (stored, Deflate or Deflate64 entries; Zip64 sizes).
  • Tar: An uncompressed POSIX, GNU or PAX tar archive.
  • TarGz: A tar archive inside one gzip stream.

ArchiveFormat::sniff

pub fn sniff(head: &[u8]) -> Option<Self>

Recognises an archive from its first bytes.

Arguments
  • head - The file's first bytes; SNIFF_BYTES is enough for every format, fewer may miss a tar archive.
Returns

The format, or None when the bytes start no format this crate expands. A gzip stream is reported as ArchiveFormat::TarGz; whether it really holds a tar archive shows when it is walked.

ArchiveFormat::looks_like_archive

pub fn looks_like_archive(head: &[u8]) -> bool

Whether bytes look like any archive or compressed container, this crate's formats or others (7z, RAR, xz, bzip2, zstd).

Used to refuse nested archives: an entry that is itself an archive is not expanded and not kept.

Arguments
  • head - The entry's first bytes.

ArchiveFormat::as_str

pub fn as_str(self) -> &'static str

The name used in logs and replies.

Implements: Clone, Copy, Debug, Deserialize<'de>, Eq, Hash, PartialEq, Serialize

NameClash

pub enum NameClash {
    Duplicate,
    FileFolder,
}

Why a path cannot join a FoldedNames set.

Variants

  • Duplicate: Another file already has the same folded name.
  • FileFolder: The path needs a folder where a file already is, or a file where a folder already is.

Implements: Clone, Copy, Debug, Deserialize<'de>, Display, Eq, Error, PartialEq, Serialize

PathRefusal

pub enum PathRefusal {
    NotUtf8,
    Empty,
    Absolute,
    Colon,
    ParentDir,
    Control,
    Reserved,
    TooLong,
    TooDeep,
}

Why a name was not admitted as an EntryPath.

Variants

  • NotUtf8: The name is not valid UTF-8.
  • Empty: Nothing is left once empty and . components are dropped (a directory entry such as ./).
  • Absolute: The name starts at a root (/, \).
  • Colon: The name holds a :, which names a drive, a stream or a device on Windows.
  • ParentDir: A component is ...
  • Control: The name holds a control character (including NUL).
  • Reserved: A component is a Windows device name or ends in a dot or a space.
  • TooLong: The name or one of its components is too long.
  • TooDeep: The name has more than MAX_DEPTH components.

Implements: Clone, Copy, Debug, Deserialize<'de>, Display, Eq, Error, Hash, PartialEq, Serialize

RefusalReason

pub enum RefusalReason {
    Path { refusal: PathRefusal },
    Clash { clash: NameClash },
    Link,
    Special,
    Encrypted,
    Compression,
    NestedArchive,
    EntryTooLarge,
    RatioExceeded,
    Overlap,
    SystemFile,
    Corrupt,
    WriteFailed,
}

Why one entry was not written.

Variants

  • Path: Its name breaks the path policy.
    • refusal: The rule it breaks.
  • Clash: Its name clashes with an earlier entry.
    • clash: How it clashes.
  • Link: It is a symbolic or hard link.
  • Special: It is a device, a FIFO or another special file.
  • Encrypted: It is encrypted.
  • Compression: It uses a compression method this crate does not decode.
  • NestedArchive: It is itself an archive; nested archives are not expanded.
  • EntryTooLarge: It would write more than Limits::max_entry_bytes.
  • RatioExceeded: It expands past Limits::max_ratio.
  • Overlap: Its data overlaps another entry's (a ZIP that reuses one stream for many names).
  • SystemFile: It is an operating system's bookkeeping file, such as __MACOSX/.
  • Corrupt: Its data could not be decoded or failed its checksum.
  • WriteFailed: The output file could not be created or written.

Implements: Clone, Copy, Debug, Deserialize<'de>, Eq, PartialEq, Serialize

ReportError

pub enum ReportError {
    TooManyEntries,
    EntryTooLarge(String),
    Total,
    Clash(String),
    RefusedName,
}

A report that breaks the limits it was produced under.

Variants

  • TooManyEntries: More entries than Limits::max_entries.
  • EntryTooLarge: One file larger than Limits::max_entry_bytes.
  • Total: The total is not the sum of the files, or is past the limit.
  • Clash: Two files clash.
  • RefusedName: A refused entry's name is too long or holds a control character.

Implements: Clone, Debug, Display, Eq, Error, PartialEq

StopReason

pub enum StopReason {
    TooManyEntries,
    TotalExceeded,
    Corrupt,
}

Why an extraction stopped before the end of the archive.

A stopped extraction's files are kept on disk only until the host reads the report; the host discards the whole archive.

Variants

  • TooManyEntries: The archive lists more than Limits::max_entries entries.
  • TotalExceeded: Writing went past Limits::max_total_bytes, or a gzip stream past its ratio allowance.
  • Corrupt: The archive's structure broke partway through.

Implements: Clone, Copy, Debug, Deserialize<'de>, Eq, PartialEq, Serialize

ExtractError

pub enum ExtractError {
    NotArchive,
    Io(Error),
}

Why an archive could not be walked at all.

Variants

  • NotArchive: The bytes start no format this crate expands.
  • Io: The archive could not be read.

Implements: Debug, Display, Error, From<Error>

Functions

extract

pub fn extract<R: Read + Seek, S: Sink>(
    reader: R,
    limits: &Limits,
    sink: &mut S,
) -> Result<ExtractReport, ExtractError>

Expands an archive through sink under limits.

The format is sniffed from the first bytes. Each entry's name is admitted by EntryPath::admit and checked against the names written before it; links, devices, encrypted entries, nested archives and system files are refused; bytes are counted as they are written. A refused entry's partial file is discarded. Reaching the entry count or the total stops the walk, and the caller should then discard everything (StopReason).

Arguments

  • reader - The archive, positioned anywhere.
  • limits - The caps for this extraction.
  • sink - Where files go.

Returns

The [ExtractReport]: files written, entries refused and why, and why the walk stopped if it did.

Errors

ExtractError::NotArchive when the format is not recognised, and ExtractError::Io when the archive cannot be read before the walk starts. A structure that breaks partway is reported as StopReason::Corrupt instead.

Constants

SNIFF_BYTES

pub const SNIFF_BYTES: usize = 265;

Bytes ArchiveFormat::sniff needs to recognise every format: the tar magic sits at offset 257.

MAX_DEPTH

pub const MAX_DEPTH: usize = 32;

Most components an admitted path may have.

MAX_PATH_BYTES

pub const MAX_PATH_BYTES: usize = 1024;

Longest admitted path, in bytes of UTF-8.

MAX_REFUSED_NAME_CHARS

pub const MAX_REFUSED_NAME_CHARS: usize = 256;

Longest name kept for a refused entry, in characters. Longer names are cut; the name is only shown to people, never used as a path.

Generated from crates/bookclerk-archive at main · View as Markdown