bookclerk_archive
Bounded extraction of the archives people upload.
An uploaded ZIP or tar file is untrusted input: its entry names may try to escape the output folder, its sizes may lie, and its contents may expand without end. This crate decides what an archive may write and stops it when it overreaches. It never opens a file itself:
EntryPath::admitturns a raw entry name into a relative path that cannot leave the output folder, or refuses it.extractwalks an archive read from anyRead + Seeksource and writes each admitted entry through aSink, counting the bytes it actually writes againstLimitsrather than trusting the archive's headers.ExtractReport::validatere-checks a report that crossed a process boundary, because the code that produced it parsed untrusted bytes.
The same code runs in the confined media worker's extract job, in the
archive_extract fuzz target (over an in-memory sink) and in the unit
tests. Nested archives are never expanded, links and devices are never
created, and encrypted entries are refused.
| Item | Kind | Summary |
|---|---|---|
Sink |
trait | Where an extraction writes its files. |
Limits |
struct | The caps one extraction runs under. |
EntryPath |
struct | A relative path that stays inside the folder it is joined to. |
FoldedNames |
struct | The files written so far, by folded name, and the folders they imply. |
ExtractReport |
struct | What one extraction did. |
RefusedEntry |
struct | One entry the extraction did not write, and why. |
WrittenEntry |
struct | One file the extraction wrote. |
DirSink |
struct | Writes files under one folder on disk, which should start empty. |
MemorySink |
struct | Keeps files in memory, for tests and the fuzz target. |
ArchiveFormat |
enum | The archive formats Bookclerk expands. |
NameClash |
enum | Why a path cannot join a FoldedNames set. |
PathRefusal |
enum | Why a name was not admitted as an EntryPath. |
RefusalReason |
enum | Why one entry was not written. |
ReportError |
enum | A report that breaks the limits it was produced under. |
StopReason |
enum | Why an extraction stopped before the end of the archive. |
ExtractError |
enum | Why an archive could not be walked at all. |
extract |
function | Expands an archive through sink under limits. |
SNIFF_BYTES |
constant | Bytes ArchiveFormat::sniff needs to recognise every format: the tar magic sits at offset 257. |
MAX_DEPTH |
constant | Most components an admitted path may have. |
MAX_PATH_BYTES |
constant | Longest admitted path, in bytes of UTF-8. |
MAX_REFUSED_NAME_CHARS |
constant | Longest name kept for a refused entry, in characters. Longer names are cut; the name is only shown to people, never used as a path. |
fixtures: Archive builders for tests and fuzz seeds, including archives a careful tool would refuse to write.
pub trait Sink {
type Writer: Write;
fn create(&mut self, path: &EntryPath) -> Result<Self::Writer>;
fn discard(&mut self, path: &EntryPath);
}
Where an extraction writes its files.
crate::extract only ever names admitted EntryPaths, checked
against earlier names, so a sink never sees two paths that clash.
type Writer: Write
The writer for one file.
fn create(&mut self, path: &EntryPath) -> Result<Self::Writer>
Creates the file at path, and any folders it sits in.
path- The file to create.
The I/O error, when the file cannot be created; the entry is then refused as corrupt and the walk goes on.
fn discard(&mut self, path: &EntryPath)
Removes a file created for an entry that was then refused.
path- A file this sink created.
Implemented by: DirSink, MemorySink
pub struct Limits {
pub max_entries: u32,
pub max_total_bytes: u64,
pub max_entry_bytes: u64,
pub max_ratio: u64,
pub ratio_floor_bytes: u64,
}
The caps one extraction runs under.
Every cap counts bytes as they are written, never sizes an archive declares. The host chooses them per job (the total is never more than the uploader's remaining quota) and sends them to the worker with the job.
Fields
max_entries: Most entries the archive may list, counting refused ones. An archive that lists more is stopped before anything is written.max_total_bytes: Most bytes the whole extraction may write. Reaching it stops the extraction.max_entry_bytes: Most bytes one entry may write. An entry that needs more is refused and its partial file discarded.max_ratio: Largest ratio of written bytes to compressed bytes an entry may reach once it is pastLimits::ratio_floor_bytes. Audio barely compresses, so an honest audiobook sits near 1:1.ratio_floor_bytes: Bytes an entry (or a whole gzip stream) may write before the ratio applies, so small, very compressible text files are not refused.
pub const UPLOAD_DEFAULTS: Self = _
The defaults for uploads: 10,000 entries, 64 GiB in all, 20 GiB per entry, 100:1 past 1 MiB.
pub fn with_total_at_most(self, total: u64) -> Self
The same caps with the total lowered to total when that is smaller,
for an uploader with less quota left than the default total.
total- The most bytes this extraction may write.
pub fn ratio_allowance(&self, compressed: u64) -> u64
Most bytes something compressed to compressed bytes may expand to.
compressed- The entry's (or stream's) compressed size in bytes.
Implements: Clone, Copy, Debug, Default, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct EntryPath(/* private */);
A relative path that stays inside the folder it is joined to.
Components are separated by /, none is empty, . or .., and none
holds a control character, a : or a \. The path is valid UTF-8, at
most MAX_PATH_BYTES long and at most MAX_DEPTH deep, and no
component is a Windows device name or ends in a dot or a space, so the
same path is safe on every filesystem a destination may use.
pub fn admit(raw: &str) -> Result<Self, PathRefusal>
Admits a name as a relative path, or says why it cannot be one.
\ is read as a separator (ZIP tools on Windows write it), empty and
. components are dropped, and everything else in the policy above is
refused rather than repaired.
raw- The name as the archive or manifest spells it.
The PathRefusal for the first rule the name breaks.
pub fn admit_bytes(raw: &[u8]) -> Result<Self, PathRefusal>
Admits a name given as bytes, which must be UTF-8.
raw- The name's bytes, as a tar header carries them.
PathRefusal::NotUtf8, or what EntryPath::admit refuses.
pub fn as_str(&self) -> &str
The path, components separated by /.
pub fn components(&self) -> impl Iterator<Item = &str>
The path's components, outermost first.
pub fn file_name(&self) -> &str
The last component.
pub fn fold_key(&self) -> String
The key two paths clash on: NFC-normalised and lower-cased, so names a case-insensitive or normalising filesystem would merge share it.
pub fn is_system_file(&self) -> bool
Whether this is an operating system's bookkeeping file rather than
content: macOS resource forks (__MACOSX/, ._name), .DS_Store,
and Windows' Thumbs.db and desktop.ini.
Implements: Clone, Debug, Deserialize<'de>, Display, Eq, Hash, Ord, PartialEq, PartialOrd, Serialize
pub struct FoldedNames {
/* private fields */
}
The files written so far, by folded name, and the folders they imply.
FoldedNames::insert refuses a path that would overwrite an earlier
file on any filesystem, or needs a folder where a file is.
pub fn new() -> Self
An empty set.
pub fn check(&self, path: &EntryPath) -> Result<(), NameClash>
Checks a file path against the set without adding it.
path- The file about to be written.
The clash, when the path cannot be written beside the others.
pub fn insert(&mut self, path: &EntryPath) -> Result<(), NameClash>
Adds a file path, after FoldedNames::check.
path- The file being written.
The clash, when the path cannot be written beside the others; the set is then unchanged.
pub fn len(&self) -> usize
How many files the set holds.
pub fn is_empty(&self) -> bool
Whether the set holds no file.
Implements: Clone, Debug, Default
pub struct ExtractReport {
pub format: ArchiveFormat,
pub written: Vec<WrittenEntry>,
pub refused: Vec<RefusedEntry>,
pub stopped: Option<StopReason>,
pub total_bytes: u64,
}
What one extraction did.
Fields
format: The format the archive was walked as.written: The files written, in archive order.refused: The entries refused, in archive order. Folders are not listed.stopped: Why the walk stopped early, if it did.total_bytes: Bytes written in all; the sum ofWrittenEntry::bytes.
pub fn new(format: ArchiveFormat) -> Self
An empty report for an archive of format.
format- The format being walked.
pub fn refuse(&mut self, raw_name: &[u8], reason: RefusalReason)
Records a refused entry, cleaning its name for display.
raw_name- The entry's name bytes.reason- Why it was refused.
pub fn validate(&self, limits: &Limits) -> Result<(), ReportError>
Checks a report produced by another process against the limits the job ran under.
Paths were already admitted when the report was deserialised; this adds the checks that span entries.
limits- The limits sent with the job.
The first ReportError found.
Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct RefusedEntry {
pub name: String,
pub reason: RefusalReason,
}
One entry the extraction did not write, and why.
Fields
name: The entry's name as the archive spells it, for people to read: invalid UTF-8 and control characters replaced, cut toMAX_REFUSED_NAME_CHARS.reason: Why it was refused.
Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct WrittenEntry {
pub path: EntryPath,
pub bytes: u64,
}
One file the extraction wrote.
Fields
path: Where it was written, relative to the output folder.bytes: How many bytes were written.
Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct DirSink {
/* private fields */
}
Writes files under one folder on disk, which should start empty.
Files are created with create_new, so nothing that already exists is
opened and no link is followed. Folders are created one component at a time and must be
real folders.
pub fn new(root: impl Into<PathBuf>) -> Self
A sink writing under root.
root- An existing, empty folder.
pub fn root(&self) -> &Path
The output folder.
Implements: Clone, Debug, Sink
pub struct MemorySink {
/* private fields */
}
Keeps files in memory, for tests and the fuzz target.
pub fn new() -> Self
An empty sink.
pub fn files(&self) -> BTreeMap<EntryPath, Vec<u8>>
The files written so far, by path.
Implements: Clone, Debug, Default, Sink
pub enum ArchiveFormat {
Zip,
Tar,
TarGz,
}
The archive formats Bookclerk expands.
The format is always decided from the bytes (ArchiveFormat::sniff),
never from the file name, so a renamed file cannot pick a decoder.
Variants
Zip: A ZIP archive (stored, Deflate or Deflate64 entries; Zip64 sizes).Tar: An uncompressed POSIX, GNU or PAX tar archive.TarGz: A tar archive inside one gzip stream.
pub fn sniff(head: &[u8]) -> Option<Self>
Recognises an archive from its first bytes.
head- The file's first bytes;SNIFF_BYTESis enough for every format, fewer may miss a tar archive.
The format, or None when the bytes start no format this crate
expands. A gzip stream is reported as ArchiveFormat::TarGz; whether
it really holds a tar archive shows when it is walked.
pub fn looks_like_archive(head: &[u8]) -> bool
Whether bytes look like any archive or compressed container, this crate's formats or others (7z, RAR, xz, bzip2, zstd).
Used to refuse nested archives: an entry that is itself an archive is not expanded and not kept.
head- The entry's first bytes.
pub fn as_str(self) -> &'static str
The name used in logs and replies.
Implements: Clone, Copy, Debug, Deserialize<'de>, Eq, Hash, PartialEq, Serialize
pub enum NameClash {
Duplicate,
FileFolder,
}
Why a path cannot join a FoldedNames set.
Variants
Duplicate: Another file already has the same folded name.FileFolder: The path needs a folder where a file already is, or a file where a folder already is.
Implements: Clone, Copy, Debug, Deserialize<'de>, Display, Eq, Error, PartialEq, Serialize
pub enum PathRefusal {
NotUtf8,
Empty,
Absolute,
Colon,
ParentDir,
Control,
Reserved,
TooLong,
TooDeep,
}
Why a name was not admitted as an EntryPath.
Variants
NotUtf8: The name is not valid UTF-8.Empty: Nothing is left once empty and.components are dropped (a directory entry such as./).Absolute: The name starts at a root (/,\).Colon: The name holds a:, which names a drive, a stream or a device on Windows.ParentDir: A component is...Control: The name holds a control character (including NUL).Reserved: A component is a Windows device name or ends in a dot or a space.TooLong: The name or one of its components is too long.TooDeep: The name has more thanMAX_DEPTHcomponents.
Implements: Clone, Copy, Debug, Deserialize<'de>, Display, Eq, Error, Hash, PartialEq, Serialize
pub enum RefusalReason {
Path { refusal: PathRefusal },
Clash { clash: NameClash },
Link,
Special,
Encrypted,
Compression,
NestedArchive,
EntryTooLarge,
RatioExceeded,
Overlap,
SystemFile,
Corrupt,
WriteFailed,
}
Why one entry was not written.
Variants
Path: Its name breaks the path policy.refusal: The rule it breaks.
Clash: Its name clashes with an earlier entry.clash: How it clashes.
Link: It is a symbolic or hard link.Special: It is a device, a FIFO or another special file.Encrypted: It is encrypted.Compression: It uses a compression method this crate does not decode.NestedArchive: It is itself an archive; nested archives are not expanded.EntryTooLarge: It would write more thanLimits::max_entry_bytes.RatioExceeded: It expands pastLimits::max_ratio.Overlap: Its data overlaps another entry's (a ZIP that reuses one stream for many names).SystemFile: It is an operating system's bookkeeping file, such as__MACOSX/.Corrupt: Its data could not be decoded or failed its checksum.WriteFailed: The output file could not be created or written.
Implements: Clone, Copy, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub enum ReportError {
TooManyEntries,
EntryTooLarge(String),
Total,
Clash(String),
RefusedName,
}
A report that breaks the limits it was produced under.
Variants
TooManyEntries: More entries thanLimits::max_entries.EntryTooLarge: One file larger thanLimits::max_entry_bytes.Total: The total is not the sum of the files, or is past the limit.Clash: Two files clash.RefusedName: A refused entry's name is too long or holds a control character.
Implements: Clone, Debug, Display, Eq, Error, PartialEq
pub enum StopReason {
TooManyEntries,
TotalExceeded,
Corrupt,
}
Why an extraction stopped before the end of the archive.
A stopped extraction's files are kept on disk only until the host reads the report; the host discards the whole archive.
Variants
TooManyEntries: The archive lists more thanLimits::max_entriesentries.TotalExceeded: Writing went pastLimits::max_total_bytes, or a gzip stream past its ratio allowance.Corrupt: The archive's structure broke partway through.
Implements: Clone, Copy, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub enum ExtractError {
NotArchive,
Io(Error),
}
Why an archive could not be walked at all.
Variants
NotArchive: The bytes start no format this crate expands.Io: The archive could not be read.
Implements: Debug, Display, Error, From<Error>
pub fn extract<R: Read + Seek, S: Sink>(
reader: R,
limits: &Limits,
sink: &mut S,
) -> Result<ExtractReport, ExtractError>
Expands an archive through sink under limits.
The format is sniffed from the first bytes. Each entry's name is admitted
by EntryPath::admit and checked against the names written before it;
links, devices, encrypted entries, nested archives and system files are
refused; bytes are counted as they are written. A refused entry's partial
file is discarded. Reaching the entry count or the total stops the walk,
and the caller should then discard everything (StopReason).
reader- The archive, positioned anywhere.limits- The caps for this extraction.sink- Where files go.
The [ExtractReport]: files written, entries refused and why, and why
the walk stopped if it did.
ExtractError::NotArchive when the format is not recognised, and
ExtractError::Io when the archive cannot be read before the walk
starts. A structure that breaks partway is reported as
StopReason::Corrupt instead.
pub const SNIFF_BYTES: usize = 265;
Bytes ArchiveFormat::sniff needs to recognise every format: the tar
magic sits at offset 257.
pub const MAX_DEPTH: usize = 32;
Most components an admitted path may have.
pub const MAX_PATH_BYTES: usize = 1024;
Longest admitted path, in bytes of UTF-8.
pub const MAX_REFUSED_NAME_CHARS: usize = 256;
Longest name kept for a refused entry, in characters. Longer names are cut; the name is only shown to people, never used as a path.
Generated from crates/bookclerk-archive at main · View as Markdown