bookclerk_model_pack
Layout of an embedding model on disk.
A text-embedding model has a vocabulary table with one row per token, and that table is most of the download: 98 MB of bekko-a8m's 130 MB. A library uses a handful of languages, so the table is shipped in parts. One model directory holds:
| File | What it is |
|---|---|
model.json |
The [PackManifest]: what the model is and the size and SHA-256 of every other file. |
graph (core.onnx) |
The published ONNX graph with its token tables turned into graph inputs. |
tokenizer (tokenizer.spm.fb) |
The whole vocabulary as a SentencePiece Lite model. |
base rows (base.rows) |
Table rows every text needs: special tokens, single characters, bytes. |
language rows (<tag>.rows) |
The other rows the texts of one language use. |
The model plugin tokenizes with the whole vocabulary, looks each token's row up in the row packs it has, and passes only those rows to the graph. Token ids for a text are therefore the published model's own ids whichever packs are present, and the output for a text whose rows are all present is bit-identical to the published model's.
Everything here is sans-IO: functions take bytes and return validated
values, so the plugin, the pack builder's tests and the fuzzers run the
same code. See docs/internals/model-packs.md.
| Item | Kind | Summary |
|---|---|---|
GraphSpec |
struct | The cut ONNX graph and the names it uses. |
LanguageTag |
struct | A language tag naming a language pack: lowercase letters and digits in --separated parts of 1 to 8 characters, at most 35 bytes (en, pt-br). |
PackFile |
struct | One file of a model directory, with what its content must be. |
PackFileName |
struct | A file name inside a model directory: 1 to 64 lowercase letters, digits, ., _ and -, starting with a letter or digit. |
PackManifest |
struct | model.json: one embedding model, cut for row packs. |
Sha256Hex |
struct | A SHA-256 digest as 64 lowercase hexadecimal characters. |
TableSpec |
struct | One token table the graph takes as an input: [tokens, width]. |
TokenizerSpec |
struct | The tokenizer and how a text is framed for the model. |
Rows |
struct | A checked row pack held in memory. |
Element |
enum | Element type of a token table. |
Pooling |
enum | How the graph's output becomes one vector per text. |
PackError |
enum | Why a model directory's file was refused. |
encode_rows |
function | Writes a row pack. |
revision |
function | The revision of a loaded model: a digest of its description, which lists the digest of every file, and of the languages loaded. |
MANIFEST_FILE |
constant | Name of the model description inside a model directory. |
MAX_LANGUAGES |
constant | Most language packs one model lists. |
MAX_LITERAL_PIECES |
constant | Most literal pieces a tokenizer lists (bekko-a8m has 142). |
MAX_MANIFEST_BYTES |
constant | Most bytes of model.json that are read. |
MAX_TABLES |
constant | Most token tables one model has (bekko-a8m has two, Harrier three). |
PACK_FORMAT |
constant | Layout version this crate reads and writes (format in model.json). |
MAX_ROWS |
constant | Most rows one pack holds. Vocabularies stay well under this (bekko-a8m has 256,000 tokens). |
MAX_ROW_BYTES |
constant | Most bytes one row has (bekko-a8m's is 388, Harrier's 410). |
ROWS_HEADER_BYTES |
constant | Bytes before the token ids: the magic and the two counts. |
ROWS_MAGIC |
constant | First eight bytes of every row pack. |
pub struct GraphSpec {
pub model: PackFile,
pub ids_input: String,
pub mask_input: String,
pub output: String,
pub pooling: Pooling,
}
The cut ONNX graph and the names it uses.
Fields
model: The graph file.ids_input: Input that takes token positions: 64-bit integers[1, tokens], numbered0..tokensbecause the tables passed hold one row per token of the text, in order.mask_input: Input that takes the attention mask: 64-bit ones,[1, tokens].output: Output read after a run: 32-bit floats.pooling: How that output becomes the text's vector.
Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct LanguageTag(/* private */);
A language tag naming a language pack: lowercase letters and digits in
--separated parts of 1 to 8 characters, at most 35 bytes (en,
pt-br).
pub fn as_str(&self) -> &str
The tag as text.
Implements: Clone, Debug, Deserialize<'de>, Eq, From<LanguageTag>, Hash, Ord, PartialEq, PartialOrd, Serialize, TryFrom<String>
pub struct PackFile {
pub file: PackFileName,
pub bytes: u64,
pub sha256: Sha256Hex,
}
One file of a model directory, with what its content must be.
Fields
file: Name inside the model directory.bytes: Length in bytes.sha256: SHA-256 of the content.
pub fn describe(file: PackFileName, content: &[u8]) -> Self
Describes content under the name file.
pub fn verify(&self, content: &[u8]) -> Result<(), PackError>
Checks that content is the file the description lists.
PackError::FileMismatch when the length or the digest differs.
Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct PackFileName(/* private */);
A file name inside a model directory: 1 to 64 lowercase letters,
digits, ., _ and -, starting with a letter or digit.
The grammar leaves no way to name a parent directory, a hidden file or a path, so joining it to the model directory stays inside it.
pub fn as_str(&self) -> &str
The name as text.
Implements: Clone, Debug, Deserialize<'de>, Eq, From<PackFileName>, Hash, Ord, PartialEq, PartialOrd, Serialize, TryFrom<String>
pub struct PackManifest {
pub format: u32,
pub model_id: String,
pub dimensions: u32,
pub max_tokens: u32,
pub graph: GraphSpec,
pub tokenizer: TokenizerSpec,
pub tables: Vec<TableSpec>,
pub base_rows: PackFile,
pub languages: BTreeMap<LanguageTag, PackFile>,
}
model.json: one embedding model, cut for row packs.
Fields
format: Layout version;PACK_FORMAT.model_id: Model id within the plugin (bekko-a8m).dimensions: Numbers per vector.max_tokens: Tokens read from one text, counting the framing pieces; the rest of the text is cut off.graph: The cut graph.tokenizer: The tokenizer.tables: The token tables, in the order their rows are concatenated in a row pack.base_rows: Rows every text needs.languages: The row pack of each language, present on disk or not.
pub fn parse(bytes: &[u8]) -> Result<Self, PackError>
Reads and checks a model.json.
bytes- The file's content.
PackError::ManifestTooLarge, PackError::ManifestSyntax,
PackError::UnsupportedFormat or the first
PackError::ManifestField rule broken.
pub fn to_json(&self) -> Result<Vec<u8>, PackError>
Writes the description as JSON that PackManifest::parse reads.
The first rule the description breaks, as PackManifest::parse
would report it.
pub fn row_bytes(&self) -> usize
Bytes of one row of a row pack: every table's row, concatenated.
Within 1 to MAX_ROW_BYTES for a checked description.
Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct Sha256Hex(/* private */);
A SHA-256 digest as 64 lowercase hexadecimal characters.
pub fn of(bytes: &[u8]) -> Self
The digest of bytes.
pub fn as_str(&self) -> &str
The digest as text.
Implements: Clone, Debug, Deserialize<'de>, Eq, From<Sha256Hex>, Hash, PartialEq, Serialize, TryFrom<String>
pub struct TableSpec {
pub input: String,
pub element: Element,
pub width: u32,
}
One token table the graph takes as an input: [tokens, width].
Fields
input: Graph input that takes the rows.element: Element type.width: Elements per row.
pub fn row_bytes(&self) -> Option<usize>
Bytes one row of this table takes, or None on overflow.
Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct TokenizerSpec {
pub model: PackFile,
pub literal_pieces: Vec<String>,
pub prefix_pieces: Vec<String>,
pub suffix_pieces: Vec<String>,
pub document_prefix: String,
pub query_prefix: String,
}
The tokenizer and how a text is framed for the model.
Fields
model: The SentencePiece Lite model with the whole vocabulary.literal_pieces: Pieces matched in the text before the tokenizer runs, longest match first; each becomes its own token and the text between them is tokenized segment by segment. A segment that starts with a space loses that space (a segment of only that space becomes the word boundary piece), as the published tokenizer does.prefix_pieces: Pieces put before the text's tokens (<bos>).suffix_pieces: Pieces put after the text's tokens (<eos>).document_prefix: Text put before a document before tokenizing.query_prefix: Text put before a search query before tokenizing.
Implements: Clone, Debug, Deserialize<'de>, Eq, PartialEq, Serialize
pub struct Rows {
/* private fields */
}
A checked row pack held in memory.
Built by Rows::parse; afterwards every lookup is a binary search
over the id column and cannot fail on the pack's content.
pub fn parse(bytes: Vec<u8>, row_bytes: usize) -> Result<Self, PackError>
Checks a row pack and takes ownership of it.
bytes- The whole file.row_bytes- Bytes per row the model's tables add up to.
The first rule of the layout the file breaks; see PackError.
pub fn len(&self) -> usize
Rows in the pack.
pub fn is_empty(&self) -> bool
Whether the pack has no rows.
pub fn row_bytes(&self) -> usize
Bytes per row.
pub fn ids(&self) -> impl Iterator<Item = u32> + '_
The token ids in the pack, ascending.
pub fn row(&self, id: u32) -> Option<&[u8]>
The row of token id.
id- A token id of the model's vocabulary.
The row's Rows::row_bytes bytes, or None when the pack does not
hold the token.
Implements: Clone, Debug, Eq, PartialEq
pub enum Element {
I8,
U8,
F32,
}
Element type of a token table.
Variants
I8: Signed 8-bit integers.U8: Unsigned 8-bit integers.F32: Little-endian IEEE 754 32-bit floats.
pub fn size(self) -> usize
Bytes one element takes.
Implements: Clone, Copy, Debug, Deserialize<'de>, Eq, Hash, PartialEq, Serialize
pub enum Pooling {
Mean,
None,
}
How the graph's output becomes one vector per text.
Variants
Mean: The output has one vector per token ([1, tokens, dimensions]); the text's vector is their mean.None: The output already is the text's vector ([1, dimensions]).
Implements: Clone, Copy, Debug, Deserialize<'de>, Eq, Hash, PartialEq, Serialize
pub enum PackError {
ManifestTooLarge,
ManifestSyntax(String),
UnsupportedFormat(u32),
ManifestField { field: &'static str, reason: &'static str },
FileMismatch(String),
RowsTruncated,
RowsMagic,
RowsWidth { expected: usize, found: usize },
RowsTooLarge,
RowsLength,
RowsOrder,
}
Why a model directory's file was refused.
Variants
ManifestTooLarge:model.jsonis longer thanMAX_MANIFEST_BYTES.ManifestSyntax:model.jsonis not the JSON document this crate reads.UnsupportedFormat:model.jsonwas written for another layout version.ManifestField: A field ofmodel.jsonbreaks a rule; the text names the field.field: Dotted path of the field (graph.output).reason: The rule it breaks.
FileMismatch: A file's length or SHA-256 differs from whatmodel.jsonlists.RowsTruncated: A row pack is shorter than its header.RowsMagic: A row pack does not start withROWS_MAGIC.RowsWidth: A row pack's rows are not the length the model's tables give.expected: Row length the model's tables add up to.found: Row length the pack's header states.
RowsTooLarge: A row pack's row count or row length is over the limits.RowsLength: A row pack's length differs from what its header implies.RowsOrder: A row pack's token ids are not strictly ascending.
Implements: Clone, Debug, Display, Eq, Error, PartialEq
pub fn encode_rows(row_bytes: usize, rows: &[(u32, &[u8])]) -> Result<Vec<u8>, PackError>
Writes a row pack.
row_bytes- Bytes per row.rows- Token id and row, in strictly ascending id order.
The file, which Rows::parse accepts.
PackError::RowsTooLarge over the limits, PackError::RowsOrder
when the ids do not ascend, and PackError::RowsWidth for a row of
another length.
pub fn revision(manifest_bytes: &[u8], languages: &[LanguageTag]) -> String
The revision of a loaded model: a digest of its description, which lists the digest of every file, and of the languages loaded.
Two outputs are comparable only when their revisions match, so the revision changes when the weights, the tokenizer or the set of loaded language packs change.
manifest_bytes- The bytes ofmodel.json.languages- Tags of the language packs loaded, in any order.
32 lowercase hexadecimal characters.
pub const MANIFEST_FILE: &str = "model.json";
Name of the model description inside a model directory.
pub const MAX_LANGUAGES: usize = 64;
Most language packs one model lists.
pub const MAX_LITERAL_PIECES: usize = 1024;
Most literal pieces a tokenizer lists (bekko-a8m has 142).
pub const MAX_MANIFEST_BYTES: usize = _;
Most bytes of model.json that are read.
pub const MAX_TABLES: usize = 8;
Most token tables one model has (bekko-a8m has two, Harrier three).
pub const PACK_FORMAT: u32 = 1;
Layout version this crate reads and writes (format in model.json).
pub const MAX_ROWS: usize = 4_000_000;
Most rows one pack holds. Vocabularies stay well under this (bekko-a8m has 256,000 tokens).
pub const MAX_ROW_BYTES: usize = _;
Most bytes one row has (bekko-a8m's is 388, Harrier's 410).
pub const ROWS_HEADER_BYTES: usize = 16;
Bytes before the token ids: the magic and the two counts.
pub const ROWS_MAGIC: [u8; 8] = _;
First eight bytes of every row pack.
Generated from crates/bookclerk-model-pack at main · View as Markdown