Embedding model packs
This page describes how a text-embedding model is stored on disk for the
first-party embedder plugin, how the plugin runs it, and how the plugin
was checked against the published models. The decision behind it is in
Models run in plugins, and the choice of models in
Embedding model benchmarks.
Note
The embedder plugin ships with Bookclerk and is on by default, but no
model is delivered with it yet. Until model delivery lands, the plugin
lists no models, and Discover keeps ranking with hashed words.
Most of a multilingual embedding model is its token table: one row of numbers for each of about 256,000 tokens. bekko-a8m's published file is 130 MB, of which the transformer is 31 MB. A library that holds books in two languages never reads most of those rows.
So the table is taken out of the graph and stored as row packs:
- every text needs the base rows: the special tokens, single characters and byte pieces;
- each language adds a language row pack with the tokens its text uses.
A node stores the graph, the tokenizer, the base rows and the row packs of the languages it was given.
The pack builder finds every vocabulary-sized table the graph reads with
input_ids and turns it into a graph input. Nothing else in the graph
changes. At run time the plugin:
- tokenizes the text with the full vocabulary, so token ids are the published model's own;
- looks up each token's row in the loaded row packs;
- passes the rows as the table inputs, one row per token in order, and
input_idsas0, 1, 2, …, so the graph's own lookup picks row n for token n; - pools the output as the description says and scales the vector to length 1.
The same code serves both chosen models, with no model-specific Rust:
| Model | Lookup node | One row | Row bytes | Output |
|---|---|---|---|---|
| bekko-a8m | Gather |
384 signed bytes and 1 float scale | 388 | one vector per token, averaged |
| Harrier 270m (4-bit) | GatherBlockQuantized |
320 packed bytes, 20 float scales and 10 zero points | 410 | the text's vector |
Models live in the plugin's data directory, one directory per model id:
models/
bekko-a8m/
model.json the model description
core.onnx the cut graph
tokenizer.spm.fb the SentencePiece Lite model, whole vocabulary
base.rows rows every text needs
en.rows rows for English (one file per language)
model.json (PackManifest in bookclerk-model-pack) holds:
- the layout version, the model id, the vector size and the token limit;
- the graph's input and output names and how its output is pooled;
- the tokenizer's framing pieces (
<bos>,<eos>), the pieces matched before tokenizing, and the text put before a document or a query; - the tables, in the order a row concatenates them;
- the name, size and SHA-256 of every other file, including each language's row pack whether it is on disk or not.
A file name is 1 to 64 lowercase letters, digits, ., _ and -,
starting with a letter or digit, so no name can leave the directory.
Unknown fields and unknown layout versions are refused.
A row pack is one little-endian file:
| Bytes | Field |
|---|---|
| 8 | the magic BCROWS1 and a zero byte |
| 4 | count: rows in the pack |
| 4 | row_bytes: bytes per row |
4 × count |
token ids, strictly ascending |
row_bytes × count |
rows, in the order of the ids |
One row is the token's row of every table, concatenated. The reader checks the whole layout once; after that a lookup is a binary search.
Vectors made by different models, or by the same model with different
languages loaded, are never compared. The plugin reports a revision
with every reply: the first 32 hexadecimal characters of a SHA-256 over
the digest of model.json and the sorted language tags whose row packs
are loaded. The host stores it with each vector.
modelslists the directories whosemodel.jsonis valid. It reads only that file, so it is cheap.runloads the model on first use: it checks every file against the size and SHA-256 inmodel.json, then starts one ONNX Runtime session with one thread. A model stays in memory until the plugin sleeps, and is loaded again when the files on disk give another revision.healthreports whether the ONNX Runtime library is beside the plugin, how many models are installed, and any directory it skipped.
A token with no row in the loaded packs is spelled with single-character
pieces, which the base rows always hold. The vector is still returned,
and the text's index is listed in the reply's fallbacks, so the caller
knows the vector is weaker and can embed the text again once the
language is added.
- ONNX Runtime is the upstream build, pinned by version and SHA-256
in
crates/bookclerk-onnxruntime/onnxruntime-pin.jsonfor Linux x86-64 and arm64, macOS arm64 and Windows x86-64.cargo ensure-onnxruntimedownloads it; staging and packaging put the shared library beside the plugin's executable, and the plugin loads it from there. There is no build for Intel macOS, so the plugin reports itself unhealthy there and serves no model. - SentencePiece Lite is compiled from the sources vendored under
third_party/sentencepiece-lite/(scripts/vendor-sentencepiece-lite.shrefreshes them).
Both are C++ and run only inside the plugin's jail, which has no network grant. Neither is linked into a host process.
Each model was cut and packed, then 69 texts were embedded twice: once by the plugin's code from the pack, and once by the published ONNX file with the published tokenizer. The texts were 60 Open Library descriptions, 7 multilingual and odd texts (Japanese, mixed scripts, emoji, blank lines) and 2 queries.
| Model | Largest difference in any number | Texts equal to 1e-6 |
|---|---|---|
| bekko-a8m | 1.8e-7 | 69 of 69 |
| Harrier 270m (4-bit) | 3.1e-7 | 69 of 69 |
With half the vocabulary removed from the packs, all 69 texts were
reported in fallbacks and the lowest cosine similarity to the true
vector was 0.57, which is why a fallback vector is flagged and not
trusted.
What a node stores before any language is added:
| Model | Cut graph | Tokenizer | Base rows | Total |
|---|---|---|---|---|
| bekko-a8m | 30.8 MB | 9.5 MB | 8.3 MB (21,225 tokens) | 48.6 MB |
| Harrier 270m (4-bit) | 98.2 MB | 10.5 MB | 10.7 MB (25,895 tokens) | 119 MB |
These runs used the pack tools on a development machine. The workflow that builds, checks and signs packs in CI comes with model delivery.
- The plugin's tests run a tiny model (a few kilobytes, the same inputs as a real one) through the pinned ONNX Runtime and compare every vector with numbers worked out apart from the plugin. They cover language packs, fallbacks, long texts, changed files and reloads.
- The tokenizer's tests compare token ids with ids the full SentencePiece library recorded.
- The staged-plugin suite installs the tiny model and runs the plugin inside its jail on Linux.
- The
model_packfuzz target checks themodel.jsonand row-pack readers against a layout check written apart from them (see Fuzzing).
- Models run in plugins for the
aientrypoint and theAIbinding. - The guest jail for how the plugin is confined.
Source: docs/internals/model-packs.md at main · View as Markdown