bookclerk_plugin_ai_embedder
Text-embedding model plugin.
The platform embedder guest serves text-embedding models on the ai
entrypoint (docs/adr/ai-models.md). It has no network access and no
settings: it serves every model directory it finds under models/ in
its data directory, with the language packs present there. The host
puts verified model files in that directory and removes them.
A run goes through three modules:
textturns a text into the model's token ids with the whole vocabulary (bookclerk-sentencepiece).modellooks each token's table rows up in the row packs on disk (bookclerk-model-pack) and spells a token whose row is missing with smaller pieces.engineruns the cut ONNX graph on those rows with the pinned ONNX Runtime shared library that sits beside the plugin's executable.
The layout of a model directory is described in
docs/internals/model-packs.md.
| Item | Kind | Summary |
|---|---|---|
ID |
constant | Plugin id advertised in describe(); models are named @embedder/<model id>. |
MODELS_DIR |
constant | Directory under the plugin's data directory that holds one directory per model. |
RUNTIME_LIBRARY |
constant | Name of the ONNX Runtime shared library beside the plugin executable on this platform (the same name bookclerk-onnxruntime installs). |
engine: Running a cut graph with ONNX Runtime.model: A model directory: what is in it, and the model loaded from it.plugin: The plugin'saientrypoint over the model directories on disk.text: From a text to the model's token ids.
pub const ID: &str = "embedder";
Plugin id advertised in describe(); models are named @embedder/<model id>.
pub const MODELS_DIR: &str = "models";
Directory under the plugin's data directory that holds one directory per model.
pub const RUNTIME_LIBRARY: &str = _;
Name of the ONNX Runtime shared library beside the plugin executable on
this platform (the same name bookclerk-onnxruntime installs).
Generated from crates/bookclerk-plugins/platform/ai-embedder at main · View as Markdown