bookclerk_plugin_ai_embedder::text
From a text to the model's token ids.
TextEncoder reproduces what the model's published tokenizer does
around SentencePiece: the text prefix for the text's role, the split on
literal pieces, the token limit and the framing pieces. The ids it
gives for a text are the published model's ids.
| Item | Kind | Summary |
|---|---|---|
TextEncoder |
struct | A model's tokenizer with the framing its description asks for. |
TextError |
enum | Why a tokenizer could not be set up or a text not encoded. |
Role |
enum | What a text is to the model; some models put a different prefix before a search query than before a document. |
pub struct TextEncoder {
/* private fields */
}
A model's tokenizer with the framing its description asks for.
pub fn new(
model: Vec<u8>,
spec: &TokenizerSpec,
max_tokens: u32,
) -> Result<Self, TextError>
Sets a tokenizer up for a model.
model- The tokenizer file's bytes, already checked against the model description.spec- The tokenizer part of the model description.max_tokens- The model's token limit, framing included.
When the bytes are not a tokenizer model, or the description names a piece the vocabulary does not have.
pub fn body_limit(&self) -> usize
Most ids of the text itself, the framing not counted.
pub fn required_ids(&self) -> impl Iterator<Item = u32> + '_
Ids the framing and the fallback always need: the framing pieces, the unknown token and the word boundary.
pub fn body_ids(&self, text: &str, role: Role) -> Result<Vec<u32>, TextError>
The ids of a text, without the framing pieces.
text- The text as the caller gave it.role- Which text prefix goes before it.
At most TextEncoder::body_limit ids; a longer text is cut off.
When the tokenizer runtime fails on the text.
pub fn frame(&self, body: Vec<u32>) -> Vec<u32>
Puts the framing pieces around a text's ids.
pub fn spell(&self, id: u32) -> Vec<u32>
Spells a token with single-character pieces, for a token whose table row is not loaded.
id- A token id the tokenizer gave.
One id per character of the token's piece: the word boundary for a boundary mark, the character's own piece when the vocabulary has one, the unknown token otherwise. An id outside the vocabulary gives the unknown token alone.
pub fn unknown_id(&self) -> u32
Id of the unknown token.
Implements: Debug
pub enum TextError {
Tokenizer(TokenizerError),
MissingPiece(String),
NoUnknownPiece,
}
Why a tokenizer could not be set up or a text not encoded.
Variants
Tokenizer: The tokenizer file is not a SentencePiece Lite model, or the runtime failed on a text.MissingPiece: The model description names a piece the vocabulary does not have.NoUnknownPiece: The vocabulary has no unknown-token piece to fall back on.
Implements: Clone, Debug, Display, Eq, Error, From<TextError>, From<TokenizerError>, PartialEq
pub enum Role {
Document,
Query,
}
What a text is to the model; some models put a different prefix before a search query than before a document.
Variants
Document: Stored content.Query: A search query matched against documents.
Implements: Clone, Copy, Debug, Eq, PartialEq
Generated from crates/bookclerk-plugins/platform/ai-embedder at main · View as Markdown