bookclerk_scan_delta
Hash-based scan deltas for storefronts with no change feed.
A store such as Libro.fm, Chirp or GraphicAudio can only list a whole
library, so every pass fetches every store page. ScanDelta keeps a
fingerprint of each title the last pass listed in the plugin's own
database (the SCAN_STATE_BINDING [[databases]] binding) and lets the
plugin send the host only the titles that are new or changed, plus the
product ids that disappeared. The store requests are the same; the host
receives, compares and writes far less.
- An empty
syncTokenis a full pass. Every title goes to the host, and the pass records one row per title. - The pass's last page marks the pass finished and returns a token
naming it (
hd1.<generation>). The host sends it on the account's next pass. - A pass with a token is a delta pass. Each page sends only titles whose
product id and fingerprint the named pass does not hold. The last page
lists the named pass's product ids that no page of this pass listed as
removed, marks this pass finished and returns its token. A delta pass is nevercomplete.
Two tables, created by [migrations]:
scan_delta_titles: one row per title a pass listed (account, pass generation, store page, product id, fingerprint).scan_delta_passes: one row per pass, finished or not, with the time it was last touched.
The account and store page columns hold hashes, never the account id or
the store's cursor. The generation of a pass is derived from the account
and the token the pass started with, so every call of one pass, resumed
or not, agrees on it without keeping state. The call that starts a pass
(empty cursor) deletes the account's rows other than the pass its token
names, and every pass of any account untouched for
RETENTION_SECONDS. A finished pass is replaced only by a later pass,
so a host that lost the last page of a pass and sends the older token
again still finds the pass it needs.
Every write is split with the binding's chunk() and every read is
sized from the session's limits (one streamed read when the session
streams, key-ordered pages when it does not), so the same code runs on
the shipped adapters, on D1 and on the smallest advertisement a host
accepts. See docs/plugins/database-limits.md.
- No binding (the operator has not approved it yet): every pass is a full pass with an empty token, as before.
- A token this crate did not write, or one whose pass is gone or never
finished:
invalid_cursor, so the host restarts once as a full pass. - A pass whose row was cleared while it ran (another pass of the account
started): its last page returns
invalid_cursorinstead of removals worked out from a partial record, and the host starts over.
| Item | Kind | Summary |
|---|---|---|
DeltaPage |
struct | What one store page of a pass sends to the host. |
ScanDelta |
struct | Hash-based delta state for one call of one account's pass. |
migrations |
function | The database_migrations registration for SCAN_STATE_BINDING. |
migrations_for |
function | The database_migrations answer for binding. |
read_keyed |
function | Every row a key-ordered query returns, read within the session's limits. |
SCAN_STATE_BINDING |
constant | [[databases]] binding first-party storefronts declare for this crate's state. |
RETENTION_SECONDS |
constant | Seconds a pass is kept after it was last touched. The host asks for a full pass at least every seven days, so an older pass is never the base of a delta; this only reclaims accounts that stopped scanning. |
pub struct DeltaPage {
pub books: Vec<ScanBook>,
pub removed: Vec<String>,
pub sync_token: String,
pub complete: bool,
}
What one store page of a pass sends to the host.
Fields
books: Titles to send: all of them on a full pass, the new or changed ones on a delta pass.removed: On the last store page of a delta pass: product ids the named pass held and no page of this pass listed, sorted.sync_token: On the last store page: the token for the account's next pass, empty when the next pass must be a full pass.complete: On the last store page: true for a full pass, which listed every title the account holds.
Implements: Clone, Debug, Default, Eq, PartialEq
pub struct ScanDelta<'a> {
/* private fields */
}
Hash-based delta state for one call of one account's pass.
Build it with ScanDelta::open at the top of scan, then pass every
store page through ScanDelta::page.
pub async fn open(
db: Option<&'a DatabaseBinding>,
params: &ScanParams,
) -> Result<Self, PluginError>
Reads the pass's token and, on the call that starts a pass, clears the account's stale rows.
db- The plugin's scan-state database, when the operator granted it.params- The scan call's params (account, cursor, sync token).
Returns invalid_cursor for a token this crate did not write or
whose pass is gone or unfinished, so the host restarts as a full
pass, and the binding's error when a database call fails.
pub async fn page(
&self,
store_cursor: &str,
books: Vec<ScanBook>,
last: bool,
) -> Result<DeltaPage, PluginError>
Records one store page and filters it; on the last one, ends the pass.
The same store page may come through more than once (the SDK's
scan_page helper fetches a large page again for each slice); each
time gives the same answer.
store_cursor- The plugin's own cursor for this store page (empty for the first).books- Every title the store page lists, in store order.last- Whether this is the last store page of the pass.
Returns invalid_cursor on the last store page when another pass of
the account cleared this one while it ran, and the binding's error
when a database call fails.
Implements: Debug
pub fn migrations() -> Vec<PluginMigration>
The database_migrations registration for SCAN_STATE_BINDING.
Applied once, in order, before the binding is handed to the plugin.
pub fn migrations_for(binding: &str) -> Vec<PluginMigration>
The database_migrations answer for binding.
migrations for SCAN_STATE_BINDING, nothing for any other binding.
pub async fn read_keyed(
db: &DatabaseBinding,
sql: &str,
binds: Vec<DbValue>,
key: &str,
) -> Result<Vec<HashMap<String, DbValue>>, PluginError>
Every row a key-ordered query returns, read within the session's limits.
sql ends AND <key> > ? ORDER BY <key> LIMIT ?, where <key> is the
unique text column named by key, which the query also returns; binds
fills the placeholders before those two. When the session streams, one read with no
lower bound and no row limit returns it all. When it does not, every read
is held to the inline caps, so the rows come in pages keyed on key
(never OFFSET), each sized from maxResultRows, halved whenever a
page's bytes pass maxResultBytes and doubled again (up to
maxResultRows) after each full page.
db- The plugin's database binding.sql- The query, shaped as above.binds- Values for the placeholders before the key bound and limit.key- Name of the returned text column the rows are ordered on.
The rows in key order, each as column name to value.
Returns the binding's error, limit_exceeded included when a single row
passes the inline caps, and internal when a paged row has no text
key column.
pub const SCAN_STATE_BINDING: &str = "SCAN_STATE";
[[databases]] binding first-party storefronts declare for this crate's
state.
pub const RETENTION_SECONDS: i64 = _;
Seconds a pass is kept after it was last touched. The host asks for a full pass at least every seven days, so an older pass is never the base of a delta; this only reclaims accounts that stopped scanning.
Generated from crates/bookclerk-scan-delta at main · View as Markdown