ADR: Event delivery follows the Workers model
- Status: Accepted (decisions 1 to 13); later phases proposed.
Amended 2026-11-18: decision 7 covers every storage-only plugin, not
only
localands3 - Date: 2026-11-11
Plugins should be able to sleep while idle, as Cloudflare Workers do, and the system should stay event-driven. That only works if the host decides which plugins an event is for, and if a plugin that isn't running still receives its events.
The host already matched events to declared [[events.consumers]] rows,
so nothing was broadcast to every plugin. Two things blurred the model:
- Subscription followed process liveness. The subscriber catalog marked a plugin enabled only while a node had its guest loaded. An instance that was deployed but not running got no deliveries at publish time and relied on the late-join reconcile to catch up.
- One word for two ideas. Docs and code said a node claims both jobs and event deliveries. A job is a command with one handler. An event is a fact that every subscriber receives. Using the same verb suggested subscribers compete for an event, which they never do.
Bookclerk follows the Cloudflare Workers model, mapping each trigger to one Workers primitive:
| Bookclerk | Workers model | Delivery rule |
|---|---|---|
[[events.consumers]] |
Events fanned out to one queue per subscriber, drained by that Worker's queue() handler |
Every subscribed instance gets its own copy |
[[queues.consumers]] |
A Queues consumer | Each message goes to the queue's one consumer |
job(job) |
A Workflow step, or a command sent to one Worker | One command, one handler |
[triggers] crons |
Cron Triggers | One fire per instance, library-wide |
| Storefront, storage, metadata and database entrypoints | Service binding RPC to a WorkerEntrypoint |
The caller awaits a reply |
Everything is asynchronous. No host thread blocks on a plugin; RPC callers await a promise, and long work is a durable job.
-
Subscriptions belong to the deployment. Each node heartbeats a catalog row for every instance its deployments run (
presentorplaced), with the consented subscriptions of what it would run (the active installed version and its grant, or a host-local tree), whether or not the guest is up. A matching event gets a delivery for that instance at publish time, and the delivery waits aspendinguntil the instance runs. Fan-out stays per instance. -
Deliveries are leased, jobs are claimed. A lease on a delivery only picks the node that runs one subscriber's copy. The host API is
lease_next_event_delivery. The persisted indexidx_event_deliveries_claimand the named atomic operationsclaimEventDelivery/claimNextEventDelivery, which database plugins implement, keep their names, since renaming them is a contract change with no behavior gain. -
Deliveries have a retention period. As with a Queues message, a delivery that was never handed to its subscriber expires (state
expired) once it has been due for longer than the consumer'smessage_retention_period: 60 seconds to 14 days, one day by default, and never more than[events].retention_days. Leased, retried and suspended deliveries keep followingmax_retriesand dead letters. Host and node consumers never expire, and a subscriber with no live catalog row waits the whole events window. Expiry is a best-effort hourly sweep that bounds the backlog, not a delivery deadline. Expired rows are kept for the events retention window as the record of what a subscriber missed. -
Plugins read the event stream to catch up. A plugin pages past events from the outbox with
env.EVENTS.read({ after | since, types, limit })(EventPublisher.read), keeping an opaque(created_at, id)cursor itself. A read returns only events of the caller's own library that the instance's consented subscriptions match by type, schema version and filter, within the invocation's account scope, and never older than[events].retention_days(truncatedsays the start was older). Each page names the events whose delivery to this plugin expired (expiredEventIds). A read is not a delivery: it changes no delivery state,deliveryAttemptis zero and the deduplication key is the event id. TheEVENTSbinding therefore exists for a consented consumer as well as a producer. Each read scans a bounded number of outbox rows and may return a short page withmoreset. A read stops 30 seconds short of now, becausecreated_atis stamped before the insert commits and a cursor must never pass a row that has yet to appear; the outbox refuses caller-chosen event ids that a cursor cannot name. The guest request and its Cap'n Proto codecs are a fuzzed boundary (event_read). -
On-demand activation, first step. Cold start was measured with the ignored
cold_starttest inbookclerk-plugin-e2e. Ready times on a 1-CPU, 1 GiB machine were about 60 ms for native guests, 75 ms for workerd JavaScript and 4 s for workerd Python. Runtime, not family, decides the cost. A plugin whose only family is integration and that exports noremoteLibraryoroidcentrypoint and is not Python is registered behind an on-demand proxy. The proxy answers identity, subscriptions, schedules and health while the guest sleeps, starts the guest on the next delivery, queue batch, cron fire or lookup, and stops it after 60 seconds with no call in flight. Placement keeps reserving budget for asleep instances. -
On-demand storefronts. A plugin whose only family is source and that is not Python sleeps the same way, behind a source proxy that shares the integration proxy's activation core. Its idle timeout is 5 minutes, since storefront work comes in bursts. Every storefront call returns once its work is done (a fetch hands back files on disk), so a call in flight is all that keeps the guest up. Account listing reads the library and never wakes the guest.
-
On-demand storage destinations. The first-party
localands3destinations, whose host side is a storage backend, sleep behind a storage proxy on the same activation core, with a 5-minute idle timeout. Every backend call starts the guest when it sleeps. A write commits inside one call, so the call in flight holds the guest from upload to commit. A read stream carries its own hold and keeps the guest up until it is dropped, and a job that lends the guest's session holds it the same way. The backend's name, instance id, scan placement and copy support are captured from the first guest, so placement and dedupe never wake it. Other storage guests have no host backend to proxy and stay running. (Amended 2026-11-18: every storage guest now has a host backend and is a destination, so every storage-only guest sleeps this way; a guest that also belongs to another family stays running.) -
Pushed sockets are host-brokered. A plugin that listens on a WebSocket can still sleep, as with Durable Objects' hibernatable WebSockets. It opens sockets through a
[sockets]binding; the host holds each one, answers declared heartbeat pairs itself, and wakes the plugin'sSocketHandleronly for a real message, handing back up to 2 KiB of attachment the plugin stored with the socket. Sockets are held by a jailed broker per plugin instance (the front door in a broker mode, under the plugin's network grant and egress checks), not by the daemon, so a socket gives a plugin no reach its grant lacks and untrusted frames are never parsed in the daemon. Remote libraries without a push channel move to a host-scheduled poll instead. The ABI shape and checks land first; the broker, the SDK WebSocket API and on-demand remote libraries woken by socket messages follow. -
On-demand remote libraries and sign-in providers. Every integration that is not Python sleeps behind the integration proxy, including plugins that export
remoteLibraryoroidc. The proxy owns the remote library's new-user poll (pollEvents): it wakes the guest every 5 minutes, several idle timeouts apart, and hands each user to the daemon's hook. Woken guests never get the hook, so a guest runs no poll of its own. A resident Python guest is still polled every 30 seconds through its session. A guest keeps what must outlive a sleep in its plugin database: Audiobookshelf stores a hash of each user it has reported, so a user created while it slept is still reported. AremoteLibraryguest therefore sleeps only when it declares a[[databases]]binding and its grant holds every one; otherwise it stays resident. A sign-in call wakes the guest like a lookup, and the guest's first successfuloidcClientsanswer is kept per proxy, since a changed package or config replaces the proxy. -
Pushed new users. A remote library may also learn of new users from a pushed socket (decision 8). Its
SocketHandler.messagereply (SocketMessageOk) carriespollEvents; when the guest sets it, the host runs the user poll at once, on the guest the message already woke, instead of waiting out the interval. Only the plugin can tell which of a service's messages announce a user (Audiobookshelf pushes many unrelated events to admins), so the plugin decides, and the scheduled poll stays as the fallback that catches anything a closed socket missed. Asks that arrive while a poll runs fold into one more poll; an ask with no poll running is dropped. -
Python start-up snapshots. A Python isolate took about 4 s to start on one CPU, too slow to sleep. workerd can save the isolate's memory after Pyodide loaded and the plugin imported its modules (a dedicated snapshot) and start later isolates from it: measured on the echo plugin with the Python SDK, about 0.6 s on one CPU (a snapshot of Pyodide alone, without the plugin's imports, gave about 1.9 s). Through the whole host path (jail, launcher, handshake and first
describe) the echo plugin's start went from about 4.8 s to about 1.2 s on one CPU. The host keeps one dedicated snapshot per installed plugin version, keyed by the workerd release and the install's files, in host-owned plugin state. The guest's jail reads only that file and writes only a fresh staging directory; the launcher saves there when there is nothing to load or the stored snapshot fails to start, and marks the file ready once the isolate answered, and the host promotes only a ready regular file. A missing or broken snapshot costs only the slow start. -
Python guests sleep. With a snapshot, a Python guest starts in about 1.2 s on one CPU through the whole host path, against about 0.1 s for a native or JavaScript guest. That is slower, but a sleeping plugin's start happens once per burst of work and its idle timeout keeps it up through the burst, so Python integrations, storefronts and the storage destinations sleep under the same rules as the others (decisions 5 to 9, which this lifts the Python exclusion from). A
remoteLibraryguest still sleeps only when its databases are granted. The first start of a new plugin version still takes the slow path while it saves the snapshot. -
A live watch keeps the local destination awake. One process may stay up while idle: the
localdestination with itswatchsetting on. Its change feed is a file-system watcher whose journal lives in the guest's memory, so stopping the guest would lose the changes it has not handed out yet. Achangespage carriesliveWatchfor such a feed. While the latest page from the running guest hascompleteandliveWatchboth true, the on-demand proxy holds one call's worth of activity on that guest (aBusyguard from the activation core), so the idle sweep leaves it running. An incomplete or non-live page, achangeserror, or the guest exiting drops the hold, and the guest then sleeps after the usual idle timeout. The watcher runs only on a local disk; on a network share or FUSE mount, or when the operating system refuses or overflows the watch, its pages saycomplete: falseand the guest sleeps like any other. The host honoursliveWatchonly from the first-party storage plugins it starts on demand, and only thelocalpackage sets it; a durable feed, such as a queue an object-store plugin drains, leaves it false. Measured on Linux with release builds: the idle guest is resident at about 3.7 MiB; a watch over a 1,000-book library (1,201 folders) adds about 0.6 MiB, one over 10,000 books (12,001 folders) about 3 MiB, and a full journal (16,384 unread changes) about 3 MiB more. The kernel also keeps roughly 1 KiB per watched folder for inotify, outside the guest.
The following phases are proposed and get their own pull requests and updates to this record:
- On-demand activation for the rest. The remaining storage guests sleep too, with an idle timeout per family.
- Batched delivery.
event(batch)andqueue(batch)receive up tomax_batch_sizemessages per call. - Richer host-side filters. Envelope selectors and per-instance narrowing that can only intersect the consented subscription.
- An instance that is deployed but failing to start holds
pendingdeliveries for the events it subscribes to only for its retention period. Deliveries that are still due when it runs drain then; older ones expire, and the plugin catches up from the event stream withenv.EVENTS.read, or resyncs from its source of truth when the gap is older than the events window. An operator who gives up on it undeploys it (its row expires within the 60 s catalog TTL). - A consumer that is running but falls behind by more than its retention
period also loses the oldest never-leased deliveries. Consumers that
expect long backlogs declare a longer
message_retention_period. - Events are scoped to one library, the library that shares its database. Every subscriber and every delivery belongs to that library, and an event never crosses to another one. Data from another library arrives through a typed import that publishes ordinary local events on the receiving side, as catalog sync does.
- Late-join reconcile remains for instances deployed after an event was published, but is no longer how a non-running instance catches up.
- Decisions 1 to 3 needed no
plugin.capnpchange:EventConsumer.eventalready takes a batch andPluginWorker.shutdownalready exists. Decision 4 appendsEventPublisher.read, its request, page and reply structs, andmaxEventCursorBytes. Decision 8 appendsBindings.sockets,Entrypoints.socketHandler,PluginCapabilities.sockets, theSocketsandSocketHandlerinterfaces, their structs and themaxSocket*limits. - Untrusted socket requests and deliveries are a new boundary with the
socket_requestfuzz target. - Decision 13 appends
ChangesPage.liveWatch. A library whoselocalinstance watches keeps that one guest resident; every other plugin still sleeps, and listing stays the backstop for anything a watch missed. - See Entrypoints, events and the ABI, Pushed sockets, Reading past events and Durable job queue.
Source: docs/adr/event-delivery-model.md at main · View as Markdown