# ADR: Event delivery follows the Workers model

- **Status:** Accepted (decisions 1 to 13); later phases proposed.
  Amended 2026-11-18: decision 7 covers every storage-only plugin, not
  only `local` and `s3`
- **Date:** 2026-11-11

## Context

Plugins should be able to sleep while idle, as Cloudflare Workers do, and
the system should stay event-driven. That only works if the host decides
which plugins an event is for, and if a plugin that isn't running still
receives its events.

The host already matched events to declared `[[events.consumers]]` rows,
so nothing was broadcast to every plugin. Two things blurred the model:

- **Subscription followed process liveness.** The subscriber catalog
  marked a plugin enabled only while a node had its guest loaded. An
  instance that was deployed but not running got no deliveries at publish
  time and relied on the late-join reconcile to catch up.
- **One word for two ideas.** Docs and code said a node *claims* both
  jobs and event deliveries. A job is a command with one handler. An
  event is a fact that every subscriber receives. Using the same verb
  suggested subscribers compete for an event, which they never do.

## Decision

Bookclerk follows the Cloudflare Workers model, mapping each trigger to
one Workers primitive:

| Bookclerk | Workers model | Delivery rule |
| --- | --- | --- |
| `[[events.consumers]]` | Events fanned out to one queue per subscriber, drained by that Worker's `queue()` handler | Every subscribed instance gets its own copy |
| `[[queues.consumers]]` | A Queues consumer | Each message goes to the queue's one consumer |
| `job(job)` | A Workflow step, or a command sent to one Worker | One command, one handler |
| `[triggers] crons` | Cron Triggers | One fire per instance, library-wide |
| Storefront, storage, metadata and database entrypoints | Service binding RPC to a `WorkerEntrypoint` | The caller awaits a reply |

Everything is asynchronous. No host thread blocks on a plugin; RPC
callers await a promise, and long work is a durable job.

1. **Subscriptions belong to the deployment.** Each node heartbeats a
   catalog row for every instance its deployments run (`present` or
   `placed`), with the consented subscriptions of what it would run (the
   active installed version and its grant, or a host-local tree), whether
   or not the guest is up. A matching event gets a delivery for that
   instance at publish time, and the delivery waits as `pending` until
   the instance runs. Fan-out stays per instance.
2. **Deliveries are leased, jobs are claimed.** A lease on a delivery
   only picks the node that runs one subscriber's copy. The host API is
   `lease_next_event_delivery`. The persisted index
   `idx_event_deliveries_claim` and the named atomic operations
   `claimEventDelivery` / `claimNextEventDelivery`, which database
   plugins implement, keep their names, since renaming them is a contract
   change with no behavior gain.
3. **Deliveries have a retention period.** As with a Queues message, a
   delivery that was never handed to its subscriber expires (state
   `expired`) once it has been due for longer than the consumer's
   `message_retention_period`: 60 seconds to 14 days, one day by default,
   and never more than `[events].retention_days`. Leased, retried and
   suspended deliveries keep following `max_retries` and dead letters.
   Host and node consumers never expire, and a subscriber with no live
   catalog row waits the whole events window. Expiry is a best-effort
   hourly sweep that bounds the backlog, not a delivery deadline. Expired
   rows are kept for the events retention window as the record of what a
   subscriber missed.

4. **Plugins read the event stream to catch up.** A plugin pages past
   events from the outbox with `env.EVENTS.read({ after | since, types,
   limit })` (`EventPublisher.read`), keeping an opaque `(created_at, id)`
   cursor itself. A read returns only events of the caller's own library
   that the instance's consented subscriptions match by type, schema
   version and filter, within the invocation's account scope, and never
   older than `[events].retention_days` (`truncated` says the start was
   older). Each page names the events whose delivery to this plugin
   expired (`expiredEventIds`). A read is not a delivery: it changes no
   delivery state, `deliveryAttempt` is zero and the deduplication key is
   the event id. The `EVENTS` binding therefore exists for a consented
   consumer as well as a producer. Each read scans a bounded number of
   outbox rows and may return a short page with `more` set. A read stops
   30 seconds short of now, because `created_at` is stamped before the
   insert commits and a cursor must never pass a row that has yet to
   appear; the outbox refuses caller-chosen event ids that a cursor cannot
   name. The guest request and its Cap'n Proto codecs are a fuzzed
   boundary (`event_read`).

5. **On-demand activation, first step.** Cold start was measured with the
   ignored `cold_start` test in `bookclerk-plugin-e2e`. Ready times on a
   1-CPU, 1 GiB machine were about 60 ms for native guests, 75 ms for
   workerd JavaScript and 4 s for workerd Python. Runtime, not family,
   decides the cost. A plugin whose only family is integration and that
   exports no `remoteLibrary` or `oidc` entrypoint and is not Python is
   registered behind an on-demand proxy. The proxy answers identity,
   subscriptions, schedules and health while the guest sleeps, starts
   the guest on the next delivery, queue batch, cron fire or lookup, and
   stops it after 60 seconds with no call in flight. Placement keeps
   reserving budget for asleep instances.
6. **On-demand storefronts.** A plugin whose only family is source and
   that is not Python sleeps the same way, behind a source proxy that
   shares the integration proxy's activation core. Its idle timeout is
   5 minutes, since storefront work comes in bursts. Every storefront
   call returns once its work is done (a fetch hands back files on
   disk), so a call in flight is all that keeps the guest up. Account
   listing reads the library and never wakes the guest.
7. **On-demand storage destinations.** The first-party `local` and `s3`
   destinations, whose host side is a storage backend, sleep behind a
   storage proxy on the same activation core, with a 5-minute idle
   timeout. Every backend call starts the guest when it sleeps. A write
   commits inside one call, so the call in flight holds the guest from
   upload to commit. A read stream carries its own hold and keeps the
   guest up until it is dropped, and a job that lends the guest's session
   holds it the same way. The backend's name, instance id, scan placement
   and copy support are captured from the first guest, so placement and
   dedupe never wake it. Other storage guests have no host backend to
   proxy and stay running. (Amended 2026-11-18: every storage guest now
   has a host backend and is a destination, so every storage-only guest
   sleeps this way; a guest that also belongs to another family stays
   running.)
8. **Pushed sockets are host-brokered.** A plugin that listens on a
   WebSocket can still sleep, as with Durable Objects' hibernatable
   WebSockets. It opens sockets through a `[sockets]` binding; the host
   holds each one, answers declared heartbeat pairs itself, and wakes the
   plugin's `SocketHandler` only for a real message, handing back up to
   2 KiB of attachment the plugin stored with the socket. Sockets are
   held by a jailed broker per plugin instance (the front door in a
   broker mode, under the plugin's network grant and egress checks), not
   by the daemon, so a socket gives a plugin no reach its grant lacks and
   untrusted frames are never parsed in the daemon. Remote libraries
   without a push channel move to a host-scheduled poll instead. The ABI
   shape and checks land first; the broker, the SDK WebSocket API and
   on-demand remote libraries woken by socket messages follow.
9. **On-demand remote libraries and sign-in providers.** Every
   integration that is not Python sleeps behind the integration proxy,
   including plugins that export `remoteLibrary` or `oidc`. The proxy
   owns the remote library's new-user poll (`pollEvents`): it wakes the
   guest every 5 minutes, several idle timeouts apart, and hands each
   user to the daemon's hook. Woken guests never get the hook, so a guest
   runs no poll of its own. A resident Python guest is still polled every
   30 seconds through its session. A guest keeps what must outlive a
   sleep in its plugin database: Audiobookshelf stores a hash of each
   user it has reported, so a user created while it slept is still
   reported. A `remoteLibrary` guest therefore sleeps only when it
   declares a `[[databases]]` binding and its grant holds every one;
   otherwise it stays resident. A sign-in call wakes the guest like a
   lookup, and the guest's first successful `oidcClients` answer is kept
   per proxy, since a changed package or config replaces the proxy.
10. **Pushed new users.** A remote library may also learn of new users
    from a pushed socket (decision 8). Its `SocketHandler.message` reply
    (`SocketMessageOk`) carries `pollEvents`; when the guest sets it, the
    host runs the user poll at once, on the guest the message already
    woke, instead of waiting out the interval. Only the plugin can tell
    which of a service's messages announce a user (Audiobookshelf pushes
    many unrelated events to admins), so the plugin decides, and the
    scheduled poll stays as the fallback that catches anything a closed
    socket missed. Asks that arrive while a poll runs fold into one more
    poll; an ask with no poll running is dropped.
11. **Python start-up snapshots.** A Python isolate took about 4 s to
    start on one CPU, too slow to sleep. workerd can save the isolate's
    memory after Pyodide loaded and the plugin imported its modules (a
    dedicated snapshot) and start later isolates from it: measured on the
    echo plugin with the Python SDK, about 0.6 s on one CPU (a snapshot of
    Pyodide alone, without the plugin's imports, gave about 1.9 s). Through
    the whole host path (jail, launcher, handshake and first `describe`) the
    echo plugin's start went from about 4.8 s to about 1.2 s on one CPU. The
    host keeps one dedicated snapshot per installed plugin version, keyed
    by the workerd release and the install's files, in host-owned plugin
    state. The guest's jail reads only that file and writes only a fresh
    staging directory; the launcher saves there when there is nothing to
    load or the stored snapshot fails to start, and marks the file ready
    once the isolate answered, and the host promotes only a ready regular
    file. A missing or broken snapshot costs only the slow start.
12. **Python guests sleep.** With a snapshot, a Python guest starts in
    about 1.2 s on one CPU through the whole host path, against about
    0.1 s for a native or JavaScript guest. That is slower, but a sleeping
    plugin's start happens once per burst of work and its idle timeout
    keeps it up through the burst, so Python integrations, storefronts and
    the storage destinations sleep under the same rules as the others
    (decisions 5 to 9, which this lifts the Python exclusion from). A
    `remoteLibrary` guest still sleeps only when its databases are granted.
    The first start of a new plugin version still takes the slow path
    while it saves the snapshot.
13. **A live watch keeps the local destination awake.** One process may
    stay up while idle: the `local` destination with its `watch` setting
    on. Its change feed is a file-system watcher whose journal lives in the
    guest's memory, so stopping the guest would lose the changes it has
    not handed out yet. A `changes` page carries `liveWatch` for such a
    feed. While the latest page from the running guest has `complete` and
    `liveWatch` both true, the on-demand proxy holds one call's worth of
    activity on that guest (a `Busy` guard from the activation core), so
    the idle sweep leaves it running. An incomplete or non-live page, a
    `changes` error, or the guest exiting drops the hold, and the guest
    then sleeps after the usual idle timeout. The watcher runs only on a
    local disk; on a network share or FUSE mount, or when the operating
    system refuses or overflows the watch, its pages say `complete: false`
    and the guest sleeps like any other. The host honours `liveWatch` only
    from the first-party storage plugins it starts on demand, and only the
    `local` package sets it; a durable feed, such as a queue an
    object-store plugin drains, leaves it false. Measured on Linux with
    release builds: the idle guest is resident at about 3.7 MiB; a watch
    over a 1,000-book library (1,201 folders) adds about 0.6 MiB, one over
    10,000 books (12,001 folders) about 3 MiB, and a full journal (16,384
    unread changes) about 3 MiB more. The kernel also keeps roughly 1 KiB
    per watched folder for inotify, outside the guest.

The following phases are proposed and get their own pull requests and
updates to this record:

- **On-demand activation for the rest.** The remaining storage guests
  sleep too, with an idle timeout per family.
- **Batched delivery.** `event(batch)` and `queue(batch)` receive up to
  `max_batch_size` messages per call.
- **Richer host-side filters.** Envelope selectors and per-instance
  narrowing that can only intersect the consented subscription.

## Consequences

- An instance that is deployed but failing to start holds `pending`
  deliveries for the events it subscribes to only for its retention
  period. Deliveries that are still due when it runs drain then; older
  ones expire, and the plugin catches up from the event stream with
  `env.EVENTS.read`, or resyncs from its source of truth when the gap is
  older than the events window. An operator who gives up on it undeploys it (its
  row expires within the 60 s catalog TTL).
- A consumer that is running but falls behind by more than its retention
  period also loses the oldest never-leased deliveries. Consumers that
  expect long backlogs declare a longer `message_retention_period`.
- Events are scoped to one library, the library that shares its database.
  Every subscriber and every delivery belongs to that library, and an
  event never crosses to another one. Data from another library arrives
  through a typed import that publishes ordinary local events on the
  receiving side, as catalog sync does.
- Late-join reconcile remains for instances deployed after an event was
  published, but is no longer how a non-running instance catches up.
- Decisions 1 to 3 needed no `plugin.capnp` change: `EventConsumer.event`
  already takes a batch and `PluginWorker.shutdown` already exists.
  Decision 4 appends `EventPublisher.read`, its request, page and reply
  structs, and `maxEventCursorBytes`. Decision 8 appends `Bindings.sockets`,
  `Entrypoints.socketHandler`, `PluginCapabilities.sockets`, the `Sockets`
  and `SocketHandler` interfaces, their structs and the `maxSocket*`
  limits.
- Untrusted socket requests and deliveries are a new boundary with the
  `socket_request` fuzz target.
- Decision 13 appends `ChangesPage.liveWatch`. A library whose `local`
  instance watches keeps that one guest resident; every other plugin
  still sleeps, and listing stays the backstop for anything a watch
  missed.
- See [Entrypoints, events and the ABI](/docs/plugins/abi/index.md#how-events-are-delivered),
  [Pushed sockets](/docs/plugins/abi/index.md#pushed-sockets-sockets),
  [Reading past events](/docs/plugins/abi/index.md#reading-past-events-enveventsread)
  and [Durable job queue](/docs/internals/jobs/index.md).
