ADR: Workers RPC plugin ABI + per-jail embedded workerd

Development These docs follow the development version and change with every merge.

Last updated View as Markdown

  • Status: Accepted (partially superseded)
  • Date: 2026-08-09
  • Updated: 2026-09-25 (native-behind-workerd is two host-spawned sibling jails joined by inherited links; nesting is removed because Seatbelt and AppContainer cannot re-apply / cross-connect)
  • Superseded in part by: plugin-capabilities-v3.md (2026-09-10) for product api_version = 3 — capability entrypoints, typed Cap'n payloads, env.EVENTS / outbox replace role factories and HOST.notify. Isolation, jail, network consent, workerd pin, and distribution-tier decisions below remain in force.

Context

Bookclerk needs a multi-language plugin model where third-party authors ship portable JS/TS (or Python) modules without per-OS binaries, while native Rust guests remain available for DRM and heavy dependencies. The host↔plugin contract must be identical across runtimes.

Decision

  1. Product api_version = 2. The ABI is an object-capability Workers RPC: role-specific classes (BookclerkPlugin, Destination, Source, JobHandler, ContentSource, Integration, Database), transferable byte streams, and invocation-scoped stub lifetime (create, invoke, dispose in one request). RPC carries bounded values and stream/stub capabilities. Cap'n Proto field/union ordinals are append-only; unknown members fail closed or return typed unsupported. describe() advertises apiVersion and supportedRoles; the verified install receipt (provenance + content hashes) is the host allowlist, not a publisher-signed manifest. Manifest api_version and describe().apiVersion must match PRODUCT_API_VERSION. Optional facilities use named rpcFeatures.
  2. Workerd is the control-plane front door; native jail is a backend.
    • Control plane: invocation, policy, binding, lifecycle, and outcome always pass through the workerd entrypoint (or a generated backend-proxy entrypoint the first-party wrapper owns).
    • Typed passthrough: for a native guest the bookclerk-workerd launcher forwards every Entrypoints family to the guest as typed Cap'n Proto without entering JavaScript memory. Observable backpressure, cancellation, checksum, and error semantics stay identical to the isolate path.
    • Cap'n Proto is the launcher↔native protocol. Direct native Cap'n Proto to the host is a host-selected compatibility fallback — the plugin cannot request it to bypass policy.
    • Bindings: authors see frozen BookclerkContext.bindings (HTTP / STORAGE / SECRETS / OAUTH) and optional ctx.native. The trusted adapter sees private AdapterEnv.PLUGIN_DESCRIBE (the manifest projection) and answers describe / open policy for native guests. Do not freeze env.NATIVE_PLUGIN. The host executor owns the process tree and both sandboxes; it launches bookclerk-workerd and the verified native guest as sibling jails joined by host-created inherited links. The launcher never nests a jail and never chooses the backend path. Plugin-controlled input cannot choose the executable or weaken the sandbox.
    • Native: guests serve plugin.capnp via capnp-rpc (serve). They do not speak newline JSON as the product ABI. Native DRM plugins do not implement Cloudflare’s private worker-interface.capnp; the host maps both adapters onto the same Rust traits.
  3. v1 newline JSON is removed. Guests speak Cap'n Proto api_version = 2 (or workerd Workers RPC mapped onto the same contract). Scalar JSON remains only as a versioned escape hatch for plugin-specific config, not as a transport envelope.
  4. Greenfield (no protocol key). No dual-stack product ABI with legacy JSON-RPC stdio framing.
  5. Isolation: instances are keyed by (plugin_id, account_id). Different accounts / security principals never execute concurrently in the same isolate. One isolate per invocation is allowed for higher-risk work. Per-invocation grants are scoped, expiring, revocable, and operation-limited; they are revoked on completion, cancellation, fence loss, suspension, or disconnect. Within one plugin instance, capability delegation is allowed (Workers RPC stubs are transferable). The OS jail remains mandatory: workerd is not a hardened sandbox. Pin advances via a daily CI job with a 7-day publish cooldown (same supply-chain posture as Dependabot); see packaging.md. Local embed only — not Cloudflare cloud execution. No JS-less stdio shim.
  6. Network:
    • Workerd: operator approves capabilities.network.domains. Isolate egress enforces the allowlist. Every redirect hop is checked, not only the initial host. Resolved IPs are checked against private / local / metadata ranges; DNS rebinding and Host/SNI mismatch are rejected (full IP/SNI enforcement in the native broker is follow-up — this ADR must not freeze initial-host-only or “native = coarse unrestricted”).
    • Brokered HTTP uses the same domain grants as isolate fetch. Raw TCP, UDP, and listen are distinct capabilities. Jail default-deny remains. Native outbound is not permanently coarse-unrestricted.
    • The OS jail for every workerd guest is NetPolicy::LocalIpc, whatever the grant says: the host↔isolate RPC bridge is a Unix socket and the isolate's egress leaves through the launcher's relay as tunnels the unsandboxed host checks and dials, so neither bookclerk-workerd nor workerd needs an IP socket (Linux seccomp refuses AF_INET, AF_INET6 and AF_PACKET; macOS denies all network but Unix sockets; Windows keeps only in-container loopback for the bridge). The grant is enforced twice: by the egress worker inside the isolate, and by the host's tunnel check outside the jail, which holds even if the isolate is escaped. A denied grant gets no host link and a blocked globalOutbound. Until per-port fetch grants exist, the host admits a fetch host on any port. This replaces the earlier NetPolicy::OutboundListen jail, which left outbound connect open at the OS layer. The generated workerd config exposes a single listen socket (rpc → bridge); compatibility flags (the Python workers pair) do not add sockets.
  7. Consent UX: CLI and web UI prompt before enable; grants persisted; capability widening re-prompts. The same covering grant is enforced at every external spawn and at privileged delivery (config / secrets / work_fs / oauth). Native outbound shows an explicit warning.
  8. compatibility_date newer than bundled workerd: warn, still load, at the pin date. workerd refuses a date past supported-compatibility-date.txt in the binary (a compatibility date selects breaking runtime changes). The author date stays in plugin.toml. The isolate runs at the applied pin date. Host surfaces such as events and jobs do not yet choose behavior from that date; when they do, later behavior changes ship with an enable date in a release.
  9. [workerd].limits: local workerd does not Cap'n Proto-enforce cpuMs / subRequests. Bookclerk clamps cpu_ms / subrequests (defaults and hard caps), injects subrequests into egress policy JSON, and the egress bridge counts hops per egress invocation (one plugin fetch() + redirect chain → 429) — Cloudflare-style per-invocation budgeting, not isolate-lifetime module state. Cross-fetch aggregation within a host RPC is deferred until that invocation unit is defined. Effective cpu_ms is logged at isolate start; OS-jail CPU enforcement is tracked separately.

Consequences

  • Manifests declare runtime = "workerd" | "native" and capability sections.
  • Catalog script archives ship plugin.toml + modules/ only.
  • Native publishers still build per-arch when they need native capabilities.
  • CI must fail on ABI schema drift vs generated SDK outputs.
  • Every byte a guest can send the host crosses a fuzzed boundary per Untrusted-input fuzzing: Cap'n Proto replies the granted HTTP channel and the mux frames (mux_peer) have targets today; CONNECT handshakes are still to add (docs/contributing/fuzzing.md).
  • CI requires the pinned workerd binary for the workerd contract suite; the job fails if the workerd adapter cannot start (BOOKCLERK_SKIP_WORKERD=1 is local native-only).

Distribution tiers

Tier Contents
Platform Hosts + jail + workerd + media-worker + sqlite + local
Product Optional first-party guests (storefronts, ABS, s3, d1, postgres)
Reference examples Echo under examples/ — CI / cargo dev --examples only; never packaged

Every guest is an external (jailed) subprocess, including platform sqlite/local.

Non-goals

  • Operator-side cargo / rustc / npm to install plugins.
  • In-process Wasm/cdylib loading in the host.
  • Marketing workerd alone as the security boundary (jail remains mandatory).
  • Cloudflare-hosted Dynamic Workers for operator plugin execution.
  • HMR for cargo dev (restart-based only).

Follow-ups (deferred)

The native-behind-workerd executor is now the production path: the host spawns every plugin through bookclerk-workerd (SpawnPlan in bookclerk-plugin-host), runtime = "native" only selects the backend behind the isolate, and a missing bookclerk-workerd / pinned workerd is a hard spawn error in every [plugins].isolation mode. Direct host↔native Cap'n Proto survives only as SpawnTransport::DirectNativeDiagnostic for tests and diagnostics; no product binary selects it.

Native-behind-workerd guests are a sibling NetPolicy::Deny jail the host starts next to the gateway, not a child of bookclerk-workerd. native_guest.rs is gone. IPC is host-created links delivered as fd:<n> / handle:<n> (BOOKCLERK_GATEWAY_GUEST_RPC, BOOKCLERK_SOCKET_PROXY). The production CONNECT mux is served by the unsandboxed host so granted loopback dials do not depend on AppContainer loopback (no CheckNetIsolation exemption). Unjailed conformance may still pass BOOKCLERK_GATEWAY_PROXY into bookclerk-workerd. Unix links are socketpairs. Windows links are two unidirectional pipes (BOOKCLERK_GATEWAY_GUEST_RPC read half, BOOKCLERK_GATEWAY_GUEST_RPC_WRITE write half, guest proxy write half BOOKCLERK_SOCKET_PROXY_WRITE) so a concurrent read cannot lock the write. The guest has no ambient AF_INET and no named endpoint. The gateway jail is LocalIpc: its host↔isolate RPC bridge is a Unix socket. Gateway TMPDIR is a host-owned session-<nonce>/ directory; guest TMPDIR is its own scratch.

Why nesting was removed

A confined process cannot apply a second OS jail on macOS or Windows:

  • macOS Seatbelt. sandbox_init from inside an existing sandbox returns EPERM. Apple DTS and Seatbelt notes state a process cannot re-apply or tighten a profile. Nested sandbox-exec fails even when the outer profile is (allow default). Profile tuning cannot fix this.
  • Windows AppContainer. Pipes created inside a container must use \\.\pipe\LOCAL\, and pipes cannot be used between different AppContainers (Win10 1709+). Creating a nested container from inside one also fails. Host↔container loopback stays blocked; same-container loopback (launcher → workerd) is the E3 contract.
  • Linux Landlock domains stack and seccomp Deny only bans socket(AF_INET/AF_INET6/AF_PACKET), which is why the nested topology passed there and hid the other two OSes.

The replacement applies each jail once from the unsandboxed host and reproduces the intersection of the old two layers (not a copy of the inner deny_spec_with). Windows Isolation::Off still requires bookclerk-jail.exe beside the host so handle handoff can run with Enforcement::Disabled.

Still deferred: OAuth/listen broker beyond the existing callback proxy, container executor, and VPS benchmarks.

Instances are keyed by (plugin_id, account_id). Shared-isolate concurrent principals are not a proven isolation boundary (stubs are transferable).

When Bookclerk defines a Cloudflare-comparable invocation unit (host RPC ≈ one Worker invocation) for cpu_ms / subrequest aggregation, plan to evolve lifecycle and tenancy as follows — no implementation in this ADR revision:

  1. Invocation-style budgeting — treat a host RPC (or an explicit job envelope) as the CF invocation boundary so isolate cpu_ms and subrequests aggregate like Workers, not as isolate-lifetime module state.
  2. Per-account instantiation — for account-bearing kinds (especially source / OAuth storefronts), prefer a jail (and workerd isolate) keyed by (plugin_id, account_id) so linked users do not share process memory or plugin-state/<PluginKey fs-id>/data. Resource caps (extraProcesses, memory, CPU) stay per instance.
  3. Lifecycle by kind — split long-lived vs one-shot:
    • Long-lived: platform DB/output and other shared infrastructure guests that amortize connection state.
    • One-shot / short-lived: high-trust-boundary or bursty work (and, eventually, many storefront account sessions) closer to Workers request/isolate churn.
  4. Until then, keep host hygiene (prefer one account per privileged RPC) and document that enabling a shared storefront guest trusts it with every account the host sends it.

Source: docs/adr/plugin-workers-rpc-workerd.md at main · View as Markdown