Summary

Durable Objects hit the Cloudflare “30 s active CPU per event” limit during large historical pulls. Once the worker runs long enough without yielding, the runtime evicts the DO and the WebSocket closes cleanly (code 1000). Because the pull stream still has thousands of events buffered, the client waits forever and our Vitest spec eventually times out.

Reference: Cloudflare SQLite DO limits – “Consuming more than 30 seconds of active CPU time between incoming network requests increases the chance your Durable Object will be evicted and reset.” https://developers.cloudflare.com/durable-objects/platform/limits/#sqlite-backed-durable-objects-general-limits

Timeline

Client pull(cursor=None)
        │
        ▼
DO emits chunk #175 (pageInfo = MoreKnown, remaining ≈3.5k)
        │
        ├── Runtime enforces 30 s CPU budget → evicts DO → sends close code 1000
        ▼
DO pull fiber exits via interrupt (“All fibers interrupted without errors.”)
        │
        └── Client stream hangs → Vitest times out after 60 s

Reproduction

CI=1 direnv exec . vitest run tests/sync-provider/src/sync-provider.test.ts \
  --testNamePattern "'Cloudflare WebSocket (DO)'"

Repeated runs consistently stall after ~7 k events during the “large batch pagination” scenario. This is the same regression that causes the property-based large-batch tests to flake in CI (see https://github.com/livestorejs/livestore/actions/runs/18123026657/job/51571735361).

Evidence

Captured 2025‑10‑01 via wrangler dev (~/.wrangler/logs/wrangler-2025-10-01_08-55-00_789.log):

TMP do pull res {"chunkIndex":175,"pageInfo":{"_tag":"MoreKnown","remaining":3500}}
TMP do ws close {"code":1000,"wasClean":true,"pullRequestIds":["180"]}
TMP do pull stream exit {"exit":"interrupt","chunks":76,"events":6500,"cause":"All fibers interrupted without errors."}

The clean close happens immediately after streaming chunk #175 even though the server reports 3 500 events still outstanding.

Impact

• Historical syncs that exceed ~30 s of DO compute hang indefinitely.
• Property-based Cloudflare sync tests in CI flake whenever the DO hits this limit (see run linked above), blocking deploys.

Proposed solution ideas

1. Automatic resume: track the last emitted cursor in the client; when the socket closes with MoreKnown, reissue pull after reconnecting.
2. Chunk/time guards: on the DO side, proactively yield after N chunks or when elapsed time approaches 24 s to let Cloudflare reset the CPU budget.
3. Back-pressure tweaks: reduce batch size or add short delays to avoid hitting the 30 s window during massive historical pulls.
4. Surface graceful failure: even with resume logic, ensure the client surfaces a deterministic error instead of hanging if the runtime closes repeatedly.

Effect RPC WebSocket Interruption Investigation

Context

• Test suite: tests/sync-provider/src/sync-provider.test.ts
• Scenario: 'Cloudflare WebSocket (DO)' sync provider > large batch pagination
• Behaviour: intermittently hangs after streaming ~7 k historic events; Vitest times out after 60 s.
• Environment: Cloudflare Durable Object running through wrangler dev with hibernated WebSockets (webSocketMode: 'hibernate').

Summary

Recent instrumentation shows the Durable Object (SyncBackendDO) emits a partial chunk with pageInfo: { _tag: 'MoreKnown', remaining: ~3k }, then the WebSocket closes cleanly with code:1000. Immediately afterwards the pull stream on the DO side is interrupted (Exit.interrupt) without emitting an Interrupt/Exit RPC message. The client library reconnects automatically, but the original pull fiber is left awaiting the remainder of the stream, leading to the 60 s timeout. This points to missing interruption handling in the Effect RPC WebSocket transport—clean closes from the DO runtime are not propagated to the caller nor used to resume pagination.

Instrumentation Added

Scope	Location	Log prefix	Purpose
Client WebSocket wrapper	packages/@livestore/utils/src/effect/WebSocket.ts	TMP ws client abort / TMP ws client onclose / TMP ws client finalizer	Records when the browser-side socket is aborted, closed by the runtime, or closed via the finalizer (includes URL, close code, exit kind).
DO pull loop	packages/@livestore/sync-cf/src/cf-worker/do/pull.ts	TMP do pull stream exit	Reports how many chunks/events were streamed and whether the stream exited via success / interrupt / failure (with cause).
DO websocket close	packages/@livestore/sync-cf/src/cf-worker/do/durable-object.ts	TMP do ws close	Logs store ID, outstanding pull request IDs, close code, reason, and wasClean.
Storage pagination + time guard	packages/@livestore/sync-cf/src/cf-worker/do/sync-storage.ts	TMP do storage page/next/empty, TMP do storage time guard hit	Confirms cursor progression and logs when we proactively stop pagination after ~80% of the CPU budget.
WebSocket RPC server scope	packages/@livestore/common-cf/src/ws-rpc/ws-rpc-server.ts	TMP ws scope close (next failing run will surface via Effect.logDebug)	Captures the exit status of the Effect scope that hosts the RPC server to determine whether the RPC layer itself registers an interrupt/failure.

Reproduced Failure (Run tmp/h001-ws-close-telemetry-run-017.log)

Key log excerpts from ~/.wrangler/logs/wrangler-2025-09-30_15-27-47_249.log:

1. Pagination in progress

   TMP do pull res {"backendId":"elD1gMUO-iKMqcsM1j0BX","chunkIndex":181,"batchSize":100,"firstSeq":26912,"lastSeq":27011,"pageInfo":{"_tag":"MoreKnown","remaining":2988}}
   TMP do storage page {"cursor":26911,"limit":256,"rawCount":256}
   TMP do storage next {"nextCursor":27167,"nextLimit":256}
   

2. Clean close from DO runtime

   TMP do ws close {"storeId":"…large-batch-test…","pullRequestIds":["203"],"pullRequestCount":1,"code":1000,"wasClean":true}
   

3. Pull stream interrupted

   TMP do pull stream exit {"storeId":"…large-batch-test…","backendId":"elD1gMUO-iKMqcsM1j0BX","exit":"interrupt","chunks":82,"events":7012,"cause":"All fibers interrupted without errors."}
   

4. Client reconnect Shortly after the close, Wrangler logs show a new Network.requestWillBeSent to the same DO endpoint and a fresh Launching WebSocket Effect RPC server. The reconnect succeeds, but the original pull fiber never recovers.

Observations

1. Clean code:1000 closes do not propagate an Interrupt/Exit RPC message to the client. The DO finalizer interrupts the streaming fiber, but the client waits indefinitely on Stream.runCollect.
2. The reconnect logic (Effect RPC socket retry) dials a new connection; however, the caller still listens on the old Stream, so the reconnection does not resume pagination.
3. Storage telemetry confirms that the pagination cursor is still far from exhaustion—this is not a logical “NoMore” termination.
4. There is no client-side handling for “socket closed mid-stream while more history remains”; the close is treated as a silent interruption, leading to the eventual Vitest timeout.
5. After adding client-side fail-fast handling (PullStreamInterruptedError), the failure now surfaces immediately with remaining events + chunk index instead of timing out, unblocking CI while we design resume semantics.

Relevant Cloudflare limits & lifecycle behaviour

• 30 s active CPU per inbound event — Each HTTP request or WebSocket message grants up to 30 s of compute time. Consuming more than 30 s of active CPU “between incoming network requests” increases the chance the Durable Object is evicted and reset (Durable Objects limits, footnote 4). Our history stream runs inside a single RPC call, so prolonged processing without new inbound messages may trigger eviction.
• Hibernation window — When idle and hibernateable, a Durable Object is hibernated after ~10 s of inactivity; WebSocket clients stay connected, but in-memory state is discarded. If the object remains non-hibernateable, it may be evicted entirely after 70–140 s of inactivity (Lifecycle of a Durable Object).
• WebSocket message size — Inbound WebSocket messages are limited to 1 MiB (Durable Objects limits). Our chunk sizes are well below this, but it constrains potential batching strategies.
• Runtime-triggered closes are expected — The hibernation guide notes that when the client closes a connection, “the runtime will close the connection too” (Use WebSockets). Clean code:1000 closures mid-stream therefore likely originate from the runtime, not our code, reinforcing the need to treat them as resumable interruptions.

Hypotheses

ID	Hypothesis	Status
H1	Cloudflare hibernation shuts down long-running hibernated sockets after ~80 streamed batches, sending a clean close without our code requesting it.	Supported by clean code:1000 close after ~82 chunks; needs confirmation via RPC scope logs.
H2	Effect RPC server/client lack resume semantics: when the socket closes, the server interrupts the stream and the client never retries the unfinished pull.	Supported by TMP do pull stream exit + lack of downstream completion.
H3	Heartbeats/pings do not guard against this scenario because the closure happens before ping failure; they simply run on the new connection.	Observed—no ping errors logged; reconnect occurs immediately.

Open Questions for the Effect RPC Team

1. Who closes the socket? Once we capture TMP ws scope close … on a failing run, we’ll know if the RPC scope sees Exit.interrupt. If it does, can we inspect the corresponding Cause inside ws-rpc-server to see whether the runtime triggered an interrupt (e.g., via ws.close) or if another component signaled shutdown?
2. Expected behaviour for streamed RPCs. Should the client treat a clean close mid-stream as a failure and retry the pull automatically? Currently it neither resumes nor surfaces an error.
3. Resume protocol. Is there built-in support (or guidance) for resuming Stream RPC calls after transport interruptions? We can reissue pull manually with the last known cursor, but the current abstraction hides the cursor from the transport layer.
4. Interrupt propagation. Would it make sense for makeEndingPullStream (or the RPC layer) to catch transport interruptions and emit an explicit Interrupt message so the client can react immediately instead of hanging?
5. Heartbeats vs. closures. With pingInterval=10s, the socket should stay alive. Do we rely on Cloudflare’s hibernation semantics to resume where we left off, and if so, do we need to hook into the resume event differently?

Suggested Next Steps

1. Reproduce with scope logging. Capture TMP ws scope close … and TMP ws client … on a failing run (post fail-fast) to feed resume design.
2. Automatic resume prototype. Build a harness that reissues pull requests from the last emitted cursor when the socket closes, validating a resumable strategy.
3. Effect RPC patch. Decide whether the RPC layer should emit explicit interrupt/failure messages when the DO runtime closes the socket to support resume semantics.

Logs & Artefacts

• Client logs: tmp/h001-ws-close-telemetry-run-017.log (initial failure), tmp/h001-ws-close-telemetry-run-0{46..55}.log (post-instrumentation passes), tmp/h001-ws-close-telemetry-run-57.log (fail-fast run).
• Wrangler logs with detailed telemetry: ~/.wrangler/logs/wrangler-2025-09-30_15-27-47_249.log, ~/.wrangler/logs/wrangler-2025-09-30_14-44-15_421.log, ~/.wrangler/logs/wrangler-2025-10-01_08-55-00_789.log.
• Instrumented source files:
  • packages/@livestore/sync-cf/src/cf-worker/do/durable-object.ts
  • packages/@livestore/sync-cf/src/cf-worker/do/pull.ts
  • packages/@livestore/sync-cf/src/cf-worker/do/sync-storage.ts
  • packages/@livestore/utils/src/effect/WebSocket.ts
  • packages/@livestore/common-cf/src/ws-rpc/ws-rpc-server.ts

This issue tracks the full fix (automatic resume + guards). The short-term fail-fast workaround in PR #713 prevents CI timeouts but does not address the underlying eviction.