A reconnect storm is what happens when many devices come back online at the same moment. Power outage in a city, a router reboot at a chain restaurant, a tower handover that resets thousands of cellular sessions: all of them produce the same shape. A flood of clients, each carrying a queue, each opening a websocket, each issuing the same first request. The server, which was idle a second ago, is now under more load than it sees in a normal week.
This essay is about what we did the first time it broke us, and the patterns that worked.
The shape of the storm#
It is worth being precise about what is happening. The clients are not malicious; they are doing what the docs told them to do. On reconnect:
- Open a fresh websocket.
- Authenticate.
- Pull the deltas they missed while offline (one HTTP request per entity type, often dozens of them).
- Push their queued writes (one request per item, or a small batch).
- Subscribe to per-tenant channels for live updates.
In normal operation these steps are spread across users and time, so the server sees a steady trickle. In a storm, a few thousand devices fire all five steps within the same five-second window. The server spends those five seconds servicing handshakes; the database spends them servicing 50,000 small queries with high lock contention; the websocket layer spends them spawning 5,000 new connections. The slowest step gates the whole thing.
The first time it happened to us, the server stayed up but the queries queued; clients started timing out, started retrying, and the retries made the storm worse. By minute three we were watching the queue grow faster than we could drain it. By minute five we hand-rolled a kill switch.
Five fixes, in the order that mattered#
We did not get them all in one go. We got the first one in panic that afternoon and the rest over the following month.
1. Jittered reconnect, on the client#
The single biggest win. Every client, on losing connection, picks a
random delay in [0, jitter_max] before its first reconnect attempt;
on subsequent failures the delay grows exponentially with another
small jitter. This breaks the synchronisation. A thousand clients no
longer reconnect at second 0; they spread across [0, 30], and the
server sees a slope instead of a wall.
Duration nextReconnectDelay({required int attempt}) {
final base = math.min(30000, 500 * math.pow(1.7, attempt).toInt());
final jitterMs = math.Random().nextInt(base ~/ 4);
return Duration(milliseconds: base + jitterMs);
}Note: jitter has to be multiplicative-ish, not additive. Adding ±200 ms to a 30 s delay barely smooths the curve; multiplying or adding a noticeable fraction does.
2. One unified pull, not dozens#
We had inherited per-entity sync endpoints (/products/list,
/orders/list, /contacts/list, …). On reconnect a single client
issued one request for each of them. A unified pull endpoint that takes a since
cursor across all entity types in one request collapses all of those requests
into one. Fewer connections, fewer queries, less ceremony.
POST /api/v1/sync/pull
{
"cursors": {
"contacts": "2026-05-08T14:22:01.123Z",
"invoices": "2026-05-08T14:22:01.500Z",
"products": "2026-05-08T14:22:00.000Z"
},
"limits": { "perEntity": 200 }
}The server returns a single bundle keyed by entity type. The client walks the bundle and applies in dependency order (contacts before invoices, products before invoice items, etc).
3. Per-tenant queues with fairness#
If one tenant is misbehaving (or has a thousand devices), they should not starve the others. Instead of a single global Horizon queue, we shard by tenant and round-robin between active tenants. A noisy tenant takes longer to drain its own queue; its noise does not bleed across.
The trade-off is that you cannot use Horizon's autoscaling per "workload" any more, since "workload" is now per tenant. Worth it.
4. A circuit breaker on the cold path#
The pull endpoint is the cold path. If its failure rate crosses a
threshold over a sliding window, trip a breaker on the server side and
return a polite "try again shortly" with a Retry-After header.
The clients honour it. The server gets its breath back. Without the
breaker, every failed pull triggers a retry, the retry adds load, the
load triggers more failures.
5. Connection-pool guardrails#
The websocket layer (Reverb in our case, but it could be any) has a finite per-process connection budget. Without limits, a storm fills the pool, new connections get refused at the OS level, and the error message at the client end says "Connection refused" — useless. Setting explicit per-tenant connection caps and per-process maximums means the server starts shedding cleanly, with a header that tells the client why.
What we measure now#
The dashboard for storm health is small:
| Metric | Healthy | Warning |
|---|---|---|
| Reconnects per second | Low and steady | Sudden spikes |
| Pull p99 latency | Stable | Climbing |
| Queue drain rate | > 1×in-flow | < 1×in-flow |
| Breaker open events / hour | None | Any |
| Pool utilisation | Comfortable headroom | Close to the cap |
The interesting one is "queue drain rate ÷ in-flow rate." If it is below 1, you are losing ground. If it stays below 1 for more than a minute or two, the storm is winning.
Why this is worth getting right#
The first time we had a storm, the team got pulled into an unplanned incident on a Tuesday morning that ended up costing engineer hours and a panicked apology to a customer. The five fixes above took real time to design, ship, and validate. The next storm, the system handled itself and we found out about it from the dashboard, not the customer. That is the trade you are buying.
If sync engines are interesting, the series page picks up where this ends.














