Broker WebSocket Reconnect and Resubscribe Patterns
Direct Answer
Broker WebSocket connections drop. Network interruptions, broker-side maintenance, idle-timeout disconnections, and client-side restarts all produce connection gaps. A WebSocket gap means your system stopped receiving order status updates, fill events, and position changes for some period. Any event that arrived during the gap, a fill, a partial fill, a cancel acknowledgment, is now missing from your internal state unless you explicitly recover it.
A robust reconnect design has three phases: reconnect with backoff (establish a new TCP connection without hammering the broker during an outage), resubscribe (restore all active subscriptions on the new connection), and reconcile (query the broker's current state via REST to detect events missed during the gap). All three phases must complete before the adapter reports itself as connected and ready to trade. Skipping reconciliation is the most common source of position state corruption in automated trading systems.
Key Takeaways
- Reconnect ≠ resume: A new WebSocket connection does not automatically restore your previous subscriptions. Each subscription must be explicitly re-sent on the new connection.
- Assume events were missed: During every gap, assume at least one fill or status change occurred that your system didn't receive. Always reconcile after reconnect rather than hoping the gap was clean.
- Exponential backoff with jitter: After a disconnect, wait before reconnecting, and randomize the wait time to prevent multiple systems from reconnecting simultaneously (thundering herd). Formula:
wait = min(cap, base × 2^attempt) + random(0, base). - Track the last known sequence number: If the broker assigns sequence numbers to events, record the last one received. On reconnect, request replay from that sequence number so you don't need to reconcile the entire order book.
- Heartbeat monitoring detects silent drops: Some WebSocket connections drop without a clean close frame, the TCP connection dies but your process doesn't know. Send pings every 15-30 seconds and close + reconnect if you don't receive a pong within the timeout.
- Subscriptions are per-connection: Store your subscription set in-memory as a Set of subscription specs. On reconnect, iterate the set and send each subscription message to the new connection before signaling readiness.
- REST reconciliation is the ground truth: After reconnecting and resubscribing, query open orders and positions via REST. Compare against your cached state. Emit synthetic events for discrepancies.
- Block trading during reconciliation: Don't allow new order submissions until reconciliation completes. An order submitted against stale position state may exceed risk limits or create an unintended position.
Core Concepts
Exponential Backoff with Jitter
Exponential backoff controls reconnect timing after a disconnect. The simplest formula doubles the wait time after each failed attempt up to a maximum: wait = min(maxWait, baseWait × 2^attempt). With baseWait = 1s and maxWait = 60s, the sequence is: 1s, 2s, 4s, 8s, 16s, 32s, 60s, 60s, … This prevents overwhelming a struggling broker API with rapid reconnects while still recovering quickly when the broker comes back.
Jitter adds randomness to the wait time: wait = min(maxWait, baseWait × 2^attempt) + random(0, baseWait). Without jitter, if 100 clients all disconnected at the same time, for example, due to a broker-side restart, they all reconnect simultaneously after the same backoff period, causing a connection storm. With jitter, the reconnects are spread across a window, reducing peak load on the broker's connection-acceptance infrastructure.
Reset the attempt counter when a connection has been stable for longer than your maximum backoff period (e.g., 5 minutes). Otherwise, a connection that drops briefly at attempt=7 will wait 60 seconds next time even if the previous gap was resolved and 20 minutes of stable operation followed. A fresh transient disconnect should start the backoff sequence from the beginning.
Implement reconnect as a state machine: CONNECTED → DISCONNECTED (on close or error) → BACKING_OFF (timing the wait) → CONNECTING (attempting TCP + WebSocket handshake) → SUBSCRIBING (sending subscription messages) → RECONCILING (REST state check) → CONNECTED. Each state transition is logged. The time spent in each state is measured and alerted on if it exceeds a threshold.
Subscription Management
WebSocket subscriptions at most broker APIs are channel-based: you send a subscribe message specifying one or more channels (order updates, trade updates, account updates, market data for specific symbols). When a connection drops, all subscriptions are lost, the broker's server-side state for your connection is deallocated. The new connection starts with zero subscriptions.
Maintain a subscription registry in your adapter: a data structure that maps subscription IDs to subscription specs. When a strategy subscribes to order updates for account X, the spec is stored. When a market data subscriber subscribes to BTC/USD quotes, that spec is stored. On reconnect, iterate the registry and send each spec as a subscribe message to the new connection. Only after all subscribe messages are sent (and acknowledged if the protocol requires acknowledgment) does the adapter proceed to reconciliation.
Subscription message formats vary by broker. Alpaca requires: {"action": "subscribe", "trades": ["AAPL"], "quotes": ["AAPL"], "updates": []}. Coinbase Advanced uses: {"type": "subscribe", "channels": ["user"], "product_ids": ["BTC-USD"]}. The subscription message is part of the adapter's translation layer, not a generic WebSocket utility.
Test subscription restoration explicitly: connect, subscribe, then use your test tooling to force-close the WebSocket (send a close frame or kill the TCP connection). Verify that the adapter reconnects and that your subscription callback receives the next event after reconnection without requiring the subscriber to re-register.
Gap Detection and REST Reconciliation
The reconciliation step is where you determine what you missed during the gap. The simplest approach: after reconnecting and resubscribing, query the broker's open orders endpoint and compare the results against your internal open order list. Orders in the broker's list that aren't in yours are orders you submitted and forgot. Orders in your list that aren't in the broker's list were filled, canceled, or expired during the gap.
For each discrepancy, emit the appropriate synthetic event. An order that was filled during the gap gets a synthetic fill event with the fill details from the REST response. An order that was canceled gets a synthetic cancel event. These synthetic events flow through the same event processing pipeline as real-time events, ensuring all downstream consumers, position tracker, P&L calculator, risk monitor, update their state correctly.
Some brokers provide a more targeted mechanism: an activity log endpoint that returns all order events since a specified timestamp. After reconnect, query for events since your last known event timestamp and replay any that your system didn't receive. This is more precise than full-state reconciliation but requires the broker to support timestamp-based event queries.
Record the timestamp of every event you receive and the timestamp of every reconnect. The gap window is from the last received event timestamp to the reconnect timestamp. Query the activity log or order state as of the start of the gap window. After reconciliation, resume treating new WebSocket events as ground truth.
Heartbeat Monitoring and Silent Disconnect Detection
TCP connections can remain in an "established" state at the OS level even after the remote end has gone away, if no data has been exchanged to trigger a timeout. A broker WebSocket that goes quiet, no fills, no status updates, no market data, can remain "connected" from your perspective while the underlying TCP session is actually dead. You won't discover this until you try to send a message and receive an error, potentially minutes later.
WebSocket protocol includes a ping/pong mechanism (RFC 6455 control frames). Send a ping frame every 15-30 seconds. If you don't receive a pong within 10 seconds, treat the connection as dead and begin the reconnect sequence. Most broker WebSocket implementations respond to pings automatically; some expect the client to initiate heartbeat messages in a broker-specific JSON format instead of protocol-level pings.
Also monitor the time since last meaningful data. If you have a subscription to an account that typically generates at least one event per minute (order updates, market data) and you've received nothing for 3 minutes, that's a signal to send a probe request and verify the connection is live. Market data subscriptions on liquid instruments like BTC/USD should produce updates many times per second; a 10-second gap in that feed is a reliable sign the connection is degraded or dead.
Log all heartbeat events and their outcomes. Heartbeat failure rates over time reveal broker infrastructure degradation patterns that often precede announced maintenance windows, giving you early warning to switch to a secondary broker before the primary fully goes offline.
Worked Scenario
- Normal operation. Your adapter is connected to Alpaca's order update WebSocket. You have a subscription to trade updates. Your system has one open GTC limit order for 100 shares of NVDA.
- Disconnect event. At 10:23:45 ET, the WebSocket connection drops with code 1006 (abnormal closure). Your adapter's close handler fires and records: disconnect_time = 10:23:45, last_event_seq = 4891, last_event_time = 10:23:40.
- Backoff phase. attempt=0, wait = 1s + random(0, 1s) = 1.3s. You sleep 1.3 seconds before attempting to reconnect.
- Reconnect attempt. At 10:23:47, you attempt a new WebSocket connection. It succeeds immediately. The adapter transitions to SUBSCRIBING state.
- Resubscribe. You send:
{"action":"subscribe","trades":["NVDA"]}. Alpaca acknowledges. You transition to RECONCILING and block order submissions. - Reconciliation. You query
GET /v2/orders?status=open. The broker returns: no open orders. But your internal state says the NVDA order is open. Discrepancy detected. - Resolution. You query
GET /v2/orders/{nvda_order_id}. The broker returns: status = "filled", fill_price = 887.50, fill_time = 10:23:43. The fill happened 2 seconds before the disconnect, you missed the fill event during the 5-second gap. - Synthetic event. Your adapter emits a synthetic fill event:
{orderId, symbol: "NVDA", side: "buy", qty: 100, price: 887.50, timestamp: "2026-08-07T10:23:43Z", synthetic: true}. Position tracker updates: NVDA position = +100 shares. Risk monitor updates. Buying power updates. - Ready to trade. Reconciliation complete. Adapter transitions to CONNECTED. Order submissions unblocked. Elapsed time from disconnect to ready: 4.2 seconds.
Measurement Framework
| Measurement | Question to Answer |
|---|---|
| Reconnect frequency (events/day) | How often is the WebSocket connection dropping, and is the rate trending up? |
| Time-to-reconnected (seconds, p50/p95) | How long does it take from disconnect to fully reconnected and reconciled? |
| Events missed per gap (count) | How many fill or status events does the adapter miss on average during a gap window? |
| Heartbeat failure rate (%) | What fraction of ping probes fail to receive a pong, indicating silent connection failure? |
| Reconciliation discrepancy rate | What fraction of reconnects discover a position discrepancy requiring a synthetic event? |
Common Failure Modes
Reconnecting Without Resubscribing
A reconnect handler that opens a new WebSocket but doesn't re-send subscription messages will receive no data on the new connection, yet will appear connected. The adapter will report status as "connected" while silently delivering nothing to event consumers. This failure is particularly hard to detect because no error is returned; the connection is simply silent.
Make resubscription an explicit, required step in the reconnect state machine. The adapter should never transition to CONNECTED status without having confirmed all subscriptions are restored. Include a timeout on the resubscription step: if subscription acknowledgments don't arrive within 5 seconds, treat the reconnect as failed and begin another backoff cycle.
Missing the Fill During the Gap Window
The most consequential failure: a fill event arrives during the WebSocket gap and is never received. The adapter reconnects, skips reconciliation (or reconciles too narrowly), and resumes operation. The strategy believes it has no position in NVDA; actually it has 100 shares. The next signal to buy NVDA results in 200 shares being bought, twice the intended size. This is a real position risk error, not just a bookkeeping error.
Reconciliation is not optional. After every reconnect, regardless of gap duration, query the broker's current open orders and positions and compare against internal state. Any discrepancy is a missed event. No gap is too short to have missed a fill, in active markets, fills happen in sub-second time windows.
Thundering Herd on Broker Restart
A broker infrastructure restart that disconnects all clients simultaneously, without jitter in the backoff, results in thousands of clients attempting to reconnect at exactly the same moment, 1 second, then 2 seconds, then 4 seconds after the disconnect. This creates wave-shaped load spikes that overwhelm the broker's connection-acceptance infrastructure, causing many reconnect attempts to fail and extending the total outage duration for all clients.
Jitter is not optional. Add random noise of at least one base-wait-period to every backoff delay. The cost is at most one extra base wait in the best case; the benefit is preventing a self-inflicted amplification of the original outage.
Trading During Reconciliation
A system that allows new order submissions while reconciliation is in progress can create impossible-to-resolve state. New orders are submitted against a position state that may be wrong. If reconciliation subsequently discovers the position is different from what the strategy assumed, the orders placed during reconciliation may be over-sizing or under-sizing relative to the true position.
Implement a hard gate: set an adapter flag isReady = false from the moment of disconnect until reconciliation completes. All order submission calls check this flag and return an error if it's false. The strategy is responsible for handling this error, either by queuing the order for submission after the gate opens, or by canceling the order intent and waiting for the next strategy evaluation cycle.
Silent Connection Death Due to No Heartbeat
A long-running connection with no heartbeat monitoring eventually enters a state where the broker's server has closed its side but the client's TCP stack still considers the connection open. The next attempt to read data will eventually time out, but this can take minutes, long enough for the system to miss entire trading sessions' worth of events while believing it's connected.
Implement heartbeats. The implementation is simple: a timer fires every 20 seconds, sends a ping, and sets a flag. If the pong doesn't arrive within 10 seconds, the flag triggers a reconnect. Test this specifically by writing a test that kills the server-side connection without sending a close frame and verifying that the client reconnects within heartbeat_interval + pong_timeout seconds.
FAQ
How long should I wait before the first reconnect attempt?
Start with 1 second as the base wait. For many transient disconnections (a brief network hiccup, a broker rolling restart), the connection will succeed on the first or second attempt at 1s or 2s. Starting shorter (100ms) risks hammering a broker during an outage; starting longer (5s) extends your data gap unnecessarily for short outages. The base wait of 1 second is a broadly used default in production WebSocket clients and provides a reasonable tradeoff.
What's the maximum reconnect wait I should use?
Cap the maximum wait at 60-120 seconds. Beyond 2 minutes, you're not protecting the broker from hammering, you're just delaying your own recovery. If the broker has been down for 3 minutes and you've been sitting in 120-second backoff waits, you haven't tried to reconnect in 2 minutes. A cap of 60 seconds means you're attempting reconnects roughly once per minute during extended outages, which is a reasonable frequency to detect broker recovery without excessive load.
Do I need to reconcile after every reconnect, even brief ones?
Yes. A 100ms gap is long enough for a fill event to arrive and be missed. Fill events on liquid instruments happen in microseconds. The extra latency of a REST reconciliation call, typically 50-200ms, is small compared to the risk of operating on stale position state. Make reconciliation unconditional; don't try to determine whether the gap was "short enough" to skip it.
What if the broker doesn't support sequence numbers or event replay?
Fall back to full-state reconciliation: query open orders, open positions, and recent activity via REST, and diff against your internal state. This is less targeted than replay but achieves the same result for the most critical state, you know what positions and orders the broker currently sees. The cost is a few extra REST calls on each reconnect. For a system that reconnects at most a few times per day. This is negligible.
Should I maintain multiple WebSocket connections for redundancy?
Maintaining a hot standby connection to the same broker adds complexity without reducing the gap window significantly, if the broker's infrastructure goes down, all connections to it drop simultaneously. A more useful redundancy strategy is maintaining a connection to a secondary broker and having failover logic that routes orders there when the primary is disconnected. Dual connections to the same broker are useful for load isolation (e.g., one connection for order updates, another for market data) but not for availability redundancy.
How do I handle a WebSocket that connects but immediately drops repeatedly?
Repeated immediate drops usually indicate an authentication problem (expired token), an IP block, or a broker-side rate limit on new connections. Inspect the WebSocket close code and message before attempting the next reconnect. Close code 1008 (policy violation) or 4xxx broker-specific codes often indicate authentication or authorization failure. These errors should not be retried with the same parameters, they require correcting the authentication state first. A circuit breaker that opens after 5 consecutive immediate drops (connection lasted under 5 seconds) prevents wasting resources on futile reconnect attempts while an authentication error persists.
How do I test WebSocket reconnect behavior in an automated test suite?
Use a local WebSocket proxy that can inject faults: drop connections, delay messages, or drop specific message types. Libraries like ws in Node.js make this straightforward. Write test cases for: (1) clean close followed by reconnect, verify resubscription; (2) dirty close (no close frame) followed by reconnect, verify heartbeat detection; (3) fill event arrives exactly at disconnect time, verify reconciliation detects the missed fill; (4) multiple rapid disconnects, verify backoff prevents immediate reconnects; (5) reconciliation discovers discrepancy, verify synthetic fill event emitted correctly.
Can I use HTTP long polling instead of WebSocket for order updates?
HTTP long polling, where you send a request and the server holds it open until an event arrives or a timeout occurs, is a fallback that some brokers provide. It works but introduces more latency per event and higher per-event overhead than a persistent WebSocket connection. If your broker offers WebSocket, use it. Fall back to long polling only if the broker doesn't offer WebSocket or if your network environment blocks WebSocket connections. Long polling also requires its own reconnect logic for the HTTP polling loop, so it doesn't eliminate the complexity, just changes its shape.
How should subscription state be stored so that resubscribing is exact?
Keep the intended subscription set as explicit application state rather than inferring it from what was sent on the previous connection. On reconnect the client replays that set and then confirms each subscription was acknowledged, treating an unacknowledged one as a failure rather than assuming success. Without this, a subscription added mid-session or one that failed quietly on the previous connection disappears after a reconnect, producing a feed that looks healthy while missing a symbol.
References
- RFC 6455: The WebSocket Protocol: The base specification defining WebSocket close codes, ping/pong frames, and connection lifecycle that all broker WebSocket implementations build on.
- AWS Builders Library: Timeouts, Retries, and Backoff with Jitter: Practical explanation of exponential backoff with jitter and why full jitter outperforms decorrelated jitter in distributed systems.
- Alpaca Markets: Trading WebSocket Streaming: Reference for a real broker WebSocket subscription format, message schema, and heartbeat requirements.
- RFC 7692: WebSocket Per-Message Compression: WebSocket extension relevant to high-volume market data streams where message compression reduces bandwidth and improves latency.
Educational Disclaimer
This guide is for educational and informational purposes only and does not constitute financial, investment, or legal advice. WebSocket behavior, heartbeat requirements, and reconnect semantics vary by broker and change with API updates. Always test reconnect behavior against your specific broker's sandbox before deploying to live trading.