Failover and Multi-Broker Design

Q: When is multi-broker design worth the added complexity?

Multi-broker design is worth the complexity when any of the following are true: (1) A single broker doesn't support all the instruments or order types your strategy requires; (2) Your strategy's success depends on continuous operation during broker outages, you can't afford to be flat for the duration of an outage; (3) You're large enough that a single broker's rate limits or capital constraints restrict your operation; (4) You're actively managing regulatory risk by avoiding excessive concentration at a single venue. For smaller systems, single-broker with good reconnect and recovery logic is usually simpler and adequate.

Q: What's the minimum capital required at the secondary broker to make failover useful?

At minimum, enough to cover your average single-day order flow for the symbols you want to route there during a failover. If your strategy typically commits $20,000 per day in new positions, the secondary broker needs at least $20,000 in buying power available. In practice, positioning 20-40% of total capital at the secondary broker provides meaningful failover capacity while allowing the primary broker to operate with the majority of available capital. Exact allocation depends on your strategy's capital requirements and your tolerance for idle capital at the secondary.

Q: How do I handle tax lot accounting when the same symbol is held at two brokers?

Tax lot accounting must be maintained per-broker, per-lot. When you sell shares at Broker A, you close lots at Broker A. When you sell shares at Broker B, you close lots at Broker B. The tax lots at one broker are completely independent of the tax lots at the other. Your tax reporting, Schedule D, Form 8949 for US traders, must aggregate realized gains from both brokers. Most brokers provide Form 1099-B covering their transactions. You'll receive one 1099-B per broker and must sum them when filing. Ensure your internal records track the acquisition date and price for each lot at each broker to reconcile against the 1099-Bs.

Q: What should I do about open GTC orders at the primary broker during an outage?

GTC orders at the primary broker remain working there during the outage. You cannot cancel them because you can't reach the broker. This means they might fill during the outage without your system knowing. When the primary recovers, reconciliation should find any fills that occurred and apply them to your position register. In the meantime. Do not submit new buy orders for the same symbols at the secondary broker, you risk doubling up if the GTC at the primary fills while you've also submitted a new buy at the secondary. Block new orders for symbols with GTC orders at the unavailable primary.

Q: How do I test failover logic safely without risking real money?

Use both brokers in paper/sandbox mode for failover testing. Configure your system with Broker A sandbox as primary and Broker B sandbox as secondary. To simulate a failover: configure the routing layer to treat Broker A as unavailable (either by forcing its circuit breaker open or by using a mock adapter that always returns circuit-open status). Submit test orders and verify they route to Broker B. Then restore Broker A and verify that recovery and reconciliation work correctly. This test requires no real capital and can be run as often as needed. Schedule a quarterly failover drill and document the results.

Q: What actually makes the decision to fail over?

A defined health signal rather than an operator noticing something is wrong. Useful inputs include consecutive request failures, latency above a threshold, a stale event stream, and reconciliation mismatches, evaluated over a window so a single failed call does not trigger a switch. The decision logic needs to run outside the component being monitored, and it needs a defined minimum dwell time after switching, otherwise an intermittent fault produces repeated flips between brokers that is worse than either state.

Q: What is a split-brain condition in a multi-broker setup?

It occurs when the system believes a broker is unreachable and routes elsewhere, while that broker is in fact still processing earlier instructions. The result is exposure at both venues that neither side of the system knows about in full. The usual mitigations are canceling or confirming outstanding orders at the primary before treating it as failed, treating an unreachable broker as an unknown state rather than a flat one, and reconciling both brokers before resuming rather than only the one now in use.

Q: How should failback to the primary broker be handled?

More cautiously than the failover itself, because the primary recovering is not the same as the primary being trustworthy. A reasonable sequence is requiring sustained health for a defined period, reconciling positions and open orders at both brokers, resuming with reduced size, and only then returning to normal routing. Automatic immediate failback is the most common cause of repeated flapping, and manual failback with automatic failover is a defensible asymmetry.

Q: What does a multi-broker design do to reconciliation and reporting?

Every downstream process that assumed one source of truth now has to aggregate. Positions must be summed across brokers with a canonical symbol mapping that works for both, cash and buying power cannot be compared against a single account, and performance reporting has to combine fills from venues with different fee structures and timestamp conventions. This is usually a larger body of work than the routing logic itself, and it is where multi-broker projects most often stall.

Direct Answer

Multi-broker design is not a backup plan for broker outages. It is a fundamental architectural decision that changes how you think about position state, risk management, and reconciliation. When you route orders to multiple brokers, your net exposure to a symbol can be split across venues: 100 shares of AAPL at Alpaca, short 50 at Interactive Brokers. Your internal position model must aggregate across venues to compute net exposure correctly. Your risk checks must compare the aggregate position, not individual venue positions, against limits.

Failover, the ability to route new orders to a secondary broker when the primary is unavailable, is achievable, but positions held at the primary broker remain there until the primary recovers. You cannot magically transfer positions between brokers. A strategy that routes new buy orders to Broker B during a Broker A outage will accumulate a new position at Broker B while the original position at Broker A is frozen and unmanageable until recovery. The strategy must be designed to handle this split-position reality, not assume it can operate as if on a single broker.

Key Takeaways

Net exposure requires cross-broker aggregation: A position held at two brokers has a net exposure equal to the sum of all venue positions. Risk checks must operate on the aggregate, not per-venue positions.
Failover is for new orders, not existing positions: When the primary broker is unavailable, you can route new orders to a secondary broker. But open positions at the primary remain there, inaccessible until the primary recovers.
Define failover triggers precisely: "Broker is down" is not a precise trigger. Define triggers: N consecutive timeouts, circuit breaker open, explicit "market closed" status from the adapter, or WebSocket-not-recovered for M minutes.
Multi-broker increases reconciliation complexity: Positions split across brokers require a cross-venue reconciliation layer that compares aggregated internal state against the sum of per-venue broker states. This is more complex than single-broker reconciliation.
Cost basis and P&L attribution become venue-specific: When you hold the same symbol at two brokers, cost basis tracking and P&L calculation must be per-venue (because each broker's tax lot tracking is separate) while risk management uses the aggregate.
Capital allocation across brokers requires explicit policy: Which broker gets how much capital? What is the rebalancing policy when capital is unequally distributed due to fills? Multi-broker adds capital allocation as an operational decision layer.
Test failover before you need it: Failover triggered for the first time during a real broker outage will not work correctly. Run failover drills in sandbox: deliberately disable the primary broker adapter and verify secondary routing, reconciliation, and recovery behavior.
Kill switch must span all brokers: An emergency stop must halt order submission to all connected brokers simultaneously. A kill switch that only stops the primary broker leaves the secondary still trading.

Core Concepts

Multi-Broker Routing Architecture

A multi-broker routing layer sits above the individual broker adapters and makes venue selection decisions. The routing layer has two operational modes: normal routing (select the best venue for each order based on capability, cost, and latency) and failover routing (select the secondary when the primary is unavailable). In normal mode, you might route all US equity orders to the primary broker, crypto to a crypto-specific exchange, and futures to a futures-only broker. In failover mode, US equity orders that would normally go to the primary are redirected to the secondary.

The routing layer needs a current-state view of each adapter: connected, disconnected, circuit-breaker-open, or degraded. The adapter's health signal drives routing decisions. Define an adapter as "unavailable" when its circuit breaker has been open for more than N seconds (e.g., 30 seconds), shorter than that and you'll trigger failover on normal transient errors; longer and you waste minutes of trading time waiting to failover.

Implement the routing layer as a strategy-agnostic component: it accepts a canonical order and returns the selected venue. The strategy doesn't know which broker executed its order; it only sees the canonical OrderAck. This decoupling allows routing logic to change (add a broker, change failover policy) without touching strategy code, the same decoupling provided by individual adapter design, extended to the multi-broker level.

Test routing decisions with dependency injection: create a mock adapter registry where you can set the state of each adapter (connected vs. disconnected) and verify that the routing layer correctly selects the secondary when the primary is disconnected. Write explicit tests for: primary available → route to primary, primary unavailable → route to secondary, both unavailable → return error, primary recovers → route back to primary.

Position Management During Primary Outage

When the primary broker is unavailable, positions held there are frozen: you cannot query their current status, cannot submit close orders, and cannot receive fill updates. Your system's internal position state for the primary-broker positions becomes stale but you can't correct it without connectivity. This is the most operationally challenging aspect of a broker outage.

Define a clear policy for each position type during a primary outage: (1) Positions with stop losses: the stop is at the broker, not your system. If the stop is a broker-managed stop order, it will still trigger at the broker even while you're disconnected. When you reconnect, reconcile to discover any stops that triggered. (2) Open GTC limit orders: they remain working at the broker during the outage. Reconcile after recovery to see their current status. (3) Intraday positions you wanted to close by session end: if you can't reach the primary during the session, these positions may roll to overnight unexpectedly. Have a human escalation process for outages that last more than 30 minutes before session close.

The net-exposure problem: if you entered 100 shares long at Broker A (now unavailable) and your strategy fires a sell signal, should you sell at Broker B? This creates a split position: long 100 at A, short 100 at B. Net exposure is zero (hedged) but you have two positions that need to be closed separately when A recovers. This may be acceptable as a temporary hedge, or may be operationally undesirable. Define a policy before you need it: "during Broker A outage. Do not open new offsetting positions at Broker B; wait for Broker A recovery and close there."

When the primary recovers, the reconnect and reconciliation sequence is particularly critical in a multi-broker context. You may have built up positions at the secondary during the outage. After reconciliation reveals the primary's current positions, compute the new aggregate net position. This aggregate may differ from the strategy's target allocation, you now need to rebalance, possibly closing the secondary positions and reestablishing at the primary, or vice versa, depending on your cost and capital allocation policy.

Cross-Broker Reconciliation

Single-broker reconciliation compares your internal state to one broker's state. Cross-broker reconciliation compares your aggregate internal state to the sum of states across all brokers. The reconciliation process: for each symbol in your internal position register, sum the quantities across all venue-tagged position records. Compare this aggregate to the strategy's intended net position. Any difference triggers investigation.

Per-venue position records are essential for regulatory compliance (tax lot tracking), broker-specific cost basis, and per-venue account management. But risk management and strategy position tracking operate on the aggregate. Your data model must support both: a position object that has a total quantity plus a breakdown by venue, with each venue-specific sub-record carrying its own cost basis and lot details.

Cross-broker fill event reconciliation is more complex because fill events from different brokers arrive through different channels (different WebSockets, different REST endpoints, different event schemas) and at different times. When two fills arrive for the same symbol from different brokers within seconds of each other, your position aggregator must correctly attribute each fill to the right venue and update the per-venue and aggregate positions without double-counting or missing either fill.

Implement a central fill event bus: all broker adapters publish normalized fill events to a single stream. The position aggregator subscribes to this stream and applies fills to the position register. The fill event includes a venue tag so the aggregator knows which venue sub-record to update. This architecture ensures the order of fill processing is deterministic (events from the stream are processed in arrival order) regardless of which broker generated them first.

Capital Allocation and Rebalancing Across Brokers

Multi-broker design requires a capital allocation policy: how much of total capital is assigned to each broker. The simplest policy is static allocation (60% Broker A, 40% Broker B) reviewed quarterly. A more sophisticated policy is dynamic: route orders to whichever broker has the most available capital for the required instrument type, rebalancing by transferring capital between brokers periodically (inter-broker transfers typically settle in 1-3 business days).

Capital allocation across brokers determines which broker can execute which orders. If all available capital is in Broker A and Broker A is unavailable, Broker B has no purchasing power to execute failover orders. For failover to work, Broker B must have capital pre-positioned, meaning you've accepted some execution friction by holding capital at the secondary rather than concentrating it in the primary for better margin efficiency.

Track capital allocation drift: fills at one broker reduce buying power there and increase it nowhere else (unless you sell at Broker B, which adds buying power to B). Over time, fills accumulate at one broker, reducing its available buying power while the other broker sits with idle capital. Define a rebalancing trigger, e.g., when one broker holds more than 70% of total capital in open positions, initiate a transfer to restore the target allocation, and implement it as part of your periodic operations rather than discovering the imbalance during a failover event.

Worked Scenario

Normal state. Strategy has 200 shares of SPY at Broker A (primary). Target allocation: 60% capital at Broker A, 40% at Broker B. All orders go to Broker A.
Outage detected. At 10:15 ET, Broker A's WebSocket disconnects. After 3 reconnect attempts over 45 seconds, the circuit breaker opens. Routing layer flags Broker A as "unavailable".
Failover activated. New order routing switches to Broker B. The 200 SPY shares at Broker A are flagged as "stranded", still open, not manageable until recovery. Trading halts for SPY specifically (no new orders) to prevent creating split positions.
Strategy evaluation paused for SPY. The strategy fires a buy signal for 50 more shares of SPY at 10:17 ET. The routing layer blocks it: "primary unavailable, SPY position stranded, no new orders until recovery." Alert sent to human operator.
New positions at Broker B. A signal fires for QQQ (no existing position). The routing layer routes the QQQ buy to Broker B. This symbol has no stranded position risk.
Recovery at 10:42 ET. Broker A reconnects. Reconciliation: Broker A confirms 200 shares of SPY at $540.80 (no fills during outage, the position is unchanged). Internal state already had 200 shares in stranded state, no discrepancy. Stranded flag cleared. SPY trading resumes.
Capital rebalancing. The QQQ position at Broker B used $30,000 of Broker B's $40,000 capital. Broker A recovered with $24,000 buying power unused. A rebalancing flag triggers a review: should QQQ be left at Broker B or transferred (closed at B, reopened at A) to restore the 60/40 allocation? Human decision required; automated transfer is scheduled for next session open.

Measurement Framework

Measurement	Question to Answer
Failover events per month	How often does the failover logic activate, and what triggered each event?
Stranded position duration (minutes)	When positions are stranded at an unavailable broker, how long until the broker recovers?
Capital allocation drift (% from target)	How far has the actual capital distribution across brokers drifted from the target allocation?
Cross-broker reconciliation discrepancy rate	What fraction of reconciliation cycles find an aggregate position mismatch across brokers?
Fill event attribution accuracy	What fraction of fill events are correctly attributed to the right venue without mis-aggregation?

Common Failure Modes

Opening Offsetting Position at Secondary During Outage

Without explicit policy to block offsetting positions during a primary outage, a strategy that sees an open long at Broker A (stranded, not updating) and receives a sell signal may execute a short sale at Broker B. This creates a zero-net-exposure position that looks risk-neutral from the aggregate perspective but is operationally complex: two positions at two brokers, each requiring a separate close transaction, with separate fee and margin implications.

Dynamic low angle view of futuristic skyscrapers in a modern cityscape. — Photo by Burst via Pexels

Block position-offsetting orders at the secondary during a primary outage for any symbol with a stranded primary position. The routing layer must maintain a "stranded symbols" list during an outage and refuse to route orders for those symbols until the primary recovers and reconciliation completes.

Kill Switch Only Halts One Broker

An emergency kill switch that only sends cancel-all to the primary broker leaves the secondary broker still active. If the kill switch is triggered because of a runaway strategy, the strategy continues running through the secondary while you believe all trading has stopped. The next order, possibly the one that caused you to trigger the kill switch, goes to the secondary and executes.

The kill switch must iterate all registered broker adapters and send cancel-all instructions to each. The implementation: for (const adapter of brokerRegistry.all()) { await adapter.cancelAllOrders(); }. Test this explicitly: with two adapters registered, trigger the kill switch and verify both receive the cancel-all instruction within the kill switch's configured timeout.

Net Exposure Not Computed Across Brokers

A risk check that queries each broker's position separately and applies limits per-broker will allow a position that exceeds the limit when viewed as an aggregate. If your risk limit is 500 shares of AAPL and you have 400 at Broker A and 200 at Broker B, per-broker checks would pass (400 < 500, 200 < 500) while the aggregate of 600 shares violates the limit.

All risk limit checks must operate on the aggregate position, not individual venue positions. The aggregation layer, the component that sums positions across venues, must be part of the risk check call path. Never expose per-venue positions directly to risk checking logic; always expose only the aggregated view.

Reconciliation After Recovery Misses Fills During Outage

The reconciliation process after Broker A recovers queries current positions and compares to internal state. But positions at the time of recovery reflect the net effect of all fills during the outage, not just the current position. If an open limit order at Broker A filled and then the resulting position was partly closed by a stop during the outage, you won't see the intermediate fills in the current position. You'll only see the final position state.

After recovery, query the activity log or trade history endpoint for all fills and orders since the last confirmed event timestamp, not just the current position. This gives you the full chronological sequence of what happened during the gap, including intermediate fills that may have created and closed positions, so you can apply them in order to your internal fill event log and compute the same final state the broker has.

FAQ

When is multi-broker design worth the added complexity?

Multi-broker design is worth the complexity when any of the following are true: (1) A single broker doesn't support all the instruments or order types your strategy requires; (2) Your strategy's success depends on continuous operation during broker outages, you can't afford to be flat for the duration of an outage; (3) You're large enough that a single broker's rate limits or capital constraints restrict your operation; (4) You're actively managing regulatory risk by avoiding excessive concentration at a single venue. For smaller systems, single-broker with good reconnect and recovery logic is usually simpler and adequate.

What's the minimum capital required at the secondary broker to make failover useful?

At minimum, enough to cover your average single-day order flow for the symbols you want to route there during a failover. If your strategy typically commits $20,000 per day in new positions, the secondary broker needs at least $20,000 in buying power available. In practice, positioning 20-40% of total capital at the secondary broker provides meaningful failover capacity while allowing the primary broker to operate with the majority of available capital. Exact allocation depends on your strategy's capital requirements and your tolerance for idle capital at the secondary.

How do I handle tax lot accounting when the same symbol is held at two brokers?

Tax lot accounting must be maintained per-broker, per-lot. When you sell shares at Broker A, you close lots at Broker A. When you sell shares at Broker B, you close lots at Broker B. The tax lots at one broker are completely independent of the tax lots at the other. Your tax reporting, Schedule D, Form 8949 for US traders, must aggregate realized gains from both brokers. Most brokers provide Form 1099-B covering their transactions. You'll receive one 1099-B per broker and must sum them when filing. Ensure your internal records track the acquisition date and price for each lot at each broker to reconcile against the 1099-Bs.

What should I do about open GTC orders at the primary broker during an outage?

GTC orders at the primary broker remain working there during the outage. You cannot cancel them because you can't reach the broker. This means they might fill during the outage without your system knowing. When the primary recovers, reconciliation should find any fills that occurred and apply them to your position register. In the meantime. Do not submit new buy orders for the same symbols at the secondary broker, you risk doubling up if the GTC at the primary fills while you've also submitted a new buy at the secondary. Block new orders for symbols with GTC orders at the unavailable primary.

How do I test failover logic safely without risking real money?

Use both brokers in paper/sandbox mode for failover testing. Configure your system with Broker A sandbox as primary and Broker B sandbox as secondary. To simulate a failover: configure the routing layer to treat Broker A as unavailable (either by forcing its circuit breaker open or by using a mock adapter that always returns circuit-open status). Submit test orders and verify they route to Broker B. Then restore Broker A and verify that recovery and reconciliation work correctly. This test requires no real capital and can be run as often as needed. Schedule a quarterly failover drill and document the results.

What actually makes the decision to fail over?

A defined health signal rather than an operator noticing something is wrong. Useful inputs include consecutive request failures, latency above a threshold, a stale event stream, and reconciliation mismatches, evaluated over a window so a single failed call does not trigger a switch. The decision logic needs to run outside the component being monitored, and it needs a defined minimum dwell time after switching, otherwise an intermittent fault produces repeated flips between brokers that is worse than either state.

What is a split-brain condition in a multi-broker setup?

It occurs when the system believes a broker is unreachable and routes elsewhere, while that broker is in fact still processing earlier instructions. The result is exposure at both venues that neither side of the system knows about in full. The usual mitigations are canceling or confirming outstanding orders at the primary before treating it as failed, treating an unreachable broker as an unknown state rather than a flat one, and reconciling both brokers before resuming rather than only the one now in use.

How should failback to the primary broker be handled?