Broker Error Taxonomy and Retry Policy

Direct Answer

Broker API errors are not all equal. Retrying a transient network error is safe and often resolves the problem within one or two attempts. Retrying a validation rejection is pointless. The broker will return the same error every time until you change the request parameters. Retrying a timeout on an order submission is dangerous, you may not know whether the order was already accepted before the timeout, meaning a retry could submit a duplicate order and double your position.

A correct retry policy classifies each error category before deciding whether to retry, and uses different strategies for each: immediate retry for transient DNS/TCP errors, exponential backoff with Retry-After for rate limits, zero retries for validation rejections, and a query-then-decide flow for ambiguous timeout states. The error code and HTTP status code alone are not enough, you need to understand what each specific code means for the order lifecycle in your broker's context.

Key Takeaways

• Never retry 4xx validation errors: A 400 Bad Request, 422 Unprocessable Entity, or broker-specific validation rejection means your request is wrong. Retrying the same request returns the same error. Fix the request, or don't retry.
• Honor Retry-After on 429: Rate limit responses include a Retry-After header or body field specifying when you may retry. Retrying before that time returns another 429. Wait exactly the specified duration, not a fixed backoff that may be shorter or longer.
• Query before retrying order submissions on timeout: A network timeout does not indicate whether the broker received and processed the request. Query the order status before retrying to determine whether to submit again or just wait for a fill event.
• 5xx server errors are usually transient: Internal server errors (500), service unavailable (503), and gateway timeouts (504) typically indicate transient broker infrastructure issues. Retry with exponential backoff, but cap the total attempts to avoid a multi-minute order delay.
• Log the full error body, not just the status code: Brokers embed structured error information in the response body, error codes, field names, constraint values. That is essential for diagnosis. Log the raw response body alongside status, timestamp, and request ID.
• Use idempotency keys to make retries safe: A client-generated order ID included in every submission allows safe retry: if the broker already has the order, it returns the existing order rather than creating a new one.
• Circuit breakers prevent retry storms: When the error rate on an endpoint exceeds a threshold, open the circuit and stop retrying. This prevents your retry logic from amplifying a broker outage.
• Classify unknown errors as ambiguous, not retryable: When you receive an error code you've never seen before and don't know whether it's transient or permanent, treat it as ambiguous. Query state before acting.

Core Concepts

The Four Error Categories

Category 1, Transient Infrastructure: Network-level errors (connection refused, DNS resolution failure, TCP reset) and HTTP 5xx responses (500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout). These errors indicate that the broker's infrastructure experienced a temporary problem. Your request may or may not have reached the broker's order processing engine. For non-order-submission requests (status queries, account queries), retry immediately with backoff. For order submissions, treat as ambiguous (see Category 4) because you can't know if the order was created.

Category 2, Rate Limits: HTTP 429 Too Many Requests. This is always safe to retry after the specified wait period. The broker is telling you explicitly that it received your request, chose not to process it, and will process it if you wait. The Retry-After header or response body field specifies the wait duration. Honor it precisely, retrying 100ms early often returns another 429. Never retry a rate limit error with a fixed backoff shorter than the specified Retry-After value.

Category 3, Validation Rejections: HTTP 400, 422, and broker-specific semantic rejections (4xx codes). These mean the broker received your request and determined it violates a rule: invalid symbol, unsupported order type, quantity below minimum, price outside valid range, account-level restriction. Retrying with identical parameters is guaranteed to fail again. The correct action is to surface the error to the strategy with the broker's error code and message, allowing the strategy to decide whether to correct and resubmit or abandon the order intent.

Category 4, Ambiguous State: Any error that doesn't clearly indicate whether the order was created or not. Timeouts are the most common: you sent an order submission request, the network timed out before you received a response, and you don't know if the broker received and processed the request before the timeout, processed it after the timeout, or never received it. Authentication errors on order submission are also ambiguous in some brokers, they may have queued the order before checking authentication. Ambiguous states require a query-before-retry flow: ask the broker for the status of any recent orders matching your parameters before deciding whether to submit again.

The Query-Before-Retry Pattern

When an order submission times out or returns an ambiguous error, the correct procedure is: (1) Wait a short delay (500ms, 2s) for any in-flight processing to complete. (2) Query the broker's open orders endpoint, filtering by your client-order-ID if supported, or by symbol and submission timestamp. (3) If found: extract the current status and treat it as if you received the original acknowledgment, no retry needed. (4) If not found after two queries spaced 1 second apart: the order was not created, and it is safe to retry the submission. (5) If found but in a terminal state (filled, canceled): emit the appropriate synthetic event and do not retry.

The query-before-retry pattern requires idempotency keys to work reliably. Without a client-order-ID that lets you find your specific order in the broker's history, you may query open orders, not find it, submit again, and then discover both orders filling simultaneously. The client-order-ID lets you ask the broker specifically: "do you have an order with this ID?" If yes. Don't retry. If no, it's safe to submit with the same ID. The broker will create the order once and deduplicate any subsequent submissions with the same ID.

Implement the query-before-retry pattern at the adapter level so every caller benefits from it automatically. The caller submits an order and awaits an OrderAck. If the adapter's internal timeout fires before the broker responds, the adapter runs the query-before-retry flow and returns the resolved OrderAck or an error to the caller. The caller never needs to implement this logic themselves.

Document in your runbooks: the max query attempts (2), the delay between queries (1 second), the timeout for declaring the order not found (5 seconds total), and the action if the order is found in an unexpected state (e.g., partially filled, emit the fill and return the partial-fill ack). These parameters are operational decisions that affect your worst-case order submission latency under ambiguous conditions.

Circuit Breaker Pattern

A circuit breaker monitors the error rate on an endpoint and automatically stops sending requests when the rate exceeds a threshold. The metaphor: an electrical circuit breaker trips when current exceeds the safe limit, cutting power to prevent damage. A software circuit breaker trips when error rate exceeds a threshold, stopping requests to prevent cascading failures and giving the broker infrastructure time to recover.

A basic circuit breaker has three states. Closed (normal): requests pass through; errors are counted. When the error rate in a rolling window (e.g., 10 errors in 60 seconds) exceeds the threshold, the circuit opens. Open: all requests return immediately with a circuit-open error; no requests reach the broker. After a timeout (e.g., 30 seconds), the circuit enters half-open state. Half-open: one request is allowed through as a probe. If it succeeds, the circuit closes; if it fails, the circuit returns to open and the timeout resets.

For broker order submission specifically, the circuit breaker must be tuned conservatively. A single duplicate order from an aggressive retry policy has real financial consequences. An error threshold of 3 failures in 30 seconds, an open duration of 60 seconds, and a single probe in half-open state is a reasonable starting point. Track the circuit state in your monitoring; alert on any transition to open state because it indicates a broker-side problem that warrants human attention.

Circuit breakers and retry logic interact: retries count as requests for circuit breaker purposes. A retry policy that retries 5 times produces 6 total requests. If all 6 fail, the circuit breaker's failure counter reflects 6 failures, potentially triggering the circuit. Ensure your circuit breaker's window and threshold are calibrated to distinguish a single failed order (6 rapid failures) from a broker-wide outage (continuous failures over 60 seconds).

Rate Limit Error Handling in Detail

A 429 from a broker contains information needed to retry correctly. The most important field is Retry-After, which may appear as an HTTP header (Retry-After: 15, meaning 15 seconds from now) or in the response body ({"error": "rate_limit", "reset_at": 1754553600}). Parse both forms and use the more specific one if both are present.

Rate limits are often per-endpoint, not global. An order submission endpoint may have a separate limit from a market data endpoint. Receiving a 429 on the order endpoint does not mean you need to pause market data requests. Your rate-limit-aware middleware should track per-endpoint counters and per-endpoint backoff state independently.

Some brokers don't return a 429 at all, they silently queue requests and process them later, potentially with significant delay. This is detectable only by measuring round-trip latency: if a broker normally responds in 50ms and suddenly responses start taking 2s, that's a sign requests are queuing. Monitor p95 latency per endpoint in addition to error rates; latency spikes often precede rate limit errors.

Proactive rate limiting, enforcing a token-bucket or leaky-bucket rate limiter in your adapter before requests reach the broker, is more reliable than reactive handling of 429 responses. Configure your adapter's rate limiter slightly below the broker's documented limit (e.g., 180 requests/minute if the limit is 200) to absorb burst variability and avoid triggering the broker's limit entirely.

Worked Scenario

1. Order submission. Your adapter submits a limit buy for 200 shares of TSLA at $245.00. A 5-second timeout fires before any response arrives.
2. Classify the error. TCP-level timeout before receiving an HTTP response. This is Category 4 (ambiguous). The adapter begins the query-before-retry flow.
3. Wait 500ms. Allow any in-flight broker processing to complete.
4. Query open orders. GET /v2/orders?status=open&symbols=TSLA. Your client-order-ID coid-7f3a2 appears in the results with status "new". The order was accepted before the timeout.
5. No retry needed. The adapter returns OrderAck{venueOrderId: "abcd1234", clientOrderId: "coid-7f3a2", status: "new"} to the caller. The order is working at the broker. Total extra latency: 0.5s wait + 0.12s query = 0.62s. The strategy proceeds normally.
6. Alternative path: not found. If the query returned no matching order, the adapter would retry the submission with the same coid-7f3a2. Alpaca would create the order fresh. If Alpaca already had a partial record for coid-7f3a2, it would return the existing record without creating a duplicate.

Measurement Framework

Common Failure Modes

Retrying a Validation Error

A retry loop that retries all errors without classification will continuously retry validation rejections. If a symbol is misspelled or an order quantity is below the minimum, each retry produces the same 400 error. The retry loop runs N times, produces N identical failed requests, consumes rate-limit budget, and ultimately times out, but never submits the correct order. During this time, the strategy may be waiting for confirmation that never comes, causing downstream logic to stall or misfire.

Classify errors before retrying. A simple rule: HTTP 4xx errors (except 429) are never retried automatically. Surface them to the caller immediately with the broker's error code and message. The caller (the adapter's consumer, usually the strategy or order manager) decides whether to correct and resubmit.

Retrying an Order Submission on Timeout Without Querying First

This is the highest-consequence failure. An order submission times out; the adapter retries without checking whether the order exists; both orders are accepted by the broker; both fill; the strategy has double the intended position at double the cost. Depending on position size, this can be a significant financial error.

Mandate the query-before-retry pattern for all order submissions. Log every instance of the pattern running, including the resolution (found or not found), and alert on cases where the order was found unexpectedly (meaning a duplicate submission was attempted and caught). Each such alert indicates the retry logic was triggered correctly and the idempotency check worked, but recurring alerts indicate your timeout threshold may be too short relative to normal broker latency.

Ignoring the Retry-After Header on 429

A system that receives a 429 and retries after a fixed 5-second backoff, rather than honoring the Retry-After header, may retry too early (getting another 429) or wait too long (unnecessarily delaying execution). In the first case, it produces a 429 storm, each premature retry generates another 429, consuming rate-limit budget and potentially extending the restriction window on some brokers.

Always parse and honor the Retry-After header. If the header specifies 30 seconds, wait 30 seconds before retrying, no more, no less. If Retry-After is absent (some brokers omit it), fall back to exponential backoff starting from 5 seconds. Log the Retry-After value alongside each 429, if it's consistently 60 seconds, your proactive rate limiter may need tuning down.

Circuit Breaker Too Sensitive, Blocking Orders During Normal Errors

A circuit breaker configured to open after 2 errors in 60 seconds will trip on normal transient error clusters that don't indicate a real outage. A single network blip that causes 2 failed order submissions will block all subsequent orders for the circuit's open duration. If the open duration is 60 seconds, a strategy that normally submits orders every 30 seconds is completely blocked for 2 minutes from what may have been a sub-second network hiccup.

Calibrate circuit breakers to your observed error baseline. If your integration normally sees 0-1 errors per 60-second window, a threshold of 5 errors in 60 seconds is a reasonable starting point, it's significantly above baseline but still well below a real outage signal. Tune the open duration to match your recovery expectations: 30 seconds for brokers that typically recover quickly from infrastructure blips, 120 seconds for brokers with longer maintenance windows.

Not Logging the Full Error Response

An error log that records only the HTTP status code and a generic error message provides almost no diagnostic value. A broker's error response body contains the specific error code, the field that failed validation, the constraint that was violated, and often a human-readable message. Without this information, diagnosing a production error requires guessing which field caused the rejection and trying different values.

Log the full HTTP response body for every error, alongside the request body (redacting sensitive fields like API keys), the HTTP status code, response headers (especially X-Request-ID or Trace-ID for broker support correlation), and a timestamp. Store these logs in a searchable format, structured JSON is ideal. The ability to search "all 400 errors on order submissions in the last 24 hours" and see the exact error codes and messages is essential for rapid diagnosis during a production incident.

FAQ

References

• AWS Builders Library: Avoiding Overload in Distributed Systems: Circuit breaker and rate limit design principles from production distributed systems engineering.
• Martin Fowler: Circuit Breaker Pattern: Canonical explanation of circuit breaker state machine design and when to use it.
• RFC 6585: Additional HTTP Status Codes: Defines HTTP 429 Too Many Requests including the Retry-After header behavior.
• Alpaca Markets: Orders API Error Codes: Example of a real broker's error code taxonomy for order submission operations.

Educational Disclaimer

This guide is for educational and informational purposes only. It does not constitute financial, investment, or legal advice. Error codes, retry behavior, and idempotency guarantees vary by broker and are subject to change. Always verify error handling behavior against current broker documentation and empirical sandbox testing.