Monitoring and Alerting for Live Algos

Direct Answer

Algo monitoring is the practice of continuously tracking a live strategy's operational and performance metrics and triggering alerts when those metrics deviate from expected ranges. It is distinct from risk controls: risk controls block or halt individual transactions based on per-trade or per-session limits; monitoring tracks cumulative trends that reveal deteriorating execution quality, signal decay, infrastructure failures, and strategy underperformance over days to weeks.

Effective monitoring covers four layers: infrastructure health (is the process running, is data arriving, are API calls succeeding?); execution quality (are fills arriving at the expected times and prices?); position integrity (do the system's holdings match the broker's records?); and strategy performance (is the live P&L trajectory consistent with backtest expectations?). Each layer requires different metrics and different alert thresholds.

Key Takeaways

• Process health monitoring — verifying the trading process is alive and producing heartbeats — is the most basic layer and must exist before any performance monitoring. A dead process that appears to be running is the most dangerous infrastructure failure.
• Data feed staleness is one of the most common and most damaging live failures. A stale feed causes the strategy to compute signals on outdated prices, often generating trades that would not have been taken on current prices.
• Slippage monitoring tracks the distribution of (fill price − signal price) across all trades. A gradual drift upward in slippage — even within the range set by pre-trade checks — signals deteriorating execution quality that may eventually erode the strategy's edge.
• Fill rate (fraction of targeted quantity that actually filled) below 85–90% over a rolling 20-trade window indicates that limit orders are being set too passively, market conditions have changed, or liquidity is lower than assumed in the backtest.
• Position drift — systematic over- or under-allocation relative to target weights — accumulates silently through partial fills, reconciliation errors, and rounding. A weekly position review comparing actual vs target weights catches this before it becomes a significant sizing deviation.
• P&L monitoring compares live daily returns against the distribution of daily returns observed in OOS backtesting at the current capital size. Sustained deviation beyond ±2 standard deviations for 5+ consecutive trading days warrants investigation.
• Alert fatigue — receiving too many alerts that turn out to be false positives — is as dangerous as receiving no alerts, because it trains the operator to ignore notifications. Threshold calibration should aim for no more than 2–3 genuine alert events per week during normal strategy operation.
• All monitoring logs should be stored persistently (not just in memory) so that a process restart does not lose the history needed to diagnose what happened before a failure.

Core Concepts

Infrastructure health monitoring

Infrastructure health covers the operational status of every component the strategy depends on: the trading process itself, the data feed connection, the broker API connection, and any supporting databases or configuration services. The primary tool for process health is the heartbeat: the trading process writes a timestamp to a file or database every N seconds, and a watchdog process checks that the timestamp is recent. If the heartbeat is stale by more than M seconds, the watchdog triggers an alert and, if configured, initiates a kill sequence or restart.

Data feed health monitoring checks that price data is arriving with expected frequency and is not anomalous. For a strategy using 1-minute bars, the feed should deliver a new bar every 60 seconds during market hours — a gap of more than 90 seconds indicates a disconnection or stale feed. Price anomaly checks flag data points that are statistically implausible: a last-trade price more than 5% from the previous close before any news event, a volume spike of 10× the 5-minute average without a corresponding price move, or a bid-ask spread wider than 3× the normal level for that symbol. These anomalies often indicate data feed errors rather than genuine market events.

Broker API health monitoring confirms that authenticated API calls are succeeding. A rising fraction of failed or rejected API calls (tracked as API error rate per hour) indicates authentication issues, rate limiting, or broker-side problems. For strategies that depend on order status polling, an API that is responding but returning stale order status creates the same execution risk as a complete disconnection — the strategy may believe an order is pending when it has already been filled or rejected.

Execution quality monitoring

Execution quality monitoring tracks the difference between intended and realized execution across every trade. The primary metric is slippage per trade: (fill price − signal mid-price) for buys, or (signal mid-price − fill price) for sells, expressed in basis points. Track this as a rolling 20-trade exponentially weighted moving average to smooth noise while remaining responsive to genuine changes. A slippage EWMA that rises by more than 50% from its initial level and stays elevated for 10+ trades warrants investigation of whether market conditions have changed, order sizing has grown relative to liquidity, or the execution timing has shifted.

Fill rate monitoring tracks what fraction of each targeted quantity actually fills. For market orders in large-cap stocks, fill rate should be near 100% — market orders against available depth fill immediately. For limit orders in smaller or less liquid names, fill rates below 80% are common and reflect the risk of non-execution inherent in passive limit strategies. The alert threshold depends on the strategy's design: a strategy built around limit orders at mid-price should expect fill rates around 60–70%; alert when the rate falls below that range for a sustained period, not on any single trade.

Order acknowledgment latency — time from order submission to broker acknowledgment — is a proxy for API health and network quality. A spike in acknowledgment latency from a normal 150 ms to 1,500 ms indicates connectivity deterioration that may affect intraday strategies. For daily strategies, acknowledgment latency matters less; for intraday strategies, sustained high latency can shift fills to significantly worse prices than the signal-time price.

Position integrity monitoring

Position integrity monitoring compares the trading system's internal position model — what it believes it holds — against the broker's confirmed position records. This comparison should happen at least once per trading session (end of day for daily strategies, every hour for intraday strategies) and after every order batch. The output is a reconciliation report: list of symbols, internal quantity, broker quantity, and any discrepancy.

Position drift is a subtler version of the same problem: even when the reconciliation shows no discrepancy (internal model matches broker), both records may show holdings that have drifted from the strategy's intended target weights. A strategy targeting 20 equal-weight positions at 5% each may, after weeks of partial fills and rounding, actually hold positions ranging from 3.8% to 6.4% of capital. Tracking the distribution of (actual weight − target weight) across all positions weekly reveals systematic biases — for example, if smaller positions consistently drift lower than target, the order sizing logic may be rounding down more than expected in illiquid names.

Strategy performance monitoring

Performance monitoring compares live results against the distribution of expected results derived from OOS backtesting. The comparison is framed in terms of standard deviations from expected: if the OOS backtest shows daily returns with mean 0.08% and standard deviation 0.40%, a live daily return of −0.95% is 2.6 standard deviations below the mean — an unusual event but within the range of expected outcomes roughly 0.5% of the time. A single such event is not alarming; five consecutive days below −1 standard deviation may indicate genuine strategy deterioration and warrants investigation.

Rolling Sharpe ratio monitoring extends this logic over time. Compute the annualized Sharpe ratio of live daily returns on a rolling 60-day window. Plot this alongside the OOS backtest's rolling Sharpe over the same window. A live rolling Sharpe that falls more than 0.5 below the backtest estimate and remains there for 30+ days is a potential signal of strategy decay or a persistent adverse regime. The challenge is that 60 days of data produces a Sharpe estimate with wide confidence intervals — statistical significance requires at least 6–12 months of live data before rolling Sharpe comparison is highly reliable.

Worked Scenario: Complete Monitoring Setup

A trader runs a weekly momentum strategy with 20 positions and $30,000 capital. Here is the full monitoring implementation:

1. Infrastructure layer: Trading process writes a heartbeat timestamp to a SQLite database every 60 seconds. A cron job checks heartbeat freshness every 5 minutes and sends an SMS via Twilio if the last heartbeat is more than 3 minutes old. Data feed health check: if any symbol in the active universe has not received a price update in more than 180 seconds during market hours, the system logs a warning and sends an email alert after 5 consecutive missing updates for any symbol.
2. Execution quality layer: After each fill, the system logs: signal mid-price, fill price, slippage in bps, fill quantity vs target. Every 10 trades, compute the EWMA slippage (lambda = 0.1). Alert fires if EWMA slippage exceeds 20 bps (vs 8 bps backtest assumption). Fill rate check: if any order fills less than 60% of target quantity, log a warning. If fill rate below 70% across any 5-trade window, send alert to review order type and limit price aggressiveness.
3. Position integrity layer: Every Friday at 4:05 PM ET, the system queries broker positions, compares against internal model, and emails a reconciliation report. Any discrepancy greater than 1 share in any position triggers an immediate SMS alert. Weight drift report: compare actual holdings weight vs 5% target per position; if any position deviates by more than 2 percentage points (actual weight above 7% or below 3%), flag for next rebalance review.
4. Performance layer: Daily at 4:15 PM, the system computes the day's P&L (realized + unrealized change) as a percentage of capital and logs it. Rolling 20-day mean and standard deviation are computed. Alert fires if daily P&L falls below (mean − 2.5 × std) — approximately once every 6 months under normal operation at this threshold. Monthly report emails the rolling 60-day Sharpe, average slippage, fill rate, and a comparison against the OOS backtest baseline.
5. Alert delivery: SMS for: process dead, daily loss limit hit, reconciliation discrepancy, slippage EWMA spike, data feed gap. Email for: fill rate warning, position weight drift, daily P&L deviation, weekly reconciliation report, monthly performance summary. No alert for: normal trades, routine fills, expected daily losses within 1 standard deviation. Result: average of 1–2 SMS alerts per month under normal operation — low enough to be taken seriously when received.

Measurement Framework

Common Failure Modes

Monitoring only the process, not the data

A trading process that is alive and connected but receiving stale or incorrect data from the feed will generate signals based on wrong prices and submit orders that would not have been taken on current market prices. Process health monitoring (heartbeat alive) is necessary but not sufficient; data health monitoring (data is fresh, plausible, and complete) must be a separate check. The most dangerous scenario is a feed that appears connected and is actively delivering data — but the data is frozen at a stale timestamp, which passes all connectivity tests while failing all freshness tests.

Alert thresholds that trigger too frequently

An alert system that sends notifications every few hours conditions the operator to ignore alerts. When a real problem fires — a process death, a data freeze, a position discrepancy — it arrives in an inbox already full of unreviewed previous alerts. Alert thresholds should be calibrated so that each alert type fires no more than once per week under normal operating conditions. If a threshold is routinely firing without identifying genuine problems, raise the threshold rather than adding it to a mental "ignore" list.

No runbook for each alert type

An alert that fires at 2 AM with no associated response procedure is nearly useless. Every alert type should have a documented runbook: what does this alert mean, what should be checked first, what actions to take if the alert is genuine, and what the expected resolution time is. A slippage EWMA alert might mean: check the last 10 fills, compare fill times against signal times, query the broker order log for any rejections, check if market conditions changed (volatility spike), and escalate to manual trading if not resolved within 30 minutes. Writing runbooks before deployment ensures the operator knows what to do under stress, rather than having to think through the diagnosis after being woken by an alert.

Assuming monitoring covers compliance

Operational monitoring tracks execution quality and system health. It does not enforce or verify regulatory compliance: order-marking requirements, pattern day trader restrictions, position reporting thresholds, or prohibited trading activity. For retail strategies operating through a standard brokerage account, the broker enforces most compliance requirements automatically. But strategies that grow in scale, add participants, or execute across multiple accounts require separate compliance tracking that is outside the scope of execution monitoring.

Not monitoring monitoring

A monitoring system can itself fail: the watchdog process may crash, the SMS service may expire its authentication token, the database where heartbeats are written may fill up. A meta-monitoring layer — typically a scheduled external check (e.g., a cron job on a second server, or a paid uptime monitoring service like UptimeRobot) that verifies the monitoring process is producing outputs — catches these second-order failures. For retail traders, a simple solution is a daily "all clear" email from the monitoring system itself: if that email does not arrive, the monitoring system has failed.

Frequently Asked Questions

Sources

• SEC Rule 15c3-5 (Market Access Rule) — risk management and monitoring requirements for automated trading
• FINRA Regulatory Notice 15-09: Supervision and control practices for algorithmic trading
• Alpaca Markets API: Order and position monitoring endpoints
• Twilio SMS API documentation — commonly used for algo alert delivery

Disclaimer

This article is for educational purposes only and does not constitute investment advice. Monitoring system design depends on strategy type, infrastructure, and broker. No monitoring framework eliminates all risk of undetected failures. Third-party tools and services referenced are for illustrative purposes only.