Direct Answer
Cleaned historical data is the result of a post-processing pipeline applied to the raw market data feed. This pipeline typically: removes bad ticks (erroneous prints with condition codes or extreme outliers), excludes extended-hours trades, applies corporate action adjustments, forward-fills brief gaps, resolves corrections by updating the original record, and may remove securities that no longer exist (survivorship bias). Each of these steps makes the historical data cleaner and more usable, but also makes it different from what a live system would have encountered at the time.
The practical consequence: a strategy backtested on cleaned historical data will generate signals at precisely calibrated indicator values, cross a level that was "cleanly" defined by cleaned data. The same strategy running live on the raw feed encounters bad ticks, gap prints, and the uncorrected original values, and at some percentage of decision points, these raw-feed imperfections shift the indicator enough that the signal does not fire, fires at a different level, or fires at a slightly different time. This is one of the persistent causes of "live-backtest divergence", the strategy performs worse live than in backtest even when the signal logic is implemented correctly.
Key Takeaways
- Cleaned historical data is a post-processed view: It represents the world as it is known today (with all corrections applied), not the world as it was known in real time. These two views differ by definition whenever any correction, revision, or vendor update has occurred since the original event.
- Bad tick removal changes volume: If a vendor's historical cleaning removes a 1,000,000-share erroneous print, the bar's volume is lower in the cleaned series than what a live system saw. Volume-based signals calibrated on cleaned history will see different volume distributions in live trading.
- Adjustment factor revisions change price history: When a vendor corrects an historical adjustment factor, all prices before that date change. A signal that uses price levels (moving averages, support/resistance) will see different absolute values in cleaned versus live data.
- Extended-hours exclusion is a common but unstated cleaning step: Many historical data vendors default to excluding pre-market and after-hours prints from their OHLCV bars. If your live system receives extended-hours prints in the first bar of the session, the live open price may differ from the historical open price in the cleaned dataset.
- Survivorship bias is the most documented but not the only form: Excluding delisted companies from historical datasets is the best-known data cleaning issue. Less obvious forms include: excluding securities below a minimum price threshold at the time of backtest construction, excluding securities with incomplete data in the backtest period, and including only securities that have "valid" price histories by today's standards.
- The solution is to backtest on data as close to raw as your signal can tolerate: For signals that are robust to individual bad ticks (e.g., daily close momentum), use cleaned data, the signal is not materially different. For signals that use intraday bar volumes or opening prices, use data with the same cleaning pipeline your live system applies, applied in real-time order (not retroactively).
- Measure the gap before going live: Compare your backtest signal outputs (indicator values and signal dates) against a simulation on the raw historical feed. The fraction of signals that would have fired at different times or not at all tells you how much of your backtest performance depends on clean data.
- Use paper trading to validate live-backtest alignment: Running the live system in paper trading mode against the live feed and comparing its signal outputs to what the backtested model predicts in the same period is the most direct validation of live-backtest consistency. Persistent divergence indicates a data pipeline difference, not a market regime change.
Core Concepts
What Cleaning Does and Does Not Preserve
Data cleaning operations can be categorized as either information-preserving (they clarify existing information without adding or removing economic content) or information-altering (they remove or modify data points that represent real, observable market events). The distinction matters for backtest validity.
Information-preserving cleaning includes: applying correct corporate action adjustment factors (transforming raw prices to reflect economic return rather than nominal price), fixing obvious data encoding errors (a price of $0.00 or $9999.99 that resulted from a binary-to-decimal conversion error), and correcting timestamps from a wrong timezone. These operations make the data more accurate without changing the economic content, a cleaned adjusted price series correctly represents the returns an investor would have experienced.
Information-altering cleaning includes: removing bad ticks from volume calculations (reducing volume from what appeared on the tape), forward-filling gaps in bars (adding synthetic prices that never traded), and excluding extended-hours prints from OHLCV bars (changing the open and close prices from what appeared on screen). These operations improve data usability for most applications but create differences between the cleaned data and the raw feed that a live system encounters.
The key question for each cleaning operation: "If I applied this same cleaning operation in real time to the live feed, would the cleaned live data match the cleaned historical data?" If the answer is yes, for example, the same bad tick filter applied in real-time produces the same outputs as applied to historical data, then the cleaned historical data is valid for backtesting a live system that applies the same filter. If the answer is no, because the cleaning was applied retrospectively (e.g., applying today's revised adjustment factors to historical prices), there is a live-backtest divergence embedded in the data.
Implementing the same cleaning pipeline in both the historical data preparation and the live data processing is the design goal. This sounds straightforward but requires discipline: every time a new cleaning step is added to the historical pipeline, it must be mirrored in the live pipeline (and vice versa), and the two pipelines must use identical logic and parameters to maintain consistency.
Common Sources of Live-Backtest Divergence from Data Differences
The most common data-source divergences are: extended-hours opening prints that appear in the live open but are excluded from historical data; bar volume differences from removed bad ticks; moving average values that differ because the historical cleaned price series uses a different adjustment factor than was in use during the backtest period; and index composition differences (the historical cleaned data reflects post-announcement membership, while the live system is seeing pre-announcement membership at each date).
Volume divergence is particularly insidious for strategies that use relative volume. If the historical cleaned data excluded a large erroneous print on a specific day, the average volume for that day in the cleaned series is lower than what appeared live. The relative volume ratio (today's volume / average volume) will compute differently on clean versus live data. A volume-surge signal calibrated to fire at "3× average volume" may fire at a different absolute volume level live than in the backtest because the average was computed from different underlying data.
Opening price divergence is common for strategies that use the open price to compute opening range signals, gap-up/gap-down signals, or intraday VWAP from open. Historical data vendors typically report the open as the first regular-session trade price, excluding any pre-market prints. The live feed shows the actual first trade in the session, which may be a pre-market print if the vendor's live data distribution includes pre-market data. The live open and historical open may differ by any pre-market price movement.
Adjustment factor timing divergence: a vendor publishes revised historical adjusted prices on T+2 after a dividend ex-date. If the backtest uses the T+2 adjusted prices to compute a moving average on the ex-date, it sees the "correct" adjusted prices. The live system, running on T+0, computes the same moving average without the dividend adjustment (which wasn't published until T+2). For strategies where the moving average crosses a level near the ex-date, the live and backtest decisions can differ depending on which version of the prices was used.
Measuring the Real-Time/Cleaned Data Gap
Quantifying the gap requires access to both raw historical data (the feed as it was received in real time) and cleaned historical data from your vendor. Comparing the two on a day-by-day, bar-by-bar basis produces a divergence profile: what fraction of bars differ between raw and clean? By how much? At which times of day are differences most common?
For daily close prices (adjusted vs. unadjusted for corporate actions), the divergence can be measured as: for each date, compare the adjusted close in the cleaned dataset to the adjusted close computed from the unadjusted price using the adjustment factors that were actually available on that date (from a PIT corporate action database). The difference gives the "phantom adjustment", the portion of the adjusted price series that was not representable in real time.
For intraday bar volumes, compare the cleaned bar volume (after bad tick removal) to the raw bar volume (direct from the feed). The ratio (cleaned / raw) shows how much volume was removed. For most sessions and most securities, this ratio is close to 1.0. For sessions with high-profile erroneous prints (flash crash events, data vendor outages), the ratio can be significantly different. Signal calibrations that depend on volume ratios should be tested across the distribution of (cleaned/raw) ratios to assess robustness.
The most direct measure is to run the same signal logic on both raw and cleaned historical data for the same period and compare the signal outputs. Track: (1) the fraction of signals that fire in both datasets on the same date/time (concordance); (2) the fraction that fire in cleaned but not raw (signals that depend on cleaned data); (3) the fraction that fire in raw but not cleaned (signals that bad ticks would incorrectly trigger in the live feed). Category (2) is particularly dangerous, it represents signals that your backtest generates but your live system will miss.
Aligning Backtest and Live Data Pipelines
The ideal architecture for a strategy that must perform consistently between backtest and live is to use the same data pipeline code for both. The live data processing code reads from the real-time feed, applies condition code filters, applies real-time corporate action adjustments (using adjustment factors available as of today), and writes to a streaming data structure. The backtest data processing code reads from the historical archived feed, applies the same code with the same parameters, and writes to the same data structure. The signal logic reads from the data structure without knowing whether it is in live or backtest mode.
This "identical pipeline" design is achievable but requires upfront architectural investment. The key requirements are: the cleaning pipeline must be parameterized so that both the live and historical modes use the same thresholds and logic; the historical archive must preserve raw feed data (not just cleaned data); and corporate action adjustments must be applied using a PIT-aware mechanism that knows what adjustment factors were available at each historical date.
For simpler implementations, the practical approach is: document every cleaning decision applied in the historical data (what bad tick filter, which extended-hours exclusion rule, which adjustment factor source and timing); implement the same decision in the live data processing code; and verify periodically that the live data matches what the cleaned historical data would show for recent sessions (where "recent" means within a period where you have both live and historical data and no major corrections have been applied yet).
Worked Scenario
A momentum team launches a live strategy after a successful backtest on cleaned OHLCV data. After 4 weeks of paper trading, the live signal matches the backtest signal only 76% of the time. They investigate the 24% divergence.
- Opening price divergence (8% of all divergences): The vendor's historical data reports opens as the first regular-session print (9:30:00 ET or later). The live feed includes 9:29:xx prints from the opening auction cross, which can differ from the 9:30:00 tape open by up to 0.5% for large-caps. The team adds a session-start filter to the live feed: only use prints with exchange timestamp ≥ 9:30:00:000 ET for the open bar, matching the historical convention.
- Volume divergence from retained bad ticks (11% of all divergences): The live feed includes prints with condition codes "Z" (out-of-sequence) and "T" (extended hours) that the historical vendor excluded. These prints inflate live bar volumes versus cleaned historical bars. Adding the same condition code filter to the live data pipeline that the vendor applies reduces volume divergence to under 2%.
- Adjustment factor timing (5% of all divergences): The team's live system applies corporate action adjustments from a vendor file published at 5:30 PM ET each day. The historical data used in the backtest was built from the same vendor's database as of today, meaning historical adjustment factors may have been revised since the original dates. For 5% of signal dates in the backtest, the adjustment factor in the current database differs from what the vendor published on the actual date. The team switches to a PIT corporate action database to resolve this.
- Residual 1.8% divergence after fixes: After implementing the three fixes, the paper trading concordance rises to 98.2%. The remaining 1.8% is attributable to sub-millisecond execution timing differences that do not affect the daily-bar signal materially.
Measurement Framework
| Measurement | Question to Answer |
|---|---|
| Signal concordance (live vs backtest, %) | What fraction of days do the live and backtested signals agree on direction, entry, and exit? |
| Opening price divergence (|live open − historical open| / historical open, %) | How often and by how much does the live open price differ from the vendor's historical open? |
| Volume divergence (live bar volume / cleaned bar volume) | What is the ratio of live to cleaned volume? Values significantly above 1 indicate bad ticks included in live but excluded in historical. |
| Adjustment factor revision frequency (revisions/year) | How often does your vendor revise historical adjustment factors, and how large are the price changes from each revision? |
| Signals triggered only in cleaned data (% of all signals) | What fraction of backtest signals would not have fired on raw data? These are signals that cannot be replicated live. |
Common Failure Modes
Assuming the Vendor's Real-Time and Historical Data Are the Same
Many data vendors offer both real-time data (for live trading) and historical data (for research). These may be produced by different pipelines, use different cleaning rules, apply adjustments at different times, and even differ in bar boundary definitions. A team that subscribes to both from the same vendor and assumes they are consistent will find divergences at every corporate action event and any day with significant bad ticks.
Before going live, explicitly compare 30-90 days of historical data from the vendor against data stored from the real-time feed for the same period. Any consistent differences identify the divergence sources. Then either adjust the historical data construction to match the real-time feed, or adjust the real-time feed processing to match the historical data, but do one or the other, not neither.
Using a Different Data Vendor for Historical vs. Live
A team builds a backtest using one vendor's historical data (with that vendor's bar-building methodology) and then subscribes to a different vendor for live data (with a different methodology). Differences in condition code filtering, bar boundary timing, open/close price definitions, and adjustment factor sources between the two vendors create systematic divergence that is not attributable to any signal logic error.
When using two different vendors, explicitly verify methodology alignment: do both vendors use the same condition codes to define regular-session trades? Do both use the same bar closing time? Do both apply adjustment factors on the same day? Document any differences and model their expected impact on signal concordance before going live.
Not Testing Signal Robustness to Data Noise
A strategy that fires signals only in highly specific conditions, e.g., when a 20-bar VWAP deviation is exactly between 1.95 and 2.05 standard deviations, is fragile to the small data differences between cleaned historical and live raw data. If the live feed's VWAP shows 1.92 standard deviations at the same moment that the cleaned historical data shows 2.00, the signal fires in the backtest but not live.
Test signal robustness by adding small random noise (±0.5% of the indicator value) to the cleaned historical data before computing signals, and measure what fraction of backtest signals survive. Low robustness, many signals disappear under small noise, indicates that the strategy's effective live performance will be significantly below the clean-data backtest. Design signals with threshold buffers or use continuous signal scores (where a value of 2.0 and a value of 1.92 produce proportionally similar action strength) rather than hard binary thresholds.
Frequently Asked Questions
What is "live-backtest divergence" and why does it happen?
Live-backtest divergence is the difference between a strategy's performance in historical simulation (backtest) and its performance in live trading. Data-layer divergence, the focus of this guide, occurs because the historical data used in the backtest is cleaner, more complete, or differently processed than the real-time data the live system encounters. Other sources of live-backtest divergence include: transaction cost underestimation (backtest assumes smaller spreads and less slippage than real fills), market impact (large orders move prices in live trading but not in backtest simulations), and latency differences (backtest assumes faster signal-to-order than the live system achieves).
Is there such a thing as "too clean" historical data?
Yes. Overly aggressive cleaning can remove data that is genuinely informative. A cleaning pipeline that removes all prints where the price moved more than 1% from the prior print will eliminate real momentum events (earnings gaps, news-driven moves) as well as bad ticks. The resulting cleaned series is too smooth, it underrepresents the true volatility and the signal's behavior during high-volatility events. Aggressive cleaning inflates Sharpe ratios in backtests by implicitly hedging out the strategy's exposure to the events that were "cleaned" away, events the live system will encounter. Calibrate cleaning aggressiveness to match what the live system can realistically filter, not to make the historical data as clean as possible.
How can I test whether my live and historical data are consistent?
Collect real-time data for a period (e.g., 30 trading days) and store the raw feed. Then compare it to what the same vendor provides in their historical data download for the same period. For each bar, compare open, high, low, close, and volume. Compute the difference as a percentage. Plot the histogram of differences. If the distributions are narrow (nearly all differences are 0%), the two are consistent. If there are systematic differences (e.g., volume is always 5% lower in historical than live), you have identified a specific cleaning step that creates divergence. Investigate each category of difference and trace it to the specific cleaning rule or timing difference that causes it.
Does using raw historical data for backtesting make backtests more accurate?
For signals that are sensitive to individual bad ticks or raw volume (e.g., volume-surge signals, single-print price signals), using raw historical data makes the backtest more accurately represent what the live system will encounter, at the cost of the backtest containing the same noise as the live feed. For signals that use aggregated, smoothed indicators (e.g., 20-day moving averages, multi-day momentum), the difference between raw and cleaned historical data has minimal effect on the indicator values, and cleaned historical data is more appropriate. Match the cleaning level of the historical data to what the live system will apply to its real-time feed.
How does survivorship bias affect cleaned historical datasets?
Survivorship bias is a form of selection cleaning: the historical dataset excludes securities that delisted, went bankrupt, or were acquired (and subsequently removed from the database). The excluded securities include both successes (acquired at a premium) and failures (bankrupt, went to zero). The surviving universe is systematically composed of companies that successfully operated for the full backtest period, which overstates average historical returns. For a typical US equity universe backtested over 10 years, excluding delisted securities can inflate annual returns by 1-3% depending on strategy type and universe. Avoiding survivorship bias requires using a database vendor that explicitly includes delisted securities in their historical universe.
What is the "open price problem" for intraday backtesting?
Many intraday backtesting strategies fire signals based on the opening price of each day or each session. The "open price" is not a single well-defined concept: it can mean the price of the first trade in the session (which may be a pre-market print), the price of the first regular-session trade (after 9:30 AM ET), the NASDAQ opening cross price, the NYSE opening auction price, or the volume-weighted average of the first N minutes. Different data vendors compute the "open" differently, and the vendor's historical open may not match what a live system sees as the first tradeable price. Clarify with your vendor exactly how the open is defined in their historical data, and implement the same definition in your live data processing.
How should I handle discovered divergences between live and backtested signals?
First, identify the source: is the divergence from data pipeline differences (the most common cause), signal logic differences (bugs in the live implementation), or market structure changes (liquidity, trading patterns that changed since the backtest period)? For data pipeline divergences, align the live and historical pipelines to use identical rules. For signal logic bugs, fix the implementation. For market structure changes, update the backtest to use more recent historical data and re-validate. Document each divergence source and its resolution. Set a concordance monitoring threshold (e.g., 95% signal concordance) and trigger an investigation if concordance drops below it in a rolling window.
Can paper trading reveal live-backtest data divergences?
Yes, and it is the most direct method. Run the live strategy in paper-trading mode against the live feed simultaneously while the backtest model runs on historical data for the same recent period. Compare signal outputs daily. Divergences that appear systematically on specific event types (ex-dates, earnings dates, high-volume days) point to specific data issues. Divergences that appear randomly suggest noise sensitivity (the signal fires near a threshold and small data differences push it to different sides). Paper trading for 20-40 business days provides a statistically meaningful sample of signal concordance, assuming the strategy generates signals at least several times per week.
Why can live and historical bar boundaries differ for the same period?
A live aggregator builds a bar from prints as they arrive and closes it on a local clock, while a vendor historical file is usually built afterwards from a complete, corrected tape using the venue own timestamps. Late prints, corrections, and any difference in which timestamp defines membership all move trades between adjacent bars. The result is two files that look like the same one-minute series and differ at the boundaries, which is enough to change any signal computed on bar closes.
References
- Shumway, T. (1997). "The Delisting Bias in CRSP Data." Journal of Finance, 52(1), 327-340. (Seminal study of survivorship bias from cleaned historical datasets)
- Maymin, Z., & Maymin, P. (2020). "Data Cleaning and the Performance of Simple Technical Rules." Journal of Portfolio Management, 46(4). (Quantification of backtest performance change from data cleaning decisions)
- de Prado, M. L. (2018). Advances in Financial Machine Learning. Wiley. Chapter 2: "The Scientific Method in Finance," including discussion of selection bias from cleaned data.
- CRSP US Stock Database: Documentation of delisting return methodology, the academic standard for survivorship-bias-free data
- Grinold, R. C., & Kahn, R. N. (2000). Active Portfolio Management, 2nd ed. McGraw-Hill. Chapter 14: implementation issues including data quality and backtest realism.
Educational Disclaimer
Data cleaning practices vary by vendor, asset class, and time period. The specific divergences described here are common but not universal. Always verify the exact cleaning methodology with your data provider and measure the live-backtest gap empirically for your specific strategy and data source.