Market Data Engineering
Market Data Engineering & Feed Quality
Your strategy is only as good as your data.
A curriculum covering the data layer of automated trading: feed types, consolidated versus exchange-direct data, timestamp synchronization, corporate action adjustments, symbol master challenges, late and corrected data handling, gap detection, point-in-time storage, and data quality assurance.
What this hub covers
Every trading strategy depends on market data. Bad data does not merely reduce alpha. It creates phantom signals, corrupts backtests, and causes live systems to act on prices that never existed. This hub covers the engineering discipline that sits between raw exchange feeds and the clean, reliable data a strategy can actually use.
You will learn the four primary feed types and when to use each, how consolidated SIP feeds differ from direct exchange connections, how timestamps are assigned and why clock drift matters, how corporate actions must be reflected in historical prices, how to detect and handle late and corrected prints, how to build point-in-time storage that prevents lookahead bias, and how to build automated QA pipelines that catch feed failures before they reach your strategy.
Key principles
- Feed types serve different latency and depth needs: Trade feeds deliver every executed print; quote feeds deliver the current best bid/offer; bar feeds aggregate trades into OHLCV; depth feeds expose the full order book. Each has a different latency profile and is appropriate for different strategy types.
- Consolidated feeds trade latency for simplicity: The SIP (CTA/UTP) delivers a single consolidated feed covering all US equity exchanges but arrives 1-10 ms later than direct exchange connections. Most strategies do not need direct feeds; those competing at sub-second latency do.
- Exchange timestamps and receipt timestamps are different things: The exchange timestamps the event when it occurs; your system timestamps receipt when the packet arrives. Clock synchronization (PTP/NTP) is required to compare the two accurately.
- Every corporate action requires backward price adjustment: Splits, dividends, mergers, and spinoffs create artificial price discontinuities. Unadjusted historical prices generate phantom signals in every backtest that uses percentage returns or moving averages.
- Tickers are not persistent identifiers: Companies change their ticker, get acquired, delist, and relist. CUSIP, ISIN, and FIGI are stable across ticker changes. A strategy that uses tickers as keys will silently join data across unrelated companies after a ticker reuse.
- Exchanges issue late, corrected, and cancelled prints: A trade print is not final when it first appears. Corrections and cancellations arrive minutes to hours later, changing the official record. Real-time signal logic must buffer and re-evaluate; historical storage must record each version with a known-as-of timestamp.
- Point-in-time storage prevents lookahead bias: Storing only the latest version of historical data and querying it in a backtest means the backtest sees corrections the strategy would not have seen in real time. Point-in-time storage attaches a known-as-of date to every record so the simulation sees the world as it was at each decision point.
- Automated QA is not optional: Feed gaps, bad ticks, and missing corporate action adjustments are common and silent. Without automated cross-feed consistency checks, range validators, and completeness monitors, data errors reach strategy research undetected and inflate apparent backtest performance.
Curriculum: Market Data Engineering & Feed Quality
Ten guides and three interactive tools covering the full data pipeline: from raw exchange feed to clean, QA-verified, point-in-time data ready for strategy research and live execution.
Guides
-
Trades, Quotes, Bars, and Depth Feeds
The four primary feed types: individual trades (time and sales), quote feeds (bid/ask/size), OHLCV bars, and order book depth feeds, their content, latency, and use cases.
Guide
-
Exchange vs. Consolidated Market Data
The difference between direct exchange feeds and consolidated SIP feeds (CTA/UTP), what each includes, their relative latency, and when each is appropriate.
Guide
-
Market Data Timestamps and Clock Synchronization
Exchange timestamps, receipt timestamps, and processing timestamps, their differences, PTP/NTP synchronization, and how clock skew introduces latency bias in backtesting.
Guide
-
Corporate Actions and Historical Price Adjustments
How splits, dividends, mergers, and spinoffs require backward price adjustments, and why failing to adjust correctly corrupts every backtest that uses historical prices.
Guide
-
Symbol Masters, Tickers, and Identifier Mapping
Ticker changes, CUSIP/ISIN/FIGI identifiers, the symbol master lifecycle, and why using tickers as persistent identifiers causes silent data errors over time.
Guide
-
Late, Corrected, and Out-of-Order Market Data
How exchanges publish late, corrected, and cancelled prints, their effect on real-time signals and historical bars, and how to handle them in both contexts.
Guide
-
Missing Data, Gaps, and Bad Tick Detection
Detecting price outliers, zero-volume gaps, and feed gaps in historical data, filtering vs imputation, and how each approach affects signal research.
Guide
-
Point-in-Time Storage and Historical Replay
Storing data with a known-as-of timestamp so backtests see only information that was available at each decision point, preventing accidental lookahead.
Guide
-
Real-Time vs. Cleaned Historical Data Differences
Why the cleaned historical data you buy from a vendor does not match what you would have seen in real-time, and how that gap can make backtest signals disappear live.
Guide
-
Market Data QA and Data Lineage
Building automated QA checks for market data pipelines: range checks, cross-feed consistency, corporate action verification, and complete data lineage documentation.
Guide
Interactive Tools
-
Feed Quality Validator
Paste synthetic OHLCV or quote data to run automated QA checks: outlier detection, gap flagging, and corporate action adjustment validation.
Tool
-
Timestamp Lag Explorer
Simulate the effect of exchange vs receipt timestamp differences on signal timing, demonstrating how clock skew and latency can shift signal entry times.
Tool
-
Corporate Action Adjustment Checker
Enter a split ratio or dividend and compute the backward price adjustment factor, then verify the adjusted series matches expected continuity.
Tool
Frequently Asked Questions
What is the difference between a trade feed and a quote feed?
A trade feed (also called time and sales) reports each completed transaction: timestamp, price, size, and exchange condition codes. A quote feed reports the current best bid and offer, including size available at each price. Trade feeds tell you what happened; quote feeds tell you what is currently available. Most trading strategies need both: trades to confirm momentum and volume, quotes to measure spread and depth before entering.
Why does using raw historical prices without corporate action adjustment corrupt backtests?
When a stock splits 2-for-1, the price halves overnight. A backtest reading unadjusted prices sees what looks like a 50% gap-down, triggering phantom sell signals and wildly distorting percentage returns. Dividend distributions create similar artificial drops. Without backward adjustment, every corporate action becomes a false signal. The adjustment factor multiplies all pre-event prices so the continuous series represents economic return without discontinuities caused by capital structure changes rather than market moves.
What is point-in-time storage and why does it prevent lookahead bias?
Point-in-time storage attaches a "known-as-of" timestamp to every record in addition to the event timestamp. When a backtest replays history, it queries only records whose known-as-of timestamp is on or before the simulation date. This means the backtest sees exactly the data that was available at each decision point, including data that arrived late or was subsequently corrected. Without point-in-time storage, loading today's clean database into a historical backtest exposes it to corrections and additions that would not have been visible in real time, inflating backtest performance.
What is the SIP feed and how does it differ from direct exchange feeds?
The Securities Information Processor (SIP) consolidates trade and quote data from all US equity exchanges into a single stream. CTA (Consolidated Tape Association) covers NYSE-listed securities; UTP (Unlisted Trading Privileges) covers Nasdaq-listed securities. Direct exchange feeds bypass the SIP and deliver data faster, typically 1 to 10 milliseconds faster, but require separate connections to each exchange. The SIP is sufficient for most retail and systematic strategies. Firms competing on execution latency in sub-second timeframes use direct feeds.
What causes bad ticks in market data and how should they be handled?
Bad ticks are erroneous price or volume prints in the raw feed. Common causes include fat-finger trades later cancelled by exchanges, test orders that leaked to the tape, data feed encoding errors, and exchange system glitches. In real-time systems, trades marked with certain condition codes (error, cancel) should be excluded from signal calculations. In historical data, bad ticks usually manifest as extreme outliers, prices far outside the session range or volumes orders of magnitude above normal, and can be detected with z-score or inter-quartile range filters applied per symbol before using the data for research.
How does clock skew between a trading system and an exchange affect backtest accuracy?
Most exchange timestamps are set by atomic clocks synchronized via PTP (Precision Time Protocol) to within microseconds of UTC. A trading system whose clock drifts even a few milliseconds will record receipt timestamps that are earlier or later than the actual market event. In backtesting, if the strategy logic uses receipt timestamps as decision points rather than exchange timestamps, even 5 milliseconds of drift can shift which bar a trade appears in, change signal trigger times, and make a strategy look faster or slower than it actually was. Synchronizing servers via PTP and preserving exchange timestamps alongside receipt timestamps is the correct approach.