Market Data QA and Data Lineage

Direct Answer

Market data QA is the set of automated checks that run continuously or on each data delivery to verify that the data meets expected quality standards before it reaches strategy computation. Data lineage is the documentation of every transformation, join, and derivation that data undergoes from its source (exchange feed or vendor API) through to the signal value used to make a trading decision. Together, QA and lineage enable two critical capabilities: catching data quality problems before they affect live trading, and explaining why a strategy produced a specific output on a specific date.

A minimal but effective QA framework includes four check categories: range checks (are prices and volumes within physically possible bounds for the security?), completeness checks (are all expected bars and securities present?), cross-feed consistency checks (do values from different sources agree within acceptable tolerances?), and corporate action checks (do price discontinuities align with known adjustment events?). Each check should have a defined threshold, a severity level (warning vs. halt), and a notification channel so that problems reach humans quickly enough to be acted on before affecting live positions.

Key Takeaways

• QA must run before data reaches the signal: A QA check that runs after the signal has already computed on bad data is too late. QA gates in the data pipeline intercept data before it enters the signal computation layer, allowing the system to either reject bad data or flag signals computed on unvalidated data.
• Range checks must be per-security and context-aware: A price range check that flags anything outside [$1, $10,000] will generate false positives for Berkshire Hathaway and miss errors in a $2 penny stock that prints at $200. Per-security expected ranges should be computed from the security's recent trading history.
• Cross-feed validation catches vendor-specific errors: If the same OHLCV bar is available from two different sources (e.g., the primary data vendor and a secondary backup), comparing the two catches errors that would be invisible if you only had one source. A 5% price discrepancy between two sources for the same bar is a red flag regardless of which is right.
• Corporate action price jumps must be validated against known events: Any daily close-to-open gap greater than 15% should be cross-referenced against a corporate action database. If the gap date matches a known split, dividend, or merger. It is expected. If no corporate action is recorded. It is a data quality problem or an unreported special event.
• Data lineage must be bidirectional: You need to trace forward (given a data source, what signals and decisions depend on it?) to understand the blast radius of a data quality event, and backward (given a signal value, what raw data contributed to it?) to explain or audit any specific decision.
• Alerting thresholds should be calibrated to severity: Low-severity issues (a single security missing one bar) generate a log entry. Medium-severity issues (10% of securities missing data) generate a notification. High-severity issues (primary feed completely down) halt trading and trigger immediate response. Calibrate these thresholds to your risk tolerance and strategy sensitivity.
• QA check results are themselves data that needs to be stored: Logging which QA checks passed and failed on each data delivery, with timestamps, creates an audit trail for investigating future problems. A data quality incident that happened 6 months ago can be investigated if the QA logs are available.
• Not all data quality issues require halting trading: A missing bar for an irrelevant security should not halt an entire strategy. The QA system should know which securities and data dimensions are "critical" (trading will halt if bad) versus "informational" (flagged for review but trading continues).

Core Concepts

Range Checks: Bounding Price and Volume

Range checks are the simplest and most reliable QA mechanism. They assert that each data field falls within a physically possible and historically plausible range. For prices: lower bound is typically 0 (no negative prices for equities), upper bound is some multiple of the recent high (e.g., 5× the 90-day high). For volumes: lower bound is 0 (no negative volume), upper bound is some multiple of the average daily volume (e.g., 10× ADV for a single bar, 5× ADV for the session total). For derived fields: the high must be ≥ the open, close, and low; the low must be ≤ the open, close, and high; the close must be between the high and low inclusive.

The most important range check is the OHLCV consistency check: within any single bar, High ≥ max(Open, Close) and Low ≤ min(Open, Close). A bar where High < Close is logically impossible, the close price is the last trade price, and if it is above the "high," the high was set incorrectly. This type of error is a data pipeline bug, not a market phenomenon, and always indicates a data quality failure that must be corrected before any indicator using that bar can be trusted.

Computing dynamic range bounds per security requires a rolling lookback. For a security with recent prices in the $40-$60 range, a plausible range check might be: flag any price below $20 (50% of recent low) or above $120 (200% of recent high). These thresholds are intentionally wide to avoid false-positive flags on genuine large moves, while still catching encoding errors that produce prices of $4.00 or $6,000.00 for a $50 stock.

Range checks should also apply to timestamps. A bar timestamped before the session open, or after the session close, in a database that is supposed to contain only regular-session data, indicates either an extended-hours print that should have been excluded or a timestamp error. Flag any bars whose timestamps fall outside the expected session window (9:30 AM to 4:00 PM ET for US equities) if the dataset is supposed to be regular-session only.

Completeness Checks: Missing Bars and Securities

Completeness checks verify that all expected data is present. For a daily bar dataset: every security in the universe should have exactly one bar per trading day. For an intraday minute bar dataset: each security should have approximately 390 bars per day (one per minute of the 9:30-4:00 session), with deviations permitted for trading halts. For tick data: the sequence number coverage should be continuous (no gaps in the message sequence).

A completeness check framework for daily bars: at end of day, query the count of distinct (security, date) pairs in the database for each trading day. Compare to the expected count (number of active securities in the universe). If the count is below a threshold (e.g., 99% of expected), alert. Separately, identify which specific securities are missing and whether they were halted or otherwise legitimately inactive. A missing bar for a security in a halt is expected; a missing bar for a security that traded normally that day is a data ingestion failure.

Universe completeness checks also verify that the security master is up to date: new listings should appear in the database within 1 trading day, delistings should be marked inactive on the delisting date. A security that delisted but is still receiving "active" bar data from the vendor is a data quality problem, the vendor is potentially sending synthetic or erroneous prices for a no-longer-trading security.

For real-time completeness monitoring, track the inter-arrival time between messages for each security. If a security's normal message rate is 5-10 messages per minute and the last message was 5 minutes ago, the feed may have dropped coverage for that security. A heartbeat-style check: if no message for security X has been received in Y seconds where Y is 3× the expected inter-arrival time, emit a "stale feed" alert for that security. This check is more sensitive than comparing end-of-day completeness and can detect feed problems during the session rather than only after the close.

Cross-Feed Consistency Checks

Cross-feed consistency checks compare data from two or more independent sources for the same security and time period. If the primary feed shows a close of $50.00 and a secondary feed shows $50.02, the 0.04% difference is likely within normal rounding tolerance. If the primary shows $50.00 and secondary shows $52.00, the 4% discrepancy indicates one of the feeds has an error, and requires investigation before either is used in signal computation.

The design of a cross-feed consistency check requires defining the comparison tolerance and the resolution action. Tolerance should be expressed as a percentage of price for price comparisons, and as an absolute count or percentage for volume comparisons. Typical tolerances: close price within 0.1%, daily volume within 2%, OHLCV high/low relationship within 0.01%. When a discrepancy exceeds tolerance, the resolution policy might be: use the primary feed by default; flag the discrepancy; if discrepancy persists for N consecutive bars, escalate to manual review and potentially switch to the secondary feed.

Cross-feed checks require maintaining two data pipelines simultaneously, which adds operational cost. The minimum viable implementation for a smaller operation: use the primary vendor's data for live trading, and once a week run a cross-vendor comparison for a sample of high-trading-activity days to catch systematic differences that are not immediately visible day-to-day. Full real-time cross-feed monitoring is more appropriate for operations where data quality failures can cause significant financial loss in a short time window.

A specific high-value cross-feed check: compare the unadjusted close price to the adjusted close price divided by the published adjustment factor. The formula: adjusted_close / factor should equal unadjusted_close. If it does not, either the adjustment factor or the adjusted price is wrong. This check catches both computational errors in adjustment factor application and missing adjustment events.

Data Lineage: Documenting the Data Pipeline

Data lineage documentation describes every transformation a data point undergoes from its source to its final use in a signal. At minimum, each signal's lineage should document: the raw data source (exchange, vendor name, feed type), the ingestion timestamp, any cleaning steps applied (which filters, which thresholds), any derived calculations (adjustment factor applied, bar construction rule), any joins with reference data (symbol master, corporate action table), and the final data structure consumed by the signal.

Automated lineage tracking is implemented in modern data engineering frameworks using metadata collection at each pipeline stage. Tools like Apache Atlas, Amundsen, or dbt (for SQL-based pipelines) can capture input-output lineage at each transformation step. For simpler implementations, a manual lineage registry, a maintained document that describes each data pipeline component and its inputs and outputs, provides the same conceptual benefit at lower automation cost, though it degrades if not kept current.

The operational use of lineage documentation is twofold. Prospectively (forward tracing): when a data quality alert fires (e.g., the primary feed is down), lineage tells you which strategies, signals, and positions depend on that data source. This is the "blast radius" assessment, how much of live trading needs to be halted or degraded while the data issue is resolved. Retrospectively (backward tracing): when investigating why a signal fired on a specific date (e.g., explaining a large position entry to risk management), lineage provides the complete record of what data was used, which version of each table was queried, and which cleaning decisions were applied. Without lineage, this investigation requires reconstructing the data state from first principles, which may be impossible if the data has since been updated.

A practical minimum viable lineage standard for a quantitative trading operation: (1) document each data source with its vendor, API endpoint, authentication method, and data delivery schedule; (2) document each data transformation step with its input tables, output tables, and the code version that performed the transformation; (3) tag each signal output with the data snapshot date and version so that any historical signal value can be traced back to the specific data that produced it; (4) maintain a change log for any modification to the pipeline, including the date of the change and its expected effect on outputs.

Worked Scenario

A systematic strategy unexpectedly generates a large short position in a stock on a Monday morning. Risk management asks the team to explain the signal. The team uses their data lineage system to trace the decision.

1. Signal trace: The signal fired on Monday at 9:32:14 ET, indicating a "momentum reversal" based on the security's adjusted close price falling below its 20-day moving average. The lineage log shows which tables were read and at what version.
2. Adjusted price check: The lineage log shows the adjusted close for Friday was $48.50, down from Thursday's adjusted close of $67.20. The team checks the corporate action database for the security. A 3-for-2 stock split ex-date is recorded for Monday. The adjustment factor applied retroactively on Friday's close is 2/3 × $72.75 (Friday's unadjusted close) = $48.50. The adjusted close is correct.
3. Moving average check: The 20-day moving average as of Friday uses adjusted closes for the prior 19 days, all correctly adjusted for the split. The MA value in the lineage log: $51.20. Since $48.50 (Friday adjusted close) < $51.20 (20-day adjusted MA), the reversal signal fired.
4. QA log review: The QA log shows a "large daily return" alert for Friday (−33.5% in unadjusted price, expected for a 3-for-2 split). The QA cross-reference confirmed this matched the corporate action database entry. The alert was logged as "expected corporate action, no action required."
5. Conclusion: The signal fired correctly given the data. However, the team realizes that the moving average level before the split was approximately $76 (unadjusted), and the current price of $73 (unadjusted, first post-split trade) is near that pre-split level, suggesting the company has not actually declined. The momentum reversal signal fired because the adjustment moved the short-term price below the adjusted MA, even though the company's economic position has not changed. The team decides to add a post-corporate-action signal suppression window of 5 days to avoid signals triggered by the adjustment mechanics rather than genuine price movements.

Measurement Framework

Common Failure Modes

Alert Fatigue from Over-Sensitive QA Checks

A QA system that generates hundreds of alerts per day, most of which turn out to be false positives or expected events, will be ignored. Operations teams that see a flood of daily alerts learn to dismiss them without reading them. When a genuine critical alert appears in the flood. It is missed. The result is the worst of both worlds: the operational overhead of running a QA system without the protection it is supposed to provide.

Calibrate QA alert sensitivity by tracking the false positive rate for each check over a 30-day baseline period. Raise thresholds for checks that consistently generate alerts on expected events (e.g., a "large daily return" alert that fires on every dividend ex-date, refine it to exclude days matching known corporate action events). Use severity tiers and ensure only P1 (halt-trading) alerts require immediate human action; P2 (investigate before tomorrow) alerts can batch into a daily digest.

Lineage Documentation That Is Not Kept Current

A lineage document written once during system design and then never updated becomes a liability rather than an asset. As pipelines evolve, new data sources added, cleaning steps modified, joins to new reference tables added, the documented lineage diverges from the actual pipeline. An investigator using the stale lineage document will trace the wrong data path and draw incorrect conclusions about why a signal fired.

The fix is to treat lineage documentation as part of the code review process. Any change to a data pipeline step. However minor, requires updating the lineage documentation as part of the same code change. Automated lineage tracking (using data engineering tools that capture metadata at runtime) is more reliable than manual documentation, because it cannot go stale as long as the tracking hooks are maintained.

Cross-Feed Checks That Use the Same Upstream Source

A "cross-feed" consistency check that compares two data feeds from the same upstream vendor is not a true cross-check. If the vendor has an error in their processing (e.g., applying the wrong adjustment factor), the error will appear in both feeds identically and the consistency check will pass. True cross-feed validation requires data from independent sources, different vendors with different data pipelines. Many operations assume they have independent sources when they actually have two subscriptions to the same underlying data.

No QA Coverage for Reference Data

Most data QA systems focus on price and volume data but neglect reference data (security master, corporate action tables, index composition). Reference data errors are often more impactful than price errors: a wrong corporate action adjustment factor affects every bar before the event date; a wrong delisting date keeps a defunct security in the active universe; a missing index rebalance creates phantom positions in an index-following strategy. QA coverage must extend to all reference data dimensions, with checks for consistency, completeness, and expected change frequency.

Frequently Asked Questions

References

• Redman, T. C. (2008). Data Driven: Profiting from Your Most Important Business Asset. Harvard Business School Press. (Framework for organizational data quality management)
• Great Expectations: Open-source data quality framework with pipeline integration, widely used for financial data QA
• dbt Data Tests documentation: SQL-based data quality testing framework with lineage tracking built in
• Dalianis, H. (2018). Clinical Text Mining: Secondary Use of Electronic Patient Records. Springer. Chapter 5: Data Quality Dimensions. (Widely cited taxonomy of data quality: completeness, accuracy, consistency, timeliness)
• FINRA Rule 3110: Supervision (requires supervision of data and system quality for broker-dealers; represents the regulatory floor for market data oversight)

Educational Disclaimer

Data quality management requirements vary by organization size, strategy type, and regulatory jurisdiction. The QA checks and lineage practices described here are educational examples. Consult with your compliance and risk management teams to determine appropriate quality standards for your specific operation.