Direct Answer

Market data quality problems fall into three categories. Price outliers (bad ticks) are individual data points with prices far outside the normal range, typically caused by erroneous trade prints, data encoding errors, or uncancelled test orders. Volume gaps are periods of zero or anomalously low volume within a session, caused by trading halts, feed interruptions, or sparse trading in illiquid securities. Feed gaps are missing records where an entire time interval has no data, caused by data ingestion failures, vendor outages, or incomplete historical datasets.

Each problem type has two potential responses: filtering (removing or ignoring the problematic data point) or imputation (replacing it with an estimated value). Filtering is safer for production signal computation, it avoids introducing synthetic data into the signal pipeline. Imputation is appropriate when the data point is needed for indicator continuity (e.g., a moving average breaks if there are NaN values in the series) and when the imputed value can be estimated reliably. Documenting which strategy was used and why is as important as the decision itself.

Key Takeaways

  • Bad ticks look like sharp spikes: A price that is 5×, 100× the neighboring prices is almost always a data error, not a real trade. Real prices can gap, but they do not usually instantaneously return to the prior level after a single print.
  • z-score and IQR are the standard outlier detectors: z-score = (price − mean) / std; flag points where |z| > 3 or 4. IQR = (Q3 − Q1); flag points outside [Q1 − 1.5×IQR, Q3 + 1.5×IQR]. IQR is more robust to the outlier itself distorting the statistics.
  • Zero-volume bars are not always gaps: Illiquid securities may have zero trades in many 1-minute bars. Zero volume in a liquid large-cap during regular session hours is a signal of a feed problem; zero volume in a micro-cap is expected. Gap detection must be calibrated to the security's normal trading pattern.
  • Halts are legitimate zero-volume periods: A trading halt (LULD halt, news halt, regulatory halt) creates a zero-volume period that is not a data error. Distinguishing legitimate halts from data gaps requires cross-referencing halt messages from the exchange or an event database.
  • Filtering by condition code is the first line of defense: Many bad ticks arrive with condition codes (cancel, error) that identify them as non-standard. Applying condition code filters before applying statistical outlier detection reduces the false-positive rate significantly.
  • Imputation introduces synthetic data: Forward-filling a missing bar with the prior close creates a flat bar with the prior close as open, high, low, and close. This is the most common imputation approach but creates a bar that never traded, if a signal fires on a "synthetic" bar. It is acting on fabricated data.
  • Gap detection must account for session boundaries: The gap between Friday's close and Monday's open is a normal weekend gap, not a data error. Day boundary logic must treat session boundaries as valid no-data periods, not feed failures.
  • Every data cleaning decision should be logged: Record which data points were removed or imputed, why, and using what method. Without this audit trail, you cannot reproduce a historical backtest result or explain why the live system behaved differently on a specific date.

Core Concepts

Price Outlier Detection: z-score, IQR, and Range Checks

The most common bad tick pattern is a spike: a single price that is far above or below neighboring prices, followed by a return to the prior price range. For a stock trading at $50, a print at $5 or $500 is almost certainly an error. Statistical detection methods identify these automatically without requiring manual inspection of millions of data points.

The z-score method computes the standard deviation of prices over a rolling window, then flags any price whose absolute z-score exceeds a threshold. Formula: z = (price − μ) / σ, where μ and σ are computed over a reference window (e.g., 20 bars). Threshold: |z| > 4 flags the point for review. A limitation is that if multiple outliers are present in the window, they inflate σ and make the z-scores of the outliers smaller, reducing detection sensitivity.

The IQR method is more robust because it uses the interquartile range rather than the standard deviation, making it resistant to the outlier distorting its own detection. Compute Q1 (25th percentile) and Q3 (75th percentile) of the reference window. Flag prices below Q1 − k×IQR or above Q3 + k×IQR, where k = 1.5 for standard outlier detection or 3.0 for extreme outliers only. The IQR method is the standard for market data cleaning when the data may contain multiple outliers in the detection window.

A simpler and often effective approach for intraday data is the range check: flag any price that is more than X% from the prior price (e.g., 5% for liquid large-caps, 20% for micro-caps). This does not require computing statistics over a window and is easy to implement in a streaming pipeline. The threshold should be calibrated per security type: for index futures, a 1% single-print move may warrant investigation; for a volatile penny stock, 20% swings are possible and real.

After detection, the decision is filter or impute. Filtering removes the outlier and treats it as missing data. The subsequent signal computation ignores the point. Imputation replaces it with an estimated value, typically the average of the neighboring prices (linear interpolation), the prior value (forward fill), or the median of the window. Filtering is almost always safer: imputed values are synthetic and can create their own artifacts if the signal is sensitive to the exact price value rather than just its presence or absence.

Volume Gap Detection: Zero Volume and Low-Activity Periods

Volume gaps appear as bars with zero volume or anomalously low volume within a period when the security should be actively trading. For a large-cap S&P 500 component, any 1-minute bar during regular trading hours with zero volume is suspicious. For a small-cap with average daily volume of 50,000 shares, zero volume in a 1-minute bar is normal for much of the day.

The practical approach is to compute each security's expected minimum volume per bar from its recent trading history. For a security averaging 5,000 shares per 1-minute bar, any bar below 50 shares (1% of average) flags as a candidate gap. This per-security calibration is more accurate than a fixed threshold that works for liquid securities but generates thousands of false positives for illiquid ones.

Trading halts create legitimate zero-volume periods. When an exchange halts trading (LULD halt, news pending, regulatory halt), no trades execute until the halt lifts. These periods appear as gaps in the data but represent actual market conditions, not data quality failures. Distinguishing halt-gaps from feed-gaps requires a halt event database: for each halt event (announced via the exchange or SIP), mark the corresponding time window as "halted" and exclude it from data quality flags. If the period is flagged as a gap and there is no halt record. It is a likely feed failure.

The operational consequence of undetected volume gaps depends on the signal. A VWAP computation that spans a 10-minute feed gap will use only the minutes when data was present, incorrectly calculating the average. A volume-weighted moving average that includes gap periods in the denominator will understate the weight of surrounding periods. Signal logic should explicitly check for and handle missing periods rather than assuming all expected bars are present.

Feed Gap Detection: Missing Bars and Sequence Breaks

A feed gap is the absence of an entire time interval's data. For minute bars, a gap is a missing row in the time series. For tick data, a gap is an interval with no messages where messages are expected. Feed gaps can result from: data vendor outages, network disruptions between exchange and vendor, database ingestion failures, or deliberate exclusions in a vendor's historical dataset.

Detection is straightforward for regularly spaced bar data: compare consecutive timestamps. If the gap between timestamp T and T+1 is greater than expected bar size × 1.1 (allowing for small rounding), a gap exists. For a 1-minute bar, any consecutive timestamp difference greater than 65 seconds (65 seconds instead of 60 to allow rounding) flags a gap. The gap length = (T+1 − T) / bar_size − 1 gives the number of missing bars.

For tick data, gap detection is more complex: the expected tick rate varies with market conditions. During normal trading, a liquid stock generates 5-50 ticks per second; during low-activity periods, ticks may be 10+ seconds apart. A sequence number gap in the feed protocol (e.g., ITCH sequence numbers jump from 1000 to 1050) is the clearest indicator of missing messages, independent of time elapsed.

The correct response to a detected feed gap depends on the gap duration and the strategy's tolerance for missing data. Short gaps (1-5 minutes) in a daily-bar strategy are usually harmless. Short gaps in an intraday strategy can cause indicators to compute on too few bars and produce stale signals. Long gaps (30+ minutes) in any real-time strategy require the system to declare "stale data" and halt trading until data resumes and indicators can be re-established. Gap handling logic must be designed as a first-class part of the live system, not an afterthought.

Choosing Between Filtering and Imputation

The filtering versus imputation decision is a tradeoff between data completeness and data accuracy. Filtering produces incomplete data (missing values where the bad point was) but ensures no synthetic values enter the pipeline. Imputation produces complete data (no NaN values) but at the cost of introducing estimated values that may not accurately reflect market conditions.

For indicator computation, many functions (moving average, standard deviation, correlation) either error on NaN values or produce NaN outputs when any input is NaN, propagating the gap through the indicator. The standard solutions are: (1) use NaN-safe functions (e.g., pandas' rolling(window, min_periods=1)) that compute over however many non-NaN values exist; (2) forward-fill missing values with the last known value; (3) impute with a conservative estimate (e.g., the midpoint of surrounding prices for a price gap; zero for a volume gap).

The choice should be documented per data type and per use case. A reasonable policy: apply condition code filtering and IQR-based outlier filtering as the first pass (filtering bad ticks without imputation); then apply forward-fill for isolated single-bar gaps in bar data (imputing a flat bar to maintain indicator continuity); and flag any gap longer than 5 bars as a data quality event requiring manual review or a different handling policy (suspend trading rather than impute a long synthetic sequence).

Worked Scenario

A research team building a daily momentum signal on a 2,000-stock universe discovers that their signal has positive returns on most stocks but several tickers show anomalously extreme performance, one showing a 3,000% monthly return. They investigate.

  1. Identify the outlier: Ticker XYZQ shows a single day close of $1.50 on Day T, then $47.50 on Day T+1, then $1.52 on Day T+2. The "return" is 3,067%, then a −96.8% reversal the next day.
  2. Check corporate actions: No split, dividend, or merger is recorded for XYZQ around this date. This rules out an adjustment issue.
  3. Check condition codes: The raw trade data for Day T+1 shows the $47.50 close came from a single print at the session close, marked with condition code "Z" (out-of-sequence). This is a delayed after-hours print from the prior day that arrived in the next session's data.
  4. IQR check: Computing the IQR of XYZQ's daily close over the prior 20 days: Q1 = $1.40, Q3 = $1.65, IQR = $0.25. Upper bound = $1.65 + 3 × $0.25 = $2.40. The $47.50 print is far above this bound, confirmed as an outlier.
  5. Fix: Apply the IQR filter as a post-processing step to the daily close price series for all 2,000 tickers. Replace flagged outlier closes with the forward-filled prior close (conservative: the signal treats the day as flat rather than acting on the phantom price). Re-run the momentum signal with the cleaned data. The XYZQ anomaly disappears; aggregate strategy performance changes by −0.3% in annual return, suggesting a handful of similar outlier events were affecting the results.
  6. Pipeline fix: Add the IQR outlier check to the data ingestion pipeline so it runs automatically on each daily update, logging all flagged points for human review.

Measurement Framework

MeasurementQuestion to Answer
Outlier rate (% of prices flagged per day)Is the frequency of price outliers consistent across days, or are there event-driven spikes that need investigation?
Volume gap frequency (bars/session/security)How often does each security show zero-volume bars, and is this consistent with its typical trading pattern?
Feed gap rate (gaps/session)How often does the data pipeline fail to deliver expected bars? Is this vendor-specific or security-specific?
Halt-flagged periods (% of gaps)What fraction of zero-volume gaps correspond to documented exchange halt events?
Post-filter missing rate (%)After filtering, what fraction of expected data points remain missing? Is this acceptable for signal computation?
Imputation frequency (imputed bars / total bars)How often is imputation applied, and is the rate stable? A rising imputation rate suggests deteriorating data quality.

Common Failure Modes

Using the Global Mean for z-score Instead of Per-Security Statistics

A team applies a global z-score filter: compute the mean and standard deviation across all 2,000 securities' close prices and flag prices more than 4 standard deviations from the global mean. This approach fails because the global mean is dominated by high-priced securities (e.g., Berkshire Hathaway at $500,000/share), and low-priced securities (e.g., a $2 penny stock) always score as extreme outliers relative to the global statistics. Outlier detection must be applied per-security, using each security's own price history for the reference distribution.

Close-up of an incomplete white puzzle with one missing piece, symbolizing challenge and strategy.
Photo by Pixabay via Pexels

Forward-Filling Gaps Without Session Boundary Checks

A forward-fill imputation that does not check session boundaries will carry Friday's close through Saturday, Sunday, and into Monday, creating two "trading days" of flat bars that never happened. These synthetic bars will appear in any rolling window that spans the weekend, inflating the number of "bars" in the window and making the rolling statistics stale. Always apply gap-filling logic only within session hours and explicitly exclude non-trading periods from the time index before computing rolling statistics.

Classifying Legitimate Illiquid-Stock Gaps as Errors

A data quality check that flags any 1-minute bar with zero volume as a feed error will generate thousands of false positives for micro-cap and small-cap securities that genuinely trade at low frequency. A security averaging 5,000 shares per day may have zero trades in 300 of the 390 1-minute bars in a regular session. Applying the same zero-volume check to liquid and illiquid securities without per-security calibration wastes review resources on non-errors and may incorrectly exclude valid data from illiquid securities.

Not Logging Data Quality Decisions

A pipeline that silently filters or imputes data without logging which points were affected makes it impossible to reproduce a specific backtest result if the data quality issue recurs or is later resolved differently. If a backtest from six months ago depended on a specific set of data quality decisions, and those decisions are not logged, you cannot confirm whether the current results match the original or have diverged due to pipeline changes. All filtering and imputation events should be written to an audit log with: timestamp, security, bar/tick identifier, original value, action taken (filter/impute), new value if imputed, and reason code.

Frequently Asked Questions

What is a "bad tick" and how common are they?

A bad tick is a price or volume data point that does not represent a real market event, it results from a data error such as an encoding mistake, a test order that leaked to the tape, or an erroneous print later cancelled. In raw trade data for US equities, bad ticks are uncommon in absolute terms, typically less than 0.01% of all prints, but because strategies may process millions of data points, even rare events cause real problems. Bad ticks tend to cluster during high-volatility periods and exchange system events, so their frequency is non-uniform across dates.

What is the difference between filtering and imputation?

Filtering removes a detected bad data point and leaves a gap (NaN or missing value) in the series. The signal computation must then handle the missing value, either by skipping it, using a reduced sample, or propagating NaN. Imputation replaces the bad point with an estimated value, typically the prior value (forward fill), the average of neighbors (linear interpolation), or a statistical estimate. Filtering is more conservative (no synthetic data enters the pipeline) but requires downstream handling of missing values. Imputation is more convenient for indicator continuity but introduces risk that the estimated value causes spurious signals if the estimate is wrong.

How do I detect feed gaps programmatically?

For regularly spaced bar data: sort by timestamp, compute the difference between consecutive timestamps, compare to the expected bar interval. Any gap greater than (expected_interval × 1.1) is a candidate gap. Divide the gap length by the expected interval to count missing bars. For tick data: use sequence numbers from the feed protocol (ITCH, OPRA, etc.), a sequence number gap indicates exactly how many messages were lost. Check for sequence gaps in real time with a counter and alert on any discontinuity.

Should I use z-score or IQR for outlier detection?

IQR is generally preferred for market data because it is resistant to masking, in a dataset with multiple outliers, the outliers themselves inflate the standard deviation in the z-score method, making each outlier's z-score smaller than it should be, reducing detection. IQR uses the 25th and 75th percentiles, which are less affected by extreme values. Use IQR with k=1.5 for moderate outlier detection or k=3.0 for extreme outliers only. Use z-score when you have high confidence that outliers are rare and isolated, and the distribution is approximately normal.

How should I handle data gaps in a live trading system?

The correct response depends on the gap duration relative to the strategy's signal window. For a gap shorter than 10% of the indicator's lookback period, forward-fill and continue trading. For a gap between 10% and 50% of the lookback, flag the signal as unreliable and reduce position size or halt new entries. For a gap longer than 50% of the lookback, halt trading on the affected security entirely and wait for enough fresh data to fully re-initialize the indicator before resuming. Document these thresholds in the strategy specification and implement them as explicit logic in the live system, not as ad hoc responses to individual events.

Can a legitimate price move look like a bad tick?

Yes. Real flash crashes, gap-ups on earnings, and short squeezes can produce price moves that are just as extreme as bad ticks. The z-score or IQR filter may incorrectly flag these as outliers and exclude them. This is a fundamental tension in data cleaning: aggressive filtering removes more bad data but also excludes more real events; conservative filtering lets more real events through but also lets more errors through. One practical resolution: apply aggressive filters to the data used for indicator computation (where one bad tick can corrupt a rolling average) and less aggressive filters to the data used for event detection (where a genuine large move should not be excluded).

How do I distinguish a trading halt from a data feed gap?

Check your exchange halt event database. Major exchanges publish halt notifications via the SIP feed and via separate halt announcements. The CTA/UTP feed includes "Trading Status" messages that indicate when a security is halted and when it resumes. If a zero-volume period coincides with a halt notification in the feed or in a separate halt database (e.g., FINRA's trading halts page). It is a legitimate halt. If no halt is recorded but the data is missing. It is likely a feed failure. For historical data, FINRA's trading halts page and the exchanges' own historical halt records can be used to validate historical gap classifications (SEC EDGAR covers corporate filings, not trading halt data).

What is the effect of bad ticks on moving averages?

A single bad tick included in a rolling window can distort the entire window's moving average for as long as the bad tick remains in the window. For a 20-bar simple moving average, a tick 10× the normal price will shift the average by approximately 0.5× the normal price for 20 bars, inflating every close-to-MA comparison for 20 bars and potentially triggering false crossover signals in both directions as the bad tick enters and exits the window. Exponential moving averages are somewhat less sensitive to bad ticks because the weight decays exponentially, but a very extreme outlier still distorts an EMA for many periods after it passes. Filtering bad ticks before computing moving averages is essential for moving-average-based signals.

How should detection thresholds be recalibrated across instruments with different volatility?

A fixed percentage threshold flags routine moves in a volatile instrument and misses genuine errors in a quiet one. Scaling the threshold by a recent volatility estimate for each instrument, such as a rolling standard deviation of returns or an average true range, makes the sensitivity comparable across a universe. The estimate itself has to be computed from data that has already passed a basic consistency check, otherwise a bad tick inflates the volatility measure that is supposed to catch it.

References

  • Barber, B. M., & Odean, T. (2000). "Trading Is Hazardous to Your Wealth." Journal of Finance, 55(2), 773-806. (Context for data quality effects on performance measurement)
  • Falkenberry, T. N. (2002). "High Frequency Data Filtering." Tick Data, Inc. white paper. (Practitioner framework for trade-level data quality filtering)
  • FINRA Trading Halts: Historical trading halt notifications for cross-referencing gap events
  • Brownlees, C. T., & Gallo, G. M. (2006). "Financial econometric analysis at ultra-high frequency: Data handling concerns." Computational Statistics & Data Analysis, 51(4), 2232-2245. (Academic treatment of high-frequency data cleaning)
  • pandas rolling documentation: min_periods parameter for handling NaN values in rolling computations

Educational Disclaimer

Data quality thresholds and cleaning methodologies are strategy- and data-source-specific. The approaches described here are educational starting points. Always calibrate filter thresholds against your specific data source and validate that cleaning decisions do not introduce systematic bias into your research.