Write a Falsifiable Trading Hypothesis
Direct Answer
A falsifiable trading hypothesis is a precise, written claim about the relationship between a signal and future price behavior that specifies in advance the exact result that would count as refutation. It names the instrument universe, data period, signal computation, entry and exit rules, cost assumptions, and a primary metric with a threshold, so that any two independent researchers applying the rules to the same data reach the same conclusion.
Without this precision, a backtest cannot validate a strategy, it can only describe what happened in one historical sample with one particular parameter configuration. Vague hypotheses produce results that can be rationalized post hoc regardless of what the data shows, which is worse than no test at all because it creates false confidence. The goal is to design the test so that a negative result provides real information about the strategy idea, not just about one specific variant that failed.
Key Takeaways
- Falsifiability requires a refutation condition: Write down exactly what result would cause you to reject the hypothesis before running the test. If no result could refute it. It is not a scientific hypothesis.
- Precision removes interpretive latitude: Every parameter, universe, look-back window, threshold, cost, must be defined so precisely that two researchers would make identical trading decisions on every historical day.
- One primary metric, chosen in advance: Selecting the metric after seeing the results and choosing the one that looks best is a form of multiple testing. Name the primary metric before running the backtest.
- The null hypothesis is no edge: The default assumption is that the signal has no predictive power. The burden is on the evidence to disprove that, not on the researcher to cherry-pick confirmations.
- Revisions start a new hypothesis: Adjusting parameters after seeing results is legitimate research, but only when logged as a new hypothesis tested on fresh or held-out data, not as a refinement of the original.
- Cost assumptions are part of the hypothesis: A hypothesis that assumes zero commissions and zero slippage is a different hypothesis from one that assumes realistic transaction costs. State them explicitly.
- Strategy ideas are not hypotheses: "Momentum tends to work" is an idea. "S&P 500 stocks ranked by 12-1 month return, rebalanced monthly, produce a Sharpe above 0.5 over 2005-2023 net of 10bps one-way costs" is a hypothesis.
- Scope determines how much a refutation tells you: A narrow hypothesis (one sector, one time period) tells you less than a broad one if refuted. But a broad hypothesis that passes is also stronger evidence of a real effect.
Core Concepts
What makes a hypothesis falsifiable
Karl Popper's criterion for a scientific claim is that it must be possible for it to be proven wrong by evidence. In trading research. This means the hypothesis must specify what observable outcome would constitute refutation. A claim like "quality stocks outperform over the long run" cannot be refuted, if it underperforms for a decade you can always say the horizon was not long enough. A claim like "stocks in the top quintile of ROE, rebalanced annually, produce a mean annual return at least 2 percentage points above the Russell 1000 over 2000-2020" can be refuted if the measured outperformance is negative or below 2%.
The falsification condition should be written before any data is examined. Once results are visible, there is a strong unconscious tendency to adjust the condition to match the result, not out of dishonesty, but because the human pattern-matching system is optimized to find explanations for what it observes, not to reject them.
Practically, the falsification condition has two parts: a direction (the signal is expected to produce positive excess returns over the benchmark) and a magnitude threshold (the Sharpe ratio of the long-short portfolio must exceed 0.3, or the information coefficient must exceed 0.05). The threshold should be set at a level that is economically meaningful, not just statistically non-zero, a Sharpe of 0.02 is technically positive but not worth pursuing after transaction costs and drawdown risk.
Specifying a threshold in advance also clarifies the difference between a strategy that is genuinely useful and one that merely passed a mechanical significance test. A Sharpe of 0.6 in the sample is much more promising than a Sharpe of 0.15, even though both might be "statistically significant" depending on sample size.
The null hypothesis and burden of proof
The null hypothesis in strategy research is the claim that the signal has no relationship with future returns, that any apparent pattern in the sample is consistent with random noise. Framing it this way places the burden of evidence on the researcher, which is the correct scientific stance. The alternative is to start from an assumption that the strategy works and look only for evidence supporting that view, which is a recipe for finding patterns in noise.
In practice, the null hypothesis for most mean-return tests can be stated as: the true population mean return of the strategy is zero (or equal to the benchmark return). The alternative is that it is positive. The test then asks whether the observed sample mean is large enough, relative to its standard error, to reject the null at a chosen significance level, typically 5% for a one-sided test, which corresponds to a t-statistic above approximately 1.65.
A Sharpe ratio above 1.0 over a 10-year backtest (roughly 120 monthly observations) typically corresponds to a t-statistic comfortably above 1.65. A Sharpe of 0.4 over the same period would produce a t-statistic near 1.3, not sufficient to reject the null at conventional levels. This matters because many published and publicly shared backtests report Sharpe ratios in the 0.3-0.6 range over short periods as though they were validated results.
The significance level itself should also be specified in advance. Choosing a 10% threshold after observing a result that passes at 10% but fails at 5% is changing the rules mid-test. Some researchers adopt a more conservative threshold (1% or 0.1%) when testing many related hypotheses, to account for the fact that stringent screening is a form of multiple testing even when each individual test uses the same data.
Structural components of a complete hypothesis
A complete hypothesis statement has seven required components. The instrument universe names every eligible asset and any filters applied before the strategy can invest in it (market cap above $1 billion, average daily dollar volume above $5 million, listed on NYSE or NASDAQ, primary share only). The data period names the start and end date and the price and accounting data frequency. The signal names the factor or indicator being tested, with its exact computation written out as a formula, not a concept, not a reference to a named publication, but the actual arithmetic.
The entry rules describe when a position is opened (on the rebalancing date when a stock enters the top decile of the ranked signal, at the next-day open to avoid execution-day close prices). The exit rules describe when a position is closed (on the next rebalancing date when it falls out of the top decile, or immediately if a hard stop-loss level is hit). The cost assumptions state the commission per share, the assumed slippage model (a fixed number of basis points, or a function of average daily volume), and any borrowing costs for short positions.
Finally, the primary evaluation metric is named, along with its threshold. "The annualized Sharpe ratio of the long-only portfolio, computed from monthly returns, must exceed 0.6 over the full in-sample period" is a complete statement. "The strategy must perform well" is not.
Scope and generalizability
A hypothesis tested on a narrow universe over a short period provides weak evidence even when it passes. If a momentum strategy is tested only on large-cap US equities from 2010 to 2022, a period of persistent upward trend and historically low volatility, a passing result is consistent both with a genuine momentum effect and with a period-specific beta to a bull market. Broadening the test to cover multiple market regimes, or testing independently on non-overlapping periods, adds evidential weight.
This does not mean every hypothesis must be tested globally before it is taken seriously. A hypothesis about US small-cap earnings surprises does not need to pass on European mid-caps to be relevant. But the scope claimed in the hypothesis should match the scope of the test, a narrow test should not be presented as evidence of a universal effect.
One useful discipline is to specify, before testing, whether the hypothesis is meant to describe a universal mechanism or a regime-specific pattern. A hypothesis claiming "low-volatility stocks outperform during bear markets in US equities" is more honest and more falsifiable than "low-volatility stocks outperform", it makes a conditional claim that can be tested on bear-market-classified periods, and it does not over-claim by asserting perpetual outperformance.
Worked Scenario
A researcher has observed that stocks with high short interest tend to experience sharp price increases after positive earnings surprises, a short squeeze dynamic. Converting this observation into a testable hypothesis:
- Universe: Russell 2000 component stocks with average daily dollar volume above $3 million, price above $5, not in the financial or utility sectors.
- Data period: 2015-01-01 to 2024-12-31, using daily price data and quarterly earnings data with a 2-day reporting lag to avoid look-ahead.
- Signal: Short interest as a percentage of float, lagged 15 days (using the semi-monthly SI reporting schedule), multiplied by earnings surprise magnitude (actual minus consensus estimate, divided by the absolute value of the consensus estimate). High signal = high short interest and high positive earnings surprise.
- Entry: Buy at the next-day open following an earnings announcement when the composite signal exceeds 1.5 standard deviations above its trailing 12-month mean.
- Exit: Close at the close of the 10th trading day following entry.
- Costs: 0.5% one-way (bid-ask spread + commissions), 1% annualized borrow cost on short interest adjustment.
- Primary metric: Mean 10-day return net of costs, with a one-sided t-test; threshold for success is a mean return above 1.5% with a t-statistic above 1.96 (5% significance, one-sided).
- Null hypothesis: The true mean 10-day return of this strategy is 0% or below.
With these specifications written before running the test, the researcher cannot later change the holding period from 10 days to 7 days because the result was better at 7, or lower the threshold from 1.5% to 1.0% because the result came in at 1.2%. If the result fails the threshold. It is refuted, and the research log reflects that with a note on what was learned.
Measurement Framework
| Measurement | Question it answers |
|---|---|
| Mean return vs benchmark (annualized) | Does the signal produce positive excess returns over the period? |
| Sharpe ratio | Is the return large relative to its volatility, suggesting consistency rather than lucky concentrated gains? |
| t-statistic of mean return | Is the observed excess return statistically distinguishable from zero at the pre-specified significance level? |
| Maximum drawdown | What is the largest peak-to-trough loss, and is it consistent with an implementation-viable strategy? |
| Win rate and average win/loss ratio | What fraction of trades are profitable, and is the ratio of average wins to average losses consistent with the strategy thesis? |
| Number of trades | Is the sample large enough for the t-statistic to be meaningful, or is the result based on too few observations? |
Common Failure Modes
The untestable narrative
Many initial strategy ideas are expressed as narratives, "when markets are fearful, oversold quality stocks bounce back strongly." Every word in this narrative is undefined: fearful (by what measure?), oversold (relative to what?), quality (which metric?), bounce back (over what horizon?), strongly (by how much?). Until each is replaced by a specific, computable definition, no test can confirm or refute the claim.
The danger is not that the idea is wrong, it might be right. The danger is that the narrative form allows any result to be accommodated. If the backtest fails, the researcher can claim the market was not fearful enough, or that the definition of quality should have used ROIC instead of ROE. Without prior specification, every failure mode is available as an excuse.
Moving the threshold post-test
A researcher specifies a Sharpe threshold of 0.8 before testing, runs the backtest, observes a Sharpe of 0.55, and then revises the threshold to 0.5 on the grounds that "the strategy is still interesting even if it doesn't meet the original standard." This is goal-post shifting. It is particularly insidious because the researcher may genuinely believe the threshold was too aggressive and the revision is reasonable.
The safest practice is to log the original threshold in a timestamped document before running the test, so any revision is visibly a departure from the protocol. If the strategy produces a Sharpe of 0.55, log the result as a partial fail with notes, and decide whether to pursue the variant with a different setup, not to retroactively declare success.
Choosing the metric post-test
Testing a strategy and then evaluating it on the metric that happens to look best is a form of multiple testing even when only one test was run. If a strategy has a Sharpe of 0.4 but a maximum drawdown recovery ratio that looks impressive, and the researcher reports the latter as the primary result because the Sharpe was disappointing, the reported result reflects selection bias in the metric, not just in the strategy.
Name the primary metric before running the test. Secondary metrics can be reported for completeness, but only the primary metric determines whether the hypothesis passed.
Scope mismatch between hypothesis and test
A hypothesis stated as "earnings momentum predicts forward returns in US equities" but tested only on the S&P 500 from 2010-2022 has a scope mismatch: the claim is broad, the test is narrow. If the result passes, the researcher's confidence in the broad claim is not warranted by the narrow test. If it fails, it also does not disprove the broad claim.
Alignment between the claimed scope and the tested scope matters most when the tested period happens to be favorable to the strategy type. Testing a long-only momentum strategy on 2010-2021, a decade-long bull market, in order to confirm a hypothesis about "momentum in US equities" conflates the strategy's behavior in a favorable regime with its behavior across all regimes.
Conflating the idea test with the implementation test
A signal can be genuinely predictive while a strategy built around it is unviable due to execution costs, capacity constraints, or portfolio construction inefficiencies. These are two separate questions. The hypothesis should specify which question is being tested. If the hypothesis is about the raw predictive power of the signal, cost assumptions can be minimal. If the hypothesis is about whether the strategy is deployable, realistic cost assumptions are required and the result is conditional on those assumptions.
Conflating the two, claiming a strategy works based on a zero-cost test, or claiming an idea is worthless based on a high-cost test that would have failed any strategy, produces misleading conclusions in both directions.
Say What Would Make You Drop It
The test of a hypothesis is whether you can say, before running anything, what result would make you abandon it. If no result would, the statement is not a hypothesis but a description with a favourable interpretation attached, and no amount of testing will make it informative.
Writing to that standard forces specificity that vaguer versions avoid. Which instruments, over which period, measured how, and compared against what? Each of those left open is a place where an eventual result can be reinterpreted to fit, and reinterpretation after the fact is indistinguishable from confirmation.
The failure that follows a well-formed hypothesis is genuinely useful, which is the part people find hardest to accept. A clear refutation closes a line of enquiry and releases the time it was consuming, whereas an ambiguous outcome invites months of further variations.
Falsifiability is a property of the statement rather than of the idea behind it. A precisely stated hypothesis can be refuted while the underlying intuition remains worth pursuing in a different form.
Frequently Asked Questions
What makes a trading hypothesis falsifiable?
A falsifiable hypothesis specifies in advance the exact observable outcome that would cause you to reject it. It names the instrument universe, the time period or look-back window, the signal, the entry and exit rules, and a primary metric with a threshold. If the backtest returns a value below the threshold, the hypothesis is refuted. If a hypothesis cannot be refuted by any possible result, for example "momentum tends to work in trending markets". It is not falsifiable and provides no information when the backtest is run.
What is the null hypothesis in strategy research?
The null hypothesis in strategy research is the claim you are trying to disprove: usually that the strategy produces no excess return over a relevant benchmark, or that the signal has no predictive relationship with future returns. The burden is on the evidence to reject this null. Starting from the assumption that the strategy works, and looking only for confirming evidence, inverts the scientific logic and produces systematically overconfident conclusions.
How specific should the hypothesis be?
Specific enough that any two researchers independently applying it to the same data would make the same trading decisions on every day in the history. That means naming the exact signal definition (including any moving average windows, percentile thresholds, or ratio formulas), the universe of stocks or assets eligible for the strategy, the rebalancing frequency, and the exit rules. If there is any ambiguity in how to apply the rules, the hypothesis is not specific enough.
Can I refine a hypothesis after the backtest?
Yes, but only as the start of a new, separately logged hypothesis, not as a revision to the original one. Revising a hypothesis after seeing the results and then presenting the revised version as though it was the original is HARKing. The correct procedure is to log the original hypothesis as refuted or inconclusive, write a new hypothesis that incorporates what you learned, and run a fresh test on held-out data if possible.
What is the difference between a strategy idea and a testable hypothesis?
A strategy idea is an informal belief, "stocks with high relative volume tend to follow through in the direction of the morning move." A testable hypothesis converts this into a precise, quantified claim: "S&P 500 component stocks with same-day relative volume above 2.0 at 10:30 AM ET, in which the first 60 minutes of price action show a net gain of at least 0.5%, will produce a mean next-60-minute return of at least +0.3% after transaction costs, over the period 2018-2023." The first cannot be disproved; the second can.
How do I handle a hypothesis that partially passes?
A partial pass, where some conditions are met but not others, or where performance is positive but below your stated threshold, should be recorded as a refutation of the original hypothesis, accompanied by a note on which aspect failed. Do not lower the threshold post hoc to make the result pass. The information that the strategy produces a Sharpe of 0.4 when you required 0.6 is useful and honest; a threshold quietly moved to 0.4 after the fact is not.
Should I specify one primary metric or several?
Specify exactly one primary metric on which the hypothesis stands or falls, plus no more than two pre-specified secondary metrics you will report for context. Evaluating a strategy on six metrics and reporting success based on whichever three were best is a form of multiple testing. The primary metric should be chosen before the backtest based on what matters most for the strategy type, total return for a long-term position strategy, Sharpe ratio for a risk-adjusted comparison, or win rate and average R:R for a short-duration trade-by-trade approach.
What should a complete hypothesis statement contain?
A complete hypothesis statement should contain: (1) the instrument universe and any eligibility filters, (2) the time period and data frequency, (3) the signal or factor being tested, including its exact computation, (4) the entry trigger, position sizing method, and exit rules, (5) the cost assumptions, commissions, slippage, and borrow cost for short positions, (6) the primary evaluation metric and the threshold that constitutes success, and (7) the null hypothesis that will be rejected if the primary metric exceeds the threshold.
Can a hypothesis be falsifiable and still be untestable in practice?
Yes, and the two are worth separating. A claim can specify exactly what would refute it while requiring data that does not exist, a history longer than the instrument has traded, or an execution assumption no account could meet. The specification is sound; the test is not available. Recognizing this early prevents a substituted proxy from being tested and reported as though it settled the original claim.
References
- Popper, K. (1959). The Logic of Scientific Discovery. Basic Books. The foundational text on falsifiability as the criterion of scientific demarcation. Chapter 4 covers the role of testing and refutation.
- Bailey, D.H., Borwein, J., Lopez de Prado, M., & Zhu, Q.J. (2014). "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance." Notices of the AMS, 61(5). Available at ams.org.
- Harvey, C.R., Liu, Y., & Zhu, H. (2016). "… and the Cross-Section of Expected Returns." Review of Financial Studies, 29(1). Discusses the multiple-testing problem in factor discovery and proposes minimum t-statistic thresholds. Available at academic.oup.com.
- Kerr, N.L. (1998). "HARKing: Hypothesizing After the Results are Known." Personality and Social Psychology Review, 2(3), 196-217. The paper that coined the HARKing term and defined its variants.
- Lopez de Prado, M. (2018). Advances in Financial Machine Learning. Wiley. Chapter 11 covers the backtest methodology and hypothesis testing in quantitative finance.
Educational Disclaimer
This guide is for educational purposes only and does not constitute investment, financial, or trading advice. All examples are illustrative and do not represent actual trading results. Past performance of any strategy described here does not predict future results. Strategy research involves significant risk of loss. Consult a qualified financial professional before making investment decisions.