Research Protocols
Research Protocols & Experiment Design
Scientific discipline for quantitative trading research.
Most strategy failures originate before the first backtest runs, in vague hypotheses, undocumented decisions, and data that silently incorporates future information. This curriculum covers the scientific infrastructure that separates systematic discovery from curve-fitting: from writing a falsifiable hypothesis and maintaining a preregistration-style research log, through controlling multiple-testing inflation, selecting appropriate benchmarks, and producing results that a colleague can independently replicate.
What this hub covers
Quantitative trading research fails most often not because the underlying idea was wrong, but because the research process was not designed to tell the difference between a genuine edge and a historical accident. A strategy chosen by testing a hundred parameter combinations and reporting the best one has not been validated, it has been fitted. A backtest built on data that incorporates subsequent earnings restatements or index changes has not been tested against what a trader could have known, it has been contaminated.
This hub covers the methodological infrastructure that makes trading research trustworthy: writing hypotheses that are falsifiable before the data is examined, keeping research logs that distinguish pre-specified analysis from post-hoc explanation, sourcing point-in-time data that reflects what was knowable on the trade date, selecting benchmarks and metrics that are appropriate for the strategy's risk exposures, applying multiple-testing corrections when comparing many variants, and documenting code and data in a way that allows independent replication. Each guide addresses one dimension of this infrastructure. The two interactive tools help you build a research protocol before a new backtest and maintain an experiment registry across iterations.
Key principles
- Falsifiability precedes testing: A hypothesis that cannot be refuted by any possible backtest result is not a scientific hypothesis. It is a narrative. Write down, before running the test, what result would cause you to reject the idea.
- Preregistration discipline prevents HARKing: Hypothesizing After Results are Known (HARKing) is the practice of choosing a hypothesis after seeing the data support it. A timestamped research log with pre-specified metrics is the practical substitute for formal preregistration in individual research.
- Point-in-time data is non-negotiable: Survivorship bias, look-ahead from earnings restatements, and index-reconstitution effects are invisible in backtests until the strategy is live and underperforming. Verify data provenance and point-in-time status before trusting any result.
- Benchmarks must match risk exposures: Apparent alpha that disappears when the benchmark is changed to one with the same factor loadings is not alpha. It is factor exposure. Compare strategies against benchmarks that represent the investable alternative, not cash or an arbitrarily chosen index.
- Multiple testing inflates false discovery rates: Selecting the best result from fifty variants at a 5% significance threshold is expected to produce false positives even when no real edge exists. Apply Bonferroni, BHY, or bootstrap-based corrections and count every meaningful variant tested, not just the ones reported.
- Robustness over optimality: A strategy whose performance degrades sharply for parameter values slightly different from the optimum is almost certainly overfit. A strategy that performs similarly across a neighborhood of parameters has a higher chance of reflecting a durable mechanism.
- Reproducibility enables accountability: If the original researcher cannot replicate their own result six months later, because the code changed, the data source updated, or the random seed was not recorded, the result cannot be trusted. Version-controlled code and pinned data snapshots are the minimum standard.
- Promote, revise, or reject, in advance: The decision to take a strategy live, revise its parameters, or archive it should be governed by criteria written before the results arrive, not justified by the results after the fact.
Curriculum: Research Protocols & Experiment Design
Eight guides and two interactive tools covering the full lifecycle of a disciplined trading research process, from initial hypothesis to final promotion or rejection decision. Each guide is self-contained and can be read in any order, though the sequence below follows the natural flow of a research project.
Guides
-
Write a Falsifiable Trading Hypothesis
How to state a strategy hypothesis precisely enough that a single backtest result can confirm or refute it, and why vague hypotheses produce uninterpretable results.
Guide
-
Research Logs and Preregistration-Style Discipline
Keeping a structured research log that separates pre-specified analysis from post-hoc explanation, the key discipline that distinguishes discovery from HARKing.
Guide
-
Data Lineage and Point-in-Time Research
Why survivorship bias, restatements, and look-ahead data corruption invalidate results, and how to source and verify point-in-time data for backtesting.
Guide
-
Choosing Benchmarks and Evaluation Metrics
Selecting appropriate benchmarks for strategy evaluation: market-cap indexes, factor benchmarks, cash hurdles, and metrics beyond Sharpe ratio.
Guide
-
Multiple Testing and Researcher Degrees of Freedom
Why testing many strategy variants inflates false-discovery rates, Bonferroni and BHY corrections, and the "researcher degrees of freedom" problem in strategy research.
Guide
-
Sensitivity Analysis and Robustness Checks
Testing whether a strategy's performance is concentrated in a narrow parameter range or robust across a neighborhood, a key signal of overfitting vs genuine edge.
Guide
-
Reproducible Trading Research
Version-controlled code, pinned data sources, seed management, and documentation standards that allow independent replication of a backtest result.
Guide
-
Decision Rules for Promote, Revise, or Reject
How to define in advance the criteria that would cause you to promote a strategy to live, revise its parameters, or reject and archive it, preventing post-hoc justification.
Guide
Interactive Tools
-
Research Protocol Generator
Answer guided prompts to generate a structured research protocol template for a new trading strategy hypothesis, ready to fill in before the backtest runs.
Tool
-
Experiment Registry Template
Interactive experiment registry: log hypothesis, method, data sources, pre-specified metrics, and outcome for each research iteration to maintain scientific discipline.
Tool
Process Is What Makes a Result Believable Later
The reason to formalise research process is that memory is unreliable and incentives are not neutral. Six months after a study, nobody remembers how many variations were tried, which decisions were taken before seeing results and which after, or what the original hypothesis actually said. A record made at the time answers those questions. A recollection does not.
The benefit arrives when a result has to be defended, including to yourself. A strategy with a documented hypothesis, a fixed evaluation rule and a log of every trial can be assessed on its merits. One without those artefacts is indistinguishable from a search that stopped when the numbers looked good.
The cost is real and worth naming. Documentation slows exploration, and the discipline is most irritating exactly when a promising idea appears. That tension is why these protocols have to be agreed in advance rather than applied when convenient.
Process does not validate a finding. A perfectly documented study of a weak idea remains a study of a weak idea, and the record makes that easier to see rather than harder.
Frequently Asked Questions
What is a research protocol in quantitative trading?
A research protocol is a written document that specifies, before a backtest runs, exactly what hypothesis is being tested, which data will be used, what the primary evaluation metric is, and what result would count as confirmation or refutation. It prevents the researcher from unconsciously shifting the goalposts after seeing the results, a practice called HARKing (Hypothesizing After Results are Known).
Why does multiple testing inflate false discovery rates in strategy research?
When you test many strategy variants and report only the one that performed best, you are selecting the winner from a large number of trials, most of which may have succeeded by chance. A 5% false-positive rate per test means that testing 20 independent variants is expected to produce at least one "significant" result by chance alone, even when no genuine edge exists. Corrections like Bonferroni or BHY adjust the significance threshold to account for the number of comparisons made.
What is point-in-time data and why does it matter for backtesting?
Point-in-time data reflects exactly what a trader would have known on any given historical date, before earnings restatements, index reconstitutions, or corporate actions were announced. Using data that incorporates subsequent revisions causes look-ahead bias: the backtest implicitly uses information that was not available at the time of the modeled trade, making historical performance appear better than it could have been in practice.
How do you choose an appropriate benchmark for strategy evaluation?
An appropriate benchmark should reflect the investable alternative a real investor would have used instead of the strategy. A long-only stock strategy should be compared to a broad market-cap index like the S&P 500, not cash. A strategy that tilts toward small-cap value stocks should be compared to a small-cap value factor benchmark, otherwise apparent alpha may simply reflect factor exposure. Strategies that run at different times of year should use appropriate seasonal variants or annualized return adjustments.
What makes a trading research result reproducible?
A reproducible result can be independently replicated by someone else given the same starting conditions. This requires version-controlled code with a pinned environment, identified and documented data sources (including the specific version or snapshot used), a fixed random seed for any stochastic processes, and a research log that documents every preprocessing and filtering decision. Without these, even the original researcher may be unable to reproduce their own result six months later.
What is HARKing and why is it a problem in strategy research?
HARKing stands for Hypothesizing After Results are Known. It describes the practice of running a backtest, observing the results, and then writing up the analysis as though the observed pattern was the original hypothesis. It is a problem because it makes noise-fitting look like genuine discovery. The strategy appears to have been validated against a pre-specified hypothesis when in reality the hypothesis was chosen because the data happened to confirm it.
Does a small research team need the same protocol discipline as a large one?
The failure modes these protocols guard against are cognitive rather than organizational, so a single researcher is exposed to all of them. What changes with team size is who the record is for. A large team needs the protocol to transfer knowledge between people; a solo researcher needs it to transfer knowledge to their own future self, who will not remember which variants were tried or why one was abandoned. The discipline is the same, the audience differs.
In what order should the practices in this cluster be adopted?
Recording what was tried comes first, because every other practice depends on knowing the search history. Writing the hypothesis and success criteria before running anything comes next, since it costs little and prevents the most common distortion. Data lineage follows, because it determines whether a result can be rebuilt at all. Formal corrections for multiple testing and full reproducibility tooling are worth adding once the earlier habits are established rather than as a starting point.
What is the cheapest version of this process that still catches real problems?
A dated text file per project, containing the hypothesis and the metric threshold written before the first run, a running list of every variant tested including failures, and a note of the data source and extraction date. That alone exposes search breadth, prevents the hypothesis from being rewritten after the fact, and makes a result rebuildable. Version control, containers and formal corrections add rigour, but the text file catches the majority of what goes wrong.