Research Logs and Preregistration-Style Discipline
Direct Answer
A research log is a structured, timestamped record that documents every meaningful decision in a strategy research project before results are visible. Its defining property is the separation between the pre-test section, written before any data is examined, and the post-test section, written after the backtest runs. This separation is what makes it possible to distinguish genuine discovery from HARKing (Hypothesizing After Results are Known), the near-universal practice of explaining results as though they were predicted in advance.
Formal preregistration, used in academic clinical and psychology research, achieves this by submitting a protocol to a public registry with a tamper-evident timestamp before the study begins. Individual traders cannot use this infrastructure, but they can replicate its core function with version-controlled text files committed before any analysis runs. The discipline is not about bureaucracy. It is about building a private record that is honest enough to trust, which is a precondition for making good decisions about which strategies to take live.
Key Takeaways
- Timestamps are the core mechanism: A log that can be edited retroactively is no better than memory. File-system timestamps are easily changed; git commit timestamps are cryptographically bound to the content and history.
- Pre-test and post-test sections are structurally separate: The pre-test section describes what will be done and what would count as success. The post-test section describes what happened. Mixing them is HARKing.
- Every meaningful variant gets its own entry: If you test ten signal variations, ten entries prevent the log from understating the search breadth, which matters for multiple-testing awareness.
- Post-hoc explanations are labeled as such: Noticing after the fact that the strategy performed badly in 2022 but well in every other year is valid observation, but it must be labeled as a post-hoc observation, not a pre-specified regime filter.
- Failed tests are logged, not deleted: A failed test is not embarrassing. It is evidence that narrowed the search space. Deleting failed entries overstates the hit rate of the research process.
- The log is the denominator for multiple-testing corrections: If you want to apply a Bonferroni or BHY correction for multiple comparisons, you need to know how many hypotheses were tested. The log provides this count.
- Minimal overhead, maximum honesty: A research log does not need to be elaborate. A plain Markdown or text file with dated entries provides the essential function without adding significant workflow friction.
- Discipline is self-protective, not self-limiting: A researcher who can point to a log showing that a strategy passed a pre-specified test has stronger grounds for confidence in the result than one who cannot. The discipline protects against overconfidence, not against finding real edges.
Core Concepts
The function of preregistration
Preregistration in science serves one purpose: to create a public, verifiable record that separates what was predicted from what was found. When a clinical trial preregisters its primary endpoint before enrollment begins, reviewers can later verify that the reported outcome matches the pre-specified one, and that the researcher did not switch from a primary endpoint that failed to a secondary one that happened to succeed. The practice emerged because the academic literature was found to be systematically biased toward positive results, in large part because researchers (not always consciously) selected the analysis that made their hypothesis look supported.
Strategy research faces an identical problem. A researcher who runs a backtest, modifies the entry rules when the result is disappointing, re-runs it with different parameters, and reports the final iteration as the validated strategy has engaged in a form of p-hacking, repeated testing until a result passes, even if each individual test was technically correct. The only structural defense against this is a time-stamped record of what the hypothesis was before any results were seen.
Individual traders cannot use the Open Science Framework or the AEA's registry of randomized trials. But they can write a protocol entry in a git-tracked file and commit it before running any code. The commit hash and timestamp in the git log provide a tamper-evident record that the content was written before the results were seen, the same functional property that makes formal preregistration useful.
The practical question is not whether this is necessary for every small exploratory analysis. It is whether, at the point of deciding to take a strategy seriously as a potential live trading candidate, there is an honest record showing that the pass was genuine and not an artifact of repeated testing. Without that record, the confidence in the result cannot be calibrated correctly.
Structure of an effective research log entry
An effective log entry has two structurally distinct blocks. The pre-test block contains: the date and time the entry was written, the research question in plain language, the specific hypothesis with all parameters as defined in the falsifiable hypothesis guide, any sub-group or regime analyses planned in advance, the primary evaluation metric and success threshold, the null hypothesis, cost assumptions, and the data source and date range. This block is committed to version control before the backtest is run.
The post-test block is a separate commit made after the results are available. It contains: the observed primary metric value, whether the hypothesis passed or failed, secondary metric values reported for context, a one-paragraph interpretation of the result, any post-hoc observations (clearly labeled as such), and the status of the hypothesis (pass, fail, inconclusive, or abandoned). If the result prompts a follow-up hypothesis, the follow-up is a new pre-test entry in a new commit, not a revision of the original.
The separation between commits is the structural guarantee. A reviewer, including the future self who is deciding whether to allocate capital to this strategy, can look at the git log and see that the pre-test block was written before the post-test block. This is a stronger claim to scientific integrity than any amount of assertion that the hypothesis was pre-specified.
The level of detail required scales with how much is riding on the result. For an early exploratory analysis, a brief entry is sufficient. For a strategy being evaluated for live deployment, the pre-test block should be detailed enough that a competent colleague could independently implement and run the same test without additional information.
Distinguishing discovery from HARKing
HARKing is pervasive in quantitative research because it does not feel like dishonesty. A researcher runs a test, observes that the strategy performed well in low-volatility periods and poorly in high-volatility ones, writes a note that "the strategy is designed for low-volatility regimes," and presents the low-volatility subset as the validated use case. The researcher genuinely believes this interpretation, but they arrived at it by looking at the data, not by predicting it. The subset that confirmed the strategy was chosen from the data that generated the result, which inflates the apparent evidence.
The test for HARKing is whether the reported analysis would have been done, with exactly those parameters, exactly that subset, exactly that metric, had the data shown a different pattern. If the answer is no, the analysis is post-hoc. This is not automatically invalid, post-hoc observation is a legitimate input to the next hypothesis, but it must be labeled as such and confirmed on fresh data before being claimed as evidence.
A research log enforces this distinction structurally. If the low-volatility regime filter was not in the pre-test block, the post-test observation that the strategy works in low volatility is labeled as an exploratory finding, not a pre-specified result. That label has consequences: the researcher knows they need to test the low-volatility version as a new hypothesis, on held-out data, before trusting it.
The asymmetric standard is also worth noting: pre-specified analyses that fail are evidence against the hypothesis. Post-hoc analyses that succeed are hypothesis-generating, not hypothesis-confirming. Treating them equivalently collapses the distinction between discovery and validation, which is the root cause of most strategy research failures.
Practical implementation with version control
The simplest implementation uses git to track a Markdown file. The workflow: before running any analysis, open the research log, write a new dated entry in the pre-test format, and commit the file with a message like "pre-test: [strategy name] [date]". After the backtest runs, write the post-test block and commit with a message like "post-test: [strategy name] [date] [pass/fail]". The git log now contains a permanent, auditable record of the sequence.
One common objection is that the researcher could simply backdate a pre-test entry after seeing results. This is true, but the effort required to commit an honest entry before the analysis is minimal, and anyone maintaining a research log primarily to deceive themselves is solving a different problem than the one addressed here. The purpose is to protect against the common case of unconscious post-hoc rationalization, not against determined fraud.
For teams, the git history provides mutual accountability, each researcher's log entries are visible to collaborators, and the commit timestamps are set by the git server rather than the local machine. Some research teams also use automated CI hooks that run a checksum on the pre-test entries and flag any retroactive modifications.
An alternative to git for individual researchers is a research notebook with pages dated in ink before use, or a cloud-based document service that stores edit history (Google Docs maintains a complete version history that cannot be permanently deleted). The key property is that the record of the pre-test state must be provably prior to the results, not asserted to be, but structurally proven.
Worked Scenario
A researcher is testing whether stocks with accelerating earnings growth (year-over-year EPS growth rate increasing for three consecutive quarters) outperform the S&P 500 over the next quarter.
- Pre-test commit (2026-07-01): Writes the hypothesis entry: universe = S&P 500 components with EPS history of at least 8 quarters, signal = three consecutive quarters of accelerating YoY EPS growth rate, entry = next-day open after the third confirming earnings report (with a 2-day lag), exit = 60 calendar days, position sizing = equal weight, cost = 15bps one-way, primary metric = mean 60-day excess return vs S&P 500 with threshold = +1.5%, significance threshold = t-stat > 1.96. Commits file with message "pre-test: EPS acceleration momentum 2026-07-01".
- Backtest run (2026-07-03): Runs the backtest over 2010-2025. Result: mean 60-day excess return = +0.4%, t-stat = 0.9. Fails the threshold on both the magnitude and statistical criteria.
- Post-test commit (2026-07-03): Writes the post-test block: "Hypothesis failed. Mean excess return 0.4% (threshold 1.5%), t-stat 0.9 (threshold 1.96). Post-hoc observation: performance appears stronger in the 2010-2018 sub-period (+1.8% mean, t-stat 2.1) but weak in 2019-2025 (-0.3%). This sub-period observation is exploratory and requires a new pre-test to evaluate." Commits.
- Follow-up pre-test (2026-07-04): Writes a new entry for the sub-period hypothesis: "Does EPS acceleration momentum outperform in periods of positive economic growth (ISM Manufacturing above 50)? Signal computation: same as original, but entries only when the trailing 3-month average ISM is above 50." Commits before running any data.
The log now honestly reflects three facts: the original hypothesis failed, a sub-period pattern was observed post-hoc, and a new pre-specified test is being designed for the conditional version. A reader can see the full chain without any single entry overstating what was demonstrated.
Measurement Framework
| Log quality metric | Question it answers |
|---|---|
| Pre-test commit timestamp vs backtest run date | Was the hypothesis genuinely specified before results were seen? |
| Number of pre-test entries vs post-test entries | Are tests being logged before they run, or only after? |
| Fraction of log entries with status "failed" or "abandoned" | Is the log capturing the full search space, including negative results? |
| Number of distinct pre-test entries per promoted strategy | How many hypotheses were considered before a passing one was found? (Relevant to multiple-testing correction) |
| Presence of "post-hoc observation" labels in post-test blocks | Is the researcher consistently distinguishing pre-specified from exploratory findings? |
| Follow-up rate on post-hoc observations | Are exploratory observations being pursued with new pre-specified tests, or quietly adopted as validated results? |
Common Failure Modes
Writing the "pre-test" entry after seeing the results
The most common failure is writing what appears to be a pre-test entry after running the backtest, either because the researcher planned to log it in advance and forgot, or because the results were compelling and retroactive documentation seemed harmless. This completely defeats the purpose of the log. The record now says a hypothesis was pre-specified when it was not, and any future evaluation of the strategy's evidence quality is wrong.
The fix is structural, not motivational: make the pre-test commit a prerequisite for running the backtest. In a disciplined setup, the code that runs the backtest checks that a pre-test log entry exists for the current experiment identifier and exits with an error if one is missing. This is not difficult to implement and removes the option of running first and logging later.
Not logging failed tests
A researcher tests 15 variations of a momentum signal and gets one that passes. Only the passing variation is logged. The research record now shows one test that passed, which appears to be strong evidence, rather than one test out of fifteen that passed, which is exactly what would be expected by chance at a 5% significance threshold even when there is no real effect.
This is the multiple-testing problem at the logging level. If failed tests are not recorded, there is no way to apply the appropriate correction when evaluating the survivor. Even when not applying a formal correction, having the count of all tests attempted provides the information needed to intuitively calibrate how much to trust the passing result.
Merging post-hoc observations into the pre-test record
After observing that a strategy performed well in a specific sub-period, a researcher adds a note to the pre-test entry: "This strategy is expected to work particularly well in low-volatility environments." Now the log appears to show that a regime filter was pre-specified, when it was actually adopted post-hoc. If the researcher then tests only the low-volatility subset and reports it as a pre-specified test, the result is doubly biased.
The pre-test entry must be read-only once the post-test commit is made. If the post-test block generates new insights, they belong in the post-test block as labeled observations, not appended to the pre-test block as though they were always there.
Treating the log as a formality rather than a tool
A researcher maintains a research log because it seems like good practice, but does not actually use it when making decisions about which strategies to develop further. The log records that a hypothesis failed, but the researcher revises and retests without creating a new pre-test entry, treating the revision as a routine parameter adjustment rather than a new hypothesis. Over time, the log drifts from the actual research process and provides no accountability.
The log has value only if it is genuinely consulted at decision points: before revising a failed strategy, before deciding to take a strategy live, before reporting a result to a collaborator or allocator. If the decision-making ignores the log, the log is overhead with no benefit. The discipline requires that the log be the authoritative record, not a post-hoc documentation exercise.
Insufficient granularity in pre-test entries
A pre-test entry that says "test momentum strategy with 12-month lookback on S&P 500" is not specific enough to prevent HARKing. If the result disappoints, the researcher can later add "except for the most recent month" (12-1 month momentum, not 12-month momentum) and claim the revised signal was pre-specified. Complete specificity, including the exact formula for every computed value, is required to prevent this.
An entry that specifies "12-month minus 1-month price momentum, defined as (P_t-22 / P_t-252) - 1 where t is the rebalancing date, applied to S&P 500 constituents as of the rebalancing date (excluding newly added members with less than 13 months of history)" cannot be silently revised without the revision being obvious. The specificity is the protection.
The Line Between Before and After
The line that gives a research log its value is the one between what was written before the data was examined and what was written after. Both halves are legitimate and only the first is evidence. Explanations produced after seeing a result are how understanding develops, and they carry no weight as confirmation of the thing they explain.
Keeping the two visibly separate is a mechanical fix for a psychological problem. Nobody recalls accurately which conclusions preceded which observations, and a document that timestamps the pre-test section removes any need to rely on memory.
The temptation the log exists to resist is quiet revision. Adjusting a stated hypothesis so that it matches whatever turned up converts a failed test into a successful one without anything having been learned, and it happens gradually rather than in a single dishonest moment.
A log records intent and does not confer rigour. A carefully separated record of a poorly designed study documents that design faithfully.
Frequently Asked Questions
What is a research log in quantitative trading?
A research log is a structured, timestamped record of every meaningful decision made before and during a research project: the hypothesis as originally stated, the data sources chosen, the signal computation, the evaluation metric and threshold, and then, after the backtest runs, the result and a conclusion. The key property is the timestamp: the pre-test section must be written and dated before any data is examined, so there is an objective record distinguishing what was specified in advance from what was decided in response to results.
What is HARKing and why is it harmful?
HARKing stands for Hypothesizing After Results are Known. It is the practice of observing a pattern in data, constructing a hypothesis that the pattern would support, and then presenting the analysis as though the hypothesis was specified before the data was examined. It is harmful because it converts noise-fitting into apparent discovery. Every random dataset contains many patterns; HARKing produces a post-hoc narrative for whichever one was found, which looks indistinguishable from genuine prediction.
How is a research log different from formal preregistration?
Formal preregistration, used in academic research, involves submitting a study protocol to a public registry before data collection begins, creating a public, time-stamped record that cannot be altered retroactively. A research log is a private equivalent: a document you maintain yourself, timestamped by your file system or a version-control commit. It lacks the third-party verification of formal preregistration but provides the same functional benefit, a record you can consult to distinguish what you planned from what you found.
What should the pre-test section of a log contain?
The pre-test section should contain: the date the entry was written, the research question or hypothesis, the exact signal computation, the universe and time period, the evaluation metric and success threshold, the null hypothesis, cost assumptions, any planned sub-group or regime analyses, and any secondary metrics that will be reported. Everything in this section must be written before the backtest is run. If a decision was made after looking at results, it belongs in the post-test section, clearly labeled as such.
Is it acceptable to update the hypothesis after a failed test?
Yes, hypothesis revision is normal science. What is not acceptable is presenting the revised hypothesis as though it was the original. The correct process is to log the original as failed, note what the result suggested, write a new hypothesis entry with a new timestamp, and run the revised test, ideally on held-out data that was not used to inspire the revision. This creates a traceable chain of iterations rather than a revisionist account.
How granular should the log entries be?
Each meaningful variant deserves its own entry. A meaningful variant is one where the signal, universe, time period, metric, or threshold differs enough that a different result would have been possible. Changing a moving average from 10 to 12 days in a systematic parameter sweep is probably one entry describing the sweep range. Changing from a mean-reversion to a momentum approach is a separate entry. The test is whether someone reading the log later could reconstruct what was tried and in what order.
Can a research log be maintained in a simple text file?
Yes. The format matters less than the discipline. A plain Markdown file committed to a version-controlled repository is arguably better than a proprietary tool because the commit history provides tamper-evident timestamps. A notebook, spreadsheet, or physical paper log also works if it is written before the test runs and never edited retroactively. The version-controlled text file has the additional advantage of being searchable, diffable, and co-located with the code it describes.
What happens to log entries for tests that were never fully run?
Log them anyway, with a status of "abandoned" and a note on why. An abandoned entry documents a search path that was considered and rejected before testing, which is relevant context for understanding the total number of ideas examined (the denominator in a multiple-testing adjustment). It also prevents accidentally re-investigating the same idea later without realizing it was already considered.
How can a pre-test section be shown to have been written before the results?
Self-attestation is weak evidence, since a file can be edited freely. Stronger options include committing the pre-test section to version control and referencing the commit hash in the final write-up, sending it to a colleague by dated message, or storing it where edits are logged. None of these are required for the log to be useful to you personally, but they matter as soon as the result is presented to anyone who has reason to be sceptical.
References
- Kerr, N.L. (1998). "HARKing: Hypothesizing After the Results are Known." Personality and Social Psychology Review, 2(3), 196-217. The foundational paper defining and categorizing HARKing. Available via many institutional libraries.
- Open Science Collaboration (2015). "Estimating the reproducibility of psychological science." Science, 349(6251). The large-scale replication study that brought reproducibility problems to broad scientific attention. Available at science.org.
- Bailey, D.H. & Lopez de Prado, M. (2012). "The Sharpe Ratio Efficient Frontier." Journal of Risk, 15(2). Includes analysis of backtest selection bias and the effect of repeated testing. Available at ssrn.com.
- Harvey, C.R. & Liu, Y. (2015). "Backtesting." Journal of Portfolio Management, 42(1), 13-28. Discusses multiple-testing and the need for discipline in the backtest process. Available at jpm.pm-research.com.
- Center for Open Science. "Preregistration." cos.io/initiatives/prereg. The academic preregistration infrastructure and rationale.
Educational Disclaimer
This guide is for educational purposes only and does not constitute investment, financial, or trading advice. All examples are illustrative and do not represent actual trading results. Past performance of any strategy described here does not predict future results. Trading involves significant risk of loss. Consult a qualified financial professional before making investment decisions.