Scenario Count, Seeds, and Reproducibility
Direct Answer
Three operational standards determine whether a Monte Carlo simulation produces credible, auditable results: the number of simulation paths (scenario count) must be large enough to estimate the relevant percentiles with acceptable precision; the random seed must be fixed and recorded before the simulation runs; and the full simulation specification must be documented so that any analyst with the same code and data can reproduce the exact results.
Without a fixed seed, two runs of the same simulation code produce different results, making every reported result non-reproducible and impossible to verify. Without enough paths, tail percentile estimates have high sampling variance, the 5th percentile from one run could differ by 5-10 percentage points from the same estimate run again. These are not theoretical concerns; they are practical requirements for defensible quantitative analysis that frequently go unfulfilled in practice.
Key Takeaways
- Scenario count requirement scales with the rarity of the percentile: 1,000 paths for the 10th percentile; 5,000 for the 5th percentile; 50,000+ for the 1st percentile or risk-of-ruin below 2%.
- Fix the seed before any simulation runs, not after: Setting the seed after examining results and selecting a favorable one is selective reporting. Adopt a convention (e.g., seed = 42) and apply it uniformly.
- Record six elements for full reproducibility: Seed, path count, simulation method, input data specification, software library and version, and run timestamp.
- Check convergence before finalizing results: Run the simulation with 1x, 2x, and 4x path counts. If key percentile estimates stabilize across the doublings, the path count is sufficient.
- PRNGs differ across languages and library versions: A seed of 42 in Python NumPy does not replicate results in R, MATLAB, or earlier NumPy versions. The PRNG algorithm is part of the reproducibility specification.
- Path count is not the bottleneck in most cases: Modern computing can run 10,000 bootstrap paths in seconds for typical strategy analysis. There is rarely a computational reason to use fewer than 5,000 paths.
- Parallel simulation requires seed management: Parallelizing across CPU cores requires giving each worker a different seed (derived from the master seed) to avoid correlated random sequences across workers.
- Reporting without seeds is not auditable: A Monte Carlo result without a reported seed cannot be verified. Treat seed reporting as mandatory, not optional.
Core Concepts
How Many Paths Are Enough: A Precision-Based Framework
The number of simulation paths needed is determined by the precision required for the specific percentile estimate you care most about. The standard error of the p-th sample percentile from N paths is approximately: SE(q_p) ≈ sqrt(p × (1−p)) / (N × f(q_p)), where f(q_p) is the probability density of the distribution at the p-th percentile. For practical purposes, this simplifies to a useful rule of thumb: the precision of a percentile estimate improves proportionally to 1/√N, and estimates of rarer percentiles require proportionally more paths to achieve the same absolute precision.
For the 10th percentile of CAGR (p = 0.10), with N = 1,000 paths: the effective sample for this percentile is approximately p × N = 100 observations. The standard error of a 10th percentile estimate based on 100 effective observations is substantial, roughly 1-2% of the metric range. For N = 5,000: the effective sample is 500, and the standard error falls by more than half. For the 5th percentile (p = 0.05) with N = 1,000: the effective sample is only 50, which may produce a 5th percentile estimate with standard error of 3-5% of the metric range, too imprecise for risk management decisions. N = 5,000 gives an effective sample of 250, which is adequate.
For the 1st percentile or ruin probability estimates below 2%, path counts of 50,000-100,000 are typically required. The 1st percentile from 10,000 paths has an effective sample of 100, similar to the 10th percentile from 1,000 paths, with corresponding sampling variance. At 100,000 paths, the effective sample is 1,000, providing adequate precision. The computational cost of 100,000 bootstrap paths is typically 10-60 seconds on modern hardware for typical strategy analyses, well within practical limits.
A convergence check procedure: run the simulation at 500, 1,000, 2,000, 5,000, and 10,000 paths. For each, record the key percentile estimates (5th and 10th percentile CAGR, 5th percentile max drawdown). If the estimates are stable, varying by less than 1-2% absolute, between the 5,000 and 10,000 path runs, 5,000 is sufficient. If they still vary substantially between 5,000 and 10,000, double to 20,000 and check again. Stop when the increment from doubling produces less than 1% absolute change in the key estimates.
Random Seeds: Why They Matter and How to Manage Them
A pseudo-random number generator (PRNG) is a deterministic algorithm that produces a sequence of numbers with statistical properties resembling true randomness, initialized by a "seed" value. Given the same seed, the same PRNG produces the exact same sequence of numbers. This determinism is essential for reproducibility: if you fix the seed before a Monte Carlo simulation and record the seed, anyone with the same code can reproduce your exact simulation results.
Without a fixed seed, each run of the simulation uses a different pseudo-random sequence, typically initialized from the system clock or a hardware entropy source. Two runs of the same simulation code with unfixed seeds will produce different distributions, different percentile estimates, and potentially different conclusions. This makes the analysis unrepeatable: the reported "5th percentile max drawdown of −38.2%" cannot be verified because running the same code again will produce a different number.
The practical implementation in Python NumPy: call numpy.random.seed(42) (or use the newer Generator API: rng = numpy.random.default_rng(42)) before any random draws in the simulation. In R: set.seed(42). In MATLAB: rng(42). The exact integer value of the seed is arbitrary, what matters is that it is fixed before the simulation begins, recorded alongside the results, and not changed after seeing results.
Seed cherry-picking, running the simulation with several seeds and reporting the one that produces the most favorable results, is a subtle form of selective reporting that inflates apparent strategy robustness. It is analogous to running many backtests and reporting only the best one. The discipline is to pre-commit to a seed (e.g., always use seed = 42 as an organizational convention) and report results from that seed without modification. If you need to verify that the results are not seed-specific, run a sensitivity check across 5-10 seeds and report the average and range of key percentile estimates, not the best single seed.
The Reproducibility Specification
For a Monte Carlo analysis to be fully reproducible, six elements must be documented alongside the results. First, the random seed. Second, the number of simulation paths. Third, the simulation method: standard bootstrap, block bootstrap (with block length), trade-order reshuffling, or parametric simulation (with the distributional assumption and fitted parameters). Fourth, the input data specification: the exact date range, return frequency, total number of observations, and the version of the historical data source used.
Fifth, the software library and version. This is frequently overlooked but critically important: numpy.random.seed(42) in NumPy 1.16 uses the Mersenne Twister PRNG; numpy.random.default_rng(42) in NumPy 1.17+ uses the PCG64 algorithm by default. The same seed in the same language can produce different results across library versions. Document the specific version (e.g., Python 3.11.2, NumPy 1.24.3) as part of the reproducibility spec. Sixth, the run timestamp, useful for auditing and version tracking, even if not strictly necessary for mathematical reproducibility.
For team-based or institutional quantitative research, these six elements should be stored in structured metadata alongside every Monte Carlo analysis result. A simple approach: store the results in a JSON or CSV file, with a metadata block that includes all six elements. When the analysis is revisited, updated, or challenged, the metadata provides a complete audit trail for reproducing the exact computation.
Parallelization and Seed Management
Parallelizing Monte Carlo simulation across multiple CPU cores is straightforward in Python using multiprocessing or concurrent.futures, and provides near-linear speedup, 8 cores can run 8× as many paths in the same wall-clock time. However, naive parallelization without seed management produces correlated results across workers: if each worker initializes its PRNG from the same global seed, all workers produce the same random sequence and the parallel runs are not independent.
The correct approach: derive worker seeds from the master seed using a deterministic function. A common pattern in NumPy's new API: create a root SeedSequence from the master seed, then spawn child sequences for each worker using seedseq.spawn(n_workers). Each child sequence initializes an independent, non-overlapping PRNG stream. The master seed still fully determines the results (given the same n_workers), so reproducibility is maintained. Document the master seed and the number of workers as part of the reproducibility spec, results may differ if the number of workers changes, because the spawned seed sequences are worker-count-dependent.
Worked Scenario
- Analysis objective: Estimate the 5th percentile max drawdown for a trend-following strategy, with ±2% absolute precision at the 5th percentile.
- Required path count calculation: For p = 0.05 and desired standard error ≤ 2% of the max drawdown metric range: using the rule of thumb N ≥ 1,000/p = 20,000 gives an effective sample of 1,000 at the 5th percentile. At a typical max drawdown standard deviation across paths of ~10%, the standard error of the 5th percentile is approximately 10% / sqrt(1,000) ≈ 0.3% absolute, well within the ±2% target. 10,000 paths is sufficient (effective sample 500, SE ≈ 0.45%).
- Seed selection: Pre-commit to seed = 2024 as the organizational standard. Record before running any simulation code.
- Convergence check: Run simulation at 1,000, 2,000, 5,000, and 10,000 paths (seed = 2024 each time). 5th percentile max drawdown estimates: −41.8%, −40.3%, −38.9%, −38.7%. Change from 5,000 to 10,000 paths: 0.2% absolute. Convergence confirmed at 10,000 paths.
- Final simulation: 10,000 paths, block bootstrap (l=15), seed 2024. 5th percentile max drawdown: −38.7%.
- Reproducibility documentation: Seed: 2024. Paths: 10,000. Method: block bootstrap, block length 15 days. Data: SPY daily returns, 2019-01-02 to 2023-12-29 (1,258 observations). Software: Python 3.11.4, NumPy 1.25.2. Run: 2026-08-07 14:23 UTC.
- Verification: Any analyst replicating the above specification can reproduce the −38.7% estimate exactly.
Measurement Framework
| Measurement | Question to Answer |
|---|---|
| Standard error of 5th percentile estimate given N paths | Is the path count sufficient for the required precision at the target percentile? |
| Change in key percentile estimates from N to 2N paths | Has the simulation converged? (Change <1-2% absolute indicates convergence) |
| Variation of key estimates across 5 different seeds at fixed N | How seed-sensitive are the results? Large variation suggests N is too small. |
| Seed documented alongside results (yes/no) | Are the results reproducible and auditable? |
| Software version documented (yes/no) | Can the results be reproduced in a future session with a potentially updated library? |
| Input data specification documented (yes/no) | Is the input data precisely identified so the same data can be used in verification? |
| Simulation run time (seconds) at chosen path count | Is the chosen path count computationally feasible within the analysis workflow? |
Common Failure Modes
Not Fixing the Seed Before the Simulation Runs
The most common and consequential reproducibility failure is omitting seed specification entirely, relying on the system's default random initialization. Without a fixed seed, running the same simulation code twice will produce different results, different distributions, different percentile estimates, and potentially different deployment recommendations. Any reported Monte Carlo result without a documented seed is non-reproducible and should be treated as anecdotal rather than auditable analysis.
Using Too Few Paths for Tail Estimates
Running 500 or 1,000 paths and reporting the 5th percentile estimate is inadequate for risk management purposes. With 1,000 paths and p = 0.05, the effective sample at the 5th percentile is only 50 observations, too few for a stable estimate. Running the same analysis again with a different seed will produce a 5th percentile estimate that differs by 5-10 percentage points. The practical solution: for any percentile at or below the 5th, use 5,000+ paths; for the 1st percentile or ruin probabilities, use 50,000+.
Running Multiple Seeds and Reporting the Favorable One
Selecting the seed that produces the most favorable-looking Monte Carlo distribution, wider confidence intervals look more pessimistic, narrower look more optimistic, is a subtle form of selective reporting analogous to p-value hacking. The discipline is to pre-commit to a seed convention and report results from that seed only. Sensitivity across seeds is valuable information but should be reported as a range (e.g., "key estimates vary by ±2% across seeds 1-10"), not as a basis for selecting the most favorable individual result.
Changing the Path Count and Not Re-Reporting
If an analysis is re-run with a different number of paths, because the initial count was insufficient, the new results should be reported alongside the old ones with the path count change noted. Silently updating a report with higher-path-count results while keeping the same seed is acceptable (and the new results are more precise); doing so while also changing the seed produces a non-reproducible update. In institutional contexts, version-control the analysis code and data specifications alongside the reported results so that updates are traceable.
Reproducibility Is a Bookkeeping Problem
Reproducibility here is a bookkeeping problem rather than a statistical one, which is partly why it gets neglected. Recording the seed, the path count, the code version and the exact input data is clerical work, and it is the only thing that lets a result be checked later, including by the person who produced it.
The seed deserves particular discipline because it can be abused quietly. Running a simulation repeatedly and keeping whichever run looked best is a form of selection that leaves no trace unless the process is logged, and in the output it is indistinguishable from a single honest run.
Path count is the other half. Too few paths leave tail estimates unstable, and stability is testable: run the same configuration under different seeds and watch whether the numbers you care about move.
None of this makes a result correct. A reproducible simulation of a badly specified model reproduces the same wrong answer reliably, which is why the bookkeeping is a precondition for evaluating a study rather than a substitute for evaluating it.
Frequently Asked Questions
How many simulation paths do I need for a stable confidence interval?
For stable estimates of the 10th and 90th percentile of common metrics (Sharpe, CAGR), 1,000-2,000 paths is typically sufficient. For the 5th percentile, use 5,000 or more. For the 1st percentile or risk-of-ruin probabilities below 5%, use 10,000-100,000 paths. The standard error of a percentile estimate from N paths scales as 1/√(N×p), rarer percentiles need proportionally more paths for the same precision.
What is a random seed and why does it matter for reproducibility?
A random seed is an integer that initializes the pseudo-random number generator (PRNG) used in simulation. Pseudo-random number generators are deterministic, given the same seed, they produce the exact same sequence of 'random' numbers. By fixing and logging the seed before each simulation run, you guarantee that the same simulation can be exactly reproduced later. Without a fixed seed, two runs of the same simulation code will produce different results, making the analysis non-reproducible and non-auditable.
Can I run multiple seeds and average the results?
Running multiple seeds and averaging results is equivalent to simply running more paths, it does not provide any additional information beyond what a single run with all those paths combined would give. The standard practice is to run one simulation with enough paths using one fixed seed, and report that seed. Running multiple seeds and reporting the best result is seed cherry-picking, a form of selective reporting that inflates apparent performance.
How do I check if my simulation has converged?
Convergence check: run the simulation with increasing path counts (100, 500, 1,000, 5,000, 10,000) and track the evolution of the key percentile estimates. When the estimate stabilizes, changing by less than a threshold (say, 1% absolute) between successive doublings of path count, the simulation has converged for that metric. Alternatively, run with 10 different seeds at your chosen path count and compute the standard deviation of the 5th percentile across those seeds.
Does path count affect the quality of individual paths, or only the distribution estimate?
Path count affects only the quality of the distribution estimate (how well you can estimate percentiles), not the quality of individual paths. Each individual path is generated by a fixed simulation model, block bootstrap or trade reshuffling, and its quality is determined by the model and data, not by how many other paths you generate. Generating more paths gives you a better-populated distribution with lower sampling variance in your percentile estimates.
What should I include in a reproducibility report for a Monte Carlo analysis?
A complete reproducibility specification should include: the random seed used; the number of paths; the simulation method (standard bootstrap, block bootstrap with block length, trade-order reshuffling); the input data description (date range, return frequency, number of observations); the software and version used; and the timestamp of the run. With these six elements, any competent analyst can reproduce the exact results.
Do different programming languages produce the same results from the same seed?
No. Random number generators differ across programming languages and even across library versions within the same language. A seed of 42 in Python NumPy will not produce the same sequence as seed 42 in R, MATLAB, or Julia. Even between NumPy versions, the PRNG algorithm changed from Mersenne Twister (legacy) to PCG64 (default since NumPy 1.17). For full reproducibility, record not just the seed but the specific language, library, and version used.
Is there a risk of cherry-picking seeds?
Yes. Running a simulation multiple times with different seeds and reporting the seed that gives the most favorable results is seed cherry-picking, a subtle form of selective reporting. It inflates the apparent robustness of the strategy and produces confidence intervals that are too narrow. To avoid this: decide on the seed before running the simulation, run it once, and report those results. If the results look different for a different seed. That is information about sampling variance, increase the path count to reduce it.
Where should the seed and scenario count live so they survive a change of analyst?
In the same artifact as the result, not in a notebook cell or a shell command that disappears with the session. Writing them into the output file itself, alongside the input data version and the code revision, means anyone reading the numbers later can see what produced them without asking. Storing them only in a personal environment recreates the original problem one step removed, because the record depends on someone still having access to that environment.
References
- NumPy documentation, Random sampling and random generator: numpy.org/doc/stable/reference/random. Covers seeding, the PCG64 algorithm, and SeedSequence for parallel simulation.
- L'Ecuyer, P. (1999). "Good Parameters and Implementations for Combined Multiple Recursive Random Number Generators." Operations Research, 47(1), 159-164. Authoritative reference on PRNG algorithms for parallel simulation.
- Lopez de Prado, M. (2018). Advances in Financial Machine Learning. Wiley. Chapter 11 discusses reproducibility standards for quantitative research, including random seed management.
- ACM Transactions on Mathematical Software, Special issue on reproducibility in scientific computing (various issues 2010-2020). dl.acm.org/journal/toms.
- Bailey, D. H., et al. (2012). "Misleading Performance Reports in Financial Research." Journal of Investment Management. Covers data dredging and selective reporting practices in quantitative finance.
Educational Disclaimer
This guide is for educational and informational purposes only. Reproducibility standards apply to simulation methodology, they do not guarantee that a reproducible simulation accurately predicts future returns. Trading involves risk. Consult a qualified financial professional before making trading or investment decisions.