Confidence Intervals for Strategy Metrics

Direct Answer

Every performance metric computed from a historical backtest, CAGR, Sharpe ratio, Sortino ratio, max drawdown, is an estimate derived from a finite sample of historical return observations. Like any statistical estimate, these metrics have uncertainty. Bootstrap confidence intervals quantify that uncertainty by showing the range of values the metric would plausibly take if the test were run on a slightly different historical sample.

A Sharpe ratio of 1.2 computed from 2 years of daily data has a 90% bootstrap confidence interval of approximately [0.6, 1.8], meaning the true Sharpe is about as likely to be near 0.6 as near 1.8, given 2 years of data. That wide interval is not a failure of the strategy; it is an honest statement of how precisely 2 years of data can estimate risk-adjusted returns. Reporting performance metrics without confidence intervals is selective disclosure, it omits the uncertainty that should drive deployment and sizing decisions.

Key Takeaways

Sample size determines CI width: Confidence intervals narrow in proportion to the square root of sample size. Doubling the data length (in years or trades) reduces CI width by approximately 30%.
The Sharpe ratio is imprecisely estimated over short periods: One year of daily data yields a 95% CI on Sharpe of approximately ±0.5, meaning a Sharpe of 1.0 is statistically consistent with anything from 0.5 to 1.5.
Max drawdown has the widest confidence intervals: As a path-dependent extreme statistic, max drawdown has very high sampling variance, its CI is typically the widest of any standard performance metric.
BCa bootstrap is preferred for skewed statistics: CAGR and max drawdown have skewed sampling distributions; BCa intervals provide better coverage than simple percentile intervals for these metrics.
A CI that includes zero for Sharpe is inconclusive: If the 95% CI on Sharpe includes zero, you cannot statistically distinguish the strategy from a zero-edge process given the historical data.
Calmar and Sortino CIs are wider than Sharpe CIs: These metrics use more extreme denominator statistics (max drawdown, downside deviation) that are less precisely estimated than full standard deviation.
Multiple strategy selection requires CI adjustment: Bootstrap CIs are valid for a pre-specified strategy evaluated once. Selecting the best of many strategies and applying standard CIs understates uncertainty, the deflated Sharpe ratio or multiple testing corrections are needed.
Report CIs alongside every point estimate: "Sharpe: 1.2 [90% CI: 0.7-1.7]" is a professional metric presentation. "Sharpe: 1.2" alone is not sufficient for a serious deployment decision.

Core Concepts

Why Performance Metrics Are Statistical Estimates

A strategy's CAGR over 3 years is not "the CAGR of the strategy". It is an estimate of the strategy's true expected annual return, computed from 3 years of realized returns. If the same strategy ran for a different 3-year window, it would produce a different CAGR estimate, because the specific sequence of market returns would differ. The historical CAGR is one draw from the distribution of possible CAGR estimates the strategy could produce over any 3-year window.

This statistical nature of performance metrics has a direct consequence: a point estimate alone (CAGR of 14.2%) is misleading without a corresponding interval estimate that quantifies the uncertainty. A 14.2% CAGR from 1 year of data is a far less reliable indicator of true long-run return than 14.2% from 10 years of data. The two estimates are numerically identical but statistically very different, the 1-year estimate has a much wider confidence interval around it.

The standard error of common performance metrics follows known patterns. For the mean return μ estimated from T periods: SE(μ) = σ/√T, where σ is the standard deviation of period returns. This means that to halve the uncertainty in the mean return estimate, you need 4× more data. For the Sharpe ratio SR = μ/σ: SE(SR) ≈ sqrt((1 + 0.5×SR²) / T) under normality assumptions. A Sharpe of 1.0 from 252 daily periods has SE ≈ 0.25; from 5 years (1,260 periods), SE ≈ 0.11. The Sharpe ratio is imprecisely estimated even from multi-year backtests.

Confidence Intervals for CAGR

The CAGR is computed from the product of (1 + r_t) across all periods t: CAGR = (∏(1 + r_t))^(1/T) − 1. Because it is a function of the product of returns (not just their sum). It is more precisely estimated than the Sharpe ratio for the same number of observations, being a function of many observations rather than an extreme statistic. The bootstrap distribution of CAGR is approximately log-normal (right-skewed), reflecting the multiplicative compounding of returns.

For a strategy with CAGR of 14.2% and daily return standard deviation of 1.1% over 5 years (1,260 trading days): the bootstrap 90% CI on CAGR spans approximately 9%, 19%. This is the range where the true CAGR is likely to lie. If a deployment decision requires the CAGR to exceed 8%, the lower bound of the CI suggests the strategy passes this requirement with high probability. If the required CAGR is 12%, the lower bound of 9% raises more uncertainty, the strategy might underperform the threshold with about 20-25% probability based on the bootstrap distribution.

BCa (bias-corrected and accelerated) bootstrap CIs are preferred for CAGR because the bootstrap distribution of CAGR is right-skewed, the upside tail is longer than the downside tail, reflecting the compounding benefit of high-return draws. Simple percentile CIs treat the distribution symmetrically and produce slightly conservative lower bounds and slightly optimistic upper bounds; BCa corrects both. For quantitative research publications or institutional strategy presentations, BCa is the appropriate standard.

Confidence Intervals for the Sharpe Ratio

The Sharpe ratio's standard error is known analytically under the assumption of i.i.d. normal returns: SE(SR) = sqrt((1 + SR²/2) / T). This gives: for SR = 0.5, T = 252: SE = sqrt((1.125)/252) ≈ 0.067, 95% CI approximately [0.37, 0.63]. For SR = 1.0, T = 252: SE ≈ sqrt(1.5/252) ≈ 0.077, 95% CI approximately [0.85, 1.15]. For SR = 2.0, T = 252: SE ≈ sqrt(3.0/252) ≈ 0.109, 95% CI approximately [1.78, 2.22].

These analytical standard errors are only valid under normality. For non-normal returns, which financial returns typically exhibit, bootstrap provides a non-parametric alternative that does not require distributional assumptions. The bootstrap CI for Sharpe is typically slightly wider than the analytical CI for the same data, reflecting the additional uncertainty from fat tails and skewness in the return distribution. The Lo (2002) correction extends the analytical Sharpe standard error to account for return autocorrelation, but for daily equity returns with limited autocorrelation, the unadjusted formula is often adequate.

Confidence Intervals for Max Drawdown and Sortino

Maximum drawdown has the highest sampling variance of the standard performance metrics. As a path-dependent statistic driven by the worst sequence of returns in the sample, max drawdown is highly sensitive to the specific returns that appear in each bootstrap sample. The resulting bootstrap distribution of max drawdown is right-skewed (long right tail), reflecting that some bootstrap samples will happen to cluster multiple large negative returns, producing very deep simulated drawdowns.

For a strategy with a historical max drawdown of −18%, the bootstrap 90% CI might span [−34%, −12%], a factor of 3 range from the worst to best percentile bounds. This wide interval is not a defect of the method; it is the honest statistical statement of how imprecisely max drawdown is estimated from a finite sample. It reinforces the point made in the drawdown distributions guide: the historical max drawdown is not "the drawdown" of the strategy, but one sample from a distribution with substantial right-tail risk.

The Sortino ratio replaces the Sharpe ratio's full standard deviation with the downside semi-deviation, the standard deviation of returns below a target threshold (often zero or the risk-free rate). Because the downside semi-deviation is estimated from a subset of observations (only the negative or below-target periods), it has higher sampling variance than the full standard deviation. Bootstrap CIs on the Sortino ratio are consequently wider than those on the Sharpe for the same sample. A strategy showing Sortino of 1.4 might have a 90% bootstrap CI of [0.6, 2.3], compared to a Sharpe CI of [0.7, 1.7] for the same data.

Worked Scenario

Strategy and data: A daily mean-reversion strategy on SPY over 4 years (1,008 trading days). Point estimates: CAGR 13.6%, Sharpe 0.97, Sortino 1.42, Max Drawdown −17.8%.
Bootstrap setup: Block bootstrap (block length 15 days, 10,000 samples, seed 2024) due to daily return autocorrelation. Method: BCa for CAGR, Max Drawdown; percentile for Sharpe and Sortino.
CAGR confidence interval: BCa 90% CI: [9.1%, 18.4%]. Lower bound (9.1%) exceeds the hurdle rate of 8%. Adequate confidence that the strategy meets the minimum return threshold.
Sharpe confidence interval: Percentile 90% CI: [0.59, 1.38]. The entire CI is above zero, statistically significant positive risk-adjusted return at the 10% level. Cannot distinguish from zero at the 5% level (95% CI needed).
Max Drawdown confidence interval: BCa 90% CI: [−31.4%, −11.2%]. The 5th percentile bound (worst 5% of bootstrap samples) reaches −38.2%. Position sizing should accommodate a drawdown of at least −30%.
Sortino confidence interval: Percentile 90% CI: [0.62, 2.31]. Wide interval, the Sortino is imprecisely estimated due to the small number of downside return observations used in the denominator.
Reporting format: "CAGR: 13.6% [90% CI: 9.1%, 18.4%] | Sharpe: 0.97 [90% CI: 0.59-1.38] | Sortino: 1.42 [90% CI: 0.62-2.31] | Max DD: −17.8% [90% CI: −31.4%, −11.2%]. Block bootstrap, l=15, 10,000 samples, seed 2024, 1,008 trading days."

Measurement Framework

Measurement	Question to Answer
90% CI lower bound on Sharpe	Is the Sharpe ratio statistically distinguishable from zero?
90% CI width on CAGR (relative to point estimate)	How precisely does the backtest estimate the annualized return?
Does 90% CI on CAGR exclude the required hurdle rate?	Does the strategy statistically satisfy the minimum return requirement?
Bootstrap p-value for Sharpe > 0	At what significance level is the positive Sharpe statistically meaningful?
90% CI on max drawdown (worst-case bound)	What drawdown level should position sizing be sized to survive?
CI width comparison: Sharpe vs. Sortino	Which metric provides a more precise characterization of the strategy's risk-adjusted return?
Standard error of bootstrap distribution for each metric	How stable is the confidence interval across different bootstrap seeds?

Common Failure Modes

Reporting Point Estimates Without Confidence Intervals

The most pervasive failure in strategy evaluation is reporting "Sharpe ratio: 1.2" without its confidence interval. The CI is not supplementary information. It is the primary result. A Sharpe of 1.2 with a 90% CI of [0.3, 2.1] cannot be meaningfully distinguished from a Sharpe of 0.5 or 1.8; it is an imprecise estimate consistent with a wide range of true values. Any deployment or allocation decision made without the CI is implicitly pretending the point estimate is more certain than the data supports.

Using Standard Bootstrap for Serially Correlated Daily Returns

Standard bootstrap applied to serially correlated daily returns underestimates CI width for path-dependent statistics like max drawdown and CAGR (over compounded paths), because it breaks up the autocorrelation structure that drives these statistics. Block bootstrap produces wider, more honest CIs for strategies evaluated on daily return data. Using standard bootstrap and claiming conservative CIs is an error that can lead to undercapitalized deployments and inadequate drawdown reserves.

Conflating CI Width With Evidence of No Edge

A very wide confidence interval reflects insufficient data, not absence of edge. A strategy with 30 trades can produce a Sharpe CI of [−0.5, 2.9] even for a genuinely excellent strategy. The correct interpretation is "insufficient data to measure," not "no edge." The response to wide CIs is to collect more data (longer backtest period, more trade history), not to conclude the strategy fails. Do not conflate statistical inconclusive with statistical evidence of no effect.

Neglecting Multiple Comparison Corrections After Strategy Selection

Bootstrap confidence intervals are statistically valid only for a pre-specified strategy evaluated once. When the strategy was selected from among 50 candidates, the standard bootstrap CIs on the selected strategy's performance substantially understate the true uncertainty, the selection process biases the historical estimate upward. The deflated Sharpe ratio (Bailey and Lopez de Prado, 2014) or White's Reality Check (White, 2000) provide multiple-comparison-corrected assessments. Applying standard CIs to a selected strategy without this correction produces spuriously narrow intervals that overstate statistical significance.

An Interval Is Not a Guarantee of Coverage

An interval built by resampling inherits every limitation of the resampling procedure. It reports the variation produced by the assumptions and the sample, not the full uncertainty about whether a strategy works. A tight interval around a performance figure means the estimate is stable within the sample, not that the result will persist outside it.

stock market business finance Confidence Intervals Strategy interval guarantee — Photo by moritz320 via Pixabay

That distinction determines how the numbers should be used. Intervals are good for comparing two strategies estimated on similar amounts of data, and for noticing when an apparently impressive figure rests on so few observations that it cannot be distinguished from a mediocre one.

Metrics differ in how well they behave under this treatment. Averages settle relatively quickly. Measures dominated by extremes, including the worst decline, are unstable and their intervals correspondingly wide, which is information rather than a defect.

Overlapping intervals for two strategies are not a formal test of difference either, and treating them as one reaches conclusions the procedure does not support.

Frequently Asked Questions

What does a 90% confidence interval on Sharpe ratio mean?

A 90% bootstrap confidence interval on the Sharpe ratio of, say, [0.6, 1.4] means that if you could repeatedly resample from the same return-generating process and compute the Sharpe for each resample, approximately 90% of those estimates would fall in a similar interval. It quantifies how much the Sharpe estimate varies due to the limited historical sample, a wide interval means the estimate is imprecise; a narrow interval means it is well-constrained.

How do I test if a strategy's Sharpe ratio is statistically significantly greater than zero?

Bootstrap hypothesis test: compute the bootstrap distribution of the Sharpe ratio. If the lower bound of the 95% confidence interval is above zero, the Sharpe is statistically significantly positive at the 5% level. Alternatively, compute the fraction of bootstrap samples with Sharpe ≤ 0. This is the p-value. A p-value below 0.05 indicates statistical significance at the 5% level.

Does a wide confidence interval mean the strategy doesn't work?

No, a wide confidence interval means the historical sample is too small to estimate the metric precisely. The strategy may have real edge that you cannot yet measure reliably. The correct response is to collect more data, not to conclude the strategy has no edge. However, if the confidence interval spans both positive and negative values, you cannot distinguish the strategy's edge from zero given the available data.

Which bootstrap method gives the most accurate confidence intervals?

The bias-corrected and accelerated (BCa) bootstrap provides the best coverage for statistics with skewed sampling distributions, including max drawdown, CAGR, and the Sharpe ratio when return distributions are non-normal. The simple percentile interval is acceptable for exploratory work but has known bias for skewed distributions. The studentized bootstrap (t-bootstrap) is accurate but requires estimating a variance function, which adds complexity.

How should I report strategy metrics in a professional context?

Report each metric as a point estimate followed by its bootstrap confidence interval and the CI level: e.g., "CAGR: 14.2% [90% CI: 9.8%, 18.7%, 5,000 bootstrap samples, seed 42]." Include the sample size (number of trades or days), the bootstrap method used (standard or block bootstrap with block length), and the seed for reproducibility. Never report point estimates alone in a professional evaluation.

Does the Sortino ratio have the same confidence interval width as the Sharpe ratio?

The Sortino ratio typically has wider confidence intervals than the Sharpe ratio for the same sample, because it uses only the downside semi-deviation in the denominator rather than the full standard deviation. The downside semi-deviation is estimated from fewer observations (only the negative returns), making it less precisely estimated. A Sortino of 1.4 might have a 90% CI of [0.6, 2.3], compared to a Sharpe of 1.0 with a CI of [0.6, 1.4] for the same strategy and sample.

Can I compare confidence intervals across strategies to rank them?

Overlapping confidence intervals do not necessarily mean two strategies perform equivalently, they mean the difference is not statistically significant given the available data. To formally compare strategies, compute the bootstrap distribution of the difference in Sharpe ratios between the two strategies and check whether zero is in the confidence interval for the difference. If the CI for (Strategy A Sharpe − Strategy B Sharpe) excludes zero, the difference is statistically significant.

What happens to confidence intervals if I select the best strategy from many?

When you test many strategy variants and report the metrics of the best-performing one, standard bootstrap CIs understate the true uncertainty. The selected strategy's historical performance is biased upward by multiple comparison effects. The deflated Sharpe ratio (Bailey and Lopez de Prado, 2014) adjusts for this by penalizing the Sharpe ratio based on the number of strategies trialed. Never treat standard bootstrap CIs as valid after a selection process without applying multiple-comparison corrections.

Should a confidence interval be computed before or after trading costs are subtracted?

After, if the interval is meant to describe what the strategy would have delivered to an account. Costs are not a constant offset that shifts the interval sideways: they scale with turnover, which varies across resampled paths, so a path with more trades loses more. Computing the interval on gross returns and then subtracting an average cost afterwards understates the spread. Applying costs inside each path before the metric is computed keeps the variation intact.

References

Lo, A. W. (2002). "The Statistics of Sharpe Ratios." Financial Analysts Journal, 58(4), 36-52. Derives the standard error of the Sharpe ratio estimator, including the autocorrelation correction.
Bailey, D. H., & Lopez de Prado, M. (2014). "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting, and Non-Normality." Journal of Portfolio Management. SSRN: ssrn.com/abstract=2460551.
Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall/CRC. Chapters 14-15 cover BCa intervals and confidence interval methods.
Harvey, C. R., Liu, Y., & Zhu, H. (2016). "… and the Cross-Section of Expected Returns." Review of Financial Studies, 29(1), 5-68. Discusses multiple comparison corrections for factor evaluation, principles applicable to strategy selection.
SciPy documentation: scipy.stats.bootstrap (BCa implementation in Python). docs.scipy.org.

Educational Disclaimer

This guide is for educational and informational purposes only. Bootstrap confidence intervals quantify statistical uncertainty in historical data, they do not predict future performance. Trading involves risk, including the possible loss of principal. Consult a qualified financial professional before making trading or investment decisions.