Primers
Statistics for comparison
Is the new version better, or was it lucky? Every comparison of two programs, two robots, two settings or two models runs into this. Runs vary: the same program on the same task takes a different time, scores a different score, fails on a different try. A difference between two results can be real, or it can be the variation you’d see between two runs of the same thing.
This page is about telling those apart: how much to trust an average, how to set up a fair comparison, how to say how sure you are, and the ways a careful-looking comparison still fools you. Pick your level at the top of the page.
High school
One run each proves little
Version A finishes a task in 50 s. Version B finishes it in 48 s. Is B faster? Not yet. Run A again and it might take 47 s. Each run has some luck in it, and one run each can’t separate luck from a real difference.
Average and spread
Run each version many times. Two numbers describe the results:
- the average (the mean: add them up and divide by how many), or the median (the middle one);
- the spread: how far the runs are from each other. A common measure is the standard deviation, roughly how far a typical run is from the average.
A difference between the averages means something only when it’s large compared with the spread. Averages of 50 s and 48 s, with runs spread by 1 s, is a clear difference. With runs spread by 10 s, it could easily be luck.
Luck looks like a pattern
Flip a fair coin 10 times. Getting 7 or more heads happens about 17 % of the time, roughly one try in six. So 7 heads in 10 flips is no evidence the coin is unfair. Small samples produce lopsided results all the time. More runs make the luck even out: with 100 flips, 70 or more heads almost never happens with a fair coin.
The same test for both
A hard task slows any version down. If A happens to get the hard tasks and B the easy ones, B looks better for no reason. So give both versions the same tasks, in the same conditions. In a simulation, that means the same seeds: the numbers that decide every random choice, so both versions face exactly the same situations.
“Right 99 % of the time”
It sounds nearly perfect. But a robot that makes 500 decisions a day and is right 99 % of the time is wrong about 5 times a day. Whether that’s good enough depends on what a wrong decision costs.
The other way round: if something worked 10 times in a row, that doesn’t show it always works. A part that fails one time in 20 gets through 10 tries without failing about 60 % of the time.
Check yourself
- A takes 50 s once, B takes 48 s once. Can you say B is faster?
- A coin gives 7 heads in 10 flips. Is it unfair?
- A system is right 99 % of the time and makes 500 decisions a day. About how many are wrong each day?
- Why should both versions be tested on the same tasks?
<details><summary>Answers</summary>
- No. One run each doesn’t show how much runs vary. Run both many times and compare the difference with the spread.
- No evidence of it. A fair coin gives 7 or more heads in 10 flips about 17 % of the time.
- 500 × 0.01 = about 5.
- So a difference in the tasks (hard ones for one version, easy for the other) isn’t mistaken for a difference in the versions.
</details>
College
Samples and the standard error
The runs you made are a sample from all the runs you could make. Their mean x̄ is an estimate of the true mean, and it varies from sample to sample. The sample standard deviation is
s = √( Σ (x − x̄)² ÷ (n − 1) )
(dividing by n − 1, not n, because x̄ was worked out from the same data). How much the mean itself varies is the standard error, s ÷ √n. Four times the runs halves it.
Confidence intervals
A 95 % confidence interval for the true mean is
x̄ ± t × s ÷ √n
where t comes from Student’s t distribution with n − 1 degrees of freedom: 2.26 for 10 runs, 2.05 for 30, close to 1.96 for many. The 95 % is about the method: intervals built this way contain the true mean in 95 % of samples. It assumes runs are independent, and that either they’re roughly normal or there are enough of them.
Comparing two versions
Unpaired, each version run on tasks of its own: the difference of the means has standard error
√( sₐ² ÷ nₐ + sᵦ² ÷ nᵦ )
and Welch’s t test uses it without assuming the two spreads are equal.
Paired, both versions run on the same tasks (the same seeds): take the difference on each task, dᵢ = aᵢ − bᵢ, and work with the differences as one sample, with mean d̄ and spread s_d. Whatever makes a task hard affects both versions, so it cancels in the difference, and s_d can be much smaller than either version’s own spread. In the “Try it” below, each version’s runs spread by about 6 s and the differences by 3.7 s.
p-values, and what they aren’t
A p-value answers: if there were no real difference, how often would chance alone give a difference at least this large? A small p (below 0.05, by convention) says the data would be unusual if there were no difference. It is not the probability that there’s no difference, and it says nothing about how big the difference is. The American Statistical Association’s statement on p-values makes these points. Report the difference and its interval, and the p-value with them if you like, never alone.
Effect size
A difference can be real and too small to matter. With enough runs, 0.1 s out of 50 s becomes “significant”. Report the difference in its own units (B is 2.2 s faster, 95 % interval 0.9 to 3.5 s) and judge whether that much matters for the job.
Success rates
For a rate (successes out of n tries, p̂ = k ÷ n), the standard error is about √( p̂(1 − p̂) ÷ n ). That breaks down near 0 % or 100 %, where the Wilson score interval does better.
A special case: zero failures in n tries. The failure rate could still be as high as about 3 ÷ n (95 % confidence): the rule of three. It comes from (1 − p)ⁿ = 0.05, which gives p ≈ −ln(0.05) ÷ n ≈ 3 ÷ n. To claim fewer than one failure in 10,000, you need about 30,000 tries without one.
Check yourself
- 25 runs: mean 40 s, standard deviation 5 s. What’s the standard error? The 95 % interval (t = 2.06)?
- Paired on 36 tasks: the differences have mean 1.5 s and standard deviation 3 s. Is the difference clear (t = 2.03)?
- A program ran 600 times with no failures. How high could its failure rate still be?
- A test gives p = 0.03. Is there a 3 % chance that B is no better than A?
- Why does pairing make the interval narrower?
<details><summary>Answers</summary>
- 5 ÷ √25 = 1 s. 40 ± 2.06 × 1: about 37.9 to 42.1 s.
- Standard error 3 ÷ √36 = 0.5 s. 1.5 ± 2.03 × 0.5 is about 0.5 to 2.5 s, which doesn’t include 0: yes.
- 3 ÷ 600 = about 0.5 %, one in 200.
- No. It says that if B were no better, a difference this large would turn up about 3 % of the time. That’s a statement about the data, not the chance the hypothesis is true.
- What makes each task hard or easy affects both versions alike, so it cancels in each task’s difference and only the run-to-run luck is left.
</details>
University
Bootstrap intervals
The t interval leans on normality or large samples. B. Efron’s bootstrap leans on the sample itself: draw a new sample of the same size from your data, with replacement, compute the statistic, repeat thousands of times, and take the middle 95 % of the results (the percentile interval). It works for medians, percentiles, ratios and other statistics with no simple formula. It assumes the runs are independent and the sample represents the population. It does poorly with very few runs or heavy tails, where refinements such as BCa intervals help.
Permutation tests
If A and B are the same, the labels “A” and “B” are arbitrary. A permutation test shuffles the labels many times and counts how often the shuffled difference is at least as large as the real one. That share is the p-value, exact under the assumption that the labels can be swapped. For paired data, the shuffle is a random sign flip of each difference. Rank-based tests, like the Mann-Whitney U test for two unpaired samples, ask whether one tends to be larger than the other, and a few extreme runs barely move them.
Common random numbers
Var(a − b) = Var(a) + Var(b) − 2 Cov(a, b). Running both versions on the same seeds makes the covariance large, so the difference varies far less than either result. In simulation this is called common random numbers. It is the cheapest way to make a comparison more precise, often worth more than many extra runs.
Power and sample size
Power is the chance a comparison detects a real difference of a given size. For a two-sided test at 5 % with 80 % power, R. Lehr’s rule gives about 16 σ² ÷ δ² runs per version (unpaired), where σ is the spread of one run and δ the difference to detect. Paired, about 8 σ_d² ÷ δ² tasks. With the “Try it” settings (σ = 6.7 s, σ_d = 4.2 s, δ = 2 s) that’s about 180 runs each unpaired, against 36 tasks paired. With 30 runs, the paired comparison detects the 2 s about 73 % of the time, and the unpaired one about 21 %. Work the sample size out before running: an underpowered comparison mostly finds nothing, and when it does find something, it overstates its size.
Many comparisons
Test 20 variants that are no better than the baseline, each at p < 0.05, and the chance at least one looks significant is 1 − 0.95²⁰ ≈ 64 %. Corrections such as Bonferroni (test each at 0.05 ÷ m) or Holm’s step-down method control that. A. Gelman and E. Loken’s “garden of forking paths” is the subtler version: choices made after seeing the data (which runs to drop, which metric to report) are comparisons too, even when only one test is formally run.
Overfitting the test
Tune against a test set again and again, keep what scores best, and the score stops being an estimate of performance on new tasks. Part of it is the method fitting those particular tasks, and part is picking the luckiest of many noisy scores (regression to the mean: run the winner again and it usually does worse). A. Blum and M. Hardt show how a public leaderboard gets overfitted this way. C. Dwork and co-authors’ “reusable holdout” limits how much each look at the test data leaks. The plain defences: a hidden test set used once, for the final number, and fresh tasks for any claim made afterwards.
Few runs, high variance
Reinforcement learning shows all of this at once. P. Henderson and co-authors found that the non-determinism of benchmark environments, together with the variance of the methods themselves, makes reported results hard to interpret, and called for significance measures and standard reporting. R. Agarwal and co-authors showed that with the handful of runs per task that heavy benchmarks allow, point estimates (mean or median scores) can lead to conclusions a fuller analysis doesn’t support. They recommend interval estimates from a stratified bootstrap, performance profiles, and the interquartile mean (the mean of the middle 50 % of runs), which is less sensitive to outliers than the mean and has a smaller interval than the median.
Check yourself
- One run’s spread is 6.7 s. How many runs per version does an unpaired comparison need to detect 2 s with 80 % power at 5 %? Paired, with the differences spread by 4.2 s?
- 20 variants are each compared with the baseline at p < 0.05, and none is really better. How likely is at least one “significant” result?
- You tune a controller on the same 50 test tasks 200 times and report the best score. Why is it optimistic, and what fixes it?
- A paired comparison’s differences are mostly small, with two huge outliers. What would you use instead of a t interval?
- Why can a comparison on 3 seeds rank two methods the wrong way round, even with no mistake in the code?
<details><summary>Answers</summary>
- 16 × 6.7² ÷ 2² ≈ 180 runs each. Paired, 8 × 4.2² ÷ 2² ≈ 36 tasks.
- 1 − 0.95²⁰ ≈ 64 %.
- The tuning fits those 50 tasks, and the best of 200 noisy scores is partly luck. Report the result on a held-out set of tasks used once, at the end.
- A bootstrap interval or a permutation (sign-flip) test, and perhaps a statistic that resists outliers, like the median or the interquartile mean of the differences.
- With 3 runs each, the run-to-run spread can be larger than the difference between the methods, so luck decides the order. The “Try it” table shows how often.
</details>
Try it
No robot needed: a simulation of two versions of a program, with a known true difference, compared the ways this page describes. You need only a container runtime. Start a shell with Python in it:
container
container run --rm -it docker.io/library/python:3.14-alpine sh
docker
docker run --rm -it python:3.14-alpine sh
podman
podman run --rm -it docker.io/library/python:3.14-alpine sh
Inside it, save the program and run it:
cat > compare.py <<'EOF'
"""Two versions of a program, each run on test scenarios and timed: is B really faster, and how sure can we be?
Made-up numbers with a known answer: B is truly 2 s faster than A (48 s against 50 s). Each scenario has its own
difficulty, which slows both versions alike (spread 6 s), and each run has its own luck on top (spread 3 s). The
seed is fixed, so every run of this program gives the same numbers.
"""
import random
import statistics
rng = random.Random(7)
A_MEAN, B_MEAN, HARD, LUCK = 50.0, 48.0, 6.0, 3.0
def scenarios(n):
return [rng.gauss(0, HARD) for _ in range(n)]
def times(mean, hard):
return [mean + h + rng.gauss(0, LUCK) for h in hard]
# 1. Pick the winner by the average of a few runs, each version on scenarios of its own. How often is it wrong?
# Theory: one run's spread is the two spreads combined, (6² + 3²) ** 0.5 = 6.7 s, and the difference of two means
# of n runs has spread 6.7 × (2 / n) ** 0.5.
print("runs each B looks slower (theory)")
for n in (3, 10, 30, 100):
wrong = sum(statistics.fmean(times(B_MEAN, scenarios(n))) > statistics.fmean(times(A_MEAN, scenarios(n)))
for _ in range(10_000))
spread = (HARD ** 2 + LUCK ** 2) ** 0.5 * (2 / n) ** 0.5
theory = statistics.NormalDist(A_MEAN - B_MEAN, spread).cdf(0)
print(f"{n:9} {wrong / 10_000:13.1%} ({theory:.1%})")
def interval(xs, n=10_000):
"""A 95 % bootstrap interval for a mean: resample xs with replacement, and keep the middle 95 % of the means."""
means = sorted(statistics.fmean(rng.choices(xs, k=len(xs))) for _ in range(n))
return means[int(0.025 * n)], means[int(0.975 * n) - 1]
# 2. One experiment, 30 runs each: separate scenarios for each version, then the same 30 scenarios for both
n = 30
a, b = times(A_MEAN, scenarios(n)), times(B_MEAN, scenarios(n))
d = statistics.fmean(a) - statistics.fmean(b)
print(f"\nseparate scenarios: A {statistics.fmean(a):.1f} s, B {statistics.fmean(b):.1f} s, A - B = {d:+.1f} s")
print(f" spread of A {statistics.stdev(a):.1f} s, of B {statistics.stdev(b):.1f} s")
boot = sorted( # resample each version's runs on its own: they aren't pairs
statistics.fmean(rng.choices(a, k=n)) - statistics.fmean(rng.choices(b, k=n)) for _ in range(10_000))
print(f" 95 % interval for A - B: {boot[250]:+.1f} to {boot[9749]:+.1f} s")
same = scenarios(n)
a, b = times(A_MEAN, same), times(B_MEAN, same)
diffs = [x - y for x, y in zip(a, b)]
lo, hi = interval(diffs)
print(f"same scenarios: A {statistics.fmean(a):.1f} s, B {statistics.fmean(b):.1f} s, A - B = {statistics.fmean(diffs):+.1f} s")
print(f" spread of the differences {statistics.stdev(diffs):.1f} s")
print(f" 95 % interval for A - B: {lo:+.1f} to {hi:+.1f} s")
# 3. A permutation test on the same-scenario runs: if A and B were the same, each difference's sign would be luck.
# Flip the signs at random and count how often the mean is at least as far from 0 as the one we saw.
seen = abs(statistics.fmean(diffs))
flips = sum(abs(statistics.fmean([x if rng.random() < 0.5 else -x for x in diffs])) >= seen for _ in range(10_000))
print(f" sign-flip test: p = {flips / 10_000:.4f}")
# 4. No failures in 300 tries. How high could the failure rate still be?
for tries in (30, 300, 3000):
top = 1 - 0.05 ** (1 / tries) # the highest rate with at least a 5 % chance of no failures in `tries`
print(f"\n0 failures in {tries}: the rate could still be up to {top:.2%} (rule of three: {3 / tries:.2%})", end="")
print()
EOF
python compare.py
runs each B looks slower (theory)
3 35.9% (35.8%)
10 25.3% (25.2%)
30 12.8% (12.4%)
100 1.7% (1.8%)
separate scenarios: A 49.4 s, B 49.7 s, A - B = -0.3 s
spread of A 5.9 s, of B 6.7 s
95 % interval for A - B: -3.4 to +2.9 s
same scenarios: A 52.0 s, B 49.7 s, A - B = +2.2 s
spread of the differences 3.7 s
95 % interval for A - B: +0.9 to +3.5 s
sign-flip test: p = 0.0023
0 failures in 30: the rate could still be up to 9.50% (rule of three: 10.00%)
0 failures in 300: the rate could still be up to 0.99% (rule of three: 1.00%)
0 failures in 3000: the rate could still be up to 0.10% (rule of three: 0.10%)
B is truly 2 s faster, and the program knows it. What each part shows:
- Picking a winner by the average. With 3 runs each, the slower version looks faster about one time in three. Even with 100 runs each, it still happens about one time in 60. The simulated rates match the formula beside them.
- Separate scenarios, 30 runs each. The estimate is −0.3 s: the wrong way round. Its interval, −3.4 to +2.9 s, holds 0 and the true 2 s alike, so this comparison can’t tell. Each version’s runs spread by about 6 to 7 s, because some scenarios are hard and some easy.
- The same 30 scenarios for both. The estimate is +2.2 s, and the interval, +0.9 to +3.5 s, leaves out 0 and holds the true 2 s. The differences spread by 3.7 s: the scenarios’ difficulty cancelled. The sign-flip test agrees (p = 0.0023). Same number of runs, a clear answer, only because the comparison was paired.
- Zero failures. The exact bound, 1 − 0.05^(1/n), and the rule of three agree closely.
Type exit to leave.
Learn more
- Python’s statistics module: means, medians, standard deviations and the normal distribution, used in the “Try it”.
- Student, The Probable Error of a Mean, Biometrika, 1908: the t distribution.
- B. L. Welch, The generalization of “Student’s” problem when several different population variances are involved, Biometrika, 1947.
- B. Efron, Bootstrap Methods: Another Look at the Jackknife, The Annals of Statistics, 1979.
- H. B. Mann and D. R. Whitney, On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other, The Annals of Mathematical Statistics, 1947.
- E. B. Wilson, Probable Inference, the Law of Succession, and Statistical Inference, Journal of the American Statistical Association, 1927: the Wilson score interval.
- J. A. Hanley and A. Lippman-Hand, If Nothing Goes Wrong, Is Everything All Right?, JAMA, 1983: the rule of three.
- R. Lehr, Sixteen S-squared over D-squared: a relation for crude sample size estimates, Statistics in Medicine, 1992.
- R. L. Wasserstein and N. A. Lazar, The ASA Statement on p-Values, The American Statistician, 2016: what a p-value does and doesn’t say.
- A. Gelman and E. Loken, The Statistical Crisis in Science, American Scientist, 2014: the garden of forking paths.
- A. Blum and M. Hardt, The Ladder: A Reliable Leaderboard for Machine Learning Competitions, 2015.
- C. Dwork, V. Feldman, M. Hardt, T. Pitassi and co-authors, The reusable holdout: Preserving validity in adaptive data analysis, Science, 2015.
- P. Henderson, R. Islam, P. Bachman, J. Pineau and co-authors, Deep Reinforcement Learning That Matters, AAAI, 2018.
- R. Agarwal and co-authors, Deep Reinforcement Learning at the Edge of the Statistical Precipice, NeurIPS, 2021: interval estimates, performance profiles and the interquartile mean, with the rliable library.