Primers
Measuring performance
Performance is how fast a program does its job, how much it can do at once, and what it uses to do it: memory, power, network. Every claim that one version is faster, or that a computer can handle the load, rests on a measurement. Measurements are easy to get and easy to get wrong. The numbers vary from run to run, the average hides the slow cases that people notice, and a test can measure something other than what you meant.
This page is about measuring well: what to measure, how to repeat it, how to report it, and the few laws that say what to expect. Pick your level at the top of the page.
High school
What to measure
Three questions cover most of it:
- How long does one job take? That’s latency: the time from asking to getting the answer. A web page that takes 1.2 seconds to load, a robot that takes 40 ms to react to an obstacle.
- How many jobs can it do in a second? That’s throughput. A server that answers 500 requests a second, a camera that gives 30 frames a second.
- What does it use? Memory, battery, network data, money.
They aren’t the same thing. A bus has worse latency than a car (you wait for it) and much better throughput (it carries fifty people). Decide which one matters before you measure.
Time it with a clock made for timing
The simplest measurement is a stopwatch in the code: read a clock, do the job, read the clock again, subtract.
Use the clock meant for measuring intervals. In Python that’s time.perf_counter(). The ordinary wall clock can
jump when the computer corrects its time over the network, and a jump in the middle of a measurement gives a wrong
answer, even a negative one.
Measure more than once
Run the same program twice and it takes a slightly different time. Other programs are running, the processor speeds up and slows down to save power, data is or isn’t already in memory. So one measurement tells you little. Measure many times and look at all of them.
Two numbers describe a set of times:
- The middle, the median: half the runs were faster, half slower.
- The spread: how far apart the fast and slow runs are.
The average can mislead
Five page loads take 1.2, 1.1, 1.3, 4.0 and 1.2 seconds. The average (mean) is 1.76 s, slower than four of the five loads. The median, 1.2 s, is what a typical load is like. But the 4.0 s load is real too, and it’s the one people remember. Report the typical time and the slow ones, not just an average.
Compare fairly
To show a change made something faster, change only that one thing. Same computer, same input, same everything else, and measure both versions the same number of times. If the difference between the versions is smaller than the spread between runs of the same version, you haven’t shown a difference.
Check yourself
- A robot’s control program must run every 20 ms. How many times a second is that?
- Program A takes 2.0 s. Program B takes 1.5 s. How much faster is B?
- Five runs take 10, 11, 10, 12 and 30 seconds. What are the mean and the median? Which describes a typical run?
- You time a change once, and it’s 3 % faster. Why isn’t that enough to say it’s faster?
<details><summary>Answers</summary>
- 1,000 ÷ 20 = 50 times a second.
- It takes 25 % less time (0.5 s of 2.0 s). Put the other way, A takes 2.0 ÷ 1.5 ≈ 1.33 times as long.
- Mean = 73 ÷ 5 = 14.6 s. Median = 11 s. The median: the 30 s run pulls the mean above four of the five.
- Runs of the same program vary, often by more than 3 %. Measure each version many times and check the difference is bigger than the spread.
</details>
College
Latency and throughput, together
For one worker doing one job at a time, throughput is 1 ÷ latency: 10 ms a job is 100 jobs a second. With many workers, or jobs that overlap, they come apart. Batching jobs often raises throughput and worsens latency, since each job waits for its batch. Always give units and say what was counted: “requests a second, measured at the server, over 60 s, 50 clients”.
Distributions and percentiles
A set of latencies is a distribution, and latency distributions usually have a long tail: most jobs are quick, and a few are much slower. Percentiles describe it:
- p50, the median: half the jobs were faster.
- p95, p99, p99.9: 95 %, 99 %, 99.9 % of jobs were at or below it.
- The maximum, the single slowest job.
The tail matters more than it looks. A page that makes 10 requests in parallel and waits for all of them is slow whenever any one is. If each request is slower than its p99 one time in 100, at least one of the ten is 1 − 0.99¹⁰ ≈ 10 % of the time. Report percentiles, and never average them: the mean of two p99s isn’t the p99 of anything.
Benchmarking
A benchmark is a repeatable measurement of a defined job. The common mistakes:
- Measuring the wrong thing. If the timed code also builds its input, you’re timing both. Set up outside the timed part.
- Too short to measure. A job that takes a microsecond needs repeating thousands of times in one timed loop, or the clock’s own resolution and overhead swamp it.
- One input. Many algorithms run at different speeds on different data. Sorting a list that’s already in order can be several times faster than sorting a shuffled one. Use inputs like the real ones.
- Cold against warm. The first run may load code and data from disk, fill caches, or (in languages with a just-in-time compiler) run slower until the compiler has done its work. Decide whether you’re measuring the first run or the steady state, and either discard the warm-up or report it.
- Work the compiler removed. In compiled languages, an optimiser deletes code whose result is never used, and the benchmark times nothing. Use the result.
Python’s timeit handles the repeating: it runs the code in a loop, repeats the loop several times, and reports
the best. The best is the run with the least interference from everything else on the machine. That’s a fair way
to compare two pieces of code, and an optimistic estimate of what a user will see.
Profiling: where the time goes
A profiler tells you which parts of a program take the time, so you speed up the right part. An
instrumenting profiler (Python’s cProfile) records every function call: exact counts, but it slows the
program and inflates small, often-called functions. A sampling profiler interrupts the program many times a
second and notes where it is: less exact, much less overhead, and fine for long runs.
Profile before optimising. The slow part is often not where you’d guess.
Amdahl’s law
If a fraction p of a program’s time can be sped up by a factor s, and the rest can’t, the whole program speeds up by
speedup = 1 ÷ ((1 − p) + p ÷ s)
With 80 % of the work spread over 4 cores, the speedup is 1 ÷ (0.2 + 0.2) = 2.5, not 4. With infinitely many cores it’s 1 ÷ 0.2 = 5. The part you didn’t speed up sets the limit.
Little’s law
For any stable system, the average number of jobs inside it (L) is the rate they arrive (λ) times the average time each spends inside (W):
L = λ × W
It needs no assumptions about the distributions. A server taking 200 requests a second, each in it for 50 ms, holds 200 × 0.05 = 10 requests at a time on average. It’s a quick check on any set of measurements: if the three numbers don’t agree, one of them is wrong.
Busy, queued, failing
For each resource (a processor, a disk, a network link, a lock), three numbers explain most slowdowns: how busy it is (utilisation), how much work is waiting for it (saturation, a queue), and how often it fails (errors). This is B. Gregg’s USE method. A resource at 100 % busy with a growing queue is the bottleneck.
Check yourself
- 80 % of a program can be split across cores. What’s the speedup on 4 cores? On a million?
- A service takes 200 requests a second and each spends 50 ms in it. How many are in it at once, on average?
- A profile says function f takes 60 % of the time. You make f twice as fast. How much faster is the program?
- Timing a sort that builds its random list inside the timed loop gave 840 µs. With the list built first, 474 µs. What went wrong in the first?
- Each of 10 parallel requests is slower than its p99 one time in 100. How often is at least one of them that slow?
<details><summary>Answers</summary>
- 1 ÷ (0.2 + 0.8 ÷ 4) = 2.5×. With a million cores, about 1 ÷ 0.2 = 5×.
- L = 200 × 0.05 = 10.
- The new time is 0.4 + 0.6 ÷ 2 = 0.7 of the old: about 1.43× faster.
- It timed building the list as well as sorting it: about 370 µs of the 840.
- 1 − 0.99¹⁰ ≈ 9.6 %, about one page in ten.
</details>
University
Queueing: why the tail blows up
A single worker, jobs arriving at random (a Poisson process, rate λ), and work times exponentially distributed with rate μ, is the M/M/1 queue. With utilisation ρ = λ ÷ μ < 1, the time a job spends in the system (waiting plus working) is exponentially distributed with rate μ − λ. So:
- the mean is 1 ÷ (μ − λ), which is (1 ÷ μ) ÷ (1 − ρ): the work time divided by the idle fraction;
- the p-th percentile is −ln(1 − p) times the mean: the median is 0.69 times it, p99 4.6 times, p99.9 6.9 times.
Going from 90 % to 95 % busy doubles the mean and every percentile. The “Try it” below simulates this and lands on the formula. For general arrival and work-time distributions (G/G/1), Kingman’s approximation gives the mean wait as about ρ ÷ (1 − ρ) × (cₐ² + cₛ²) ÷ 2 × (the mean work time), where cₐ and cₛ are the coefficients of variation of the gaps between arrivals and of the work times. Wait grows with both utilisation and variability. That’s why a system that looks fine at 70 % busy falls over at 95 %. Variation counts too, within limits: with random arrivals (cₐ = 1), making exponential work times (cₛ = 1) perfectly regular (cₛ = 0) halves the wait. Near saturation, a small cut in the mean work time lowers ρ and can cut the wait by far more.
The tail at scale
J. Dean and L. A. Barroso’s point: when one user request fans out to many servers, the user waits for the slowest, so rare slow responses become common slow pages. Two ways to measure it honestly: report the latency at the point the user sees it, and report high percentiles of that. The fixes they describe include hedged requests (send a second copy if the first is slower than the p95, and take whichever answers first) and keeping background work (garbage collection, compaction) off the request path.
Coordinated omission
Many load generators are closed: a client sends a request, waits for the answer, then sends the next. If the server stalls for one second, a client meant to send 100 requests a second sends just one during the stall, and records one slow sample instead of the 100 slow experiences real users would have had. The tool has coordinated with the system it measures, and omitted the very samples that make up the tail. Its p99 can be off by orders of magnitude. G. Tene named this. The fix is an open load generator that sends on a schedule regardless of answers, and measures each latency from when the request should have been sent (wrk2 does this).
Statistics for a benchmark
A. Georges, D. Buytaert and L. Eeckhout showed that common ways of reporting results (the best run, the mean of a few runs) can reverse which of two versions looks faster. Treat each benchmark result as a sample:
- Repeat at the level where the variation is. Runs within one process vary less than separate process starts, which vary less than reboots.
- Report a mean or median with a confidence interval. Ten runs with mean 412 µs and standard deviation 9 µs give a 95 % interval of about 412 ± 2.26 × 9 ÷ √10 ≈ 412 ± 6.4 µs (2.26 is Student’s t for 9 degrees of freedom).
- Call two versions different only when the intervals (or a proper test of the difference) say so.
Measurement bias
T. Mytkowicz, A. Diwan, M. Hauswirth and P. Sweeney found that things that shouldn’t matter can change a program’s measured speed enough to flip a conclusion: the size of the environment variables, or the order the object files were linked in. They shift where code and data land in memory, and with it how caches and branch predictors behave. The defence is to vary the setup across runs (randomise it), so the bias becomes spread you can see instead of an offset you can’t.
The roofline model
S. Williams, A. Waterman and D. Patterson’s roofline model bounds a program’s speed by two ceilings: the processor’s peak arithmetic rate, and memory bandwidth times the program’s arithmetic intensity (operations per byte moved from memory). A machine with 100 GFLOP/s peak and 20 GB/s of bandwidth runs a kernel doing 0.25 FLOP a byte at most min(100, 0.25 × 20) = 5 GFLOP/s: it’s memory-bound, and a faster processor won’t help. The ridge, where the ceilings meet, is at 100 ÷ 20 = 5 FLOP a byte.
Deadlines: real-time systems
A control loop that must run every 10 ms cares about its worst case, not its average. Two measures: jitter, how much the start times vary, and deadline misses, how often a run finishes late. A measured maximum is not a bound: the worst case may simply not have happened during the test. Hard real-time systems use worst-case execution time analysis, and designs that make timing predictable (fixed priorities, no allocation in the loop, interrupts kept short), and then measure to check, with margin.
Check yourself
- An M/M/1 server’s work takes 10 ms on average. What’s the mean time in the system at 90 % busy? At 95 %?
- A closed load tester with one client sending 100 requests a second hits a 1 s stall. How many slow samples does it record, and how many did users have?
- Ten runs give a mean of 412 µs with a standard deviation of 9 µs. What’s the 95 % confidence interval?
- 100 GFLOP/s peak, 20 GB/s memory bandwidth, a kernel at 0.25 FLOP a byte. What bounds it, and at what rate?
- A loop’s longest measured run in an hour of testing is 7 ms against a 10 ms deadline. Is it safe?
<details><summary>Answers</summary>
- 10 ÷ (1 − 0.9) = 100 ms; 10 ÷ (1 − 0.95) = 200 ms. Five points of utilisation doubled it.
- About one, against about 100: every request that should have gone out during the stall was slow.
- 412 ± 2.26 × 9 ÷ √10 ≈ 412 ± 6.4 µs.
- Memory bandwidth: 0.25 × 20 = 5 GFLOP/s, far under the peak.
- Not shown. An hour of measurement is a sample, not a bound. It needs worst-case analysis, or a design that bounds the time, plus testing under the worst load you can create.
</details>
Try it
No robot needed: a simulation of one worker taking jobs at random, at four levels of busy, and Python’s timeit
on a sort. You need only a container runtime. Start a shell with Python in it:
container
container run --rm -it docker.io/library/python:3.14-alpine sh
docker
docker run --rm -it python:3.14-alpine sh
podman
podman run --rm -it docker.io/library/python:3.14-alpine sh
Inside it, save the simulation and run it:
cat > busy.py <<'EOF'
"""One worker, jobs arriving at random: how long a job waits and runs, as the worker gets busier.
Each job takes 10 ms of work on average (exponentially distributed), and jobs arrive at random at a steady rate.
Busy is the share of the time the worker is working. The seed is fixed, so every run gives the same numbers.
"""
import math
import random
import statistics
WORK_MS = 10.0
JOBS = 200_000
def run(busy, seed=1):
rng = random.Random(seed)
gap_ms = WORK_MS / busy # the average time between arrivals
now = free_at = 0.0
times = []
for _ in range(JOBS):
now += rng.expovariate(1 / gap_ms) # the next job arrives
start = max(now, free_at) # it waits if the worker is still busy
free_at = start + rng.expovariate(1 / WORK_MS)
times.append(free_at - now) # waiting plus working
return times
def pct(xs, p):
"""The p-th percentile: p % of the values are at or below it."""
xs = sorted(xs)
return xs[min(len(xs) - 1, math.ceil(p / 100 * len(xs)) - 1)]
print("busy mean p50 p99 p99.9 (ms; theory: mean)")
for busy in (0.5, 0.8, 0.9, 0.95):
t = run(busy)
theory = WORK_MS / (1 - busy)
print(f"{busy:4.0%} {statistics.fmean(t):6.1f} {pct(t, 50):6.1f} {pct(t, 99):6.1f} {pct(t, 99.9):7.1f}"
f" ({theory:.0f})")
EOF
python busy.py
busy mean p50 p99 p99.9 (ms; theory: mean)
50% 20.0 13.9 91.8 140.1 (20)
80% 49.7 34.1 235.9 391.5 (50)
90% 98.6 67.8 485.7 726.7 (100)
95% 220.4 135.7 1193.8 1944.4 (200)
The work never changes: 10 ms a job on average. At half busy, a typical job is done in 14 ms and the slowest in a hundred under 92 ms. At 95 % busy the typical job takes ten times as long, and p99 is over a second. The mean tracks the formula, 10 ÷ (1 − busy). At 50 % busy the theory’s median and p99 are 13.9 and 92 ms, and the simulation lands on them. Near 95 % it’s further off, because jobs there wait behind long queues that build and drain slowly, and 200,000 jobs is a short sample of that.
Now time a sort three ways:
python -m timeit -r 7 "sorted(range(10000, 0, -1))"
python -m timeit -r 7 "import random; r = random.Random(1); sorted(r.random() for _ in range(10000))"
python -m timeit -r 7 -s "import random; r = random.Random(1); xs = [r.random() for _ in range(10000)]" "sorted(xs)"
2000 loops, best of 7: 114 usec per loop
500 loops, best of 7: 840 usec per loop
500 loops, best of 7: 474 usec per loop
Your numbers will differ: they depend on the computer. The ratios are the point. The same 10,000 items sort about four
times faster in reverse order than shuffled, because Python’s sort finds runs that are already in order. The second
line builds its list inside the timed code, so over 40 % of what it reports is building the list, not sorting. The
third builds it first (-s, setup) and times the sort alone.
Type exit to leave.
Learn more
- Python’s timeit and
profilers: repeated timing, why it reports the best, and
cProfile. - G. M. Amdahl, Validity of the single processor approach to achieving large scale computing capabilities, AFIPS 1967: the argument behind Amdahl’s law.
- J. D. C. Little, A Proof for the Queuing Formula: L = λW, Operations Research, 1961.
- J. Dean and L. A. Barroso, The Tail at Scale, Communications of the ACM, 2013: fan-out, tail latency, and hedged requests.
- B. Gregg, The USE Method: utilisation, saturation and errors for every resource.
- G. Tene’s wrk2: a load generator that sends at a constant rate, and its README on coordinated omission.
- A. Georges, D. Buytaert and L. Eeckhout, Statistically Rigorous Java Performance Evaluation, OOPSLA 2007: confidence intervals for benchmarks.
- T. Mytkowicz, A. Diwan, M. Hauswirth and P. F. Sweeney, Producing Wrong Data Without Doing Anything Obviously Wrong!, ASPLOS 2009: measurement bias from the environment and link order.
- S. Williams, A. Waterman and D. Patterson, Roofline, Communications of the ACM, 2009: an insightful visual performance model for multicore architectures.