Primers
Testing software
A test is a small program that runs a piece of your program and checks the answer. Tests catch a mistake while it is still cheap: before the code is merged, before it runs on a robot, before someone else builds on it. They also let you change code without fear, because the tests tell you when a change broke something that used to work.
A test can only show that a bug is there, never that none is left. Most of the skill is picking cases that would catch the bugs you’re likely to write, and checking that each test would really fail if the code were wrong.
The examples on this page test one small function from coordinates and poses: the turn from one heading to another, the short way round. Pick your level at the top of the page.
High school
What a test is
Say you’ve written a function that works out how far a robot should turn to face a new direction. You want
turn(350, 10) to give 20: from 350° to 10° is 20° to the left, across zero. A test is a few lines that call the
function with those numbers and compare what comes back with what you expected:
def test_the_short_way_across_zero(self):
self.assertEqual(turn(350, 10), 20)
If the function gives 20, the test passes. If it gives anything else, the test fails, and says what it got instead. You keep the tests next to the code and run them every time you change it, all at once, with one command.
Set up, act, check
Most tests have three parts:
- Set up what the test needs: some numbers, a list, a pretend sensor reading.
- Act: call the code being tested.
- Check the result against the answer you worked out yourself, beforehand.
The check is the part that matters. A test that calls the code and checks nothing passes whatever the code does.
Picking the cases
You can’t test every possible input, so you choose. Good tests cover three kinds of case:
- The ordinary: a normal input the code sees every day.
turn(0, 90)should be 90. - The edges: where the behaviour changes. For the turn, that’s crossing zero (350 to 10) and exactly half a turn (0 to 180: left or right?). Most bugs live at edges.
- The wrong: inputs the code shouldn’t get but might. An empty list, a negative distance, a heading of 720°. Decide what the code should do with them, then test that it does.
When a test fails
A failing test is good news: it found a problem before anyone else did. Read what it says. It names the test, the line, and what it expected against what it got:
FAIL: test_the_short_way_across_zero
AssertionError: -340 != 20
Either the code is wrong or the test’s expected answer is. Work out which before you change anything. Never “fix” a failing test by changing its expected answer to whatever the code gave, unless you’ve checked the code is right.
A test that can’t fail proves nothing
Here’s a test that looks fine:
def test_turn_gives_a_number(self):
self.assertIsInstance(turn(350, 10), int)
It passes when turn gives 20. It also passes when turn gives −340, the wrong answer the long way round. It
can’t tell right from wrong, so it protects nothing. The way to find out is to break the code on purpose and run
the tests: a good test fails.
Check yourself
- A function
average(numbers)gives the mean of a list. Name three cases worth a test. - A test checks only that
turn(350, 10)gives back a number. The code gives −340. Does the test pass? - You’ve found that
turn(0, 180)gives a different answer every time someone edits the code. What do you do first? - A test passed yesterday and fails today. Nobody changed the code. Give two possible reasons.
<details><summary>Answers</summary>
- An ordinary list (2, 4, 6 gives 4), one number (5 gives 5), and the wrong case, an empty list: decide what it should do (an error, most likely) and test that.
- Yes. −340 is a number. The test can’t tell a right answer from a wrong one.
- Decide the right answer (say −180, turning right) and write a test for it. Then nobody can change it without the test failing.
- Any two of: it depends on today’s date or the time, on a random number, on a network or a website, or on what another test left behind.
</details>
College
Unit, integration and end to end
Tests come in sizes:
- Unit tests check one function or class on its own. They run in milliseconds and point straight at the problem when they fail.
- Integration tests check that parts work together: your code with a real database, a real message broker, a real file.
- End-to-end tests run the whole system the way a user would: start everything, click or send commands, check what comes out.
The bigger the test, the more it proves about the real system, and the slower, flakier and harder to diagnose it is. The usual advice is a pyramid: many unit tests, fewer integration tests, a handful of end-to-end tests.
Running tests
Python ships with unittest. A test is a method whose name starts with test, in a class that subclasses
unittest.TestCase, in a file whose name starts with test:
python -m unittest # find and run every test
python -m unittest -v # and name each one
python -m unittest test_turns.Turn.test_the_edges # just one
pytest is the other common runner: plain assert statements instead of assertEqual, clearer failure reports,
and fixtures. It also runs unittest tests unchanged.
Name a test for what it shows, as a sentence: test_past_half_a_turn_comes_round_the_other_side. When it fails,
the name tells you what broke before you open the file.
Each test on its own
A test should pass or fail by itself, in any order, on any machine. Tests that share state break that: one test leaves a file, a global variable or a database row behind, and another test passes only because of it, or fails only after it.
Give each test fresh state. In unittest, setUp runs before each test and tearDown after it. Write to a
temporary directory, not a fixed path. Start each test from an empty database. A test that passes alone and fails
in the full run, or the other way round, is the sign of shared state.
Fakes, stubs and mocks
Unit tests shouldn’t need the network, a sensor or a motor. You replace those with a test double, a stand-in you control:
- A stub gives canned answers: a sensor that always reads 412 mm.
- A fake is a working, simpler version: an in-memory database instead of a real one.
- A mock records how it was called, so the test can check it: “the motor was told to stop, once”.
The easiest code to test needs no doubles at all. Split the logic (turn counts into distance, choose the turn) from the parts that touch the world (read the encoder, drive the motor), and test the logic directly. Doubles have a cost: they check what you assumed the real thing does. If the real thing changes, the doubles don’t, and the tests keep passing. An integration test against the real thing catches that.
Write the failing test first
When you fix a bug, write the test for it before the fix, and run it. It should fail, the same way the bug does. Then fix the code and watch it pass. That order proves two things: the test really catches this bug, and the fix really fixes it. A test written after the fix might pass either way, and you’d never know.
Writing every test first, before the code, is called test-driven development: red (a failing test), green (the simplest code that passes), refactor (tidy up with the tests as a safety net).
Checking an old test is the same idea run backwards: switch the fix off, run the tests, and check at least one fails. If none do, the fix is unprotected.
Continuous integration
Continuous integration (CI) runs the whole suite automatically on every change pushed to a shared repository, on a clean machine. A change merges only when CI is green. That catches “works on my machine”: a file you forgot to commit, a library version that differs, a test that only passed because of something left on your laptop.
Coverage, and what it misses
A coverage tool records which lines (or branches) the tests ran. Lines no test ran are untested for sure.
The reverse isn’t true: a line that ran isn’t checked. A test that calls turn and checks nothing gives
turn full coverage. Coverage also can’t see code that’s missing: if you forgot to handle the empty list, there’s
no line for it to report. Use coverage to find gaps, not as a score that says the tests are good.
Check yourself
- A function reads bytes from a sensor over a serial port and turns them into millimetres. How do you unit test the conversion without the sensor?
- Coverage says every line ran. Name two kinds of bug it could still miss.
- Two tests pass when run alone. Run together, the second fails. What’s the likely cause, and the fix?
- Why run the new test before the fix, not only after?
- A mock stands in for a web service and the tests pass. Then the service changes the format of its replies. Do the tests notice? What would?
<details><summary>Answers</summary>
- Split it: a function that takes bytes and gives millimetres, tested directly with bytes you write in the test. Or pass the reader in, and give the test a stub that returns those bytes.
- Any two of: a line that ran but whose result no test checked; code that’s missing altogether (an unhandled
case); a wrong boundary (
<for<=) where the line runs either way. - Shared state: the first test leaves something behind (a file, a global, a row). Give each test fresh state,
with
setUpandtearDownor a temporary directory. - To see it fail. That proves it catches this bug. A test you’ve only seen pass might pass with the bug too.
- No: the mock still gives the old format. An integration test against the real service would, or a check of recorded replies against the service now and then.
</details>
University
What testing can and can’t show
A test runs the program on a handful of inputs out of a space that’s usually far too large to cover. turn takes
two floating-point numbers: about 2¹²⁸ pairs. So a passing suite is evidence, not proof. E. W. Dijkstra’s point,
from 1970: testing can show the presence of bugs, never their absence. Proof needs other tools: types, formal
specifications, model checking, or proof assistants. They cost far more and are used where a bug would cost more
still (aircraft, chips, cryptography).
Testing’s job, then, is to be a good sample: inputs chosen where bugs are likely, and checks that would notice them.
The oracle problem
Every test needs an oracle, something that says what the right answer is. Writing 20 into the test is a
hand-worked oracle, and it doesn’t scale: you can’t hand-work ten thousand cases, and for some programs (a path
planner, a simulator, a trained model) nobody knows the exact right answer.
Two ways out:
- Properties. Instead of exact answers, state what must be true for every input. For
turn(h, g): the result is in [−180, 180), and turning by it from h ends up facing g: (h +turn(h, g)− g) is a whole number of turns. A test checks the properties over many inputs. - Metamorphic relations. State how the output must change when the input changes. Turning from g back to h
should be the opposite turn:
turn(g, h) == -turn(h, g). Careful: that one is false at exactly half a turn.turn(0, 180)andturn(180, 0)are both −180, because [−180, 180) holds −180 and not +180. Writing the relation down is how you find out which edge cases you hadn’t decided.
Property-based testing
A property-based testing library (QuickCheck in Haskell, from 2000; Hypothesis in Python) generates the inputs
for you: a hundred random cases per property by default, within the range you give, and biased toward the values
that often break code (zero, the ends of the range, and with no range, huge numbers and NaN). When a case fails, it
shrinks it: tries smaller and simpler inputs until it finds the simplest one that still fails, so one input that
doesn’t matter comes back as 0.0 and the report points straight at the other.
from hypothesis import given, strategies as st
angles = st.floats(-1e6, 1e6)
@given(angles, angles)
def test_every_turn_is_at_most_half_a_turn(h, g):
assert -180 <= turn(h, g) < 180
The property is false on floats, at one value. wrap(-180.00000000000003), the float just below −180, gives
180.0, outside the range: the remainder comes out a hair under 360 and rounds up to 360. The function is correct on
whole numbers. When the test above was run for this page, Hypothesis’s hundred random cases passed without
hitting that value: generated inputs are still a sample. Had it hit it, the shrunk report would read something
like h=0.0, g=-180.00000000000003. A case for the float on each side of each edge
(math.nextafter(-180, -math.inf)) finds it at once.
Testing the tests: mutation testing
Coverage asks whether the tests ran the code. Mutation testing asks whether they’d notice it was wrong. A tool
makes many small changes to the code, one at a time (each a mutant): < to <=, + to -, a constant changed,
a line deleted. It runs the suite against each. If some test fails, the mutant is killed. If every test still
passes, the mutant survived, and the suite has a blind spot exactly there.
The mutation score is the killed mutants over the total, leaving out equivalent mutants: changes that don’t
change behaviour (for example, < to <= where the two values can never be equal). Deciding whether a mutant is
equivalent is undecidable in general, so a person looks at the survivors. Two hypotheses justify the method: that
programmers write code close to correct (the competent programmer hypothesis), and that tests which catch small
faults also catch most of the bigger faults made of them (the coupling effect). The cost is a full suite run per
mutant, so tools run only the tests that cover the changed line.
Switching a bug fix off and checking a test fails is mutation testing by hand, one mutant: the one that matters.
Flaky tests
A flaky test passes and fails on the same code. Studies of open-source projects put the commonest causes as
waiting for something asynchronous (a fixed sleep(0.1) that’s usually long enough), concurrency (two threads in an
order nobody planned for), and test-order dependence (shared state, from the college level). Others: time and
dates, random numbers without a fixed seed, the network, floating-point results that differ across machines.
Flakiness compounds. A suite of 400 tests, each failing at random one run in 200, is all green only 0.995⁴⁰⁰ ≈ 13 % of the time. A team learns to rerun until green, and then a real failure gets rerun away too.
The fixes are the same few ideas: wait on the condition (with a timeout), not on a fixed delay; inject the clock and the random source so the test controls them; fix the seed and print it, so a failure can be replayed; isolate each test’s state.
Software that drives hardware
Code that runs a machine adds timing, physics and parts that wear. Tests are layered the same way as before, by how much of the real thing they include:
- On the host: the logic compiled and run on a computer, with the hardware stubbed. Fast, every change.
- Software in the loop (SIL): the real control code against a simulation of the machine. It tests the loop’s behaviour over time: does the speed settle, does it stop in time?
- Hardware in the loop (HIL): the real controller board, with its inputs and outputs wired to a simulator or a test rig instead of the machine. It tests timing, drivers and the board itself.
- On the machine: the real thing, on a bench or a test track, last and least often.
What’s tested lower down doesn’t need repeating higher up. What only shows up higher (a motor’s real inertia, a sensor’s real noise, an interrupt that runs late) needs a test there, with margins: “stops within 30 cm” rather than “stops at 27.4 cm”.
Check yourself
- Give properties a test could check for a function that sorts a list. Why isn’t “the output is in order” enough?
- A mutant changes a
<to<=and survives. Give two explanations. - A test sends a message, sleeps 100 ms, then checks it arrived. It fails about one run in 200. How do you fix it without making the suite slower?
- A suite has 1,000 tests, each independently flaky one run in 1,000. Roughly how often is a run all green?
turnis tested on every whole-number heading and goal from −720 to 720. Is it now proved correct?
<details><summary>Answers</summary>
- The output is in order, and it holds the same items as the input, each as many times. In order alone is met by returning an empty list.
- No test checks that boundary (no case where the two values are equal), or the mutant is equivalent: the values can never be equal there, so nothing could tell the two apart.
- Wait on the condition: check for the message in a loop, or block on an event, with a timeout of a second or more. A passing run returns as soon as it arrives, usually faster than 100 ms, and a slow run still passes.
- 0.999¹⁰⁰⁰ ≈ e⁻¹ ≈ 37 %.
- No. It’s proved for those integers only (the loop is exhaustive over them). Floats are another input space, and the float just below −180 fails, above.
</details>
Try it
No robot needed: the turn function, five tests for it, and what happens when the function is broken on purpose. You need only a container runtime. Start a shell with Python in it:
container
container run --rm -it docker.io/library/python:3.14-alpine sh
docker
docker run --rm -it python:3.14-alpine sh
podman
podman run --rm -it docker.io/library/python:3.14-alpine sh
Inside it, save the function and its tests:
cat > turns.py <<'EOF'
"""The turn from one heading to another, the short way round, in degrees."""
def wrap(a):
"""An angle in degrees, brought into [-180, 180)."""
return (a + 180) % 360 - 180
def turn(heading, goal):
"""How far to turn to face goal: positive is to the left (counter-clockwise)."""
return wrap(goal - heading)
EOF
cat > test_turns.py <<'EOF'
import unittest
from turns import turn, wrap
class Wrap(unittest.TestCase):
def test_angles_already_in_range_are_left_alone(self):
self.assertEqual(wrap(10), 10)
def test_past_half_a_turn_comes_round_the_other_side(self):
self.assertEqual(wrap(190), -170)
self.assertEqual(wrap(-190), 170)
def test_the_edges(self):
self.assertEqual(wrap(180), -180) # 180 itself is outside [-180, 180)
self.assertEqual(wrap(-180), -180)
self.assertEqual(wrap(360), 0)
class Turn(unittest.TestCase):
def test_the_short_way_across_zero(self):
self.assertEqual(turn(350, 10), 20) # 20 to the left, not 340 to the right
self.assertEqual(turn(10, 350), -20)
def test_every_turn_is_at_most_half_a_turn(self):
for heading in range(-720, 721, 7):
for goal in range(-720, 721, 11):
t = turn(heading, goal)
case = f"from {heading} to {goal}, turn {t}"
self.assertTrue(-180 <= t < 180, case)
self.assertEqual((heading + t - goal) % 360, 0, case) # and it ends up facing the goal
if __name__ == "__main__":
unittest.main()
EOF
python -m unittest -v
test_every_turn_is_at_most_half_a_turn (test_turns.Turn.test_every_turn_is_at_most_half_a_turn) ... ok
test_the_short_way_across_zero (test_turns.Turn.test_the_short_way_across_zero) ... ok
test_angles_already_in_range_are_left_alone (test_turns.Wrap.test_angles_already_in_range_are_left_alone) ... ok
test_past_half_a_turn_comes_round_the_other_side (test_turns.Wrap.test_past_half_a_turn_comes_round_the_other_side) ... ok
test_the_edges (test_turns.Wrap.test_the_edges) ... ok
----------------------------------------------------------------------
Ran 5 tests in 0.010s
OK
Four tests check exact answers someone worked out by hand. The fifth checks two properties over about 27,000 pairs of headings: every turn is at most half a turn, and it ends up facing the goal.
Now break the function, so wrap gives back the angle unchanged, and run the tests again. The grep keeps just the
name of each failing test and what it got:
sed -i 's/ return (a + 180) % 360 - 180/ return a/' turns.py
python -m unittest 2>&1 | grep -E "^(FAIL|AssertionError|Ran)"
FAIL: test_every_turn_is_at_most_half_a_turn (test_turns.Turn.test_every_turn_is_at_most_half_a_turn)
AssertionError: False is not true : from -720 to -533, turn 187
FAIL: test_the_short_way_across_zero (test_turns.Turn.test_the_short_way_across_zero)
AssertionError: -340 != 20
FAIL: test_past_half_a_turn_comes_round_the_other_side (test_turns.Wrap.test_past_half_a_turn_comes_round_the_other_side)
AssertionError: 190 != -170
FAIL: test_the_edges (test_turns.Wrap.test_the_edges)
AssertionError: 180 != -180
Ran 5 tests in 0.001s
FAILED (failures=4)
Four of the five fail, and each says what went wrong: the long way round (−340 for 20), and a turn of 187°. The
one that still passes, wrap(10), checks an angle already in range, which the broken function leaves alone too.
It’s a fine test, just not one that could catch this bug. Run one failing test by its name for the full report:
python -m unittest test_turns.Turn.test_the_short_way_across_zero
F
======================================================================
FAIL: test_the_short_way_across_zero (test_turns.Turn.test_the_short_way_across_zero)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/test_turns.py", line 22, in test_the_short_way_across_zero
self.assertEqual(turn(350, 10), 20) # 20 to the left, not 340 to the right
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
AssertionError: -340 != 20
----------------------------------------------------------------------
Ran 1 test in 0.000s
FAILED (failures=1)
Put the function back and everything passes again:
sed -i 's/ return a$/ return (a + 180) % 360 - 180/' turns.py
python -m unittest
.....
----------------------------------------------------------------------
Ran 5 tests in 0.009s
OK
Type exit to leave.
Learn more
- Python’s unittest: test cases,
setUpandtearDown, running one test, andunittest.mock’s stand-ins. - pytest: plain
assert, fixtures, and runningunittestsuites. - Hypothesis: property-based testing for Python, with generated inputs and shrinking.
- K. Claessen and J. Hughes, QuickCheck: a lightweight tool for random testing of Haskell programs, ICFP 2000: where property-based testing started.
- Y. Jia and M. Harman, An Analysis and Survey of the Development of Mutation Testing, IEEE TSE, 2011: mutants, equivalent mutants, and the coupling effect.
- Q. Luo, F. Hariri, L. Eloussi and D. Marinov, An Empirical Analysis of Flaky Tests, FSE 2014: what makes tests flaky, from real projects’ fixes.
- M. Fowler, Mocks Aren’t Stubs and Test Pyramid: the kinds of test double, and the sizes of test.
- E. W. Dijkstra, Notes on Structured Programming, 1970: the source of the point that testing shows the presence of bugs, never their absence.