Properties and test-first

From examples to properties

An example test checks one input you chose.

A property states a rule that must hold for every valid input, and a test checks it on many. Several shapes recur. A round trip: decoding what you encoded gives back the original.

An invariant: the output of a sort is ordered and holds the same items. A bound: Relay's retry delay is always between one and sixty seconds. Agreement with a simpler version: a fast function returns what a slow, obviously correct one returns.

Idempotence: normalizing twice gives the same result as once. Writing a property makes you say what the contract really is, which an example lets you skip. Even without a library, a loop over inputs from a seeded generator checks a property and fails the same way on every run.

import random

def test_delay_stays_within_bounds():
    rng = random.Random(7)
    for _ in range(1_000):
        attempt = rng.randint(0, 10_000)
        assert 1 <= retry_delay(attempt) <= 60

Let a tool search, then shrink

A property-based testing library such as Hypothesis generates the inputs for you.

You describe their shape with strategies, such as lists of strings, and the test runs on a hundred generated examples by default, including awkward ones like empty lists and very large numbers. When an input fails, the library shrinks it: it keeps trying simpler versions that still fail and reports the simplest it finds, so a failure on a list of forty items is reported as one on two. In its plainest form, shrinking removes a part of the input and keeps the smaller version whenever it still fails.

Hypothesis saves failing inputs and tries them first on the next run, and @example pins a case you always want checked. A passing property is still a sample, not a proof, and a weak one passes for wrong code: “the result has no duplicates” holds for a function that returns an empty list.

from hypothesis import given, strategies as st

@given(st.lists(st.text()))
def test_dedup_keeps_every_item_once(items):
    result = stable_dedup(items)
    assert len(result) == len(set(items))
    assert set(result) == set(items)

Write the test first

Test-driven development turns tests into the way you design a change.

Keep a short list of behaviors still to add. Red: write one small test for the next behavior and run it, and watch it fail for the reason you expect; a test you have never seen fail may not be testing anything. Green: write the simplest code that makes it pass, even if it is clumsy.

Refactor: improve the names and structure, in both the code and the tests, while every test stays green. Then pick the next behavior. The small steps keep each failure close to its cause, and the tests describe what callers need before the implementation exists.

Properties fit the cycle too: when a property finds a failure, add the shrunk input as an explicit example, fix the code, and keep the example as a regression test.

# Red: written before stable_dedup exists, so it fails.
def test_dedup_keeps_the_first_occurrence():
    assert stable_dedup(["b", "a", "b"]) == ["b", "a"]

# Green: the simplest code that passes.
def stable_dedup(items):
    seen, result = set(), []
    for item in items:
        if item not in seen:
            seen.add(item)
            result.append(item)
    return result