Debugging as an experiment
Move from a symptom to a reproducible explanation using controlled observations.
BY THE END, YOU CAN
- Reduce a failure to a repeatable case
- Form a falsifiable debugging hypothesis
- Distinguish root cause from a downstream symptom
Diagram description: Reproduce → Hypothesis; Hypothesis → Experiment; Experiment → Observe; Observe → Hypothesis (refine); Observe → Fix + test (confirmed)
Reproduce before repairing
A bug report is often a symptom: 'jobs sometimes disappear.' Capture the input, observed result, expected result, and relevant environment.
Seek the smallest repeatable example that preserves the failure. Removing irrelevant setup reduces the number of possible causes. If the failure depends on concurrency or timing, record the ordering of events rather than assuming it is random.
A reproduction is a laboratory instrument: it lets you compare explanations and verify a repair without relying on memory or confidence.
Ask a question the program can answer
A useful hypothesis makes a prediction. 'We remove the job before recording completion, so a crash between those steps loses it' predicts a specific failure window.
Add a breakpoint, inspect state, or instrument that boundary. Change one relevant variable at a time. Read a traceback from the exception back through the call chain and inspect actual values, not the values you expected.
Logging can reveal history that a debugger misses, while a debugger can inspect details that were never logged.
Close the loop
Once the cause is understood, create a regression test that demonstrates the broken behavior.
Apply the smallest coherent fix and verify the test fails on the old behavior and passes on the corrected behavior. Then inspect nearby assumptions: was the error isolated, or does the same pattern appear elsewhere? For a production incident, a quick mitigation may come before a full explanation, but label it honestly.
Restarting a worker can restore service without resolving the reason the worker stalled in the first place.
COMMON FAILURE MODE
Making several speculative edits at once destroys the evidence needed to know which change mattered.
TRY IT YOURSELF
A worker removes a job, performs its effect, and then saves completion. Enumerate the observable outcomes of crashing before and after each step.
Make the call in the laboratory.
The two-question checkpoint, your notes, and your saved progress are in the laboratory. Free, with no account.
Open in the laboratory ↗CONNECT IT TO RELAY
Introduce one controlled failure into Relay, create a minimal reproduction, record the causal explanation, and add a regression test.