Incidents, recovery & learning
Restore service with evidence, communicate clearly, and turn a failure into a better system.
BY THE END, YOU CAN
- Separate mitigation, diagnosis, and recovery verification
- Define recovery point and recovery time objectives
- Write a blameless account with concrete corrective actions
Diagram description: Detect: 0–8; Mitigate: 8–25 roll back; Recover: 25–70 drain backlog; Learn: 80–120 blameless review; 0: alert
Stabilize before optimizing
During an incident, prioritize reducing user harm and preserving enough evidence to understand what happened.
Assign clear coordination, investigation, and communication responsibilities when the incident is large enough to need them. Record a timeline of observed facts and actions. A rollback, admission limit, or dependency bypass may be an appropriate mitigation before the full cause is known.
Change one significant variable at a time when possible and verify its effect. Avoid a rush of uncoordinated fixes that make the system state impossible to interpret.
Recovery is a behavior you test
A recovery time objective describes the desired time to restore service; a recovery point objective describes the acceptable amount of data loss measured back from the disruption.
Backups alone do not prove either objective can be met. Practice restoration, verify data integrity, and ensure credentials and procedures are available when the primary environment is impaired. For Relay, recovery must also reconcile queued, claimed, and externally completed work.
A restored database may bring back jobs whose external effects already happened, so replay safety remains important.
Explain the conditions, then change them
A useful post-incident review describes impact, timeline, contributing conditions, detection, response, and follow-up work.
Blamelessness means examining why an action made sense with the information and system available, while still assigning owners to improvements. Avoid the non-explanation 'human error' and the vague action 'be more careful.' Prefer changes such as a tested restore procedure, a guarded migration, or an alert for stalled queue age. Verify follow-up effectiveness rather than treating a list of tickets as proof the system is safer.
COMMON FAILURE MODE
Declaring recovery when processes are running can miss corrupted data, stuck jobs, and users whose work still has not completed.
TRY IT YOURSELF
Write a mini incident report for a deployment that doubles queue age. Include a factual timeline, a mitigation, a recovery check, and one prevention action with an owner.
Make the call in the laboratory.
The two-question checkpoint, your notes, and your saved progress are in the laboratory. Free, with no account.
Open in the laboratory ↗CONNECT IT TO RELAY
Run a Relay failure exercise: stop a worker mid-job, recover pending work, and document duplicate-effect risks and verified recovery behavior.