Observability that answers questions
Connect metrics, logs, and traces to the decisions an operator actually needs to make.
BY THE END, YOU CAN
- Choose signals from user-visible behavior
- Distinguish metrics, logs, and traces
- Control cardinality and sensitive data
Diagram description: Why are jobs late? → Metrics; Why are jobs late? → Traces; Why are jobs late? → Logs; Metrics → Answer; Traces → Answer; Logs → Answer
Begin with a question
Observability is useful when it helps explain what the system is doing from its outputs.
Start with operational questions: are jobs being accepted, are they completing on time, and where are failures concentrated? Traffic, errors, latency, and saturation provide a useful starting set of signals. For a queue, oldest waiting-job age may reveal user pain more directly than CPU utilization.
A healthy HTTP endpoint can coexist with a worker fleet that has stopped processing, so observe the end-to-end outcome as well as individual components.
Use the signals together
Metrics aggregate measurements over time and support trends and alerts.
Logs capture events and context. Traces connect work across component boundaries and help locate delays along a path. Carry correlation context from intake through queue storage to worker execution, while remembering that asynchronous work may outlive the original request.
A job ID belongs in relevant logs or trace attributes; using every job ID as a metric label can create unbounded time-series cardinality and make monitoring expensive or unusable.
Instrument the failure paths
Record acceptance, claim, retry, completion, and permanent failure with consistent meanings.
Separate queue wait from execution duration so a slow system points toward scheduling or handling rather than one blended number. Measure distributions and tail latency; averages can hide a small group of users waiting far longer. Avoid logging payload secrets or unnecessary personal information.
Alerts should indicate an actionable condition with an owner and a useful runbook. A dashboard no one can interpret during an incident is decoration, however polished it looks.
COMMON FAILURE MODE
Averages and success-only logs can make a system look healthy while slow or failed jobs disappear from the measurement.
TRY IT YOURSELF
Choose three Relay metrics, two structured log events, and one trace path. For each, write the operational question it answers and any sensitive data to exclude.
Make the call in the laboratory.
The two-question checkpoint, your notes, and your saved progress are in the laboratory. Free, with no account.
Open in the laboratory ↗CONNECT IT TO RELAY
Instrument Relay's queue wait, execution duration, completion outcomes, and oldest pending-job age with consistent definitions.