Runbook 05 · Observability

Monitor the user journey before the server trivia

A small-team monitoring baseline: prove the service works, expose risk early, and alert only when a person can take useful action.

RB–05 / v1.0Output: Monitor + alert policyPrint-friendly

Start here: name the smallest external journey that proves users receive value. A green CPU graph does not prove checkout, login, sync, or delivery works.

A connected service signal moving through normal, degraded, and down states.
Pair every color with a written state and user impact. Color is reinforcement, not the only signal.

The minimum signal set

Availability

Can a user complete the critical journey from outside the production network? Prefer a synthetic check that exercises DNS, TLS, routing, application logic, and a safe read or write.

Correctness

Does the response contain valid, current results—not merely HTTP 200? Check data freshness, queue age, scheduled outputs, and business invariants.

Latency

How long does the journey take at meaningful percentiles? Set thresholds from user impact and normal baselines, not round numbers copied from another service.

Saturation

Which finite resources cause failure: workers, connections, disk, memory, rate limits, queue depth, or vendor quota? Alert on time-to-exhaustion when possible.

Alert policy

QuestionRequired answer
What user harm does this signal predict or confirm?A named journey, scope, and time sensitivity.
Who receives it?A rotation or person who is actually accountable now.
What can they do?A linked runbook with first checks and a safe action.
When does it escalate?A timebox and backup owner.
When is it closed?A recovery condition, not just a quiet graph.

Baseline checklist

  • External uptime and certificate-expiry check.
  • Primary journey success and latency.
  • Error rate by meaningful operation.
  • Queue age, scheduled job freshness, and failed background work.
  • Database connection, storage, and backup freshness.
  • Vendor status or quota where a dependency can stop service.
  • Cost or usage anomaly where runaway consumption creates operational risk.
  • Alert delivery test to the real on-call destination.

Monitor record

Signal:
User journey / risk:
Query or check:
Normal range:
Warning condition:
Incident condition:
Evaluation window:
Owner / rotation:
Escalation after:
Runbook:
Recovery condition:
Last delivery test:
Review date:

Alert hygiene

  • Delete or downgrade alerts that never produce action.
  • Group symptoms of one failure so responders receive one incident, not fifty pages.
  • Review thresholds after incidents and material traffic changes.
  • Test notification delivery and escalation on a schedule.
  • Never put secrets, personal data, or customer content in alert payloads.

No one trusts the dashboards?
Hoyack can help rebuild monitoring around the real service path.

Discuss monitoring