Runbook 01 · Incident response
The first 15 minutes, under control
A compact incident kit for naming ownership, bounding impact, recording actions, and deciding when recovery is real.
Use this when: a primary user journey is failing, a dependency is unstable, or the team is making changes faster than it is recording evidence.
0–5 minutes: stabilize the response
- Name one incident lead with authority to sequence work.
- Open one timestamped log in UTC and put the link where responders can find it.
- Write the user-visible symptom in one sentence. Do not start with an internal theory.
- Pause unrelated deploys, migrations, and configuration changes.
- Pick an initial severity and publish the next update time.
5–10 minutes: bound the problem
- Test the primary user journey from outside the production network.
- Compare the last known-good state with recent deploys, flags, certificates, DNS, vendors, queues, and capacity.
- Separate facts, hypotheses, and actions in the log.
- Choose the safest reversible move: rollback, failover, feature disable, traffic reduction, or observation.
10–15 minutes: communicate and act
- Assign an action owner and a separate communications owner when staffing allows.
- State what is known, unknown, affected, and underway.
- Make one meaningful change at a time. Record the command or control used and the result.
- Define recovery checks before declaring recovery.
Severity ladder
| Level | Use when | Operating posture |
|---|---|---|
| SEV 1 | Critical journey unavailable, data at risk, or broad security exposure suspected. | Dedicated incident lead; freeze unrelated changes; continuous response; frequent updates. |
| SEV 2 | Material degradation, limited outage, or workaround exists. | Named owner; timeboxed updates; protect against escalation. |
| SEV 3 | Low-impact defect or warning with no immediate user harm. | Track in normal work; monitor; define escalation signal. |
Copy-ready incident log
Incident:
Started (UTC):
Incident lead:
Severity:
User impact:
Last known good:
Recent changes:
TIME (UTC) | OBSERVATION / ACTION | OWNER | RESULT
-----------|----------------------|-------|-------
| | |
Next update:
Recovery checks:
Status update template
We are investigating [user-visible symptom] affecting [scope]. Current impact: [what users can/cannot do]. Known: [facts only]. Unknown: [open questions]. Action underway: [current step]. Next update by: [time and timezone].
Recovery gate
- The primary user journey succeeds from outside the system.
- Error rate, latency, queues, and capacity are stable for an agreed observation window.
- Data integrity checks pass, including any delayed work or replayed messages.
- Monitoring is active and the incident lead approves closure.
- A follow-up owner and postmortem date are recorded.
Need another operator in the room?
Ask Hoyack for a service rescue review.