Runbook 01 · Incident response

The first 15 minutes, under control

A compact incident kit for naming ownership, bounding impact, recording actions, and deciding when recovery is real.

RB–01 / v1.0Output: Incident log + status updatePrint-friendly

Use this when: a primary user journey is failing, a dependency is unstable, or the team is making changes faster than it is recording evidence.

0–5 minutes: stabilize the response

  • Name one incident lead with authority to sequence work.
  • Open one timestamped log in UTC and put the link where responders can find it.
  • Write the user-visible symptom in one sentence. Do not start with an internal theory.
  • Pause unrelated deploys, migrations, and configuration changes.
  • Pick an initial severity and publish the next update time.

5–10 minutes: bound the problem

  • Test the primary user journey from outside the production network.
  • Compare the last known-good state with recent deploys, flags, certificates, DNS, vendors, queues, and capacity.
  • Separate facts, hypotheses, and actions in the log.
  • Choose the safest reversible move: rollback, failover, feature disable, traffic reduction, or observation.

10–15 minutes: communicate and act

  • Assign an action owner and a separate communications owner when staffing allows.
  • State what is known, unknown, affected, and underway.
  • Make one meaningful change at a time. Record the command or control used and the result.
  • Define recovery checks before declaring recovery.

Severity ladder

LevelUse whenOperating posture
SEV 1Critical journey unavailable, data at risk, or broad security exposure suspected.Dedicated incident lead; freeze unrelated changes; continuous response; frequent updates.
SEV 2Material degradation, limited outage, or workaround exists.Named owner; timeboxed updates; protect against escalation.
SEV 3Low-impact defect or warning with no immediate user harm.Track in normal work; monitor; define escalation signal.

Copy-ready incident log

Incident:
Started (UTC):
Incident lead:
Severity:
User impact:
Last known good:
Recent changes:

TIME (UTC) | OBSERVATION / ACTION | OWNER | RESULT
-----------|----------------------|-------|-------
           |                      |       |

Next update:
Recovery checks:

Status update template

We are investigating [user-visible symptom] affecting [scope].
Current impact: [what users can/cannot do].
Known: [facts only].
Unknown: [open questions].
Action underway: [current step].
Next update by: [time and timezone].

Recovery gate

  • The primary user journey succeeds from outside the system.
  • Error rate, latency, queues, and capacity are stable for an agreed observation window.
  • Data integrity checks pass, including any delayed work or replayed messages.
  • Monitoring is active and the incident lead approves closure.
  • A follow-up owner and postmortem date are recorded.

Need another operator in the room?
Ask Hoyack for a service rescue review.

Email Hoyack