Runbook: [Service name]
What it does | |
Who notices first | |
Dashboard | |
Logs | |
Recent deploys | |
Repository | |
Depends on | |
Depended on by | |
Service level | |
Incident channel | |
Owning team | |
Last reviewed |
First five minutes
Acknowledge the page, so it doesn't escalate.
Look at the dashboard: what changed, and when did it start?
Check for a recent change near that time: a deploy, a config change, a feature flag.
Set a severity from the table below. When unsure, choose the higher one.
Say you're on it in the incident channel: what fired, the severity, and that you're investigating.
Severity | What it looks like for this service | Response |
SEV1 | ||
SEV2 | ||
SEV3 |
Never do these alone
Get the incident commander on the call first, even at 3 a.m.:
Restore from a backup
Fail over the primary database
Delete, purge or replay messages
Anything else that can't be undone in a few minutes
Alerts
One entry per alert, named exactly as it pages. Copy down to the divider.
See an example alert entry
Alert: checkout-api 5xx rate above 2% for 5 minutes
Means: More than 2 in 100 checkout requests fail on the server.
Impact: Customers see an error at payment, and the order is lost.
Severity: SEV2; SEV1 above 20%, or in every region at once.
Graph: Checkout dashboard › Errors by route
1. Check for a deploy in the last hour: deployctl history checkout-api --since 1h. Normal: none, or one that finished more than an hour ago.
2. If a deploy lines up with the spike, roll it back (see Procedures). Errors should fall below 0.5% within 5 minutes.
3. If the errors are all TaxVendorTimeout, switch tax to cached rates: flags set tax.cached_rates on. Errors should clear within 2 minutes.
4. Not resolved after 15 minutes: page the secondary and post what you tried in #inc-checkout.
[Alert name]
Means | |
Impact | |
Severity | |
Graph |
Check:
kubectl get pods -n <namespace> -l app=<service>Normal looks like:
If it's [likely cause]:
If it's [another cause]:
Not fixed in [N] minutes: escalate (see Escalation) and post what you tried in the incident channel.
Procedures
The commands are Kubernetes examples: use your own.
Roll back a deploy
Find the last good version:
kubectl rollout history deployment/<service> -n <namespace>Roll back to it:
kubectl rollout undo deployment/<service> -n <namespace> --to-revision=<revision>Check that errors and latency are back to normal on the dashboard.
Post in the incident channel what you rolled back and why.
[Procedure name]
Restart, scale up, fail over or restore from a backup.
Before you start:
Run:
kubectl scale deployment/<service> -n <namespace> --replicas=<count>Check:
To undo:
Escalation
When | Who | How |
Page not acknowledged in [N] minutes | ||
Not mitigated in [N] minutes, or SEV1 | ||
A dependency is at fault | ||
A vendor is down | ||
Customers are affected |
After the incident
Log what you ran, and when, in the incident channel or ticket
Fix any step in this runbook that was wrong, missing or slow
Open a postmortem for SEV1 and SEV2, or as your team's policy says
Give every follow-up action one owner and a due date
Revision history
Date | Author | Change |
First version | ||