Running an incident management program
An incident management program keeps the structure ready between incidents. It sets one severity scale for every kind of incident and keeps a trained Incident Commander on call.
It fixes when each audience hears from the response, reviews every serious incident, and reports the numbers to leadership. Prepare covers people, authority and tools. This page covers the standing parts around them.
One severity scale
Outages, security incidents, privacy events, enforcement failures and safety events compete for the same executives, lawyers and engineers. Rate them on one scale so that a SEV 1 means the same thing to everyone. Rate each row that applies and take the highest.
| Impact | SEV 1 | SEV 2 | SEV 3 | SEV 4 |
|---|---|---|---|---|
| Service | Core service down or unusable for most users | A major feature down, or core service degraded for many users | Degraded for some users, with a workaround | No user impact |
| Data, security or privacy | Confirmed access to regulated or restricted data by someone without approval | Suspected access to regulated or restricted data, or confirmed access to internal data | One account or host compromised and contained | A blocked attempt, or a policy breach with no data exposed |
| Enforcement: content, fraud or policy | Enforcement failing at scale, or applied wrongly to many users | Enforcement failing for one feature or region | One enforcement error with limited reach | A single case |
| Safety | Any risk to life or physical safety | A physical process kept safe only by manual control | A safety control degraded, with a backup in place | A safety alarm found to be false |
| Legal and public | A notification deadline has started, or national media coverage | A regulator or government body has asked questions, or trade press coverage | Customer complaints about the issue | None expected |
Anyone may declare an incident and propose a severity. The Incident Commander confirms or changes it within 15 minutes, and anyone may ask for a higher one.
Severity sets how fast and how high the incident is reported. Complexity type sets how large the organization grows. Sizing the response has the reporting deadlines for each level and the rules for choosing the type.
The Incident Commander rotation
Put qualified Incident Commanders on a paging rotation that covers every hour, separate from the engineering and security on-call rotations. The commander on call takes every SEV 1 and SEV 2, whatever kind of incident it is, and gives the technical lead to the team that owns the problem.
| Seat | Covers | Paged for |
|---|---|---|
| Primary Incident Commander | Takes command, sets objectives, owns the update schedule | Every SEV 1 and SEV 2 |
| Secondary Incident Commander | Backs up the primary, takes over at shift change or when two incidents run at once | SEV 1, or any incident when the primary is committed |
| Communications Lead | Drafts updates for every audience from the commander's facts | Every SEV 1 and SEV 2 |
| Executive on call | Makes the decisions above the commander's delegated authority | Every SEV 1, and a SEV 2 when a decision needs them |
| Legal on call | Notification deadlines, privilege, contact with regulators | Any incident touching regulated data, regulators or litigation |
To join the rotation, a commander completes FEMA's IS-100 and IS-200 or an internal equivalent, shadows two incidents, and commands one tabletop while a qualified commander watches. Size the rotation with the staffing math.
Communications standards
Publish the update schedule before the incident, so nobody negotiates it during one. Each update says what happened, the impact now, what is being done, what the reader needs to do or decide, and the time of the next update. Mark each fact as confirmed, probable or unknown.
| Audience | SEV 1 | SEV 2 | Channel and approver |
|---|---|---|---|
| Executives | First by the severity deadline, then every 30 minutes until mitigated, then every 2 hours | First by the severity deadline, then every hour until mitigated, then every 4 hours | Incident channel or email, approved by the Incident Commander |
| Board | The same day, then daily | At the next scheduled update, unless the executive on call decides sooner | Through the executive sponsor, reviewed by counsel |
| Regulators and government | By the deadline in each obligation | By the deadline in each obligation | Through counsel. See notification deadlines |
| Users | Status page within 30 minutes when users are affected, then every hour | Status page within an hour when users are affected | Public Information Officer, cleared by counsel when data is involved |
| Employees | Within the hour, then with each executive update | When their work is affected | Internal channel, from the Public Information Officer |
| Partners and suppliers | When they are affected or involved | When they are affected or involved | Liaison |
Public statements say what users see and what they should do. A cause goes public only once it is confirmed. A statement that data is safe goes out only after someone has checked the stored data.
When a supplier caused the failure, see when a partner runs your infrastructure before naming them.
Board and government reporting
Reports to a board or a government body carry weight and get reread later, sometimes by people looking for inconsistencies. Write them from the incident record and keep one version of the facts.
- Start from the latest status summary (ICS 209). Nothing goes in that the record does not support.
- Keep facts apart from assessments, and give the confidence and the evidence for each assessment.
- State what is still unknown and when you expect to know it.
- Counsel reviews each report before release. Record who approved it and when.
- Keep every version sent, with its recipients, in the incident record. When a fact changes, the next report says what changed and why.
Incident review board
A standing review board keeps reviews honest and actions moving. It meets every week, accepts each SEV 1 and SEV 2 review before publication, and chases overdue actions.
| Forum | How often | Who | Decides |
|---|---|---|---|
| Incident review board | Weekly | Program lead as chair, the commander rotation, service owners, security, and legal when needed | Accepts reviews, assigns actions, escalates overdue ones |
| Leadership review | Monthly | Executive sponsor and the engineering, security, legal and communications leaders | Funds larger fixes, accepts risks that will stay unfixed |
| Board committee | Quarterly | The board committee that oversees risk or security | Hears trends, major incidents and overdue actions |
Reviews follow Learn. A review is due within five business days of resolution for a SEV 1 and 10 for a SEV 2.
Near misses
Record near misses the same way as incidents. A near miss is an alert that fired late and was caught by luck, a change rolled back minutes before it spread, or a failover that worked because someone happened to be awake. They cost nothing and expose the same weaknesses as incidents.
Review them at the weekly board and count them in the metrics.
Metrics
Measure every incident the same way so that incidents can be compared. Report the median and the worst case for each severity every month.
| Metric | Measured as | Tells you |
|---|---|---|
| Time to declare | First signal to declaration | Whether detection reaches someone who acts |
| Time to command | Declaration to a named Incident Commander | Whether the rotation answers |
| Time to first executive update | Declaration to the first update sent | Whether the update schedule works |
| Updates on time | Share of scheduled updates sent by their stated time | Whether audiences can trust the schedule |
| Time to mitigate | Declaration to the end of user impact | How long users felt it |
| Time to resolve | Declaration to closing the incident | How long the organization carried it |
| Reviews on time | Share of reviews published by their due date | Whether learning keeps pace |
| Actions closed on time | Share of corrective actions closed by their due date | Whether fixes happen |
| Repeat incidents | Incidents whose review cites an earlier action left open | What open actions cost |
| Near misses recorded | Count per month | Whether people report without fear |
The PIVTR-D metrics cover the technical response.
First 90 days for a new program lead
| By | Do | Deliver |
|---|---|---|
| Day 30 | Meet the leads of every team that responds or gets informed. Read the last 10 serious incident reviews. Shadow the commander rotation. Map the current severity scales, paging paths, tools and reporting duties. | A one-page current state with its gaps, agreed with the executive sponsor |
| Day 60 | Take commander shifts. Run a tabletop that crosses two kinds of incident, such as the regional outage, where an outage looks like lost money. Publish the update schedule and a report checklist agreed with counsel. | The tabletop after-action report. The schedule and checklist in use |
| Day 90 | Propose one severity scale where the current ones disagree. Stand up the review board. Report the first metrics. | The severity proposal and the first monthly metrics report |
Built from Google's Incident Management Guide, the Site Reliability Engineering chapters on managing incidents and postmortem culture, NIMS, and the HSEEP after-action process. The severity rows, intervals and deadlines on this page are starting values. Set your own and publish them.