ICS
1Prepare

Running an incident management program

An incident management program keeps the structure ready between incidents. It sets one severity scale for every kind of incident and keeps a trained Incident Commander on call.

It fixes when each audience hears from the response, reviews every serious incident, and reports the numbers to leadership. Prepare covers people, authority and tools. This page covers the standing parts around them.

One severity scale

Outages, security incidents, privacy events, enforcement failures and safety events compete for the same executives, lawyers and engineers. Rate them on one scale so that a SEV 1 means the same thing to everyone. Rate each row that applies and take the highest.

ImpactSEV 1SEV 2SEV 3SEV 4
ServiceCore service down or unusable for most usersA major feature down, or core service degraded for many usersDegraded for some users, with a workaroundNo user impact
Data, security or privacyConfirmed access to regulated or restricted data by someone without approvalSuspected access to regulated or restricted data, or confirmed access to internal dataOne account or host compromised and containedA blocked attempt, or a policy breach with no data exposed
Enforcement: content, fraud or policyEnforcement failing at scale, or applied wrongly to many usersEnforcement failing for one feature or regionOne enforcement error with limited reachA single case
SafetyAny risk to life or physical safetyA physical process kept safe only by manual controlA safety control degraded, with a backup in placeA safety alarm found to be false
Legal and publicA notification deadline has started, or national media coverageA regulator or government body has asked questions, or trade press coverageCustomer complaints about the issueNone expected

Anyone may declare an incident and propose a severity. The Incident Commander confirms or changes it within 15 minutes, and anyone may ask for a higher one.

Severity sets how fast and how high the incident is reported. Complexity type sets how large the organization grows. Sizing the response has the reporting deadlines for each level and the rules for choosing the type.

The Incident Commander rotation

Put qualified Incident Commanders on a paging rotation that covers every hour, separate from the engineering and security on-call rotations. The commander on call takes every SEV 1 and SEV 2, whatever kind of incident it is, and gives the technical lead to the team that owns the problem.

SeatCoversPaged for
Primary Incident CommanderTakes command, sets objectives, owns the update scheduleEvery SEV 1 and SEV 2
Secondary Incident CommanderBacks up the primary, takes over at shift change or when two incidents run at onceSEV 1, or any incident when the primary is committed
Communications LeadDrafts updates for every audience from the commander's factsEvery SEV 1 and SEV 2
Executive on callMakes the decisions above the commander's delegated authorityEvery SEV 1, and a SEV 2 when a decision needs them
Legal on callNotification deadlines, privilege, contact with regulatorsAny incident touching regulated data, regulators or litigation

To join the rotation, a commander completes FEMA's IS-100 and IS-200 or an internal equivalent, shadows two incidents, and commands one tabletop while a qualified commander watches. Size the rotation with the staffing math.

Communications standards

Publish the update schedule before the incident, so nobody negotiates it during one. Each update says what happened, the impact now, what is being done, what the reader needs to do or decide, and the time of the next update. Mark each fact as confirmed, probable or unknown.

AudienceSEV 1SEV 2Channel and approver
ExecutivesFirst by the severity deadline, then every 30 minutes until mitigated, then every 2 hoursFirst by the severity deadline, then every hour until mitigated, then every 4 hoursIncident channel or email, approved by the Incident Commander
BoardThe same day, then dailyAt the next scheduled update, unless the executive on call decides soonerThrough the executive sponsor, reviewed by counsel
Regulators and governmentBy the deadline in each obligationBy the deadline in each obligationThrough counsel. See notification deadlines
UsersStatus page within 30 minutes when users are affected, then every hourStatus page within an hour when users are affectedPublic Information Officer, cleared by counsel when data is involved
EmployeesWithin the hour, then with each executive updateWhen their work is affectedInternal channel, from the Public Information Officer
Partners and suppliersWhen they are affected or involvedWhen they are affected or involvedLiaison

Public statements say what users see and what they should do. A cause goes public only once it is confirmed. A statement that data is safe goes out only after someone has checked the stored data.

When a supplier caused the failure, see when a partner runs your infrastructure before naming them.

Board and government reporting

Reports to a board or a government body carry weight and get reread later, sometimes by people looking for inconsistencies. Write them from the incident record and keep one version of the facts.

  • Start from the latest status summary (ICS 209). Nothing goes in that the record does not support.
  • Keep facts apart from assessments, and give the confidence and the evidence for each assessment.
  • State what is still unknown and when you expect to know it.
  • Counsel reviews each report before release. Record who approved it and when.
  • Keep every version sent, with its recipients, in the incident record. When a fact changes, the next report says what changed and why.

Incident review board

A standing review board keeps reviews honest and actions moving. It meets every week, accepts each SEV 1 and SEV 2 review before publication, and chases overdue actions.

ForumHow oftenWhoDecides
Incident review boardWeeklyProgram lead as chair, the commander rotation, service owners, security, and legal when neededAccepts reviews, assigns actions, escalates overdue ones
Leadership reviewMonthlyExecutive sponsor and the engineering, security, legal and communications leadersFunds larger fixes, accepts risks that will stay unfixed
Board committeeQuarterlyThe board committee that oversees risk or securityHears trends, major incidents and overdue actions

Reviews follow Learn. A review is due within five business days of resolution for a SEV 1 and 10 for a SEV 2.

Near misses

Record near misses the same way as incidents. A near miss is an alert that fired late and was caught by luck, a change rolled back minutes before it spread, or a failover that worked because someone happened to be awake. They cost nothing and expose the same weaknesses as incidents.

Review them at the weekly board and count them in the metrics.

Metrics

Measure every incident the same way so that incidents can be compared. Report the median and the worst case for each severity every month.

MetricMeasured asTells you
Time to declareFirst signal to declarationWhether detection reaches someone who acts
Time to commandDeclaration to a named Incident CommanderWhether the rotation answers
Time to first executive updateDeclaration to the first update sentWhether the update schedule works
Updates on timeShare of scheduled updates sent by their stated timeWhether audiences can trust the schedule
Time to mitigateDeclaration to the end of user impactHow long users felt it
Time to resolveDeclaration to closing the incidentHow long the organization carried it
Reviews on timeShare of reviews published by their due dateWhether learning keeps pace
Actions closed on timeShare of corrective actions closed by their due dateWhether fixes happen
Repeat incidentsIncidents whose review cites an earlier action left openWhat open actions cost
Near misses recordedCount per monthWhether people report without fear

The PIVTR-D metrics cover the technical response.

First 90 days for a new program lead

ByDoDeliver
Day 30Meet the leads of every team that responds or gets informed. Read the last 10 serious incident reviews. Shadow the commander rotation. Map the current severity scales, paging paths, tools and reporting duties.A one-page current state with its gaps, agreed with the executive sponsor
Day 60Take commander shifts. Run a tabletop that crosses two kinds of incident, such as the regional outage, where an outage looks like lost money. Publish the update schedule and a report checklist agreed with counsel.The tabletop after-action report. The schedule and checklist in use
Day 90Propose one severity scale where the current ones disagree. Stand up the review board. Report the first metrics.The severity proposal and the first monthly metrics report

Built from Google's Incident Management Guide, the Site Reliability Engineering chapters on managing incidents and postmortem culture, NIMS, and the HSEEP after-action process. The severity rows, intervals and deadlines on this page are starting values. Set your own and publish them.

Adapted from the National Incident Management System, Third Edition (FEMA, 2017). Cyber adaptation, examples and templates by Habib Tora, licensed under CC BY 4.0. Not published by, endorsed by, or affiliated with FEMA or the Department of Homeland Security. Structure after Google's SRE Incident Management Guide. Companion to PIVTR-D.