Tabletop: data center flood
An 80-minute exercise with no attacker in it. A burst water main floods a data center and runs for a day, and the same structure that runs a ransomware response runs this one.
It tests taking command, safety authority, sizing, the crisis team, spending above delegated limits, public information and handover. The organization and every detail below are fictional.
Set up the room
Seat one person in each position: Incident Commander, Operations, Planning, Logistics, Public Information, Liaison, Finance/Admin and Safety, plus one executive for the crisis team and one who delegated the Incident Commander's authority. Choose the Incident Commander from facilities, to show that command goes to whoever is qualified for the incident in front of them. A facilitator reads the injects and a note-taker records the time of every decision.
The injects compress about 24 hours into 64 minutes, one every eight minutes, followed by a 15-minute hot wash. Keep the "Look for" column away from the players.
The scenario
Northgate Supply, the distributor from the ransomware tabletop, runs its primary data center in two halls on the lower level of a leased building in Chicago, with a secondary site in Dallas. Order entry and warehouse management run in hall B.
Facilities has an on-call engineer, and IT has an on-call rotation. Neither has worked an incident with the other.
Injects
| Clock | Inject | Look for | Seats tested |
|---|---|---|---|
| 0:00 | Tuesday 05:40. The facilities on-call engineer gets a water alarm from the lower level of the Chicago data center building. Temperature sensors on two racks in hall B go offline. | The facilities engineer takes command out loud, opens an activity log and calls the IT on-call. Nobody enters standing water near powered equipment. The ICS 201 is started. | Incident Commander, Safety Officer |
| 0:08 | 06:10. Water is 10 centimeters deep in hall B. The building engineer wants to cut power to the hall to protect staff. Order entry and warehouse management run there. | The power cut goes ahead, because people come first and the Safety Officer can stop unsafe work. The IC names an Operations Lead from IT to fail systems over to Dallas. The incident is sized at Type 4. | Incident Commander, Safety Officer, Operations |
| 0:16 | 07:00. Order entry fails over to Dallas. Its database does not, and shipping will stop when the day shift starts at 08:00. | Up to three objectives are written, each with a measure and a time. The crisis team stands up to run manual shipping. The business, through the crisis team, sets which system comes back first. | Incident Commander, crisis team, Operations |
| 0:24 | 08:30. The city says the water main repair will take 36 hours. Building management says hall B stays without power until it is dry and inspected, which will not be before Wednesday morning. | The incident is re-sized to Type 3, because it will run past one operational period. A Planning Section Chief is named and the planning cycle starts. Logistics orders pumps, dehumidifiers and temporary cooling. Finance/Admin opens a cost code. | Incident Commander, Planning, Logistics, Finance/Admin |
| 0:32 | 10:00. The property insurer and the landlord both want to inspect hall B. A vendor offers emergency rack relocation for $180,000. | Liaison handles the insurer and the landlord. The spend is above the IC's delegated limit, so it goes to the executive who delegated authority, with options and the cost of waiting. | Liaison, Finance/Admin, Incident Commander |
| 0:40 | 12:00. A local news crew films water pouring from the building. Customers post that their orders are not being confirmed. | A Public Information Officer is named. The first ICS 209 is issued and the interim statement and customer notice are drawn from it. Staff are told where to send questions from outside. | Public Information Officer, Planning, crisis team |
| 0:48 | 15:00. A contractor moving equipment in hall A gets a mild shock from a wet power strip. | The Safety Officer stops work in hall A. The contractor gets first aid and a medical check. The hazard is isolated before work resumes, and the safety message in the IAP is rewritten. | Safety Officer, Logistics, Incident Commander |
| 0:56 | Wednesday 06:00. Hall B is dry and power is back. The servers need inspection before restart. Several responders have worked since Tuesday morning. | Servers restart only after inspection and owner sign-off. The Safety Officer names everyone past the shift limit and relief is rostered. The demobilization plan starts, with check-out for the contractors. | Operations, Safety Officer, Planning |
Hot wash
Start the hot wash at 1:05. Ask each question, record the answers, and let the note-taker's decision times settle disagreements.
| # | Item |
|---|---|
| 1 | How long from the water alarm to someone taking command out loud? |
| 2 | Who decided to cut power, and was the Safety Officer's authority clear? |
| 3 | When was the incident re-sized to Type 3, and what triggered it? |
| 4 | Did the crisis team or incident command decide the restore order, and was that the right place? |
| 5 | How did the $180,000 decision reach someone with authority, and how long did it take? |
| 6 | Did the news crew, customers and staff hear the same facts? |
| 7 | Who worked past the shift limit, and who noticed? |
Write the findings into the after-action report and improvement plan within a week, with an owner and a date on every action.