Tabletop: regional outage
An 80-minute exercise for a consumer service that runs on a hosting partner. A power failure at the partner's facility takes down one region, and what users see makes the outage look like lost money.
It tests declaring on user impact, telling a display error from data loss, and failing over without the partner's estimate. It also tests an attacker using the confusion, a status page that disagrees with the partner's, a regulator's questions, and naming the partner.
The organizations and every detail below are fictional.
Set up the room
Seat one person in each position: Incident Commander, Operations, Planning, Public Information, Liaison, Legal advisor, Finance/Admin and Intelligence/Investigations, plus one executive for the crisis team. Take the Incident Commander from the commander rotation, and the Operations Section Chief from site reliability engineering. A facilitator reads the injects and speaks for Carrow Cloud, and a note-taker records the time of every decision.
The injects compress about 37 hours into 64 minutes, one every eight minutes, followed by a 15-minute hot wash. Keep the "Look for" column away from the players.
The scenario
Lanternly runs a consumer payments and messaging app with 30 million users in the United States. Carrow Cloud hosts it in two regions, East and Central, under a dedicated hosting contract.
Lanternly has a commander rotation, a status page and a published severity scale. Carrow and Lanternly have never exercised together. Times are Eastern.
Injects
| Clock | Inject | Look for | Seats tested |
|---|---|---|---|
| 0:00 | Saturday 19:40. Error rates climb in Lanternly's East region. Payment confirmations stall, and the app shows a balance of $0.00 for some users. The on-call site reliability engineer pages the Incident Commander rotation. | The commander on call takes command out loud and declares SEV 1 on user impact. Operations and Communications Leads are named and the incident record is opened. Security is asked to check whether this is an attack. | Incident Commander, Operations, Communications |
| 0:08 | 19:55. Carrow Cloud's status page reports "a power event affecting one facility in the East region." Lanternly's support queue passes 4,000 tickets. Posts saying "Lanternly lost my money" start trending. | One person joins Carrow's support case and bridge and reports to Liaison. Operations checks whether stored balances are wrong or only shown wrong, before anyone says either. The first status page post says what users see, what to do and when the next update comes. | Liaison, Operations, Public Information Officer |
| 0:16 | 20:30. Operations confirms stored balances are intact. The balance service times out and the app falls back to showing zero. Failover to the Central region is possible, but Central has capacity for about 60 percent of peak load. | The Incident Commander decides on failover with load shedding on Lanternly's own evidence, without waiting for Carrow's estimate. The crisis team sets which features shed first. The status page now says balances are safe, because someone checked. | Incident Commander, Operations, crisis team |
| 0:24 | 21:15. The failover sets off a retry storm from mobile clients, and Central slows for everyone. Security sees a spike in failed logins from a botnet testing stolen passwords while users retry. | Operations throttles client retries. Credential-stuffing defense becomes its own group under Operations with its own supervisor. The incident is sized at Type 3 and a Planning Section Chief is named. | Operations, Intelligence/Investigations, Incident Commander |
| 0:32 | 23:00. Carrow's status page turns green: "Power restored." Lanternly is still degraded while queues drain and caches rebuild. An executive asks why Lanternly's status page still shows an incident. | Lanternly keeps its status open until its own measures show recovery. The Incident Commander briefs the executive with those measures and the next update time. The night shift is named and briefed. | Incident Commander, Public Information Officer, Planning |
| 0:40 | Sunday 08:30. A state financial regulator emails asking whether customer funds were at risk and wants a written account by Monday 17:00. A reporter asks whether Lanternly was hacked. | The Legal advisor owns the reply to the regulator. It is written from the incident record, keeps facts apart from assessments, and says what is still unknown. The reporter gets the same facts as the status page. | Legal advisor, Public Information Officer, Planning |
| 0:48 | 11:00. Carrow's preliminary report says a transfer switch failed during a utility outage. Carrow asks Lanternly to leave the cause out of its statements until Carrow's own statement at 15:00. | Liaison and the Public Information Officer agree timing with Carrow's communications lead as a joint information system. Lanternly's statement covers its own service and names the cause after 15:00. | Liaison, Public Information Officer, Incident Commander |
| 0:56 | Monday 09:00. Service has been normal since Sunday 02:00. The executive group asks whether Central should carry full load on its own, at an added $2.4 million a year. | The Incident Commander closes the incident and schedules the review. The capacity question goes to the executive group as a decision with options and the risk of each. Finance/Admin files for service credits and requests Carrow's post-incident report. | Incident Commander, Finance/Admin, executive group |
Hot wash
Start the hot wash at 1:05. Ask each question, record the answers, and let the note-taker's decision times settle disagreements.
| # | Item |
|---|---|
| 1 | How long from the first alert to a declared SEV 1 with a named Incident Commander? |
| 2 | When did the team know stored balances were safe, and how long before the status page said so? |
| 3 | Who decided to fail over, and on what evidence? |
| 4 | Did the credential-stuffing work stay coordinated with the failover? |
| 5 | Why did Lanternly recover after Carrow did, and did the executives understand why? |
| 6 | Did the regulator, the reporter and users hear the same facts? |
| 7 | Who agreed when to name Carrow, and was the agreement kept? |
Write the findings into the after-action report and improvement plan within a week, with an owner and a date on every action.