When a partner runs your infrastructure
When a cloud provider, hosting company or managed service provider runs the infrastructure that failed, two incidents run at once. The partner commands its response. You command yours: your users, your data, your executives and your regulators.
The job is keeping the two joined so neither waits on the other.
Choose the structure
| Partner relationship | Structure | Partner's place |
|---|---|---|
| Public cloud shared with many customers | Separate commands | Outside your organization. A Technical Specialist tracks the provider's status page and support case and reports to Planning |
| Dedicated hosting, or a managed service that runs your systems | Assisting organization (NIMS: assisting agency) | Inside Operations. The partner's lead runs a group under your Operations Section Chief for work in your environment |
| A partner with authority over part of the response, by contract or regulation | Unified command | A member of unified command, agreeing objectives with you |
Decide the structure for each major partner in advance and write it into the contract and the communications plan. Whatever the structure, Liaison owns the relationship and keeps the partner's contacts, commitments and requests in one place.
Before the incident
| # | Item | Owner |
|---|---|---|
| 1 | Escalation contacts named in both directions, with backups, and tested each quarter | Liaison |
| 2 | Your severity levels mapped to the partner's support severities, so a SEV 1 opens their highest case | Liaison |
| 3 | Contract sets how fast the partner notifies you, what the notice contains, and how often it updates | Legal advisor |
| 4 | Logs and telemetry the partner shares during an incident listed, with how fast you receive them | Operations |
| 5 | Actions the partner may take in your environment on its own authority listed, and the ones that need your Operations Section Chief | Incident Commander |
| 6 | Agreement on who may name the partner, or the partner's cause, in public statements | Public Information Officer |
| 7 | The partner's post-incident report promised by a set number of days after resolution | Legal advisor |
| 8 | Failover you control tested without the partner's help | Operations |
| 9 | A joint exercise each year, such as the regional outage tabletop | Planning |
First hour when the partner is the cause
| # | Item | Owner |
|---|---|---|
| 1 | Highest-severity support case opened, with the case number in the activity log | Operations |
| 2 | One person assigned to the partner's bridge or case, reporting to Liaison, so the Incident Commander hears one summary | Liaison |
| 3 | Four facts asked of the partner: scope, start time, current state, and the time of the next update | Liaison |
| 4 | Failover and load shedding decided on your own evidence, without waiting for the partner's estimate | Incident Commander |
| 5 | What users see checked against what is stored. A zero count, a missing balance or a lost message may be a display fallback over intact data. Verify before saying either | Operations |
| 6 | Security checks whether the failure is an attack, and keeps watching after ruling it out, because attackers use outages as cover | Intelligence/Investigations |
| 7 | Notification deadlines checked, since loss of availability of regulated services or data can start one | Legal advisor |
One set of facts
Users see you as responsible, whoever caused the failure. Tell them what they see, what to do and when you will update them.
Name the partner, or its cause, only once the partner has published it or agreed in writing. Until then, "a failure at our infrastructure provider" is accurate and fair. When both organizations speak to the same audiences, run a joint information system with the partner's communications lead.
The partner's status page tracks the partner's service. Yours may stay broken after theirs recovers, while queues drain, caches rebuild and client retries settle. Keep your own status open until your own measures show recovery.
After the incident
- Ask for the partner's post-incident report by the date the contract sets, and check it against your timeline.
- Run a joint review for any SEV 1. Each side also keeps its own blameless review.
- Put actions that depend on the partner into your improvement plan, each with an owner on your side.
- Finance/Admin claims service credits and records costs for any insurance claim.
- Revisit the structure. A partner you kept outside may need a seat next time.
- Waiting for the partner's estimate before deciding on failover. Their estimate covers their service, and your recovery can take longer.
- Repeating a cause the partner has not confirmed. Early causes change.
- Letting several engineers join the partner's bridge on their own. One person reports what the partner says.
Built from the NIMS guidance on unified command, assisting agencies and the joint information system, and from Google's Incident Management Guide.