ICS
2Respond and manage

When a partner runs your infrastructure

When a cloud provider, hosting company or managed service provider runs the infrastructure that failed, two incidents run at once. The partner commands its response. You command yours: your users, your data, your executives and your regulators.

The job is keeping the two joined so neither waits on the other.

Choose the structure

Partner relationshipStructurePartner's place
Public cloud shared with many customersSeparate commandsOutside your organization. A Technical Specialist tracks the provider's status page and support case and reports to Planning
Dedicated hosting, or a managed service that runs your systemsAssisting organization (NIMS: assisting agency)Inside Operations. The partner's lead runs a group under your Operations Section Chief for work in your environment
A partner with authority over part of the response, by contract or regulationUnified commandA member of unified command, agreeing objectives with you

Decide the structure for each major partner in advance and write it into the contract and the communications plan. Whatever the structure, Liaison owns the relationship and keeps the partner's contacts, commitments and requests in one place.

Before the incident

#ItemOwner
1Escalation contacts named in both directions, with backups, and tested each quarterLiaison
2Your severity levels mapped to the partner's support severities, so a SEV 1 opens their highest caseLiaison
3Contract sets how fast the partner notifies you, what the notice contains, and how often it updatesLegal advisor
4Logs and telemetry the partner shares during an incident listed, with how fast you receive themOperations
5Actions the partner may take in your environment on its own authority listed, and the ones that need your Operations Section ChiefIncident Commander
6Agreement on who may name the partner, or the partner's cause, in public statementsPublic Information Officer
7The partner's post-incident report promised by a set number of days after resolutionLegal advisor
8Failover you control tested without the partner's helpOperations
9A joint exercise each year, such as the regional outage tabletopPlanning

First hour when the partner is the cause

#ItemOwner
1Highest-severity support case opened, with the case number in the activity logOperations
2One person assigned to the partner's bridge or case, reporting to Liaison, so the Incident Commander hears one summaryLiaison
3Four facts asked of the partner: scope, start time, current state, and the time of the next updateLiaison
4Failover and load shedding decided on your own evidence, without waiting for the partner's estimateIncident Commander
5What users see checked against what is stored. A zero count, a missing balance or a lost message may be a display fallback over intact data. Verify before saying eitherOperations
6Security checks whether the failure is an attack, and keeps watching after ruling it out, because attackers use outages as coverIntelligence/Investigations
7Notification deadlines checked, since loss of availability of regulated services or data can start oneLegal advisor

One set of facts

Users see you as responsible, whoever caused the failure. Tell them what they see, what to do and when you will update them.

Name the partner, or its cause, only once the partner has published it or agreed in writing. Until then, "a failure at our infrastructure provider" is accurate and fair. When both organizations speak to the same audiences, run a joint information system with the partner's communications lead.

The partner's status page tracks the partner's service. Yours may stay broken after theirs recovers, while queues drain, caches rebuild and client retries settle. Keep your own status open until your own measures show recovery.

After the incident

  • Ask for the partner's post-incident report by the date the contract sets, and check it against your timeline.
  • Run a joint review for any SEV 1. Each side also keeps its own blameless review.
  • Put actions that depend on the partner into your improvement plan, each with an owner on your side.
  • Finance/Admin claims service credits and records costs for any insurance claim.
  • Revisit the structure. A partner you kept outside may need a seat next time.
Watch out
  • Waiting for the partner's estimate before deciding on failover. Their estimate covers their service, and your recovery can take longer.
  • Repeating a cause the partner has not confirmed. Early causes change.
  • Letting several engineers join the partner's bridge on their own. One person reports what the partner says.

Built from the NIMS guidance on unified command, assisting agencies and the joint information system, and from Google's Incident Management Guide.

Adapted from the National Incident Management System, Third Edition (FEMA, 2017). Cyber adaptation, examples and templates by Habib Tora, licensed under CC BY 4.0. Not published by, endorsed by, or affiliated with FEMA or the Department of Homeland Security. Structure after Google's SRE Incident Management Guide. Companion to PIVTR-D.