A postmortem, or post-incident review, captures the key information around an incident, its root cause(s), cost and follow-up. Below is a suggested set of sections and supporting material, together with some best practice to follow. This template is aimed at IT, datacentre and buildings functions, but may be applied more widely.
As mentioned earlier, the quality of your telemetry will determine the quality of the resulting review. Ideally you will build your incident timeline from telemetry, rather than trying to piece it back together from memory.
PODTECH
Turn Infrastructure Data Into Insight
PODTECH builds bespoke technology solutions for critical infrastructure. These range from building monitoring to advanced analytics to enable informed, strategic decisions.
VisitTL;DR
- Apply this process to all incidents that meet or exceed a defined level of severity. You'll need to capture key details, write a high-level description, quantify the effect, establish a timeline, identify contributing factors, highlight positives, and define actions with an assignee, target date and criteria for completion.
- Adopt a blame-free culture. Your aim is to describe the series of events and conditions that led to the incident, but avoid singling out individuals or groups. See Google's SRE book.
- Complete the postmortem within 5 business days of the incident being fully remediated. You should prepare the document ahead of the postmortem meeting.
- Expect multiple contributing factors and ensure there is one designated assignee for each remediation.
- For datacentre or buildings incidents, consider additional elements around health and safety, facility impacts and third-party dependencies. You should also compare actual time-to-recovery to your RTO.
Table of Contents
- What a post-incident review is for
- When to run one
- The incident postmortem template
- How to run the review meeting
- Adapting the template to datacentre, building and safety events
- Worked example: a cooling alarm that nobody saw
- Turning findings into closed actions
- From postmortems to prevention
- Perspective
- Put your findings into a system that closes them
- FAQ
- Sources
What a post-incident review is for
There are three main objectives of conducting a postmortem: to record the incident, ensure the relevant contributing factors have been identified, and drive remediation activity. A secondary consideration is to demonstrate to your customer(s) and senior leadership that the issue has been fully diagnosed and addressed.
In the context of a critical infrastructure environment, the review should take place after the business is no longer impacted by the incident. The CRR guidance from CISA, with the SEI at CMU, recommends that the review includes those involved in the incident response and recovery, as well as any customers or parties impacted. Finally, it should be used to formally close off the incident.
Of course, if you are able to revert to a previous state or switch to a standby system, you will have restored the impacted service(s). But this will provide no insight as to how the original degradation occurred. That's what the postmortem is intended to address.
When to run one
You will need to decide on a minimum level of incident that warrants a postmortem, and consistently apply that. Typically you might look to conduct a detailed postmortem on Level 1+ incidents, and a reduced version for lower levels. This decision should be made prior to any incident occurring, otherwise there is a tendency for all involved to want to waive the requirement.
An incident should trigger a postmortem if:
- One or more customers experienced an outage, or the incident resulted in the RTO being exceeded.
- An individual suffered harm, or came close to doing so. Alternatively, any other health and safety equipment triggered, or failed to trigger.
- Any element of datacentre cooling, power supply or associated monitoring was compromised, irrespective of whether the customer was aware.
- The incident was only contained due to chance or the actions of a small number of people. Near-misses can be highly informative.
The incident postmortem template
You can use the following as a basis for your own document. Further detail is given in the notes below each field.
INCIDENT POSTMORTEM
- Brief overview (< 3 lines)
- Effect of incident (quantified, or "unable to determine")
- How incident was identified (e.g. alert received, customer escalation)
- Timeline of incident
- Initial trigger
- Contributing factors
- Remedial actions taken to contain and resolve the incident
- Positive observations
- Areas for improvement
- Elements that prevented greater impact
- Follow-on actions required (assignee, due date, priority, reference and definition of complete)
- Relevant artifacts (log links, graphs, Slack threads, video clips)
Key details and overview
At the outset, capture the above fields to facilitate searchability of the postmortem. Add the duration of the incident, from initial point of impact through to the final point of restoration. Follow this with an overview of no more than two to three lines, which could be digested by a Director in roughly 30 seconds. Describe the nature of the failure, its length and the extent of impact. This can be drafted towards the end of the exercise.
Effect of incident (quantified, or "unable to determine")
Vagueness will undermine the whole exercise. Be specific and note the most severe category impacted, services, systems and sites, customers and tenants, number of alerts and events lost, SLAs impacted and tickets created. Where the answer isn't readily apparent, state "unknown" and nominate an individual to return with the value. Leaving it blank encourages others to make an assumption.
In a datacentre environment, consider also the number of server racks and data hall(s) impacted, deviations from expected environmental conditions such as temperature or voltage, and whether any customer SLA was active. In a built asset, note the number of workers on site, any evacuation, casualties and near-miss(es).
Detection
State the means by which the incident came to light. Whether it's via an automated alert, a report from a customer, or simply a member of staff noticing a water leak en route to their desk, each will lead to different conclusions. Determine the time elapsed between the onset of impact and the point of detection, and identify how this could be reduced by roughly 50%.
In highly monitored environments, this will likely highlight an absent data point, under-configured alerting or an alert that has been lost.
Establish the timeline
Populate this with objective facts only. Do not rely on memory:
- List events with references to the relevant logging source, alert, correspondence, JIRA or equivalent.
- Adopt a single common timestamp, with UTC being a good choice for multi-site and rostered teams.
- Extend the timeline backwards from the initial detection. It may include planned works, certificate expiry, software deployment, roster change and similar context.
- Note actions that confirmed resolution, rather than merely actions that addressed the issue.
- Append further material points for reference at the end of the document.
Ensure that any disparate systems used to generate the timeline, such as BMS, electrical and IT platforms, are reconciled for any clock skew. Otherwise you risk constructing an erroneous narrative.
Cause(s)
Pinpoint the inciting incident and identify all of the pre-requisite conditions required for the failure to result in the observed impact. There will typically be multiple, spanning technical, operational and exogenous causes.
Apply the Five Whys method, beginning with the initial impact and drilling down through successive whys. Cease once the cause(s) relate to a change in behaviour, working practice or architectural decision. Resist the urge to summarise this to a single root cause. There will almost certainly be additional deficiencies that will persist into subsequent incidents.
Mitigation and positive takeaways
Clearly differentiate between the temporary measure(s) taken to restore operations and the definitive corrective action(s). Having identified these, describe the areas where the organisation performed satisfactorily, for example effective communications or an appropriate runbook being executed. These should feed into future postmortems.
Negative takeaways and fortuitous circumstances
Identify failings in the organisation's ability to manage the incident, from delayed notification of key stakeholders, lack of clarity over responsibility and authority, inability to obtain requisite permissions, to missed or disregarded alarms.
Equally, highlight any fortunate breaks, such as the incident occurring outside a period of high load, or the availability of a replacement asset. Both require follow-up.
Follow-on actions
To ensure that the effort expended is justified, each of the above negatives must translate into a discrete, actionable item. For example, "Improve monitoring" is insufficient. "Implement alarm to notify duty engineer when temperature rises above X within the cold aisle containment" is an acceptable starting point.
Additionally, specify an accountable owner, target date and level of urgency, and attach a corresponding JIRA. Finally, define the criteria that will signal completion of the action, such as re-validation of the relevant control, assurance of coverage or a simulated scenario. Group these under Prevent, Detect, Contain and Recover. If the bulk fall into the latter, revisit the preceding steps.
Conducting the review
Plan this in advance and conduct no later than five working days following the incident. PagerDuty's guidance recommends five business days, while GitLab state that incident reviews should be concluded within five working days of the incident being closed, publishing an RCA for incidents of the highest severity within seven days. Ideally, the review should be prepared in advance for the session to focus on deliberation.
Following the structure proposed in CISA's resource guide, the key discussion points are:
- Attendees and their remit
- Summary of the incident
- Review and amend the established timeline
- Evaluate containment and recovery relative to RTO
- Consider lessons learned
- Pinpoint root causes for any deficiencies identified
- Define remedial steps, nominate owners and set deadlines
- Manage outstanding issues through to completion
It is best practice that the host of the session does not have ownership of the system under consideration. Call out staff by name, but frame your queries around their observations and knowledge only.
Attendees
Per the AWS Well-Architected Framework, this will include the key decision-makers and day-to-day owners of the system, any other internal parties impacted, representatives of the customers and the team(s) responsible for the resolution. AWS additionally advises that conclusions are made anonymous and circulated company-wide to maximise benefit.
At GitLab, there is a designated starter, driver and sign-off of the process. Furthermore, all remediation actions must be attributed to a specific team for the postmortem to be considered complete.
Why it should be blame-free
Underpinning a postmortem is the premise that all parties involved did their best given their position and understanding of the situation at the time. As described by Google in their SRE book, if you create an environment where staff are afraid to come forward about a potential issue, you will inevitably discover multiple such issues that have gone unreported. Think of all the close calls occurring in a live datacentre, or at a worksite.
Extending the post-mortem beyond IT
As discussed, IT-focused postmortems will conclude once the affected service(s) have been returned to normal. For other asset classes, additional aspects need to be considered. CISA lists staff welfare, including evacuation, casualties and care of the injured, staff management, media and other external communications, accommodation of the disruption by building and third-party providers, and return to the required RTO. From here, the output can be considered in the context of BC and DR arrangements.
Depending on the circumstances, each of the following categories may feature:
- Staff presence and safety: headcount, assets and staff evacuated, any injuries and near-misses, and the efficacy of any safeguarding controls. There are many considerations to take account of when building and running safe datacentres, some of which should form part of your pre-requisite hazard register.
- Building services: for example HVAC, electrical distribution, air separation, water ingress and firefighting. You will want to capture the telemetry trace to understand the progression of the event.
- Third parties: compare the actual time taken by utility, telecom, site owner and other suppliers to respond to the disruption versus their contractual commitments.
- Achieving the RTO: note the desired and achieved duration, and hence any shortfall. You may well find that you haven't previously exercised your contingency plans, and so this event has provided an opportunity to assess your ability to recover. Read more on creating and executing successful DR drills.
- Support chain: who was escalated to, in sequence, and how quickly was each tier able to accept the incident? See our article on defining appropriate support escalation processes for high-availability sites.
Any breakdowns to the physical estate should be captured in the postmortem as rigorously as code changes. For example, if a main supply line fractured in a plant area, what checks failed to identify this earlier? Plumbing experts suggest that even routine plumbing faults encountered in commercial buildings can be significant enough to feature in contingency planning and DR.
Sample findings
The following is hypothetical, although closely mirrors a genuine incident experienced by one of our customers. At 02:14, the primary chiller circuit supplying a co-location suite failed. A secondary system continued to provide sufficient cooling for a period of approximately 40 minutes. The cold aisle exceeded the maximum defined value at 02:51, triggering an alert. This was displayed on a monitoring console, with no notification to duty staff. An engineer detected the fault at 03:20.
Some indicative conclusions:
- The failure resulted in the loss of one active chiller for 66 minutes. While there was no impact to the applicable SLA, the water entering the suites approached the maximum specified value.
- Fault detection was delayed: 69 minutes elapsed between the point of failure and discovery by an individual.
- Contributing reasons included: the relevant pump was due for testing in the previous month; the alert was configured to notify a digital interface; one duty engineer was responsible for monitoring two adjacent suites; and existing remedial work had been triaged based purely on financial implication.
- We were fortunate that the prevailing external conditions allowed the standby chiller to continue servicing the hall.
- Remediation: configure alerts to page the relevant duty contact(s) to eliminate the likelihood of recurrence; implement predictive analytics on the pumps to flag degradation; and conduct a walk-through with the applicable duty engineer(s). All actions should have associated acceptance criteria.
There is nothing here that identifies any individual as being solely culpable for this event, merely items that can be remediated.
Ensuring the remediation actions are completed
Typically, the output of a postmortem will simply be filed away and little follow-up occurs, resulting in a re-occurrence of the original problem. To minimise this, consider the following.
First, log all tasks in your existing workflow management tool(s) and allow them to compete against other items for resource. Second, schedule regular triage of outstanding items and escalate any lagging items up the chain. Finally, complete each item only once you have objective proof that it's resolved. You cannot rely on verbal confirmation alone, but rather need to see the relevant test, monitoring or scenario re-executed.
Document the lesson(s) learned. This should be distributed to the rest of the business, anonymised, and if required externally, for example under contractual obligation or to maintain customer trust. Distributing lessons learned helps restore faith in the organisation and demonstrates learning.
Automating postmortems
As the volume of postmortems increases, producing them manually will take longer. In addition, the supporting information will typically be located in disparate locations. By instrumenting your environment to provide a historical record, and identifying commonalities between postmortems, you can reduce the time spent. We cover some approaches in our blog post.
Data
All of the postmortems that I've encountered that are likely to stand the test of time rely on adequate levels of baseline information being collected prior to the event. Your ability to create a factual narrative will be constrained by the instrumentation in place.
This is true whether you're running a datacentre or writing code. If you can aggregate the telemetry feeds from the various disciplines, such as BMS, PMS and NMS, onto a unified timestamp, then the subsequent investigation can focus entirely on why the issue occurred, rather than trying to piece together a sequence of events. Our 99.9% availability guarantee and 250+ successful implementations are a direct result of elevating this black box to first-class status.
Likewise, we consider the resulting set of actions to form part of the deliverable of a postmortem. These could range from implementing a revised escalation path, replacing a dated solution, to implementing an automated process. In any case, these must be clearly defined, assigned and demonstrated as complete.
Harry
Close the loop using your technology stack
Common themes will emerge from most postmortems, including absent detection mechanisms, lack of connectivity between relevant platforms and inefficient processes. At PODTECH we develop bespoke solutions for highly-available environments, covering telemetry, containment, BMS, PMS and DCIM advisory.
Where appropriate, we can assist to implement these recommendations via our Legacy Modernisation and Enterprise Automation offerings. Contact us via podtech.com or browse our range of products on our Products page, including our telemetry offering and POD View Liquid Monitoring.
FAQ
Why write a postmortem vs conduct a post-incident review?
They are effectively synonymous. As noted in PagerDuty's article, the terms Post-incident review, Learning review, After-action review and Incident report are all used. Simply pick one and ensure everyone knows this is the nomenclature to apply when searching for prior instances.
When should I conduct the postmortem?
Ideally within five business days of closing the incident and no later, consistent with the examples referenced previously. Of course the sooner the better given the risk of lost memory and log purging. Some organisations may choose to share a preliminary account within seven days of the incident for significant events.
Will my postmortem identify an individual who caused the incident?
No. Focus your description on the task(s) performed and the circumstances, for example no applicable alerting, deficient documentation or operator tiredness. As highlighted by the SRE book, assigning blame will encourage staff to be less forthcoming. However, if you can identify the enabling circumstances, there is an opportunity to address these.
What are the expected outputs from a postmortem?
One or more actions for each root cause identified. Obviously these will depend on your budget, but you should define an action, nominate a responsible party, target date and how you'll confirm its completion. Better to have a few actions completed than many left dangling.
Further Reading
- CRR Supplemental Resource Guide, Volume 5: Incident Management (CISA)
- Postmortem culture: learning from failure (Google SRE book)
- Conduct post-incident analysis for continuous improvement (AWS Well-Architected DevOps Guidance)
- Incident Review (The GitLab Handbook)
- Postmortem Template (PagerDuty Docs)
- What Is a Postmortem (PagerDuty Docs)
- A Guide to the Incident Postmortem Process (Atlassian)
