Incident Response Runbooks: How to Build Ones That Work

An incident response runbook is a documented and executable play book for one specific type of incident. It specifies who does what, what to look for, what commands or systems to utilize, what the expected result is and when to stop or escalate. A well-written runbook allows a trained responder to safely act at 3 a.m. without having to improvise. A poorly written runbook is a policy document that no one can follow when the monitoring tools are down.

This guide covers what must be in a runbook, how to create your first five runbooks, how runbooks differ from playbooks and plans, and how things change when your incident impacts data centres, building systems or industrial controls.

TL;DR

• A runbook is a set of operations, not a story book policy. Every step requires an owner, a prerequisite, a where or command, an expected outcome and a stop or escalate condition.
• Guidance from sources like NIST, AWS and Microsoft is converging around a common framework comprised of prerequisites, workflow, verification and playbooks or steps.
• Begin with runbooks that cover events you are likely to encounter. Verify each runbook in a real environment, and place it under version control.
• Having only an IT-scope runbook in critical infrastructure is half baked. Escalation needs to hit facilities, engineering, safety, legal and executives if impacted.

Table of Contents

• Runbook, playbook or plan: what is the difference?
• Why NIST’s 2025 revision matters for your runbooks
• What every runbook needs
• Building your first five runbooks
• A worked example: compromised service account
• When the incident reaches the physical world
• Escalation: write it down before you need it
• Testing, version control and automation
• Our perspective: the software layer between failure and success
• Where PODTECH can help
• A maintenance routine that keeps runbooks alive
• FAQ
• Sources

Runbook, playbook or plan: what is the difference?

These terms get thrown around pretty loosely by most people. Search results make that obvious: you'll find the same page titled “playbook” sometimes and “runbook” other times. But having a working distinction helps your team avoid debating semantics during an incident.

Plans are where you put authority into practice. Playbooks are where you detail your approach to a particular class of attack such as ransomware or credential compromise. Runbooks are where you store the keystrokes. Most organisations combine the latter two. That's fine, as long as you make the document executable and you actually write down who has decision authority.

Why NIST’s 2025 revision matters for your runbooks

NIST released the final SP 800-61 Rev. 3 on April 3, 2025. The big idea is that incident response should be integrated with cybersecurity risk management processes and other CSF 2.0 functions, not opened as a playbook only after an alert. Guidance documents from years back are still out there using the Rev. 2 phase model, so expect to know both vocabularies.

The lesson for runbook authors is that every runbook should tie back to business risk and service ownership. It should document which service it is protecting, who owns that service, what business impact will trigger escalation, what the recovery priority is and how lessons are fed back into preparation. Executives should request those fields when creating runbooks, not just technical commands.

What every runbook needs

AWS Well-Architected guidance outlines what makes an effective runbook. Microsoft’s incident response guidance organizes a similar concept into four tiers. Together they provide a practical template.

1. Scenario, goal and scope

Enter the incident type and desired outcome. “Privileged service account had its credentials compromised: containment within 30 minutes, no lateral movement” is a goal. “Manage account issues” is not. Internalize any target times as your own goals set by your risk owners.

2. Prerequisites

This is the layer teams miss and the layer that fails most frequently. Enumerate the logs that should be available, the detections that will trigger the alert, the tools, and the roles and permissions someone needs before beginning. Then validate them regularly. The procedure that is technically correct but cannot be executed during an outage because you assumed access, telemetry, contacts or permissions is failure number one.

3. Intake paths

CISA’s federal incident and vulnerability response playbooks acknowledge that incidents can be responded to from automated alerts, user reports, contractor or third-party reports, situational-awareness updates, known compromised infrastructure, loss of services or threat-hunting results. A runbook beginning only at “SIEM alert fires” ignores the phone call from the site engineer that a controller is acting funny.

4. Workflow with phases

Things happen in loops during incident response. CISA breaks it into declaration or detection and analysis, containment, eradication and recovery, and post-incident activity, noting that you will probably loop between some of these steps before declaring closure. Map your workflow as a flow diagram with decisions in it. Don't try to draw it as a straight line. Make the places where the responder will loop back explicit.

5. Step detail

For each action record:

• Owner: a role, not a name.
• Location or command: the system, console path, query or script.
• Expected result: how the desired outcome should look, so the responder knows whether to proceed.
• Stop or rollback condition: when to halt and who to call.
• Evidence to capture: logs, screenshots, timestamps.
• Escalation path: who is next, and by what channel.

6. Completion checklist

Microsoft provides a checklist layer for responders to validate that each necessary step was completed. This also acts as your audit trail for regulated environments.

Building your first five runbooks

AWS suggests creating runbooks for incidents you are likely to face, such as denial of service, ransomware and credential compromise, as well as specific alert types you know you will receive from your own detection tools. Don't try to create runbooks for everything. Choose five this way.

1. List your last 12 months of incidents and near misses. Facts trump fiction.
2. Include the scenarios that are at the top of your risk register, even if they have not occurred yet.
3. Rank by likelihood and impact, and take the top five.
4. Assign one author who does the work and one reviewer who would actually run it. The reviewer tests whether it is usable.
5. Run a tabletop exercise on each before you call it done.

A sensible first set for most enterprises:

• Compromised privileged credentials
• Ransomware on a file or application server
• Phishing leading to a mailbox compromise
• Denial of service against a public service
• Loss of monitoring or telemetry for a critical system

The hardest element to notice is your environment. If your environment is invisible, every other runbook is essentially running blindfolded.

A worked example: compromised service account

Below is a shortened version of the anatomy of one runbook. Tweak as needed for your identity platform.

Situation and objective: One of your privileged service accounts is active during an unusual time. Identify whether this is compromised and, if so, contain and recover the service.

Requirements: Existing authentication logs stored and searchable; someone on-call with permissions to disable accounts or rotate passwords; up-to-date contact on the service owner; tested break-glass account.

Steps:

1. Create incident or ticket. Owner: whoever is on-call. Resolution: incident number and Incident Commander identified.
2. Look up the last 72 hours of sign-in and API activity for the account. Expected result: list of source addresses, hosts and actions taken. Stop and escalate if the account accessed systems outside documented scope.
3. Query the service owner about recent changes. Expected outcome: confirmed innocent explanation or documented lack thereof.
4. Turn off the account or revoke its permissions, then rotate its secrets. Verify the behavior of the dependent service beforehand so you know what will break. Stop condition: before taking action, if disabling the account will interrupt a safety-relevant or production system, escalate to the system owner to make that decision.
5. Search for the same indicators across other accounts and hosts.
6. Restore service with new credentials, then confirm normal operation.
7. Close with a completion checklist and schedule a review.

Take note of step 4. This is where the pure IT-flavoured runbook collides with operational reality.

When the incident reaches the physical world

If you manage data centres, building systems or industrial facilities, a stolen identity, rogue controller or telemetry outage can impact environmental controls, production processes and safety systems. The recently released CISA’s Dams Sector Crisis Management Handbook from April 2025 frames crisis management as preparing for incidents that will have severe consequences to people, infrastructure, systems, operations, stakeholders, brand and reputation. Written for a single sector, the concepts still apply: IT activities should align with facility, safety and business continuity responses.

Concretely, your runbooks for these environments should add:

• System dependency notes. What BMS, PMS or NMS components will this action affect? Segregating a network segment could have thermal ramifications if a building management system controls cooling and power.
• Safety gates. Steps with potential impact to humans or physical plant are gated for sign-off by the named engineering or safety owner.
• Expanded escalation. Facilities, engineering, safety, legal, communications and executives, depending on impact.
• Multi-site coordination. If an event occurs at multiple sites there has to be ownership of the collective picture. This guide on coordinating multi-site protection considers that topic from the site-systems perspective.

Business continuity links matter too. Executive accountability and stakeholder roles play a part on construction and building projects as well, such as in this summary of an emergency response plan for commercial buildings.

The same argument can be made for remote access. Most operational events start with a discovered remote path, so your runbooks should cite the controls outlined here: identity-first controls for remote monitoring across IT and OT, as well as long-term hardening activities in a 3 to 12 month industrial IoT security roadmap.

Escalation: write it down before you need it

A runbook is only as strong as its handoffs. Define severities in business terms. Name the role that can declare each severity. Give each severity level its own channel and response expectation. We have our own thoughts on this which you can read here in our article on support escalation workflows for critical infrastructure.

A simple severity table to adapt:

Limit the table to one page. If it requires a meeting to understand it, nobody will look at it at 3 a.m.

Testing, version control and automation

AWS points out that playbooks change as steps get validated, and validated steps become automation candidates. Sequence your actions accordingly. Automating something you didn't test just speeds up failure.

Version control. Keep runbooks in a central repository that includes review history and owner metadata. Date stamp every revision. Include the date of last test at the top.

Test against real life. Execute each runbook through a tabletop exercise, then have responders execute the queries during a technical drill. Document every instance where access, logging or a contact was incorrect, and remediate these prerequisites first.

Do automation in phases. Start by automating read-only steps like evidence collection and enrichment. Follow with reversible containment automation. Leave anything that touches a production system or safety system gated by human approval.

Use public tools for the containment layer. Released 30 July 2025, CISA’s free Eviction Strategies Tool helps defenders author customised containment and eviction plans based on MITRE ATT&CK techniques or simply a free-form description of attacker activity. It uses more than 100 curated countermeasure actions and exports to JSON, Word, Excel and Markdown, which makes it useful for a version-controlled runbook repository. Looking for starting templates? Grab the cloud-centric examples from AWS’s sample runbooks repo and tailor them to your own risks, tools and processes.

Our perspective: the software layer between failure and success

We write software for data centres, buildings and other mission-critical systems. That means we see the same pattern replicated in runbooks that we see in systems. Documents don't tend to fail because they're wrong. They fail because they are unvalidated. Someone assumed a dashboard would be there, a sensor feed would be up to date, or a contact list would be correct.

Three convictions shape how we approach this:

• Telemetry is table stakes, not a nice-to-have. If your runbook includes the step “check the cooling status” and the source of that status is an integration nobody has tested since day-one commissioning, then that step is fictional. That’s why integration of BMS, PMS and NMS data into a single pane of glass is so important, and why we insist that monitoring teams be involved from day one, as covered in our post on datacentre mobilisation.
• Cyber and facility response is one process. Where our customers operate, the engineer who can cybersecurely isolate a network segment is usually not the engineer who knows what racks that network segment cools. The runbook brings them together.
• Runbooks must be machine readable wherever possible. Machine-readable steps with owners, expected results and stop conditions are the only ones that can be safely automated later.

If your runbook cannot explain to a responder what normal is, it cannot explain to them when to stop.

— Harry

Where PODTECH can help

Runbooks are only as good as the systems underneath. If your monitoring data is siloed into legacy BMS, PMS and NMS tools then integration is job one. Our BMS and PMS integration services can connect those dots, and our legacy modernisation services can refresh ageing platforms you can no longer trust when responding to incidents. Our enterprise automation software is designed for automating validated steps. To learn more about your environment, start with our critical infrastructure datacentre solutions.

A maintenance routine that keeps runbooks alive

Runbooks rot. People leave. Systems evolve and phone numbers expire. Regular maintenance avoids that:

• After every incident: update the runbook that you used. While it's fresh, note what was missing, what was wrong and what was slow.
• After each significant change such as adding a new identity provider, replacing a controller or adding a new monitoring platform, review every runbook that references it.
• Quarterly: verify prerequisites, contacts and access for the top runbooks.

Yearly: conduct an exercise that involves executives and facility owners, then compare against last year's results.

Track three simple metrics: percentage of runbooks exercised in the last 12 months, number of prerequisite failures discovered per exercise, and elapsed time from incident declaration to first containment effort. They'll tell you far more than word count ever will.

FAQ

What should an incident response runbook include?

At the very least include the situation and objective, requirements such as logs, tools and permissions, notification and stakeholder duties, response steps with expected results, and a completion checklist. Both AWS and Microsoft describe almost the exact same format. Include an owner, a stop condition and an escalation path for each step.

Are incident response runbooks the same as playbooks?

Close, but many organizations use those terms synonymously. A playbook outlines the strategy and roles for a class of incident. A runbook details the exact steps, queries and commands for a single scenario. If you combine them, make sure to keep the decision makers and the prescriptive detail cleanly delineated in the document.

How often should runbooks be tested?

Run every test at least annually, and after any significant change to the systems the test covers. Check prerequisites such as access and contacts even more frequently, since those things become stale faster. Use every actual incident as a test and revise the runbook after.

Which framework should we base our runbooks on?

US organisations often begin with NIST. The latest SP 800-61 Rev. 3 links incident response back to CSF 2.0 and risk management. CISA’s playbooks lay out a usable phase structure you can emulate. Check with your compliance team for sector-specific regulatory requirements.

Sources

An incident handling guide published by the National Institute of Standards and Technology (NIST).

Recommendations and considerations for creating an incident response capability as part of an organization's cybersecurity risk management strategy.

• AWS Well-Architected Framework: security incident response playbooks
• Microsoft Learn: incident response playbooks
• CISA: Cybersecurity Incident and Vulnerability Response Playbooks
• CISA: Eviction Strategies Tool announcement
• CISA: Dams Sector Crisis Management Handbook
• AWS sample incident response runbooks

Recommended

• Five Step NIST Aligned Runbook for Infrastructure Operators Using MLOps
• Support Escalation Workflows for Critical Infrastructure
• IT operations optimization workflow: 2026 enterprise guide
• How to enhance enterprise security in 2026