Incident management I: preparation, detection and triage

Every organization will have security incidents. What separates a manageable incident from a crisis is usually not the attacker's skill but the defender's readiness: whether people know who is in charge, whether alerts reach someone who can act, whether the organization can tell quickly how serious an event is, and whether decision makers are told at the right moment. Incident management is the organizational process that provides that readiness.

This lesson and the next treat incident response as a management process, which is how the CISSP outline frames it in sub-domain 7.6 (conduct incident management). This lesson covers the front half: the frameworks, preparation, detection, triage, severity and escalation. Lesson 6 covers the back half: containment decisions, eradication and recovery at a process level, communication and lessons learned. Lesson 8 covers legal and regulatory reporting. ITIL-140 described incident management as an IT service practice focused on restoring service; a security incident adds concerns about evidence, legal duties, adversaries and confidentiality, so the two processes must connect but are not identical.

Frameworks: NIST SP 800-61r3 and ISO/IEC 27035

The best known guide is NIST Special Publication 800-61. Revision 3, published in April 2025, replaced the long-familiar four-phase life cycle of revision 2 (preparation; detection and analysis; containment, eradication and recovery; post-incident activity) with a model built on the six functions of the NIST Cybersecurity Framework (CSF) 2.0. In revision 3, Govern, Identify and Protect are the preparation that happens before an incident; Detect, Respond and Recover are the incident response activities themselves; and lessons learned feed improvement back into all six functions. The change reflects a view that incident response is part of overall cybersecurity risk management rather than a separate activity run by a specialist team.

The international equivalent is the ISO/IEC 27035 series, information security incident management. Part 1 sets out principles and a process of plan and prepare; detect and report; assess and decide; respond; and learn lessons. Part 2 gives guidelines for planning and preparing, and Part 3 addresses incident response operations. Organizations certified to ISO/IEC 27001 often use 27035 because Annex A of 27001 includes controls for incident management planning, assessment, response, learning and evidence collection.

The CISSP outline lists the steps as detection, response, mitigation, reporting, recovery, remediation and lessons learned. Exam questions often test ordering, for example that containment comes before eradication, or that lessons learned is a required step rather than an optional extra. The terminology differs between frameworks, but the sequence of thinking is the same.

Preparation: policy, plan, team and tools

Preparation is where most of the value in incident management is created. An incident response policy, approved by senior management, defines what counts as an incident, gives the incident response team its authority (for example, to disconnect a system or engage outside help) and sets reporting obligations for staff. An incident response plan turns the policy into a working arrangement: roles, severity definitions, escalation paths, communication rules, contact lists, and links to the business continuity and crisis management plans. Playbooks give step-by-step process guidance for common incident types such as phishing compromise, ransomware, lost devices, data exposure in the cloud and insider misuse.

The team itself can be centralized, distributed or partly outsourced. Many organizations keep a small internal core (an incident manager, security analysts, and liaisons to IT, legal, privacy, communications and human resources) and hold retainers with external responders, forensic firms and breach counsel. Retainers are arranged in advance because negotiating contracts and confidentiality terms in the middle of a crisis wastes hours. Cyber insurance policies often require the use of approved vendors and prompt notification to the insurer, so the plan must reflect those terms.

Preparation also includes practice. Tabletop exercises walk decision makers through a realistic scenario so that gaps in the plan, such as an out-of-date contact list or unclear authority to take a customer-facing system offline, are found before a real incident. Exercises with executives are particularly valuable because the hardest incident decisions, such as whether to shut down operations, how to communicate publicly or whether to involve law enforcement, are business decisions, not technical ones.

Detection and reporting

Incidents are detected through many channels: alerts from the SIEM and endpoint tools, reports from employees who notice something odd, calls from customers, notices from partners or vendors, threat intelligence, and sometimes contact from law enforcement or a journalist. A good program makes reporting easy (a single phone number, address and button in the mail client), thanks people for reports even when they turn out to be harmless, and makes clear that staff will not be punished for reporting their own honest mistakes, such as clicking a link. Late reporting is one of the most common reasons incidents grow.

Two widely used measures of detection and response are mean time to detect (MTTD), the average time from when an incident begins to when it is detected, and mean time to respond or recover (MTTR), the average time from detection to containment or restoration. Definitions vary, so a program must state precisely which start and end points it uses. Medians are often more informative than means because a single long-running intrusion can dominate an average.

from datetime import datetime as dt
from statistics import mean, median
# Synthetic incident timeline records: start, detected, contained (hours resolution)
incidents = [
    ('IR-301', '2026-03-02 08:00', '2026-03-02 10:30', '2026-03-02 15:00'),
    ('IR-302', '2026-03-09 22:00', '2026-03-10 07:00', '2026-03-10 12:00'),
    ('IR-303', '2026-04-01 13:00', '2026-04-01 13:20', '2026-04-01 16:20'),
    ('IR-304', '2026-04-15 02:00', '2026-05-20 09:00', '2026-05-21 18:00'),
    ('IR-305', '2026-05-03 11:00', '2026-05-03 12:00', '2026-05-03 20:00'),
]
f = '%Y-%m-%d %H:%M'
ttd, ttc = [], []
for rid, s, d, c in incidents:
    s, d, c = dt.strptime(s, f), dt.strptime(d, f), dt.strptime(c, f)
    ttd.append((d - s).total_seconds() / 3600)
    ttc.append((c - d).total_seconds() / 3600)
    print(f'{rid}: detect {ttd[-1]:7.1f} h   contain {ttc[-1]:5.1f} h')
print(f'MTTD mean {mean(ttd):6.1f} h, median {median(ttd):4.1f} h')
print(f'MTTC mean {mean(ttc):6.1f} h, median {median(ttc):4.1f} h')

IR-301: detect     2.5 h   contain   4.5 h
IR-302: detect     9.0 h   contain   5.0 h
IR-303: detect     0.3 h   contain   3.0 h
IR-304: detect   847.0 h   contain  33.0 h
IR-305: detect     1.0 h   contain   8.0 h
MTTD mean  172.0 h, median  2.5 h
MTTC mean   10.7 h, median  5.0 h

One long-undetected intrusion, IR-304, pushes the mean time to detect to roughly a week even though the median is a few hours. Reporting both figures, and discussing outliers individually, gives leadership a truer picture than a single average.

Triage, severity and escalation

Triage is the rapid assessment of a reported event to decide whether it is an incident, how serious it is, and who needs to be involved. Severity is usually decided from two dimensions: impact (what is affected, how badly, how many people, which data) and urgency or scope (whether the situation is spreading, whether it is ongoing, whether there is a deadline). Many organizations express this as a matrix whose cells map to severity levels, each of which carries required response times, an incident lead and an escalation and notification list.

Severity should be revisited as facts emerge. An event first classed as SEV-3 because one laptop showed a WARN-level alert may become SEV-1 when investigators learn the laptop belonged to a payroll administrator and that data left the network. Escalation in both directions must be easy and blame-free: it is far better to escalate early and stand down than to discover that executives learned of a serious breach from the news.

# Classify events with the severity matrix and list who must be notified
impact_levels = ['low', 'moderate', 'high', 'severe']
scope_levels = ['single', 'department', 'organization']
matrix = [[4, 4, 3], [3, 3, 2], [2, 1, 1], [1, 1, 1]]
notify = {1: 'CISO, CIO, legal, privacy, comms, exec sponsor', 2: 'CISO, system owner, legal/privacy as needed',
          3: 'system owner, security manager', 4: 'ticket only'}
events = [
    ('ERROR malware blocked on one kiosk', 'low', 'single'),
    ('WARN unusual login pattern for finance team', 'moderate', 'department'),
    ('ERROR customer database exposed to internet', 'high', 'organization'),
    ('ERROR ransom note on file servers at two sites', 'severe', 'department'),
]
for text, imp, scope in events:
    sev = matrix[impact_levels.index(imp)][scope_levels.index(scope)]
    print(f'SEV-{sev}  {text}\n        notify: {notify[sev]}')

SEV-4  ERROR malware blocked on one kiosk
        notify: ticket only
SEV-3  WARN unusual login pattern for finance team
        notify: system owner, security manager
SEV-1  ERROR customer database exposed to internet
        notify: CISO, CIO, legal, privacy, comms, exec sponsor
SEV-1  ERROR ransom note on file servers at two sites
        notify: CISO, CIO, legal, privacy, comms, exec sponsor

Analyst capacity and the triage queue

Triage happens in a queue, and queues behave in ways that surprise managers. When alerts arrive faster than analysts can handle them, waiting time does not grow gently; it grows without limit. Even when average capacity is slightly above average demand, random bunching of arrivals produces long waits at busy times. Queueing theory captures this with utilization: the ratio of arrival rate to service capacity. As utilization approaches 100 %, waiting time rises steeply. A SOC that plans its staffing to run at 95 % utilization will regularly fail its response targets.

import random
random.seed(42)
# Simulate a triage queue for one 8-hour shift at different staffing levels
def simulate(analysts, arrivals_per_hour=11, minutes_per_alert=20, hours=8, runs=200):
    waits = []
    for _ in range(runs):
        t, free_at = 0.0, [0.0] * analysts
        while True:
            t += random.expovariate(arrivals_per_hour / 60)
            if t > hours * 60:
                break
            i = min(range(analysts), key=lambda k: free_at[k])
            start = max(t, free_at[i])
            waits.append(start - t)
            free_at[i] = start + random.expovariate(1 / minutes_per_alert)
    waits.sort()
    return sum(waits) / len(waits), waits[int(0.9 * len(waits))]
for n in (4, 5, 6):
    util = 11 * 20 / 60 / n
    avg, p90 = simulate(n)
    print(f'{n} analysts  utilization {util:4.0%}  mean wait {avg:5.1f} min  90th pct {p90:5.1f} min')

4 analysts  utilization  92%  mean wait  19.4 min  90th pct  56.4 min
5 analysts  utilization  73%  mean wait   5.7 min  90th pct  20.1 min
6 analysts  utilization  61%  mean wait   1.8 min  90th pct   6.0 min

The simulation shows the managerial point: moving from four to five analysts cuts waits dramatically, while the sixth analyst gives a smaller gain. Improving rule precision, as Lesson 3 showed, reduces the arrival rate and has the same effect as hiring. Automation through SOAR reduces handling time per alert. Both are usually cheaper than adding staff, and both make analysts' work more meaningful.