Before
- React to one alert
- Make undocumented changes
- Restore the visible host
- Close when alerts stop
Module 14 of 17
Containment, Eradication, and Recovery
Coordinate a safe incident response using NIST SP 800-61 Rev. 3 and evidence-based decisions.
Start here
Learning objectives
- Place incident-response work within the six NIST Cybersecurity Framework 2.0 Functions used by SP 800-61 Rev. 3.
- Separate detection, analysis, containment, eradication, recovery, and improvement decisions.
- Choose containment actions by balancing evidence preservation, business impact, and risk of continued exposure.
- Define recovery acceptance criteria and observe for recurrence instead of treating restoration as closure.
- Create an incident decision log that records facts, uncertainty, authorization, owner, and time.
After
- Maintain an evidence-based scope
- Authorize and log decisions
- Remove the failed trust path
- Verify staged recovery
Incident Response Is a System Capability
NIST SP 800-61 Rev. 3 integrates incident response with the NIST Cybersecurity Framework 2.0. Govern, Identify, and Protect support preparation and improvement. Detect, Respond, and Recover cover the active incident lifecycle, while lessons from an event feed back into every Function. The important shift is that response is not a document opened after an alert. It is an organizational capability designed, funded, exercised, and improved before an incident.
For developers, preparation includes asset and service ownership, trustworthy builds, logging contracts, release records, recovery objectives, dependency inventories, access paths, and tested rollback. When these are missing, responders spend the first hours reconstructing basic system knowledge instead of reducing harm.
This module uses a tabletop with fictional systems and synthetic evidence. It does not ask learners to investigate a real compromise or run a suspicious artifact.
Build an Evidence-Based Scope
Scope is a living statement of what is known, suspected, and not yet observed. Start with the triggering evidence, its source, collection health, time range, affected identity or asset, and confidence. Separate direct facts from analyst interpretation. Record alternative explanations and the evidence that would change the current assessment.
A timeline should normalize time zones and preserve original timestamps. Link each entry to an evidence identifier rather than copying fragments without provenance. Gaps matter: missing telemetry, an unmonitored segment, or an unknown asset owner should be written as explicit uncertainty rather than silently assumed safe.
Severity combines technical scope with business consequence. A low-volume event in a release-signing service can be more urgent than many noisy workstation alerts. Engage service owners, legal, privacy, communications, and leadership according to the incident plan, not according to improvisation in a chat channel.
Containment Is a Reversible Risk Decision
Containment limits further harm while preserving the ability to understand and recover. Possible authorized actions include isolating a managed asset through the approved endpoint platform, disabling a compromised identity through the identity team, pausing a release channel, revoking an artifact, or applying a temporary network policy. This course discusses those decisions; it does not provide intrusion or destructive commands.
For each option, record expected risk reduction, customer and service impact, evidence impact, prerequisites, authority, owner, rollback, and signals that show whether it worked. Prefer actions that are narrowly scoped, observable, and reversible. Emergency changes still need a decision record.
Do not let a desire for perfect attribution delay urgent protection, but do not erase evidence blindly. The incident lead decides with the appropriate owners. A responder should never act on a third-party system or personal account without explicit authorization.
Eradication Addresses the Cause
Eradication removes the conditions that allow the incident to continue or return. Reimaging one host is not sufficient if a compromised build credential, vulnerable update path, unsafe dependency policy, or unauthorized release remains. Trace the affected trust relationships and identify which controls failed or were absent.
Use known-good sources. Rebuild from reviewed source in a controlled pipeline, verify provenance and signatures, rotate affected credentials through the owning platform, remove unauthorized artifacts through approved management systems, and patch the exposed path. Each action needs verification evidence.
Keep eradication distinct from punishment. The purpose of the review is to improve systems and decisions, not to find an individual to blame. Blame suppresses the weak signals teams need in the next incident.
Recovery Requires Acceptance Criteria
Recovery restores business capability with known risk. Define criteria before reconnecting or reopening a release path: approved clean build, verified configuration, required patches, credential state, telemetry health, smoke tests, data validation, owner approval, and rollback readiness. Restore in controlled stages when possible.
Observe for recurrence using the indicators and behaviors identified during analysis, but do not declare success merely because alerts stop. Confirm that telemetry is flowing, detection logic still evaluates, dependencies are reachable, and customers see the expected behavior. Track temporary controls until each is removed or made permanent through normal change management.
After stabilization, document what happened, why existing controls allowed it, what reduced impact, what slowed response, and which changes have owners and dates. Validate improvements in a later exercise. A lesson without an owner and verification plan is only a note.
Suggested Study Plan (6 hours)
The published duration includes active practice, not video playback alone. Complete each block with the course-owned evidence and retain the stated deliverable so another reviewer can reproduce your reasoning.
| Study block | Time | Required evidence |
|---|---|---|
| Guided lesson and primary-source review | 1h 3m | Annotated notes that separate observations, hypotheses, limits, and version-sensitive facts. |
| Worked evidence walkthroughs | 53 minutes | Reproduce the lesson's tables or decision flow and challenge at least two assumptions. |
| Independent practice rounds | 53 minutes | Apply the method to two alternate records in the sanitized evidence pack and compare the conclusions. |
| Required lab | 2h 30m | A scoped incident statement, evidence-linked timeline, containment decision matrix, authorized action log, eradication plan, recovery checklist, and owned improvement backlog. |
| Knowledge check and review | 41 minutes | Answer the evidence check, review the rubric, and record one production follow-up. |
Evidence Check
Question: A team restored a service from a clean image and received no alerts for one hour. What evidence is still needed before declaring recovery complete?
Reveal the defensive reasoning
The team should verify the root cause was addressed, affected identities and artifacts were handled, the restored build and configuration are known-good, data integrity and service tests pass, telemetry and detections are healthy, staged acceptance criteria are met, owners approve the risk, rollback remains available, and enhanced monitoring covers the agreed observation period.
Primary References
Use these primary sources for the current standard or tool behavior. The course records framework versions so mappings can be reviewed when upstream guidance changes.
Real world
Where this shows up
- Pausing and validating a software release after a synthetic integrity alert
- Coordinating application, identity, endpoint, legal, and communications owners during an incident
- Defining staged service recovery and rollback criteria
- Turning incident lessons into tested platform and SDLC improvements
Production notes
Keep these close
- Pre-authorize common containment actions and maintain current service and escalation ownership.
- Test backups, rebuild paths, signing-key recovery, and telemetry restoration - not only data restoration.
- Use one incident clock, evidence identifiers, and an auditable decision log.
- Track temporary controls as expiring changes so emergency exceptions do not become permanent debt.
Common mistakes
What usually breaks
- Treating the first alert as a complete scope or attribution statement
- Isolating or deleting systems without authority, impact review, or evidence planning
- Rebuilding an endpoint while leaving the compromised trust path unchanged
- Declaring recovery because alerts stopped without checking telemetry health
Security risks
Threats to watch
- Over-broad containment can cause avoidable customer or safety impact.
- Delayed containment can allow the simulated impact to expand.
- Uncontrolled evidence handling can undermine privacy, integrity, or later review.
- Premature reconnection can restore the same exposure.
Tradeoffs
Design choices you should be able to defend
Immediate broad isolation
Pros
- Can reduce exposure quickly
- Simple operational message
Cons
- Can interrupt critical services
- May remove access needed for evidence collection
Targeted staged containment
Pros
- Reduces business impact
- Can preserve evidence and observability
Cons
- Needs trustworthy scope and coordination
- May leave residual risk longer
Observe before action
Pros
- Can improve understanding
- Avoids unnecessary disruption
Cons
- Can allow impact to continue
- Requires explicit risk acceptance and monitoring
Think like an engineer
Questions to answer before shipping
- Which containment action reduces the most risk while preserving a reversible path?
- What evidence would prove that the root cause, not only the symptom, was removed?
- How do you know a quiet dashboard reflects recovery rather than broken collection?
- Which missing platform capability made this incident harder than it needed to be?
Key terms
Vocabulary used in this module
Containment
Authorized action that limits further incident impact while considering business and evidence needs.
Eradication
Removal of the artifacts, access, vulnerabilities, or process failures that allow an incident to continue or recur.
Recovery
Controlled restoration of business capability with verified acceptance criteria and monitoring.
Decision log
Time-ordered record of incident facts, choices, authority, owners, rationale, and outcomes.
Recovery acceptance criteria
Observable conditions that must be met before a system or capability returns to service.
Exercises
Practice inside the lesson
Run a Synthetic Developer-Platform Incident Tabletop
Use a fictional timeline and service map to make authorized containment, eradication, and recovery decisions with explicit evidence and uncertainty.
- Confirm the tabletop boundary: all names, systems, logs, and observables are fictional and no real account or environment may be queried.
- Classify each evidence card as fact, interpretation, uncertainty, or missing telemetry, then create a normalized timeline.
- Identify affected business capabilities, owners, trust relationships, and the current confidence of the scope.
- Compare at least three containment options using risk reduction, service impact, evidence impact, authority, reversibility, and verification signals.
- Choose an authorized containment plan and write every action, owner, timestamp, reason, and rollback in the decision log.
- Define eradication work that addresses the simulated root cause rather than only the visible endpoint.
- Write staged recovery acceptance criteria, enhanced-monitoring checks, customer communication triggers, and a rollback threshold.
- Finish with three improvement actions mapped to Govern, Identify, Protect, Detect, Respond, or Recover, each with an owner and validation date.
Expected evidence
A scoped incident statement, evidence-linked timeline, containment decision matrix, authorized action log, eradication plan, recovery checklist, and owned improvement backlog.
Assessment criteria
- Facts, interpretations, unknowns, and missing telemetry are visibly separated.
- Containment decisions balance risk reduction, service impact, evidence preservation, authorization, and rollback.
- Eradication addresses the simulated trust or process failure instead of only replacing one asset.
- Recovery criteria verify build, identity, data, service, and telemetry health before closure.
- Improvement actions have an accountable owner, due date, and later validation method.
Course-owned resources
Teardown
- Delete temporary tabletop card copies and draft notes.
- Retain only the approved fictional timeline, decision log, recovery checklist, and improvement backlog.
Recap
Key takeaways
- NIST SP 800-61 Rev. 3 treats incident response as part of all six CSF 2.0 Functions.
- Scope must show facts, interpretations, unknowns, and telemetry gaps separately.
- Containment is an authorized risk decision with business, evidence, and rollback consequences.
- Recovery is complete only when acceptance criteria, telemetry health, and root-cause controls are verified.
Related resources