incident response

Conducting Post-Incident Lessons Learned

Facilitate structured post-incident reviews to identify root causes, document what worked and failed, and produce actionable recommendations to improve future incident response.

after-action-reviewincident-responselessons-learnedpost-incidentprocess-improvement
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

When to Use

  • After any security incident has been fully resolved and recovery completed
  • Following tabletop exercises or IR simulations
  • After significant near-miss events
  • Quarterly review of accumulated incident trends
  • When IR playbooks need updating based on real-world experience

Prerequisites

  • Incident fully resolved (containment, eradication, recovery complete)
  • Incident timeline and documentation gathered
  • All incident responders available for review session
  • Meeting space for collaborative discussion
  • Incident ticketing system data for metrics analysis

Workflow

Step 1: Gather Incident Data

# Export incident timeline from ticketing system
curl -s "https://thehive.local/api/v1/case/$CASE_ID/timeline" \
  -H "Authorization: Bearer $THEHIVE_API_KEY" | jq '.' > incident_timeline.json
 
# Extract detection and response metrics from SIEM
index=notable incident_id="IR-2024-042"
| stats min(_time) as first_alert, max(_time) as last_alert,
  count as total_alerts, dc(src) as unique_sources
 
# Compile all responder actions and timestamps
grep -E "timestamp|action|analyst" /var/log/ir/IR-2024-042/*.json | \
  python3 -m json.tool > compiled_actions.json

Step 2: Conduct Blameless Post-Mortem Meeting

Structured Agenda (90 minutes):
1. Incident summary (5 min) - Factual overview
2. Timeline walkthrough (20 min) - Chronological events
3. What worked well (15 min) - Positive outcomes
4. What needs improvement (15 min) - Gaps and failures
5. Root cause analysis (15 min) - 5 Whys or fishbone
6. Action items (10 min) - Specific improvements with owners
7. Playbook updates (10 min) - Changes to IR procedures
 
Blameless Principles:
- Focus on systems and processes, not individuals
- Assume best intentions with available information
- Seek to understand, not to blame

Step 3: Perform Root Cause Analysis

# 5 Whys analysis example:
# Why 1: Why did ransomware encrypt production servers?
#   Answer: Attacker had domain admin credentials
# Why 2: Why did attacker have domain admin credentials?
#   Answer: Kerberoasted a service account and cracked it
# Why 3: Why was the service account password crackable?
#   Answer: Used a 12-character dictionary-based password
# Why 4: Why was the service account password weak?
#   Answer: No enforcement of service account password policy
# Why 5: Why was there no service account password policy?
#   Answer: PAM was not implemented for service accounts
# ROOT CAUSE: Lack of privileged access management

Step 4: Calculate Response Metrics

from datetime import datetime
events = {
    'compromise': '2024-01-10 14:00:00',
    'detection': '2024-01-15 08:30:00',
    'triage': '2024-01-15 08:45:00',
    'containment': '2024-01-15 09:30:00',
    'eradication': '2024-01-16 14:00:00',
    'recovery': '2024-01-18 16:00:00',
    'closure': '2024-01-25 10:00:00',
}
fmt = '%Y-%m-%d %H:%M:%S'
times = {k: datetime.strptime(v, fmt) for k, v in events.items()}
print(f"Dwell Time: {times['detection'] - times['compromise']}")
print(f"MTTD: {times['triage'] - times['detection']}")
print(f"MTTC: {times['containment'] - times['detection']}")
print(f"MTTR: {times['recovery'] - times['eradication']}")
print(f"Total Duration: {times['closure'] - times['detection']}")

Step 5: Document Findings and Create Action Items

# Create tracked action items in project management
curl -X POST "https://jira.local/rest/api/2/issue" \
  -H "Authorization: Bearer $JIRA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "fields": {
      "project": {"key": "SEC"},
      "summary": "Implement PAM for service accounts (IR-2024-042)",
      "issuetype": {"name": "Task"},
      "priority": {"name": "High"},
      "assignee": {"name": "security_engineer"},
      "duedate": "2024-03-15"
    }
  }'

Step 6: Update Playbooks and Detection Rules

# New Sigma detection rule based on incident learnings
title: Kerberoasting Activity Detected
status: stable
description: Detects Kerberoasting based on IR-2024-042 lessons
logsource:
  product: windows
  service: security
detection:
  selection:
    EventID: 4769
    TicketEncryptionType: '0x17'
  condition: selection
level: high
tags:
  - attack.credential_access
  - attack.t1558.003

Key Concepts

Concept Description
Blameless Post-Mortem Reviewing incidents focusing on systems, not blaming individuals
Root Cause Analysis Identifying the fundamental reason the incident occurred
5 Whys Iterative questioning technique to find root cause
MTTD Mean Time to Detect - time from compromise to detection
MTTC Mean Time to Contain - time from detection to containment
MTTR Mean Time to Recover - time from eradication to full recovery
Continuous Improvement Iterating on IR processes based on real incident data

Tools & Systems

Tool Purpose
TheHive/ServiceNow Incident timeline and documentation
Jira/Azure DevOps Action item tracking
Confluence/SharePoint Lessons learned documentation
Splunk/Elastic Incident metrics and detection improvement
Sigma Detection rule development

Common Scenarios

  1. Ransomware Post-Mortem: Review entire kill chain from initial access to encryption. Identify detection gaps and backup failures.
  2. Phishing Campaign Review: Analyze why users clicked, why email filters missed it, and how to improve training.
  3. Cloud Misconfiguration Incident: Review IaC pipeline, CSPM coverage, and change management process.
  4. Insider Threat Review: Examine DLP effectiveness, access control gaps, and user monitoring capabilities.
  5. Third-Party Breach Impact: Review vendor risk assessment process and data sharing agreements.

Output Format

  • Post-incident review meeting minutes
  • Root cause analysis document
  • Incident metrics report (MTTD, MTTC, MTTR)
  • Action items list with owners and deadlines
  • Updated IR playbooks and detection rules
  • Executive summary for leadership
Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 3

api-reference.md1.5 KB

Post-Incident Lessons Learned — API Reference

Libraries

Library Install Purpose
requests pip install requests API calls to ticketing/SIEM systems
jinja2 pip install Jinja2 Report template rendering
matplotlib pip install matplotlib Timeline and metric visualization

Key Metrics

Metric Formula Target
MTTD Detection time - Incident start < 30 minutes
MTTC Containment time - Detection time < 60 minutes
MTTR Resolution time - Detection time < 4 hours
Dwell Time Detection time - Initial compromise < 24 hours

NIST SP 800-61 Phases

Phase Activities
Preparation Playbooks, tools, training
Detection & Analysis Alert triage, scoping, evidence collection
Containment Short-term and long-term isolation
Eradication & Recovery Root cause removal, system restoration
Post-Incident Lessons learned, action items, metrics

Report Template Sections

Section Content
Executive Summary Impact, scope, duration
Timeline Chronological event sequence
Root Cause 5-Whys or fishbone analysis
Action Items Prioritized P1/P2/P3 with owners

External References

standards.md1.2 KB

Standards References - Post-Incident Lessons Learned

NIST SP 800-61 Rev. 2 - Section 3.4 Post-Incident Activity

  • 3.4.1: Lessons Learned meetings after each significant incident
  • 3.4.2: Using Collected Incident Data for trending and metrics
  • Recommends formal review within days of resolution

NIST SP 800-61 Rev. 3 - Continuous Improvement

  • Recover (RC) function: Learning from incidents
  • RC.CO-03: Recovery activities and progress communicated
  • Emphasis on continuous improvement of IR capabilities

SANS PICERL - Lessons Learned Phase

  • Phase 6: Final phase of incident handling
  • Formal review with all stakeholders
  • Document improvements and update procedures

MITRE ATT&CK - Detection Gap Analysis

  • Map incident techniques to ATT&CK framework
  • Identify detection gaps in current monitoring
  • Develop new detection rules based on observed TTPs

ISO 27001 - Clause 10: Improvement

  • 10.1: Nonconformity and corrective action
  • 10.2: Continual improvement
  • Requires organizations to learn from security incidents

Google SRE Post-Mortem Culture

  • Blameless approach to incident review
  • Focus on systemic issues rather than human error
  • Document and share learnings broadly
workflows.md1.9 KB

Post-Incident Lessons Learned - Detailed Workflow

Pre-Meeting Preparation (1-3 days before)

  1. Compile complete incident timeline from all sources
  2. Gather all communication logs (email, chat, phone)
  3. Export incident metrics from ticketing system
  4. Collect detection data from SIEM/EDR
  5. Identify all participants and send calendar invites

Meeting Facilitation Guide

Ground Rules

  1. Blameless discussion - focus on processes and systems
  2. Everyone's perspective is valued equally
  3. Objective review of facts, not opinions
  4. All observations documented in real-time
  5. Action items must have owners and deadlines

Discussion Framework

  1. What was the incident? (5 min) - Brief factual summary
  2. Walk the timeline (20 min) - Chronological event review
  3. What went well? (15 min) - Effective actions and decisions
  4. What could improve? (15 min) - Gaps and failures
  5. Root cause deep dive (15 min) - 5 Whys or fishbone diagram
  6. Action items (10 min) - Assigned improvements
  7. Playbook updates (10 min) - Procedural changes

Key Metrics Framework

Metric Formula Industry Benchmark
Dwell Time Detection - Initial Compromise Median: 10 days (Mandiant)
MTTD Triage Complete - First Alert Target: < 15 min (P1)
MTTC Containment Complete - Detection Target: < 4 hours
MTTR Recovery Complete - Eradication Target: < 48 hours
Total Duration Closure - Detection Target: < 7 days

Action Item Categories

Process

  • Updated playbooks and runbooks
  • Communication plan updates
  • Escalation criteria changes

Technology

  • New detection rules
  • Tool improvements
  • Monitoring expansion
  • Automation opportunities

People

  • Training needs
  • Staffing gaps
  • Cross-training requirements

Follow-Up Schedule

  • 1 week: Action items tracked in project system
  • 1 month: First progress review
  • 3 months: Validate improvements with tabletop
  • 6 months: Re-evaluate metrics

Scripts 2

agent.py5.7 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""Post-incident lessons learned analysis agent."""

import json
import argparse
from datetime import datetime


def analyze_incident_timeline(incident_data):
    """Analyze incident timeline for response gaps."""
    gaps = []
    if not incident_data:
        return gaps
    events = incident_data.get("timeline", [])
    for i in range(1, len(events)):
        prev_time = datetime.fromisoformat(events[i-1]["timestamp"])
        curr_time = datetime.fromisoformat(events[i]["timestamp"])
        delta_minutes = (curr_time - prev_time).total_seconds() / 60
        if delta_minutes > 30:
            gaps.append({
                "between": f"{events[i-1]['action']} -> {events[i]['action']}",
                "gap_minutes": round(delta_minutes),
                "severity": "HIGH" if delta_minutes > 120 else "MEDIUM",
                "recommendation": "Reduce response time with automated playbooks",
            })
    return gaps


def calculate_metrics(incident_data):
    """Calculate key incident response metrics."""
    timeline = incident_data.get("timeline", [])
    if len(timeline) < 2:
        return {}
    detect_time = None
    contain_time = None
    resolve_time = None
    for event in timeline:
        action = event.get("action", "").lower()
        ts = datetime.fromisoformat(event["timestamp"])
        if "detect" in action and not detect_time:
            detect_time = ts
        if "contain" in action and not contain_time:
            contain_time = ts
        if "resolve" in action or "close" in action:
            resolve_time = ts
    start = datetime.fromisoformat(timeline[0]["timestamp"])
    metrics = {}
    if detect_time:
        metrics["mttd_minutes"] = round((detect_time - start).total_seconds() / 60)
    if contain_time and detect_time:
        metrics["mttc_minutes"] = round((contain_time - detect_time).total_seconds() / 60)
    if resolve_time and detect_time:
        metrics["mttr_minutes"] = round((resolve_time - detect_time).total_seconds() / 60)
    return metrics


def generate_action_items(gaps, metrics):
    """Generate prioritized action items from analysis."""
    items = []
    if metrics.get("mttd_minutes", 0) > 60:
        items.append({
            "priority": "P1",
            "area": "Detection",
            "action": "Deploy automated detection rules to reduce MTTD below 30 minutes",
            "owner": "SOC Engineering",
        })
    if metrics.get("mttc_minutes", 0) > 120:
        items.append({
            "priority": "P1",
            "area": "Containment",
            "action": "Implement automated containment playbook in SOAR platform",
            "owner": "IR Team",
        })
    for gap in gaps:
        if gap["severity"] == "HIGH":
            items.append({
                "priority": "P2",
                "area": "Process",
                "action": f"Address {gap['gap_minutes']}min gap in {gap['between']}",
                "owner": "IR Manager",
            })
    items.append({
        "priority": "P3",
        "area": "Training",
        "action": "Schedule tabletop exercise within 30 days based on incident scenario",
        "owner": "Security Training",
    })
    return items


def generate_report_template():
    """Generate lessons learned report template."""
    return {
        "sections": [
            {"title": "Executive Summary", "content": "Brief overview of incident and impact"},
            {"title": "Incident Timeline", "content": "Chronological sequence of events"},
            {"title": "Root Cause Analysis", "content": "Underlying cause identification"},
            {"title": "What Went Well", "content": "Effective response actions"},
            {"title": "What Needs Improvement", "content": "Gaps and failures identified"},
            {"title": "Action Items", "content": "Prioritized remediation tasks with owners and deadlines"},
            {"title": "Metrics", "content": "MTTD, MTTC, MTTR measurements"},
            {"title": "Appendix", "content": "Supporting evidence, IOCs, detection rules"},
        ],
    }


def run_analysis(incident_file):
    """Execute post-incident lessons learned analysis."""
    print(f"\n{'='*60}")
    print(f"  POST-INCIDENT LESSONS LEARNED")
    print(f"  Generated: {datetime.utcnow().isoformat()} UTC")
    print(f"{'='*60}\n")

    incident_data = {}
    if incident_file:
        with open(incident_file, "r") as f:
            incident_data = json.load(f)

    metrics = calculate_metrics(incident_data)
    print(f"--- RESPONSE METRICS ---")
    for k, v in metrics.items():
        print(f"  {k}: {v} minutes")

    gaps = analyze_incident_timeline(incident_data)
    print(f"\n--- TIMELINE GAPS ({len(gaps)}) ---")
    for g in gaps:
        print(f"  [{g['severity']}] {g['between']}: {g['gap_minutes']} min gap")

    items = generate_action_items(gaps, metrics)
    print(f"\n--- ACTION ITEMS ({len(items)}) ---")
    for item in items:
        print(f"  [{item['priority']}] {item['area']}: {item['action']}")

    template = generate_report_template()
    print(f"\n--- REPORT SECTIONS ---")
    for s in template["sections"]:
        print(f"  - {s['title']}")

    return {"metrics": metrics, "gaps": gaps, "action_items": items, "template": template}


def main():
    parser = argparse.ArgumentParser(description="Post-Incident Lessons Learned Agent")
    parser.add_argument("--incident-file", help="Incident data JSON file")
    parser.add_argument("--output", help="Save report to JSON file")
    args = parser.parse_args()

    report = run_analysis(args.incident_file)
    if args.output:
        with open(args.output, "w") as f:
            json.dump(report, f, indent=2, default=str)
        print(f"\n[+] Report saved to {args.output}")


if __name__ == "__main__":
    main()
process.py9.2 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""
Post-Incident Lessons Learned Automation Script

Generates structured post-incident review reports including:
- Incident metrics calculation (MTTD, MTTC, MTTR)
- Timeline compilation
- Action item tracking
- Trend analysis across incidents

Requirements:
    pip install requests jinja2
"""

import argparse
import json
import logging
import os
from collections import Counter
from datetime import datetime, timezone
from typing import Optional

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger("lessons_learned")


class IncidentMetrics:
    """Calculate incident response metrics from timeline data."""

    def __init__(self, timeline: dict):
        self.timeline = timeline
        self.fmt = "%Y-%m-%dT%H:%M:%S"

    def _parse(self, key: str) -> Optional[datetime]:
        val = self.timeline.get(key)
        if val:
            try:
                return datetime.strptime(val, self.fmt)
            except ValueError:
                return datetime.fromisoformat(val)
        return None

    def calculate(self) -> dict:
        compromise = self._parse("compromise_time")
        detection = self._parse("detection_time")
        triage = self._parse("triage_time")
        containment = self._parse("containment_time")
        eradication = self._parse("eradication_time")
        recovery = self._parse("recovery_time")
        closure = self._parse("closure_time")

        metrics = {}
        if compromise and detection:
            metrics["dwell_time_hours"] = round((detection - compromise).total_seconds() / 3600, 2)
        if detection and triage:
            metrics["mttd_minutes"] = round((triage - detection).total_seconds() / 60, 2)
        if detection and containment:
            metrics["mttc_hours"] = round((containment - detection).total_seconds() / 3600, 2)
        if eradication and recovery:
            metrics["mttr_hours"] = round((recovery - eradication).total_seconds() / 3600, 2)
        if containment and eradication:
            metrics["eradication_hours"] = round((eradication - containment).total_seconds() / 3600, 2)
        if detection and closure:
            metrics["total_duration_hours"] = round((closure - detection).total_seconds() / 3600, 2)
            metrics["total_duration_days"] = round(metrics["total_duration_hours"] / 24, 1)

        return metrics


class RootCauseAnalyzer:
    """Structure root cause analysis using 5 Whys technique."""

    def __init__(self):
        self.whys = []

    def add_why(self, question: str, answer: str):
        self.whys.append({"level": len(self.whys) + 1, "question": question, "answer": answer})

    def get_root_cause(self) -> str:
        if self.whys:
            return self.whys[-1]["answer"]
        return "Root cause not determined"

    def to_dict(self) -> dict:
        return {
            "method": "5 Whys",
            "analysis": self.whys,
            "root_cause": self.get_root_cause(),
        }


class LessonsLearnedReport:
    """Generate comprehensive post-incident lessons learned report."""

    def __init__(self, incident_id: str):
        self.incident_id = incident_id
        self.report = {
            "incident_id": incident_id,
            "report_date": datetime.now(timezone.utc).isoformat(),
            "incident_summary": "",
            "timeline": {},
            "metrics": {},
            "what_worked": [],
            "what_failed": [],
            "root_cause_analysis": {},
            "action_items": [],
            "playbook_updates": [],
            "detection_improvements": [],
        }

    def set_summary(self, summary: str):
        self.report["incident_summary"] = summary

    def set_timeline(self, timeline: dict):
        self.report["timeline"] = timeline
        calculator = IncidentMetrics(timeline)
        self.report["metrics"] = calculator.calculate()

    def add_positive(self, item: str):
        self.report["what_worked"].append(item)

    def add_improvement(self, item: str):
        self.report["what_failed"].append(item)

    def set_root_cause(self, rca: RootCauseAnalyzer):
        self.report["root_cause_analysis"] = rca.to_dict()

    def add_action_item(self, title: str, owner: str, priority: str,
                        deadline: str, category: str):
        self.report["action_items"].append({
            "title": title,
            "owner": owner,
            "priority": priority,
            "deadline": deadline,
            "category": category,
            "status": "open",
        })

    def add_playbook_update(self, playbook: str, change: str):
        self.report["playbook_updates"].append({"playbook": playbook, "change": change})

    def add_detection_improvement(self, rule_name: str, description: str, technique: str):
        self.report["detection_improvements"].append({
            "rule_name": rule_name,
            "description": description,
            "mitre_technique": technique,
        })

    def generate_markdown(self) -> str:
        m = self.report["metrics"]
        md = f"# Post-Incident Lessons Learned Report\n\n"
        md += f"## Incident: {self.incident_id}\n"
        md += f"**Report Date:** {self.report['report_date']}\n\n"
        md += f"## Summary\n{self.report['incident_summary']}\n\n"

        md += f"## Response Metrics\n"
        md += f"| Metric | Value |\n|--------|-------|\n"
        for k, v in m.items():
            label = k.replace("_", " ").title()
            md += f"| {label} | {v} |\n"

        md += f"\n## What Worked Well\n"
        for item in self.report["what_worked"]:
            md += f"- {item}\n"

        md += f"\n## What Needs Improvement\n"
        for item in self.report["what_failed"]:
            md += f"- {item}\n"

        md += f"\n## Root Cause Analysis\n"
        rca = self.report["root_cause_analysis"]
        if rca:
            md += f"**Method:** {rca.get('method', 'N/A')}\n\n"
            for why in rca.get("analysis", []):
                md += f"**Why {why['level']}:** {why['question']}\n"
                md += f"  **Answer:** {why['answer']}\n\n"
            md += f"**Root Cause:** {rca.get('root_cause', 'N/A')}\n"

        md += f"\n## Action Items\n"
        md += f"| Title | Owner | Priority | Deadline | Status |\n"
        md += f"|-------|-------|----------|----------|--------|\n"
        for ai in self.report["action_items"]:
            md += f"| {ai['title']} | {ai['owner']} | {ai['priority']} | {ai['deadline']} | {ai['status']} |\n"

        return md

    def save(self, output_dir: str):
        os.makedirs(output_dir, exist_ok=True)
        json_path = os.path.join(output_dir, f"lessons_learned_{self.incident_id}.json")
        md_path = os.path.join(output_dir, f"lessons_learned_{self.incident_id}.md")

        with open(json_path, "w") as f:
            json.dump(self.report, f, indent=2)
        with open(md_path, "w") as f:
            f.write(self.generate_markdown())

        logger.info(f"Report saved: {json_path}")
        logger.info(f"Markdown saved: {md_path}")


class IncidentTrendAnalyzer:
    """Analyze trends across multiple incidents."""

    def __init__(self, incidents: list):
        self.incidents = incidents

    def analyze(self) -> dict:
        if not self.incidents:
            return {"error": "No incidents to analyze"}

        types = Counter(i.get("type", "unknown") for i in self.incidents)
        severities = Counter(i.get("severity", "unknown") for i in self.incidents)
        root_causes = Counter(i.get("root_cause_category", "unknown") for i in self.incidents)

        dwell_times = [i.get("dwell_time_hours", 0) for i in self.incidents if i.get("dwell_time_hours")]
        mttc_values = [i.get("mttc_hours", 0) for i in self.incidents if i.get("mttc_hours")]

        return {
            "total_incidents": len(self.incidents),
            "by_type": dict(types),
            "by_severity": dict(severities),
            "by_root_cause": dict(root_causes),
            "avg_dwell_time_hours": round(sum(dwell_times) / len(dwell_times), 2) if dwell_times else None,
            "avg_mttc_hours": round(sum(mttc_values) / len(mttc_values), 2) if mttc_values else None,
        }


def main():
    parser = argparse.ArgumentParser(description="Post-Incident Lessons Learned Generator")
    parser.add_argument("--incident-id", required=True, help="Incident ID")
    parser.add_argument("--summary", default="", help="Incident summary")
    parser.add_argument("--timeline-file", help="JSON file with incident timeline")
    parser.add_argument("--output-dir", default="./lessons_learned_output")

    args = parser.parse_args()

    report = LessonsLearnedReport(args.incident_id)
    report.set_summary(args.summary or f"Post-incident review for {args.incident_id}")

    if args.timeline_file and os.path.exists(args.timeline_file):
        with open(args.timeline_file) as f:
            timeline = json.load(f)
        report.set_timeline(timeline)
    else:
        logger.info("No timeline file provided. Create a JSON with keys: "
                     "compromise_time, detection_time, triage_time, containment_time, "
                     "eradication_time, recovery_time, closure_time")

    report.save(args.output_dir)
    print(f"Lessons learned report generated in: {args.output_dir}")


if __name__ == "__main__":
    main()

Assets 1

template.mdtext/markdown · 2.1 KB
Keep exploring