ot ics security

Implementing OT Incident Response Playbook

Develop and implement OT-specific incident response playbooks aligned with SANS PICERL framework, IEC 62443, and NIST SP 800-82 that address unique ICS challenges including safety-critical systems, limited downtime tolerance, and coordination between IT SOC, OT engineering, and plant operations teams.

icsiec62443incident-responsenistot-securityplaybooksafety-criticalsans
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

When to Use

  • When building OT-specific incident response procedures for the first time
  • When existing IT IR playbooks do not address ICS/SCADA-specific requirements
  • When preparing for OT ransomware scenarios like EKANS or LockerGoga
  • When aligning IR procedures with IEC 62443 and NERC CIP incident reporting requirements
  • When conducting post-incident reviews to improve OT IR capabilities

Do not use for IT-only incident response without OT components (use standard NIST 800-61 playbooks), for day-to-day OT security monitoring (see implementing-dragos-platform-for-ot-monitoring), or for tabletop exercise design (see performing-ics-tabletop-exercise).

Prerequisites

  • OT asset inventory with criticality ratings and safety system identification
  • Defined roles: OT IR Lead, IT SOC Analyst, Plant Operations Manager, Process Safety Engineer
  • Communication plan including out-of-band channels (OT incidents may compromise IT communications)
  • Known-good backups of PLC programs, HMI configurations, and historian data
  • Contact information for ICS vendors, Dragos/Claroty support, and CISA ICS-CERT

Workflow

Step 1: Define OT-Specific Incident Classification and Response Procedures

#!/usr/bin/env python3
"""OT Incident Response Playbook Engine.
 
Implements structured OT incident response procedures following
SANS PICERL lifecycle with ICS-specific considerations for safety,
availability, and cross-team coordination.
"""
 
import json
import sys
from datetime import datetime
from enum import Enum
from typing import Dict, List, Optional
 
 
class OTIncidentSeverity(Enum):
    SEV1_SAFETY = "SEV1-SAFETY"  # Safety system compromise
    SEV2_PROCESS = "SEV2-PROCESS"  # Active process manipulation
    SEV3_ACCESS = "SEV3-ACCESS"  # Unauthorized OT access
    SEV4_RECON = "SEV4-RECON"  # Reconnaissance in OT network
    SEV5_IT_SPILLOVER = "SEV5-IT-SPILLOVER"  # IT incident with OT exposure
 
 
class OTIncidentCategory(Enum):
    RANSOMWARE = "ransomware"
    MALWARE_ICS = "malware_ics_specific"
    UNAUTHORIZED_ACCESS = "unauthorized_ot_access"
    PROCESS_MANIPULATION = "process_manipulation"
    SIS_COMPROMISE = "safety_system_compromise"
    DATA_EXFILTRATION = "ot_data_exfiltration"
    SUPPLY_CHAIN = "supply_chain_compromise"
    INSIDER_THREAT = "insider_threat"
 
 
# PICERL phase definitions for OT
PICERL_PHASES = {
    "preparation": {
        "description": "Readiness activities before an incident occurs",
        "ot_specific": [
            "Maintain offline backups of all PLC programs and HMI configurations",
            "Document safe shutdown procedures for each process area",
            "Establish out-of-band communication (satellite phone, analog radio)",
            "Pre-stage forensic tools that work in air-gapped OT networks",
            "Maintain spare PLCs and engineering workstations",
            "Conduct quarterly OT tabletop exercises",
        ],
    },
    "identification": {
        "description": "Detect and confirm the OT security incident",
        "ot_specific": [
            "Correlate OT IDS alerts with process anomalies from historian data",
            "Verify if process deviations are cyber-caused vs operational",
            "Check Safety Instrumented Systems (SIS) status and integrity",
            "Review engineering workstation logs for unauthorized access",
            "Examine PLC mode changes (RUN/STOP/PROGRAM transitions)",
            "Assess whether the incident is IT-only or has crossed into OT",
        ],
    },
    "containment": {
        "description": "Limit the spread and impact of the incident",
        "ot_specific": [
            "NEVER shut down OT systems without plant operations approval",
            "Isolate affected segments at the industrial firewall (not by powering off)",
            "Switch PLCs to LOCAL/MANUAL mode if remote manipulation is suspected",
            "Disconnect IT-OT conduits at the DMZ while maintaining intra-OT communication",
            "Preserve forensic evidence before any remediation actions",
            "Maintain safety system operation throughout containment",
        ],
    },
    "eradication": {
        "description": "Remove the threat from OT systems",
        "ot_specific": [
            "Compare running PLC programs against known-good backups",
            "Rebuild compromised engineering workstations from golden images",
            "Verify historian data integrity for evidence of manipulation",
            "Check for persistence mechanisms in OT-specific locations (startup scripts, scheduled tasks on HMIs)",
            "Validate firmware integrity on PLCs and RTUs",
            "Coordinate with ICS vendor for rootkit-level remediation if needed",
        ],
    },
    "recovery": {
        "description": "Restore OT operations to normal",
        "ot_specific": [
            "Restore PLC programs from verified offline backups",
            "Bring processes back online in stages with engineering oversight",
            "Monitor process variables closely during restart for anomalies",
            "Validate safety system functionality before resuming automatic operation",
            "Re-enable IT-OT connectivity only after OT is verified clean",
            "Document any process variable drift during the incident",
        ],
    },
    "lessons_learned": {
        "description": "Post-incident review and improvement",
        "ot_specific": [
            "Conduct joint IT/OT post-incident review within 2 weeks",
            "Update detection rules based on observed attack techniques",
            "Revise network segmentation if lateral movement was successful",
            "Update PLC backup schedules based on recovery time experienced",
            "Report to CISA ICS-CERT and sector ISAC as required",
            "Test updated playbook within 90 days",
        ],
    },
}
 
 
class OTIncident:
    """Represents an active OT security incident."""
 
    def __init__(self, title: str, severity: OTIncidentSeverity,
                 category: OTIncidentCategory, affected_systems: List[str]):
        self.id = f"OT-IR-{datetime.now().strftime('%Y%m%d-%H%M%S')}"
        self.title = title
        self.severity = severity
        self.category = category
        self.affected_systems = affected_systems
        self.created = datetime.now().isoformat()
        self.current_phase = "identification"
        self.timeline = []
        self.decisions = []
        self.containment_actions = []
 
    def log_event(self, phase: str, action: str, actor: str, notes: str = ""):
        """Log an incident response action."""
        entry = {
            "timestamp": datetime.now().isoformat(),
            "phase": phase,
            "action": action,
            "actor": actor,
            "notes": notes,
        }
        self.timeline.append(entry)
        return entry
 
    def log_decision(self, decision: str, rationale: str, approved_by: str):
        """Log a critical decision during incident response."""
        entry = {
            "timestamp": datetime.now().isoformat(),
            "decision": decision,
            "rationale": rationale,
            "approved_by": approved_by,
        }
        self.decisions.append(entry)
        return entry
 
 
class OTPlaybookEngine:
    """Executes OT incident response playbooks."""
 
    def __init__(self):
        self.playbooks = self._build_playbooks()
 
    def _build_playbooks(self) -> Dict:
        """Build category-specific OT IR playbooks."""
        return {
            OTIncidentCategory.RANSOMWARE: {
                "name": "OT Ransomware Response",
                "reference": "SANS ICS Ransomware Defense Playbook",
                "immediate_actions": [
                    "DO NOT pay ransom without executive and legal approval",
                    "Disconnect IT-OT conduit at DMZ firewalls immediately",
                    "Verify SIS/safety systems are operating independently",
                    "Switch critical processes to manual/local control",
                    "Preserve ransom note and encrypted file samples for forensics",
                    "Assess if ransomware has reached Level 2 or below",
                ],
                "containment_steps": [
                    "Block lateral movement by disabling SMB/RDP between OT hosts",
                    "Isolate affected VLANs while maintaining critical process communication",
                    "Disable remote access VPN to OT environment",
                    "Check if backup infrastructure is intact (ransomware targets backups)",
                    "Inventory which OT systems are encrypted vs still operational",
                ],
                "recovery_priority": [
                    "1. Safety Instrumented Systems (SIS)",
                    "2. Critical process controllers (PLCs in continuous process)",
                    "3. HMIs for operator visibility",
                    "4. Historian for data continuity",
                    "5. Engineering workstations",
                    "6. IT-OT connectivity (last)",
                ],
                "reporting": [
                    "CISA: report within 72 hours per CIRCIA",
                    "Sector ISAC: share IOCs within 24 hours",
                    "NERC (if applicable): report within 1 hour for BES impact",
                ],
            },
            OTIncidentCategory.SIS_COMPROMISE: {
                "name": "Safety System Compromise Response",
                "reference": "TRITON/TRISIS Incident Lessons Learned",
                "immediate_actions": [
                    "IMMEDIATELY alert Process Safety team",
                    "Verify physical safety devices are functional (relief valves, rupture discs)",
                    "Consider controlled process shutdown if SIS integrity is uncertain",
                    "Isolate SIS network from all other networks",
                    "Check if SIS is in bypass mode or has been disarmed",
                    "Engage SIS vendor emergency support (Schneider Triconex, HIMA, etc)",
                ],
                "containment_steps": [
                    "Physically disconnect the SIS engineering workstation from network",
                    "Capture forensic image of SIS engineering workstation",
                    "Verify SIS controller firmware and logic against factory baseline",
                    "Check for unauthorized TriStation/safety protocol connections",
                    "Inspect all engineering workstations for TRITON indicators",
                ],
                "recovery_priority": [
                    "1. Verify all physical safety barriers are intact",
                    "2. Reload SIS logic from offline backup (verified against vendor baseline)",
                    "3. Full SIS proof test before returning to service",
                    "4. Independent verification by process safety engineer",
                ],
                "reporting": [
                    "CISA ICS-CERT: immediate notification for SIS-targeting attack",
                    "Process safety regulator (OSHA, HSE): as required by jurisdiction",
                    "SIS vendor: engage for root cause analysis",
                ],
            },
        }
 
    def execute_playbook(self, incident: OTIncident):
        """Execute the appropriate playbook for an incident."""
        playbook = self.playbooks.get(incident.category)
        if not playbook:
            print(f"[!] No specific playbook for {incident.category.value}. Using generic OT IR procedures.")
            return
 
        print(f"\n{'='*70}")
        print(f"OT INCIDENT RESPONSE PLAYBOOK ACTIVATED")
        print(f"{'='*70}")
        print(f"Incident ID: {incident.id}")
        print(f"Title: {incident.title}")
        print(f"Severity: {incident.severity.value}")
        print(f"Category: {incident.category.value}")
        print(f"Playbook: {playbook['name']}")
        print(f"Reference: {playbook['reference']}")
        print(f"Activated: {incident.created}")
        print(f"Affected Systems: {', '.join(incident.affected_systems)}")
 
        print(f"\n--- IMMEDIATE ACTIONS (Execute within first 15 minutes) ---")
        for i, action in enumerate(playbook["immediate_actions"], 1):
            print(f"  {i}. {action}")
 
        print(f"\n--- CONTAINMENT STEPS ---")
        for i, step in enumerate(playbook["containment_steps"], 1):
            print(f"  {i}. {step}")
 
        print(f"\n--- RECOVERY PRIORITY ORDER ---")
        for item in playbook["recovery_priority"]:
            print(f"  {item}")
 
        print(f"\n--- REPORTING REQUIREMENTS ---")
        for req in playbook["reporting"]:
            print(f"  - {req}")
 
        # Print PICERL phase guidance
        print(f"\n--- PICERL PHASE CHECKLIST ---")
        for phase, info in PICERL_PHASES.items():
            print(f"\n  [{phase.upper()}] {info['description']}")
            for item in info["ot_specific"][:3]:
                print(f"    - {item}")
 
 
if __name__ == "__main__":
    engine = OTPlaybookEngine()
 
    # Example: OT Ransomware incident
    incident = OTIncident(
        title="Ransomware detected on Level 3 historian servers",
        severity=OTIncidentSeverity.SEV2_PROCESS,
        category=OTIncidentCategory.RANSOMWARE,
        affected_systems=["HIST-01", "HIST-02", "ENG-WS-03", "HMI-AREA1"],
    )
 
    engine.execute_playbook(incident)

Key Concepts

Term Definition
PICERL SANS incident response lifecycle: Preparation, Identification, Containment, Eradication, Recovery, Lessons Learned
ICS4ICS Incident Command System for Industrial Control Systems -- adapts FEMA ICS to OT cybersecurity response
Safety Instrumented System (SIS) Independent safety controller that prevents hazardous conditions; compromising SIS can cause physical harm
Manual/Local Mode Operating PLCs with local panel controls instead of remote SCADA; used when remote access is compromised
CIRCIA Cyber Incident Reporting for Critical Infrastructure Act requiring reporting to CISA within 72 hours
Known-Good Backup Verified, offline copy of PLC programs and configurations used as the trusted baseline for recovery

Common Scenarios

Scenario: Ransomware Spreads from IT to OT Level 3

Context: Ransomware encrypts enterprise IT systems and spreads through an inadequately protected IT/OT conduit to Level 3 historian servers. HMIs at Level 2 begin showing connectivity errors.

Approach:

  1. Activate the OT IR playbook for ransomware immediately
  2. Sever IT-OT connectivity at the DMZ firewall (both north and south firewalls)
  3. Verify PLCs are still running and process is stable (PLCs run independently of IT)
  4. Switch operators to local HMI panels if networked HMIs are affected
  5. Assess which Level 2/3 systems are encrypted vs operational
  6. Prioritize restoring HMI visibility, then historian, then engineering workstations
  7. Restore from offline backups -- never attempt to decrypt using attacker-provided tools without sandbox testing
  8. Report to CISA within 72 hours per CIRCIA requirements

Pitfalls: Do not shut down PLCs to "protect" them from ransomware -- PLCs run firmware, not Windows, and are typically unaffected by ransomware. Shutting down PLCs disrupts the physical process. Never reconnect IT-OT conduit until the IT side is fully remediated.

Output Format

OT INCIDENT RESPONSE REPORT
==============================
Incident ID: OT-IR-YYYYMMDD-HHMMSS
Severity: SEV[1-5]
Category: [category]
Status: [Active/Contained/Eradicated/Recovered/Closed]
 
TIMELINE:
  [timestamp] - [phase] - [action] - [actor]
 
AFFECTED SYSTEMS:
  Safety Systems: [status]
  Process Controllers: [status]
  HMI/SCADA: [status]
  Historian: [status]
 
DECISIONS LOG:
  [timestamp] - [decision] - [rationale] - [approver]
 
CONTAINMENT ACTIONS TAKEN:
  1. [action and timestamp]
 
RECOVERY STATUS:
  [system] - [restored/pending] - [ETA]
Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 1

api-reference.md2.6 KB

API Reference: Implementing OT Incident Response Playbook

ICS-CERT Incident Categories

Category Severity Description Response Time
Unauthorized Access P1 - Critical PLC/HMI/SIS unauthorized access Immediate
Malware/Ransomware P1 - Critical OT network malware (EKANS, Triton) Immediate
DoS/DDoS P1 - Critical OT communication disruption Immediate
Network Intrusion P2 - High IT-OT boundary breach < 4 hours
Reconnaissance P3 - Medium OT network scanning detected < 24 hours
Policy Violation P4 - Low Unauthorized configuration change < 72 hours

Purdue Model Containment Zones

Level Name Containment Action
L0 Physical Process Manual control, verify SIS
L1 Basic Control (PLC, SIS) Isolate network, do NOT power off
L2 Supervisory (HMI, SCADA) Disconnect HMI, activate backup
L3 Operations (Historian) Isolate segment, preserve logs
L3.5 DMZ Sever IT-OT bridge
L4-5 Enterprise IT Standard IT IR procedures

NIST SP 800-82 IR Controls

Control Title OT Consideration
IR-1 IR Policy Must address safety-critical systems
IR-4 Incident Handling Include OT engineering team
IR-5 Incident Monitoring Passive monitoring only in OT
IR-6 Incident Reporting CISA ICS-CERT within 72 hours
IR-8 IR Plan Separate OT and IT playbooks

SANS PICERL Framework for OT

Phase OT-Specific Actions
Preparation Maintain PLC backup programs, define safe states
Identification Correlate OT alerts with process anomalies
Containment Network isolation without process disruption
Eradication Restore from known-good PLC/HMI configurations
Recovery Staged restart with operator verification
Lessons Learned Update OT-specific TTPs and detection rules

Reporting Obligations

Authority Timeframe Trigger
CISA ICS-CERT 72 hours Critical infrastructure impact
Sector ISAC 48 hours Sector-relevant threat
TSA (pipeline) 12 hours Pipeline cybersecurity incident
NERC (electric) 1 hour Cyber Security Incident

References

Scripts 1

agent.py5.7 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""OT Incident Response Playbook Agent - executes ICS/SCADA incident response procedures."""

import json
import argparse
import logging
from datetime import datetime

logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)

OT_INCIDENT_TYPES = {
    "unauthorized_plc_access": {"severity": "critical", "safety_impact": True},
    "hmi_compromise": {"severity": "critical", "safety_impact": True},
    "historian_breach": {"severity": "high", "safety_impact": False},
    "engineering_workstation_malware": {"severity": "critical", "safety_impact": True},
    "network_anomaly": {"severity": "medium", "safety_impact": False},
    "firmware_tampering": {"severity": "critical", "safety_impact": True},
    "ransomware_ot": {"severity": "critical", "safety_impact": True},
    "dos_scada": {"severity": "high", "safety_impact": True},
}


def assess_incident(incident_type, affected_assets):
    incident_info = OT_INCIDENT_TYPES.get(incident_type, {"severity": "medium", "safety_impact": False})
    has_safety_system = any(a.get("type") in ("SIS", "safety_plc", "emergency_shutdown") for a in affected_assets)
    return {
        "incident_type": incident_type,
        "base_severity": incident_info["severity"],
        "safety_impact": incident_info["safety_impact"] or has_safety_system,
        "affected_assets": len(affected_assets),
        "escalated_severity": "critical" if has_safety_system else incident_info["severity"],
        "requires_plant_shutdown": has_safety_system and incident_info["safety_impact"],
    }


def generate_containment_steps(incident_type, assessment):
    steps = [
        {"step": 1, "action": "Notify OT operations and plant safety manager", "priority": "immediate"},
        {"step": 2, "action": "Verify safety instrumented systems (SIS) are operational", "priority": "immediate"},
        {"step": 3, "action": "Document current process state and control values", "priority": "immediate"},
    ]
    if incident_type in ("unauthorized_plc_access", "firmware_tampering"):
        steps.extend([
            {"step": 4, "action": "Isolate affected PLCs from network (do NOT power off)", "priority": "high"},
            {"step": 5, "action": "Switch affected processes to manual control", "priority": "high"},
            {"step": 6, "action": "Verify PLC program integrity against known-good backup", "priority": "high"},
        ])
    elif incident_type == "ransomware_ot":
        steps.extend([
            {"step": 4, "action": "Isolate IT/OT boundary immediately", "priority": "immediate"},
            {"step": 5, "action": "Verify Level 0-1 devices are unaffected", "priority": "immediate"},
            {"step": 6, "action": "Preserve forensic evidence from affected HMIs", "priority": "high"},
        ])
    elif incident_type == "hmi_compromise":
        steps.extend([
            {"step": 4, "action": "Disconnect compromised HMI from control network", "priority": "immediate"},
            {"step": 5, "action": "Activate backup HMI or manual operations", "priority": "high"},
        ])
    else:
        steps.extend([
            {"step": 4, "action": "Isolate affected network segment", "priority": "high"},
            {"step": 5, "action": "Enable enhanced monitoring on OT network", "priority": "medium"},
        ])
    return steps


def generate_recovery_plan(incident_type, affected_assets):
    return {
        "pre_recovery_checks": [
            "Verify all safety systems are functional",
            "Confirm process is in safe state",
            "Validate backup integrity before restoration",
        ],
        "recovery_steps": [
            "Restore from known-good configuration backups",
            "Re-validate PLC programs against engineering documentation",
            "Perform staged restart with operator verification",
            "Monitor process values against baseline for 24 hours",
        ],
        "post_recovery_validation": [
            "Compare process parameters to pre-incident baseline",
            "Run safety system functional tests",
            "Verify all control loops are operating correctly",
        ],
    }


def generate_report(assessment, containment, recovery):
    return {
        "timestamp": datetime.utcnow().isoformat(),
        "playbook": "OT Incident Response",
        "incident_assessment": assessment,
        "containment_steps": containment,
        "recovery_plan": recovery,
        "regulatory_notifications": [
            "CISA ICS-CERT (within 72 hours for critical infrastructure)",
            "Sector-specific ISAC notification",
        ] if assessment["safety_impact"] else [],
    }


def main():
    parser = argparse.ArgumentParser(description="OT Incident Response Playbook Agent")
    parser.add_argument("--incident-type", required=True, choices=list(OT_INCIDENT_TYPES.keys()))
    parser.add_argument("--affected-assets", help="JSON file listing affected assets")
    parser.add_argument("--output", default="ot_ir_playbook.json")
    args = parser.parse_args()

    assets = []
    if args.affected_assets:
        with open(args.affected_assets) as f:
            assets = json.load(f)

    assessment = assess_incident(args.incident_type, assets)
    containment = generate_containment_steps(args.incident_type, assessment)
    recovery = generate_recovery_plan(args.incident_type, assets)
    report = generate_report(assessment, containment, recovery)
    with open(args.output, "w") as f:
        json.dump(report, f, indent=2, default=str)
    logger.info("OT IR: %s, severity %s, safety impact: %s",
                args.incident_type, assessment["escalated_severity"], assessment["safety_impact"])
    print(json.dumps(report, indent=2, default=str))


if __name__ == "__main__":
    main()
Keep exploring