incident response

Testing Ransomware Recovery Procedures

Test and validate ransomware recovery procedures including backup restore operations, RTO/RPO target verification, recovery sequencing, and clean restore validation to ensure organizational resilience against destructive ransomware attacks.

backupdisaster-recoveryincident-responseransomwareresiliencerporto
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

When to Use

Use this skill when:

  • Validating that ransomware recovery plans actually work under realistic conditions
  • Measuring RTO (Recovery Time Objective) and RPO (Recovery Point Objective) against business requirements
  • Testing backup restore operations to confirm data integrity and completeness after simulated encryption
  • Conducting tabletop exercises or live recovery drills for ransomware scenarios
  • Auditing disaster recovery readiness as part of compliance or cyber insurance requirements

Do not use for active incident response during a live ransomware attack. Use dedicated IR playbooks instead.

Prerequisites

  • Isolated recovery test environment (air-gapped or network-segmented lab)
  • Access to backup infrastructure (Veeam, Commvault, Rubrik, AWS Backup, Azure Backup)
  • Documented RTO/RPO targets per application tier from business impact analysis
  • Backup copies available for restore testing (production replicas or test snapshots)
  • Recovery runbooks with step-by-step procedures for each critical system

Workflow

Step 1: Define Recovery Test Scope

Identify critical systems and their tiered recovery targets:

Tier System Type RTO Target RPO Target Example
Tier 1 Mission-critical < 1 hour < 15 min Active Directory, core database
Tier 2 Business-critical < 4 hours < 1 hour ERP, email, CRM
Tier 3 Business-operational < 24 hours < 4 hours File shares, internal apps
Tier 4 Non-critical < 72 hours < 24 hours Dev/test, analytics

Step 2: Prepare Test Environment

# Verify isolated recovery network is segmented
# No routes to production should exist
ip route show | grep -v "192.168.100.0/24"  # recovery VLAN only
 
# Verify backup catalog is accessible
restic snapshots --repo s3:s3.amazonaws.com/backup-bucket --password-file /etc/restic/pw
# Or for Veeam:
# Get-VBRBackup | Where-Object {$_.JobType -eq "Backup"} | Select Name, LastPointCreationTime

Step 3: Execute Restore and Measure RTO

For each tiered system, measure the full recovery timeline:

  1. Detection to Decision - Time from simulated alert to restore decision
  2. Backup Locate - Time to identify and select the correct clean restore point
  3. Restore Execution - Time to restore data/VM/application from backup
  4. Validation - Time to verify data integrity and application functionality
  5. Service Restoration - Time until the system is fully operational
Recovery Timeline Measurement:
  T0: Incident declared (simulated ransomware detection)
  T1: Recovery team assembled and backup identified
  T2: Restore initiated from clean backup
  T3: Restore completed, integrity checks passed
  T4: Application validated and service restored
 
  Actual RTO = T4 - T0
  Actual RPO = T0 - backup_timestamp

Step 4: Validate Data Integrity Post-Restore

# Compare file counts between backup manifest and restored data
find /restored/data -type f | wc -l
# Compare against pre-backup manifest
 
# Verify database consistency after restore
pg_isready -h localhost -p 5432
psql -c "SELECT count(*) FROM critical_table;" -d restored_db
 
# Hash verification of critical files
sha256sum /restored/data/critical_config.xml
# Compare against known-good hash from backup manifest

Step 5: Test Credential Rotation and Security Hardening

After restore, validate that security controls are re-established:

  1. Rotate all service account passwords and API keys
  2. Verify MFA is enabled on all administrative accounts
  3. Confirm EDR/AV agents are running and reporting to management console
  4. Validate firewall rules block known C2 indicators
  5. Check that restored systems have latest security patches

Step 6: Document Results and Calculate Gap

Recovery Test Report:
  System: [Name]
  Tier: [1-4]
  RTO Target: [target]    Actual RTO: [measured]    Gap: [delta]
  RPO Target: [target]    Actual RPO: [measured]    Gap: [delta]
  Data Integrity: [PASS/FAIL]
  Application Validation: [PASS/FAIL]
  Security Controls Restored: [PASS/FAIL]
 
  Status: [MEETS TARGET / EXCEEDS TARGET / FAILS TARGET]
  Remediation Required: [description if FAILS]

Key Concepts

Term Definition
RTO Recovery Time Objective: maximum acceptable downtime for a system after a disaster
RPO Recovery Point Objective: maximum acceptable data loss measured in time
WRT Work Recovery Time: time to verify system integrity after restore completes
MTD Maximum Tolerable Downtime: absolute limit before unacceptable business impact
Clean Restore Point A backup verified to be free of ransomware artifacts or encryption
Recovery Sequencing The order in which interdependent systems must be restored
Air-Gapped Backup Backup stored on media physically disconnected from the network

Tools & Systems

Tool Purpose
Veeam Backup & Replication VM and physical server backup and restore
Commvault Enterprise data protection and recovery orchestration
Rubrik Cloud-native backup with ransomware recovery SLA
AWS Backup Centralized backup for AWS services
Azure Backup Microsoft cloud backup with immutable vault
Restic Open-source encrypted backup tool
Velero Kubernetes cluster backup and restore

Common Pitfalls

  • Not testing restores regularly: Backups that are never tested often fail when needed. Test quarterly at minimum.
  • Ignoring recovery sequencing: Restoring an application before its database dependency causes cascading failures.
  • Skipping credential rotation: Restored systems may contain compromised credentials that allow re-infection.
  • Using production network for testing: Recovery tests on production networks risk spreading simulated or real infections.
  • Measuring RTO without WRT: Restore completion is not recovery completion. Include validation and hardening time.
  • No immutable backups: If ransomware can encrypt or delete backups, recovery is impossible. Use air-gapped or immutable storage.

References

Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 1

api-reference.md3.8 KB

API Reference: Testing Ransomware Recovery Procedures

CLI Usage

# Generate hash manifest for a directory (pre-backup baseline)
python agent.py --hash-dir /data/critical-app -o manifest_baseline.json
 
# Compare original manifest against restored data
python agent.py --compare manifest_baseline.json manifest_restored.json
 
# Check if a service is running after restore
python agent.py --check-service postgresql
 
# Check database connectivity after restore
python agent.py --check-db postgresql:localhost:5432
 
# Run full recovery drill from config
python agent.py --config drill_config.json -o recovery_report.json

Drill Configuration Format

{
  "systems": [
    {
      "name": "core-database",
      "tier": 1,
      "rto_target_seconds": 3600,
      "rpo_target_seconds": 900,
      "backup_timestamp_epoch": 1711000000,
      "restore_directory": "/restored/core-db",
      "manifest_file": "/manifests/core-db-baseline.json",
      "services": ["postgresql"],
      "database": {
        "type": "postgresql",
        "host": "localhost",
        "port": 5432
      }
    },
    {
      "name": "web-application",
      "tier": 2,
      "rto_target_seconds": 14400,
      "rpo_target_seconds": 3600,
      "restore_directory": "/restored/webapp",
      "services": ["nginx", "gunicorn"]
    }
  ]
}

Recovery Phases Tracked

Phase Timestamp Key Description
Incident Declaration incident_declared Simulated ransomware detection time
Backup Identification backup_identified Clean restore point located
Restore Initiated restore_initiated Backup restore process started
Restore Completed restore_completed Data fully written to target
Service Restored service_restored Application validated and operational

RTO/RPO Calculation

Actual RTO = service_restored - incident_declared
Actual RPO = incident_declared - backup_timestamp
 
RTO Met = Actual RTO <= RTO Target
RPO Met = Actual RPO <= RPO Target

Tier Definitions

Tier RTO Range RPO Range System Classification
1 < 1 hour < 15 min Mission-critical (AD, core DB)
2 < 4 hours < 1 hour Business-critical (ERP, email)
3 < 24 hours < 4 hours Business-operational (file shares)
4 < 72 hours < 24 hours Non-critical (dev/test, analytics)

Hash Manifest Format

{
  "config/app.yaml": "a3f2b8c9d1e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0",
  "data/users.db": "1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2",
  "bin/server": "PERMISSION_DENIED"
}

Validation Checks

Check Description Pass Criteria
file_count Files present in restored directory count > 0
integrity_check Hash comparison vs baseline manifest No missing or modified files
service_* System service running post-restore Service status is RUNNING/active
database_connectivity Database port reachable TCP connection succeeds

Report Output Schema

{
  "report_date": "2026-03-19T12:00:00+00:00",
  "drill_type": "ransomware_recovery_validation",
  "systems_tested": 2,
  "systems_meeting_rto": 2,
  "systems_meeting_rpo": 1,
  "overall_pass": false,
  "results": [
    {
      "system_name": "core-database",
      "tier": 1,
      "rto_target_seconds": 3600,
      "actual_rto_seconds": 2400.5,
      "rto_met": true,
      "rpo_met": true,
      "validations": {},
      "errors": []
    }
  ]
}

References

Scripts 1

agent.py11.3 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""Agent for testing and validating ransomware recovery procedures.

Measures RTO/RPO against targets, validates backup restore integrity,
tracks recovery sequencing, and generates compliance reports.
"""

import argparse
import hashlib
import json
import os
import subprocess
import sys
import time
from datetime import datetime, timezone
from pathlib import Path


class RecoveryTest:
    """Represents a single system recovery test with timing and validation."""

    def __init__(self, system_name, tier, rto_target_seconds, rpo_target_seconds):
        self.system_name = system_name
        self.tier = tier
        self.rto_target = rto_target_seconds
        self.rpo_target = rpo_target_seconds
        self.timestamps = {}
        self.validations = {}
        self.errors = []

    def mark(self, phase):
        """Record a timestamp for a recovery phase."""
        self.timestamps[phase] = time.time()

    def validate(self, check_name, passed, detail=""):
        """Record a validation result."""
        self.validations[check_name] = {"passed": passed, "detail": detail}

    def actual_rto(self):
        """Calculate actual RTO from incident declaration to service restored."""
        t0 = self.timestamps.get("incident_declared")
        t4 = self.timestamps.get("service_restored")
        if t0 and t4:
            return t4 - t0
        return None

    def actual_rpo(self, backup_timestamp_epoch):
        """Calculate actual RPO from last backup to incident declaration."""
        t0 = self.timestamps.get("incident_declared")
        if t0 and backup_timestamp_epoch:
            return t0 - backup_timestamp_epoch
        return None

    def to_dict(self, backup_timestamp_epoch=None):
        rto = self.actual_rto()
        rpo = self.actual_rpo(backup_timestamp_epoch)
        return {
            "system_name": self.system_name,
            "tier": self.tier,
            "rto_target_seconds": self.rto_target,
            "rpo_target_seconds": self.rpo_target,
            "actual_rto_seconds": round(rto, 2) if rto else None,
            "actual_rpo_seconds": round(rpo, 2) if rpo else None,
            "rto_met": rto <= self.rto_target if rto else None,
            "rpo_met": rpo <= self.rpo_target if rpo else None,
            "timestamps": {
                k: datetime.fromtimestamp(v, tz=timezone.utc).isoformat()
                for k, v in self.timestamps.items()
            },
            "validations": self.validations,
            "errors": self.errors,
        }


def compute_file_hashes(directory, algorithm="sha256"):
    """Compute hashes for all files in a directory for integrity verification."""
    hashes = {}
    dir_path = Path(directory)
    if not dir_path.is_dir():
        return {"error": f"Directory not found: {directory}"}

    for fpath in sorted(dir_path.rglob("*")):
        if fpath.is_file():
            h = hashlib.new(algorithm)
            try:
                with open(fpath, "rb") as f:
                    for chunk in iter(lambda: f.read(65536), b""):
                        h.update(chunk)
                rel = str(fpath.relative_to(dir_path))
                hashes[rel] = h.hexdigest()
            except PermissionError:
                hashes[str(fpath.relative_to(dir_path))] = "PERMISSION_DENIED"
    return hashes


def compare_manifests(original_manifest, restored_manifest):
    """Compare two hash manifests to detect missing, added, or changed files."""
    missing = []
    modified = []
    added = []

    for fname, orig_hash in original_manifest.items():
        if fname not in restored_manifest:
            missing.append(fname)
        elif restored_manifest[fname] != orig_hash:
            modified.append(fname)

    for fname in restored_manifest:
        if fname not in original_manifest:
            added.append(fname)

    return {
        "total_original": len(original_manifest),
        "total_restored": len(restored_manifest),
        "missing_files": missing,
        "modified_files": modified,
        "added_files": added,
        "integrity_pass": len(missing) == 0 and len(modified) == 0,
    }


def check_service_health(service_name):
    """Check if a service is running and responsive."""
    if sys.platform == "win32":
        try:
            result = subprocess.run(
                ["sc", "query", service_name],
                capture_output=True, text=True, timeout=10
            )
            running = "RUNNING" in result.stdout
            return {"service": service_name, "running": running, "platform": "windows"}
        except (subprocess.SubprocessError, FileNotFoundError):
            return {"service": service_name, "running": False, "error": "check failed"}
    else:
        try:
            result = subprocess.run(
                ["systemctl", "is-active", service_name],
                capture_output=True, text=True, timeout=10
            )
            active = result.stdout.strip() == "active"
            return {"service": service_name, "running": active, "platform": "linux"}
        except (subprocess.SubprocessError, FileNotFoundError):
            return {"service": service_name, "running": False, "error": "check failed"}


def check_database_connectivity(db_type, host="localhost", port=None):
    """Verify database is accessible after restore."""
    ports = {"postgresql": 5432, "mysql": 3306, "mssql": 1433}
    port = port or ports.get(db_type, 5432)

    import socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.settimeout(5)
    try:
        result = sock.connect_ex((host, port))
        return {
            "database": db_type,
            "host": host,
            "port": port,
            "reachable": result == 0,
        }
    except socket.error as e:
        return {"database": db_type, "host": host, "port": port, "reachable": False,
                "error": str(e)}
    finally:
        sock.close()


def run_recovery_drill(config):
    """Execute a recovery drill based on a configuration dict."""
    results = []

    for system in config.get("systems", []):
        test = RecoveryTest(
            system_name=system["name"],
            tier=system.get("tier", 3),
            rto_target_seconds=system.get("rto_target_seconds", 14400),
            rpo_target_seconds=system.get("rpo_target_seconds", 3600),
        )

        test.mark("incident_declared")
        print(f"[*] Recovery drill started for: {system['name']}")

        # Phase: Locate backup
        test.mark("backup_identified")
        backup_ts = system.get("backup_timestamp_epoch", time.time() - 3600)

        # Phase: Validate restore directory if provided
        restore_dir = system.get("restore_directory")
        if restore_dir and os.path.isdir(restore_dir):
            test.mark("restore_initiated")
            hashes = compute_file_hashes(restore_dir)
            file_count = len([v for v in hashes.values() if v != "PERMISSION_DENIED"])
            test.validate("file_count", file_count > 0,
                          f"{file_count} files found in restored directory")
            test.mark("restore_completed")

            # Compare with manifest if provided
            manifest_path = system.get("manifest_file")
            if manifest_path and os.path.isfile(manifest_path):
                with open(manifest_path, "r") as f:
                    original_manifest = json.load(f)
                comparison = compare_manifests(original_manifest, hashes)
                test.validate("integrity_check", comparison["integrity_pass"],
                              json.dumps(comparison, indent=2))
        else:
            test.validate("restore_directory", False,
                          f"Directory not found: {restore_dir}")

        # Phase: Check services
        for svc in system.get("services", []):
            health = check_service_health(svc)
            test.validate(f"service_{svc}", health.get("running", False),
                          json.dumps(health))

        # Phase: Check database
        db = system.get("database")
        if db:
            db_check = check_database_connectivity(
                db.get("type", "postgresql"),
                db.get("host", "localhost"),
                db.get("port"),
            )
            test.validate("database_connectivity", db_check["reachable"],
                          json.dumps(db_check))

        test.mark("service_restored")
        results.append(test.to_dict(backup_ts))
        print(f"[*] Recovery drill completed for: {system['name']}")

    return results


def generate_report(results, output_path=None):
    """Generate a recovery test report."""
    report = {
        "report_date": datetime.now(timezone.utc).isoformat(),
        "drill_type": "ransomware_recovery_validation",
        "systems_tested": len(results),
        "systems_meeting_rto": sum(1 for r in results if r.get("rto_met")),
        "systems_meeting_rpo": sum(1 for r in results if r.get("rpo_met")),
        "overall_pass": all(
            r.get("rto_met") and r.get("rpo_met") for r in results
            if r.get("rto_met") is not None
        ),
        "results": results,
    }

    if output_path:
        with open(output_path, "w") as f:
            json.dump(report, f, indent=2)
        print(f"[*] Report saved to {output_path}")

    return report


def main():
    parser = argparse.ArgumentParser(
        description="Ransomware Recovery Procedure Testing Agent"
    )
    parser.add_argument("--config", help="JSON config file for recovery drill")
    parser.add_argument("--hash-dir", help="Compute file hashes for a directory")
    parser.add_argument("--compare", nargs=2, metavar=("ORIGINAL", "RESTORED"),
                        help="Compare two hash manifest JSON files")
    parser.add_argument("--check-service", help="Check if a system service is running")
    parser.add_argument("--check-db", help="Check database connectivity (type:host:port)")
    parser.add_argument("--output", "-o", help="Output report file path")
    args = parser.parse_args()

    print("[*] Ransomware Recovery Procedure Testing Agent")

    if args.hash_dir:
        hashes = compute_file_hashes(args.hash_dir)
        print(json.dumps(hashes, indent=2))
        if args.output:
            with open(args.output, "w") as f:
                json.dump(hashes, f, indent=2)
            print(f"[*] Hash manifest saved to {args.output}")
        return

    if args.compare:
        with open(args.compare[0], "r") as f:
            orig = json.load(f)
        with open(args.compare[1], "r") as f:
            restored = json.load(f)
        result = compare_manifests(orig, restored)
        print(json.dumps(result, indent=2))
        return

    if args.check_service:
        result = check_service_health(args.check_service)
        print(json.dumps(result, indent=2))
        return

    if args.check_db:
        parts = args.check_db.split(":")
        db_type = parts[0]
        host = parts[1] if len(parts) > 1 else "localhost"
        port = int(parts[2]) if len(parts) > 2 else None
        result = check_database_connectivity(db_type, host, port)
        print(json.dumps(result, indent=2))
        return

    if args.config:
        with open(args.config, "r") as f:
            config = json.load(f)
        results = run_recovery_drill(config)
        report = generate_report(results, args.output)
        print(json.dumps(report, indent=2))
        return

    parser.print_help()


if __name__ == "__main__":
    main()
Keep exploring