incident response

Validating Backup Integrity for Recovery

Validate backup integrity through cryptographic hash verification, automated restore testing, corruption detection, and recoverability checks to ensure backups are reliable for disaster recovery and ransomware response scenarios.

backupdisaster-recoveryhash-verificationincident-responseintegrityrestore-testing
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

When to Use

Use this skill when:

  • Verifying backup integrity before relying on backups for ransomware recovery
  • Building automated backup validation pipelines that run after each backup job
  • Auditing backup infrastructure to confirm recoverability for compliance (SOC 2, ISO 27001, NIST CSF RC.RP-03)
  • Detecting silent data corruption (bit rot) in backup storage before a disaster occurs
  • Validating that immutable or air-gapped backups have not been tampered with

Do not use for initial backup configuration or scheduling. This skill focuses on post-backup validation.

Prerequisites

  • Access to backup storage (local, NAS, S3, Azure Blob, GCS)
  • Python 3.9+ with hashlib (standard library)
  • Backup manifests or baseline hash files for comparison
  • Isolated restore environment for restore testing
  • Backup tool CLI access (restic, borgbackup, rclone, or vendor-specific)

Workflow

Step 1: Generate Baseline Hash Manifest

Create a cryptographic fingerprint of every file at backup time:

# Generate SHA-256 manifest for a directory
find /data/production -type f -exec sha256sum {} \; > /manifests/prod_baseline_$(date +%Y%m%d).sha256
 
# Verify manifest format
head -5 /manifests/prod_baseline_20260319.sha256
# e3b0c44298fc1c149afbf4c8996fb924...  /data/production/config.yaml
# a7ffc6f8bf1ed76651c14756a061d662...  /data/production/database.sql

Step 2: Verify Backup Archive Integrity

Check that the backup archive itself is not corrupted:

# Restic: verify backup repository integrity
restic -r s3:s3.amazonaws.com/backup-bucket check --read-data
 
# Borg: verify backup archive
borg check --verify-data /backup/repo::archive-2026-03-19
 
# Tar with gzip: verify archive integrity
gzip -t backup_20260319.tar.gz && echo "Archive OK" || echo "Archive CORRUPTED"
 
# AWS S3: verify object checksums
aws s3api head-object --bucket backup-bucket --key daily/2026-03-19.tar.gz \
  --checksum-mode ENABLED

Step 3: Perform Restore Test to Isolated Environment

# Restore to isolated test directory
restic -r s3:s3.amazonaws.com/backup-bucket restore latest --target /restore-test/
 
# Generate hash manifest of restored data
find /restore-test -type f -exec sha256sum {} \; > /manifests/restored_$(date +%Y%m%d).sha256
 
# Compare baseline and restored manifests
diff <(sort /manifests/prod_baseline_20260319.sha256) \
     <(sort /manifests/restored_20260319.sha256)

Step 4: Validate Data Completeness

# Count files in original vs restored
echo "Original: $(find /data/production -type f | wc -l) files"
echo "Restored: $(find /restore-test -type f | wc -l) files"
 
# Check total size
echo "Original: $(du -sh /data/production | cut -f1)"
echo "Restored: $(du -sh /restore-test | cut -f1)"
 
# Database consistency check after restore
pg_restore --list backup.dump | wc -l  # Count objects in dump
psql -c "SELECT schemaname, tablename FROM pg_tables WHERE schemaname='public';" restored_db

Step 5: Detect Ransomware Artifacts in Backups

Before trusting a backup for recovery, scan for ransomware indicators:

# Check for common ransomware file extensions
find /restore-test -type f \( \
  -name "*.encrypted" -o -name "*.locked" -o -name "*.crypt" \
  -o -name "*.ransom" -o -name "*.pay" -o -name "*.wncry" \
  -o -name "*.cerber" -o -name "*.locky" -o -name "*.zepto" \
\) -print
 
# Check for ransom notes
find /restore-test -type f \( \
  -name "README_TO_DECRYPT*" -o -name "HOW_TO_RECOVER*" \
  -o -name "DECRYPT_INSTRUCTIONS*" -o -name "HELP_DECRYPT*" \
\) -print
 
# Check file entropy (high entropy = possible encryption)
# Files with entropy > 7.9 out of 8.0 are likely encrypted
python agent.py --entropy-scan /restore-test

Step 6: Automate and Schedule Validation

# cron-based validation schedule
# Run nightly after backup window
0 4 * * * /opt/backup-validator/agent.py --validate-latest --notify-on-failure
# Weekly full restore test
0 6 * * 0 /opt/backup-validator/agent.py --full-restore-test --config /etc/backup-validator/config.json

Key Concepts

Term Definition
Hash Manifest File containing cryptographic hashes (SHA-256) for every file in a dataset, used as integrity baseline
Bit Rot Gradual data corruption on storage media that silently alters file contents
Immutable Backup Backup that cannot be modified or deleted for a defined retention period
Restore Test Process of recovering data from backup to an isolated environment to verify recoverability
File Entropy Measure of randomness in file contents; encrypted files have entropy near 8.0 bits/byte
3-2-1 Rule Keep 3 copies of data, on 2 different media types, with 1 offsite copy
Backup Chain Sequence of full and incremental backups that must all be intact for recovery

Tools & Systems

Tool Purpose
Restic Encrypted, deduplicated backup with built-in integrity verification
BorgBackup Deduplicating backup with archive verification
Rclone Cloud storage sync with checksum verification
AWS S3 Object Lock Immutable backup storage with WORM compliance
Azure Immutable Blob Tamper-proof backup storage for compliance
sha256sum Standard hash computation for file integrity
pg_restore PostgreSQL backup validation and restore testing

Common Pitfalls

  • Never testing restores: The most common failure mode. Backups that are never restored are untested assumptions.
  • Checking only archive integrity, not data integrity: A valid tar.gz can contain corrupted file contents. Always hash individual files.
  • Trusting last backup without scanning for ransomware: Backups may contain encrypted files if the infection predates the backup.
  • Ignoring incremental chain integrity: A single corrupted incremental backup can break the entire restore chain.
  • No alerting on validation failures: Backup validation must be monitored with alerts, not just logged silently.
  • Using MD5 for integrity: MD5 is cryptographically broken. Use SHA-256 or SHA-3 for integrity verification.

References

  • NIST SP 800-184: Guide for Cybersecurity Event Recovery
  • NIST CSF 2.0 RC.RP-03: Backup Integrity Verification
  • CIS Controls v8: Control 11 - Data Recovery
  • CISA Ransomware Guide: https://www.cisa.gov/stopransomware
Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 1

api-reference.md3.9 KB

API Reference: Validating Backup Integrity for Recovery

CLI Usage

# Generate SHA-256 hash manifest for a directory
python agent.py --generate-manifest /data/production -o manifest.json
 
# Generate manifest with SHA-512
python agent.py --generate-manifest /data/production --algorithm sha512 -o manifest.json
 
# Compare baseline vs restored manifest
python agent.py --compare baseline_manifest.json restored_manifest.json
 
# Run full backup validation suite
python agent.py --validate /restore-test --baseline baseline_manifest.json -o report.json
 
# Scan for ransomware artifacts in restored data
python agent.py --ransomware-scan /restore-test
 
# Scan for high-entropy (possibly encrypted) files
python agent.py --entropy-scan /restore-test --entropy-threshold 7.9

Hash Algorithms Supported

Algorithm Digest Size Use Case
sha256 256 bits Default; standard integrity verification
sha512 512 bits Higher security; larger files
sha3_256 256 bits NIST post-quantum recommendation
blake2b 512 bits Faster alternative; high performance

Manifest Format

{
  "directory": "/data/production",
  "algorithm": "sha256",
  "generated_at": "2026-03-19T04:00:00+00:00",
  "total_files": 1523,
  "errors": 0,
  "hashes": {
    "config/app.yaml": "a3f2b8c9d1e4f5a6...",
    "data/users.db": "1b2c3d4e5f6a7b8c...",
    "logs/access.log": "ERROR:Permission denied"
  }
}

Comparison Result Format

{
  "baseline_files": 1523,
  "restored_files": 1520,
  "missing_files": ["logs/audit.log", "tmp/cache.db", "data/session.bin"],
  "missing_count": 3,
  "modified_files": [
    {
      "file": "config/app.yaml",
      "baseline": "a3f2b8c9...",
      "restored": "7e8f9a0b..."
    }
  ],
  "modified_count": 1,
  "added_files": [],
  "added_count": 0,
  "integrity_pass": false
}

Entropy Scan Output

{
  "directory": "/restore-test",
  "threshold": 7.9,
  "files_scanned": 1200,
  "suspicious_count": 3,
  "suspicious_files": [
    {
      "file": "data/report.docx.encrypted",
      "entropy": 7.98,
      "size_bytes": 524288
    }
  ]
}

Entropy Reference Values

Entropy Range Interpretation
0.0 - 1.0 Highly repetitive data (empty files, padding)
1.0 - 5.0 Structured text (config files, logs, source code)
5.0 - 7.0 Binary data (executables, images, databases)
7.0 - 7.8 Compressed data (zip, gzip, jpg)
7.8 - 8.0 Encrypted or fully random data (ransomware indicator)

Ransomware Scan Output

{
  "ransomware_extensions": [
    "documents/report.docx.locked",
    "data/backup.sql.encrypted"
  ],
  "ransom_notes": [
    "HOW_TO_RECOVER_YOUR_FILES.txt"
  ],
  "total_scanned": 1523,
  "clean": false
}

Known Ransomware Extensions Detected

.encrypted, .locked, .crypt, .ransom, .pay, .wncry, .wcry, .cerber, .locky, .zepto, .osiris, .aesir, .thor, .odin, .crypz, .crypted, .enc, .crypto, .lockbit

Full Validation Report Schema

{
  "timestamp": "2026-03-19T04:30:00+00:00",
  "directory": "/restore-test",
  "checks": {
    "file_stats": {
      "total_files": 1523,
      "total_size_bytes": 1073741824,
      "total_size_mb": 1024.0,
      "pass": true
    },
    "integrity": {
      "integrity_pass": true,
      "missing_count": 0,
      "modified_count": 0
    },
    "ransomware_scan": {
      "clean": true,
      "total_scanned": 1523
    },
    "entropy_scan": {
      "files_scanned": 1200,
      "suspicious_count": 0
    }
  },
  "overall_pass": true
}

References

Scripts 1

agent.py10.5 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""Agent for validating backup integrity for disaster recovery.

Computes cryptographic hashes, compares manifests, detects corruption,
scans for ransomware artifacts, measures file entropy, and validates
backup recoverability.
"""

import argparse
import hashlib
import json
import math
import os
from collections import Counter
from datetime import datetime, timezone
from pathlib import Path


RANSOMWARE_EXTENSIONS = {
    ".encrypted", ".locked", ".crypt", ".ransom", ".pay",
    ".wncry", ".wcry", ".cerber", ".locky", ".zepto",
    ".osiris", ".aesir", ".thor", ".odin", ".crypz",
    ".crypted", ".enc", ".crypto", ".lockbit",
}

RANSOM_NOTE_PATTERNS = [
    "README_TO_DECRYPT", "HOW_TO_RECOVER", "DECRYPT_INSTRUCTIONS",
    "HELP_DECRYPT", "RECOVERY_INSTRUCTIONS", "RESTORE_FILES",
    "READ_ME_TO_DECRYPT", "YOUR_FILES_ARE_ENCRYPTED",
    "!README!", "DECRYPT_YOUR_FILES",
]


def compute_file_hash(filepath, algorithm="sha256"):
    """Compute cryptographic hash of a single file."""
    h = hashlib.new(algorithm)
    try:
        with open(filepath, "rb") as f:
            for chunk in iter(lambda: f.read(65536), b""):
                h.update(chunk)
        return h.hexdigest()
    except (PermissionError, OSError) as e:
        return f"ERROR:{e}"


def generate_manifest(directory, algorithm="sha256"):
    """Generate hash manifest for all files in a directory."""
    manifest = {}
    dir_path = Path(directory)
    if not dir_path.is_dir():
        return {"error": f"Directory not found: {directory}"}

    total = 0
    errors = 0
    for fpath in sorted(dir_path.rglob("*")):
        if fpath.is_file():
            total += 1
            digest = compute_file_hash(str(fpath), algorithm)
            rel = str(fpath.relative_to(dir_path))
            manifest[rel] = digest
            if digest.startswith("ERROR:"):
                errors += 1

    return {
        "directory": str(directory),
        "algorithm": algorithm,
        "generated_at": datetime.now(timezone.utc).isoformat(),
        "total_files": total,
        "errors": errors,
        "hashes": manifest,
    }


def compare_manifests(baseline_path, restored_path):
    """Compare two manifest files to detect integrity issues."""
    with open(baseline_path, "r") as f:
        baseline = json.load(f)
    with open(restored_path, "r") as f:
        restored = json.load(f)

    base_hashes = baseline.get("hashes", baseline)
    rest_hashes = restored.get("hashes", restored)

    missing = []
    modified = []
    added = []

    for fname, base_hash in base_hashes.items():
        if fname not in rest_hashes:
            missing.append(fname)
        elif rest_hashes[fname] != base_hash:
            modified.append({"file": fname, "baseline": base_hash,
                             "restored": rest_hashes[fname]})

    for fname in rest_hashes:
        if fname not in base_hashes:
            added.append(fname)

    integrity_pass = len(missing) == 0 and len(modified) == 0
    return {
        "baseline_files": len(base_hashes),
        "restored_files": len(rest_hashes),
        "missing_files": missing,
        "missing_count": len(missing),
        "modified_files": modified,
        "modified_count": len(modified),
        "added_files": added,
        "added_count": len(added),
        "integrity_pass": integrity_pass,
    }


def calculate_entropy(filepath):
    """Calculate Shannon entropy of a file (0-8 bits per byte)."""
    try:
        with open(filepath, "rb") as f:
            data = f.read()
    except (PermissionError, OSError):
        return None

    if not data:
        return 0.0

    byte_counts = Counter(data)
    length = len(data)
    entropy = 0.0
    for count in byte_counts.values():
        p = count / length
        if p > 0:
            entropy -= p * math.log2(p)
    return round(entropy, 4)


def entropy_scan(directory, threshold=7.9):
    """Scan directory for files with suspiciously high entropy (possible encryption)."""
    suspicious = []
    scanned = 0
    dir_path = Path(directory)

    for fpath in dir_path.rglob("*"):
        if not fpath.is_file():
            continue
        if fpath.stat().st_size < 1024:
            continue
        scanned += 1
        ent = calculate_entropy(str(fpath))
        if ent is not None and ent >= threshold:
            suspicious.append({
                "file": str(fpath.relative_to(dir_path)),
                "entropy": ent,
                "size_bytes": fpath.stat().st_size,
            })

    return {
        "directory": str(directory),
        "threshold": threshold,
        "files_scanned": scanned,
        "suspicious_count": len(suspicious),
        "suspicious_files": suspicious[:100],
    }


def scan_ransomware_artifacts(directory):
    """Scan restored backup for ransomware indicators."""
    findings = {
        "ransomware_extensions": [],
        "ransom_notes": [],
        "total_scanned": 0,
    }
    dir_path = Path(directory)

    for fpath in dir_path.rglob("*"):
        if not fpath.is_file():
            continue
        findings["total_scanned"] += 1

        if fpath.suffix.lower() in RANSOMWARE_EXTENSIONS:
            findings["ransomware_extensions"].append(
                str(fpath.relative_to(dir_path))
            )

        for pattern in RANSOM_NOTE_PATTERNS:
            if pattern.lower() in fpath.name.lower():
                findings["ransom_notes"].append(
                    str(fpath.relative_to(dir_path))
                )
                break

    findings["clean"] = (
        len(findings["ransomware_extensions"]) == 0
        and len(findings["ransom_notes"]) == 0
    )
    return findings


def validate_backup(directory, baseline_manifest=None, check_ransomware=True,
                    check_entropy=True, entropy_threshold=7.9):
    """Run full backup validation suite."""
    results = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "directory": str(directory),
        "checks": {},
    }

    # File count and size
    dir_path = Path(directory)
    if not dir_path.is_dir():
        return {"error": f"Directory not found: {directory}"}

    total_files = sum(1 for _ in dir_path.rglob("*") if _.is_file())
    total_size = sum(f.stat().st_size for f in dir_path.rglob("*") if f.is_file())
    results["checks"]["file_stats"] = {
        "total_files": total_files,
        "total_size_bytes": total_size,
        "total_size_mb": round(total_size / (1024 * 1024), 2),
        "pass": total_files > 0,
    }

    # Manifest comparison
    if baseline_manifest and os.path.isfile(baseline_manifest):
        current = generate_manifest(directory)
        current_path = str(dir_path / ".current_manifest.json")
        with open(current_path, "w") as f:
            json.dump(current, f)
        comparison = compare_manifests(baseline_manifest, current_path)
        results["checks"]["integrity"] = comparison
        os.remove(current_path)
    else:
        results["checks"]["integrity"] = {"skipped": True,
                                           "reason": "No baseline manifest provided"}

    # Ransomware artifact scan
    if check_ransomware:
        results["checks"]["ransomware_scan"] = scan_ransomware_artifacts(directory)

    # Entropy scan
    if check_entropy:
        results["checks"]["entropy_scan"] = entropy_scan(directory, entropy_threshold)

    # Overall verdict
    checks = results["checks"]
    results["overall_pass"] = (
        checks.get("file_stats", {}).get("pass", False)
        and checks.get("integrity", {}).get("integrity_pass", True)
        and checks.get("ransomware_scan", {}).get("clean", True)
        and checks.get("entropy_scan", {}).get("suspicious_count", 0) == 0
    )

    return results


def main():
    parser = argparse.ArgumentParser(
        description="Backup Integrity Validation Agent"
    )
    parser.add_argument("--generate-manifest",
                        help="Generate hash manifest for a directory")
    parser.add_argument("--compare", nargs=2, metavar=("BASELINE", "RESTORED"),
                        help="Compare two manifest JSON files")
    parser.add_argument("--validate", help="Run full validation on a backup directory")
    parser.add_argument("--baseline", help="Baseline manifest for comparison")
    parser.add_argument("--entropy-scan", help="Scan directory for high-entropy files")
    parser.add_argument("--entropy-threshold", type=float, default=7.9,
                        help="Entropy threshold (default: 7.9)")
    parser.add_argument("--ransomware-scan",
                        help="Scan directory for ransomware artifacts")
    parser.add_argument("--algorithm", default="sha256",
                        choices=["sha256", "sha512", "sha3_256", "blake2b"],
                        help="Hash algorithm (default: sha256)")
    parser.add_argument("--output", "-o", help="Output file path")
    args = parser.parse_args()

    print("[*] Backup Integrity Validation Agent")
    result = None

    if args.generate_manifest:
        result = generate_manifest(args.generate_manifest, args.algorithm)
        print(f"[*] Generated manifest: {result.get('total_files', 0)} files")

    elif args.compare:
        result = compare_manifests(args.compare[0], args.compare[1])
        status = "PASS" if result["integrity_pass"] else "FAIL"
        print(f"[*] Integrity check: {status}")
        if result["missing_count"]:
            print(f"[!] Missing files: {result['missing_count']}")
        if result["modified_count"]:
            print(f"[!] Modified files: {result['modified_count']}")

    elif args.validate:
        result = validate_backup(
            args.validate,
            baseline_manifest=args.baseline,
            entropy_threshold=args.entropy_threshold,
        )
        status = "PASS" if result.get("overall_pass") else "FAIL"
        print(f"[*] Overall validation: {status}")

    elif args.entropy_scan:
        result = entropy_scan(args.entropy_scan, args.entropy_threshold)
        print(f"[*] Scanned {result['files_scanned']} files, "
              f"{result['suspicious_count']} suspicious")

    elif args.ransomware_scan:
        result = scan_ransomware_artifacts(args.ransomware_scan)
        status = "CLEAN" if result["clean"] else "INFECTED"
        print(f"[*] Ransomware scan: {status}")

    else:
        parser.print_help()
        return

    if result:
        output = json.dumps(result, indent=2)
        if args.output:
            with open(args.output, "w") as f:
                f.write(output)
            print(f"[*] Results saved to {args.output}")
        else:
            print(output)


if __name__ == "__main__":
    main()
Keep exploring