malware analysis

Detecting Rootkit Activity

Detects rootkit presence on compromised systems by identifying hidden processes, hooked system calls, modified kernel structures, hidden files, and covert network connections using memory forensics, cross-view detection, and integrity checking techniques. Activates for requests involving rootkit detection, hidden process discovery, kernel integrity checking, or system call hook analysis.

detectionkernel-analysismalwarememory-forensicsrootkit
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

When to Use

  • System shows signs of compromise but standard tools (Task Manager, netstat) show nothing abnormal
  • Antivirus/EDR detects rootkit signatures but cannot identify the specific hiding mechanism
  • Memory forensics reveals discrepancies between kernel data structures and user-mode tool output
  • Investigating a persistent threat that survives remediation attempts and system reboots
  • Validating system integrity after a suspected kernel-level compromise

Do not use as a first-line detection method; start with standard malware triage and escalate to rootkit analysis when hiding behavior is suspected.

Prerequisites

  • Volatility 3 for memory forensics and kernel structure analysis
  • GMER or Rootkit Revealer (Windows) for live system scanning
  • rkhunter and chkrootkit (Linux) for filesystem and process integrity checks
  • Sysinternals tools (Process Explorer, Autoruns, RootkitRevealer) for Windows analysis
  • Memory dump from the suspected system (WinPmem, LiME)
  • Clean baseline of the OS for comparison (known-good kernel module hashes)

Workflow

Step 1: Cross-View Detection for Hidden Processes

Compare process lists from different data sources to find discrepancies:

# Volatility: Compare process enumeration methods
# pslist - walks ActiveProcessLinks (EPROCESS linked list - what rootkits manipulate)
vol3 -f memory.dmp windows.pslist > pslist_output.txt
 
# psscan - scans physical memory for EPROCESS pool tags (rootkit-resistant)
vol3 -f memory.dmp windows.psscan > psscan_output.txt
 
# Compare outputs to find hidden processes
python3 << 'PYEOF'
pslist_pids = set()
psscan_pids = set()
 
with open("pslist_output.txt") as f:
    for line in f:
        parts = line.split()
        if len(parts) > 1 and parts[1].isdigit():
            pslist_pids.add(int(parts[1]))
 
with open("psscan_output.txt") as f:
    for line in f:
        parts = line.split()
        if len(parts) > 1 and parts[1].isdigit():
            psscan_pids.add(int(parts[1]))
 
hidden = psscan_pids - pslist_pids
if hidden:
    print(f"[!] HIDDEN PROCESSES DETECTED (in psscan but not pslist):")
    for pid in hidden:
        print(f"    PID: {pid}")
else:
    print("[*] No hidden processes detected via cross-view analysis")
PYEOF

Step 2: Detect System Call Hooking

Identify hooks in the System Service Descriptor Table (SSDT) and Import Address Tables:

# Check SSDT for hooked system calls
vol3 -f memory.dmp windows.ssdt
 
# Identify hooks pointing outside ntoskrnl.exe or win32k.sys
vol3 -f memory.dmp windows.ssdt | grep -v "ntoskrnl\|win32k"
 
# Check for Inline hooks (detour patching)
vol3 -f memory.dmp windows.apihooks --pid 4  # System process
 
# IDT (Interrupt Descriptor Table) analysis
vol3 -f memory.dmp windows.idt
 
# Check for IRP (I/O Request Packet) hooking on drivers
vol3 -f memory.dmp windows.driverscan
vol3 -f memory.dmp windows.driverirp
Types of Rootkit Hooks:
━━━━━━━━━━━━━━━━━━━━━
SSDT Hook:         Modifies System Service Descriptor Table entries to redirect
                   system calls through rootkit code (filters process/file listings)
 
IAT Hook:          Patches Import Address Table of a process to intercept API calls
                   before they reach the kernel
 
Inline Hook:       Overwrites the first bytes of a function with a JMP to rootkit code
                   (detour/trampoline technique)
 
IRP Hook:          Intercepts I/O Request Packets to filter disk/network operations
                   at the driver level
 
DKOM:              Direct Kernel Object Manipulation - unlinking structures like
                   EPROCESS from the ActiveProcessLinks list without hooking

Step 3: Analyze Kernel Modules and Drivers

Identify unauthorized kernel drivers that may be rootkit components:

# List all loaded kernel modules
vol3 -f memory.dmp windows.modules
 
# Scan for drivers in memory (including hidden/unlinked)
vol3 -f memory.dmp windows.driverscan
 
# Compare module lists to find hidden drivers
vol3 -f memory.dmp windows.modscan > modscan.txt
vol3 -f memory.dmp windows.modules > modules.txt
 
# Check driver signatures and verify against known-good baselines
vol3 -f memory.dmp windows.verinfo
 
# Dump suspicious driver for static analysis
vol3 -f memory.dmp windows.moddump --base 0xFFFFF80012340000 --dump

Step 4: Detect File and Registry Hiding

Identify files and registry keys hidden by the rootkit:

# Linux rootkit detection with rkhunter
rkhunter --check --skip-keypress --report-warnings-only
 
# chkrootkit scanning
chkrootkit -q
 
# Windows: Compare filesystem views
# Live system file listing vs Volatility filescan
vol3 -f memory.dmp windows.filescan > mem_files.txt
 
# Check for hidden registry keys
vol3 -f memory.dmp windows.registry.hivelist
vol3 -f memory.dmp windows.registry.printkey --key "SYSTEM\CurrentControlSet\Services"
 
# Look for hidden services (loaded but not in service registry)
vol3 -f memory.dmp windows.svcscan | grep -i "kernel"

Step 5: Network Connection Analysis

Find hidden network connections and backdoors:

# Memory-based network connection enumeration
vol3 -f memory.dmp windows.netscan
 
# Compare with live netstat (if available) to find hidden connections
# Hidden connections: present in memory but not shown by netstat
 
# Look for raw sockets (often used by rootkits for covert communication)
vol3 -f memory.dmp windows.netscan | grep RAW
 
# Check for network filter drivers (NDIS hooks)
vol3 -f memory.dmp windows.driverscan | grep -i "ndis\|tcpip\|afd"
 
# Analyze callback routines registered by drivers
vol3 -f memory.dmp windows.callbacks

Step 6: Integrity Verification

Verify system file and kernel integrity:

# Check kernel code integrity (compare in-memory kernel to on-disk copy)
vol3 -f memory.dmp windows.moddump --base 0xFFFFF80070000000 --dump
# Compare SHA-256 of dumped ntoskrnl.exe with known-good copy
 
# Windows: System File Checker (on live system)
sfc /scannow
 
# Linux: Package integrity verification
rpm -Va  # RPM-based systems
debsums -c  # Debian-based systems
 
# Compare critical system binaries
find /bin /sbin /usr/bin /usr/sbin -type f -exec sha256sum {} \; > current_hashes.txt
# Compare against baseline: diff baseline_hashes.txt current_hashes.txt
 
# YARA scan for known rootkit signatures
vol3 -f memory.dmp yarascan.YaraScan --yara-file rootkit_rules.yar

Key Concepts

Term Definition
Rootkit Malware designed to maintain persistent, privileged access while hiding its presence from system administrators and security tools
DKOM Direct Kernel Object Manipulation; technique of modifying kernel data structures (e.g., unlinking EPROCESS) to hide objects without hooking
SSDT Hooking Replacing entries in the System Service Descriptor Table to intercept and filter system call results (hide processes, files, connections)
Inline Hooking Patching the first instructions of a function with a jump to rootkit code; the rootkit can filter the function output before returning
Cross-View Detection Comparing results from multiple enumeration methods (linked list walk vs memory scan) to identify discrepancies caused by hiding
Kernel Driver Code running in kernel mode (Ring 0) with full system access; rootkits use malicious drivers to gain kernel-level control
Bootkits Rootkits that infect the boot process (MBR, VBR, or UEFI firmware) to load before the operating system and security tools

Tools & Systems

  • Volatility: Memory forensics framework providing cross-view detection, SSDT analysis, and kernel structure inspection for rootkit detection
  • GMER: Free Windows rootkit detection tool scanning for SSDT hooks, IDT hooks, IRP hooks, and hidden processes/files/registry
  • rkhunter: Linux rootkit detection tool checking for known rootkit signatures, suspicious files, and system binary modifications
  • chkrootkit: Linux tool for detecting rootkit presence through signature-based and anomaly-based checks
  • Sysinternals RootkitRevealer: Microsoft tool comparing Windows API results with raw filesystem/registry scans to find discrepancies

Common Scenarios

Scenario: Investigating a System Where Standard Tools Show No Compromise

Context: An endpoint shows network beaconing to a known C2 IP in firewall logs, but the local EDR, Task Manager, and netstat show no suspicious processes or connections. A memory dump has been acquired for analysis.

Approach:

  1. Run Volatility psscan and compare with pslist to identify processes hidden via DKOM
  2. Run windows.ssdt to check for system call hooks that filter process and network listings
  3. Run windows.malfind to detect injected code in legitimate processes
  4. Run windows.netscan to find network connections hidden from user-mode tools
  5. Run windows.driverscan to identify malicious kernel drivers enabling the hiding
  6. Dump the rootkit driver and analyze with Ghidra to understand its hooking mechanism
  7. Check for boot persistence (MBR/VBR modifications, UEFI firmware implants)

Pitfalls:

  • Running detection tools on the live compromised system (rootkit may hide from or subvert them)
  • Assuming kernel integrity because no SSDT hooks are found (rootkit may use DKOM or inline hooks instead)
  • Not checking for both user-mode and kernel-mode rootkit components (many rootkits have both)
  • Trusting the rootkit scanner results on a live system; always verify with offline memory forensics

Output Format

ROOTKIT DETECTION ANALYSIS REPORT
====================================
Dump File:        memory.dmp
System:           Windows 10 21H2 x64
Analysis Tool:    Volatility 3.2
 
CROSS-VIEW DETECTION
Process List Comparison:
  pslist processes:  127
  psscan processes:  129
  [!] HIDDEN PROCESSES: 2
    PID 6784: sysmon64.exe (hidden rootkit component)
    PID 6812: netfilter.exe (hidden network filter)
 
SSDT HOOK ANALYSIS
[!] Entry 0x004A (NtQuerySystemInformation) hooked -> driver.sys+0x1200
[!] Entry 0x0055 (NtQueryDirectoryFile) hooked -> driver.sys+0x1400
[!] Entry 0x0119 (NtDeviceIoControlFile) hooked -> driver.sys+0x1600
Hook Target: driver.sys at 0xFFFFF800ABCD0000 (unsigned, suspicious)
 
KERNEL DRIVER ANALYSIS
[!] driver.sys - No digital signature, loaded at 0xFFFFF800ABCD0000
    Size: 45,056 bytes
    SHA-256: abc123def456...
    IRP Hooks: IRP_MJ_CREATE, IRP_MJ_DEVICE_CONTROL
    Registry: HKLM\SYSTEM\CurrentControlSet\Services\MalDriver
 
HIDDEN NETWORK CONNECTIONS
PID 6812: 10.1.5.42:49152 -> 185.220.101.42:443 (ESTABLISHED)
  - Not visible via netstat or user-mode tools
  - Filtered by NtDeviceIoControlFile SSDT hook
 
ROOTKIT CAPABILITIES
- Process hiding (DKOM + SSDT)
- File hiding (NtQueryDirectoryFile hook)
- Network connection hiding (NtDeviceIoControlFile hook)
- Kernel-mode persistence (driver service)
 
REMEDIATION
- Boot from clean media for offline remediation
- Remove malicious driver from offline registry
- Verify MBR/VBR/UEFI integrity for boot persistence
- Full system rebuild recommended for kernel-level compromise
Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 1

api-reference.md2.3 KB

Rootkit Detection API Reference

Volatility 3 - Rootkit Analysis Plugins

# Process enumeration - compare for hidden processes
vol3 -f memory.dmp windows.pslist     # EPROCESS linked list (rootkit-manipulable)
vol3 -f memory.dmp windows.psscan     # Pool tag scanning (rootkit-resistant)
 
# SSDT hook detection
vol3 -f memory.dmp windows.ssdt
 
# Kernel module listing
vol3 -f memory.dmp windows.modules
vol3 -f memory.dmp windows.modscan    # Scan for hidden modules
 
# Driver IRP hook detection
vol3 -f memory.dmp windows.driverirp
 
# Callback enumeration
vol3 -f memory.dmp windows.callbacks
 
# IDT (Interrupt Descriptor Table) check
vol3 -f memory.dmp windows.idt
 
# Injected code detection
vol3 -f memory.dmp windows.malfind

Cross-View Detection Method

Step 1: Enumerate with pslist (uses EPROCESS ActiveProcessLinks)
Step 2: Enumerate with psscan (scans pool tags in physical memory)
Step 3: Compare PID sets
Step 4: PIDs in psscan but NOT in pslist = hidden by DKOM rootkit

Linux Rootkit Detection Tools

# rkhunter
rkhunter --update                       # Update signatures
rkhunter --check --skip-keypress        # Full scan
rkhunter --check --report-warnings-only # Warnings only
 
# chkrootkit
chkrootkit                              # Full scan
chkrootkit -q                           # Quiet (only infected)
 
# Unhide (process and port hiding detection)
unhide proc     # Compare /proc, ps, syscall enumeration
unhide sys      # System call brute force
unhide-tcp      # Hidden TCP/UDP ports

Rootkit Types

Type Hides In Detection Method
User-mode LD_PRELOAD, IAT hooks Cross-view, strace
Kernel-mode DKOM, SSDT hooks Memory forensics
Bootkits MBR/VBR/UEFI Firmware integrity
Hypervisor Below OS Timing analysis

DKOM (Direct Kernel Object Manipulation)

Rootkit unlinking technique:
EPROCESS(prev).Flink -> EPROCESS(hidden).Flink  (skip hidden)
EPROCESS(next).Blink -> EPROCESS(hidden).Blink  (skip hidden)
 
Process disappears from pslist but remains in physical memory (psscan finds it)

Memory Acquisition

# Windows - WinPmem
winpmem_mini_x64.exe memdump.raw
 
# Linux - LiME
insmod lime.ko "path=/tmp/memory.lime format=lime"
 
# Linux - /proc/kcore
dd if=/proc/kcore of=/evidence/memory.raw bs=1M

Scripts 1

agent.py8.0 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""Rootkit detection agent using cross-view analysis and integrity checking."""

import json
import os
import subprocess
import sys
from datetime import datetime


def run_volatility_pslist(memory_dump):
    """List processes using ActiveProcessLinks (EPROCESS linked list)."""
    cmd = ["vol3", "-f", memory_dump, "windows.pslist"]
    try:
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
        processes = []
        for line in result.stdout.splitlines():
            parts = line.split()
            if len(parts) >= 4 and parts[0].isdigit():
                processes.append({"pid": int(parts[0]), "ppid": int(parts[1]),
                                  "name": parts[2], "threads": parts[3] if len(parts) > 3 else ""})
        return {"method": "pslist", "count": len(processes), "processes": processes}
    except FileNotFoundError:
        return {"error": "Volatility 3 not installed"}
    except subprocess.TimeoutExpired:
        return {"error": "Timed out"}


def run_volatility_psscan(memory_dump):
    """Scan physical memory for EPROCESS pool tags (rootkit-resistant)."""
    cmd = ["vol3", "-f", memory_dump, "windows.psscan"]
    try:
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
        processes = []
        for line in result.stdout.splitlines():
            parts = line.split()
            if len(parts) >= 4 and parts[0].startswith("0x"):
                processes.append({"offset": parts[0], "pid": parts[1],
                                  "ppid": parts[2] if len(parts) > 2 else "",
                                  "name": parts[3] if len(parts) > 3 else ""})
        return {"method": "psscan", "count": len(processes), "processes": processes}
    except FileNotFoundError:
        return {"error": "Volatility 3 not installed"}
    except subprocess.TimeoutExpired:
        return {"error": "Timed out"}


def cross_view_detection(memory_dump):
    """Compare pslist vs psscan to find hidden processes (DKOM rootkits)."""
    pslist = run_volatility_pslist(memory_dump)
    psscan = run_volatility_psscan(memory_dump)

    if "error" in pslist or "error" in psscan:
        return {"error": "Could not complete cross-view analysis",
                "pslist": pslist, "psscan": psscan}

    pslist_pids = set(str(p["pid"]) for p in pslist.get("processes", []))
    psscan_pids = set(str(p.get("pid", "")) for p in psscan.get("processes", []))

    hidden = psscan_pids - pslist_pids
    hidden_processes = [
        p for p in psscan.get("processes", [])
        if str(p.get("pid", "")) in hidden
    ]

    return {
        "pslist_count": len(pslist_pids),
        "psscan_count": len(psscan_pids),
        "hidden_processes": hidden_processes,
        "hidden_count": len(hidden_processes),
        "alert": "ROOTKIT DETECTED - Hidden processes found" if hidden_processes else "No hidden processes detected",
    }


def check_ssdt_hooks(memory_dump):
    """Check for SSDT (System Service Descriptor Table) hooks."""
    cmd = ["vol3", "-f", memory_dump, "windows.ssdt"]
    try:
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
        hooks = []
        for line in result.stdout.splitlines():
            if "UNKNOWN" in line.upper() or line.count("\\") == 0:
                parts = line.split()
                if len(parts) >= 3:
                    hooks.append({"entry": " ".join(parts)})
        return {"ssdt_hooks": hooks, "count": len(hooks)}
    except FileNotFoundError:
        return {"error": "Volatility 3 not installed"}
    except subprocess.TimeoutExpired:
        return {"error": "Timed out"}


def check_kernel_modules(memory_dump):
    """List loaded kernel modules and detect unsigned/suspicious ones."""
    cmd = ["vol3", "-f", memory_dump, "windows.modules"]
    try:
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
        modules = []
        for line in result.stdout.splitlines():
            parts = line.split()
            if len(parts) >= 3 and parts[0].startswith("0x"):
                modules.append({"base": parts[0], "size": parts[1],
                                "name": parts[2] if len(parts) > 2 else ""})
        return {"modules": modules, "count": len(modules)}
    except FileNotFoundError:
        return {"error": "Volatility 3 not installed"}
    except subprocess.TimeoutExpired:
        return {"error": "Timed out"}


def run_rkhunter():
    """Run rkhunter for Linux rootkit detection."""
    try:
        result = subprocess.run(
            ["rkhunter", "--check", "--skip-keypress", "--report-warnings-only"],
            capture_output=True, text=True, timeout=120
        )
        warnings = [line.strip() for line in result.stdout.splitlines() if "Warning" in line]
        return {
            "tool": "rkhunter",
            "warnings": warnings,
            "warning_count": len(warnings),
            "exit_code": result.returncode,
        }
    except FileNotFoundError:
        return {"error": "rkhunter not installed (apt install rkhunter)"}
    except subprocess.TimeoutExpired:
        return {"error": "rkhunter timed out"}


def run_chkrootkit():
    """Run chkrootkit for Linux rootkit detection."""
    try:
        result = subprocess.run(
            ["chkrootkit", "-q"], capture_output=True, text=True, timeout=120
        )
        infected = [line.strip() for line in result.stdout.splitlines()
                     if "INFECTED" in line.upper()]
        return {
            "tool": "chkrootkit",
            "infected": infected,
            "infected_count": len(infected),
            "exit_code": result.returncode,
        }
    except FileNotFoundError:
        return {"error": "chkrootkit not installed (apt install chkrootkit)"}
    except subprocess.TimeoutExpired:
        return {"error": "chkrootkit timed out"}


def check_hidden_files_linux():
    """Check for hidden files and directories that may indicate a rootkit."""
    suspicious = []
    check_dirs = ["/tmp", "/dev/shm", "/var/tmp"]
    for d in check_dirs:
        if not os.path.exists(d):
            continue
        try:
            for entry in os.listdir(d):
                if entry.startswith(".") and entry not in (".", ".."):
                    full_path = os.path.join(d, entry)
                    suspicious.append({
                        "path": full_path,
                        "is_dir": os.path.isdir(full_path),
                        "size": os.path.getsize(full_path) if os.path.isfile(full_path) else 0,
                    })
        except PermissionError:
            continue
    return {"hidden_files": suspicious, "count": len(suspicious)}


def generate_report(memory_dump=None):
    """Generate comprehensive rootkit detection report."""
    report = {"timestamp": datetime.utcnow().isoformat() + "Z"}

    if memory_dump:
        report["cross_view"] = cross_view_detection(memory_dump)
        report["ssdt_hooks"] = check_ssdt_hooks(memory_dump)
        report["kernel_modules"] = check_kernel_modules(memory_dump)
    else:
        report["rkhunter"] = run_rkhunter()
        report["chkrootkit"] = run_chkrootkit()
        report["hidden_files"] = check_hidden_files_linux()

    return report


if __name__ == "__main__":
    action = sys.argv[1] if len(sys.argv) > 1 else "help"
    if action == "cross-view" and len(sys.argv) > 2:
        print(json.dumps(cross_view_detection(sys.argv[2]), indent=2, default=str))
    elif action == "malfind" and len(sys.argv) > 2:
        print(json.dumps(run_volatility_pslist(sys.argv[2]), indent=2, default=str))
    elif action == "ssdt" and len(sys.argv) > 2:
        print(json.dumps(check_ssdt_hooks(sys.argv[2]), indent=2, default=str))
    elif action == "rkhunter":
        print(json.dumps(run_rkhunter(), indent=2))
    elif action == "chkrootkit":
        print(json.dumps(run_chkrootkit(), indent=2))
    elif action == "report":
        mem = sys.argv[2] if len(sys.argv) > 2 else None
        print(json.dumps(generate_report(mem), indent=2, default=str))
    else:
        print("Usage: agent.py [cross-view <mem>|malfind <mem>|ssdt <mem>|rkhunter|chkrootkit|report [mem]]")
Keep exploring