container security

Detecting Container Drift at Runtime

Detect unauthorized modifications to running containers by monitoring for binary execution drift, file system changes, and configuration deviations from the original container image.

container-driftcontainer-securitydrift-detectionfalcoimmutable-containerskubernetesmicrosoft-defenderruntime-security
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

Overview

Container drift occurs when running containers deviate from their original image state through unauthorized file modifications, unexpected binary execution, configuration changes, or package installations. Since containers should be treated as immutable infrastructure, any drift is a potential indicator of compromise. Detection techniques leverage the DIE (Detect, Isolate, Evict) model -- an immutable workload should not change during runtime, so any observed change is potentially evidence of malicious activity.

When to Use

  • When investigating security incidents that require detecting container drift at runtime
  • When building detection rules or threat hunting queries for this domain
  • When SOC analysts need structured procedures for this analysis type
  • When validating security monitoring coverage for related attack techniques

Prerequisites

  • Kubernetes cluster v1.24+ with runtime security tooling
  • Falco or Sysdig for runtime drift detection
  • Container image registry with image manifests available
  • Familiarity with Linux filesystem layers and OverlayFS

Core Concepts

Types of Container Drift

  1. Binary drift: Execution of binaries not present in the original image (downloaded malware, compiled tools)
  2. File drift: Creation, modification, or deletion of files in the container filesystem
  3. Configuration drift: Changes to environment variables, mounted secrets, or runtime parameters
  4. Package drift: Installation of new packages via apt, yum, pip, or npm at runtime
  5. Network drift: New listening ports or outbound connections not expected for the workload

Detection Methods

Image-Based Comparison: Compare the running container's filesystem against its source image to identify added, modified, or removed files.

Behavioral Monitoring: Use eBPF or kernel-level monitoring to detect process execution, file access, and network activity that deviates from expected behavior.

Digest Verification: Continuously verify that running container image digests match the approved deployment manifests.

Implementation with Falco

Detecting New Binary Execution

- rule: Drift Detected (Container Image Modified Binary)
  desc: Detect execution of a binary not present in the original container image
  condition: >
    spawned_process and
    container and
    not proc.pname in (container_entrypoint) and
    proc.is_exe_upper_layer = true
  output: >
    Drift detected: new binary executed in container
    (user=%user.name command=%proc.cmdline container=%container.name
     image=%container.image.repository:%container.image.tag
     exe_path=%proc.exepath)
  priority: WARNING
  tags: [container, drift]
 
- rule: Container Shell Spawned
  desc: Detect interactive shell in a container that should be immutable
  condition: >
    spawned_process and
    container and
    proc.name in (bash, sh, dash, zsh, csh, ksh) and
    not proc.pname in (container_entrypoint)
  output: >
    Shell spawned in container (user=%user.name shell=%proc.name
     container=%container.name image=%container.image.repository)
  priority: WARNING
  tags: [container, drift, shell]

Detecting Package Manager Usage

- rule: Package Manager Execution in Container
  desc: Detect use of package managers indicating drift
  condition: >
    spawned_process and
    container and
    proc.name in (apt, apt-get, yum, dnf, apk, pip, pip3, npm, gem, cargo)
  output: >
    Package manager executed in container (user=%user.name
     command=%proc.cmdline container=%container.name
     image=%container.image.repository)
  priority: ERROR
  tags: [container, drift, package-manager]

Detecting File System Modifications

- rule: Container File System Write
  desc: Detect writes to container upper layer filesystem
  condition: >
    open_write and
    container and
    fd.typechar = 'f' and
    not fd.name startswith /tmp and
    not fd.name startswith /var/log and
    not fd.name startswith /proc
  output: >
    File write in container (user=%user.name file=%fd.name
     container=%container.name)
  priority: NOTICE
  tags: [container, drift, filesystem]

Implementation with Kubernetes Enforcement

Read-Only Root Filesystem

Prevent drift by making container filesystems immutable:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: immutable-app
spec:
  template:
    spec:
      containers:
        - name: app
          image: app:v1.0@sha256:abc123...
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            runAsNonRoot: true
          volumeMounts:
            - name: tmp
              mountPath: /tmp
            - name: cache
              mountPath: /var/cache
      volumes:
        - name: tmp
          emptyDir:
            sizeLimit: 100Mi
        - name: cache
          emptyDir:
            sizeLimit: 50Mi

Pod Security Standards Enforcement

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Image Digest Verification

Continuous Digest Monitoring

#!/bin/bash
# Compare running container digests against approved manifest
 
NAMESPACE="production"
 
kubectl get pods -n "$NAMESPACE" -o json | jq -r '
  .items[] |
  .spec.containers[] |
  "\(.image) \(.imageID)"
' | while read IMAGE IMAGE_ID; do
  APPROVED_DIGEST=$(kubectl get deploy -n "$NAMESPACE" -o json | \
    jq -r ".items[].spec.template.spec.containers[] | select(.image==\"$IMAGE\") | .image")
 
  if [[ "$IMAGE" != *"@sha256:"* ]]; then
    echo "[WARN] Container using mutable tag: $IMAGE"
  fi
done

Microsoft Defender for Containers Integration

For Azure Kubernetes environments, Microsoft Defender provides built-in binary drift detection:

{
  "alertType": "K8S.NODE_ImageBinaryDrift",
  "severity": "Medium",
  "description": "Binary executed that was not part of the original container image",
  "remediationSteps": [
    "Investigate the binary origin and purpose",
    "Check if the container was compromised",
    "Rebuild the container from a clean image",
    "Enable readOnlyRootFilesystem"
  ]
}

Drift Response Playbook

  1. Detect: Alert fires on drift event (Falco, Defender, Sysdig)
  2. Validate: Confirm the drift is not from an approved process (init containers, config reloads)
  3. Isolate: Apply a deny-all NetworkPolicy to the affected pod
  4. Investigate: Capture container filesystem diff and process list
  5. Evict: Delete the drifted pod (ReplicaSet will recreate from clean image)
  6. Remediate: Fix the root cause (patch vulnerability, update image, tighten RBAC)

References

Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 3

api-reference.md2.0 KB

API Reference: Detecting Container Drift at Runtime

Docker SDK for Python

import docker
client = docker.from_env()
 
# List running containers
containers = client.containers.list()
 
# Get container details
container = client.containers.get("container_id")
container.attrs         # full inspection dict
container.image.id      # image SHA256
container.image.tags    # ['app:v1.0']
 
# Filesystem diff (vs original image)
diff = container.diff()
# Returns: [{"Path": "/tmp/new_file", "Kind": 1}]
# Kind: 0=Modified, 1=Added, 2=Deleted
 
# Container inspection fields
container.attrs["HostConfig"]["Privileged"]       # bool
container.attrs["HostConfig"]["ReadonlyRootfs"]   # bool
container.attrs["Config"]["Image"]                # image reference

Docker CLI Commands

# Filesystem changes since creation
docker diff <container>     # A=Added, C=Changed, D=Deleted
 
# Running processes
docker top <container> -eo pid,user,comm,args
 
# Image digest verification
docker inspect --format='{{.Image}}' <container>

Falco Drift Detection Rules

# Detect binary not in original image
condition: spawned_process and container and proc.is_exe_upper_layer = true
 
# Detect package manager usage
condition: spawned_process and container and proc.name in (apt, yum, pip, npm)
 
# Detect shell spawn
condition: spawned_process and container and proc.name in (bash, sh, dash)

Kubernetes Security Context

securityContext:
  readOnlyRootFilesystem: true     # prevent drift
  allowPrivilegeEscalation: false
  runAsNonRoot: true
  capabilities:
    drop: ["ALL"]

Drift Severity Classification

Indicator Severity
Privileged container CRITICAL
Sensitive file modified (/etc/shadow) CRITICAL
Binary added to system path HIGH
Package manager executed HIGH
Root shell active MEDIUM
Mutable root filesystem MEDIUM

CLI Usage

python agent.py --container my-app-container
python agent.py --container abc123 --all
standards.md1.2 KB

Standards and References - Container Drift Detection

Industry Standards

NIST SP 800-190: Application Container Security Guide

  • Section 3.3: Containers modified at runtime indicate compromise
  • Section 4.2: Monitor containers for unauthorized changes
  • Recommends treating containers as immutable infrastructure

CIS Kubernetes Benchmark v1.9

  • Control 5.2.8: Minimize container readOnlyRootFilesystem
  • Control 5.7.3: Apply security contexts to pods
  • Control 5.7.4: Default namespace restrictions

MITRE ATT&CK for Containers

  • T1610: Deploy Container -- unauthorized container deployment
  • T1611: Escape to Host -- container boundary violation
  • T1059.004: Unix Shell execution in containers
  • T1105: Ingress Tool Transfer -- downloading tools into containers

Compliance Mapping

Requirement Framework Drift Detection Capability
Change detection PCI DSS 11.5 File integrity monitoring in containers
Unauthorized software SOC 2 CC6.8 Binary execution drift alerts
Configuration management ISO 27001 A.12.1 Image digest verification
Incident detection NIST CSF DE.CM-7 Runtime behavioral anomaly detection
workflows.md1.7 KB

Workflows - Container Drift Detection

Detection Workflow

  1. Container image deployed with known-good state
  2. Runtime monitor (Falco/Sysdig) tracks all process executions and file changes
  3. Events compared against baseline: original image manifest + expected runtime behavior
  4. Drift events classified by severity (binary drift = HIGH, config drift = MEDIUM)
  5. Alerts sent to SIEM/SOC with full container context
  6. Automated response: isolate pod network, capture forensics, evict pod

Implementation Phases

Phase 1: Visibility (Weeks 1-2)

  • Deploy Falco with drift detection rules in alert-only mode
  • Collect baseline of normal container behavior per workload
  • Identify legitimate runtime changes (log files, temp files, caches)
  • Create allowlists for expected runtime modifications

Phase 2: Detection (Weeks 3-4)

  • Enable drift detection alerts with tuned thresholds
  • Integrate with SIEM for correlation and dashboarding
  • Build runbooks for drift investigation
  • Conduct tabletop exercises with container drift scenarios

Phase 3: Prevention (Weeks 5-8)

  • Enable readOnlyRootFilesystem on all production workloads
  • Deploy Pod Security Standards in enforce mode
  • Implement image digest pinning in all manifests
  • Enable automated pod eviction for confirmed drift events

Incident Response for Drift Events

  1. Triage: Is the drift from a legitimate operation or potential compromise?
  2. Contain: Apply NetworkPolicy deny-all to affected pod
  3. Collect: Capture container filesystem diff, process tree, network connections
  4. Analyze: Compare drifted files against malware signatures and IoCs
  5. Remediate: Delete compromised pod, scan all pods in namespace
  6. Recover: Deploy clean image, verify no persistence mechanisms

Scripts 2

agent.py5.8 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""Container drift detection agent using Docker SDK.

Compares running container filesystem against the original image to detect
binary drift, file modifications, and package installations.
"""

import argparse
import json
import subprocess
import sys
from datetime import datetime

try:
    import docker
except ImportError:
    print("Install docker SDK: pip install docker")
    sys.exit(1)

PACKAGE_MANAGERS = {"apt", "apt-get", "yum", "dnf", "apk", "pip", "pip3", "npm", "gem"}
SHELLS = {"bash", "sh", "dash", "zsh", "csh", "ash"}
SUSPICIOUS_BINARIES = {"curl", "wget", "nc", "ncat", "netcat", "socat", "python",
                       "perl", "gcc", "cc", "make", "nmap", "tcpdump"}


def get_container_diff(client, container_id):
    container = client.containers.get(container_id)
    try:
        diff = container.diff()
    except docker.errors.APIError:
        diff = []
    changes = {"added": [], "modified": [], "deleted": []}
    for entry in diff or []:
        path = entry.get("Path", "")
        kind = entry.get("Kind", 0)
        if kind == 0:
            changes["modified"].append(path)
        elif kind == 1:
            changes["added"].append(path)
        elif kind == 2:
            changes["deleted"].append(path)
    return changes


def get_running_processes(container_id):
    try:
        result = subprocess.run(
            ["docker", "top", container_id, "-eo", "pid,user,comm,args"],
            capture_output=True, text=True, timeout=10)
        if result.returncode != 0:
            return []
        lines = result.stdout.strip().split("\n")[1:]
        processes = []
        for line in lines:
            parts = line.split(None, 3)
            if len(parts) >= 3:
                processes.append({
                    "pid": parts[0], "user": parts[1],
                    "command": parts[2], "args": parts[3] if len(parts) > 3 else ""
                })
        return processes
    except (subprocess.TimeoutExpired, FileNotFoundError):
        return []


def detect_drift_indicators(changes, processes):
    findings = []
    for path in changes.get("added", []):
        basename = path.rsplit("/", 1)[-1]
        if basename in SUSPICIOUS_BINARIES:
            findings.append({"type": "suspicious_binary_added", "path": path,
                             "severity": "HIGH"})
        if path.startswith("/usr/bin/") or path.startswith("/usr/sbin/"):
            findings.append({"type": "binary_added_to_system_path", "path": path,
                             "severity": "HIGH"})
    for path in changes.get("modified", []):
        if path in ("/etc/passwd", "/etc/shadow", "/etc/sudoers"):
            findings.append({"type": "sensitive_file_modified", "path": path,
                             "severity": "CRITICAL"})
        if path.startswith("/etc/cron"):
            findings.append({"type": "cron_modified", "path": path, "severity": "HIGH"})

    for proc in processes:
        cmd = proc.get("command", "")
        if cmd in PACKAGE_MANAGERS:
            findings.append({"type": "package_manager_running", "process": cmd,
                             "severity": "HIGH"})
        if cmd in SHELLS and proc.get("user") == "root":
            findings.append({"type": "root_shell_active", "process": cmd,
                             "severity": "MEDIUM"})
    return findings


def check_image_digest(client, container_id):
    container = client.containers.get(container_id)
    image_id = container.image.id
    image_tags = container.image.tags
    attrs = container.attrs
    config_image = attrs.get("Config", {}).get("Image", "")
    uses_digest = "@sha256:" in config_image
    return {
        "image_id": image_id[:19],
        "image_tags": image_tags,
        "config_image": config_image,
        "uses_immutable_digest": uses_digest,
        "privileged": attrs.get("HostConfig", {}).get("Privileged", False),
        "read_only_rootfs": attrs.get("HostConfig", {}).get("ReadonlyRootfs", False),
    }


def audit_container(client, container_id):
    changes = get_container_diff(client, container_id)
    processes = get_running_processes(container_id)
    findings = detect_drift_indicators(changes, processes)
    image_info = check_image_digest(client, container_id)

    if not image_info.get("read_only_rootfs"):
        findings.append({"type": "mutable_rootfs", "severity": "MEDIUM",
                         "detail": "readOnlyRootFilesystem not enabled"})
    if image_info.get("privileged"):
        findings.append({"type": "privileged_container", "severity": "CRITICAL",
                         "detail": "Container running in privileged mode"})

    total_changes = sum(len(v) for v in changes.values())
    risk = "CRITICAL" if any(f["severity"] == "CRITICAL" for f in findings) else \
           "HIGH" if total_changes > 20 or any(f["severity"] == "HIGH" for f in findings) else \
           "MEDIUM" if total_changes > 5 else "LOW"

    return {
        "container_id": container_id,
        "timestamp": datetime.utcnow().isoformat() + "Z",
        "filesystem_changes": changes,
        "total_changes": total_changes,
        "running_processes": processes,
        "image_info": image_info,
        "findings": findings,
        "risk_level": risk,
    }


def main():
    parser = argparse.ArgumentParser(description="Container Drift Detector")
    parser.add_argument("--container", required=True, help="Container ID or name")
    parser.add_argument("--all", action="store_true", help="Audit all running containers")
    args = parser.parse_args()

    client = docker.from_env()
    results = []
    if args.all:
        for c in client.containers.list():
            results.append(audit_container(client, c.id))
    else:
        results.append(audit_container(client, args.container))

    print(json.dumps(results, indent=2))


if __name__ == "__main__":
    main()
process.py7.6 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""
Container Drift Detection Tool

Compares running containers against their original image state
to detect filesystem drift, unexpected processes, and configuration changes.
"""

import json
import subprocess
import sys
import argparse
from datetime import datetime
from collections import defaultdict


def run_command(cmd: list[str], timeout: int = 30) -> str:
    """Execute a command and return stdout."""
    try:
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
        return result.stdout.strip()
    except (subprocess.TimeoutExpired, FileNotFoundError):
        return ""


def get_running_containers(namespace: str = "") -> list[dict]:
    """Get running containers from Kubernetes."""
    cmd = ["kubectl", "get", "pods", "-o", "json"]
    if namespace:
        cmd.extend(["-n", namespace])
    else:
        cmd.append("--all-namespaces")

    output = run_command(cmd)
    if not output:
        return []

    try:
        data = json.loads(output)
        containers = []
        for pod in data.get("items", []):
            ns = pod["metadata"]["namespace"]
            pod_name = pod["metadata"]["name"]
            for cs in pod.get("status", {}).get("containerStatuses", []):
                containers.append({
                    "namespace": ns,
                    "pod": pod_name,
                    "container": cs["name"],
                    "image": cs["image"],
                    "imageID": cs.get("imageID", ""),
                    "ready": cs.get("ready", False),
                    "restartCount": cs.get("restartCount", 0),
                })
        return containers
    except (json.JSONDecodeError, KeyError):
        return []


def check_image_tag_drift(containers: list[dict]) -> list[dict]:
    """Detect containers using mutable tags instead of digests."""
    findings = []
    for c in containers:
        if "@sha256:" not in c["image"]:
            findings.append({
                "severity": "MEDIUM",
                "type": "mutable_tag",
                "namespace": c["namespace"],
                "pod": c["pod"],
                "container": c["container"],
                "image": c["image"],
                "description": f"Container uses mutable tag instead of digest: {c['image']}"
            })
    return findings


def check_readonly_filesystem(namespace: str = "") -> list[dict]:
    """Check which containers lack readOnlyRootFilesystem."""
    cmd = ["kubectl", "get", "pods", "-o", "json"]
    if namespace:
        cmd.extend(["-n", namespace])
    else:
        cmd.append("--all-namespaces")

    output = run_command(cmd)
    if not output:
        return []

    findings = []
    try:
        data = json.loads(output)
        for pod in data.get("items", []):
            ns = pod["metadata"]["namespace"]
            pod_name = pod["metadata"]["name"]
            for container in pod["spec"].get("containers", []):
                sc = container.get("securityContext", {})
                if not sc.get("readOnlyRootFilesystem", False):
                    findings.append({
                        "severity": "MEDIUM",
                        "type": "writable_filesystem",
                        "namespace": ns,
                        "pod": pod_name,
                        "container": container["name"],
                        "description": f"Container {container['name']} has writable root filesystem"
                    })
    except (json.JSONDecodeError, KeyError):
        pass

    return findings


def check_restart_anomalies(containers: list[dict], threshold: int = 3) -> list[dict]:
    """Detect containers with high restart counts (potential crash loops from drift)."""
    findings = []
    for c in containers:
        if c["restartCount"] >= threshold:
            findings.append({
                "severity": "LOW",
                "type": "high_restarts",
                "namespace": c["namespace"],
                "pod": c["pod"],
                "container": c["container"],
                "restarts": c["restartCount"],
                "description": f"Container has {c['restartCount']} restarts (may indicate instability from drift)"
            })
    return findings


def check_pod_security_standards(namespace: str = "") -> list[dict]:
    """Check namespace-level Pod Security Standards enforcement."""
    output = run_command(["kubectl", "get", "namespaces", "-o", "json"])
    if not output:
        return []

    findings = []
    try:
        data = json.loads(output)
        for ns in data.get("items", []):
            ns_name = ns["metadata"]["name"]
            if namespace and ns_name != namespace:
                continue
            labels = ns["metadata"].get("labels", {})
            enforce = labels.get("pod-security.kubernetes.io/enforce", "")
            if enforce not in ("restricted", "baseline"):
                findings.append({
                    "severity": "MEDIUM",
                    "type": "no_pss_enforcement",
                    "namespace": ns_name,
                    "enforce_level": enforce or "none",
                    "description": f"Namespace {ns_name} lacks Pod Security Standards enforcement"
                })
    except (json.JSONDecodeError, KeyError):
        pass

    return findings


def generate_report(containers: list[dict], all_findings: list[dict],
                    output_format: str = "text") -> str:
    """Generate drift detection report."""
    critical = [f for f in all_findings if f["severity"] == "CRITICAL"]
    high = [f for f in all_findings if f["severity"] == "HIGH"]
    medium = [f for f in all_findings if f["severity"] == "MEDIUM"]
    low = [f for f in all_findings if f["severity"] == "LOW"]

    if output_format == "json":
        return json.dumps({
            "timestamp": datetime.utcnow().isoformat(),
            "containers_scanned": len(containers),
            "findings": {"critical": len(critical), "high": len(high),
                        "medium": len(medium), "low": len(low)},
            "details": all_findings
        }, indent=2)

    lines = []
    lines.append("=" * 70)
    lines.append("CONTAINER DRIFT DETECTION REPORT")
    lines.append(f"Generated: {datetime.utcnow().isoformat()}")
    lines.append("=" * 70)
    lines.append(f"\nContainers Scanned: {len(containers)}")
    lines.append(f"Findings: {len(critical)} Critical, {len(high)} High, {len(medium)} Medium, {len(low)} Low")

    for severity, items in [("CRITICAL", critical), ("HIGH", high), ("MEDIUM", medium), ("LOW", low)]:
        if items:
            lines.append(f"\n## {severity} Findings")
            for f in items:
                lines.append(f"  [{f['type']}] {f['description']}")
                if "pod" in f:
                    lines.append(f"    Namespace: {f.get('namespace', 'N/A')} | Pod: {f.get('pod', 'N/A')}")

    lines.append("\n" + "=" * 70)
    return "\n".join(lines)


def main():
    parser = argparse.ArgumentParser(description="Container Drift Detection Tool")
    parser.add_argument("--namespace", default="", help="Kubernetes namespace to scan")
    parser.add_argument("--format", choices=["text", "json"], default="text")
    parser.add_argument("--restart-threshold", type=int, default=3)
    args = parser.parse_args()

    containers = get_running_containers(args.namespace)
    all_findings = []
    all_findings.extend(check_image_tag_drift(containers))
    all_findings.extend(check_readonly_filesystem(args.namespace))
    all_findings.extend(check_restart_anomalies(containers, args.restart_threshold))
    all_findings.extend(check_pod_security_standards(args.namespace))

    report = generate_report(containers, all_findings, args.format)
    print(report)


if __name__ == "__main__":
    main()

Assets 1

template.mdtext/markdown · 0.7 KB
Keep exploring