phishing defense

Detecting Spearphishing with Email Gateway

Spearphishing targets specific individuals using personalized, researched content that bypasses generic spam filters. Email security gateways (SEGs) like Microsoft Defender for Office 365, Proofpoint, Mimecast, and Barracuda provide advanced detection capabilities including behavioral analysis, URL detonation, attachment sandboxing, and impersonation detection. This skill covers configuring these gateways to detect and block targeted phishing attacks.

awarenessdmarcemail-gatewayemail-securityphishingsocial-engineeringspearphishing
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

Overview

Spearphishing targets specific individuals using personalized, researched content that bypasses generic spam filters. Email security gateways (SEGs) like Microsoft Defender for Office 365, Proofpoint, Mimecast, and Barracuda provide advanced detection capabilities including behavioral analysis, URL detonation, attachment sandboxing, and impersonation detection. This skill covers configuring these gateways to detect and block targeted phishing attacks.

When to Use

  • When investigating security incidents that require detecting spearphishing with email gateway
  • When building detection rules or threat hunting queries for this domain
  • When SOC analysts need structured procedures for this analysis type
  • When validating security monitoring coverage for related attack techniques

Prerequisites

  • Access to email security gateway admin console
  • Understanding of email flow architecture (MX records, transport rules)
  • Familiarity with SPF/DKIM/DMARC authentication
  • Knowledge of common spearphishing techniques and pretexts

Key Concepts

Spearphishing Characteristics

  • Targeted recipients: Specific individuals, often executives or finance staff
  • Researched pretexts: References to real projects, colleagues, or events
  • Impersonation: Spoofs trusted senders (CEO, vendor, partner)
  • Low volume: Few emails to avoid pattern-based detection
  • Urgent tone: Creates pressure to act quickly

Gateway Detection Layers

  1. Reputation filtering: IP/domain/URL reputation scoring
  2. Authentication checks: SPF, DKIM, DMARC validation
  3. Content analysis: NLP-based analysis of email body
  4. Impersonation detection: Display name and domain similarity matching
  5. URL analysis: Real-time URL detonation and redirect following
  6. Attachment sandboxing: Behavioral analysis of attachments in isolated environments
  7. Behavioral analytics: Anomaly detection in communication patterns

Workflow

Step 1: Configure Impersonation Protection

Microsoft Defender for Office 365:
  Security > Anti-phishing policies > Impersonation settings
  - Enable user impersonation protection for VIPs
  - Enable domain impersonation protection
  - Add protected users (CEO, CFO, HR Director)
  - Set action: Quarantine message
 
Proofpoint:
  Email Protection > Impostor Classifier
  - Enable display name spoofing detection
  - Configure lookalike domain detection
  - Set Impostor threshold sensitivity

Step 2: Configure URL Protection

  • Enable Safe Links / URL rewriting
  • Enable time-of-click URL detonation
  • Block newly registered domains (< 30 days)
  • Enable URL redirect chain following

Step 3: Configure Attachment Sandboxing

  • Enable Safe Attachments / attachment sandboxing
  • Configure dynamic delivery (deliver body, hold attachments)
  • Set sandbox detonation timeout to 60+ seconds
  • Block macro-enabled Office documents from external senders

Step 4: Create Custom Detection Rules

Use the scripts/process.py to analyze email gateway logs, identify spearphishing patterns, and generate custom detection rules.

Step 5: Configure Alert and Response Actions

  • Real-time alerts for impersonation attempts
  • Automatic quarantine for high-confidence detections
  • User notification with safety tips
  • Integration with SIEM for correlation

Tools & Resources

Validation

  • Impersonation protection correctly identifies spoofed VIP display names
  • URL detonation catches malicious links in test phishing emails
  • Attachment sandboxing detects weaponized documents
  • Custom rules trigger on known spearphishing patterns
  • SIEM integration receives gateway alerts
Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 3

api-reference.md2.2 KB

API Reference: Spearphishing Detection via Email Gateway

Python email Module

Parse EML file

import email
from email import policy
 
with open("message.eml", "rb") as f:
    msg = email.message_from_binary_file(f, policy=policy.default)

Security Headers

Header Purpose
Received-SPF SPF check result
Authentication-Results SPF, DKIM, DMARC combined
DKIM-Signature DKIM signing info
ARC-Authentication-Results ARC chain results
X-Mailer Client used to send
Return-Path Envelope sender

Authentication-Results Values

Authentication-Results: mx.google.com;
    dkim=pass header.d=example.com;
    spf=pass smtp.mailfrom=example.com;
    dmarc=pass

SPF Record Lookup

dig TXT example.com | grep "v=spf1"
# v=spf1 include:_spf.google.com ~all

SPF Results

Result Meaning
pass Authorized sender
fail Unauthorized (reject)
softfail Unauthorized (accept with mark)
neutral No assertion
none No SPF record

DKIM Verification

opendkim-testkey -d example.com -s selector -vvv

DMARC Record

dig TXT _dmarc.example.com
# v=DMARC1; p=reject; rua=mailto:dmarc@example.com

Microsoft Defender for Office 365 API

Get email threat assessment

POST https://graph.microsoft.com/v1.0/informationProtection/threatAssessmentRequests
Authorization: Bearer {token}
 
{
  "contentType": "mail",
  "expectedAssessment": "block",
  "category": "phishing",
  "mailInfo": {
    "internetMessageId": "<message-id>"
  }
}

Proofpoint TAP API

Get blocked messages

GET https://tap-api-v2.proofpoint.com/v2/siem/messages/blocked
    ?sinceSeconds=3600
Authorization: Basic {base64_credentials}

Response Fields

Field Description
spamScore Spam confidence (0-100)
phishScore Phishing confidence (0-100)
threatsInfoMap Threat details array
fromAddress Envelope sender

Mimecast API — URL Protection

Decode Mimecast URL

POST https://api.mimecast.com/api/ttp/url/decode-url
Authorization: MC {access-key}:{secret-key}
 
{
  "data": [{"url": "https://protect.mimecast.com/..."}]
}
standards.md2.4 KB

Standards & References: Detecting Spearphishing with Email Gateway

MITRE ATT&CK References

  • T1566.001: Phishing: Spearphishing Attachment
  • T1566.002: Phishing: Spearphishing Link
  • T1566.003: Phishing: Spearphishing via Service
  • T1598.002: Phishing for Information: Spearphishing Attachment
  • T1598.003: Phishing for Information: Spearphishing Link
  • T1534: Internal Spearphishing

NIST Guidelines

  • NIST SP 800-177 Rev.1: Trustworthy Email
  • NIST SP 800-53 Rev.5: SI-8 Spam Protection, SI-3 Malicious Code Protection
  • NIST CSF: PR.AT (Awareness & Training), DE.CM (Security Continuous Monitoring)

CIS Controls v8

  • CIS Control 9: Email and Web Browser Protections
    • 9.1: Ensure only approved browsers and email clients are used
    • 9.2: Use DNS filtering services
    • 9.3: Maintain and enforce network-based URL filters
    • 9.6: Block unnecessary file types
    • 9.7: Deploy and maintain email server anti-malware protections

Email Gateway Feature Matrix

Feature Microsoft Defender Proofpoint Mimecast Barracuda
Impersonation detection Anti-phishing policy Impostor Classifier Brand Exploit Protect Impersonation Protection
URL detonation Safe Links URL Defense URL Protect Link Protection
Attachment sandbox Safe Attachments Targeted Attack Protection Attachment Protect Advanced Threat Protection
DMARC enforcement Built-in Built-in DMARC Analyzer Built-in
AI/ML detection Yes (multiple models) NexusAI Yes Yes
User reporting Report Message add-in PhishAlarm Built-in Phishline
SIEM integration Microsoft Sentinel Splunk, QRadar Splunk, Sentinel Various
Auto-remediation (ZAP) Yes CLEAR Yes Yes

Detection Indicators for Spearphishing

Indicator Weight Description
Display name spoofing VIP High From name matches protected user but different email
Lookalike domain High Domain differs by 1-2 characters from legitimate
First-time sender to VIP Medium No prior communication history
Urgency keywords Medium "urgent", "immediately", "wire transfer", "confidential"
Reply-to mismatch High Reply-to differs from From address
External sender with internal branding High Email mimics internal templates
Newly registered domain High Sending domain < 30 days old
Authentication failure High SPF/DKIM/DMARC fail
workflows.md3.7 KB

Workflows: Detecting Spearphishing with Email Gateway

Workflow 1: Multi-Layer Detection Pipeline

Inbound Email Arrives at Gateway
  |
  v
[Layer 1: Connection Filtering]
  +-- Check sender IP reputation
  +-- Check RBL/DNSBL blacklists
  +-- Rate limiting / throttling
  |
  v
[Layer 2: Authentication]
  +-- Verify SPF alignment
  +-- Verify DKIM signature
  +-- Evaluate DMARC policy
  +-- Check ARC headers (forwarded mail)
  |
  v
[Layer 3: Impersonation Detection]
  +-- Compare display name against VIP list
  +-- Check domain similarity (Levenshtein distance)
  +-- Evaluate sender reputation
  +-- First-time sender analysis
  |
  v
[Layer 4: Content Analysis]
  +-- NLP analysis for urgency/social engineering
  +-- Business context anomaly detection
  +-- Keyword pattern matching
  +-- Language analysis
  |
  v
[Layer 5: URL Analysis]
  +-- URL reputation check
  +-- Domain age verification
  +-- Real-time URL detonation
  +-- Redirect chain following
  +-- Visual similarity to legitimate sites
  |
  v
[Layer 6: Attachment Analysis]
  +-- File type validation
  +-- Sandbox detonation
  +-- Macro analysis
  +-- Embedded object detection
  |
  v
[Decision Engine]
  +-- Aggregate scores from all layers
  +-- Apply organizational policy
  |
  +-- DELIVER: Low risk
  +-- TAG: Add warning banner
  +-- QUARANTINE: Moderate risk
  +-- BLOCK: High risk, drop message

Workflow 2: VIP Impersonation Detection

Email arrives with From display name matching VIP list
  |
  v
[Check: Is sending domain authorized for this VIP?]
  |
  +-- YES: Check DKIM/SPF --> If pass, deliver normally
  |
  +-- NO: Impersonation suspected
       |
       v
  [Calculate domain similarity score]
       |
       +-- Exact match (different email): CRITICAL - Block
       +-- Lookalike domain (1-2 char diff): HIGH - Quarantine
       +-- Similar but different: MEDIUM - Tag with warning
       |
       v
  [Additional checks]
       +-- Has this sender emailed before?
       +-- Is the sending infrastructure legitimate?
       +-- Does email content match typical VIP communication?
       |
       v
  [Action: Quarantine + Alert SOC + Notify recipient manager]

Workflow 3: Spearphishing Response

Gateway detects potential spearphishing
  |
  v
[Automated Response]
  +-- Quarantine message
  +-- Generate alert in SIEM
  +-- Extract IOCs (sender, domain, URLs, hashes)
  |
  v
[SOC Analyst Review]
  +-- Review quarantined message
  +-- Analyze full headers
  +-- Investigate sending infrastructure
  +-- Check if other users received similar emails
  |
  +-- FALSE POSITIVE
  |     +-- Release from quarantine
  |     +-- Whitelist if legitimate
  |     +-- Update detection rules
  |
  +-- CONFIRMED SPEARPHISHING
        +-- Block sender domain organization-wide
        +-- Search mailboxes for similar messages (retroactive)
        +-- Auto-purge any delivered copies (ZAP)
        +-- Notify targeted users
        +-- Submit IOCs to threat intelligence
        +-- Check for any successful credential compromise
        +-- Update VIP protection list if needed

Workflow 4: Gateway Tuning Cycle

Monthly Review
  |
  +-- Pull detection statistics from gateway
  +-- Analyze false positive rate
  +-- Analyze false negative rate (user-reported misses)
  +-- Review quarantine volumes
  |
  v
[Identify gaps]
  +-- New impersonation patterns?
  +-- New sending domains to whitelist/blacklist?
  +-- Policy thresholds too aggressive/permissive?
  |
  v
[Adjust configuration]
  +-- Update VIP protection list (new hires, departures)
  +-- Tune sensitivity thresholds
  +-- Add custom transport rules
  +-- Update URL/domain blocklists
  |
  v
[Validate changes]
  +-- Send test phishing emails
  +-- Verify legitimate mail still flows
  +-- Document changes

Scripts 2

agent.py5.6 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""Agent for detecting spearphishing emails using email gateway log analysis."""

import argparse
import email
import json
import os
import re
from datetime import datetime, timezone
from email import policy


EXECUTIVE_TITLES = [
    "ceo", "cfo", "cto", "cio", "coo", "president", "director",
    "vp", "vice president", "managing director", "partner",
]
URGENCY_KEYWORDS = [
    "urgent", "immediate", "asap", "right away", "time sensitive",
    "confidential", "do not share", "wire transfer", "bank account",
    "update payment", "invoice attached", "past due",
]
SUSPICIOUS_EXTENSIONS = {
    ".exe", ".scr", ".bat", ".cmd", ".ps1", ".vbs", ".js",
    ".hta", ".lnk", ".iso", ".img", ".dll", ".msi",
}


def parse_email_headers(eml_path):
    """Extract security-relevant headers from an email."""
    with open(eml_path, "rb") as f:
        msg = email.message_from_binary_file(f, policy=policy.default)
    headers = {
        "from": msg.get("From", ""),
        "to": msg.get("To", ""),
        "subject": msg.get("Subject", ""),
        "reply_to": msg.get("Reply-To", ""),
        "return_path": msg.get("Return-Path", ""),
        "received_spf": msg.get("Received-SPF", ""),
        "dkim_signature": msg.get("DKIM-Signature", ""),
        "authentication_results": msg.get("Authentication-Results", ""),
        "x_mailer": msg.get("X-Mailer", ""),
        "message_id": msg.get("Message-ID", ""),
    }
    return headers, msg


def check_authentication(headers):
    """Verify SPF, DKIM, DMARC authentication results."""
    issues = []
    auth_results = headers.get("authentication_results", "").lower()
    if "spf=fail" in auth_results or "spf=softfail" in auth_results:
        issues.append("SPF validation failed")
    if "dkim=fail" in auth_results:
        issues.append("DKIM validation failed")
    if "dmarc=fail" in auth_results:
        issues.append("DMARC validation failed")
    if not headers.get("dkim_signature"):
        issues.append("No DKIM signature present")
    from_domain = re.search(r"@([\w.-]+)", headers.get("from", ""))
    reply_domain = re.search(r"@([\w.-]+)", headers.get("reply_to", ""))
    if from_domain and reply_domain and from_domain.group(1) != reply_domain.group(1):
        issues.append(f"Reply-To domain mismatch: {reply_domain.group(1)} vs {from_domain.group(1)}")
    return issues


def check_content_indicators(msg):
    """Analyze email body for spearphishing content indicators."""
    indicators = []
    body = ""
    for part in msg.walk():
        if part.get_content_type() in ("text/plain", "text/html"):
            payload = part.get_payload(decode=True)
            if payload:
                body += payload.decode("utf-8", errors="replace")

    body_lower = body.lower()
    for kw in URGENCY_KEYWORDS:
        if kw in body_lower:
            indicators.append(f"Urgency keyword: '{kw}'")

    urls = re.findall(r'https?://[^\s<>"\']+', body)
    for url in urls[:10]:
        if re.search(r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}', url):
            indicators.append(f"URL with IP address: {url[:80]}")
        if len(url) > 100:
            indicators.append(f"Unusually long URL: {url[:80]}...")

    attachments = []
    for part in msg.walk():
        fname = part.get_filename()
        if fname:
            ext = os.path.splitext(fname)[1].lower()
            attachments.append(fname)
            if ext in SUSPICIOUS_EXTENSIONS:
                indicators.append(f"Suspicious attachment: {fname}")
            if ".." in fname or ext == ".exe.pdf":
                indicators.append(f"Double extension: {fname}")

    return indicators, attachments


def analyze_email(eml_path):
    """Full spearphishing analysis of a single email."""
    headers, msg = parse_email_headers(eml_path)
    auth_issues = check_authentication(headers)
    content_indicators, attachments = check_content_indicators(msg)

    all_indicators = auth_issues + content_indicators
    score = len(all_indicators) * 15
    risk = "CRITICAL" if score >= 75 else "HIGH" if score >= 50 else "MEDIUM" if score >= 25 else "LOW"

    return {
        "file": eml_path,
        "from": headers["from"],
        "to": headers["to"],
        "subject": headers["subject"],
        "auth_issues": auth_issues,
        "content_indicators": content_indicators,
        "attachments": attachments,
        "risk_score": min(score, 100),
        "risk_level": risk,
    }


def main():
    parser = argparse.ArgumentParser(
        description="Detect spearphishing via email gateway analysis"
    )
    parser.add_argument("input", help=".eml file or directory")
    parser.add_argument("--output", "-o", help="Output JSON report")
    parser.add_argument("--verbose", "-v", action="store_true")
    args = parser.parse_args()

    print("[*] Spearphishing Detection Agent")
    results = []

    if os.path.isdir(args.input):
        for root, _, files in os.walk(args.input):
            for f in files:
                if f.lower().endswith(".eml"):
                    results.append(analyze_email(os.path.join(root, f)))
    else:
        results.append(analyze_email(args.input))

    flagged = [r for r in results if r["risk_level"] in ("HIGH", "CRITICAL")]
    report = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "emails_scanned": len(results),
        "spearphishing_detected": len(flagged),
        "results": results,
    }

    print(f"[*] Scanned {len(results)} emails, {len(flagged)} flagged")

    if args.output:
        with open(args.output, "w") as f:
            json.dump(report, f, indent=2)
        print(f"[*] Report saved to {args.output}")
    else:
        print(json.dumps(report, indent=2))


if __name__ == "__main__":
    main()
process.py21.1 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""
Spearphishing Detection Engine for Email Gateway Logs

Analyzes email gateway logs to detect spearphishing patterns including
impersonation, lookalike domains, and behavioral anomalies. Generates
custom detection rules and threat reports.

Usage:
    python process.py analyze --log-file gateway_log.json
    python process.py detect --email-file email.eml
    python process.py rules --output detection_rules.yaml
"""

import argparse
import json
import re
import sys
import hashlib
from datetime import datetime, timezone
from pathlib import Path
from dataclasses import dataclass, field, asdict
from typing import Optional
from collections import defaultdict

try:
    import requests
    HAS_REQUESTS = True
except ImportError:
    HAS_REQUESTS = False


@dataclass
class SpearphishingIndicator:
    """A detected spearphishing indicator."""
    indicator_type: str = ""
    description: str = ""
    severity: str = "medium"
    confidence: float = 0.0
    raw_value: str = ""
    mitre_technique: str = ""


@dataclass
class DomainSimilarity:
    """Domain similarity analysis result."""
    original_domain: str = ""
    suspicious_domain: str = ""
    distance: int = 0
    technique: str = ""
    confidence: float = 0.0


@dataclass
class EmailAnalysis:
    """Complete spearphishing analysis for a single email."""
    message_id: str = ""
    from_address: str = ""
    from_display_name: str = ""
    from_domain: str = ""
    to_address: str = ""
    subject: str = ""
    date: str = ""
    indicators: list = field(default_factory=list)
    risk_score: float = 0.0
    risk_level: str = "low"
    domain_similarities: list = field(default_factory=list)
    is_spearphishing: bool = False


# VIP list for impersonation detection (configure per organization)
DEFAULT_VIP_NAMES = [
    "CEO", "CFO", "CTO", "CISO", "COO",
    "Chief Executive", "Chief Financial", "Chief Technology",
    "President", "Vice President", "Director",
]

# Common urgency keywords in spearphishing
URGENCY_KEYWORDS = [
    r'\burgent\b', r'\bimmediately\b', r'\basap\b', r'\btime.?sensitive\b',
    r'\bwire\s+transfer\b', r'\bbank\s+transfer\b', r'\bgift\s+card\b',
    r'\bconfidential\b', r'\bdo\s+not\s+share\b', r'\bsecret\b',
    r'\bpayment\b', r'\binvoice\b', r'\boverdue\b', r'\bfinal\s+notice\b',
    r'\baccount\s+suspen', r'\bverify\s+your\b', r'\bconfirm\s+your\b',
    r'\bunusual\s+activity\b', r'\bsecurity\s+alert\b',
]

# Legitimate domains for similarity comparison
COMMON_TARGETS = [
    "microsoft.com", "google.com", "apple.com", "amazon.com",
    "paypal.com", "netflix.com", "linkedin.com", "facebook.com",
    "dropbox.com", "docusign.com", "zoom.us", "slack.com",
    "office365.com", "outlook.com", "gmail.com",
]


def levenshtein_distance(s1: str, s2: str) -> int:
    """Calculate Levenshtein edit distance between two strings."""
    if len(s1) < len(s2):
        return levenshtein_distance(s2, s1)
    if len(s2) == 0:
        return len(s1)
    prev_row = range(len(s2) + 1)
    for i, c1 in enumerate(s1):
        curr_row = [i + 1]
        for j, c2 in enumerate(s2):
            insertions = prev_row[j + 1] + 1
            deletions = curr_row[j] + 1
            substitutions = prev_row[j] + (c1 != c2)
            curr_row.append(min(insertions, deletions, substitutions))
        prev_row = curr_row
    return prev_row[-1]


def detect_homograph(domain: str) -> list:
    """Detect IDN homograph attacks in domain names."""
    homograph_map = {
        'a': ['@', 'а', 'ɑ'],   # Cyrillic а, Latin alpha
        'e': ['е', 'ё', 'э'],   # Cyrillic variants
        'o': ['о', '0', 'ο'],   # Cyrillic о, zero, Greek omicron
        'p': ['р', 'ρ'],        # Cyrillic р, Greek rho
        'c': ['с', 'ç'],        # Cyrillic с
        'x': ['х', 'χ'],        # Cyrillic х, Greek chi
        'y': ['у', 'γ'],        # Cyrillic у
        'i': ['і', 'і', '1', 'l'],
    }
    findings = []
    for i, char in enumerate(domain):
        for latin, lookalikes in homograph_map.items():
            if char in lookalikes:
                findings.append({
                    "position": i,
                    "character": char,
                    "looks_like": latin,
                    "type": "homograph"
                })
    return findings


def check_domain_similarity(domain: str, known_domains: list = None) -> list:
    """Check if a domain is similar to known legitimate domains."""
    if known_domains is None:
        known_domains = COMMON_TARGETS

    similarities = []
    domain_base = domain.split(".")[0] if "." in domain else domain

    for known in known_domains:
        known_base = known.split(".")[0]
        dist = levenshtein_distance(domain_base.lower(), known_base.lower())

        if 0 < dist <= 2:
            technique = "typosquatting"
            if len(domain_base) != len(known_base):
                technique = "character_addition" if len(domain_base) > len(known_base) else "character_omission"
            elif dist == 1:
                for i, (a, b) in enumerate(zip(domain_base, known_base)):
                    if a != b:
                        if i > 0 and domain_base[i-1:i+1] == known_base[i:i-2:-1] if i > 0 else False:
                            technique = "transposition"
                        break

            confidence = max(0, 1.0 - dist * 0.3)
            similarities.append(DomainSimilarity(
                original_domain=known,
                suspicious_domain=domain,
                distance=dist,
                technique=technique,
                confidence=confidence
            ))

    # Check for homographs
    homographs = detect_homograph(domain)
    if homographs:
        similarities.append(DomainSimilarity(
            original_domain="(homograph detection)",
            suspicious_domain=domain,
            distance=0,
            technique="homograph",
            confidence=0.9
        ))

    return sorted(similarities, key=lambda x: x.distance)


def detect_urgency(text: str) -> list:
    """Detect urgency patterns in email text."""
    findings = []
    text_lower = text.lower()
    for pattern in URGENCY_KEYWORDS:
        matches = re.findall(pattern, text_lower, re.IGNORECASE)
        if matches:
            findings.append({
                "pattern": pattern,
                "matches": matches,
                "count": len(matches)
            })
    return findings


def detect_impersonation(display_name: str, vip_names: list = None) -> list:
    """Check if display name impersonates a VIP."""
    if vip_names is None:
        vip_names = DEFAULT_VIP_NAMES

    findings = []
    name_lower = display_name.lower()

    for vip in vip_names:
        if vip.lower() in name_lower:
            findings.append({
                "matched_vip": vip,
                "display_name": display_name,
                "confidence": 0.8
            })

    return findings


def analyze_email(headers: dict, body: str = "",
                  vip_names: list = None,
                  known_domains: list = None) -> EmailAnalysis:
    """Analyze a single email for spearphishing indicators."""
    analysis = EmailAnalysis()

    analysis.from_address = headers.get("from", "")
    analysis.from_display_name = headers.get("from_display_name", "")
    analysis.from_domain = headers.get("from_domain", "")
    analysis.to_address = headers.get("to", "")
    analysis.subject = headers.get("subject", "")
    analysis.date = headers.get("date", "")
    analysis.message_id = headers.get("message_id", "")

    # Extract domain from from_address if not provided
    if not analysis.from_domain and analysis.from_address:
        match = re.search(r'@([\w.-]+)', analysis.from_address)
        if match:
            analysis.from_domain = match.group(1).lower()

    # Extract display name if not provided
    if not analysis.from_display_name and analysis.from_address:
        match = re.match(r'"?([^"<]+)"?\s*<', analysis.from_address)
        if match:
            analysis.from_display_name = match.group(1).strip()

    score = 0.0

    # Check 1: Domain similarity
    if analysis.from_domain:
        similarities = check_domain_similarity(analysis.from_domain, known_domains)
        analysis.domain_similarities = similarities
        for sim in similarities:
            if sim.distance <= 1:
                analysis.indicators.append(SpearphishingIndicator(
                    indicator_type="lookalike_domain",
                    description=f"Domain '{sim.suspicious_domain}' is {sim.distance} edit(s) "
                                f"from '{sim.original_domain}' ({sim.technique})",
                    severity="critical" if sim.distance == 1 else "high",
                    confidence=sim.confidence,
                    raw_value=sim.suspicious_domain,
                    mitre_technique="T1566.002"
                ))
                score += 30 * sim.confidence
            elif sim.distance == 2:
                analysis.indicators.append(SpearphishingIndicator(
                    indicator_type="similar_domain",
                    description=f"Domain '{sim.suspicious_domain}' resembles "
                                f"'{sim.original_domain}' (distance={sim.distance})",
                    severity="medium",
                    confidence=sim.confidence,
                    raw_value=sim.suspicious_domain,
                    mitre_technique="T1566.002"
                ))
                score += 15 * sim.confidence

    # Check 2: VIP impersonation
    if analysis.from_display_name:
        impersonations = detect_impersonation(analysis.from_display_name, vip_names)
        for imp in impersonations:
            analysis.indicators.append(SpearphishingIndicator(
                indicator_type="vip_impersonation",
                description=f"Display name '{imp['display_name']}' matches VIP "
                            f"keyword '{imp['matched_vip']}'",
                severity="high",
                confidence=imp["confidence"],
                raw_value=analysis.from_display_name,
                mitre_technique="T1566.001"
            ))
            score += 25 * imp["confidence"]

    # Check 3: Urgency indicators in subject
    urgency_subject = detect_urgency(analysis.subject)
    if urgency_subject:
        analysis.indicators.append(SpearphishingIndicator(
            indicator_type="urgency_subject",
            description=f"Subject contains {len(urgency_subject)} urgency pattern(s)",
            severity="medium",
            confidence=min(len(urgency_subject) * 0.3, 0.9),
            raw_value=analysis.subject,
            mitre_technique="T1566"
        ))
        score += min(len(urgency_subject) * 5, 20)

    # Check 4: Urgency indicators in body
    if body:
        urgency_body = detect_urgency(body)
        if urgency_body:
            total_matches = sum(u["count"] for u in urgency_body)
            analysis.indicators.append(SpearphishingIndicator(
                indicator_type="urgency_body",
                description=f"Body contains {total_matches} urgency keyword(s) "
                            f"across {len(urgency_body)} pattern(s)",
                severity="medium",
                confidence=min(total_matches * 0.15, 0.9),
                raw_value=f"{total_matches} matches",
                mitre_technique="T1566"
            ))
            score += min(total_matches * 3, 15)

    # Check 5: Authentication failures
    auth_results = headers.get("authentication_results", "")
    if auth_results:
        if "spf=fail" in auth_results.lower() or "spf=softfail" in auth_results.lower():
            analysis.indicators.append(SpearphishingIndicator(
                indicator_type="spf_failure",
                description="SPF authentication failed",
                severity="high",
                confidence=0.7,
                raw_value=auth_results,
                mitre_technique="T1566"
            ))
            score += 20

        if "dkim=fail" in auth_results.lower():
            analysis.indicators.append(SpearphishingIndicator(
                indicator_type="dkim_failure",
                description="DKIM authentication failed",
                severity="high",
                confidence=0.7,
                raw_value=auth_results,
                mitre_technique="T1566"
            ))
            score += 20

        if "dmarc=fail" in auth_results.lower():
            analysis.indicators.append(SpearphishingIndicator(
                indicator_type="dmarc_failure",
                description="DMARC authentication failed",
                severity="critical",
                confidence=0.8,
                raw_value=auth_results,
                mitre_technique="T1566"
            ))
            score += 25

    # Check 6: Reply-to mismatch
    reply_to = headers.get("reply_to", "")
    if reply_to and analysis.from_address:
        reply_domain = ""
        match = re.search(r'@([\w.-]+)', reply_to)
        if match:
            reply_domain = match.group(1).lower()
        if reply_domain and reply_domain != analysis.from_domain:
            analysis.indicators.append(SpearphishingIndicator(
                indicator_type="reply_to_mismatch",
                description=f"Reply-To domain ({reply_domain}) differs from "
                            f"From domain ({analysis.from_domain})",
                severity="high",
                confidence=0.85,
                raw_value=f"From: {analysis.from_domain}, Reply-To: {reply_domain}",
                mitre_technique="T1566"
            ))
            score += 20

    # Calculate final risk
    analysis.risk_score = min(score, 100)
    if analysis.risk_score >= 70:
        analysis.risk_level = "critical"
        analysis.is_spearphishing = True
    elif analysis.risk_score >= 50:
        analysis.risk_level = "high"
        analysis.is_spearphishing = True
    elif analysis.risk_score >= 30:
        analysis.risk_level = "medium"
    elif analysis.risk_score >= 10:
        analysis.risk_level = "low"
    else:
        analysis.risk_level = "clean"

    return analysis


def generate_detection_rules(indicators_db: list) -> str:
    """Generate YAML detection rules from accumulated indicators."""
    rules = []

    # Group by indicator type
    by_type = defaultdict(list)
    for ind in indicators_db:
        by_type[ind["indicator_type"]].append(ind)

    rule_id = 1
    for ind_type, indicators in by_type.items():
        values = list(set(ind.get("raw_value", "") for ind in indicators if ind.get("raw_value")))
        if not values:
            continue

        rule = {
            "id": f"SPEAR-{rule_id:04d}",
            "name": f"Spearphishing {ind_type.replace('_', ' ').title()} Detection",
            "type": ind_type,
            "severity": indicators[0].get("severity", "medium"),
            "mitre": indicators[0].get("mitre_technique", "T1566"),
            "description": indicators[0].get("description", ""),
            "action": "quarantine" if indicators[0].get("severity") in ("high", "critical") else "tag",
            "values_count": len(values),
            "sample_values": values[:5]
        }
        rules.append(rule)
        rule_id += 1

    # Format as YAML-like output
    output_lines = ["# Auto-generated Spearphishing Detection Rules",
                    f"# Generated: {datetime.now(timezone.utc).isoformat()}",
                    f"# Total rules: {len(rules)}", ""]

    for rule in rules:
        output_lines.append(f"- id: {rule['id']}")
        output_lines.append(f"  name: \"{rule['name']}\"")
        output_lines.append(f"  type: {rule['type']}")
        output_lines.append(f"  severity: {rule['severity']}")
        output_lines.append(f"  mitre: {rule['mitre']}")
        output_lines.append(f"  action: {rule['action']}")
        output_lines.append(f"  indicators_count: {rule['values_count']}")
        output_lines.append(f"  sample_values:")
        for val in rule["sample_values"]:
            output_lines.append(f"    - \"{val}\"")
        output_lines.append("")

    return "\n".join(output_lines)


def format_analysis_report(analysis: EmailAnalysis) -> str:
    """Format analysis as text report."""
    lines = []
    lines.append("=" * 60)
    lines.append("  SPEARPHISHING DETECTION REPORT")
    lines.append("=" * 60)
    lines.append(f"  Risk Level: {analysis.risk_level.upper()} "
                 f"(Score: {analysis.risk_score:.0f}/100)")
    lines.append(f"  Verdict: {'SPEARPHISHING DETECTED' if analysis.is_spearphishing else 'NOT DETECTED'}")
    lines.append("")
    lines.append(f"  From: {analysis.from_display_name} <{analysis.from_address}>")
    lines.append(f"  To: {analysis.to_address}")
    lines.append(f"  Subject: {analysis.subject}")
    lines.append(f"  Date: {analysis.date}")
    lines.append("")

    if analysis.indicators:
        lines.append(f"[INDICATORS] ({len(analysis.indicators)} found)")
        for i, ind in enumerate(analysis.indicators, 1):
            lines.append(f"  {i}. [{ind.severity.upper()}] {ind.description}")
            lines.append(f"     Type: {ind.indicator_type} | "
                         f"MITRE: {ind.mitre_technique} | "
                         f"Confidence: {ind.confidence:.0%}")
    else:
        lines.append("[INDICATORS] None found")

    if analysis.domain_similarities:
        lines.append(f"\n[DOMAIN ANALYSIS]")
        for sim in analysis.domain_similarities:
            lines.append(f"  {sim.suspicious_domain} ~ {sim.original_domain} "
                         f"(distance={sim.distance}, technique={sim.technique})")

    lines.append("=" * 60)
    return "\n".join(lines)


def main():
    parser = argparse.ArgumentParser(description="Spearphishing Detection Engine")
    subparsers = parser.add_subparsers(dest="command")

    analyze_parser = subparsers.add_parser("analyze", help="Analyze gateway log file")
    analyze_parser.add_argument("--log-file", required=True, help="JSON log file")
    analyze_parser.add_argument("--output", "-o", help="Output file")

    detect_parser = subparsers.add_parser("detect", help="Detect spearphishing in single email")
    detect_parser.add_argument("--from", dest="from_addr", required=True)
    detect_parser.add_argument("--from-name", default="")
    detect_parser.add_argument("--to", dest="to_addr", default="")
    detect_parser.add_argument("--subject", default="")
    detect_parser.add_argument("--body", default="")
    detect_parser.add_argument("--auth-results", default="")

    domain_parser = subparsers.add_parser("check-domain", help="Check domain similarity")
    domain_parser.add_argument("domain", help="Domain to check")

    rules_parser = subparsers.add_parser("rules", help="Generate detection rules from log")
    rules_parser.add_argument("--log-file", required=True)
    rules_parser.add_argument("--output", "-o", help="Output rules file")

    parser.add_argument("--json", action="store_true", help="Output as JSON")

    args = parser.parse_args()

    if args.command == "analyze":
        with open(args.log_file, "r") as f:
            log_entries = json.load(f)

        results = []
        for entry in log_entries:
            headers = entry.get("headers", entry)
            body = entry.get("body", "")
            analysis = analyze_email(headers, body)
            results.append(analysis)

        spearphishing_count = sum(1 for r in results if r.is_spearphishing)
        print(f"Analyzed {len(results)} emails, detected {spearphishing_count} spearphishing attempts")
        print()

        for analysis in results:
            if analysis.is_spearphishing:
                if args.json:
                    print(json.dumps(asdict(analysis), indent=2, default=str))
                else:
                    print(format_analysis_report(analysis))
                print()

    elif args.command == "detect":
        headers = {
            "from": args.from_addr,
            "from_display_name": args.from_name,
            "to": args.to_addr,
            "subject": args.subject,
            "authentication_results": args.auth_results,
        }
        analysis = analyze_email(headers, args.body)
        if args.json:
            print(json.dumps(asdict(analysis), indent=2, default=str))
        else:
            print(format_analysis_report(analysis))

    elif args.command == "check-domain":
        similarities = check_domain_similarity(args.domain)
        if similarities:
            print(f"Domain '{args.domain}' is similar to:")
            for sim in similarities:
                print(f"  - {sim.original_domain} (distance={sim.distance}, "
                      f"technique={sim.technique}, confidence={sim.confidence:.0%})")
        else:
            print(f"Domain '{args.domain}' has no known similarities")

    elif args.command == "rules":
        with open(args.log_file, "r") as f:
            log_entries = json.load(f)

        all_indicators = []
        for entry in log_entries:
            analysis = analyze_email(entry.get("headers", entry), entry.get("body", ""))
            for ind in analysis.indicators:
                all_indicators.append(asdict(ind))

        rules = generate_detection_rules(all_indicators)
        if args.output:
            with open(args.output, "w") as f:
                f.write(rules)
            print(f"Rules written to {args.output}")
        else:
            print(rules)

    else:
        parser.print_help()


if __name__ == "__main__":
    main()

Assets 1

template.mdtext/markdown · 1.9 KB
Keep exploring