red teaming

Performing Open Source Intelligence Gathering

Open Source Intelligence (OSINT) gathering is the first active phase of a red team engagement, where operators collect publicly available information about the target organization to identify attack surfaces, potential targets for social engineering, technology stacks, and credential exposures. Effective OSINT directly shapes initial access strategies and reduces operational risk.

adversary-simulationexploitationmitre-attackosintpost-exploitationreconnaissancered-team
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

Legal Notice: This skill is for authorized security testing and educational purposes only. Unauthorized use against systems you do not own or have written permission to test is illegal and may violate computer fraud laws.

Overview

Open Source Intelligence (OSINT) gathering is the first active phase of a red team engagement, where operators collect publicly available information about the target organization to identify attack surfaces, potential targets for social engineering, technology stacks, and credential exposures. Effective OSINT directly shapes initial access strategies and reduces operational risk.

When to Use

  • When conducting security assessments that involve performing open source intelligence gathering
  • When following incident response procedures for related security events
  • When performing scheduled security testing or auditing activities
  • When validating security controls through hands-on testing

Prerequisites

  • Familiarity with red teaming concepts and tools
  • Access to a test or lab environment for safe execution
  • Python 3.8+ with required dependencies installed
  • Appropriate authorization for any testing activities

Objectives

  • Enumerate the target organization's external attack surface (domains, IPs, cloud assets)
  • Identify employees and their roles for social engineering targeting
  • Discover leaked credentials, API keys, and sensitive documents
  • Map the organization's technology stack and vendors
  • Identify physical locations, office layouts, and access control details
  • Build target profiles for spearphishing campaign development

Core Concepts

OSINT Categories

Category Sources Value
Domain Intelligence DNS records, WHOIS, CT logs, subdomain enumeration Network attack surface
Personnel Intelligence LinkedIn, social media, conference talks, publications Social engineering targets
Credential Intelligence Breach databases, paste sites, GitHub leaks Valid credential discovery
Technology Intelligence Job postings, Wappalyzer, Shodan, Censys Vulnerability identification
Physical Intelligence Google Maps, social media photos, Glassdoor Physical access planning
Document Intelligence SEC filings, public documents, metadata extraction Organizational structure

MITRE ATT&CK Mapping

  • T1595.001 - Active Scanning: Scanning IP Blocks
  • T1595.002 - Active Scanning: Vulnerability Scanning
  • T1592 - Gather Victim Host Information
  • T1589 - Gather Victim Identity Information
  • T1590 - Gather Victim Network Information
  • T1591 - Gather Victim Org Information
  • T1593 - Search Open Websites/Domains
  • T1594 - Search Victim-Owned Websites
  • T1596 - Search Open Technical Databases

Workflow

Phase 1: Domain and Network Reconnaissance

  1. Perform WHOIS lookups for target domains
  2. Enumerate subdomains using Certificate Transparency logs, DNS brute-force, and web scraping
  3. Identify IP ranges and ASN ownership
  4. Scan for exposed services using Shodan/Censys
  5. Check for cloud storage buckets (S3, Azure Blob, GCS)
  6. Map CDN and hosting providers

Phase 2: Personnel and Social Intelligence

  1. Enumerate employees via LinkedIn, company website, and conference speaker lists
  2. Identify email naming conventions
  3. Discover personal social media accounts of key targets
  4. Map organizational hierarchy and reporting structure
  5. Identify recently hired IT/security personnel
  6. Check for conference presentations and technical publications

Phase 3: Credential and Data Leak Discovery

  1. Search breach databases (Have I Been Pwned, DeHashed)
  2. Check paste sites (Pastebin, GitHub Gists)
  3. Search GitHub/GitLab for leaked secrets and API keys
  4. Look for exposed configuration files and backups
  5. Check for leaked internal documents via Google dorking

Phase 4: Technology Stack Identification

  1. Analyze job postings for technology mentions
  2. Use Wappalyzer/BuiltWith for web technology fingerprinting
  3. Check for exposed admin panels and development environments
  4. Identify VPN and remote access technologies
  5. Map cloud services and SaaS applications

Tools and Resources

Tool Purpose Type
Amass Subdomain enumeration and network mapping Open Source
Subfinder Passive subdomain discovery Open Source
theHarvester Email, subdomain, and name harvesting Open Source
Maltego Visual link analysis and data correlation Commercial
SpiderFoot Automated OSINT collection Open Source
Shodan Internet-connected device search Commercial
Censys Internet asset discovery Commercial
Recon-ng Web reconnaissance framework Open Source
GitDorker GitHub secret scanning Open Source
Photon Web crawler for OSINT Open Source

Validation Criteria

  • Complete list of target domains and subdomains
  • Employee list with roles and email addresses
  • Technology stack identified
  • Credential leak assessment completed
  • Attack surface map documented
  • OSINT report compiled for engagement team
Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 3

api-reference.md1.4 KB

API Reference — Performing Open Source Intelligence Gathering

Libraries Used

  • requests: HTTP requests for tech fingerprinting and social media checks
  • dns.resolver (dnspython): DNS record enumeration and subdomain discovery
  • python-whois: Domain WHOIS registration data
  • re: Email pattern extraction
  • socket: Network connectivity

CLI Interface

python agent.py whois --domain example.com
python agent.py dns --domain example.com
python agent.py email --domain example.com
python agent.py tech --url https://example.com
python agent.py social --name "John Doe"

Core Functions

whois_lookup(domain) — Domain registration data

Returns registrar, creation/expiration dates, name servers, registrant info.

dns_enumeration(domain) — DNS record and subdomain discovery

Queries 7 record types. Tests 15 common subdomain prefixes.

email_harvest(domain) — Email address discovery

Uses Hunter.io API and regex pattern matching.

technology_fingerprint(url) — Web technology identification

Detects: web server, framework, CMS. Audits 6 security headers.

social_media_search(target_name) — Profile enumeration

Checks: LinkedIn, Twitter/X, GitHub, Facebook, Instagram.

Security Headers Checked

Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, X-Content-Type-Options, X-XSS-Protection, Referrer-Policy

Dependencies

pip install requests dnspython python-whois
standards.md3.1 KB

Standards and Framework References

MITRE ATT&CK - Reconnaissance (TA0043)

Technique ID Name Description
T1595.001 Active Scanning: Scanning IP Blocks Scanning target IP ranges for active hosts
T1595.002 Active Scanning: Vulnerability Scanning Scanning for vulnerabilities on discovered hosts
T1592.001 Gather Victim Host Information: Hardware Identifying target hardware configurations
T1592.002 Gather Victim Host Information: Software Identifying installed software and versions
T1592.004 Gather Victim Host Information: Client Configurations Discovering client-side configurations
T1589.001 Gather Victim Identity Information: Credentials Searching for exposed credentials
T1589.002 Gather Victim Identity Information: Email Addresses Harvesting email addresses
T1589.003 Gather Victim Identity Information: Employee Names Collecting employee information
T1590.001 Gather Victim Network Information: Domain Properties DNS and domain enumeration
T1590.002 Gather Victim Network Information: DNS DNS record collection
T1590.004 Gather Victim Network Information: Network Topology Mapping network architecture
T1590.005 Gather Victim Network Information: IP Addresses Identifying target IP addresses
T1591.001 Gather Victim Org Information: Determine Physical Locations Physical location mapping
T1591.002 Gather Victim Org Information: Business Relationships Identifying vendors and partners
T1591.004 Gather Victim Org Information: Identify Roles Mapping organizational roles
T1593.001 Search Open Websites/Domains: Social Media Social media intelligence
T1593.002 Search Open Websites/Domains: Search Engines Google dorking and search engine recon
T1594 Search Victim-Owned Websites Analyzing target websites
T1596.001 Search Open Technical Databases: DNS/Passive DNS Passive DNS intelligence
T1596.005 Search Open Technical Databases: Scan Databases Shodan, Censys, ZoomEye queries
T1597.001 Search Closed Sources: Threat Intel Vendors Threat intelligence platform queries

PTES - Intelligence Gathering

Level 1: Passive Information Gathering

  • WHOIS lookups
  • DNS enumeration
  • Search engine queries
  • Social media analysis
  • Public records review

Level 2: Semi-Passive Information Gathering

  • Website analysis and spidering
  • Metadata extraction from documents
  • Job posting analysis
  • Technology stack identification

Level 3: Active Information Gathering

  • Port scanning
  • Service enumeration
  • Web application fingerprinting
  • Active subdomain brute-forcing

OSSTMM - Information Security Testing

Section 5: Human Security Testing

  • Social engineering reconnaissance
  • Personnel profiling
  • Communication channel mapping

Section 6: Physical Security Testing

  • Location reconnaissance
  • Access control assessment
  • Surveillance analysis

NIST SP 800-115 Section 3: Review Techniques

  • Documentation review
  • Log review
  • Ruleset review
  • System configuration review
  • Network sniffing (passive)
workflows.md6.6 KB

OSINT Gathering Workflows

Workflow 1: Domain and Infrastructure Reconnaissance

Step 1: Passive DNS and WHOIS

# WHOIS lookup
whois targetdomain.com
 
# DNS record enumeration
dig targetdomain.com ANY
dig targetdomain.com MX
dig targetdomain.com TXT
dig targetdomain.com NS
 
# Reverse DNS
dig -x <IP_ADDRESS>
 
# Zone transfer attempt
dig axfr @ns1.targetdomain.com targetdomain.com

Step 2: Subdomain Enumeration

# Using Subfinder for passive enumeration
subfinder -d targetdomain.com -o subdomains.txt
 
# Using Amass for comprehensive enumeration
amass enum -passive -d targetdomain.com -o amass_results.txt
 
# Certificate Transparency log search
curl -s "https://crt.sh/?q=%.targetdomain.com&output=json" | jq -r '.[].name_value' | sort -u
 
# Using httpx to probe discovered subdomains
cat subdomains.txt | httpx -status-code -title -tech-detect -o live_subdomains.txt

Step 3: IP Range and ASN Discovery

# ASN lookup
whois -h whois.radb.net -- '-i origin AS12345'
 
# BGP prefix lookup via Hurricane Electric
curl -s "https://bgp.he.net/AS12345#_prefixes"
 
# Shodan search for organization
shodan search "org:Target Corporation" --fields ip_str,port,product

Step 4: Cloud Asset Discovery

# AWS S3 bucket enumeration
python3 cloud_enum.py -k targetcorp -l cloud_results.txt
 
# Azure blob storage check
for name in targetcorp targetcorp-dev targetcorp-backup; do
  curl -s -o /dev/null -w "%{http_code}" "https://${name}.blob.core.windows.net/"
done
 
# GCP bucket check
gsutil ls gs://targetcorp-*

Workflow 2: Personnel Intelligence

Step 1: Employee Enumeration

# theHarvester for email and name harvesting
theHarvester -d targetdomain.com -b all -l 500 -f harvest_results
 
# LinkedIn enumeration (manual + tools)
# Use LinkedIn search operators:
# site:linkedin.com/in "targetcorp" "security engineer"
# site:linkedin.com/in "targetcorp" "system administrator"
 
# CrossLinked for LinkedIn name harvesting
python3 crosslinked.py -f '{first}.{last}@targetdomain.com' "Target Corporation"

Step 2: Email Validation

# Verify email format using Hunter.io API
curl "https://api.hunter.io/v2/domain-search?domain=targetdomain.com&api_key=YOUR_KEY"
 
# SMTP verification (careful - can be logged)
# Use tools like EmailHippo or NeverBounce for passive verification

Step 3: Social Media Profiling

# Sherlock for username enumeration across platforms
python3 sherlock username --timeout 5 --output sherlock_results.txt
 
# Social media searching
# Twitter advanced search: from:username targetcorp
# Instagram: #targetcorp
# GitHub: org:targetcorp

Workflow 3: Credential and Data Leak Discovery

Step 1: Breach Database Search

# Have I Been Pwned API check
curl "https://haveibeenpwned.com/api/v3/breachedaccount/user@targetdomain.com" \
  -H "hibp-api-key: YOUR_KEY"
 
# DeHashed search (requires subscription)
curl "https://api.dehashed.com/search?query=domain:targetdomain.com" \
  -u email:api_key

Step 2: GitHub Secret Scanning

# GitDorker for GitHub dorking
python3 GitDorker.py -tf tokens.txt -d dorks/alldorksv3 -q targetdomain.com
 
# truffleHog for repository scanning
trufflehog github --org=targetcorp --only-verified
 
# Manual GitHub dorking
# Search: "targetdomain.com" password
# Search: "targetdomain.com" api_key
# Search: "targetcorp" filename:.env
# Search: "targetcorp" filename:wp-config.php

Step 3: Google Dorking

# Sensitive files
site:targetdomain.com filetype:pdf
site:targetdomain.com filetype:xlsx
site:targetdomain.com filetype:docx confidential
 
# Configuration files
site:targetdomain.com filetype:xml
site:targetdomain.com filetype:conf
site:targetdomain.com filetype:env
 
# Login pages and admin panels
site:targetdomain.com inurl:admin
site:targetdomain.com inurl:login
site:targetdomain.com intitle:"index of"
 
# Error messages with sensitive info
site:targetdomain.com "error" "sql" "syntax"
site:targetdomain.com "php error" "on line"

Workflow 4: Technology Stack Identification

Step 1: Web Technology Fingerprinting

# Wappalyzer CLI
wappalyzer https://targetdomain.com
 
# WhatWeb for technology identification
whatweb targetdomain.com -v
 
# Nuclei for technology detection
nuclei -u https://targetdomain.com -t technologies/

Step 2: Service and Version Detection

# Nmap service detection (active - requires authorization)
nmap -sV -sC -p- targetdomain.com -oA nmap_results
 
# Shodan host lookup
shodan host <IP_ADDRESS>
 
# Censys host search
censys search "services.tls.certificates.leaf_data.subject.organization:Target Corp"

Step 3: Job Posting Analysis

# Search job boards for technology mentions:
# LinkedIn Jobs: "Target Corporation" AND ("AWS" OR "Azure" OR "GCP")
# Indeed: "Target Corporation" "security" tools
# Glassdoor: Target Corporation technology stack
 
# Look for mentions of:
# - Cloud platforms (AWS, Azure, GCP)
# - Security tools (CrowdStrike, Carbon Black, Splunk)
# - Development languages and frameworks
# - Network equipment vendors (Cisco, Palo Alto, Fortinet)
# - Identity providers (Okta, Azure AD, Ping Identity)

Workflow 5: Physical Intelligence

Step 1: Location Mapping

# Google Maps reconnaissance:
# - Office locations and building layouts
# - Parking areas and entry points
# - Nearby businesses for staging
# - Delivery entrance locations
 
# Google Street View:
# - Access control systems (card readers, turnstiles)
# - Security camera locations
# - Badge/lanyard colors and designs
# - Building signage

Step 2: Document Metadata Extraction

# ExifTool for document metadata
exiftool -r -ext pdf -ext docx -ext xlsx ./downloaded_documents/
 
# FOCA for metadata analysis (Windows)
# Import documents and analyze:
# - Author names and usernames
# - Software versions
# - Internal file paths
# - Printer names and network paths

Workflow 6: OSINT Report Compilation

Report Structure

1. Executive Summary
   - Key findings overview
   - Risk assessment
 
2. Attack Surface Map
   - External infrastructure diagram
   - Domain and subdomain inventory
   - Exposed services and applications
 
3. Personnel Intelligence
   - Key personnel profiles
   - Email address list
   - Organizational chart
 
4. Credential Exposure
   - Breach database findings
   - Leaked secrets and API keys
   - Password pattern analysis
 
5. Technology Stack
   - Identified technologies and versions
   - Known vulnerabilities for detected versions
   - Security tool coverage gaps
 
6. Recommended Attack Vectors
   - Prioritized initial access options
   - Social engineering target list
   - Technical vulnerability targets

Scripts 2

agent.py6.7 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
# For authorized penetration testing and educational environments only.
# Usage against targets without prior mutual consent is illegal.
# It is the end user's responsibility to obey all applicable local, state and federal laws.
"""Agent for performing open source intelligence (OSINT) gathering."""

import json
import argparse
import re

try:
    import requests
except ImportError:
    requests = None

try:
    import dns.resolver
    HAS_DNS = True
except ImportError:
    HAS_DNS = False


def whois_lookup(domain):
    """Perform WHOIS lookup for domain registration info."""
    try:
        import whois
        w = whois.whois(domain)
        return {
            "domain": domain, "registrar": w.registrar,
            "creation_date": str(w.creation_date),
            "expiration_date": str(w.expiration_date),
            "name_servers": w.name_servers if isinstance(w.name_servers, list) else [w.name_servers],
            "status": w.status if isinstance(w.status, list) else [w.status],
            "registrant": w.get("org", w.get("name", "")),
        }
    except ImportError:
        return {"error": "python-whois not installed — pip install python-whois"}
    except Exception as e:
        return {"domain": domain, "error": str(e)}


def dns_enumeration(domain):
    """Enumerate DNS records for intelligence gathering."""
    if not HAS_DNS:
        return {"error": "dnspython not installed — pip install dnspython"}
    records = {}
    for rtype in ["A", "AAAA", "MX", "NS", "TXT", "SOA", "CNAME"]:
        try:
            answers = dns.resolver.resolve(domain, rtype)
            records[rtype] = [str(r) for r in answers]
        except Exception:
            pass
    subs = ["www", "mail", "ftp", "vpn", "api", "dev", "staging", "admin",
            "portal", "test", "owa", "remote", "webmail", "autodiscover", "sip"]
    found = []
    for sub in subs:
        try:
            answers = dns.resolver.resolve(f"{sub}.{domain}", "A")
            found.append({"subdomain": f"{sub}.{domain}", "ips": [str(r) for r in answers]})
        except Exception:
            pass
    return {"domain": domain, "records": records, "subdomains": found}


def email_harvest(domain):
    """Search for email addresses associated with a domain."""
    if not requests:
        return {"error": "requests not installed"}
    emails = set()
    try:
        resp = requests.get(f"https://api.hunter.io/v2/domain-search?domain={domain}&api_key=demo", timeout=15)
        if resp.status_code == 200:
            data = resp.json()
            for email in data.get("data", {}).get("emails", []):
                emails.add(email.get("value", ""))
    except Exception:
        pass
    pattern = re.compile(rf"[a-zA-Z0-9._%+-]+@{re.escape(domain)}", re.I)
    search_urls = [f"https://www.google.com/search?q=%22%40{domain}%22&num=20"]
    return {
        "domain": domain,
        "emails_found": list(emails)[:20],
        "email_count": len(emails),
        "search_pattern": f"*@{domain}",
    }


def technology_fingerprint(url):
    """Fingerprint web technologies from HTTP response headers."""
    if not requests:
        return {"error": "requests not installed"}
    try:
        resp = requests.get(url, timeout=10, allow_redirects=True)
        headers = dict(resp.headers)
        technologies = []
        server = headers.get("Server", "")
        if server:
            technologies.append({"category": "Web Server", "value": server})
        powered = headers.get("X-Powered-By", "")
        if powered:
            technologies.append({"category": "Framework", "value": powered})
        if "wp-content" in resp.text or "wordpress" in resp.text.lower():
            technologies.append({"category": "CMS", "value": "WordPress"})
        if "drupal" in resp.text.lower():
            technologies.append({"category": "CMS", "value": "Drupal"})
        security_headers = {}
        for h in ["Strict-Transport-Security", "Content-Security-Policy", "X-Frame-Options",
                   "X-Content-Type-Options", "X-XSS-Protection", "Referrer-Policy"]:
            security_headers[h] = headers.get(h, "MISSING")
        return {
            "url": url, "status": resp.status_code,
            "technologies": technologies,
            "security_headers": security_headers,
            "cookies": [{"name": c.name, "secure": c.secure, "httponly": "HttpOnly" in str(c._rest)}
                        for c in resp.cookies][:10],
        }
    except Exception as e:
        return {"url": url, "error": str(e)}


def social_media_search(target_name):
    """Generate social media profile URLs for a target name."""
    username = target_name.lower().replace(" ", "")
    platforms = {
        "linkedin": f"https://www.linkedin.com/in/{username}",
        "twitter": f"https://twitter.com/{username}",
        "github": f"https://github.com/{username}",
        "facebook": f"https://www.facebook.com/{username}",
        "instagram": f"https://www.instagram.com/{username}",
    }
    results = []
    for platform, url in platforms.items():
        if requests:
            try:
                resp = requests.head(url, timeout=5, allow_redirects=True)
                exists = resp.status_code == 200
                results.append({"platform": platform, "url": url, "exists": exists})
            except Exception:
                results.append({"platform": platform, "url": url, "exists": "unknown"})
        else:
            results.append({"platform": platform, "url": url, "exists": "unchecked"})
    return {"target": target_name, "profiles": results}


def main():
    parser = argparse.ArgumentParser(description="OSINT Gathering Agent")
    sub = parser.add_subparsers(dest="command")
    w = sub.add_parser("whois", help="WHOIS lookup")
    w.add_argument("--domain", required=True)
    d = sub.add_parser("dns", help="DNS enumeration")
    d.add_argument("--domain", required=True)
    e = sub.add_parser("email", help="Email harvesting")
    e.add_argument("--domain", required=True)
    t = sub.add_parser("tech", help="Technology fingerprinting")
    t.add_argument("--url", required=True)
    s = sub.add_parser("social", help="Social media search")
    s.add_argument("--name", required=True)
    args = parser.parse_args()
    if args.command == "whois":
        result = whois_lookup(args.domain)
    elif args.command == "dns":
        result = dns_enumeration(args.domain)
    elif args.command == "email":
        result = email_harvest(args.domain)
    elif args.command == "tech":
        result = technology_fingerprint(args.url)
    elif args.command == "social":
        result = social_media_search(args.name)
    else:
        parser.print_help()
        return
    print(json.dumps(result, indent=2, default=str))


if __name__ == "__main__":
    main()
process.py20.9 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""
OSINT Gathering Automation Tool

Performs automated open source intelligence collection including:
- Subdomain enumeration via Certificate Transparency logs
- DNS record collection
- WHOIS information gathering
- Technology fingerprinting
- Google dorking query generation
- Email pattern discovery
- Shodan/Censys integration

Usage:
    python process.py --domain targetdomain.com --output ./osint_report
    python process.py --domain targetdomain.com --modules all
    python process.py --domain targetdomain.com --modules dns,subdomains,emails

Requirements:
    pip install requests dnspython whois beautifulsoup4 rich
"""

import argparse
import json
import re
import socket
import sys
from datetime import datetime
from pathlib import Path
from typing import Any
from urllib.parse import urlparse

try:
    import dns.resolver
    import requests
    from bs4 import BeautifulSoup
    from rich.console import Console
    from rich.table import Table
    from rich.panel import Panel
    from rich.progress import Progress, SpinnerColumn, TextColumn
except ImportError:
    print("[!] Missing dependencies. Install with:")
    print("    pip install requests dnspython beautifulsoup4 rich")
    sys.exit(1)

console = Console()
SESSION = requests.Session()
SESSION.headers.update(
    {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
    }
)


def resolve_dns_records(domain: str) -> dict:
    """Collect DNS records for a domain."""
    records = {}
    record_types = ["A", "AAAA", "MX", "NS", "TXT", "SOA", "CNAME", "SRV"]

    for rtype in record_types:
        try:
            answers = dns.resolver.resolve(domain, rtype)
            records[rtype] = [str(rdata) for rdata in answers]
        except (dns.resolver.NoAnswer, dns.resolver.NXDOMAIN, dns.resolver.NoNameservers):
            pass
        except Exception:
            pass

    return records


def enumerate_subdomains_ct(domain: str) -> list[str]:
    """Enumerate subdomains using Certificate Transparency logs via crt.sh."""
    subdomains = set()

    try:
        url = f"https://crt.sh/?q=%.{domain}&output=json"
        response = SESSION.get(url, timeout=30)
        if response.status_code == 200:
            data = response.json()
            for entry in data:
                name_value = entry.get("name_value", "")
                for name in name_value.split("\n"):
                    name = name.strip().lower()
                    if name.endswith(f".{domain}") or name == domain:
                        # Skip wildcard entries
                        if not name.startswith("*"):
                            subdomains.add(name)
    except Exception as e:
        console.print(f"[yellow][!] crt.sh query failed: {e}[/yellow]")

    # Also try common subdomain prefixes
    common_prefixes = [
        "www", "mail", "ftp", "smtp", "pop", "imap", "webmail",
        "vpn", "remote", "portal", "admin", "dev", "staging",
        "test", "api", "app", "blog", "shop", "store", "cdn",
        "ns1", "ns2", "dns", "mx", "exchange", "owa", "autodiscover",
        "sso", "login", "auth", "git", "gitlab", "jenkins",
        "jira", "confluence", "wiki", "docs", "support", "help",
    ]

    for prefix in common_prefixes:
        subdomain = f"{prefix}.{domain}"
        try:
            dns.resolver.resolve(subdomain, "A")
            subdomains.add(subdomain)
        except Exception:
            pass

    return sorted(subdomains)


def perform_whois_lookup(domain: str) -> dict:
    """Perform WHOIS lookup for a domain."""
    try:
        import whois as python_whois
        w = python_whois.whois(domain)
        result = {
            "domain_name": str(w.domain_name) if w.domain_name else "N/A",
            "registrar": str(w.registrar) if w.registrar else "N/A",
            "creation_date": str(w.creation_date) if w.creation_date else "N/A",
            "expiration_date": str(w.expiration_date) if w.expiration_date else "N/A",
            "name_servers": w.name_servers if w.name_servers else [],
            "registrant_org": str(w.org) if w.org else "N/A",
            "registrant_country": str(w.country) if w.country else "N/A",
            "emails": w.emails if w.emails else [],
        }
        return result
    except Exception as e:
        console.print(f"[yellow][!] WHOIS lookup failed: {e}[/yellow]")
        return {}


def discover_email_format(domain: str) -> dict:
    """Attempt to discover email format patterns using Hunter.io free tier."""
    result = {
        "domain": domain,
        "format_guess": [],
        "discovered_emails": [],
    }

    # Common email format patterns
    common_formats = [
        "{first}.{last}@" + domain,
        "{first}{last}@" + domain,
        "{f}{last}@" + domain,
        "{first}_{last}@" + domain,
        "{first}@" + domain,
        "{last}@" + domain,
    ]
    result["format_guess"] = common_formats

    # Try to discover emails from web pages
    try:
        response = SESSION.get(f"https://{domain}", timeout=10)
        if response.status_code == 200:
            # Extract emails from page content
            email_pattern = re.compile(
                rf"[a-zA-Z0-9._%+-]+@{re.escape(domain)}",
                re.IGNORECASE
            )
            emails = email_pattern.findall(response.text)
            result["discovered_emails"] = list(set(emails))
    except Exception:
        pass

    # Search for emails on common pages
    common_pages = ["/contact", "/about", "/team", "/about-us", "/contact-us"]
    for page in common_pages:
        try:
            response = SESSION.get(f"https://{domain}{page}", timeout=10)
            if response.status_code == 200:
                email_pattern = re.compile(
                    rf"[a-zA-Z0-9._%+-]+@{re.escape(domain)}",
                    re.IGNORECASE
                )
                emails = email_pattern.findall(response.text)
                result["discovered_emails"].extend(emails)
        except Exception:
            pass

    result["discovered_emails"] = list(set(result["discovered_emails"]))
    return result


def fingerprint_web_technologies(domain: str) -> dict:
    """Identify web technologies using HTTP headers and response analysis."""
    tech = {
        "server": None,
        "powered_by": None,
        "frameworks": [],
        "security_headers": {},
        "cookies": [],
        "cdn": None,
        "cms": None,
    }

    try:
        response = SESSION.get(f"https://{domain}", timeout=10, allow_redirects=True)
        headers = response.headers

        # Server identification
        tech["server"] = headers.get("Server", "Not disclosed")
        tech["powered_by"] = headers.get("X-Powered-By", "Not disclosed")

        # Security headers
        security_headers = [
            "Strict-Transport-Security",
            "Content-Security-Policy",
            "X-Frame-Options",
            "X-Content-Type-Options",
            "X-XSS-Protection",
            "Referrer-Policy",
            "Permissions-Policy",
            "Cross-Origin-Opener-Policy",
            "Cross-Origin-Resource-Policy",
        ]

        for header in security_headers:
            value = headers.get(header)
            tech["security_headers"][header] = value if value else "MISSING"

        # Cookie analysis
        for cookie in response.cookies:
            tech["cookies"].append(
                {
                    "name": cookie.name,
                    "secure": cookie.secure,
                    "httponly": "httponly" in cookie._rest,
                    "domain": cookie.domain,
                }
            )

        # CDN detection
        cdn_indicators = {
            "cloudflare": ["cf-ray", "cf-cache-status"],
            "akamai": ["x-akamai-transformed"],
            "cloudfront": ["x-amz-cf-id", "x-amz-cf-pop"],
            "fastly": ["x-served-by", "x-fastly-request-id"],
            "incapsula": ["x-iinfo"],
        }

        for cdn, indicators in cdn_indicators.items():
            for indicator in indicators:
                if indicator in [h.lower() for h in headers]:
                    tech["cdn"] = cdn
                    break

        # CMS detection from HTML
        html = response.text.lower()
        if "wp-content" in html or "wordpress" in html:
            tech["cms"] = "WordPress"
        elif "drupal" in html:
            tech["cms"] = "Drupal"
        elif "joomla" in html:
            tech["cms"] = "Joomla"
        elif "shopify" in html:
            tech["cms"] = "Shopify"

        # Framework detection
        if "react" in html or "reactdom" in html:
            tech["frameworks"].append("React")
        if "angular" in html or "ng-" in html:
            tech["frameworks"].append("Angular")
        if "vue" in html or "vuejs" in html:
            tech["frameworks"].append("Vue.js")
        if "jquery" in html:
            tech["frameworks"].append("jQuery")
        if "bootstrap" in html:
            tech["frameworks"].append("Bootstrap")

    except Exception as e:
        console.print(f"[yellow][!] Web fingerprinting failed: {e}[/yellow]")

    return tech


def generate_google_dorks(domain: str) -> list[str]:
    """Generate Google dorking queries for the target domain."""
    dorks = [
        # Sensitive files
        f'site:{domain} filetype:pdf',
        f'site:{domain} filetype:xlsx',
        f'site:{domain} filetype:docx',
        f'site:{domain} filetype:csv',
        f'site:{domain} filetype:sql',
        f'site:{domain} filetype:log',
        f'site:{domain} filetype:bak',
        f'site:{domain} filetype:conf',
        f'site:{domain} filetype:env',
        f'site:{domain} filetype:xml',
        # Configuration and credentials
        f'site:{domain} inurl:admin',
        f'site:{domain} inurl:login',
        f'site:{domain} inurl:wp-admin',
        f'site:{domain} inurl:wp-login',
        f'site:{domain} intitle:"index of"',
        f'site:{domain} intitle:"dashboard"',
        f'site:{domain} inurl:config',
        f'site:{domain} inurl:setup',
        # Error messages
        f'site:{domain} "error" "sql syntax"',
        f'site:{domain} "php error" "on line"',
        f'site:{domain} "ORA-" "error"',
        f'site:{domain} "mysql" "error"',
        f'site:{domain} "stack trace" "at"',
        # Sensitive information
        f'site:{domain} "confidential"',
        f'site:{domain} "internal use only"',
        f'site:{domain} "not for distribution"',
        f'site:{domain} "password" filetype:txt',
        f'site:{domain} "api_key" OR "apikey" OR "api-key"',
        # Infrastructure
        f'site:{domain} inurl:vpn',
        f'site:{domain} inurl:remote',
        f'site:{domain} inurl:portal',
        f'site:{domain} inurl:citrix',
        f'site:{domain} inurl:owa',
        # GitHub leaks
        f'"{domain}" password site:github.com',
        f'"{domain}" api_key site:github.com',
        f'"{domain}" secret site:github.com',
        f'"{domain}" token site:github.com',
        f'"{domain}" site:pastebin.com',
        # Cloud storage
        f'site:s3.amazonaws.com "{domain.split(".")[0]}"',
        f'site:blob.core.windows.net "{domain.split(".")[0]}"',
        f'site:storage.googleapis.com "{domain.split(".")[0]}"',
    ]
    return dorks


def check_security_txt(domain: str) -> dict | None:
    """Check for security.txt file per RFC 9116."""
    urls = [
        f"https://{domain}/.well-known/security.txt",
        f"https://{domain}/security.txt",
    ]

    for url in urls:
        try:
            response = SESSION.get(url, timeout=10)
            if response.status_code == 200 and "contact" in response.text.lower():
                return {
                    "url": url,
                    "content": response.text[:2000],
                }
        except Exception:
            pass

    return None


def check_robots_txt(domain: str) -> dict | None:
    """Check robots.txt for interesting paths."""
    try:
        response = SESSION.get(f"https://{domain}/robots.txt", timeout=10)
        if response.status_code == 200:
            disallowed = []
            for line in response.text.split("\n"):
                if line.strip().lower().startswith("disallow:"):
                    path = line.split(":", 1)[1].strip()
                    if path and path != "/":
                        disallowed.append(path)
            return {
                "content": response.text[:2000],
                "disallowed_paths": disallowed,
            }
    except Exception:
        pass
    return None


def generate_report(domain: str, results: dict, output_dir: Path):
    """Generate comprehensive OSINT report."""
    report = f"""# OSINT Report: {domain}
## Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}

---

## 1. Domain Information

### WHOIS Data
"""

    whois_data = results.get("whois", {})
    if whois_data:
        for key, value in whois_data.items():
            report += f"- **{key}**: {value}\n"
    else:
        report += "WHOIS data not available.\n"

    report += "\n### DNS Records\n\n"
    dns_data = results.get("dns", {})
    if dns_data:
        for rtype, records in dns_data.items():
            report += f"#### {rtype} Records\n"
            for record in records:
                report += f"- `{record}`\n"
            report += "\n"

    report += "## 2. Subdomain Enumeration\n\n"
    subdomains = results.get("subdomains", [])
    report += f"**Total subdomains discovered:** {len(subdomains)}\n\n"
    for sub in subdomains:
        try:
            ip = socket.gethostbyname(sub)
            report += f"- `{sub}` -> `{ip}`\n"
        except Exception:
            report += f"- `{sub}` -> [unresolvable]\n"

    report += "\n## 3. Web Technology Fingerprint\n\n"
    tech = results.get("technology", {})
    if tech:
        report += f"- **Server:** {tech.get('server', 'N/A')}\n"
        report += f"- **Powered By:** {tech.get('powered_by', 'N/A')}\n"
        report += f"- **CMS:** {tech.get('cms', 'None detected')}\n"
        report += f"- **CDN:** {tech.get('cdn', 'None detected')}\n"
        report += f"- **Frameworks:** {', '.join(tech.get('frameworks', [])) or 'None detected'}\n\n"

        report += "### Security Headers\n\n"
        report += "| Header | Status |\n|--------|--------|\n"
        for header, value in tech.get("security_headers", {}).items():
            status = "MISSING" if value == "MISSING" else "Present"
            report += f"| {header} | {status} |\n"

    report += "\n## 4. Email Intelligence\n\n"
    email_data = results.get("emails", {})
    if email_data:
        report += "### Discovered Emails\n"
        for email in email_data.get("discovered_emails", []):
            report += f"- `{email}`\n"
        report += "\n### Likely Email Formats\n"
        for fmt in email_data.get("format_guess", []):
            report += f"- `{fmt}`\n"

    report += "\n## 5. Google Dorking Queries\n\n"
    dorks = results.get("dorks", [])
    for dork in dorks[:20]:
        report += f"- `{dork}`\n"

    report += "\n## 6. Additional Findings\n\n"
    security_txt = results.get("security_txt")
    if security_txt:
        report += f"### security.txt Found\n- URL: {security_txt['url']}\n\n"

    robots_txt = results.get("robots_txt")
    if robots_txt:
        report += "### Interesting robots.txt Paths\n"
        for path in robots_txt.get("disallowed_paths", []):
            report += f"- `{path}`\n"

    report += f"""
---

## 7. Recommendations for Attack Planning

### Priority Initial Access Vectors
1. Review discovered subdomains for vulnerable web applications
2. Validate credential leaks against target systems
3. Use discovered email format for spearphishing campaign
4. Investigate missing security headers for potential exploitation
5. Check disallowed paths from robots.txt for sensitive content

### Social Engineering Targets
- Use discovered personnel for targeted phishing
- Leverage technology stack knowledge for pretexting
- Utilize physical location data for physical access testing

---

*Report generated by OSINT Automation Tool*
*Classification: CONFIDENTIAL - Red Team Use Only*
"""

    report_path = output_dir / f"osint_report_{domain}.md"
    with open(report_path, "w") as f:
        f.write(report)

    console.print(f"[green][+] Report saved to: {report_path}[/green]")

    # Save raw data as JSON
    json_path = output_dir / f"osint_data_{domain}.json"
    serializable_results = {}
    for key, value in results.items():
        try:
            json.dumps(value)
            serializable_results[key] = value
        except (TypeError, ValueError):
            serializable_results[key] = str(value)

    with open(json_path, "w") as f:
        json.dump(serializable_results, f, indent=2, default=str)

    console.print(f"[green][+] Raw data saved to: {json_path}[/green]")


def main():
    parser = argparse.ArgumentParser(
        description="OSINT Gathering Automation Tool"
    )
    parser.add_argument("--domain", required=True, help="Target domain")
    parser.add_argument(
        "--output", default="./osint_output", help="Output directory"
    )
    parser.add_argument(
        "--modules",
        default="all",
        help="Comma-separated modules: dns,subdomains,whois,emails,tech,dorks,all",
    )

    args = parser.parse_args()
    output_dir = Path(args.output)
    output_dir.mkdir(parents=True, exist_ok=True)

    modules = args.modules.split(",") if args.modules != "all" else [
        "dns", "subdomains", "whois", "emails", "tech", "dorks", "security_txt", "robots_txt"
    ]

    console.print(
        Panel(
            f"[bold red]OSINT Gathering Tool[/bold red]\n"
            f"Target: {args.domain}\n"
            f"Modules: {', '.join(modules)}\n"
            f"Output: {args.output}",
            title="Configuration",
        )
    )

    results = {}

    with Progress(
        SpinnerColumn(),
        TextColumn("[progress.description]{task.description}"),
        console=console,
    ) as progress:

        if "dns" in modules:
            task = progress.add_task("[cyan]Collecting DNS records...", total=None)
            results["dns"] = resolve_dns_records(args.domain)
            progress.update(task, completed=True, description="[green]DNS records collected")

        if "subdomains" in modules:
            task = progress.add_task("[cyan]Enumerating subdomains...", total=None)
            results["subdomains"] = enumerate_subdomains_ct(args.domain)
            progress.update(
                task,
                completed=True,
                description=f"[green]Found {len(results['subdomains'])} subdomains",
            )

        if "whois" in modules:
            task = progress.add_task("[cyan]Performing WHOIS lookup...", total=None)
            results["whois"] = perform_whois_lookup(args.domain)
            progress.update(task, completed=True, description="[green]WHOIS data collected")

        if "emails" in modules:
            task = progress.add_task("[cyan]Discovering email formats...", total=None)
            results["emails"] = discover_email_format(args.domain)
            progress.update(task, completed=True, description="[green]Email discovery complete")

        if "tech" in modules:
            task = progress.add_task("[cyan]Fingerprinting technologies...", total=None)
            results["technology"] = fingerprint_web_technologies(args.domain)
            progress.update(task, completed=True, description="[green]Technology fingerprint complete")

        if "dorks" in modules:
            task = progress.add_task("[cyan]Generating Google dorks...", total=None)
            results["dorks"] = generate_google_dorks(args.domain)
            progress.update(
                task,
                completed=True,
                description=f"[green]Generated {len(results['dorks'])} dork queries",
            )

        if "security_txt" in modules:
            task = progress.add_task("[cyan]Checking security.txt...", total=None)
            results["security_txt"] = check_security_txt(args.domain)
            progress.update(task, completed=True, description="[green]security.txt check complete")

        if "robots_txt" in modules:
            task = progress.add_task("[cyan]Checking robots.txt...", total=None)
            results["robots_txt"] = check_robots_txt(args.domain)
            progress.update(task, completed=True, description="[green]robots.txt check complete")

    generate_report(args.domain, results, output_dir)

    # Display summary table
    table = Table(title=f"OSINT Summary: {args.domain}")
    table.add_column("Category", style="cyan")
    table.add_column("Count/Status", style="green")

    table.add_row("DNS Record Types", str(len(results.get("dns", {}))))
    table.add_row("Subdomains Found", str(len(results.get("subdomains", []))))
    table.add_row("Emails Discovered", str(len(results.get("emails", {}).get("discovered_emails", []))))
    table.add_row("Google Dorks Generated", str(len(results.get("dorks", []))))
    table.add_row("security.txt", "Found" if results.get("security_txt") else "Not Found")
    table.add_row("robots.txt", "Found" if results.get("robots_txt") else "Not Found")

    console.print(table)


if __name__ == "__main__":
    main()

Assets 1

template.mdtext/markdown · 4.0 KB
Keep exploring