npx skills add mukul975/Anthropic-Cybersecurity-SkillsMITRE ATT&CK
Legal Notice: This skill is for authorized security testing and educational purposes only. Unauthorized use against systems you do not own or have written permission to test is illegal and may violate computer fraud laws.
Overview
Open Source Intelligence (OSINT) gathering is the first active phase of a red team engagement, where operators collect publicly available information about the target organization to identify attack surfaces, potential targets for social engineering, technology stacks, and credential exposures. Effective OSINT directly shapes initial access strategies and reduces operational risk.
When to Use
- When conducting security assessments that involve performing open source intelligence gathering
- When following incident response procedures for related security events
- When performing scheduled security testing or auditing activities
- When validating security controls through hands-on testing
Prerequisites
- Familiarity with red teaming concepts and tools
- Access to a test or lab environment for safe execution
- Python 3.8+ with required dependencies installed
- Appropriate authorization for any testing activities
Objectives
- Enumerate the target organization's external attack surface (domains, IPs, cloud assets)
- Identify employees and their roles for social engineering targeting
- Discover leaked credentials, API keys, and sensitive documents
- Map the organization's technology stack and vendors
- Identify physical locations, office layouts, and access control details
- Build target profiles for spearphishing campaign development
Core Concepts
OSINT Categories
| Category | Sources | Value |
|---|---|---|
| Domain Intelligence | DNS records, WHOIS, CT logs, subdomain enumeration | Network attack surface |
| Personnel Intelligence | LinkedIn, social media, conference talks, publications | Social engineering targets |
| Credential Intelligence | Breach databases, paste sites, GitHub leaks | Valid credential discovery |
| Technology Intelligence | Job postings, Wappalyzer, Shodan, Censys | Vulnerability identification |
| Physical Intelligence | Google Maps, social media photos, Glassdoor | Physical access planning |
| Document Intelligence | SEC filings, public documents, metadata extraction | Organizational structure |
MITRE ATT&CK Mapping
- T1595.001 - Active Scanning: Scanning IP Blocks
- T1595.002 - Active Scanning: Vulnerability Scanning
- T1592 - Gather Victim Host Information
- T1589 - Gather Victim Identity Information
- T1590 - Gather Victim Network Information
- T1591 - Gather Victim Org Information
- T1593 - Search Open Websites/Domains
- T1594 - Search Victim-Owned Websites
- T1596 - Search Open Technical Databases
Workflow
Phase 1: Domain and Network Reconnaissance
- Perform WHOIS lookups for target domains
- Enumerate subdomains using Certificate Transparency logs, DNS brute-force, and web scraping
- Identify IP ranges and ASN ownership
- Scan for exposed services using Shodan/Censys
- Check for cloud storage buckets (S3, Azure Blob, GCS)
- Map CDN and hosting providers
Phase 2: Personnel and Social Intelligence
- Enumerate employees via LinkedIn, company website, and conference speaker lists
- Identify email naming conventions
- Discover personal social media accounts of key targets
- Map organizational hierarchy and reporting structure
- Identify recently hired IT/security personnel
- Check for conference presentations and technical publications
Phase 3: Credential and Data Leak Discovery
- Search breach databases (Have I Been Pwned, DeHashed)
- Check paste sites (Pastebin, GitHub Gists)
- Search GitHub/GitLab for leaked secrets and API keys
- Look for exposed configuration files and backups
- Check for leaked internal documents via Google dorking
Phase 4: Technology Stack Identification
- Analyze job postings for technology mentions
- Use Wappalyzer/BuiltWith for web technology fingerprinting
- Check for exposed admin panels and development environments
- Identify VPN and remote access technologies
- Map cloud services and SaaS applications
Tools and Resources
| Tool | Purpose | Type |
|---|---|---|
| Amass | Subdomain enumeration and network mapping | Open Source |
| Subfinder | Passive subdomain discovery | Open Source |
| theHarvester | Email, subdomain, and name harvesting | Open Source |
| Maltego | Visual link analysis and data correlation | Commercial |
| SpiderFoot | Automated OSINT collection | Open Source |
| Shodan | Internet-connected device search | Commercial |
| Censys | Internet asset discovery | Commercial |
| Recon-ng | Web reconnaissance framework | Open Source |
| GitDorker | GitHub secret scanning | Open Source |
| Photon | Web crawler for OSINT | Open Source |
Validation Criteria
- Complete list of target domains and subdomains
- Employee list with roles and email addresses
- Technology stack identified
- Credential leak assessment completed
- Attack surface map documented
- OSINT report compiled for engagement team
References and resources
Everything below is rendered for inspection. Script files are read-only and never run.
References 3
api-reference.md1.4 KB
API Reference — Performing Open Source Intelligence Gathering
Libraries Used
- requests: HTTP requests for tech fingerprinting and social media checks
- dns.resolver (dnspython): DNS record enumeration and subdomain discovery
- python-whois: Domain WHOIS registration data
- re: Email pattern extraction
- socket: Network connectivity
CLI Interface
python agent.py whois --domain example.com
python agent.py dns --domain example.com
python agent.py email --domain example.com
python agent.py tech --url https://example.com
python agent.py social --name "John Doe"Core Functions
whois_lookup(domain) — Domain registration data
Returns registrar, creation/expiration dates, name servers, registrant info.
dns_enumeration(domain) — DNS record and subdomain discovery
Queries 7 record types. Tests 15 common subdomain prefixes.
email_harvest(domain) — Email address discovery
Uses Hunter.io API and regex pattern matching.
technology_fingerprint(url) — Web technology identification
Detects: web server, framework, CMS. Audits 6 security headers.
social_media_search(target_name) — Profile enumeration
Checks: LinkedIn, Twitter/X, GitHub, Facebook, Instagram.
Security Headers Checked
Strict-Transport-Security, Content-Security-Policy, X-Frame-Options, X-Content-Type-Options, X-XSS-Protection, Referrer-Policy
Dependencies
pip install requests dnspython python-whoisstandards.md3.1 KB
Standards and Framework References
MITRE ATT&CK - Reconnaissance (TA0043)
| Technique ID | Name | Description |
|---|---|---|
| T1595.001 | Active Scanning: Scanning IP Blocks | Scanning target IP ranges for active hosts |
| T1595.002 | Active Scanning: Vulnerability Scanning | Scanning for vulnerabilities on discovered hosts |
| T1592.001 | Gather Victim Host Information: Hardware | Identifying target hardware configurations |
| T1592.002 | Gather Victim Host Information: Software | Identifying installed software and versions |
| T1592.004 | Gather Victim Host Information: Client Configurations | Discovering client-side configurations |
| T1589.001 | Gather Victim Identity Information: Credentials | Searching for exposed credentials |
| T1589.002 | Gather Victim Identity Information: Email Addresses | Harvesting email addresses |
| T1589.003 | Gather Victim Identity Information: Employee Names | Collecting employee information |
| T1590.001 | Gather Victim Network Information: Domain Properties | DNS and domain enumeration |
| T1590.002 | Gather Victim Network Information: DNS | DNS record collection |
| T1590.004 | Gather Victim Network Information: Network Topology | Mapping network architecture |
| T1590.005 | Gather Victim Network Information: IP Addresses | Identifying target IP addresses |
| T1591.001 | Gather Victim Org Information: Determine Physical Locations | Physical location mapping |
| T1591.002 | Gather Victim Org Information: Business Relationships | Identifying vendors and partners |
| T1591.004 | Gather Victim Org Information: Identify Roles | Mapping organizational roles |
| T1593.001 | Search Open Websites/Domains: Social Media | Social media intelligence |
| T1593.002 | Search Open Websites/Domains: Search Engines | Google dorking and search engine recon |
| T1594 | Search Victim-Owned Websites | Analyzing target websites |
| T1596.001 | Search Open Technical Databases: DNS/Passive DNS | Passive DNS intelligence |
| T1596.005 | Search Open Technical Databases: Scan Databases | Shodan, Censys, ZoomEye queries |
| T1597.001 | Search Closed Sources: Threat Intel Vendors | Threat intelligence platform queries |
PTES - Intelligence Gathering
Level 1: Passive Information Gathering
- WHOIS lookups
- DNS enumeration
- Search engine queries
- Social media analysis
- Public records review
Level 2: Semi-Passive Information Gathering
- Website analysis and spidering
- Metadata extraction from documents
- Job posting analysis
- Technology stack identification
Level 3: Active Information Gathering
- Port scanning
- Service enumeration
- Web application fingerprinting
- Active subdomain brute-forcing
OSSTMM - Information Security Testing
Section 5: Human Security Testing
- Social engineering reconnaissance
- Personnel profiling
- Communication channel mapping
Section 6: Physical Security Testing
- Location reconnaissance
- Access control assessment
- Surveillance analysis
NIST SP 800-115 Section 3: Review Techniques
- Documentation review
- Log review
- Ruleset review
- System configuration review
- Network sniffing (passive)
workflows.md6.6 KB
OSINT Gathering Workflows
Workflow 1: Domain and Infrastructure Reconnaissance
Step 1: Passive DNS and WHOIS
# WHOIS lookup
whois targetdomain.com
# DNS record enumeration
dig targetdomain.com ANY
dig targetdomain.com MX
dig targetdomain.com TXT
dig targetdomain.com NS
# Reverse DNS
dig -x <IP_ADDRESS>
# Zone transfer attempt
dig axfr @ns1.targetdomain.com targetdomain.comStep 2: Subdomain Enumeration
# Using Subfinder for passive enumeration
subfinder -d targetdomain.com -o subdomains.txt
# Using Amass for comprehensive enumeration
amass enum -passive -d targetdomain.com -o amass_results.txt
# Certificate Transparency log search
curl -s "https://crt.sh/?q=%.targetdomain.com&output=json" | jq -r '.[].name_value' | sort -u
# Using httpx to probe discovered subdomains
cat subdomains.txt | httpx -status-code -title -tech-detect -o live_subdomains.txtStep 3: IP Range and ASN Discovery
# ASN lookup
whois -h whois.radb.net -- '-i origin AS12345'
# BGP prefix lookup via Hurricane Electric
curl -s "https://bgp.he.net/AS12345#_prefixes"
# Shodan search for organization
shodan search "org:Target Corporation" --fields ip_str,port,productStep 4: Cloud Asset Discovery
# AWS S3 bucket enumeration
python3 cloud_enum.py -k targetcorp -l cloud_results.txt
# Azure blob storage check
for name in targetcorp targetcorp-dev targetcorp-backup; do
curl -s -o /dev/null -w "%{http_code}" "https://${name}.blob.core.windows.net/"
done
# GCP bucket check
gsutil ls gs://targetcorp-*Workflow 2: Personnel Intelligence
Step 1: Employee Enumeration
# theHarvester for email and name harvesting
theHarvester -d targetdomain.com -b all -l 500 -f harvest_results
# LinkedIn enumeration (manual + tools)
# Use LinkedIn search operators:
# site:linkedin.com/in "targetcorp" "security engineer"
# site:linkedin.com/in "targetcorp" "system administrator"
# CrossLinked for LinkedIn name harvesting
python3 crosslinked.py -f '{first}.{last}@targetdomain.com' "Target Corporation"Step 2: Email Validation
# Verify email format using Hunter.io API
curl "https://api.hunter.io/v2/domain-search?domain=targetdomain.com&api_key=YOUR_KEY"
# SMTP verification (careful - can be logged)
# Use tools like EmailHippo or NeverBounce for passive verificationStep 3: Social Media Profiling
# Sherlock for username enumeration across platforms
python3 sherlock username --timeout 5 --output sherlock_results.txt
# Social media searching
# Twitter advanced search: from:username targetcorp
# Instagram: #targetcorp
# GitHub: org:targetcorpWorkflow 3: Credential and Data Leak Discovery
Step 1: Breach Database Search
# Have I Been Pwned API check
curl "https://haveibeenpwned.com/api/v3/breachedaccount/user@targetdomain.com" \
-H "hibp-api-key: YOUR_KEY"
# DeHashed search (requires subscription)
curl "https://api.dehashed.com/search?query=domain:targetdomain.com" \
-u email:api_keyStep 2: GitHub Secret Scanning
# GitDorker for GitHub dorking
python3 GitDorker.py -tf tokens.txt -d dorks/alldorksv3 -q targetdomain.com
# truffleHog for repository scanning
trufflehog github --org=targetcorp --only-verified
# Manual GitHub dorking
# Search: "targetdomain.com" password
# Search: "targetdomain.com" api_key
# Search: "targetcorp" filename:.env
# Search: "targetcorp" filename:wp-config.phpStep 3: Google Dorking
# Sensitive files
site:targetdomain.com filetype:pdf
site:targetdomain.com filetype:xlsx
site:targetdomain.com filetype:docx confidential
# Configuration files
site:targetdomain.com filetype:xml
site:targetdomain.com filetype:conf
site:targetdomain.com filetype:env
# Login pages and admin panels
site:targetdomain.com inurl:admin
site:targetdomain.com inurl:login
site:targetdomain.com intitle:"index of"
# Error messages with sensitive info
site:targetdomain.com "error" "sql" "syntax"
site:targetdomain.com "php error" "on line"Workflow 4: Technology Stack Identification
Step 1: Web Technology Fingerprinting
# Wappalyzer CLI
wappalyzer https://targetdomain.com
# WhatWeb for technology identification
whatweb targetdomain.com -v
# Nuclei for technology detection
nuclei -u https://targetdomain.com -t technologies/Step 2: Service and Version Detection
# Nmap service detection (active - requires authorization)
nmap -sV -sC -p- targetdomain.com -oA nmap_results
# Shodan host lookup
shodan host <IP_ADDRESS>
# Censys host search
censys search "services.tls.certificates.leaf_data.subject.organization:Target Corp"Step 3: Job Posting Analysis
# Search job boards for technology mentions:
# LinkedIn Jobs: "Target Corporation" AND ("AWS" OR "Azure" OR "GCP")
# Indeed: "Target Corporation" "security" tools
# Glassdoor: Target Corporation technology stack
# Look for mentions of:
# - Cloud platforms (AWS, Azure, GCP)
# - Security tools (CrowdStrike, Carbon Black, Splunk)
# - Development languages and frameworks
# - Network equipment vendors (Cisco, Palo Alto, Fortinet)
# - Identity providers (Okta, Azure AD, Ping Identity)Workflow 5: Physical Intelligence
Step 1: Location Mapping
# Google Maps reconnaissance:
# - Office locations and building layouts
# - Parking areas and entry points
# - Nearby businesses for staging
# - Delivery entrance locations
# Google Street View:
# - Access control systems (card readers, turnstiles)
# - Security camera locations
# - Badge/lanyard colors and designs
# - Building signageStep 2: Document Metadata Extraction
# ExifTool for document metadata
exiftool -r -ext pdf -ext docx -ext xlsx ./downloaded_documents/
# FOCA for metadata analysis (Windows)
# Import documents and analyze:
# - Author names and usernames
# - Software versions
# - Internal file paths
# - Printer names and network pathsWorkflow 6: OSINT Report Compilation
Report Structure
1. Executive Summary
- Key findings overview
- Risk assessment
2. Attack Surface Map
- External infrastructure diagram
- Domain and subdomain inventory
- Exposed services and applications
3. Personnel Intelligence
- Key personnel profiles
- Email address list
- Organizational chart
4. Credential Exposure
- Breach database findings
- Leaked secrets and API keys
- Password pattern analysis
5. Technology Stack
- Identified technologies and versions
- Known vulnerabilities for detected versions
- Security tool coverage gaps
6. Recommended Attack Vectors
- Prioritized initial access options
- Social engineering target list
- Technical vulnerability targetsScripts 2
agent.py6.7 KB
#!/usr/bin/env python3
# For authorized penetration testing and educational environments only.
# Usage against targets without prior mutual consent is illegal.
# It is the end user's responsibility to obey all applicable local, state and federal laws.
"""Agent for performing open source intelligence (OSINT) gathering."""
import json
import argparse
import re
try:
import requests
except ImportError:
requests = None
try:
import dns.resolver
HAS_DNS = True
except ImportError:
HAS_DNS = False
def whois_lookup(domain):
"""Perform WHOIS lookup for domain registration info."""
try:
import whois
w = whois.whois(domain)
return {
"domain": domain, "registrar": w.registrar,
"creation_date": str(w.creation_date),
"expiration_date": str(w.expiration_date),
"name_servers": w.name_servers if isinstance(w.name_servers, list) else [w.name_servers],
"status": w.status if isinstance(w.status, list) else [w.status],
"registrant": w.get("org", w.get("name", "")),
}
except ImportError:
return {"error": "python-whois not installed — pip install python-whois"}
except Exception as e:
return {"domain": domain, "error": str(e)}
def dns_enumeration(domain):
"""Enumerate DNS records for intelligence gathering."""
if not HAS_DNS:
return {"error": "dnspython not installed — pip install dnspython"}
records = {}
for rtype in ["A", "AAAA", "MX", "NS", "TXT", "SOA", "CNAME"]:
try:
answers = dns.resolver.resolve(domain, rtype)
records[rtype] = [str(r) for r in answers]
except Exception:
pass
subs = ["www", "mail", "ftp", "vpn", "api", "dev", "staging", "admin",
"portal", "test", "owa", "remote", "webmail", "autodiscover", "sip"]
found = []
for sub in subs:
try:
answers = dns.resolver.resolve(f"{sub}.{domain}", "A")
found.append({"subdomain": f"{sub}.{domain}", "ips": [str(r) for r in answers]})
except Exception:
pass
return {"domain": domain, "records": records, "subdomains": found}
def email_harvest(domain):
"""Search for email addresses associated with a domain."""
if not requests:
return {"error": "requests not installed"}
emails = set()
try:
resp = requests.get(f"https://api.hunter.io/v2/domain-search?domain={domain}&api_key=demo", timeout=15)
if resp.status_code == 200:
data = resp.json()
for email in data.get("data", {}).get("emails", []):
emails.add(email.get("value", ""))
except Exception:
pass
pattern = re.compile(rf"[a-zA-Z0-9._%+-]+@{re.escape(domain)}", re.I)
search_urls = [f"https://www.google.com/search?q=%22%40{domain}%22&num=20"]
return {
"domain": domain,
"emails_found": list(emails)[:20],
"email_count": len(emails),
"search_pattern": f"*@{domain}",
}
def technology_fingerprint(url):
"""Fingerprint web technologies from HTTP response headers."""
if not requests:
return {"error": "requests not installed"}
try:
resp = requests.get(url, timeout=10, allow_redirects=True)
headers = dict(resp.headers)
technologies = []
server = headers.get("Server", "")
if server:
technologies.append({"category": "Web Server", "value": server})
powered = headers.get("X-Powered-By", "")
if powered:
technologies.append({"category": "Framework", "value": powered})
if "wp-content" in resp.text or "wordpress" in resp.text.lower():
technologies.append({"category": "CMS", "value": "WordPress"})
if "drupal" in resp.text.lower():
technologies.append({"category": "CMS", "value": "Drupal"})
security_headers = {}
for h in ["Strict-Transport-Security", "Content-Security-Policy", "X-Frame-Options",
"X-Content-Type-Options", "X-XSS-Protection", "Referrer-Policy"]:
security_headers[h] = headers.get(h, "MISSING")
return {
"url": url, "status": resp.status_code,
"technologies": technologies,
"security_headers": security_headers,
"cookies": [{"name": c.name, "secure": c.secure, "httponly": "HttpOnly" in str(c._rest)}
for c in resp.cookies][:10],
}
except Exception as e:
return {"url": url, "error": str(e)}
def social_media_search(target_name):
"""Generate social media profile URLs for a target name."""
username = target_name.lower().replace(" ", "")
platforms = {
"linkedin": f"https://www.linkedin.com/in/{username}",
"twitter": f"https://twitter.com/{username}",
"github": f"https://github.com/{username}",
"facebook": f"https://www.facebook.com/{username}",
"instagram": f"https://www.instagram.com/{username}",
}
results = []
for platform, url in platforms.items():
if requests:
try:
resp = requests.head(url, timeout=5, allow_redirects=True)
exists = resp.status_code == 200
results.append({"platform": platform, "url": url, "exists": exists})
except Exception:
results.append({"platform": platform, "url": url, "exists": "unknown"})
else:
results.append({"platform": platform, "url": url, "exists": "unchecked"})
return {"target": target_name, "profiles": results}
def main():
parser = argparse.ArgumentParser(description="OSINT Gathering Agent")
sub = parser.add_subparsers(dest="command")
w = sub.add_parser("whois", help="WHOIS lookup")
w.add_argument("--domain", required=True)
d = sub.add_parser("dns", help="DNS enumeration")
d.add_argument("--domain", required=True)
e = sub.add_parser("email", help="Email harvesting")
e.add_argument("--domain", required=True)
t = sub.add_parser("tech", help="Technology fingerprinting")
t.add_argument("--url", required=True)
s = sub.add_parser("social", help="Social media search")
s.add_argument("--name", required=True)
args = parser.parse_args()
if args.command == "whois":
result = whois_lookup(args.domain)
elif args.command == "dns":
result = dns_enumeration(args.domain)
elif args.command == "email":
result = email_harvest(args.domain)
elif args.command == "tech":
result = technology_fingerprint(args.url)
elif args.command == "social":
result = social_media_search(args.name)
else:
parser.print_help()
return
print(json.dumps(result, indent=2, default=str))
if __name__ == "__main__":
main()
process.py20.9 KB
#!/usr/bin/env python3
"""
OSINT Gathering Automation Tool
Performs automated open source intelligence collection including:
- Subdomain enumeration via Certificate Transparency logs
- DNS record collection
- WHOIS information gathering
- Technology fingerprinting
- Google dorking query generation
- Email pattern discovery
- Shodan/Censys integration
Usage:
python process.py --domain targetdomain.com --output ./osint_report
python process.py --domain targetdomain.com --modules all
python process.py --domain targetdomain.com --modules dns,subdomains,emails
Requirements:
pip install requests dnspython whois beautifulsoup4 rich
"""
import argparse
import json
import re
import socket
import sys
from datetime import datetime
from pathlib import Path
from typing import Any
from urllib.parse import urlparse
try:
import dns.resolver
import requests
from bs4 import BeautifulSoup
from rich.console import Console
from rich.table import Table
from rich.panel import Panel
from rich.progress import Progress, SpinnerColumn, TextColumn
except ImportError:
print("[!] Missing dependencies. Install with:")
print(" pip install requests dnspython beautifulsoup4 rich")
sys.exit(1)
console = Console()
SESSION = requests.Session()
SESSION.headers.update(
{
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
)
def resolve_dns_records(domain: str) -> dict:
"""Collect DNS records for a domain."""
records = {}
record_types = ["A", "AAAA", "MX", "NS", "TXT", "SOA", "CNAME", "SRV"]
for rtype in record_types:
try:
answers = dns.resolver.resolve(domain, rtype)
records[rtype] = [str(rdata) for rdata in answers]
except (dns.resolver.NoAnswer, dns.resolver.NXDOMAIN, dns.resolver.NoNameservers):
pass
except Exception:
pass
return records
def enumerate_subdomains_ct(domain: str) -> list[str]:
"""Enumerate subdomains using Certificate Transparency logs via crt.sh."""
subdomains = set()
try:
url = f"https://crt.sh/?q=%.{domain}&output=json"
response = SESSION.get(url, timeout=30)
if response.status_code == 200:
data = response.json()
for entry in data:
name_value = entry.get("name_value", "")
for name in name_value.split("\n"):
name = name.strip().lower()
if name.endswith(f".{domain}") or name == domain:
# Skip wildcard entries
if not name.startswith("*"):
subdomains.add(name)
except Exception as e:
console.print(f"[yellow][!] crt.sh query failed: {e}[/yellow]")
# Also try common subdomain prefixes
common_prefixes = [
"www", "mail", "ftp", "smtp", "pop", "imap", "webmail",
"vpn", "remote", "portal", "admin", "dev", "staging",
"test", "api", "app", "blog", "shop", "store", "cdn",
"ns1", "ns2", "dns", "mx", "exchange", "owa", "autodiscover",
"sso", "login", "auth", "git", "gitlab", "jenkins",
"jira", "confluence", "wiki", "docs", "support", "help",
]
for prefix in common_prefixes:
subdomain = f"{prefix}.{domain}"
try:
dns.resolver.resolve(subdomain, "A")
subdomains.add(subdomain)
except Exception:
pass
return sorted(subdomains)
def perform_whois_lookup(domain: str) -> dict:
"""Perform WHOIS lookup for a domain."""
try:
import whois as python_whois
w = python_whois.whois(domain)
result = {
"domain_name": str(w.domain_name) if w.domain_name else "N/A",
"registrar": str(w.registrar) if w.registrar else "N/A",
"creation_date": str(w.creation_date) if w.creation_date else "N/A",
"expiration_date": str(w.expiration_date) if w.expiration_date else "N/A",
"name_servers": w.name_servers if w.name_servers else [],
"registrant_org": str(w.org) if w.org else "N/A",
"registrant_country": str(w.country) if w.country else "N/A",
"emails": w.emails if w.emails else [],
}
return result
except Exception as e:
console.print(f"[yellow][!] WHOIS lookup failed: {e}[/yellow]")
return {}
def discover_email_format(domain: str) -> dict:
"""Attempt to discover email format patterns using Hunter.io free tier."""
result = {
"domain": domain,
"format_guess": [],
"discovered_emails": [],
}
# Common email format patterns
common_formats = [
"{first}.{last}@" + domain,
"{first}{last}@" + domain,
"{f}{last}@" + domain,
"{first}_{last}@" + domain,
"{first}@" + domain,
"{last}@" + domain,
]
result["format_guess"] = common_formats
# Try to discover emails from web pages
try:
response = SESSION.get(f"https://{domain}", timeout=10)
if response.status_code == 200:
# Extract emails from page content
email_pattern = re.compile(
rf"[a-zA-Z0-9._%+-]+@{re.escape(domain)}",
re.IGNORECASE
)
emails = email_pattern.findall(response.text)
result["discovered_emails"] = list(set(emails))
except Exception:
pass
# Search for emails on common pages
common_pages = ["/contact", "/about", "/team", "/about-us", "/contact-us"]
for page in common_pages:
try:
response = SESSION.get(f"https://{domain}{page}", timeout=10)
if response.status_code == 200:
email_pattern = re.compile(
rf"[a-zA-Z0-9._%+-]+@{re.escape(domain)}",
re.IGNORECASE
)
emails = email_pattern.findall(response.text)
result["discovered_emails"].extend(emails)
except Exception:
pass
result["discovered_emails"] = list(set(result["discovered_emails"]))
return result
def fingerprint_web_technologies(domain: str) -> dict:
"""Identify web technologies using HTTP headers and response analysis."""
tech = {
"server": None,
"powered_by": None,
"frameworks": [],
"security_headers": {},
"cookies": [],
"cdn": None,
"cms": None,
}
try:
response = SESSION.get(f"https://{domain}", timeout=10, allow_redirects=True)
headers = response.headers
# Server identification
tech["server"] = headers.get("Server", "Not disclosed")
tech["powered_by"] = headers.get("X-Powered-By", "Not disclosed")
# Security headers
security_headers = [
"Strict-Transport-Security",
"Content-Security-Policy",
"X-Frame-Options",
"X-Content-Type-Options",
"X-XSS-Protection",
"Referrer-Policy",
"Permissions-Policy",
"Cross-Origin-Opener-Policy",
"Cross-Origin-Resource-Policy",
]
for header in security_headers:
value = headers.get(header)
tech["security_headers"][header] = value if value else "MISSING"
# Cookie analysis
for cookie in response.cookies:
tech["cookies"].append(
{
"name": cookie.name,
"secure": cookie.secure,
"httponly": "httponly" in cookie._rest,
"domain": cookie.domain,
}
)
# CDN detection
cdn_indicators = {
"cloudflare": ["cf-ray", "cf-cache-status"],
"akamai": ["x-akamai-transformed"],
"cloudfront": ["x-amz-cf-id", "x-amz-cf-pop"],
"fastly": ["x-served-by", "x-fastly-request-id"],
"incapsula": ["x-iinfo"],
}
for cdn, indicators in cdn_indicators.items():
for indicator in indicators:
if indicator in [h.lower() for h in headers]:
tech["cdn"] = cdn
break
# CMS detection from HTML
html = response.text.lower()
if "wp-content" in html or "wordpress" in html:
tech["cms"] = "WordPress"
elif "drupal" in html:
tech["cms"] = "Drupal"
elif "joomla" in html:
tech["cms"] = "Joomla"
elif "shopify" in html:
tech["cms"] = "Shopify"
# Framework detection
if "react" in html or "reactdom" in html:
tech["frameworks"].append("React")
if "angular" in html or "ng-" in html:
tech["frameworks"].append("Angular")
if "vue" in html or "vuejs" in html:
tech["frameworks"].append("Vue.js")
if "jquery" in html:
tech["frameworks"].append("jQuery")
if "bootstrap" in html:
tech["frameworks"].append("Bootstrap")
except Exception as e:
console.print(f"[yellow][!] Web fingerprinting failed: {e}[/yellow]")
return tech
def generate_google_dorks(domain: str) -> list[str]:
"""Generate Google dorking queries for the target domain."""
dorks = [
# Sensitive files
f'site:{domain} filetype:pdf',
f'site:{domain} filetype:xlsx',
f'site:{domain} filetype:docx',
f'site:{domain} filetype:csv',
f'site:{domain} filetype:sql',
f'site:{domain} filetype:log',
f'site:{domain} filetype:bak',
f'site:{domain} filetype:conf',
f'site:{domain} filetype:env',
f'site:{domain} filetype:xml',
# Configuration and credentials
f'site:{domain} inurl:admin',
f'site:{domain} inurl:login',
f'site:{domain} inurl:wp-admin',
f'site:{domain} inurl:wp-login',
f'site:{domain} intitle:"index of"',
f'site:{domain} intitle:"dashboard"',
f'site:{domain} inurl:config',
f'site:{domain} inurl:setup',
# Error messages
f'site:{domain} "error" "sql syntax"',
f'site:{domain} "php error" "on line"',
f'site:{domain} "ORA-" "error"',
f'site:{domain} "mysql" "error"',
f'site:{domain} "stack trace" "at"',
# Sensitive information
f'site:{domain} "confidential"',
f'site:{domain} "internal use only"',
f'site:{domain} "not for distribution"',
f'site:{domain} "password" filetype:txt',
f'site:{domain} "api_key" OR "apikey" OR "api-key"',
# Infrastructure
f'site:{domain} inurl:vpn',
f'site:{domain} inurl:remote',
f'site:{domain} inurl:portal',
f'site:{domain} inurl:citrix',
f'site:{domain} inurl:owa',
# GitHub leaks
f'"{domain}" password site:github.com',
f'"{domain}" api_key site:github.com',
f'"{domain}" secret site:github.com',
f'"{domain}" token site:github.com',
f'"{domain}" site:pastebin.com',
# Cloud storage
f'site:s3.amazonaws.com "{domain.split(".")[0]}"',
f'site:blob.core.windows.net "{domain.split(".")[0]}"',
f'site:storage.googleapis.com "{domain.split(".")[0]}"',
]
return dorks
def check_security_txt(domain: str) -> dict | None:
"""Check for security.txt file per RFC 9116."""
urls = [
f"https://{domain}/.well-known/security.txt",
f"https://{domain}/security.txt",
]
for url in urls:
try:
response = SESSION.get(url, timeout=10)
if response.status_code == 200 and "contact" in response.text.lower():
return {
"url": url,
"content": response.text[:2000],
}
except Exception:
pass
return None
def check_robots_txt(domain: str) -> dict | None:
"""Check robots.txt for interesting paths."""
try:
response = SESSION.get(f"https://{domain}/robots.txt", timeout=10)
if response.status_code == 200:
disallowed = []
for line in response.text.split("\n"):
if line.strip().lower().startswith("disallow:"):
path = line.split(":", 1)[1].strip()
if path and path != "/":
disallowed.append(path)
return {
"content": response.text[:2000],
"disallowed_paths": disallowed,
}
except Exception:
pass
return None
def generate_report(domain: str, results: dict, output_dir: Path):
"""Generate comprehensive OSINT report."""
report = f"""# OSINT Report: {domain}
## Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}
---
## 1. Domain Information
### WHOIS Data
"""
whois_data = results.get("whois", {})
if whois_data:
for key, value in whois_data.items():
report += f"- **{key}**: {value}\n"
else:
report += "WHOIS data not available.\n"
report += "\n### DNS Records\n\n"
dns_data = results.get("dns", {})
if dns_data:
for rtype, records in dns_data.items():
report += f"#### {rtype} Records\n"
for record in records:
report += f"- `{record}`\n"
report += "\n"
report += "## 2. Subdomain Enumeration\n\n"
subdomains = results.get("subdomains", [])
report += f"**Total subdomains discovered:** {len(subdomains)}\n\n"
for sub in subdomains:
try:
ip = socket.gethostbyname(sub)
report += f"- `{sub}` -> `{ip}`\n"
except Exception:
report += f"- `{sub}` -> [unresolvable]\n"
report += "\n## 3. Web Technology Fingerprint\n\n"
tech = results.get("technology", {})
if tech:
report += f"- **Server:** {tech.get('server', 'N/A')}\n"
report += f"- **Powered By:** {tech.get('powered_by', 'N/A')}\n"
report += f"- **CMS:** {tech.get('cms', 'None detected')}\n"
report += f"- **CDN:** {tech.get('cdn', 'None detected')}\n"
report += f"- **Frameworks:** {', '.join(tech.get('frameworks', [])) or 'None detected'}\n\n"
report += "### Security Headers\n\n"
report += "| Header | Status |\n|--------|--------|\n"
for header, value in tech.get("security_headers", {}).items():
status = "MISSING" if value == "MISSING" else "Present"
report += f"| {header} | {status} |\n"
report += "\n## 4. Email Intelligence\n\n"
email_data = results.get("emails", {})
if email_data:
report += "### Discovered Emails\n"
for email in email_data.get("discovered_emails", []):
report += f"- `{email}`\n"
report += "\n### Likely Email Formats\n"
for fmt in email_data.get("format_guess", []):
report += f"- `{fmt}`\n"
report += "\n## 5. Google Dorking Queries\n\n"
dorks = results.get("dorks", [])
for dork in dorks[:20]:
report += f"- `{dork}`\n"
report += "\n## 6. Additional Findings\n\n"
security_txt = results.get("security_txt")
if security_txt:
report += f"### security.txt Found\n- URL: {security_txt['url']}\n\n"
robots_txt = results.get("robots_txt")
if robots_txt:
report += "### Interesting robots.txt Paths\n"
for path in robots_txt.get("disallowed_paths", []):
report += f"- `{path}`\n"
report += f"""
---
## 7. Recommendations for Attack Planning
### Priority Initial Access Vectors
1. Review discovered subdomains for vulnerable web applications
2. Validate credential leaks against target systems
3. Use discovered email format for spearphishing campaign
4. Investigate missing security headers for potential exploitation
5. Check disallowed paths from robots.txt for sensitive content
### Social Engineering Targets
- Use discovered personnel for targeted phishing
- Leverage technology stack knowledge for pretexting
- Utilize physical location data for physical access testing
---
*Report generated by OSINT Automation Tool*
*Classification: CONFIDENTIAL - Red Team Use Only*
"""
report_path = output_dir / f"osint_report_{domain}.md"
with open(report_path, "w") as f:
f.write(report)
console.print(f"[green][+] Report saved to: {report_path}[/green]")
# Save raw data as JSON
json_path = output_dir / f"osint_data_{domain}.json"
serializable_results = {}
for key, value in results.items():
try:
json.dumps(value)
serializable_results[key] = value
except (TypeError, ValueError):
serializable_results[key] = str(value)
with open(json_path, "w") as f:
json.dump(serializable_results, f, indent=2, default=str)
console.print(f"[green][+] Raw data saved to: {json_path}[/green]")
def main():
parser = argparse.ArgumentParser(
description="OSINT Gathering Automation Tool"
)
parser.add_argument("--domain", required=True, help="Target domain")
parser.add_argument(
"--output", default="./osint_output", help="Output directory"
)
parser.add_argument(
"--modules",
default="all",
help="Comma-separated modules: dns,subdomains,whois,emails,tech,dorks,all",
)
args = parser.parse_args()
output_dir = Path(args.output)
output_dir.mkdir(parents=True, exist_ok=True)
modules = args.modules.split(",") if args.modules != "all" else [
"dns", "subdomains", "whois", "emails", "tech", "dorks", "security_txt", "robots_txt"
]
console.print(
Panel(
f"[bold red]OSINT Gathering Tool[/bold red]\n"
f"Target: {args.domain}\n"
f"Modules: {', '.join(modules)}\n"
f"Output: {args.output}",
title="Configuration",
)
)
results = {}
with Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
console=console,
) as progress:
if "dns" in modules:
task = progress.add_task("[cyan]Collecting DNS records...", total=None)
results["dns"] = resolve_dns_records(args.domain)
progress.update(task, completed=True, description="[green]DNS records collected")
if "subdomains" in modules:
task = progress.add_task("[cyan]Enumerating subdomains...", total=None)
results["subdomains"] = enumerate_subdomains_ct(args.domain)
progress.update(
task,
completed=True,
description=f"[green]Found {len(results['subdomains'])} subdomains",
)
if "whois" in modules:
task = progress.add_task("[cyan]Performing WHOIS lookup...", total=None)
results["whois"] = perform_whois_lookup(args.domain)
progress.update(task, completed=True, description="[green]WHOIS data collected")
if "emails" in modules:
task = progress.add_task("[cyan]Discovering email formats...", total=None)
results["emails"] = discover_email_format(args.domain)
progress.update(task, completed=True, description="[green]Email discovery complete")
if "tech" in modules:
task = progress.add_task("[cyan]Fingerprinting technologies...", total=None)
results["technology"] = fingerprint_web_technologies(args.domain)
progress.update(task, completed=True, description="[green]Technology fingerprint complete")
if "dorks" in modules:
task = progress.add_task("[cyan]Generating Google dorks...", total=None)
results["dorks"] = generate_google_dorks(args.domain)
progress.update(
task,
completed=True,
description=f"[green]Generated {len(results['dorks'])} dork queries",
)
if "security_txt" in modules:
task = progress.add_task("[cyan]Checking security.txt...", total=None)
results["security_txt"] = check_security_txt(args.domain)
progress.update(task, completed=True, description="[green]security.txt check complete")
if "robots_txt" in modules:
task = progress.add_task("[cyan]Checking robots.txt...", total=None)
results["robots_txt"] = check_robots_txt(args.domain)
progress.update(task, completed=True, description="[green]robots.txt check complete")
generate_report(args.domain, results, output_dir)
# Display summary table
table = Table(title=f"OSINT Summary: {args.domain}")
table.add_column("Category", style="cyan")
table.add_column("Count/Status", style="green")
table.add_row("DNS Record Types", str(len(results.get("dns", {}))))
table.add_row("Subdomains Found", str(len(results.get("subdomains", []))))
table.add_row("Emails Discovered", str(len(results.get("emails", {}).get("discovered_emails", []))))
table.add_row("Google Dorks Generated", str(len(results.get("dorks", []))))
table.add_row("security.txt", "Found" if results.get("security_txt") else "Not Found")
table.add_row("robots.txt", "Found" if results.get("robots_txt") else "Not Found")
console.print(table)
if __name__ == "__main__":
main()