Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-SkillsFramework mappings
MITRE ATT&CK
When to Use
- During the reconnaissance phase of penetration testing or bug bounty hunting
- When mapping the external attack surface of a target organization
- Before performing vulnerability scanning on discovered subdomains
- When building an asset inventory for continuous security monitoring
- During red team engagements requiring passive information gathering
Prerequisites
- Go 1.21+ installed for building from source
- Subfinder v2 installed (
go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest) - API keys configured for passive sources (Shodan, Censys, VirusTotal, SecurityTrails, Chaos)
- Provider configuration file at
$HOME/.config/subfinder/provider-config.yaml - Network access to passive DNS and certificate transparency sources
- httpx or httprobe for validating discovered subdomains
Workflow
Step 1 — Install and Configure Subfinder
# Install subfinder
go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
# Verify installation
subfinder -version
# Configure API keys for enhanced results
mkdir -p $HOME/.config/subfinder
cat > $HOME/.config/subfinder/provider-config.yaml << 'EOF'
shodan:
- YOUR_SHODAN_API_KEY
censys:
- YOUR_CENSYS_API_ID:YOUR_CENSYS_API_SECRET
virustotal:
- YOUR_VT_API_KEY
securitytrails:
- YOUR_ST_API_KEY
chaos:
- YOUR_CHAOS_API_KEY
EOFStep 2 — Run Basic Subdomain Enumeration
# Single domain enumeration
subfinder -d example.com -o subdomains.txt
# Multiple domains from a file
subfinder -dL domains.txt -o all_subdomains.txt
# Use all passive sources (slower but more thorough)
subfinder -d example.com -all -o subdomains_all.txt
# Silent mode for piping to other tools
subfinder -d example.com -silent | httpx -silent -status-codeStep 3 — Filter and Customize Source Selection
# Use specific sources only
subfinder -d example.com -s crtsh,virustotal,shodan -o filtered.txt
# Exclude specific sources
subfinder -d example.com -es github -o results.txt
# Enable recursive subdomain enumeration
subfinder -d example.com -recursive -o recursive_subs.txt
# Match specific patterns
subfinder -d example.com -m "api,dev,staging" -o matched.txtStep 4 — Control Rate Limiting and Output Format
# Rate limit to avoid API throttling
subfinder -d example.com -rate-limit 10 -t 5 -o rate_limited.txt
# JSON output for programmatic processing
subfinder -d example.com -oJ -o subdomains.json
# Output with source information
subfinder -d example.com -cs -o subdomains_with_sources.txt
# Collect results in a directory per domain
subfinder -dL domains.txt -oD ./results/Step 5 — Validate Discovered Subdomains with httpx
# Pipe subfinder output to httpx for live validation
subfinder -d example.com -silent | httpx -silent -status-code -title -tech-detect -o live_hosts.txt
# Check for specific ports
subfinder -d example.com -silent | httpx -ports 80,443,8080,8443 -o web_services.txt
# Resolve IP addresses
subfinder -d example.com -silent | dnsx -a -resp -o resolved.txtStep 6 — Integrate with Broader Recon Pipeline
# Chain with nuclei for vulnerability scanning
subfinder -d example.com -silent | httpx -silent | nuclei -t cves/ -o vulns.txt
# Combine with amass for comprehensive enumeration
subfinder -d example.com -o subfinder_results.txt
amass enum -passive -d example.com -o amass_results.txt
cat subfinder_results.txt amass_results.txt | sort -u > combined_subdomains.txt
# Screenshot discovered hosts
subfinder -d example.com -silent | httpx -silent | gowitness file -f - -P screenshots/Key Concepts
| Concept | Description |
|---|---|
| Passive Enumeration | Discovering subdomains without directly querying target DNS servers |
| Certificate Transparency | Public logs of SSL/TLS certificates revealing subdomain names |
| DNS Aggregation | Collecting subdomain data from multiple passive DNS databases |
| Recursive Enumeration | Discovering subdomains of subdomains for deeper coverage |
| Source Providers | External APIs and databases queried for subdomain intelligence |
| CNAME Records | Canonical name records that may reveal additional infrastructure |
| Wildcard DNS | DNS configuration returning results for any subdomain query |
Tools & Systems
| Tool | Purpose |
|---|---|
| Subfinder | Primary passive subdomain enumeration engine |
| httpx | HTTP probe tool for validating live subdomains |
| dnsx | DNS resolution and validation toolkit |
| Nuclei | Template-based vulnerability scanner for discovered hosts |
| Amass | Complementary subdomain enumeration with active/passive modes |
| gowitness | Web screenshot utility for visual reconnaissance |
| Shodan | Internet-wide scanning database for subdomain intelligence |
| crt.sh | Certificate transparency log search engine |
Common Scenarios
- Bug Bounty Reconnaissance — Enumerate all subdomains of a target program scope to identify forgotten or misconfigured assets that may contain vulnerabilities
- Attack Surface Mapping — Build a comprehensive inventory of externally accessible subdomains for ongoing security monitoring and risk assessment
- Cloud Asset Discovery — Identify subdomains pointing to cloud services (AWS, Azure, GCP) that may be vulnerable to subdomain takeover
- CI/CD Integration — Automate subdomain monitoring in pipelines to detect new subdomains and alert on changes to the attack surface
- Merger & Acquisition Due Diligence — Map the complete external footprint of an acquisition target during security assessment
Output Format
## Subdomain Enumeration Report
- **Target Domain**: example.com
- **Total Subdomains Found**: 247
- **Live Hosts**: 183
- **Unique IP Addresses**: 42
- **Sources Used**: crt.sh, VirusTotal, Shodan, SecurityTrails, Censys
### Discovered Subdomains
| Subdomain | IP Address | Status Code | Technology |
|-----------|-----------|-------------|------------|
| api.example.com | 10.0.1.5 | 200 | Nginx, Node.js |
| staging.example.com | 10.0.2.10 | 403 | Apache |
| dev.example.com | 10.0.3.15 | 200 | Express |
### Recommendations
- Remove DNS records for decommissioned subdomains
- Investigate subdomains with CNAME pointing to unclaimed services
- Restrict access to development and staging environmentsSource materials
References and resources
Everything below is rendered for inspection. Script files are read-only and never run.
References 3
api-reference.md2.0 KB
API Reference: Subdomain Enumeration with Subfinder
Subfinder CLI Options
| Flag | Description |
|---|---|
-d <domain> |
Target domain to enumerate |
-dL <file> |
File containing list of domains |
-o <file> |
Output file for results |
-oJ |
JSON lines output format |
-oD <dir> |
Output directory (one file per domain) |
-all |
Use all passive sources (slower, more thorough) |
-silent |
Show only subdomains in output |
-recursive |
Enumerate subdomains of discovered subdomains |
-s <src1,src2> |
Use only specified sources |
-es <src> |
Exclude specific sources |
-cs |
Show source for each subdomain |
-rate-limit <n> |
Max requests per second |
-t <n> |
Number of concurrent threads |
httpx CLI Options
| Flag | Description |
|---|---|
-l <file> |
Input file with hosts |
-ports <p1,p2> |
Ports to probe |
-status-code |
Show HTTP status code |
-title |
Show page title |
-tech-detect |
Detect web technologies |
-json |
JSON output format |
-silent |
Suppress banner |
dnsx CLI Options
| Flag | Description |
|---|---|
-l <file> |
Input file with hosts |
-a |
Resolve A records |
-cname |
Resolve CNAME records |
-resp |
Show response data |
-json |
JSON output |
Passive Sources
| Source | API Key Required |
|---|---|
| crt.sh | No |
| VirusTotal | Yes |
| Shodan | Yes |
| SecurityTrails | Yes |
| Censys | Yes |
| Chaos (ProjectDiscovery) | Yes |
| AlienVault OTX | No |
| HackerTarget | No |
Python Libraries
| Library | Version | Purpose |
|---|---|---|
subprocess |
stdlib | Execute subfinder, httpx, dnsx CLI |
json |
stdlib | Parse JSON lines output |
pathlib |
stdlib | File path management |
References
- Subfinder GitHub: https://github.com/projectdiscovery/subfinder
- httpx GitHub: https://github.com/projectdiscovery/httpx
- dnsx GitHub: https://github.com/projectdiscovery/dnsx
- Subfinder Config: https://docs.projectdiscovery.io/tools/subfinder/install
standards.md1.5 KB
Standards & References — Subdomain Enumeration with Subfinder
Industry Standards
- OWASP Testing Guide v4.2 — OTG-INFO-004: Enumerate applications on web server through subdomain discovery
- PTES (Penetration Testing Execution Standard) — Intelligence Gathering phase requiring comprehensive asset enumeration
- NIST SP 800-115 — Technical Guide to Information Security Testing and Assessment, passive reconnaissance methods
- MITRE ATT&CK T1596 — Search Open Technical Databases for target infrastructure information
Tool References
- Subfinder GitHub: https://github.com/projectdiscovery/subfinder
- ProjectDiscovery Documentation: https://docs.projectdiscovery.io/tools/subfinder/overview
- Certificate Transparency RFC 6962: https://www.rfc-editor.org/rfc/rfc6962
- crt.sh Certificate Search: https://crt.sh/
API Provider Documentation
- Shodan API: https://developer.shodan.io/api
- Censys Search API: https://search.censys.io/api
- VirusTotal API v3: https://docs.virustotal.com/reference/overview
- SecurityTrails API: https://docs.securitytrails.com/reference
- Chaos ProjectDiscovery: https://chaos.projectdiscovery.io/
Regulatory Considerations
- Passive subdomain enumeration does not involve active scanning and is generally legal
- Always verify scope and authorization before proceeding to active enumeration
- Bug bounty programs define specific scope for subdomain testing
- GDPR may apply when collecting data that reveals organizational structure
workflows.md1.6 KB
Workflows — Subdomain Enumeration with Subfinder
Standard Enumeration Workflow
- Configure API keys in provider-config.yaml for maximum source coverage
- Run subfinder with
-allflag against target domain(s) - Deduplicate results and remove out-of-scope entries
- Validate live hosts with httpx including status codes and technologies
- Resolve DNS records with dnsx to map IP infrastructure
- Screenshot live hosts with gowitness for visual review
- Feed live hosts into vulnerability scanner (nuclei) for automated checks
Continuous Monitoring Workflow
- Schedule subfinder runs via cron (daily or weekly)
- Compare new results against baseline subdomain list
- Alert on newly discovered subdomains via webhook or email
- Automatically scan new subdomains for known vulnerabilities
- Update asset inventory with new discoveries
Bug Bounty Recon Pipeline
- Collect target domains from bug bounty program scope
- Run subfinder + amass + findomain for maximum coverage
- Merge and deduplicate all results
- Filter results to in-scope assets only
- Probe for live HTTP services with httpx
- Run nuclei templates for quick wins
- Manually investigate interesting subdomains (dev, staging, api, admin)
Integration Commands
# Full pipeline example
subfinder -d target.com -all -silent | \
httpx -silent -status-code -title -tech-detect | \
tee live_hosts.txt | \
nuclei -t cves/ -t exposures/ -t misconfigurations/ -o findings.txt
# Delta monitoring
subfinder -d target.com -silent > today_subs.txt
comm -13 <(sort baseline_subs.txt) <(sort today_subs.txt) > new_subs.txtScripts 2
agent.py6.4 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""Agent for subdomain enumeration using subfinder and httpx.
Runs ProjectDiscovery subfinder for passive subdomain discovery,
validates live hosts with httpx, resolves DNS, and generates
an attack surface report.
"""
import subprocess
import json
import sys
from datetime import datetime
from pathlib import Path
from collections import defaultdict
class SubdomainEnumerationAgent:
"""Enumerates subdomains using subfinder and validates with httpx."""
def __init__(self, domain, output_dir="./recon"):
self.domain = domain
self.output_dir = Path(output_dir)
self.output_dir.mkdir(parents=True, exist_ok=True)
self.subdomains = []
self.live_hosts = []
def run_subfinder(self, all_sources=False, recursive=False):
"""Run subfinder for passive subdomain enumeration."""
out_file = self.output_dir / f"{self.domain}_subdomains.txt"
cmd = ["subfinder", "-d", self.domain, "-o", str(out_file), "-silent"]
if all_sources:
cmd.append("-all")
if recursive:
cmd.append("-recursive")
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
if out_file.exists():
self.subdomains = [
line.strip() for line in out_file.read_text().splitlines()
if line.strip()
]
return {"count": len(self.subdomains),
"output_file": str(out_file),
"stderr": result.stderr[:200] if result.stderr else ""}
except FileNotFoundError:
return {"error": "subfinder not installed. Install: go install -v "
"github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest"}
except subprocess.TimeoutExpired:
return {"error": "subfinder timed out after 300s"}
def run_subfinder_json(self):
"""Run subfinder with JSON output for source tracking."""
cmd = ["subfinder", "-d", self.domain, "-oJ", "-silent"]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
entries = []
for line in result.stdout.strip().splitlines():
try:
entry = json.loads(line)
entries.append(entry)
host = entry.get("host", "")
if host and host not in self.subdomains:
self.subdomains.append(host)
except json.JSONDecodeError:
continue
return entries
except (FileNotFoundError, subprocess.TimeoutExpired):
return []
def validate_with_httpx(self, ports="80,443"):
"""Validate discovered subdomains with httpx probe."""
if not self.subdomains:
return []
input_file = self.output_dir / f"{self.domain}_to_probe.txt"
input_file.write_text("\n".join(self.subdomains))
out_file = self.output_dir / f"{self.domain}_live.json"
cmd = ["httpx", "-l", str(input_file), "-ports", ports,
"-status-code", "-title", "-json", "-o", str(out_file), "-silent"]
try:
subprocess.run(cmd, capture_output=True, text=True, timeout=600)
if out_file.exists():
for line in out_file.read_text().splitlines():
try:
self.live_hosts.append(json.loads(line))
except json.JSONDecodeError:
continue
return self.live_hosts
except (FileNotFoundError, subprocess.TimeoutExpired):
return []
def resolve_dns(self):
"""Resolve subdomains to IP addresses using dnsx."""
if not self.subdomains:
return []
input_file = self.output_dir / f"{self.domain}_to_resolve.txt"
input_file.write_text("\n".join(self.subdomains))
cmd = ["dnsx", "-l", str(input_file), "-a", "-resp", "-json", "-silent"]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
records = []
for line in result.stdout.strip().splitlines():
try:
records.append(json.loads(line))
except json.JSONDecodeError:
continue
return records
except (FileNotFoundError, subprocess.TimeoutExpired):
return []
def detect_takeover_candidates(self):
"""Identify subdomains potentially vulnerable to takeover."""
candidates = []
cloud_cnames = ["amazonaws.com", "azurewebsites.net", "cloudfront.net",
"herokuapp.com", "github.io", "s3.amazonaws.com",
"blob.core.windows.net", "cloudapp.azure.com"]
dns_records = self.resolve_dns()
for record in dns_records:
cname = record.get("cname", "")
if any(cloud in cname for cloud in cloud_cnames):
candidates.append({
"subdomain": record.get("host", ""),
"cname": cname,
"risk": "Potential subdomain takeover"
})
return candidates
def generate_report(self):
"""Generate attack surface enumeration report."""
status_dist = defaultdict(int)
for host in self.live_hosts:
code = host.get("status_code", 0)
status_dist[str(code)] += 1
report = {
"domain": self.domain,
"report_date": datetime.utcnow().isoformat(),
"total_subdomains": len(self.subdomains),
"live_hosts": len(self.live_hosts),
"status_distribution": dict(status_dist),
"subdomains": self.subdomains[:100],
"live_host_details": self.live_hosts[:50],
"takeover_candidates": self.detect_takeover_candidates(),
}
report_path = self.output_dir / f"{self.domain}_report.json"
with open(report_path, "w") as f:
json.dump(report, f, indent=2)
print(json.dumps(report, indent=2))
return report
def main():
if len(sys.argv) < 2:
print("Usage: agent.py <domain> [output_dir]")
sys.exit(1)
domain = sys.argv[1]
output_dir = sys.argv[2] if len(sys.argv) > 2 else "./recon"
agent = SubdomainEnumerationAgent(domain, output_dir)
agent.run_subfinder(all_sources=True)
agent.validate_with_httpx()
agent.generate_report()
if __name__ == "__main__":
main()
process.py5.7 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""
Subdomain Enumeration Pipeline with Subfinder
Automates subdomain discovery, validation, and reporting.
"""
import subprocess
import json
import csv
import sys
import os
from datetime import datetime
from pathlib import Path
def run_subfinder(domain: str, output_dir: str, use_all_sources: bool = False) -> list:
"""Run subfinder against a target domain and return discovered subdomains."""
os.makedirs(output_dir, exist_ok=True)
output_file = os.path.join(output_dir, f"{domain}_subdomains.txt")
cmd = ["subfinder", "-d", domain, "-silent"]
if use_all_sources:
cmd.append("-all")
result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
subdomains = [line.strip() for line in result.stdout.strip().split("\n") if line.strip()]
with open(output_file, "w") as f:
f.write("\n".join(subdomains))
print(f"[+] Found {len(subdomains)} subdomains for {domain}")
return subdomains
def validate_with_httpx(subdomains: list, output_dir: str) -> list:
"""Validate discovered subdomains using httpx."""
input_data = "\n".join(subdomains)
cmd = [
"httpx", "-silent", "-status-code", "-title",
"-tech-detect", "-json", "-no-color"
]
result = subprocess.run(
cmd, input=input_data, capture_output=True, text=True, timeout=600
)
live_hosts = []
for line in result.stdout.strip().split("\n"):
if line.strip():
try:
host_data = json.loads(line)
live_hosts.append(host_data)
except json.JSONDecodeError:
continue
output_file = os.path.join(output_dir, "live_hosts.json")
with open(output_file, "w") as f:
json.dump(live_hosts, f, indent=2)
print(f"[+] Validated {len(live_hosts)} live hosts")
return live_hosts
def detect_takeover_candidates(subdomains: list) -> list:
"""Identify subdomains with CNAME records pointing to potentially claimable services."""
takeover_services = [
"amazonaws.com", "azurewebsites.net", "cloudfront.net",
"herokuapp.com", "github.io", "gitlab.io", "pantheon.io",
"shopify.com", "surge.sh", "fastly.net", "ghost.io",
"myshopify.com", "zendesk.com", "readme.io", "bitbucket.io"
]
candidates = []
for subdomain in subdomains:
try:
result = subprocess.run(
["dig", "+short", "CNAME", subdomain],
capture_output=True, text=True, timeout=10
)
cname = result.stdout.strip()
if cname:
for service in takeover_services:
if service in cname:
candidates.append({
"subdomain": subdomain,
"cname": cname,
"service": service
})
break
except (subprocess.TimeoutExpired, FileNotFoundError):
continue
return candidates
def generate_report(domain: str, subdomains: list, live_hosts: list,
takeover_candidates: list, output_dir: str) -> str:
"""Generate a markdown report of enumeration results."""
report_path = os.path.join(output_dir, f"{domain}_report.md")
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
with open(report_path, "w") as f:
f.write(f"# Subdomain Enumeration Report: {domain}\n\n")
f.write(f"**Date**: {timestamp}\n\n")
f.write(f"## Summary\n")
f.write(f"- **Total Subdomains Discovered**: {len(subdomains)}\n")
f.write(f"- **Live Hosts**: {len(live_hosts)}\n")
f.write(f"- **Takeover Candidates**: {len(takeover_candidates)}\n\n")
f.write("## Live Hosts\n\n")
f.write("| URL | Status | Title | Technologies |\n")
f.write("|-----|--------|-------|--------------|\n")
for host in live_hosts:
url = host.get("url", "N/A")
status = host.get("status_code", "N/A")
title = host.get("title", "N/A")
techs = ", ".join(host.get("tech", [])) if host.get("tech") else "N/A"
f.write(f"| {url} | {status} | {title} | {techs} |\n")
if takeover_candidates:
f.write("\n## Potential Subdomain Takeover Candidates\n\n")
f.write("| Subdomain | CNAME | Service |\n")
f.write("|-----------|-------|---------|\n")
for candidate in takeover_candidates:
f.write(f"| {candidate['subdomain']} | {candidate['cname']} | {candidate['service']} |\n")
f.write("\n## All Discovered Subdomains\n\n")
for sub in sorted(subdomains):
f.write(f"- {sub}\n")
print(f"[+] Report saved to {report_path}")
return report_path
def main():
if len(sys.argv) < 2:
print("Usage: python process.py <domain> [output_dir] [--all]")
sys.exit(1)
domain = sys.argv[1]
output_dir = sys.argv[2] if len(sys.argv) > 2 and not sys.argv[2].startswith("--") else "./recon_output"
use_all = "--all" in sys.argv
print(f"[*] Starting subdomain enumeration for {domain}")
subdomains = run_subfinder(domain, output_dir, use_all)
if not subdomains:
print("[-] No subdomains found. Exiting.")
sys.exit(0)
print("[*] Validating live hosts with httpx...")
live_hosts = validate_with_httpx(subdomains, output_dir)
print("[*] Checking for subdomain takeover candidates...")
takeover_candidates = detect_takeover_candidates(subdomains)
print("[*] Generating report...")
generate_report(domain, subdomains, live_hosts, takeover_candidates, output_dir)
print(f"[+] Enumeration complete. Results saved to {output_dir}/")
if __name__ == "__main__":
main()
Assets 1
template.mdtext/markdown · 1.5 KBKeep exploring