web application security

Performing Subdomain Enumeration with Subfinder

Enumerate subdomains of target domains using ProjectDiscovery's Subfinder passive reconnaissance tool to map the attack surface during security assessments.

attack-surfacebug-bountyosintpassive-reconreconnaissancesubdomain-enumerationsubfinder
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

When to Use

  • During the reconnaissance phase of penetration testing or bug bounty hunting
  • When mapping the external attack surface of a target organization
  • Before performing vulnerability scanning on discovered subdomains
  • When building an asset inventory for continuous security monitoring
  • During red team engagements requiring passive information gathering

Prerequisites

  • Go 1.21+ installed for building from source
  • Subfinder v2 installed (go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest)
  • API keys configured for passive sources (Shodan, Censys, VirusTotal, SecurityTrails, Chaos)
  • Provider configuration file at $HOME/.config/subfinder/provider-config.yaml
  • Network access to passive DNS and certificate transparency sources
  • httpx or httprobe for validating discovered subdomains

Workflow

Step 1 — Install and Configure Subfinder

# Install subfinder
go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest
 
# Verify installation
subfinder -version
 
# Configure API keys for enhanced results
mkdir -p $HOME/.config/subfinder
cat > $HOME/.config/subfinder/provider-config.yaml << 'EOF'
shodan:
  - YOUR_SHODAN_API_KEY
censys:
  - YOUR_CENSYS_API_ID:YOUR_CENSYS_API_SECRET
virustotal:
  - YOUR_VT_API_KEY
securitytrails:
  - YOUR_ST_API_KEY
chaos:
  - YOUR_CHAOS_API_KEY
EOF

Step 2 — Run Basic Subdomain Enumeration

# Single domain enumeration
subfinder -d example.com -o subdomains.txt
 
# Multiple domains from a file
subfinder -dL domains.txt -o all_subdomains.txt
 
# Use all passive sources (slower but more thorough)
subfinder -d example.com -all -o subdomains_all.txt
 
# Silent mode for piping to other tools
subfinder -d example.com -silent | httpx -silent -status-code

Step 3 — Filter and Customize Source Selection

# Use specific sources only
subfinder -d example.com -s crtsh,virustotal,shodan -o filtered.txt
 
# Exclude specific sources
subfinder -d example.com -es github -o results.txt
 
# Enable recursive subdomain enumeration
subfinder -d example.com -recursive -o recursive_subs.txt
 
# Match specific patterns
subfinder -d example.com -m "api,dev,staging" -o matched.txt

Step 4 — Control Rate Limiting and Output Format

# Rate limit to avoid API throttling
subfinder -d example.com -rate-limit 10 -t 5 -o rate_limited.txt
 
# JSON output for programmatic processing
subfinder -d example.com -oJ -o subdomains.json
 
# Output with source information
subfinder -d example.com -cs -o subdomains_with_sources.txt
 
# Collect results in a directory per domain
subfinder -dL domains.txt -oD ./results/

Step 5 — Validate Discovered Subdomains with httpx

# Pipe subfinder output to httpx for live validation
subfinder -d example.com -silent | httpx -silent -status-code -title -tech-detect -o live_hosts.txt
 
# Check for specific ports
subfinder -d example.com -silent | httpx -ports 80,443,8080,8443 -o web_services.txt
 
# Resolve IP addresses
subfinder -d example.com -silent | dnsx -a -resp -o resolved.txt

Step 6 — Integrate with Broader Recon Pipeline

# Chain with nuclei for vulnerability scanning
subfinder -d example.com -silent | httpx -silent | nuclei -t cves/ -o vulns.txt
 
# Combine with amass for comprehensive enumeration
subfinder -d example.com -o subfinder_results.txt
amass enum -passive -d example.com -o amass_results.txt
cat subfinder_results.txt amass_results.txt | sort -u > combined_subdomains.txt
 
# Screenshot discovered hosts
subfinder -d example.com -silent | httpx -silent | gowitness file -f - -P screenshots/

Key Concepts

Concept Description
Passive Enumeration Discovering subdomains without directly querying target DNS servers
Certificate Transparency Public logs of SSL/TLS certificates revealing subdomain names
DNS Aggregation Collecting subdomain data from multiple passive DNS databases
Recursive Enumeration Discovering subdomains of subdomains for deeper coverage
Source Providers External APIs and databases queried for subdomain intelligence
CNAME Records Canonical name records that may reveal additional infrastructure
Wildcard DNS DNS configuration returning results for any subdomain query

Tools & Systems

Tool Purpose
Subfinder Primary passive subdomain enumeration engine
httpx HTTP probe tool for validating live subdomains
dnsx DNS resolution and validation toolkit
Nuclei Template-based vulnerability scanner for discovered hosts
Amass Complementary subdomain enumeration with active/passive modes
gowitness Web screenshot utility for visual reconnaissance
Shodan Internet-wide scanning database for subdomain intelligence
crt.sh Certificate transparency log search engine

Common Scenarios

  1. Bug Bounty Reconnaissance — Enumerate all subdomains of a target program scope to identify forgotten or misconfigured assets that may contain vulnerabilities
  2. Attack Surface Mapping — Build a comprehensive inventory of externally accessible subdomains for ongoing security monitoring and risk assessment
  3. Cloud Asset Discovery — Identify subdomains pointing to cloud services (AWS, Azure, GCP) that may be vulnerable to subdomain takeover
  4. CI/CD Integration — Automate subdomain monitoring in pipelines to detect new subdomains and alert on changes to the attack surface
  5. Merger & Acquisition Due Diligence — Map the complete external footprint of an acquisition target during security assessment

Output Format

## Subdomain Enumeration Report
- **Target Domain**: example.com
- **Total Subdomains Found**: 247
- **Live Hosts**: 183
- **Unique IP Addresses**: 42
- **Sources Used**: crt.sh, VirusTotal, Shodan, SecurityTrails, Censys
 
### Discovered Subdomains
| Subdomain | IP Address | Status Code | Technology |
|-----------|-----------|-------------|------------|
| api.example.com | 10.0.1.5 | 200 | Nginx, Node.js |
| staging.example.com | 10.0.2.10 | 403 | Apache |
| dev.example.com | 10.0.3.15 | 200 | Express |
 
### Recommendations
- Remove DNS records for decommissioned subdomains
- Investigate subdomains with CNAME pointing to unclaimed services
- Restrict access to development and staging environments
Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 3

api-reference.md2.0 KB

API Reference: Subdomain Enumeration with Subfinder

Subfinder CLI Options

Flag Description
-d <domain> Target domain to enumerate
-dL <file> File containing list of domains
-o <file> Output file for results
-oJ JSON lines output format
-oD <dir> Output directory (one file per domain)
-all Use all passive sources (slower, more thorough)
-silent Show only subdomains in output
-recursive Enumerate subdomains of discovered subdomains
-s <src1,src2> Use only specified sources
-es <src> Exclude specific sources
-cs Show source for each subdomain
-rate-limit <n> Max requests per second
-t <n> Number of concurrent threads

httpx CLI Options

Flag Description
-l <file> Input file with hosts
-ports <p1,p2> Ports to probe
-status-code Show HTTP status code
-title Show page title
-tech-detect Detect web technologies
-json JSON output format
-silent Suppress banner

dnsx CLI Options

Flag Description
-l <file> Input file with hosts
-a Resolve A records
-cname Resolve CNAME records
-resp Show response data
-json JSON output

Passive Sources

Source API Key Required
crt.sh No
VirusTotal Yes
Shodan Yes
SecurityTrails Yes
Censys Yes
Chaos (ProjectDiscovery) Yes
AlienVault OTX No
HackerTarget No

Python Libraries

Library Version Purpose
subprocess stdlib Execute subfinder, httpx, dnsx CLI
json stdlib Parse JSON lines output
pathlib stdlib File path management

References

standards.md1.5 KB

Standards & References — Subdomain Enumeration with Subfinder

Industry Standards

  • OWASP Testing Guide v4.2 — OTG-INFO-004: Enumerate applications on web server through subdomain discovery
  • PTES (Penetration Testing Execution Standard) — Intelligence Gathering phase requiring comprehensive asset enumeration
  • NIST SP 800-115 — Technical Guide to Information Security Testing and Assessment, passive reconnaissance methods
  • MITRE ATT&CK T1596 — Search Open Technical Databases for target infrastructure information

Tool References

API Provider Documentation

Regulatory Considerations

  • Passive subdomain enumeration does not involve active scanning and is generally legal
  • Always verify scope and authorization before proceeding to active enumeration
  • Bug bounty programs define specific scope for subdomain testing
  • GDPR may apply when collecting data that reveals organizational structure
workflows.md1.6 KB

Workflows — Subdomain Enumeration with Subfinder

Standard Enumeration Workflow

  1. Configure API keys in provider-config.yaml for maximum source coverage
  2. Run subfinder with -all flag against target domain(s)
  3. Deduplicate results and remove out-of-scope entries
  4. Validate live hosts with httpx including status codes and technologies
  5. Resolve DNS records with dnsx to map IP infrastructure
  6. Screenshot live hosts with gowitness for visual review
  7. Feed live hosts into vulnerability scanner (nuclei) for automated checks

Continuous Monitoring Workflow

  1. Schedule subfinder runs via cron (daily or weekly)
  2. Compare new results against baseline subdomain list
  3. Alert on newly discovered subdomains via webhook or email
  4. Automatically scan new subdomains for known vulnerabilities
  5. Update asset inventory with new discoveries

Bug Bounty Recon Pipeline

  1. Collect target domains from bug bounty program scope
  2. Run subfinder + amass + findomain for maximum coverage
  3. Merge and deduplicate all results
  4. Filter results to in-scope assets only
  5. Probe for live HTTP services with httpx
  6. Run nuclei templates for quick wins
  7. Manually investigate interesting subdomains (dev, staging, api, admin)

Integration Commands

# Full pipeline example
subfinder -d target.com -all -silent | \
  httpx -silent -status-code -title -tech-detect | \
  tee live_hosts.txt | \
  nuclei -t cves/ -t exposures/ -t misconfigurations/ -o findings.txt
 
# Delta monitoring
subfinder -d target.com -silent > today_subs.txt
comm -13 <(sort baseline_subs.txt) <(sort today_subs.txt) > new_subs.txt

Scripts 2

agent.py6.4 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""Agent for subdomain enumeration using subfinder and httpx.

Runs ProjectDiscovery subfinder for passive subdomain discovery,
validates live hosts with httpx, resolves DNS, and generates
an attack surface report.
"""

import subprocess
import json
import sys
from datetime import datetime
from pathlib import Path
from collections import defaultdict


class SubdomainEnumerationAgent:
    """Enumerates subdomains using subfinder and validates with httpx."""

    def __init__(self, domain, output_dir="./recon"):
        self.domain = domain
        self.output_dir = Path(output_dir)
        self.output_dir.mkdir(parents=True, exist_ok=True)
        self.subdomains = []
        self.live_hosts = []

    def run_subfinder(self, all_sources=False, recursive=False):
        """Run subfinder for passive subdomain enumeration."""
        out_file = self.output_dir / f"{self.domain}_subdomains.txt"
        cmd = ["subfinder", "-d", self.domain, "-o", str(out_file), "-silent"]
        if all_sources:
            cmd.append("-all")
        if recursive:
            cmd.append("-recursive")

        try:
            result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
            if out_file.exists():
                self.subdomains = [
                    line.strip() for line in out_file.read_text().splitlines()
                    if line.strip()
                ]
            return {"count": len(self.subdomains),
                    "output_file": str(out_file),
                    "stderr": result.stderr[:200] if result.stderr else ""}
        except FileNotFoundError:
            return {"error": "subfinder not installed. Install: go install -v "
                    "github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest"}
        except subprocess.TimeoutExpired:
            return {"error": "subfinder timed out after 300s"}

    def run_subfinder_json(self):
        """Run subfinder with JSON output for source tracking."""
        cmd = ["subfinder", "-d", self.domain, "-oJ", "-silent"]
        try:
            result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
            entries = []
            for line in result.stdout.strip().splitlines():
                try:
                    entry = json.loads(line)
                    entries.append(entry)
                    host = entry.get("host", "")
                    if host and host not in self.subdomains:
                        self.subdomains.append(host)
                except json.JSONDecodeError:
                    continue
            return entries
        except (FileNotFoundError, subprocess.TimeoutExpired):
            return []

    def validate_with_httpx(self, ports="80,443"):
        """Validate discovered subdomains with httpx probe."""
        if not self.subdomains:
            return []

        input_file = self.output_dir / f"{self.domain}_to_probe.txt"
        input_file.write_text("\n".join(self.subdomains))
        out_file = self.output_dir / f"{self.domain}_live.json"

        cmd = ["httpx", "-l", str(input_file), "-ports", ports,
               "-status-code", "-title", "-json", "-o", str(out_file), "-silent"]
        try:
            subprocess.run(cmd, capture_output=True, text=True, timeout=600)
            if out_file.exists():
                for line in out_file.read_text().splitlines():
                    try:
                        self.live_hosts.append(json.loads(line))
                    except json.JSONDecodeError:
                        continue
            return self.live_hosts
        except (FileNotFoundError, subprocess.TimeoutExpired):
            return []

    def resolve_dns(self):
        """Resolve subdomains to IP addresses using dnsx."""
        if not self.subdomains:
            return []
        input_file = self.output_dir / f"{self.domain}_to_resolve.txt"
        input_file.write_text("\n".join(self.subdomains))
        cmd = ["dnsx", "-l", str(input_file), "-a", "-resp", "-json", "-silent"]
        try:
            result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
            records = []
            for line in result.stdout.strip().splitlines():
                try:
                    records.append(json.loads(line))
                except json.JSONDecodeError:
                    continue
            return records
        except (FileNotFoundError, subprocess.TimeoutExpired):
            return []

    def detect_takeover_candidates(self):
        """Identify subdomains potentially vulnerable to takeover."""
        candidates = []
        cloud_cnames = ["amazonaws.com", "azurewebsites.net", "cloudfront.net",
                        "herokuapp.com", "github.io", "s3.amazonaws.com",
                        "blob.core.windows.net", "cloudapp.azure.com"]
        dns_records = self.resolve_dns()
        for record in dns_records:
            cname = record.get("cname", "")
            if any(cloud in cname for cloud in cloud_cnames):
                candidates.append({
                    "subdomain": record.get("host", ""),
                    "cname": cname,
                    "risk": "Potential subdomain takeover"
                })
        return candidates

    def generate_report(self):
        """Generate attack surface enumeration report."""
        status_dist = defaultdict(int)
        for host in self.live_hosts:
            code = host.get("status_code", 0)
            status_dist[str(code)] += 1

        report = {
            "domain": self.domain,
            "report_date": datetime.utcnow().isoformat(),
            "total_subdomains": len(self.subdomains),
            "live_hosts": len(self.live_hosts),
            "status_distribution": dict(status_dist),
            "subdomains": self.subdomains[:100],
            "live_host_details": self.live_hosts[:50],
            "takeover_candidates": self.detect_takeover_candidates(),
        }

        report_path = self.output_dir / f"{self.domain}_report.json"
        with open(report_path, "w") as f:
            json.dump(report, f, indent=2)
        print(json.dumps(report, indent=2))
        return report


def main():
    if len(sys.argv) < 2:
        print("Usage: agent.py <domain> [output_dir]")
        sys.exit(1)
    domain = sys.argv[1]
    output_dir = sys.argv[2] if len(sys.argv) > 2 else "./recon"
    agent = SubdomainEnumerationAgent(domain, output_dir)
    agent.run_subfinder(all_sources=True)
    agent.validate_with_httpx()
    agent.generate_report()


if __name__ == "__main__":
    main()
process.py5.7 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""
Subdomain Enumeration Pipeline with Subfinder
Automates subdomain discovery, validation, and reporting.
"""

import subprocess
import json
import csv
import sys
import os
from datetime import datetime
from pathlib import Path


def run_subfinder(domain: str, output_dir: str, use_all_sources: bool = False) -> list:
    """Run subfinder against a target domain and return discovered subdomains."""
    os.makedirs(output_dir, exist_ok=True)
    output_file = os.path.join(output_dir, f"{domain}_subdomains.txt")

    cmd = ["subfinder", "-d", domain, "-silent"]
    if use_all_sources:
        cmd.append("-all")

    result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
    subdomains = [line.strip() for line in result.stdout.strip().split("\n") if line.strip()]

    with open(output_file, "w") as f:
        f.write("\n".join(subdomains))

    print(f"[+] Found {len(subdomains)} subdomains for {domain}")
    return subdomains


def validate_with_httpx(subdomains: list, output_dir: str) -> list:
    """Validate discovered subdomains using httpx."""
    input_data = "\n".join(subdomains)
    cmd = [
        "httpx", "-silent", "-status-code", "-title",
        "-tech-detect", "-json", "-no-color"
    ]

    result = subprocess.run(
        cmd, input=input_data, capture_output=True, text=True, timeout=600
    )

    live_hosts = []
    for line in result.stdout.strip().split("\n"):
        if line.strip():
            try:
                host_data = json.loads(line)
                live_hosts.append(host_data)
            except json.JSONDecodeError:
                continue

    output_file = os.path.join(output_dir, "live_hosts.json")
    with open(output_file, "w") as f:
        json.dump(live_hosts, f, indent=2)

    print(f"[+] Validated {len(live_hosts)} live hosts")
    return live_hosts


def detect_takeover_candidates(subdomains: list) -> list:
    """Identify subdomains with CNAME records pointing to potentially claimable services."""
    takeover_services = [
        "amazonaws.com", "azurewebsites.net", "cloudfront.net",
        "herokuapp.com", "github.io", "gitlab.io", "pantheon.io",
        "shopify.com", "surge.sh", "fastly.net", "ghost.io",
        "myshopify.com", "zendesk.com", "readme.io", "bitbucket.io"
    ]

    candidates = []
    for subdomain in subdomains:
        try:
            result = subprocess.run(
                ["dig", "+short", "CNAME", subdomain],
                capture_output=True, text=True, timeout=10
            )
            cname = result.stdout.strip()
            if cname:
                for service in takeover_services:
                    if service in cname:
                        candidates.append({
                            "subdomain": subdomain,
                            "cname": cname,
                            "service": service
                        })
                        break
        except (subprocess.TimeoutExpired, FileNotFoundError):
            continue

    return candidates


def generate_report(domain: str, subdomains: list, live_hosts: list,
                    takeover_candidates: list, output_dir: str) -> str:
    """Generate a markdown report of enumeration results."""
    report_path = os.path.join(output_dir, f"{domain}_report.md")
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")

    with open(report_path, "w") as f:
        f.write(f"# Subdomain Enumeration Report: {domain}\n\n")
        f.write(f"**Date**: {timestamp}\n\n")
        f.write(f"## Summary\n")
        f.write(f"- **Total Subdomains Discovered**: {len(subdomains)}\n")
        f.write(f"- **Live Hosts**: {len(live_hosts)}\n")
        f.write(f"- **Takeover Candidates**: {len(takeover_candidates)}\n\n")

        f.write("## Live Hosts\n\n")
        f.write("| URL | Status | Title | Technologies |\n")
        f.write("|-----|--------|-------|--------------|\n")
        for host in live_hosts:
            url = host.get("url", "N/A")
            status = host.get("status_code", "N/A")
            title = host.get("title", "N/A")
            techs = ", ".join(host.get("tech", [])) if host.get("tech") else "N/A"
            f.write(f"| {url} | {status} | {title} | {techs} |\n")

        if takeover_candidates:
            f.write("\n## Potential Subdomain Takeover Candidates\n\n")
            f.write("| Subdomain | CNAME | Service |\n")
            f.write("|-----------|-------|---------|\n")
            for candidate in takeover_candidates:
                f.write(f"| {candidate['subdomain']} | {candidate['cname']} | {candidate['service']} |\n")

        f.write("\n## All Discovered Subdomains\n\n")
        for sub in sorted(subdomains):
            f.write(f"- {sub}\n")

    print(f"[+] Report saved to {report_path}")
    return report_path


def main():
    if len(sys.argv) < 2:
        print("Usage: python process.py <domain> [output_dir] [--all]")
        sys.exit(1)

    domain = sys.argv[1]
    output_dir = sys.argv[2] if len(sys.argv) > 2 and not sys.argv[2].startswith("--") else "./recon_output"
    use_all = "--all" in sys.argv

    print(f"[*] Starting subdomain enumeration for {domain}")
    subdomains = run_subfinder(domain, output_dir, use_all)

    if not subdomains:
        print("[-] No subdomains found. Exiting.")
        sys.exit(0)

    print("[*] Validating live hosts with httpx...")
    live_hosts = validate_with_httpx(subdomains, output_dir)

    print("[*] Checking for subdomain takeover candidates...")
    takeover_candidates = detect_takeover_candidates(subdomains)

    print("[*] Generating report...")
    generate_report(domain, subdomains, live_hosts, takeover_candidates, output_dir)

    print(f"[+] Enumeration complete. Results saved to {output_dir}/")


if __name__ == "__main__":
    main()

Assets 1

template.mdtext/markdown · 1.5 KB
Keep exploring