threat intelligence

Analyzing Certificate Transparency for Phishing

Monitor Certificate Transparency logs using crt.sh and Certstream to detect phishing domains, lookalike certificates, and unauthorized certificate issuance targeting your organization.

certificate-transparencycertstreamcrt-shct-logsdomain-monitoringphishingsslthreat-intelligence
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

Overview

Certificate Transparency (CT) is an Internet security standard that creates a public, append-only log of all issued SSL/TLS certificates. Monitoring CT logs enables early detection of phishing domains that register certificates mimicking legitimate brands, unauthorized certificate issuance for owned domains, and certificate-based attack infrastructure. This skill covers querying CT logs via crt.sh, real-time monitoring with Certstream, building automated alerting for suspicious certificates, and integrating findings into threat intelligence workflows.

When to Use

  • When investigating security incidents that require analyzing certificate transparency for phishing
  • When building detection rules or threat hunting queries for this domain
  • When SOC analysts need structured procedures for this analysis type
  • When validating security monitoring coverage for related attack techniques

Prerequisites

  • Python 3.9+ with requests, certstream, tldextract, Levenshtein libraries
  • Access to crt.sh (https://crt.sh/) for historical CT log queries
  • Certstream (https://certstream.calidog.io/) for real-time monitoring
  • List of organization domains and brand keywords to monitor
  • Understanding of SSL/TLS certificate structure and issuance process

Key Concepts

Certificate Transparency Logs

CT logs are cryptographically assured, publicly auditable, append-only records of TLS certificate issuance. Major CAs (Let's Encrypt, DigiCert, Sectigo, Google Trust Services) submit all issued certificates to multiple CT logs. As of 2025, Chrome and Safari require CT for all publicly trusted certificates.

Phishing Detection via CT

Attackers register lookalike domains and obtain free certificates (often from Let's Encrypt) to make phishing sites appear legitimate with HTTPS. CT monitoring detects these early because the certificate appears in logs before the phishing campaign launches, providing a window for proactive blocking.

crt.sh Database

crt.sh is a free web interface and PostgreSQL database operated by Sectigo that indexes CT logs. It supports wildcard searches (%.example.com), direct SQL queries, and JSON API responses. It tracks certificate issuance, expiration, and revocation across all major CT logs.

Workflow

Step 1: Query crt.sh for Certificate History

import requests
import json
from datetime import datetime
import tldextract
 
class CTLogMonitor:
    CRT_SH_URL = "https://crt.sh"
 
    def __init__(self, monitored_domains, brand_keywords):
        self.monitored_domains = monitored_domains
        self.brand_keywords = [k.lower() for k in brand_keywords]
 
    def query_crt_sh(self, domain, include_expired=False):
        """Query crt.sh for certificates matching a domain."""
        params = {
            "q": f"%.{domain}",
            "output": "json",
        }
        if not include_expired:
            params["exclude"] = "expired"
 
        resp = requests.get(self.CRT_SH_URL, params=params, timeout=30)
        if resp.status_code == 200:
            certs = resp.json()
            print(f"[+] crt.sh: {len(certs)} certificates for *.{domain}")
            return certs
        return []
 
    def find_suspicious_certs(self, domain):
        """Find certificates that may be phishing attempts."""
        certs = self.query_crt_sh(domain)
        suspicious = []
 
        for cert in certs:
            common_name = cert.get("common_name", "").lower()
            name_value = cert.get("name_value", "").lower()
            issuer = cert.get("issuer_name", "")
            not_before = cert.get("not_before", "")
            not_after = cert.get("not_after", "")
 
            # Check for exact domain matches (legitimate)
            extracted = tldextract.extract(common_name)
            cert_domain = f"{extracted.domain}.{extracted.suffix}"
            if cert_domain == domain:
                continue  # Legitimate certificate
 
            # Flag suspicious patterns
            flags = []
            if domain.replace(".", "") in common_name.replace(".", ""):
                flags.append("contains target domain string")
            if any(kw in common_name for kw in self.brand_keywords):
                flags.append("contains brand keyword")
            if "let's encrypt" in issuer.lower():
                flags.append("free CA (Let's Encrypt)")
 
            if flags:
                suspicious.append({
                    "common_name": cert.get("common_name", ""),
                    "name_value": cert.get("name_value", ""),
                    "issuer": issuer,
                    "not_before": not_before,
                    "not_after": not_after,
                    "serial": cert.get("serial_number", ""),
                    "flags": flags,
                    "crt_sh_id": cert.get("id", ""),
                    "crt_sh_url": f"https://crt.sh/?id={cert.get('id', '')}",
                })
 
        print(f"[+] Found {len(suspicious)} suspicious certificates")
        return suspicious
 
monitor = CTLogMonitor(
    monitored_domains=["mycompany.com", "mycompany.org"],
    brand_keywords=["mycompany", "mybrand", "myproduct"],
)
suspicious = monitor.find_suspicious_certs("mycompany.com")
for cert in suspicious[:5]:
    print(f"  [{cert['common_name']}] Flags: {cert['flags']}")

Step 2: Real-Time Monitoring with Certstream

import certstream
import Levenshtein
import re
from datetime import datetime
 
class CertstreamMonitor:
    def __init__(self, watched_domains, brand_keywords, similarity_threshold=0.8):
        self.watched_domains = [d.lower() for d in watched_domains]
        self.brand_keywords = [k.lower() for k in brand_keywords]
        self.threshold = similarity_threshold
        self.alerts = []
 
    def start_monitoring(self, max_alerts=100):
        """Start real-time CT log monitoring."""
        print("[*] Starting Certstream monitoring...")
        print(f"    Watching: {self.watched_domains}")
        print(f"    Keywords: {self.brand_keywords}")
 
        def callback(message, context):
            if message["message_type"] == "certificate_update":
                data = message["data"]
                leaf = data.get("leaf_cert", {})
                all_domains = leaf.get("all_domains", [])
 
                for domain in all_domains:
                    domain_lower = domain.lower().strip("*.")
                    if self._is_suspicious(domain_lower):
                        alert = {
                            "domain": domain,
                            "all_domains": all_domains,
                            "issuer": leaf.get("issuer", {}).get("O", ""),
                            "fingerprint": leaf.get("fingerprint", ""),
                            "not_before": leaf.get("not_before", ""),
                            "detected_at": datetime.now().isoformat(),
                            "reason": self._get_reason(domain_lower),
                        }
                        self.alerts.append(alert)
                        print(f"  [ALERT] {domain} - {alert['reason']}")
 
                        if len(self.alerts) >= max_alerts:
                            raise KeyboardInterrupt
 
        try:
            certstream.listen_for_events(callback, url="wss://certstream.calidog.io/")
        except KeyboardInterrupt:
            print(f"\n[+] Monitoring stopped. {len(self.alerts)} alerts collected.")
        return self.alerts
 
    def _is_suspicious(self, domain):
        """Check if domain is suspicious relative to watched domains."""
        for watched in self.watched_domains:
            # Exact keyword match
            watched_base = watched.split(".")[0]
            if watched_base in domain and domain != watched:
                return True
 
            # Levenshtein distance (typosquatting detection)
            domain_base = tldextract.extract(domain).domain
            similarity = Levenshtein.ratio(watched_base, domain_base)
            if similarity >= self.threshold and domain_base != watched_base:
                return True
 
        # Brand keyword match
        for keyword in self.brand_keywords:
            if keyword in domain:
                return True
 
        return False
 
    def _get_reason(self, domain):
        """Determine why domain was flagged."""
        reasons = []
        for watched in self.watched_domains:
            watched_base = watched.split(".")[0]
            if watched_base in domain:
                reasons.append(f"contains '{watched_base}'")
            domain_base = tldextract.extract(domain).domain
            similarity = Levenshtein.ratio(watched_base, domain_base)
            if similarity >= self.threshold and domain_base != watched_base:
                reasons.append(f"similar to '{watched}' ({similarity:.0%})")
        for kw in self.brand_keywords:
            if kw in domain:
                reasons.append(f"brand keyword '{kw}'")
        return "; ".join(reasons) if reasons else "unknown"
 
cs_monitor = CertstreamMonitor(
    watched_domains=["mycompany.com"],
    brand_keywords=["mycompany", "mybrand"],
    similarity_threshold=0.75,
)
alerts = cs_monitor.start_monitoring(max_alerts=50)

Step 3: Enumerate Subdomains from CT Logs

def enumerate_subdomains_ct(domain):
    """Discover all subdomains from Certificate Transparency logs."""
    params = {"q": f"%.{domain}", "output": "json"}
    resp = requests.get("https://crt.sh", params=params, timeout=30)
 
    if resp.status_code != 200:
        return []
 
    certs = resp.json()
    subdomains = set()
    for cert in certs:
        name_value = cert.get("name_value", "")
        for name in name_value.split("\n"):
            name = name.strip().lower()
            if name.endswith(f".{domain}") or name == domain:
                name = name.lstrip("*.")
                subdomains.add(name)
 
    sorted_subs = sorted(subdomains)
    print(f"[+] CT subdomain enumeration for {domain}: {len(sorted_subs)} subdomains")
    return sorted_subs
 
subdomains = enumerate_subdomains_ct("example.com")
for sub in subdomains[:20]:
    print(f"  {sub}")

Step 4: Generate CT Intelligence Report

def generate_ct_report(suspicious_certs, certstream_alerts, domain):
    report = f"""# Certificate Transparency Intelligence Report
## Target Domain: {domain}
## Generated: {datetime.now().isoformat()}
 
## Summary
- Suspicious certificates found: {len(suspicious_certs)}
- Real-time alerts triggered: {len(certstream_alerts)}
 
## Suspicious Certificates (crt.sh)
| Common Name | Issuer | Flags | crt.sh Link |
|------------|--------|-------|-------------|
"""
    for cert in suspicious_certs[:20]:
        flags = "; ".join(cert.get("flags", []))
        report += (f"| {cert['common_name']} | {cert['issuer'][:30]} "
                   f"| {flags} | [View]({cert['crt_sh_url']}) |\n")
 
    report += f"""
## Real-Time Certstream Alerts
| Domain | Issuer | Reason | Detected |
|--------|--------|--------|----------|
"""
    for alert in certstream_alerts[:20]:
        report += (f"| {alert['domain']} | {alert['issuer']} "
                   f"| {alert['reason']} | {alert['detected_at'][:19]} |\n")
 
    report += """
## Recommendations
1. Add flagged domains to DNS sinkhole / web proxy blocklist
2. Submit takedown requests for confirmed phishing domains
3. Monitor CT logs continuously for new certificate registrations
4. Implement CAA DNS records to restrict certificate issuance for your domains
5. Deploy DMARC to prevent email spoofing from lookalike domains
"""
    with open(f"ct_report_{domain.replace('.','_')}.md", "w") as f:
        f.write(report)
    print(f"[+] CT report saved")
    return report
 
generate_ct_report(suspicious, alerts if 'alerts' in dir() else [], "mycompany.com")

Validation Criteria

  • crt.sh queries return certificate data for target domains
  • Suspicious certificates identified based on lookalike patterns
  • Certstream real-time monitoring detects new phishing certificates
  • Subdomain enumeration produces comprehensive list from CT logs
  • Alerts generated with reason classification
  • CT intelligence report created with actionable recommendations

References

Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 1

api-reference.md2.8 KB

API Reference: Certificate Transparency Phishing Detection

crt.sh API

Search Certificates

# JSON output
curl "https://crt.sh/?q=%.example.com&output=json"
 
# Exclude expired
curl "https://crt.sh/?q=%.example.com&output=json&exclude=expired"
 
# Exact match
curl "https://crt.sh/?q=example.com&output=json"

Response Fields

Field Description
id Certificate ID in crt.sh database
common_name Certificate CN
name_value All SANs (newline-separated)
issuer_name Certificate Authority
not_before Validity start
not_after Validity end
serial_number Certificate serial

Certstream - Real-time CT Monitoring

Python Client

import certstream
 
def callback(message, context):
    if message["message_type"] == "certificate_update":
        data = message["data"]
        domains = data["leaf_cert"]["all_domains"]
        for domain in domains:
            if "example" in domain:
                print(f"[ALERT] {domain}")
 
certstream.listen_for_events(callback, url="wss://certstream.calidog.io/")

Message Fields

Field Path
Domains data.leaf_cert.all_domains
Issuer data.leaf_cert.issuer.O
Subject data.leaf_cert.subject.CN
Fingerprint data.leaf_cert.fingerprint
Source data.source.name

CT Log Servers

Log Operator URL
Argon Google ct.googleapis.com/logs/argon2024
Xenon Google ct.googleapis.com/logs/xenon2024
Nimbus Cloudflare ct.cloudflare.com/logs/nimbus2024
Oak Let's Encrypt oak.ct.letsencrypt.org/2024h1
Yeti DigiCert yeti2024.ct.digicert.com/log

Phishing Detection Techniques

Homoglyph / IDN Attacks

Original Lookalike Technique
example.com examp1e.com Character substitution (l→1)
google.com gооgle.com Cyrillic о (U+043E)
paypal.com paypa1.com l→1 substitution
microsoft.com mіcrosoft.com Cyrillic і (U+0456)

dnstwist Integration

dnstwist -r -f json example.com   # Generate and resolve permutations
dnstwist -w wordlist.txt example.com  # Dictionary-based

Certificate Details Lookup

# Get full certificate from crt.sh
curl "https://crt.sh/?d=<cert_id>"
 
# OpenSSL inspection
openssl s_client -connect domain.com:443 -servername domain.com </dev/null 2>/dev/null | \
  openssl x509 -noout -text

Suspicious Indicators

Pattern Risk Level
Free CA + new domain + brand keyword HIGH
Wildcard cert on recently registered domain HIGH
Multiple certs for slight domain variants MEDIUM
IDN/punycode domain mimicking brand HIGH
Cert issued same day as domain registration MEDIUM

Scripts 1

agent.py7.3 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""Certificate Transparency monitoring agent for phishing detection.

Queries crt.sh for certificates matching target domains, detects lookalike
certificates, and identifies potential phishing infrastructure.
"""

import json
import sys
from collections import defaultdict

try:
    import requests
    HAS_REQUESTS = True
except ImportError:
    HAS_REQUESTS = False


def query_crtsh(domain, wildcard=True, expired=False):
    """Query crt.sh for certificates matching a domain."""
    if not HAS_REQUESTS:
        return []
    query = f"%.{domain}" if wildcard else domain
    params = {"q": query, "output": "json"}
    if not expired:
        params["exclude"] = "expired"
    try:
        resp = requests.get("https://crt.sh/", params=params, timeout=30)
        resp.raise_for_status()
        return resp.json()
    except (requests.RequestException, json.JSONDecodeError) as e:
        return [{"error": str(e)}]


def find_lookalike_domains(target_domain, ct_results):
    """Identify certificates for domains that look similar to the target."""
    base = target_domain.split(".")[0].lower()
    lookalikes = []
    for cert in ct_results:
        cn = cert.get("common_name", "").lower()
        names = cert.get("name_value", "").lower().split("\n")
        for name in [cn] + names:
            name = name.strip()
            if not name or name == target_domain:
                continue
            similarity = calculate_similarity(base, name.split(".")[0])
            if similarity > 0.6 and name != target_domain:
                lookalikes.append({
                    "domain": name,
                    "similarity": round(similarity, 3),
                    "issuer": cert.get("issuer_name", ""),
                    "not_before": cert.get("not_before", ""),
                    "not_after": cert.get("not_after", ""),
                    "cert_id": cert.get("id"),
                })
    seen = set()
    unique = []
    for l in sorted(lookalikes, key=lambda x: -x["similarity"]):
        if l["domain"] not in seen:
            seen.add(l["domain"])
            unique.append(l)
    return unique


def calculate_similarity(s1, s2):
    """Calculate string similarity using Levenshtein-like ratio."""
    if s1 == s2:
        return 1.0
    len1, len2 = len(s1), len(s2)
    if len1 == 0 or len2 == 0:
        return 0.0
    matrix = [[0] * (len2 + 1) for _ in range(len1 + 1)]
    for i in range(len1 + 1):
        matrix[i][0] = i
    for j in range(len2 + 1):
        matrix[0][j] = j
    for i in range(1, len1 + 1):
        for j in range(1, len2 + 1):
            cost = 0 if s1[i-1] == s2[j-1] else 1
            matrix[i][j] = min(matrix[i-1][j] + 1, matrix[i][j-1] + 1,
                               matrix[i-1][j-1] + cost)
    distance = matrix[len1][len2]
    return 1.0 - distance / max(len1, len2)


HOMOGLYPH_MAP = {
    "a": ["а", "@", "4"], "e": ["е", "3"], "o": ["о", "0"],
    "i": ["і", "1", "l"], "l": ["1", "i", "I"],
    "s": ["5", "$"], "t": ["7"], "g": ["9", "q"],
}


def detect_homoglyph_domains(target_domain, ct_results):
    """Detect domains using homoglyph/IDN attacks against target."""
    findings = []
    base = target_domain.split(".")[0].lower()
    for cert in ct_results:
        names = cert.get("name_value", "").lower().split("\n")
        for name in names:
            name = name.strip()
            if not name or name == target_domain:
                continue
            name_base = name.split(".")[0]
            if len(name_base) == len(base):
                diffs = sum(1 for a, b in zip(base, name_base) if a != b)
                if 0 < diffs <= 2:
                    findings.append({
                        "domain": name,
                        "char_differences": diffs,
                        "cert_id": cert.get("id"),
                        "issuer": cert.get("issuer_name", ""),
                    })
    return findings


def analyze_issuer_patterns(ct_results):
    """Analyze certificate issuer patterns for anomalies."""
    issuer_counts = defaultdict(int)
    free_cas = ["Let's Encrypt", "ZeroSSL", "Buypass"]
    for cert in ct_results:
        issuer = cert.get("issuer_name", "Unknown")
        issuer_counts[issuer] += 1
    free_ca_certs = sum(
        count for issuer, count in issuer_counts.items()
        if any(ca.lower() in issuer.lower() for ca in free_cas)
    )
    return {
        "issuers": dict(issuer_counts),
        "total_certs": len(ct_results),
        "free_ca_count": free_ca_certs,
        "free_ca_ratio": round(free_ca_certs / max(len(ct_results), 1), 3),
    }


def detect_wildcard_abuse(ct_results):
    """Detect suspicious wildcard certificate patterns."""
    wildcards = []
    for cert in ct_results:
        cn = cert.get("common_name", "")
        if cn.startswith("*."):
            wildcards.append({
                "domain": cn,
                "issuer": cert.get("issuer_name", ""),
                "not_before": cert.get("not_before", ""),
            })
    return wildcards


def generate_report(target_domain, ct_results):
    """Generate comprehensive CT monitoring report."""
    lookalikes = find_lookalike_domains(target_domain, ct_results)
    homoglyphs = detect_homoglyph_domains(target_domain, ct_results)
    issuer_analysis = analyze_issuer_patterns(ct_results)
    wildcards = detect_wildcard_abuse(ct_results)

    risk_score = 0
    risk_score += min(len(lookalikes) * 10, 40)
    risk_score += min(len(homoglyphs) * 15, 30)
    risk_score += 20 if issuer_analysis["free_ca_ratio"] > 0.8 else 0
    risk_score = min(risk_score, 100)

    return {
        "target_domain": target_domain,
        "total_certificates": len(ct_results),
        "lookalike_domains": lookalikes[:20],
        "homoglyph_domains": homoglyphs[:20],
        "issuer_analysis": issuer_analysis,
        "wildcard_certs": wildcards[:10],
        "risk_score": risk_score,
        "risk_level": "HIGH" if risk_score >= 60 else "MEDIUM" if risk_score >= 30 else "LOW",
    }


if __name__ == "__main__":
    print("=" * 60)
    print("Certificate Transparency Phishing Detection Agent")
    print("crt.sh queries, lookalike detection, homoglyph analysis")
    print("=" * 60)

    domain = sys.argv[1] if len(sys.argv) > 1 else None

    if not domain:
        print("\n[DEMO] Usage: python agent.py <target_domain>")
        print("  e.g. python agent.py example.com")
        sys.exit(0)

    if not HAS_REQUESTS:
        print("[!] Install requests: pip install requests")
        sys.exit(1)

    print(f"\n[*] Querying crt.sh for: {domain}")
    results = query_crtsh(domain)
    print(f"[*] Found {len(results)} certificates")

    report = generate_report(domain, results)

    print(f"\n--- Lookalike Domains ({len(report['lookalike_domains'])}) ---")
    for l in report["lookalike_domains"][:10]:
        print(f"  [{l['similarity']:.3f}] {l['domain']} (issuer: {l['issuer'][:40]})")

    print(f"\n--- Homoglyph Domains ({len(report['homoglyph_domains'])}) ---")
    for h in report["homoglyph_domains"][:10]:
        print(f"  [diff={h['char_differences']}] {h['domain']}")

    print(f"\n--- Issuer Analysis ---")
    for issuer, count in sorted(report["issuer_analysis"]["issuers"].items(),
                                 key=lambda x: -x[1])[:5]:
        print(f"  {count:4d} | {issuer[:60]}")

    print(f"\n[*] Risk Score: {report['risk_score']}/100 ({report['risk_level']})")
Keep exploring