devsecops

Implementing Semgrep for Custom SAST Rules

Write custom Semgrep SAST rules in YAML to detect application-specific vulnerabilities, enforce coding standards, and integrate into CI/CD pipelines.

code-securitycustom-rulesdevsecopssastsemgrepstatic-analysis
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

Overview

Semgrep is an open-source static analysis tool that uses pattern-matching to find bugs, enforce code standards, and detect security vulnerabilities. Custom rules are written in YAML using Semgrep's pattern syntax, making it accessible without requiring compiler knowledge. It supports 30+ languages including Python, JavaScript, Go, Java, and C.

When to Use

  • When deploying or configuring implementing semgrep for custom sast rules capabilities in your environment
  • When establishing security controls aligned to compliance requirements
  • When building or improving security architecture for this domain
  • When conducting security assessments that require this implementation

Prerequisites

  • Python 3.8+ or Docker
  • Semgrep CLI installed
  • Target codebase in a supported language

Installation

# Install via pip
pip install semgrep
 
# Install via Homebrew
brew install semgrep
 
# Run via Docker
docker run -v "${PWD}:/src" returntocorp/semgrep semgrep --config auto /src
 
# Verify
semgrep --version

Running Semgrep

# Auto-detect rules for your code
semgrep --config auto .
 
# Use Semgrep registry rules
semgrep --config r/python.lang.security
 
# Use custom rule file
semgrep --config my-rules.yaml .
 
# Use multiple configs
semgrep --config auto --config ./custom-rules/ .
 
# JSON output
semgrep --config auto --json . > results.json
 
# SARIF output for GitHub
semgrep --config auto --sarif . > results.sarif
 
# Filter by severity
semgrep --config auto --severity ERROR .

Writing Custom Rules

Basic Pattern Matching

# rules/sql-injection.yaml
rules:
  - id: sql-injection-string-format
    languages: [python]
    severity: ERROR
    message: |
      Potential SQL injection via string formatting.
      Use parameterized queries instead.
    pattern: |
      cursor.execute(f"..." % ...)
    metadata:
      cwe: ["CWE-89"]
      owasp: ["A03:2021"]
      category: security
    fix: |
      cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,))

Pattern Operators

rules:
  - id: hardcoded-secret-in-code
    languages: [python, javascript, typescript]
    severity: ERROR
    message: Hardcoded secret detected in source code
    patterns:
      - pattern-either:
          - pattern: $VAR = "..."
          - pattern: $VAR = '...'
      - metavariable-regex:
          metavariable: $VAR
          regex: (?i)(password|secret|api_key|token|aws_secret)
      - pattern-not: $VAR = ""
      - pattern-not: $VAR = "changeme"
      - pattern-not: $VAR = "PLACEHOLDER"
    metadata:
      cwe: ["CWE-798"]
      category: security

Taint Analysis

rules:
  - id: xss-taint-tracking
    languages: [python]
    severity: ERROR
    message: User input flows to HTML response without sanitization
    mode: taint
    pattern-sources:
      - pattern: request.args.get(...)
      - pattern: request.form.get(...)
      - pattern: request.form[...]
    pattern-sinks:
      - pattern: return render_template_string(...)
      - pattern: Markup(...)
    pattern-sanitizers:
      - pattern: bleach.clean(...)
      - pattern: escape(...)
    metadata:
      cwe: ["CWE-79"]
      owasp: ["A03:2021"]

Multiple Language Rule

rules:
  - id: insecure-random
    languages: [python, javascript, go, java]
    severity: WARNING
    message: |
      Using insecure random number generator. Use cryptographically
      secure alternatives for security-sensitive operations.
    pattern-either:
      # Python
      - pattern: random.random()
      - pattern: random.randint(...)
      # JavaScript
      - pattern: Math.random()
      # Go
      - pattern: math/rand.Intn(...)
      # Java
      - pattern: new java.util.Random()
    metadata:
      cwe: ["CWE-330"]

Enforce Coding Standards

rules:
  - id: require-error-handling
    languages: [go]
    severity: WARNING
    message: Error return value not checked
    pattern: |
      $VAR, _ := $FUNC(...)
    fix: |
      $VAR, err := $FUNC(...)
      if err != nil {
        return fmt.Errorf("$FUNC failed: %w", err)
      }
 
  - id: no-console-log-in-production
    languages: [javascript, typescript]
    severity: WARNING
    message: Remove console.log before merging to production
    pattern: console.log(...)
    paths:
      exclude:
        - "tests/*"
        - "*.test.*"

JWT Security Rules

rules:
  - id: jwt-none-algorithm
    languages: [python]
    severity: ERROR
    message: JWT decoded without algorithm verification - allows token forgery
    patterns:
      - pattern: jwt.decode($TOKEN, ..., algorithms=["none"], ...)
    metadata:
      cwe: ["CWE-347"]
 
  - id: jwt-no-verification
    languages: [python]
    severity: ERROR
    message: JWT decoded with verification disabled
    patterns:
      - pattern: jwt.decode($TOKEN, ..., options={"verify_signature": False}, ...)
    metadata:
      cwe: ["CWE-345"]

Rule Testing

# rules/test-sql-injection.yaml
rules:
  - id: sql-injection-format-string
    languages: [python]
    severity: ERROR
    message: SQL injection via format string
    pattern: |
      cursor.execute(f"...{$VAR}...")
 
# Test annotation in test file:
# test-sql-injection.py
def bad_query(user_id):
    # ruleid: sql-injection-format-string
    cursor.execute(f"SELECT * FROM users WHERE id = {user_id}")
 
def good_query(user_id):
    # ok: sql-injection-format-string
    cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,))
# Run rule tests
semgrep --test rules/
 
# Test specific rule
semgrep --config rules/sql-injection.yaml --test

CI/CD Integration

GitHub Actions

name: Semgrep SAST
on: [pull_request]
 
jobs:
  semgrep:
    runs-on: ubuntu-latest
    container:
      image: returntocorp/semgrep
    steps:
      - uses: actions/checkout@v4
 
      - name: Run Semgrep
        run: |
          semgrep --config auto \
            --config ./custom-rules/ \
            --sarif --output results.sarif \
            --severity ERROR \
            .
 
      - name: Upload SARIF
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: results.sarif

GitLab CI

semgrep:
  stage: test
  image: returntocorp/semgrep
  script:
    - semgrep --config auto --config ./custom-rules/ --json --output semgrep.json .
  artifacts:
    reports:
      sast: semgrep.json

Configuration File

# .semgrep.yaml
rules:
  - id: my-org-rules
    # ... rules here
 
# .semgrepignore
tests/
node_modules/
vendor/
*.min.js

Best Practices

  1. Start with auto config then add custom rules for org-specific patterns
  2. Test rules with # ruleid: and # ok: annotations
  3. Use taint mode for data flow vulnerabilities (XSS, SQLi, SSRF)
  4. Include metadata (CWE, OWASP) for vulnerability classification
  5. Provide fix suggestions with the fix key where possible
  6. Exclude test files to reduce false positives
  7. Version control rules in a shared repository
  8. Run in CI as a blocking check for ERROR severity findings
Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 2

api-reference.md5.1 KB

API Reference: Semgrep Custom SAST Rules

Libraries Used

Library Purpose
subprocess Execute semgrep CLI scans
json Parse semgrep JSON output
yaml Read and write custom Semgrep rule files
pathlib Handle source code and rule file paths

Installation

# Python package
pip install semgrep
 
# Homebrew (macOS)
brew install semgrep
 
# Docker
docker pull semgrep/semgrep:latest

CLI Reference

Core Commands

# Scan with auto-detected rules
semgrep scan --config auto --json --output results.json /path/to/code
 
# Scan with specific rulesets from Semgrep Registry
semgrep scan --config p/python --config p/owasp-top-ten /path/to/code
 
# Scan with a custom rule file
semgrep scan --config my-rules.yaml /path/to/code
 
# Scan with multiple configs
semgrep scan --config p/security-audit --config ./custom-rules/ /path/to/code

Key CLI Flags

Flag Description
--config, -c Rule source: registry key, YAML file, or directory
--json Output results in JSON format
--sarif Output in SARIF format (for CI/CD integration)
--output, -o Write results to file
--severity Filter by severity: INFO, WARNING, ERROR
--include Only scan files matching glob pattern
--exclude Skip files matching glob pattern
--lang Restrict scan to specific language
--max-target-bytes Skip files larger than N bytes
--timeout Per-rule timeout in seconds (default: 5)
--jobs, -j Number of parallel jobs
--verbose, -v Show detailed scan progress
--metrics off Disable anonymous metrics

Custom Rule Syntax

Basic Pattern Rule

rules:
  - id: hardcoded-password
    pattern: password = "..."
    message: "Hardcoded password detected — use environment variables"
    languages: [python]
    severity: ERROR
    metadata:
      cwe: ["CWE-798: Use of Hard-coded Credentials"]
      owasp: ["A07:2021 - Identification and Authentication Failures"]

Pattern Operators

rules:
  - id: sql-injection-format-string
    patterns:
      - pattern: |
          cursor.execute($QUERY % ...)
      - pattern-not: |
          cursor.execute("..." % ())
    message: "SQL injection via string formatting — use parameterized queries"
    languages: [python]
    severity: ERROR
 
  - id: unsafe-deserialization
    pattern-either:
      - pattern: pickle.loads(...)
      - pattern: pickle.load(...)
      - pattern: yaml.load(..., Loader=yaml.Loader)
      - pattern: yaml.unsafe_load(...)
    message: "Unsafe deserialization — may allow remote code execution"
    languages: [python]
    severity: ERROR
 
  - id: missing-timeout-requests
    patterns:
      - pattern: requests.$METHOD(...)
      - pattern-not: requests.$METHOD(..., timeout=..., ...)
    message: "HTTP request without timeout — may hang indefinitely"
    languages: [python]
    severity: WARNING

Metavariable Patterns

rules:
  - id: eval-user-input
    patterns:
      - pattern: |
          $INPUT = request.$METHOD(...)
          ...
          eval($INPUT)
    message: "User input passed to eval() — command injection risk"
    languages: [python]
    severity: ERROR

Python Integration

import subprocess
import json
 
def run_semgrep(target_path, config="auto", severity=None):
    cmd = [
        "semgrep", "scan",
        "--config", config,
        "--json",
        "--metrics", "off",
        str(target_path),
    ]
    if severity:
        cmd.extend(["--severity", severity])
 
    result = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
    output = json.loads(result.stdout)
    return output.get("results", [])
 
def summarize_findings(results):
    by_severity = {"ERROR": [], "WARNING": [], "INFO": []}
    for r in results:
        sev = r.get("extra", {}).get("severity", "INFO")
        by_severity[sev].append({
            "rule": r["check_id"],
            "file": r["path"],
            "line": r["start"]["line"],
            "message": r["extra"]["message"],
        })
    return by_severity

Semgrep Registry Rule Packs

Pack Description
p/python Python-specific security and correctness rules
p/javascript JavaScript/TypeScript rules
p/owasp-top-ten OWASP Top 10 vulnerability patterns
p/security-audit Broad security audit rules across languages
p/secrets Secret and credential detection
p/ci Rules optimized for CI/CD pipelines
p/docker Dockerfile security best practices
p/terraform Terraform IaC security rules

Output Format

{
  "results": [
    {
      "check_id": "python.lang.security.audit.eval-detected",
      "path": "app/views.py",
      "start": {"line": 42, "col": 5},
      "end": {"line": 42, "col": 28},
      "extra": {
        "message": "Detected eval() usage — avoid with untrusted input",
        "severity": "ERROR",
        "metadata": {
          "cwe": ["CWE-95"],
          "owasp": ["A03:2021 - Injection"]
        }
      }
    }
  ],
  "errors": [],
  "stats": {
    "findings": 3,
    "errors": 0,
    "total_time": 2.45
  }
}
standards.md1.1 KB

Standards - Semgrep Custom SAST Rules

OWASP Top 10 (2021) Coverage

Category Semgrep Detection
A01 Broken Access Control Authorization bypass patterns
A02 Cryptographic Failures Weak crypto, hardcoded secrets
A03 Injection SQL, XSS, command injection (taint mode)
A04 Insecure Design Missing input validation
A05 Security Misconfiguration Debug mode, insecure defaults
A06 Vulnerable Components Deprecated API usage
A07 Auth Failures JWT misconfig, session issues
A08 Software/Data Integrity Deserialization, unsigned data
A09 Logging Failures Missing audit logging
A10 SSRF Server-side request forgery (taint mode)

CWE Coverage

Common CWEs detectable via Semgrep custom rules: CWE-79 (XSS), CWE-89 (SQLi), CWE-798 (Hardcoded Credentials), CWE-330 (Insecure Random), CWE-502 (Deserialization), CWE-918 (SSRF)

NIST SP 800-53 Rev 5

  • SA-11: Developer Security Testing
  • SA-15: Development Process, Standards, and Tools

Compliance

  • PCI DSS v4.0 Req 6.3.2: Secure development with automated tools
  • SOC 2 CC8.1: Change management with code scanning

Scripts 1

agent.py7.6 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""Semgrep SAST scanning agent.

Wraps the Semgrep CLI to perform static application security testing
using built-in rulesets and custom rules. Parses JSON output to produce
structured vulnerability findings with severity, CWE, and OWASP mappings.
"""
import argparse
import json
import os
import subprocess
import sys
from datetime import datetime, timezone


def find_semgrep_binary():
    """Locate the semgrep binary on the system."""
    custom_path = os.environ.get("SEMGREP_PATH")
    if custom_path and os.path.isfile(custom_path):
        return custom_path
    for name in ["semgrep", "semgrep.exe"]:
        for directory in os.environ.get("PATH", "").split(os.pathsep):
            full_path = os.path.join(directory, name)
            if os.path.isfile(full_path):
                return full_path
    print("[!] semgrep not found. Install: pip install semgrep", file=sys.stderr)
    sys.exit(1)


def run_scan(semgrep_bin, target, configs=None, severity=None,
             exclude=None, include=None, max_target_bytes=None,
             timeout_per_rule=None, verbose=False):
    """Run semgrep scan and return JSON results."""
    cmd = [semgrep_bin, "scan", "--json"]

    if configs:
        for cfg in configs:
            cmd.extend(["--config", cfg])
    else:
        cmd.extend(["--config", "auto"])

    if severity:
        cmd.extend(["--severity", severity])
    if exclude:
        for pattern in exclude:
            cmd.extend(["--exclude", pattern])
    if include:
        for pattern in include:
            cmd.extend(["--include", pattern])
    if max_target_bytes:
        cmd.extend(["--max-target-bytes", str(max_target_bytes)])
    if timeout_per_rule:
        cmd.extend(["--timeout", str(timeout_per_rule)])
    if verbose:
        cmd.append("--verbose")

    cmd.append(target)

    print(f"[*] Running: {' '.join(cmd)}")
    result = subprocess.run(
        cmd,
        capture_output=True,
        text=True,
        timeout=900,
    )
    return result.stdout, result.stderr, result.returncode


def parse_findings(raw_json):
    """Parse semgrep JSON output into structured findings."""
    findings = []
    if not raw_json:
        return findings, {}
    try:
        data = json.loads(raw_json)
    except json.JSONDecodeError:
        return findings, {}

    errors = data.get("errors", [])
    for result in data.get("results", []):
        metadata = result.get("extra", {}).get("metadata", {})
        finding = {
            "rule_id": result.get("check_id", "unknown"),
            "message": result.get("extra", {}).get("message", ""),
            "severity": result.get("extra", {}).get("severity", "WARNING"),
            "path": result.get("path", ""),
            "start_line": result.get("start", {}).get("line", 0),
            "end_line": result.get("end", {}).get("line", 0),
            "matched_code": result.get("extra", {}).get("lines", ""),
            "fix": result.get("extra", {}).get("fix", ""),
            "cwe": metadata.get("cwe", []),
            "owasp": metadata.get("owasp", []),
            "confidence": metadata.get("confidence", ""),
            "references": metadata.get("references", []),
            "category": metadata.get("category", ""),
            "technology": metadata.get("technology", []),
        }
        findings.append(finding)

    stats = {
        "total_findings": len(findings),
        "files_scanned": data.get("paths", {}).get("scanned", []),
        "files_scanned_count": len(data.get("paths", {}).get("scanned", [])),
        "errors": len(errors),
        "parse_errors": [e.get("message", "") for e in errors[:5]],
    }
    return findings, stats


def format_summary(findings, stats, target):
    """Print human-readable scan summary."""
    print(f"\n{'='*60}")
    print(f"  Semgrep SAST Scan Report")
    print(f"{'='*60}")
    print(f"  Target       : {target}")
    print(f"  Files Scanned: {stats.get('files_scanned_count', 0)}")
    print(f"  Findings     : {len(findings)}")
    print(f"  Parse Errors : {stats.get('errors', 0)}")

    severity_counts = {}
    for f in findings:
        sev = f.get("severity", "WARNING")
        severity_counts[sev] = severity_counts.get(sev, 0) + 1

    print(f"\n  By Severity:")
    for sev in ["ERROR", "WARNING", "INFO"]:
        count = severity_counts.get(sev, 0)
        if count > 0:
            print(f"    {sev:10s}: {count}")

    by_rule = {}
    for f in findings:
        by_rule.setdefault(f["rule_id"], []).append(f)

    print(f"\n  Top Rules ({len(by_rule)} unique):")
    for rule, items in sorted(by_rule.items(), key=lambda x: -len(x[1]))[:10]:
        short_rule = rule.split(".")[-1] if "." in rule else rule
        print(f"    {short_rule:45s}: {len(items)} hit(s)")

    by_file = {}
    for f in findings:
        by_file.setdefault(f["path"], []).append(f)

    print(f"\n  Most Affected Files ({len(by_file)} files):")
    for filepath, items in sorted(by_file.items(), key=lambda x: -len(x[1]))[:10]:
        print(f"    {filepath:50s}: {len(items)} finding(s)")

    if findings:
        print(f"\n  Critical/Error Findings:")
        for f in findings[:15]:
            if f["severity"] == "ERROR":
                cwe = f["cwe"][0] if f["cwe"] else ""
                print(f"    {f['path']}:{f['start_line']} [{cwe}] {f['rule_id']}")

    return severity_counts


def main():
    parser = argparse.ArgumentParser(
        description="Semgrep SAST scanning agent"
    )
    parser.add_argument("--target", required=True,
                        help="Path to source code directory or file to scan")
    parser.add_argument("--config", nargs="+", default=None,
                        help="Semgrep config(s): auto, p/security-audit, p/owasp-top-ten, or path to .yaml")
    parser.add_argument("--severity", choices=["INFO", "WARNING", "ERROR"],
                        help="Minimum severity to report")
    parser.add_argument("--exclude", nargs="+",
                        help="File patterns to exclude (e.g., tests/ vendor/)")
    parser.add_argument("--include", nargs="+",
                        help="File patterns to include (e.g., *.py *.js)")
    parser.add_argument("--timeout-per-rule", type=int, default=30,
                        help="Timeout per rule in seconds (default: 30)")
    parser.add_argument("--output", "-o", help="Output JSON report path")
    parser.add_argument("--verbose", "-v", action="store_true")
    args = parser.parse_args()

    semgrep_bin = find_semgrep_binary()
    print(f"[*] Using semgrep: {semgrep_bin}")

    raw_json, stderr, exit_code = run_scan(
        semgrep_bin, args.target, args.config, args.severity,
        args.exclude, args.include, timeout_per_rule=args.timeout_per_rule,
        verbose=args.verbose
    )

    findings, stats = parse_findings(raw_json)
    severity_counts = format_summary(findings, stats, args.target)

    report = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "tool": "semgrep",
        "target": args.target,
        "configs": args.config or ["auto"],
        "stats": stats,
        "severity_counts": severity_counts,
        "findings_count": len(findings),
        "findings": findings,
        "risk_level": (
            "CRITICAL" if severity_counts.get("ERROR", 0) > 5
            else "HIGH" if severity_counts.get("ERROR", 0) > 0
            else "MEDIUM" if severity_counts.get("WARNING", 0) > 0
            else "LOW"
        ),
    }

    if args.output:
        with open(args.output, "w") as f:
            json.dump(report, f, indent=2)
        print(f"\n[+] Report saved to {args.output}")
    elif args.verbose:
        print(json.dumps(report, indent=2))

    print(f"\n[*] Risk Level: {report['risk_level']}")


if __name__ == "__main__":
    main()
Keep exploring