npx skills add mukul975/Anthropic-Cybersecurity-SkillsMITRE ATT&CK
Overview
Semgrep is an open-source static analysis tool that uses pattern-matching to find bugs, enforce code standards, and detect security vulnerabilities. Custom rules are written in YAML using Semgrep's pattern syntax, making it accessible without requiring compiler knowledge. It supports 30+ languages including Python, JavaScript, Go, Java, and C.
When to Use
- When deploying or configuring implementing semgrep for custom sast rules capabilities in your environment
- When establishing security controls aligned to compliance requirements
- When building or improving security architecture for this domain
- When conducting security assessments that require this implementation
Prerequisites
- Python 3.8+ or Docker
- Semgrep CLI installed
- Target codebase in a supported language
Installation
# Install via pip
pip install semgrep
# Install via Homebrew
brew install semgrep
# Run via Docker
docker run -v "${PWD}:/src" returntocorp/semgrep semgrep --config auto /src
# Verify
semgrep --versionRunning Semgrep
# Auto-detect rules for your code
semgrep --config auto .
# Use Semgrep registry rules
semgrep --config r/python.lang.security
# Use custom rule file
semgrep --config my-rules.yaml .
# Use multiple configs
semgrep --config auto --config ./custom-rules/ .
# JSON output
semgrep --config auto --json . > results.json
# SARIF output for GitHub
semgrep --config auto --sarif . > results.sarif
# Filter by severity
semgrep --config auto --severity ERROR .Writing Custom Rules
Basic Pattern Matching
# rules/sql-injection.yaml
rules:
- id: sql-injection-string-format
languages: [python]
severity: ERROR
message: |
Potential SQL injection via string formatting.
Use parameterized queries instead.
pattern: |
cursor.execute(f"..." % ...)
metadata:
cwe: ["CWE-89"]
owasp: ["A03:2021"]
category: security
fix: |
cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,))Pattern Operators
rules:
- id: hardcoded-secret-in-code
languages: [python, javascript, typescript]
severity: ERROR
message: Hardcoded secret detected in source code
patterns:
- pattern-either:
- pattern: $VAR = "..."
- pattern: $VAR = '...'
- metavariable-regex:
metavariable: $VAR
regex: (?i)(password|secret|api_key|token|aws_secret)
- pattern-not: $VAR = ""
- pattern-not: $VAR = "changeme"
- pattern-not: $VAR = "PLACEHOLDER"
metadata:
cwe: ["CWE-798"]
category: securityTaint Analysis
rules:
- id: xss-taint-tracking
languages: [python]
severity: ERROR
message: User input flows to HTML response without sanitization
mode: taint
pattern-sources:
- pattern: request.args.get(...)
- pattern: request.form.get(...)
- pattern: request.form[...]
pattern-sinks:
- pattern: return render_template_string(...)
- pattern: Markup(...)
pattern-sanitizers:
- pattern: bleach.clean(...)
- pattern: escape(...)
metadata:
cwe: ["CWE-79"]
owasp: ["A03:2021"]Multiple Language Rule
rules:
- id: insecure-random
languages: [python, javascript, go, java]
severity: WARNING
message: |
Using insecure random number generator. Use cryptographically
secure alternatives for security-sensitive operations.
pattern-either:
# Python
- pattern: random.random()
- pattern: random.randint(...)
# JavaScript
- pattern: Math.random()
# Go
- pattern: math/rand.Intn(...)
# Java
- pattern: new java.util.Random()
metadata:
cwe: ["CWE-330"]Enforce Coding Standards
rules:
- id: require-error-handling
languages: [go]
severity: WARNING
message: Error return value not checked
pattern: |
$VAR, _ := $FUNC(...)
fix: |
$VAR, err := $FUNC(...)
if err != nil {
return fmt.Errorf("$FUNC failed: %w", err)
}
- id: no-console-log-in-production
languages: [javascript, typescript]
severity: WARNING
message: Remove console.log before merging to production
pattern: console.log(...)
paths:
exclude:
- "tests/*"
- "*.test.*"JWT Security Rules
rules:
- id: jwt-none-algorithm
languages: [python]
severity: ERROR
message: JWT decoded without algorithm verification - allows token forgery
patterns:
- pattern: jwt.decode($TOKEN, ..., algorithms=["none"], ...)
metadata:
cwe: ["CWE-347"]
- id: jwt-no-verification
languages: [python]
severity: ERROR
message: JWT decoded with verification disabled
patterns:
- pattern: jwt.decode($TOKEN, ..., options={"verify_signature": False}, ...)
metadata:
cwe: ["CWE-345"]Rule Testing
# rules/test-sql-injection.yaml
rules:
- id: sql-injection-format-string
languages: [python]
severity: ERROR
message: SQL injection via format string
pattern: |
cursor.execute(f"...{$VAR}...")
# Test annotation in test file:
# test-sql-injection.py
def bad_query(user_id):
# ruleid: sql-injection-format-string
cursor.execute(f"SELECT * FROM users WHERE id = {user_id}")
def good_query(user_id):
# ok: sql-injection-format-string
cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,))# Run rule tests
semgrep --test rules/
# Test specific rule
semgrep --config rules/sql-injection.yaml --testCI/CD Integration
GitHub Actions
name: Semgrep SAST
on: [pull_request]
jobs:
semgrep:
runs-on: ubuntu-latest
container:
image: returntocorp/semgrep
steps:
- uses: actions/checkout@v4
- name: Run Semgrep
run: |
semgrep --config auto \
--config ./custom-rules/ \
--sarif --output results.sarif \
--severity ERROR \
.
- name: Upload SARIF
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: results.sarifGitLab CI
semgrep:
stage: test
image: returntocorp/semgrep
script:
- semgrep --config auto --config ./custom-rules/ --json --output semgrep.json .
artifacts:
reports:
sast: semgrep.jsonConfiguration File
# .semgrep.yaml
rules:
- id: my-org-rules
# ... rules here
# .semgrepignore
tests/
node_modules/
vendor/
*.min.jsBest Practices
- Start with auto config then add custom rules for org-specific patterns
- Test rules with
# ruleid:and# ok:annotations - Use taint mode for data flow vulnerabilities (XSS, SQLi, SSRF)
- Include metadata (CWE, OWASP) for vulnerability classification
- Provide fix suggestions with the
fixkey where possible - Exclude test files to reduce false positives
- Version control rules in a shared repository
- Run in CI as a blocking check for ERROR severity findings
References and resources
Everything below is rendered for inspection. Script files are read-only and never run.
References 2
api-reference.md5.1 KB
API Reference: Semgrep Custom SAST Rules
Libraries Used
| Library | Purpose |
|---|---|
subprocess |
Execute semgrep CLI scans |
json |
Parse semgrep JSON output |
yaml |
Read and write custom Semgrep rule files |
pathlib |
Handle source code and rule file paths |
Installation
# Python package
pip install semgrep
# Homebrew (macOS)
brew install semgrep
# Docker
docker pull semgrep/semgrep:latestCLI Reference
Core Commands
# Scan with auto-detected rules
semgrep scan --config auto --json --output results.json /path/to/code
# Scan with specific rulesets from Semgrep Registry
semgrep scan --config p/python --config p/owasp-top-ten /path/to/code
# Scan with a custom rule file
semgrep scan --config my-rules.yaml /path/to/code
# Scan with multiple configs
semgrep scan --config p/security-audit --config ./custom-rules/ /path/to/codeKey CLI Flags
| Flag | Description |
|---|---|
--config, -c |
Rule source: registry key, YAML file, or directory |
--json |
Output results in JSON format |
--sarif |
Output in SARIF format (for CI/CD integration) |
--output, -o |
Write results to file |
--severity |
Filter by severity: INFO, WARNING, ERROR |
--include |
Only scan files matching glob pattern |
--exclude |
Skip files matching glob pattern |
--lang |
Restrict scan to specific language |
--max-target-bytes |
Skip files larger than N bytes |
--timeout |
Per-rule timeout in seconds (default: 5) |
--jobs, -j |
Number of parallel jobs |
--verbose, -v |
Show detailed scan progress |
--metrics off |
Disable anonymous metrics |
Custom Rule Syntax
Basic Pattern Rule
rules:
- id: hardcoded-password
pattern: password = "..."
message: "Hardcoded password detected — use environment variables"
languages: [python]
severity: ERROR
metadata:
cwe: ["CWE-798: Use of Hard-coded Credentials"]
owasp: ["A07:2021 - Identification and Authentication Failures"]Pattern Operators
rules:
- id: sql-injection-format-string
patterns:
- pattern: |
cursor.execute($QUERY % ...)
- pattern-not: |
cursor.execute("..." % ())
message: "SQL injection via string formatting — use parameterized queries"
languages: [python]
severity: ERROR
- id: unsafe-deserialization
pattern-either:
- pattern: pickle.loads(...)
- pattern: pickle.load(...)
- pattern: yaml.load(..., Loader=yaml.Loader)
- pattern: yaml.unsafe_load(...)
message: "Unsafe deserialization — may allow remote code execution"
languages: [python]
severity: ERROR
- id: missing-timeout-requests
patterns:
- pattern: requests.$METHOD(...)
- pattern-not: requests.$METHOD(..., timeout=..., ...)
message: "HTTP request without timeout — may hang indefinitely"
languages: [python]
severity: WARNINGMetavariable Patterns
rules:
- id: eval-user-input
patterns:
- pattern: |
$INPUT = request.$METHOD(...)
...
eval($INPUT)
message: "User input passed to eval() — command injection risk"
languages: [python]
severity: ERRORPython Integration
import subprocess
import json
def run_semgrep(target_path, config="auto", severity=None):
cmd = [
"semgrep", "scan",
"--config", config,
"--json",
"--metrics", "off",
str(target_path),
]
if severity:
cmd.extend(["--severity", severity])
result = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
output = json.loads(result.stdout)
return output.get("results", [])
def summarize_findings(results):
by_severity = {"ERROR": [], "WARNING": [], "INFO": []}
for r in results:
sev = r.get("extra", {}).get("severity", "INFO")
by_severity[sev].append({
"rule": r["check_id"],
"file": r["path"],
"line": r["start"]["line"],
"message": r["extra"]["message"],
})
return by_severitySemgrep Registry Rule Packs
| Pack | Description |
|---|---|
p/python |
Python-specific security and correctness rules |
p/javascript |
JavaScript/TypeScript rules |
p/owasp-top-ten |
OWASP Top 10 vulnerability patterns |
p/security-audit |
Broad security audit rules across languages |
p/secrets |
Secret and credential detection |
p/ci |
Rules optimized for CI/CD pipelines |
p/docker |
Dockerfile security best practices |
p/terraform |
Terraform IaC security rules |
Output Format
{
"results": [
{
"check_id": "python.lang.security.audit.eval-detected",
"path": "app/views.py",
"start": {"line": 42, "col": 5},
"end": {"line": 42, "col": 28},
"extra": {
"message": "Detected eval() usage — avoid with untrusted input",
"severity": "ERROR",
"metadata": {
"cwe": ["CWE-95"],
"owasp": ["A03:2021 - Injection"]
}
}
}
],
"errors": [],
"stats": {
"findings": 3,
"errors": 0,
"total_time": 2.45
}
}standards.md1.1 KB
Standards - Semgrep Custom SAST Rules
OWASP Top 10 (2021) Coverage
| Category | Semgrep Detection |
|---|---|
| A01 Broken Access Control | Authorization bypass patterns |
| A02 Cryptographic Failures | Weak crypto, hardcoded secrets |
| A03 Injection | SQL, XSS, command injection (taint mode) |
| A04 Insecure Design | Missing input validation |
| A05 Security Misconfiguration | Debug mode, insecure defaults |
| A06 Vulnerable Components | Deprecated API usage |
| A07 Auth Failures | JWT misconfig, session issues |
| A08 Software/Data Integrity | Deserialization, unsigned data |
| A09 Logging Failures | Missing audit logging |
| A10 SSRF | Server-side request forgery (taint mode) |
CWE Coverage
Common CWEs detectable via Semgrep custom rules: CWE-79 (XSS), CWE-89 (SQLi), CWE-798 (Hardcoded Credentials), CWE-330 (Insecure Random), CWE-502 (Deserialization), CWE-918 (SSRF)
NIST SP 800-53 Rev 5
- SA-11: Developer Security Testing
- SA-15: Development Process, Standards, and Tools
Compliance
- PCI DSS v4.0 Req 6.3.2: Secure development with automated tools
- SOC 2 CC8.1: Change management with code scanning
Scripts 1
agent.py7.6 KB
#!/usr/bin/env python3
"""Semgrep SAST scanning agent.
Wraps the Semgrep CLI to perform static application security testing
using built-in rulesets and custom rules. Parses JSON output to produce
structured vulnerability findings with severity, CWE, and OWASP mappings.
"""
import argparse
import json
import os
import subprocess
import sys
from datetime import datetime, timezone
def find_semgrep_binary():
"""Locate the semgrep binary on the system."""
custom_path = os.environ.get("SEMGREP_PATH")
if custom_path and os.path.isfile(custom_path):
return custom_path
for name in ["semgrep", "semgrep.exe"]:
for directory in os.environ.get("PATH", "").split(os.pathsep):
full_path = os.path.join(directory, name)
if os.path.isfile(full_path):
return full_path
print("[!] semgrep not found. Install: pip install semgrep", file=sys.stderr)
sys.exit(1)
def run_scan(semgrep_bin, target, configs=None, severity=None,
exclude=None, include=None, max_target_bytes=None,
timeout_per_rule=None, verbose=False):
"""Run semgrep scan and return JSON results."""
cmd = [semgrep_bin, "scan", "--json"]
if configs:
for cfg in configs:
cmd.extend(["--config", cfg])
else:
cmd.extend(["--config", "auto"])
if severity:
cmd.extend(["--severity", severity])
if exclude:
for pattern in exclude:
cmd.extend(["--exclude", pattern])
if include:
for pattern in include:
cmd.extend(["--include", pattern])
if max_target_bytes:
cmd.extend(["--max-target-bytes", str(max_target_bytes)])
if timeout_per_rule:
cmd.extend(["--timeout", str(timeout_per_rule)])
if verbose:
cmd.append("--verbose")
cmd.append(target)
print(f"[*] Running: {' '.join(cmd)}")
result = subprocess.run(
cmd,
capture_output=True,
text=True,
timeout=900,
)
return result.stdout, result.stderr, result.returncode
def parse_findings(raw_json):
"""Parse semgrep JSON output into structured findings."""
findings = []
if not raw_json:
return findings, {}
try:
data = json.loads(raw_json)
except json.JSONDecodeError:
return findings, {}
errors = data.get("errors", [])
for result in data.get("results", []):
metadata = result.get("extra", {}).get("metadata", {})
finding = {
"rule_id": result.get("check_id", "unknown"),
"message": result.get("extra", {}).get("message", ""),
"severity": result.get("extra", {}).get("severity", "WARNING"),
"path": result.get("path", ""),
"start_line": result.get("start", {}).get("line", 0),
"end_line": result.get("end", {}).get("line", 0),
"matched_code": result.get("extra", {}).get("lines", ""),
"fix": result.get("extra", {}).get("fix", ""),
"cwe": metadata.get("cwe", []),
"owasp": metadata.get("owasp", []),
"confidence": metadata.get("confidence", ""),
"references": metadata.get("references", []),
"category": metadata.get("category", ""),
"technology": metadata.get("technology", []),
}
findings.append(finding)
stats = {
"total_findings": len(findings),
"files_scanned": data.get("paths", {}).get("scanned", []),
"files_scanned_count": len(data.get("paths", {}).get("scanned", [])),
"errors": len(errors),
"parse_errors": [e.get("message", "") for e in errors[:5]],
}
return findings, stats
def format_summary(findings, stats, target):
"""Print human-readable scan summary."""
print(f"\n{'='*60}")
print(f" Semgrep SAST Scan Report")
print(f"{'='*60}")
print(f" Target : {target}")
print(f" Files Scanned: {stats.get('files_scanned_count', 0)}")
print(f" Findings : {len(findings)}")
print(f" Parse Errors : {stats.get('errors', 0)}")
severity_counts = {}
for f in findings:
sev = f.get("severity", "WARNING")
severity_counts[sev] = severity_counts.get(sev, 0) + 1
print(f"\n By Severity:")
for sev in ["ERROR", "WARNING", "INFO"]:
count = severity_counts.get(sev, 0)
if count > 0:
print(f" {sev:10s}: {count}")
by_rule = {}
for f in findings:
by_rule.setdefault(f["rule_id"], []).append(f)
print(f"\n Top Rules ({len(by_rule)} unique):")
for rule, items in sorted(by_rule.items(), key=lambda x: -len(x[1]))[:10]:
short_rule = rule.split(".")[-1] if "." in rule else rule
print(f" {short_rule:45s}: {len(items)} hit(s)")
by_file = {}
for f in findings:
by_file.setdefault(f["path"], []).append(f)
print(f"\n Most Affected Files ({len(by_file)} files):")
for filepath, items in sorted(by_file.items(), key=lambda x: -len(x[1]))[:10]:
print(f" {filepath:50s}: {len(items)} finding(s)")
if findings:
print(f"\n Critical/Error Findings:")
for f in findings[:15]:
if f["severity"] == "ERROR":
cwe = f["cwe"][0] if f["cwe"] else ""
print(f" {f['path']}:{f['start_line']} [{cwe}] {f['rule_id']}")
return severity_counts
def main():
parser = argparse.ArgumentParser(
description="Semgrep SAST scanning agent"
)
parser.add_argument("--target", required=True,
help="Path to source code directory or file to scan")
parser.add_argument("--config", nargs="+", default=None,
help="Semgrep config(s): auto, p/security-audit, p/owasp-top-ten, or path to .yaml")
parser.add_argument("--severity", choices=["INFO", "WARNING", "ERROR"],
help="Minimum severity to report")
parser.add_argument("--exclude", nargs="+",
help="File patterns to exclude (e.g., tests/ vendor/)")
parser.add_argument("--include", nargs="+",
help="File patterns to include (e.g., *.py *.js)")
parser.add_argument("--timeout-per-rule", type=int, default=30,
help="Timeout per rule in seconds (default: 30)")
parser.add_argument("--output", "-o", help="Output JSON report path")
parser.add_argument("--verbose", "-v", action="store_true")
args = parser.parse_args()
semgrep_bin = find_semgrep_binary()
print(f"[*] Using semgrep: {semgrep_bin}")
raw_json, stderr, exit_code = run_scan(
semgrep_bin, args.target, args.config, args.severity,
args.exclude, args.include, timeout_per_rule=args.timeout_per_rule,
verbose=args.verbose
)
findings, stats = parse_findings(raw_json)
severity_counts = format_summary(findings, stats, args.target)
report = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"tool": "semgrep",
"target": args.target,
"configs": args.config or ["auto"],
"stats": stats,
"severity_counts": severity_counts,
"findings_count": len(findings),
"findings": findings,
"risk_level": (
"CRITICAL" if severity_counts.get("ERROR", 0) > 5
else "HIGH" if severity_counts.get("ERROR", 0) > 0
else "MEDIUM" if severity_counts.get("WARNING", 0) > 0
else "LOW"
),
}
if args.output:
with open(args.output, "w") as f:
json.dump(report, f, indent=2)
print(f"\n[+] Report saved to {args.output}")
elif args.verbose:
print(json.dumps(report, indent=2))
print(f"\n[*] Risk Level: {report['risk_level']}")
if __name__ == "__main__":
main()