soc operations

Triaging Security Alerts in Splunk

Triages security alerts in Splunk Enterprise Security by classifying severity, investigating notable events, correlating related telemetry, and making escalation or closure decisions using SPL queries and the Incident Review dashboard. Use when SOC analysts face queued alerts from correlation searches, need to prioritize investigation order, or must document triage decisions for handoff to Tier 2/3 analysts.

alert-triagecorrelation-searchincident-reviewnotable-eventssiemsocsplunk
Install this skill
npx skills add mukul975/Anthropic-Cybersecurity-Skills
Framework mappings

When to Use

Use this skill when:

  • SOC Tier 1 analysts need to process the Incident Review queue in Splunk Enterprise Security (ES)
  • Notable events require rapid severity classification and initial investigation before escalation
  • Alert volume exceeds capacity and analysts need a systematic triage methodology
  • Management requests metrics on alert disposition (true positive, false positive, benign)

Do not use for deep forensic investigation — escalate to Tier 2/3 after initial triage confirms malicious activity.

Prerequisites

  • Splunk Enterprise Security 7.x+ with Incident Review dashboard configured
  • CIM-normalized data sources (Windows Event Logs, firewall, proxy, endpoint)
  • Role with ess_analyst capability for notable event status updates
  • Familiarity with SPL (Search Processing Language)

Workflow

Step 1: Access Incident Review and Prioritize Queue

Open the Incident Review dashboard in Splunk ES. Sort notable events by urgency (calculated from severity x priority). Apply filters to focus on unassigned events:

| `notable`
| search status="new" OR status="unassigned"
| sort - urgency
| table _time, rule_name, src, dest, user, urgency, status
| head 50

Focus on Critical and High urgency events first. Group related alerts by src or dest to identify attack chains rather than treating each alert independently.

Step 2: Investigate the Notable Event Context

For each notable event, pivot to raw events. Example for a brute force alert:

index=wineventlog sourcetype="WinEventLog:Security" EventCode=4625
src_ip="192.168.1.105"
earliest=-1h latest=now
| stats count by src_ip, dest, user, status
| where count > 10
| sort - count

Check if the source IP is internal (lateral movement) or external (perimeter attack). Cross-reference with asset and identity lookups:

| `notable`
| search rule_name="Brute Force Access Behavior Detected"
| lookup asset_lookup_by_cidr ip AS src OUTPUT category, owner, priority
| lookup identity_lookup_expanded identity AS user OUTPUT department, managedBy
| table _time, src, dest, user, category, owner, department

Step 3: Correlate Across Data Sources

Check if the same source appears in other telemetry:

index=proxy OR index=firewall src="192.168.1.105" earliest=-24h
| stats count by index, sourcetype, action, dest_port
| sort - count

Look for corroborating evidence: Did the same IP also trigger DNS anomalies, proxy blocks, or endpoint detection alerts?

index=main sourcetype="cisco:asa" src="192.168.1.105" action=blocked earliest=-24h
| timechart span=1h count by dest_port

Step 4: Check Threat Intelligence Enrichment

Query the threat intelligence framework for known IOCs:

| `notable`
| search search_name="Threat - Threat Intelligence Match - Rule"
| lookup threat_intel_by_ip ip AS src OUTPUT threat_collection, threat_description, threat_key
| table _time, src, dest, threat_collection, threat_description, weight
| where weight >= 3

For domains, check against threat lists:

| tstats count from datamodel=Web where Web.url="*evil-domain.com*" by Web.src, Web.url, Web.status
| rename Web.* AS *

Step 5: Classify and Disposition the Alert

Update the notable event status in Incident Review:

Disposition Criteria Action
True Positive Corroborating evidence confirms malicious activity Escalate to Tier 2, create incident ticket
Benign True Positive Alert fired correctly but activity is authorized (e.g., pen test) Close with comment, add suppression if recurring
False Positive Alert logic matched benign behavior Close, tune correlation search, document pattern
Undetermined Insufficient data to classify Assign to Tier 2 with investigation notes

Update via Splunk ES UI or REST API:

| sendalert update_notable_event param.status="2" param.urgency="critical"
  param.comment="Confirmed brute force from compromised workstation. Escalated to IR-2024-0431."
  param.owner="analyst_jdoe"

Step 6: Document Triage Findings

Record in the notable event comment field:

  • Source/destination involved
  • Data sources examined
  • Correlation findings (related alerts, TI matches)
  • Disposition rationale
  • Next steps for escalation
| `notable`
| search rule_name="Brute Force*" status="closed"
| stats count by status_label, disposition
| addtotal

Step 7: Track Triage Metrics

Monitor triage performance over time:

| `notable`
| where status_end > 0
| eval triage_time = status_end - _time
| stats avg(triage_time) AS avg_triage_sec, median(triage_time) AS med_triage_sec,
        count by rule_name, status_label
| eval avg_triage_min = round(avg_triage_sec/60, 1)
| sort - count
| table rule_name, status_label, count, avg_triage_min

Key Concepts

Term Definition
Notable Event Splunk ES alert generated by a correlation search that meets defined risk or threshold criteria
Urgency Calculated field combining event severity with asset/identity priority (Critical/High/Medium/Low/Informational)
Correlation Search Scheduled SPL query that detects threat patterns and generates notable events when conditions match
CIM Common Information Model — Splunk's normalized field naming convention enabling cross-source queries
Disposition Final classification of an alert: true positive, false positive, benign true positive, or undetermined
MTTD/MTTR Mean Time to Detect / Mean Time to Respond — key SOC metrics measuring detection and resolution speed

Tools & Systems

  • Splunk Enterprise Security: SIEM platform providing Incident Review dashboard, correlation searches, and risk-based alerting
  • Splunk SOAR (Phantom): Orchestration platform for automating triage playbooks and enrichment actions
  • Asset & Identity Framework: Splunk ES lookup tables mapping IPs to asset owners and users to departments for context enrichment
  • Threat Intelligence Framework: Splunk ES module ingesting STIX/TAXII feeds and matching IOCs against notable events

Common Scenarios

  • Brute Force Alerts: Correlate EventCode 4625 (failed logon) with 4624 (successful logon) from same source to determine if attack succeeded
  • Malware Detection: Cross-reference endpoint AV alert with proxy logs for C2 callback confirmation
  • Data Exfiltration Alert: Check outbound data volume from DLP and proxy logs against user baseline
  • Privilege Escalation: Correlate EventCode 4672 (special privileges assigned) with 4720 (account created) from non-admin users
  • Lateral Movement: Map EventCode 4648 (explicit credential logon) across multiple destinations from single source

Output Format

TRIAGE REPORT — Notable Event #NE-2024-08921
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Alert:        Brute Force Access Behavior Detected
Time:         2024-03-15 14:23:07 UTC
Source:       192.168.1.105 (WORKSTATION-042, Finance Dept)
Destination:  10.0.5.20 (DC-PRIMARY, Domain Controller)
User:         jsmith (Finance Analyst)
 
Investigation:
  - 847 failed logons (4625) in 12 minutes from src
  - Successful logon (4624) at 14:35:02 after brute force
  - No proxy/DNS anomalies from src in prior 24h
  - Source not on threat intel lists
 
Disposition:  TRUE POSITIVE — Compromised credential
Action:       Escalated to Tier 2, ticket IR-2024-0431 created
              Account jsmith disabled pending password reset
Source materials

References and resources

Everything below is rendered for inspection. Script files are read-only and never run.

References 1

api-reference.md2.1 KB

API Reference: Triaging Security Alerts in Splunk

splunklib (Splunk SDK for Python)

Installation

pip install splunk-sdk

Connection

import splunklib.client as client
service = client.connect(host="localhost", port=8089,
                         username="admin", password="password")

Running Searches

# Blocking search (wait for results)
job = service.jobs.create(query, exec_mode="blocking")
 
# Parse results
import splunklib.results as results
for result in results.JSONResultsReader(job.results(output_mode="json")):
    if isinstance(result, dict):
        print(result)

Search Parameters

Parameter Description
exec_mode blocking (wait) or normal (async)
earliest_time Search time range start (e.g., -24h)
latest_time Search time range end (e.g., now)
output_mode json, xml, or csv

Key SPL Commands for Triage

Command Purpose
`notable` Macro to access ES notable events
lookup asset_lookup_by_cidr Enrich with asset information
lookup identity_lookup_expanded Enrich with identity context
lookup threat_intel_by_ip Check IP against threat feeds
tstats Fast datamodel statistics
sendalert update_notable_event Update notable event status

Notable Event Status Values

Value Status
0 Unassigned
1 New
2 In Progress
3 Pending
4 Resolved
5 Closed

Disposition Categories

Disposition Criteria
True Positive Confirmed malicious activity
Benign True Positive Alert correct but activity authorized
False Positive Benign behavior matched detection logic
Undetermined Insufficient data to classify

References

Scripts 1

agent.py9.3 KB
Display-only source. This catalog never executes bundled scripts.
#!/usr/bin/env python3
"""Agent for triaging security alerts in Splunk Enterprise Security."""

import splunklib.client as splunk_client
import splunklib.results as splunk_results
import json
import sys
import argparse
from datetime import datetime


def connect_splunk(host, port, username, password):
    """Connect to Splunk Enterprise instance."""
    try:
        service = splunk_client.connect(
            host=host, port=port, username=username, password=password,
            autologin=True,
        )
        print(f"[*] Connected to Splunk {host}:{port}")
        return service
    except Exception as e:
        print(f"[-] Connection failed: {e}")
        sys.exit(1)


def get_notable_events(service, status="new", limit=50):
    """Query notable events from Splunk ES Incident Review."""
    query = f"""| `notable`
| search status="{status}"
| sort - urgency
| table _time, rule_name, src, dest, user, urgency, status, event_id
| head {limit}"""
    print(f"\n[*] Fetching notable events (status={status})...")
    job = service.jobs.create(query, exec_mode="blocking")
    results = []
    for result in splunk_results.JSONResultsReader(job.results(output_mode="json")):
        if isinstance(result, dict):
            results.append(result)
            print(f"  [{result.get('urgency', '?')}] {result.get('rule_name', 'Unknown')} "
                  f"| src={result.get('src', 'N/A')} dest={result.get('dest', 'N/A')}")
    print(f"[*] Retrieved {len(results)} notable events")
    return results


def investigate_brute_force(service, src_ip, hours=1):
    """Investigate brute force activity from a source IP."""
    query = f"""search index=wineventlog sourcetype="WinEventLog:Security" EventCode=4625
src_ip="{src_ip}" earliest=-{hours}h latest=now
| stats count by src_ip, dest, user, status
| where count > 5
| sort - count"""
    print(f"\n[*] Investigating brute force from {src_ip}...")
    job = service.jobs.create(query, exec_mode="blocking")
    results = []
    for result in splunk_results.JSONResultsReader(job.results(output_mode="json")):
        if isinstance(result, dict):
            results.append(result)
            print(f"  {result.get('src_ip')} -> {result.get('dest')} "
                  f"user={result.get('user')} count={result.get('count')}")

    success_query = f"""search index=wineventlog sourcetype="WinEventLog:Security" EventCode=4624
src_ip="{src_ip}" earliest=-{hours}h latest=now
| stats count by src_ip, dest, user
| where count > 0"""
    success_job = service.jobs.create(success_query, exec_mode="blocking")
    for result in splunk_results.JSONResultsReader(success_job.results(output_mode="json")):
        if isinstance(result, dict):
            print(f"  [!] SUCCESSFUL logon: {result.get('user')} on {result.get('dest')}")
    return results


def correlate_across_sources(service, src_ip, hours=24):
    """Correlate alerts across multiple data sources for a given IP."""
    query = f"""search (index=proxy OR index=firewall OR index=dns) src="{src_ip}" earliest=-{hours}h
| stats count by index, sourcetype, action, dest_port
| sort - count"""
    print(f"\n[*] Correlating across sources for {src_ip}...")
    job = service.jobs.create(query, exec_mode="blocking")
    results = []
    for result in splunk_results.JSONResultsReader(job.results(output_mode="json")):
        if isinstance(result, dict):
            results.append(result)
            print(f"  {result.get('index')}/{result.get('sourcetype')}: "
                  f"action={result.get('action')} port={result.get('dest_port')} "
                  f"count={result.get('count')}")
    return results


def check_threat_intel(service, indicator, indicator_type="ip"):
    """Check an indicator against Splunk ES threat intelligence."""
    field_map = {"ip": "src", "domain": "url", "hash": "file_hash"}
    field = field_map.get(indicator_type, "src")
    query = f"""| `notable`
| search search_name="Threat*" {field}="{indicator}"
| lookup threat_intel_by_{indicator_type} {indicator_type} AS {field}
  OUTPUT threat_collection, threat_description, weight
| table _time, {field}, threat_collection, threat_description, weight
| where weight >= 1"""
    print(f"\n[*] Checking threat intelligence for {indicator}...")
    job = service.jobs.create(query, exec_mode="blocking")
    matches = []
    for result in splunk_results.JSONResultsReader(job.results(output_mode="json")):
        if isinstance(result, dict):
            matches.append(result)
            print(f"  [!] TI Match: {result.get('threat_collection', 'Unknown')} "
                  f"(weight: {result.get('weight', '?')})")
    if not matches:
        print("  [+] No threat intelligence matches")
    return matches


def enrich_with_asset_identity(service, src_ip=None, username=None):
    """Enrich an alert with asset and identity context."""
    results = {}
    if src_ip:
        query = f"""| inputlookup asset_lookup_by_cidr
| where cidrmatch(cidr, "{src_ip}")
| table cidr, category, owner, priority, lat, long"""
        print(f"\n[*] Enriching asset info for {src_ip}...")
        job = service.jobs.create(query, exec_mode="blocking")
        for result in splunk_results.JSONResultsReader(job.results(output_mode="json")):
            if isinstance(result, dict):
                results["asset"] = result
                print(f"  Asset: {result.get('category', 'Unknown')} "
                      f"owner={result.get('owner', 'N/A')} priority={result.get('priority', 'N/A')}")

    if username:
        query = f"""| inputlookup identity_lookup_expanded
| search identity="{username}"
| table identity, first, last, department, managedBy, email"""
        print(f"[*] Enriching identity info for {username}...")
        job = service.jobs.create(query, exec_mode="blocking")
        for result in splunk_results.JSONResultsReader(job.results(output_mode="json")):
            if isinstance(result, dict):
                results["identity"] = result
                print(f"  User: {result.get('first', '')} {result.get('last', '')} "
                      f"dept={result.get('department', 'N/A')}")
    return results


def get_triage_metrics(service, days=30):
    """Get triage performance metrics."""
    query = f"""| `notable`
| where status_end > 0
| eval triage_time = status_end - _time
| stats avg(triage_time) AS avg_sec, median(triage_time) AS med_sec,
        count by rule_name, status_label
| eval avg_min = round(avg_sec/60, 1)
| sort - count
| head 20
| table rule_name, status_label, count, avg_min"""
    print(f"\n[*] Fetching triage metrics (last {days} days)...")
    job = service.jobs.create(query, exec_mode="blocking",
                               earliest_time=f"-{days}d", latest_time="now")
    for result in splunk_results.JSONResultsReader(job.results(output_mode="json")):
        if isinstance(result, dict):
            print(f"  {result.get('rule_name', 'Unknown')}: "
                  f"{result.get('count', 0)} alerts, avg triage: {result.get('avg_min', '?')} min")


def generate_triage_report(notable, correlations, ti_matches, enrichment, output_path):
    """Generate a structured triage report."""
    report = {
        "triage_date": datetime.now().isoformat(),
        "notable_events": notable,
        "correlations": correlations,
        "threat_intel_matches": ti_matches,
        "enrichment": enrichment,
    }
    with open(output_path, "w") as f:
        json.dump(report, f, indent=2, default=str)
    print(f"\n[*] Triage report saved to {output_path}")


def main():
    parser = argparse.ArgumentParser(description="Splunk ES Alert Triage Agent")
    parser.add_argument("action", choices=["queue", "investigate", "correlate", "threat-intel",
                                           "enrich", "metrics", "full-triage"])
    parser.add_argument("--host", default="localhost", help="Splunk host")
    parser.add_argument("--port", type=int, default=8089, help="Splunk management port")
    parser.add_argument("--username", default="admin")
    parser.add_argument("--password", required=True)
    parser.add_argument("--src-ip", help="Source IP to investigate")
    parser.add_argument("--user", help="Username to enrich")
    parser.add_argument("--indicator", help="IOC to check against threat intel")
    parser.add_argument("--status", default="new", help="Notable event status filter")
    parser.add_argument("-o", "--output", default="triage_report.json")
    args = parser.parse_args()

    service = connect_splunk(args.host, args.port, args.username, args.password)

    if args.action == "queue":
        get_notable_events(service, args.status)
    elif args.action == "investigate":
        investigate_brute_force(service, args.src_ip)
    elif args.action == "correlate":
        correlate_across_sources(service, args.src_ip)
    elif args.action == "threat-intel":
        check_threat_intel(service, args.indicator)
    elif args.action == "enrich":
        enrich_with_asset_identity(service, args.src_ip, args.user)
    elif args.action == "metrics":
        get_triage_metrics(service)
    elif args.action == "full-triage":
        notable = get_notable_events(service, args.status)
        corr = correlate_across_sources(service, args.src_ip) if args.src_ip else []
        ti = check_threat_intel(service, args.src_ip) if args.src_ip else []
        enrich = enrich_with_asset_identity(service, args.src_ip, args.user)
        generate_triage_report(notable, corr, ti, enrich, args.output)


if __name__ == "__main__":
    main()
Keep exploring