Network Traffic Analysis for Threat Detection: Beyond Signature-Based Tools
Network traffic analysis (NTA) — also called Network Detection and Response (NDR) in vendor contexts — examines network flows, packets, and behavioral baselines to detect threats that endpoint agents miss, cannot see, or have not yet received signatures for. For environments with unmanaged devices, OT systems, or sophisticated attackers actively evading EDR, network telemetry is often the most reliable detection surface available. This guide covers the methodology of NTA-based threat detection: what data to collect, how to establish baselines, what anomalies are worth chasing, and how NTA integrates with the broader detection stack.
Flow Data vs. Full Packet Capture: Choosing the Right Telemetry
Network telemetry exists on a spectrum from lightweight flow records to full packet capture, each with different detection capabilities and storage requirements.
NetFlow / IPFIX (flow records): Records connection metadata: source/dest IP, port, protocol, byte count, packet count, duration. No payload. Storage: approximately 1-5 GB per day per 1 Gbps of traffic. Detection uses: volume anomalies, port scanning, beaconing (regular intervals to the same destination), east-west lateral movement patterns, data exfiltration by volume.
DNS query logs: DNS is the single richest free detection source in most environments. Every domain lookup, successful or not, is a record of intent. Detection uses: DGA domain identification, DNS tunneling, C2 beaconing via DNS, newly registered domain lookups, subpoena-level forensic reconstruction of what a host tried to reach.
TLS metadata (without decryption): SNI (Server Name Indication) in the TLS handshake reveals the target domain even in encrypted traffic. JA3/JA3S fingerprints characterize the TLS client and server behavior without decryption. Detection uses: malware C2 over HTTPS (specific JA3 fingerprints match known malware families), self-signed certificate connections, non-browser TLS from browser processes.
Full packet capture (PCAP): Complete payload capture. Storage: 50-500 GB per day per 1 Gbps of traffic depending on compression. Detection uses: everything above, plus payload inspection, protocol anomaly detection, credential extraction from cleartext protocols. Typically limited to 24-72 hour retention at the tap point, longer in tiered storage for specific sessions.
Practical recommendation: Deploy flow collection everywhere, DNS logging everywhere, TLS metadata at internet egress, and PCAP at chokepoints (internet gateway, sensitive segment boundaries, OT/IT boundary).
Establishing Behavioral Baselines
Anomaly detection requires knowing what normal looks like. Baselines must be established before detection is meaningful.
Host-level baselines:
- Normal destination IP/domain set per host (web servers should not initiate outbound connections to random internet IPs)
- Normal protocols per host (a domain controller initiating HTTP/S to external IPs is unusual)
- Normal byte volume per connection and per hour
- Normal connection frequency to each destination
Network segment baselines:
- Expected traffic patterns between segments (OT should not communicate with DMZ; development network should not access production database)
- Expected total bandwidth per link at different times of day
- Expected DNS query rate per subnet
Beaconing baseline: Legitimate software phones home at irregular intervals. Malware C2 often beacons at precise regular intervals. Calculate connection frequency distribution per source-destination pair. Highly regular intervals (low variance in inter-connection timing) are a beaconing indicator even when the destination passes reputation checks.
Baselines take 2-4 weeks of clean traffic to establish meaningfully. They must be refreshed quarterly and whenever network topology changes significantly.
Briefings like this, every morning before 9am.
Threat intel, active CVEs, and campaign alerts, distilled for practitioners. 50,000+ subscribers. No noise.
High-Value Detection Patterns for NTA
Not all anomalies warrant investigation. Focus detection logic on patterns with known attacker relevance:
DNS-based C2 detection:
- High query volume to a single domain or domain family (DNS tunneling)
- Queries to algorithmically generated domain names (DGA): long random subdomains, high entropy, no associated A record history
- Queries to domains registered within the past 30 days from internal hosts
- NXDOMAIN burst (mass failed lookups characteristic of DGA malware scanning for its C2)
Beaconing detection:
- Regular outbound connections (sub-60-second intervals, variance below 5%) to a single external IP
- HTTPS beaconing to IPs without valid PTR records or with mismatched certificates
- HTTP beaconing with minimal data transfer (heartbeat pattern)
Lateral movement detection:
- SMB connections from workstations to other workstations (peer-to-peer SMB is a lateral movement indicator)
- WMI remote execution (TCP 135 + high RPC port connections between workstations)
- RDP connections from servers to workstations (reversed expected direction)
- Authentication sweeps (one source attempting authentication to many internal targets within a short window)
Data exfiltration detection:
- Outbound transfer volume exceeding historical baseline for that host by more than 3 standard deviations
- Large data transfers to cloud storage (Dropbox, Google Drive, OneDrive) from systems that do not normally use them
- DNS exfiltration: queries with unusually long subdomains (encoded payload)
- Uploads to new external IP destinations at unusual hours
TLS Inspection: When and How to Decrypt
The majority of malicious traffic is now encrypted. TLS inspection allows payload analysis but introduces complexity, performance overhead, and privacy considerations.
SSL/TLS interception options:
Inline SSL inspection (man-in-the-middle): Proxy device (Palo Alto, Zscaler, Fortinet) intercepts and re-encrypts traffic. Full payload visibility. Requires certificate deployment to endpoints. Breaks certificate pinning in some applications (banking apps, some SaaS tools). Latency impact: 2-10ms per connection depending on hardware.
Passive TLS metadata analysis (no decryption): Analyze JA3/JA3S fingerprints, SNI, certificate characteristics, and traffic patterns without decrypting. Tools: Zeek (formerly Bro), Suricata, Stamus Networks. Detection power is lower than full inspection but avoids breaking applications and does not require certificate deployment.
Selective decryption: Decrypt traffic to high-risk categories (newly registered domains, low-reputation IPs, anonymizing proxies) while leaving trusted destinations (Microsoft, Google, Salesforce) uninterrupted. Best balance of coverage and operational overhead.
For most enterprise environments: deploy passive TLS metadata analysis everywhere, selective inline inspection at internet egress for high-risk traffic categories.
Integrating NTA with SIEM and EDR
NTA in isolation catches network-level indicators but lacks context about what process generated the traffic or what the user was doing. The detection value multiplies when correlated with endpoint and identity data.
NTA + EDR correlation:
- EDR identifies the process making the connection; NTA identifies where it connected. Combining these catches malware that uses trusted processes (lolbins) to make network connections that look benign from either source alone.
- Example: EDR shows
mshta.exemaking an outbound connection; NTA shows the connection is beaconing at 30-second intervals to an IP registered two days ago. Either alone is ambiguous; together they are high confidence.
NTA + Identity correlation:
- Authentication logs from AD or your IdP placed alongside NTA lateral movement detections. SMB peer-to-peer plus authentication events from a new account or after a suspicious login event escalates the priority.
NTA + Threat intelligence:
- Enrich all external connection events with IP and domain reputation at ingestion time. Flag connections to IPs in threat intel feeds (C2 infrastructure, malware distribution, anonymizers) for immediate triage.
SIEM integration: Ingest flow data, DNS logs, and TLS metadata into SIEM alongside endpoint and identity data. Build correlation rules that require multiple data sources. Single-source detections produce more noise; multi-source correlations produce better signal.
NDR Platform Evaluation: What to Look For
Network Detection and Response (NDR) platforms (ExtraHop, Darktrace, Corelight, Vectra AI, Cisco Secure Network Analytics) automate much of the NTA workflow. Evaluation criteria that matter:
Telemetry breadth: Does the platform ingest both flow data and full packet metadata? Can it process east-west (internal) traffic, not just north-south (perimeter)?
Baseline learning period: How long before behavioral baselines are established? What happens to detection quality during that period?
Alert fidelity: What is the true positive rate in your specific environment? Request a proof-of-concept with your actual traffic before committing.
Detection methodology transparency: Does the vendor explain what each detection model is looking for, or is it a black box? Unexplainable detections cannot be tuned.
SIEM and SOAR integration: Does the platform send enriched alerts to your SIEM, or does it require analysts to maintain a separate investigation interface?
Encrypted traffic analysis: What is the platform's approach to encrypted traffic — passive metadata only, or optional decryption? Does it support JA3/JA3S fingerprinting?
Storage and retention: What is the default retention period for flow records and packet capture? What are the storage costs for extended retention?
The bottom line
Network traffic analysis is not a replacement for endpoint security or signature-based detection — it is the detection surface that catches what those tools miss. Deploy flow data and DNS logging universally, establish behavioral baselines before tuning detection logic, focus detection on patterns with demonstrated attacker relevance (beaconing, DNS-based C2, lateral movement), and correlate NTA findings with EDR and identity data before assigning priority. The combination is substantially more powerful than any single source.
Frequently asked questions
What is network traffic analysis (NTA) in cybersecurity?
Network traffic analysis examines network flows, packets, and metadata to detect threats based on behavioral anomalies rather than known-bad signatures. It catches attacker activity — C2 beaconing, lateral movement, data exfiltration — that endpoint agents miss because the traffic crosses the network even when endpoints are managed.
What is the difference between NTA and NDR?
Network Traffic Analysis (NTA) is the discipline; Network Detection and Response (NDR) is the product category that automates NTA at scale. NDR platforms ingest network telemetry, build behavioral baselines, surface anomalies, and integrate with SIEM/SOAR for response. NTA can be done manually with open-source tools like Zeek and Suricata; NDR platforms productize that capability.
Do I need full packet capture for effective network-based threat detection?
No. Flow data (NetFlow/IPFIX), DNS query logs, and TLS metadata detect the majority of high-value threat patterns — beaconing, lateral movement, DNS tunneling, DGA domains — without full payload capture. PCAP adds coverage for payload-dependent detections and forensic reconstruction but requires far more storage. Deploy flow and DNS everywhere; PCAP at chokepoints.
How do you detect C2 beaconing in network traffic?
Beaconing detection looks for highly regular outbound connections — low variance in inter-connection timing — to a single external destination. Legitimate software phones home at irregular intervals; malware C2 often beacons on a precise schedule. Calculate the variance of connection intervals per source-destination pair. Low-variance patterns to low-reputation destinations warrant investigation.
Can NTA detect threats in encrypted traffic?
Yes, to a meaningful extent without decryption. TLS metadata — SNI (target domain), JA3/JA3S fingerprints (TLS client/server behavior), certificate characteristics, and traffic volume patterns — reveals a great deal about encrypted connections. Known malware families have characteristic JA3 fingerprints. Suspicious patterns: self-signed certs, beaconing cadence in HTTPS traffic, non-browser TLS from browser processes.
What open-source tools support network traffic analysis?
Zeek (formerly Bro) generates rich connection logs, DNS logs, TLS metadata, and protocol-level records from PCAP or live capture. Suricata provides signature-based detection plus flow data. Arkime (formerly Moloch) provides PCAP indexing and search. Elastic SIEM has built-in support for Zeek and Suricata log ingestion. These tools together provide enterprise-grade NTA capability without commercial NDR licensing.
Sources & references
- The Practice of Network Security Monitoring — Richard Bejtlich
- MITRE ATT&CK Network-Based Techniques
- Gartner Network Detection and Response Market Guide 2025
- CISA Network Monitoring Guidance
Free resources
Critical CVE Reference Card 2025–2026
25 actively exploited vulnerabilities with CVSS scores, exploit status, and patch availability. Print it, pin it, share it with your SOC team.
Ransomware Incident Response Playbook
Step-by-step 24-hour IR checklist covering detection, containment, eradication, and recovery. Built for SOC teams, IR leads, and CISOs.
Get threat intel before your inbox does.
50,000+ security professionals read Decryption Digest for early warnings on zero-days, ransomware, and nation-state campaigns. Free, weekly, no spam.
Unsubscribe anytime. We never sell your data.
Founder & Cybersecurity Evangelist, Decryption Digest
Cybersecurity professional with expertise in threat intelligence, vulnerability research, and enterprise security. Covers zero-days, ransomware, and nation-state operations for 50,000+ security professionals weekly.
The Mythos Brief is free.
AI that finds 27-year-old zero-days. What it means for your security program.