Linux Network Packet Capture and Protocol Analysis: A Practical Troubleshooting Guide from tcpdump to Wireshark

Overview

2 AM. Your phone rings. The order system is timing out across the board — CPU is fine, memory is fine, disk I/O is fine. Restarting services does nothing. Rolling back does nothing. You stare at the dashboard; every metric is green. Only the users are screaming.

Nine times out of ten, it is a network-layer problem. And what you need is a pair of eyes that can actually see the packets.

tcpdump is those eyes. It is not new — it has been around since the 1990s — but it remains the most hard-core network diagnostic tool on Linux. Let me be blunt: if you only know how to read monitoring charts and cannot capture packets, you are blind when a network-layer issue hits.

This article skips the fancy concepts and goes straight to practice: how to capture with tcpdump, how to write BPF filters, how to analyze pcap files with Wireshark, what the TCP three-way handshake and TLS handshake look like in a capture, and the tricks I learned the hard way.

tcpdump Basics: Understand What You Are Capturing

What Is Packet Capture

Under normal conditions, only the OS kernel processes every data frame the NIC receives — user-space programs cannot see the raw content. Packet capture turns on promiscuous mode on the NIC, telling the kernel to hand copies of all packets flowing through the interface to the capture tool.

Think of it this way: normally the NIC is a mailman who only delivers packages addressed to you. With promiscuous mode on, it becomes a surveillance camera that photographs every package passing down the street — whether it is for you or not.

tcpdump vs Wireshark: Different Jobs

The standard workflow in practice: capture on the server with tcpdump and save as pcap, scp it to your local machine, open with Wireshark. I have used this combo over five hundred times in the past seven years.

Installation and Permissions

Most Linux distros come with tcpdump pre-installed. Check:

which tcpdump || echo "Not installed"
tcpdump --version

If not installed:

# Debian/Ubuntu
sudo apt-get install -y tcpdump

# CentOS/RHEL
sudo yum install -y tcpdump

Permission management is a gotcha. tcpdump needs the CAP_NET_RAW capability to capture packets, so by default only root can use it. But in production, you cannot give everyone root access. The recommended approach is a sudo whitelist:

# Create a dedicated sudoers config
sudo visudo -f /etc/sudoers.d/tcpdump

# Allow members of the ops group to run tcpdump without a password
%ops ALL=(root) NOPASSWD: /usr/sbin/tcpdump

Now ops team members can run sudo tcpdump without full root privileges. Do not take the lazy route of chmod +s /usr/sbin/tcpdump — that opens a root backdoor for every user on the system.

Core Parameters Cheatsheet

Here is a reference table; each parameter is explained below.

Common Parameter Combinations

# The most commonly used production capture command
sudo tcpdump -i eth0 -nn -s 0 -w capture.pcap

# View HTTP traffic in real time (text protocols only)
sudo tcpdump -i eth0 -nn -A -s 0 'tcp port 80'

# Capture 1000 packets then stop
sudo tcpdump -i eth0 -nn -c 1000 -w capture.pcap

# Rotate hourly, keep at most 24 files
sudo tcpdump -i eth0 -nn -s 0 -G 3600 -W 24 -w /var/log/tcpdump/capture_%Y%m%d_%H.pcap

Why -n and -nn Matter

Without -n, tcpdump tries to reverse-resolve IP addresses to hostnames. In production, this means every captured packet triggers a DNS query. If you are troubleshooting a DNS issue — congratulations, your capture tool itself is generating a DNS storm.

Adding -nn goes further: it also skips resolving port numbers. Without it, tcpdump displays 443 as https, which looks convenient but causes confusion when writing filter rules. -nn forces all-numeric output — clean and unambiguous.

BPF Filters: The Art of Precision Capture

BPF (Berkeley Packet Filter) is tcpdump’s filtering engine. It runs in kernel space, so packets that do not match the rules are dropped before being copied to user space. This makes BPF filtering extremely performant — you can filter in real time on a 10GbE NIC without packet loss.

Filter Syntax Basics

A BPF filter expression consists of three components:

Type    Direction  Protocol  Value
host    src        tcp       192.168.1.100

Common Filter Examples

# Capture all traffic for a specific host
sudo tcpdump -i eth0 -nn host 10.0.0.5

# Capture traffic with source IP 10.0.0.5
sudo tcpdump -i eth0 -nn src host 10.0.0.5

# Capture TCP traffic to destination port 443
sudo tcpdump -i eth0 -nn 'tcp dst port 443'

# Capture traffic between two hosts
sudo tcpdump -i eth0 -nn 'host 10.0.0.5 and host 10.0.0.6'

# Capture DNS queries (UDP 53)
sudo tcpdump -i eth0 -nn 'udp port 53'

# Capture ICMP (ping)
sudo tcpdump -i eth0 -nn icmp

# Exclude SSH traffic (avoid capture feedback loop)
sudo tcpdump -i eth0 -nn 'not port 22'

# Capture a specific subnet
sudo tcpdump -i eth0 -nn 'net 10.0.0.0/24'

# Combined filter: traffic between 10.0.0.5 and 10.0.0.6 on port 443
sudo tcpdump -i eth0 -nn 'host 10.0.0.5 and host 10.0.0.6 and tcp port 443'

TCP Flag Filtering

This is key to diagnosing connection problems. The TCP header has 6 flag bits, and BPF can filter on each precisely:

# Capture only SYN packets (who is initiating connections?)
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0'

# Capture only RST packets (connections being reset)
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & tcp-rst != 0'

# Capture only FIN packets (who is closing connections?)
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & tcp-fin != 0'

In my experience, SYN filtering is extremely effective for diagnosing connection timeouts. Once I dealt with a microservice that had intermittent timeouts — monitoring showed nothing wrong. I filtered on SYN for five minutes and found the client was sending SYN, waiting 3 seconds with no SYN-ACK, then retrying — the server was genuinely not responding. Further investigation revealed the server’s conntrack table was full.

Packet Size Filtering

# Capture packets larger than 1400 bytes (investigate large-packet fragmentation)
sudo tcpdump -i eth0 -nn 'greater 1400'

# Capture packets smaller than 60 bytes (possibly abnormal ACK-only packets)
sudo tcpdump -i eth0 -nn 'less 60'

Common Filter Pitfalls

Pitfall 1: Parentheses not quoted

# Wrong — shell eats the parentheses
sudo tcpdump -i eth0 host 10.0.0.5 and (port 80 or port 443)

# Correct — wrap in quotes
sudo tcpdump -i eth0 'host 10.0.0.5 and (port 80 or port 443)'

Pitfall 2: Direction reversed

src host 10.0.0.5 means the source IP is 10.0.0.5; dst host 10.0.0.5 means the destination IP is 10.0.0.5. Get this wrong and you capture the wrong direction, then analyze empty air for ages.

Pitfall 3: Forgetting to exclude SSH traffic

When you SSH into a server and run tcpdump, tcpdump’s own output travels through the SSH channel. If you do not exclude port 22, you capture your own SSH traffic, creating a positive feedback loop that floods the terminal.

# Safe approach
sudo tcpdump -i eth0 -nn 'not port 22 and not port 53' -w capture.pcap

Capture Strategies: Do Not Capture Blindly in Production

Disk and Performance Considerations

Full packet capture on a 10GbE NIC can generate several GB per minute. Without filtering, disk fills up fast. This is not theory — I have seen someone run tcpdump -i eth0 -w capture.pcap overnight, only to find the root partition at 100% the next morning, taking down the entire business.

Production capture rules:

1. Always add filter conditions, even if just excluding SSH traffic
2. Use -c to limit packet count, or -G + -W for rotation
3. Write to a dedicated directory — not /tmp, which may be tmpfs in containers and consume RAM
4. Monitor remaining disk space — run a simple script as a safety net

# Safe production capture setup
sudo mkdir -p /var/log/tcpdump
sudo tcpdump -i eth0 -nn -s 0 \
  -G 300 -W 48 \
  -w /var/log/tcpdump/capture_%Y%m%d_%H%M%S.pcap \
  'host 10.0.0.5 and tcp port 443'

# -G 300: rotate every 5 minutes
# -W 48: keep at most 48 files (4 hours)
# Filter specific host and port, no irrelevant traffic

The snaplen Trap

The -s parameter controls how many bytes of each packet are captured. The default varies by version:

• Old versions default to 68 or 96 bytes — headers only
• New versions default to 262144 bytes — essentially full packets

If you use an old tcpdump to capture a TLS handshake, the certificate packet often exceeds 1500 bytes, and a small snaplen truncates the certificate chain in ServerHello. Always use -s 0 when troubleshooting TLS:

# -s 0 means no truncation, capture full packets
sudo tcpdump -i eth0 -nn -s 0 -w tls_capture.pcap 'tcp port 443'

Wireshark Analysis: From pcap to Root Cause

tcpdump + Wireshark Workflow

# Step 1: Capture on the server
sudo tcpdump -i eth0 -nn -s 0 -w /tmp/capture.pcap 'host 10.0.0.5 and tcp port 443'

# Step 2: Download to local
scp user@server:/tmp/capture.pcap ./

# Step 3: Open with Wireshark locally
wireshark capture.pcap &

If you do not want to scp every time, you can stream in real time via an SSH pipe:

# Remote capture, open directly in local Wireshark
ssh user@server 'sudo tcpdump -i eth0 -nn -s 0 -w - "host 10.0.0.5"' | wireshark -k -i -

This works on both macOS and Linux. -w - tells tcpdump to write pcap data to stdout, piped to Wireshark’s stdin; -k -i - tells Wireshark to read from stdin and start immediately. The latency is about 1-2 seconds in practice — very handy for real-time troubleshooting.

Wireshark Display Filters

Note: Wireshark’s display filter syntax is completely different from tcpdump’s BPF syntax. This is the most common source of confusion for beginners.

Common Wireshark display filters:

# HTTP requests only
http.request

# TLS handshake only
tls.handshake

# TCP retransmissions only
tcp.analysis.retransmission

# TCP duplicate ACKs only
tcp.analysis.duplicate_ack

# Specific HTTP path
http.request.uri contains "api/order"

# Specific TCP stream
tcp.stream eq 42

# Connection resets
tcp.flags.reset == 1

Right-Click Follow TCP Stream

This is Wireshark’s most used feature. Right-click any TCP packet → Follow → TCP Stream, and Wireshark assembles all packets from that TCP connection in chronological order into a conversation — client data on the left, server data on the right.

For plaintext protocols like HTTP, this lets you read the complete request-response exchange like a chat log. But TLS traffic is encrypted, so following a TCP stream only shows encrypted data.

Protocol Analysis in Practice

What the TCP Three-Way Handshake Looks Like in a Capture

A normal TCP connection establishment looks like this in a capture:

10:30:01.123456 IP 10.0.0.5.54321 > 10.0.0.6.443: Flags [S], seq 1234567890, win 64240, [mss 1460,sackOK,TS val 1234567890 ecr 0,nop,wscale 7], length 0
10:30:01.123567 IP 10.0.0.6.443 > 10.0.0.5.54321: Flags [S.], seq 9876543210, ack 1234567891, win 65160, [mss 1460,sackOK,TS val 9876543210 ecr 1234567890,nop,wscale 7], length 0
10:30:01.123578 IP 10.0.0.5.54321 > 10.0.0.6.443: Flags [.], ack 9876543211, win 502, length 0

Interpretation:

Flag quick reference:

• [S] = SYN
• [.] = ACK
• [S.] = SYN + ACK
• [P.] = PSH + ACK (has data to send)
• [F.] = FIN + ACK (I want to close)
• [R.] = RST + ACK (connection error, reset)
• [R] = RST (direct reset, no ACK)

Diagnosing TCP Connection Timeouts

If a service is timing out but you do not know which layer is the problem, capture SYN packets first:

sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & tcp-syn != 0' -c 100

Normally, every SYN should be followed by a SYN-ACK. If you see SYN without SYN-ACK, here is what it means:

Diagnosing TLS Handshake Failures

TLS handshake is far more complex than TCP. A complete TLS 1.2 handshake requires 2 RTTs:

# Capture TLS handshake traffic
sudo tcpdump -i eth0 -nn -s 0 -w tls.pcap 'tcp port 443'

Open in Wireshark, filter for tls.handshake, and you will see:

Common TLS troubleshooting:

Scenario 1: Expired Certificate

In Wireshark, filter tls.handshake.type == 11 (Certificate), expand the certificate details, and check the validity.not_after field. If the certificate has expired, that is your problem.

You can also check from the command line with tshark:

# Use tshark to parse TLS certificates
tshark -r tls.pcap -Y "tls.handshake.type == 11" -T fields -e tls.handshake.certificate

Scenario 2: SNI Mismatch

SNI (Server Name Indication) is the target domain the client carries in ClientHello. If the server has multiple virtual hosts but SNI is misconfigured, it may return the wrong certificate. Filter tls.handshake.type == 1 (ClientHello) to check the SNI field:

tshark -r tls.pcap -Y "tls.handshake.type == 1" -T fields -e tls.handshake.extensions_server_name

Scenario 3: Cipher Suite Negotiation Failure

If ClientHello is sent and an Alert message comes back immediately, the cipher suites did not agree. Check whether the ClientHello’s Cipher Suites list and the server’s supported list have an empty intersection.

HTTP Traffic Analysis

Although most traffic uses HTTPS now, internal service-to-service HTTP communication is still common. The -A flag lets you see HTTP content directly:

sudo tcpdump -i eth0 -nn -A -s 0 'tcp port 80 and host 10.0.0.5'

The output looks like this:

GET /api/v1/orders?status=pending HTTP/1.1
Host: order-service.internal
User-Agent: curl/7.81.0
Accept: */*
Connection: keep-alive

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 142

{"orders": [...]}

But do not leave -A output running in a production terminal long-term — it is not only slow but may expose sensitive information (tokens, passwords) in terminal logs.

DNS Troubleshooting

DNS is a high-incidence area for network problems. A DNS query capture is straightforward:

sudo tcpdump -i eth0 -nn -l 'udp port 53' | grep -E 'A\?|AAAA\?'

-l enables line-buffered mode for real-time output. A? indicates an A record query; AAAA? indicates an IPv6 query.

# Capture DNS queries and responses
sudo tcpdump -i eth0 -nn 'udp port 53' -c 20

Example output:

12:30:01.123 IP 10.0.0.5.43210 > 8.8.8.8.53: 45231+ A? api.example.com. (33)
12:30:01.156 IP 8.8.8.8.53 > 10.0.0.5.43210: 45231 2/0/0 A 93.184.216.34, A 93.184.216.35 (69)

• 45231+ — the + indicates a recursive query
• 2/0/0 — 2 answer records, 0 authority records, 0 additional records
• If the response is 0/0/0 or 0/1/0 (NXDOMAIN), the domain does not exist

Case Studies

Case 1: Intermittent 502 Between Services

Symptom: Service A calls Service B’s HTTP API and gets intermittent 502 Bad Gateway, at a rate of about 1 in 1000.

Investigation:

# Capture on Service B's machine
sudo tcpdump -i eth0 -nn -s 0 -w 502.pcap 'host <Service-A-IP> and tcp port 8080'

After reproducing the issue, opened in Wireshark with filter tcp.analysis.retransmission or tcp.analysis.duplicate_ack or tcp.flags.reset == 1, and found:

1. TCP three-way handshake was normal
2. After the client sent the HTTP request, the server ACKed but did not return data
3. The client waited 15 seconds, then sent FIN to close the connection
4. Nginx (reverse proxy) returned 502 after Service B timed out

Root cause: Service B’s connection pool was too small. During peak hours, connections were queued, exceeding Nginx’s proxy_read_timeout. No network change needed — just increase the connection pool.

Case 2: Cross-Datacenter Latency Spike

Symptom: RPC calls from Beijing datacenter to Shanghai datacenter went from 30ms to 200ms+.

# Bidirectional capture
sudo tcpdump -i eth0 -nn -s 0 -w latency.pcap 'host <Shanghai-DC-IP>'

Opened in Wireshark and used tcp.analysis.ack_rtt to check RTT distribution. Found some packets with ACK RTT over 200ms, and these packets had TTL values 3 hops higher than normal packets.

Root cause: Dynamic routing policy change caused some traffic to take an extra 3 hops, increasing latency. Fixed after the network team corrected the routing policy.

Case 3: ARP Spoofing Causing Intermittent Network Outages

Symptom: A server intermittently could not reach the gateway, each outage lasting 10-30 seconds before self-recovery.

# Capture ARP packets
sudo tcpdump -i eth0 -nn arp

Under normal conditions, ARP requests are not frequent within the cache validity period. But the capture showed:

14:20:01 IP (via eth0) arp who-has 10.0.0.1 tell 10.0.0.5
14:20:01 IP (via eth0) arp reply 10.0.0.1 is-at 00:11:22:33:44:55  ← normal gateway MAC
14:20:03 IP (via eth0) arp reply 10.0.0.1 is-at 66:77:88:99:aa:bb  ← abnormal MAC!

Root cause: Another machine had a misconfigured IP, and its ARP response preemptively overwrote the correct gateway MAC address, redirecting traffic to the wrong machine. Fixed after correcting that machine’s IP configuration.

tshark: Command-Line Wireshark

Not every scenario allows a GUI. tshark is the command-line version of Wireshark, with full protocol dissection capability, suitable for analyzing pcap files directly on servers.

# Install
sudo apt-get install -y tshark   # Debian/Ubuntu
sudo yum install -y wireshark     # CentOS/RHEL (includes tshark)

# Protocol hierarchy statistics
tshark -r capture.pcap -q -z io,phs

# Extract all HTTP request URLs
tshark -r capture.pcap -Y "http.request" -T fields -e http.host -e http.request.uri

# Count TCP retransmissions
tshark -r capture.pcap -Y "tcp.analysis.retransmission" | wc -l

# View TLS SNI
tshark -r capture.pcap -Y "tls.handshake.type == 1" -T fields -e tls.handshake.extensions_server_name

# Top 10 IPs by packet count
tshark -r capture.pcap -q -z conv,ip | sort -t'|' -k1 -rn | head -10

Advanced Techniques

Using ngrep for Content Matching

If you only want to see traffic containing a specific string, ngrep is more convenient than tcpdump:

# Install
sudo apt-get install -y ngrep

# Capture HTTP traffic containing "error"
sudo ngrep -d eth0 -W byline 'error' 'tcp port 80'

# Capture requests containing a specific token
sudo ngrep -d eth0 -W byline 'Authorization: Bearer eyJ...' 'tcp port 443'

Note that ngrep only works on plaintext protocols. HTTPS traffic is encrypted; ngrep cannot see the content.

Combined ss + tcpdump Diagnosis

Check current connection state first, then capture targeted traffic:

# View all connections to 10.0.0.6
ss -tn state established '( dst 10.0.0.6 )'

# Count TIME-WAIT connections
ss -tn state time-wait | wc -l

# Count SYN-RECV connections (possible SYN flood)
ss -tn state syn-recv | wc -l

If you find a large number of SYN-RECV connections, it means the server received SYN but did not get the third ACK. This could be an attack or packet loss in the intermediate network.

Capture Performance Benchmarking

On a 10GbE NIC, capture performance depends on BPF filter complexity and snaplen:

# Use dropwatch to check BPF filter drops
sudo dropwatch -l kas

# If you see many "kfree_skb" at __netif_receive_skb_core,
# the kernel is dropping before BPF filtering — reduce capture volume

If you genuinely need full capture but performance cannot keep up, consider PF_RING or AF_PACKET mmap mode. These techniques bypass kernel copying and DMA directly from the NIC to user-space memory. Facebook’s fbtaxii system and Netflix’s internal tools use similar approaches.

Production Environment Considerations

1. Do not capture all traffic. Full capture on a 10GbE NIC generates tens of GB in minutes, and production traffic may contain sensitive information (API tokens, user data). Precise filtering is your first line of defense.

2. pcap files contain sensitive information. pcap files include complete network data — everything outside HTTPS is plaintext. Do not leave pcap files in publicly accessible locations; do not transmit them through insecure channels. Delete after use.

3. Use rotation for long captures. Combine -G + -W for time-based rotation and file count limits to prevent disk exhaustion.

4. Capturing host traffic in container environments. Capturing inside a container only sees traffic within the container’s network namespace. To capture host traffic, use hostNetwork: true or run tcpdump directly on the host.

# Capture inside a K8s Pod (requires hostNetwork or privileged mode)
kubectl exec -it <pod> -- tcpdump -i eth0 -nn -c 100

# Capture Pod traffic from the host
# First find the Pod's IP
kubectl get pod <pod> -o jsonpath='{.status.podIP}'
# Then capture traffic for that IP on the host
sudo tcpdump -i eth0 -nn 'host <pod-IP>'

5. Timezone issues. tcpdump timestamps default to local time. If the server is in a different timezone than you, timestamps will be off by several hours. Use -tttt to display the date:

# Display full date and time
sudo tcpdump -i eth0 -nn -tttt

# Output: 2026-07-14 10:30:01.123456 IP ...

Summary

tcpdump and Wireshark are the core tool combo for SREs troubleshooting network problems. tcpdump handles efficient capture on production servers; Wireshark handles deep analysis locally. The workflow is simple: capture on the server → scp download → analyze in Wireshark.

Key takeaways:

• BPF filters are your first line of defense — they reduce irrelevant traffic and prevent disk exhaustion. Always quote parentheses in filter expressions.
• TCP flag filtering is key to diagnosing connection issues. SYN to see who initiates, RST to see who rejects, FIN to see who closes.
• -nn is a production must — it avoids the performance overhead and noise of DNS reverse lookups.
• -s 0 ensures TLS handshake packets are not truncated — large certificate chains often exceed 1500 bytes.
• pcap files contain sensitive information — everything outside HTTPS is plaintext. Delete after use.
• tshark replaces Wireshark in headless environments — full protocol dissection and field extraction.
• Long captures must use -G + -W rotation — otherwise you will be woken up at 2 AM not by a business outage, but by a full disk.

There is no shortcut to improving packet capture skills — just practice. Next time you hit a network issue, do not rush to restart services. Capture some packets first — eight times out of ten, the answer is in the pcap.

References and Acknowledgments

The following resources were consulted during the writing of this article. Credit to the original authors:

1. tcpdump Official Manual — tcpdump.org, the authoritative documentation for BPF filter syntax and command parameters
2. Wireshark User’s Guide — Wireshark.org, complete documentation for display filters and protocol dissection
3. tcpdump Practical Capture Techniques — CSDN, production environment tcpdump usage experience and troubleshooting cases
4. Network Analysis Duo: tcpdump Combined with Wireshark Protocol Analysis Guide — CSDN, practical summary of the tcpdump and Wireshark collaborative workflow
5. Network Security Analysis: tcpdump Command Precision pcap Filtering — CSDN, advanced BPF filtering techniques and pcap file analysis