Overview
It’s 2 AM and alerts are firing. Your service can’t connect to the database, curl times out, but ping to the IP works fine. Is it a network issue or an application issue? Nine times out of ten, it’s DNS resolution gone wrong.
DNS resolution is the most fundamental yet most overlooked piece of Linux infrastructure. Nobody thinks about it until it breaks—and when it does, the entire system acts like it suddenly has amnesia: every domain becomes unrecognizable. What makes it worse is that DNS problems manifest in wildly different ways: some throw Temporary failure in name resolution, some mysteriously hang for 5 seconds, and some fail intermittently. Without understanding the resolution chain, troubleshooting becomes a guessing game.
This article breaks down the complete Linux DNS resolution chain: from /etc/hosts to nsswitch, from resolv.conf to systemd-resolved, from dig to tcpdump. Each component comes with real-world examples. By the end, you’ll be able to independently troubleshoot DNS issues in production.
The Complete Linux DNS Resolution Chain
Let’s address one question first: when you type curl http://example.com in the terminal, what exactly does the system do?
DNS resolution doesn’t happen in a single step. It goes through multiple layers. Understanding this chain is the prerequisite for troubleshooting. Think of it like sending a package: you write “Chaoyang District, Beijing” as the address, but the courier doesn’t inherently know where that is—it first checks the postal code zone, then the street, then the building number. DNS works the same way, searching layer by layer.
Resolution Priority
When a Linux system resolves a domain name, it follows this default order:
- Local hosts file (
/etc/hosts)—highest priority, returns immediately on match - DNS cache—if nscd or systemd-resolved is installed, the cache is checked first
- Configured DNS servers (nameserver in
/etc/resolv.conf)—network query is the last resort
But this order isn’t hardcoded. The actual resolution order is controlled by the hosts line in /etc/nsswitch.conf:
# View nsswitch configuration
cat /etc/nsswitch.conf | grep hosts
# Typical output
hosts: files dns myhostname
This line means: check files (/etc/hosts) first, then dns (DNS servers), and finally use the myhostname mechanism. If the order were dns files, the system would query DNS first, which changes performance characteristics entirely.
Gotcha: Some systems have nsswitch.conf with a hosts line missing the
dnskeyword. The system never performs DNS queries and only recognizes records in the hosts file. The symptom:digworks fine butcurlfails.
A Complete DNS Query Flow
Taking curl http://example.com as an example, the full flow:
Application (curl)
│
▼
glibc getaddrinfo() ← C standard library resolution entry point
│
├─→ /etc/nsswitch.conf ← Determines query order
│
├─→ /etc/hosts ← Step 1: Check local hosts file
│ (Return immediately if matched)
│
├─→ nscd / systemd-resolved ← Step 2: Check local cache
│ (Return immediately if cache hit)
│
└─→ /etc/resolv.conf ← Step 3: Perform DNS query
│
▼
nameserver-specified DNS server
│
▼
Recursive query: root → TLD → authoritative server
│
▼
Return IP address
Key insight: dig and nslookup do not use glibc. They directly construct DNS protocol packets and send them to a specified DNS server. This is why sometimes dig works but curl fails—the two take different paths.
resolv.conf Behavior Details
/etc/resolv.conf is the core configuration file for DNS queries. It looks simple, but there’s more to it than meets the eye:
# Typical resolv.conf
nameserver 8.8.8.8
nameserver 1.1.1.1
search example.com internal.example.com
options timeout:2 attempts:3
| Parameter | Description | Default | Production Recommendation |
|---|---|---|---|
nameserver | DNS server address | None | Configure at least 2 |
search | Short domain suffix completion | None | Configure per actual domain |
domain | Local domain | None | Mutually exclusive with search |
options timeout | Single query timeout (seconds) | 5 | Recommend setting to 2 |
options attempts | Retry count | 2 | Recommend setting to 3 |
options rotate | Round-robin across nameservers | Off | Enable for HA scenarios |
The most insidious behavior: nameserver supports up to 3 IPv4 addresses (kernel limit), and anything beyond the third is silently discarded. It’s also not load-balanced—only when the first one times out does it fall back to the second. If the first nameserver is an internal DNS that’s unreachable, every query waits 5 seconds for timeout before switching.
War story: Once in production, all requests were hanging for 5 seconds. Investigation revealed the first nameserver in resolv.conf pointed to a decommissioned DNS server. Reordering to put the working one first solved it instantly.
systemd-resolved: Friend or Foe
Modern Linux distributions (Ubuntu 18.04+, RHEL 8+, Fedora 33+) enable systemd-resolved by default, which takes over DNS resolution. The upside: DNS caching, DNS-over-TLS, per-interface DNS configuration. The downside: troubleshooting adds an extra middleman.
First, check if the system is using systemd-resolved:
# Method 1: Check if resolv.conf is a symlink
ls -l /etc/resolv.conf
# If the output looks like this, systemd-resolved has taken over
# lrwxrwxrwx 1 root root 39 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
# Method 2: Check service status
systemctl is-active systemd-resolved
# Method 3: View actual DNS configuration (this is the real one)
resolvectl status
The resolvectl status output is information-rich:
Global
Protocols: LLMNR=resolve -fL MDNS=resolve -fL DNSOverTLS=no
resolv.conf mode: stub
Current DNS Server: 8.8.8.8
DNS Servers: 8.8.8.8 8.8.4.4
DNS Domain: example.com
Link 2 (eth0)
Current Scopes: DNS
Protocols: LLMNR=resolve MDNS=resolve DNSOverTLS=no
Current DNS Server: 192.168.1.1
DNS Servers: 192.168.1.1
DNS Domain: internal.example.com
Classic trap: systemd-resolved process is running, occupying
127.0.0.53:53, but hasn’t actually configured an upstream DNS or is in a failed state. resolv.conf points to it, and all requests hang at connect timeout (default 5 seconds). You manually edit resolv.conf tonameserver 8.8.8.8, but it quietly overwrites your change.
Solutions:
# Check real configuration—if it shows "No current DNS server", that's the problem
resolvectl status
# Temporary workaround: switch to stub mode
sudo ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
# Or disable systemd-resolved entirely (traditional mode)
sudo systemctl disable --now systemd-resolved
sudo rm /etc/resolv.conf
echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf
Troubleshooting Toolset
DNS troubleshooting has a mature toolset. You won’t use every tool every time, but you need to know which tool fits which scenario.
dig: The Primary DNS Troubleshooting Weapon
dig (Domain Information Groper) is the most powerful DNS troubleshooting tool. It directly constructs DNS protocol packets and sends them to a specified server, bypassing the system resolver.
Basic query:
# Basic query showing full resolution process
dig example.com
# Key output fields:
# ;; ->>HEADER<<- opcode: QUERY, status: NOERROR
# ;; QUESTION SECTION: ← What you queried
# ;; ANSWER SECTION: ← What was returned
# ;; SERVER: 8.8.8.8#53 ← DNS server that actually responded
# ;; Query time: 23 msec ← Response time
Query a specific DNS server to eliminate local configuration interference:
# Query using 8.8.8.8, bypassing local DNS configuration
dig @8.8.8.8 example.com
# Query using 1.1.1.1 to compare results
dig @1.1.1.1 example.com
# If two public DNS servers return different results, local DNS may be hijacked
Query specific record types:
# A record (IPv4 address)
dig example.com A
# AAAA record (IPv6 address)
dig example.com AAAA
# MX record (mail server)
dig example.com MX
# TXT record (text record, often used for verification)
dig example.com TXT
# NS record (name server)
dig example.com NS
# SOA record (start of authority)
dig example.com SOA
Trace the complete resolution path, seeing each hop from root to final result:
dig +trace example.com
# Sample output (simplified):
# ;; QUERY: 1, ANSWER: 13
# . 518400 IN NS a.root-servers.net. ← Root servers
# com. 172800 IN NS a.gtld-servers.net. ← .com TLD
# example.com. 172800 IN NS a.iana-servers.net. ← Authoritative server
# example.com. 300 IN A 93.184.216.34 ← Final result
+trace is a powerful tool for troubleshooting DNS chain issues. If a hop times out, you can pinpoint exactly which layer has the problem.
Reverse query (IP to domain name):
# Reverse DNS query
dig -x 8.8.8.8
# Output:
# ;; ANSWER SECTION:
# 8.8.8.8.in-addr.arpa. 86400 IN PTR dns.google.
DNSSEC query to verify if a domain has DNS signing enabled:
dig +dnssec example.com
# If the response contains the ad flag (Authenticated Data), DNSSEC is enabled
Minimal output, just the IP address:
dig +short example.com
# 93.184.216.34
nslookup: Lightweight Tool for Quick Verification
nslookup has simpler functionality than dig, but its interactive mode is convenient for batch queries:
# Basic query
nslookup example.com
# Specify DNS server
nslookup example.com 8.8.8.8
# Query specific record type
nslookup -type=MX example.com
# Interactive mode (can query continuously)
nslookup
> server 8.8.8.8
> example.com
> example.com MX
> exit
Key difference between dig and nslookup: nslookup defaults to the system resolver (affected by nsswitch.conf and systemd-resolved), while dig defaults to directly connecting to a DNS server. Comparing the two during troubleshooting reveals many hidden issues. If nslookup is slow but dig is fast, the local resolution chain has a problem.
host: The Most Concise Query Tool
# Basic query, most concise output
host example.com
# Query specific records
host -t MX example.com
host -t NS example.com
# Verbose output
host -v example.com
# Reverse query
host 8.8.8.8
resolvectl: systemd-resolved专用工具
If the system uses systemd-resolved, resolvectl is a must-know tool:
# View DNS configuration status
resolvectl status
# Query a specific domain
resolvectl query example.com
# View cache statistics
resolvectl statistics
# Clear DNS cache
resolvectl flush-caches
# Reset link configuration
resolvectl revert eth0
tcpdump: The Ultimate Packet Capture Tool
When all other tools fail to identify the problem, capture packets directly to see actual DNS traffic:
# Capture DNS traffic (UDP port 53)
sudo tcpdump -i any -n port 53 -vv
# Capture DNS queries for a specific domain
sudo tcpdump -i any -n port 53 -vv | grep "example.com"
# Capture DNS responses, see returned IPs
sudo tcpdump -i any -n port 53 -vv -X
# Save to file for Wireshark analysis
sudo tcpdump -i any -n port 53 -w dns.pcap
Pro tip: Use tcpdump to confirm whether DNS query packets are actually being sent, to whom, and whether responses are received. If no query packets are going out at all, the problem is at the application layer or glibc; if packets are sent but no response comes back, the problem is with the network or DNS server.
Common Failure Scenarios and Troubleshooting Workflows
Scenario 1: Domain Completely Unresolvable
Symptom: curl reports Temporary failure in name resolution or Could not resolve host.
Troubleshooting flow:
# Step 1: Confirm it's a DNS issue (IP works but domain doesn't)
ping -c 2 8.8.8.8 # IP works
ping -c 2 google.com # Domain doesn't work → DNS issue
# Step 2: Check resolv.conf
cat /etc/resolv.conf
# Confirm there's a valid nameserver
# Step 3: Test directly against a DNS server
dig @8.8.8.8 google.com
# If direct query succeeds but system resolution fails → local config issue
# If direct query also fails → network issue or DNS server unreachable
# Step 4: Check network connectivity
ping -c 2 $(awk '/^nameserver/{print $2; exit}' /etc/resolv.conf)
# If DNS server is unreachable → network routing issue
# Step 5: Check if firewall blocks UDP port 53
sudo iptables -L -n | grep 53
# or
sudo ufw status
Scenario 2: DNS Resolution is Extremely Slow
Symptom: All requests mysteriously hang, occasional timeouts. SSH login is also slow.
This is the most common hidden DNS problem. Causes typically include:
Cause 1: First nameserver in resolv.conf is unreachable
# Test response time of each nameserver
for ns in $(awk '/^nameserver/{print $2}' /etc/resolv.conf); do
echo -n "$ns: "
dig @$ns example.com +stats 2>/dev/null | grep "Query time" || echo "timeout"
done
# If the first one times out, modify resolv.conf to reorder
# Or add options timeout:1 to reduce timeout wait
Cause 2: IPv6 resolution issues
Many applications try IPv6 (AAAA records) first. If the system doesn’t support IPv6 or IPv6 DNS is unreachable, it waits for the AAAA query to time out before falling back to A record queries.
# Check if IPv6 is enabled
ip -6 addr show
# If IPv6 is not needed, disable it to avoid AAAA query timeouts
# Method 1: Add options in resolv.conf (not recommended, may be overwritten)
# Method 2: Disable IPv6 at the application layer
# Disable IPv6 for curl
curl -4 http://example.com
# Disable IPv6 for wget
wget -4 http://example.com
Cause 3: Extra queries caused by ndots configuration
The options ndots:N parameter in resolv.conf controls when to perform fully qualified domain name queries. The default is 1 (older systems) or 5 (systemd-resolved systems).
# View ndots configuration
cat /etc/resolv.conf | grep options
# If ndots:5, it means:
# When querying a short name like "db", the system first tries:
# db.search-domain1.com ← First tries search suffix completion
# db.search-domain2.com
# ...
# db ← Finally queries the bare domain
# Each failed completion generates a DNS query, totaling 6 queries
This problem is especially severe in Kubernetes environments. A Pod’s default resolv.conf looks like this:
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
This means querying example.com (only 1 dot, less than 5) will first attempt:
# Actually triggers 4 unnecessary DNS queries
example.com.default.svc.cluster.local → NXDOMAIN
example.com.svc.cluster.local → NXDOMAIN
example.com.cluster.local → NXDOMAIN
example.com ← Finally queries the real domain
Production lesson: In a K8s cluster, an application was slow making external requests. Investigation revealed 3 extra DNS queries per request. Solution: either modify dnsPolicy and dnsConfig at the Pod level, or use fully qualified domain names (append
.suffix, e.g.,example.com.) for external domains.
Scenario 3: Intermittent Resolution Failures
Symptom: Works sometimes, fails sometimes, no pattern. This is the hardest to troubleshoot.
# Continuously monitor DNS resolution, recording failure times
while true; do
timestamp=$(date '+%Y-%m-%d %H:%M:%S')
result=$(dig +short +time=2 +tries=1 example.com 2>&1)
if [ -z "$result" ] || echo "$result" | grep -q "timed out"; then
echo "[$timestamp] FAILED: $result"
else
echo "[$timestamp] OK: $result"
fi
sleep 5
done
Common causes of intermittent failures:
- DNS server overload: Multiple machines querying the same DNS server simultaneously, peak response times exceed capacity
- UDP packet loss: DNS defaults to UDP, high packet loss rates cause intermittent failures
- Connection tracking table full: nf_conntrack table being full causes UDP packets to be dropped
- DNS server QPS exceeded: Single DNS server exceeding its query rate limit
# Check connection tracking table
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
# If count is close to max, increase the limit
echo 262144 | sudo tee /proc/sys/net/netfilter/nf_conntrack_max
Scenario 4: nslookup Works but curl Fails
This is the most confusing scenario. Root cause: nslookup/dig directly connect to DNS servers, while curl goes through the system glibc getaddrinfo(), which is affected by nsswitch.conf and systemd-resolved.
# Step 1: Confirm nsswitch configuration
cat /etc/nsswitch.conf | grep hosts
# Ensure it includes dns:
# hosts: files dns myhostname
# Step 2: If using systemd-resolved, check if it's healthy
resolvectl status
systemctl status systemd-resolved
# Step 3: Check if nscd (Name Service Cache Daemon) is interfering
systemctl status nscd
# nscd's cache may be stale but not refreshed, returning old data
sudo nscd -i hosts # Clear hosts cache
Scenario 5: resolv.conf Automatically Overwritten
You manually modified /etc/resolv.conf, but the configuration disappears after a network restart or machine reboot. This is the classic DNS pitfall.
# First check if resolv.conf is a symlink
ls -l /etc/resolv.conf
# If it's a link to systemd-resolved:
# lrwxrwxrwx ... /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf
# Don't edit it directly; modify systemd-resolved configuration instead
# Modify systemd-resolved configuration
sudo vim /etc/systemd/resolved.conf
# [Resolve]
# DNS=8.8.8.8 1.1.1.1
# FallbackDNS=8.8.4.4
# Domains=example.com
# Restart the service to apply
sudo systemctl restart systemd-resolved
For systems managed by NetworkManager:
# View current connections
nmcli connection show
# Modify DNS configuration
nmcli connection modify "Wired connection 1" ipv4.dns "8.8.8.8 1.1.1.1"
nmcli connection modify "Wired connection 1" ipv4.ignore-auto-dns yes
# Restart the connection to apply
nmcli connection down "Wired connection 1" && nmcli connection up "Wired connection 1"
For DHCP-obtained DNS:
# Prevent DHCP from overriding DNS configuration
# In /etc/dhcp/dhclient.conf, add:
# prepend domain-name-servers 8.8.8.8 1.1.1.1;
# Or completely override:
# supersede domain-name-servers 8.8.8.8 1.1.1.1;
Production DNS Configuration Best Practices
Server DNS Configuration Recommendations
# /etc/resolv.conf recommended configuration (non-systemd-resolved environments)
nameserver 8.8.8.8
nameserver 1.1.1.1
search internal.example.com
options timeout:2 attempts:3 rotate
# timeout:2 → 2 second timeout, don't wait the default 5 seconds
# attempts:3 → Maximum 3 retries
# rotate → Round-robin across nameservers, avoid overloading the first one
# /etc/systemd/resolved.conf recommended configuration (systemd-resolved environments)
[Resolve]
DNS=8.8.8.8 1.1.1.1
FallbackDNS=8.8.4.4 9.9.9.9
Domains=example.com
DNSOverTLS=opportunistic
Cache=yes
CacheFromLocalhost=no
Kubernetes Pod DNS Configuration
# For Pods with frequent external requests, customize DNS config to avoid ndots issues
apiVersion: v1
kind: Pod
metadata:
name: external-api-client
spec:
dnsPolicy: None # Disable default DNS policy
dnsConfig:
nameservers:
- 8.8.8.8
- 1.1.1.1
searches:
- default.svc.cluster.local
- svc.cluster.local
options:
- name: ndots
value: "2" # Lower ndots to avoid unnecessary queries
- name: timeout
value: "2"
- name: attempts
value: "3"
containers:
- name: app
image: myapp:latest
DNS Monitoring Metrics
Production environments should monitor DNS resolution health:
# Bash script: DNS health check (can be integrated with Prometheus Node Exporter textfile collector)
#!/bin/bash
# dns-health-check.sh
METRICS_FILE="/var/lib/node_exporter/textfile/dns_health.prom"
# Test DNS resolution latency
for domain in "internal.example.com" "google.com" "kubernetes.default.svc"; do
start=$(date +%s%N)
result=$(dig +short +time=2 +tries=1 "$domain" 2>/dev/null)
end=$(date +%s%N)
latency_ms=$(( (end - start) / 1000000 ))
if [ -n "$result" ]; then
status=1
else
status=0
latency_ms=9999
fi
# Generate metric (replace special characters in domain name with underscores)
safe_domain=$(echo "$domain" | tr '.' '_')
echo "dns_resolve_success{domain=\"${safe_domain}\"} ${status}" >> "$METRICS_FILE"
echo "dns_resolve_latency_ms{domain=\"${safe_domain}\"} ${latency_ms}" >> "$METRICS_FILE"
done
DNS Cache Management
# Clear systemd-resolved cache
resolvectl flush-caches
# Clear nscd cache
sudo nscd -i hosts
# Clear dnsmasq cache
sudo systemctl restart dnsmasq
# Clear BIND cache
sudo rndc flush
Cache strategy recommendation: Servers should have DNS caching enabled (systemd-resolved enables it by default), with TTL following the upstream DNS setting. Don’t set the cache TTL too high, or domain changes won’t take effect promptly. I’ve seen someone set the cache TTL to 24 hours—after a DNS switch, half the data center’s applications were still connecting to the old IP, and it took a full day to troubleshoot.
DNS Troubleshooting Quick Reference
| Symptom | First Check | Possible Cause | Command |
|---|---|---|---|
| Domain completely unresolvable | resolv.conf | nameserver empty or wrong | dig @8.8.8.8 domain |
| Slow resolution (5s hang) | First nameserver in resolv.conf | DNS server unreachable | for ns in ...; do dig @$ns ... |
| nslookup works but curl fails | nsswitch.conf | hosts line missing dns | cat /etc/nsswitch.conf | grep hosts |
| resolv.conf overwritten | Is it a symlink | systemd-resolved/NetworkManager | ls -l /etc/resolv.conf |
| Intermittent failures | conntrack/UDP loss | nf_conntrack table full | cat /proc/sys/net/netfilter/nf_conntrack_count |
| K8s Pod slow resolution | ndots config | ndots:5 causing extra queries | Check Pod resolv.conf |
| IPv6 causing slowness | AAAA query timeout | System doesn’t support IPv6 but tries AAAA | ip -6 addr show |
| DNS hijacking | dig vs nslookup results | ISP DNS pollution | dig @8.8.8.8 vs dig @local |
Troubleshooting Flowchart
Domain resolution failure
│
├─ Does ping to IP work?
│ ├─ No → Network issue, not DNS
│ └─ Yes ↓
│
├─ Does dig @8.8.8.8 domain work?
│ ├─ Yes → Local configuration issue
│ │ ├─ Check /etc/resolv.conf
│ │ ├─ Check /etc/nsswitch.conf
│ │ └─ Check systemd-resolved
│ └─ No → DNS server or network issue
│ ├─ Ping DNS server
│ ├─ Check firewall UDP 53
│ └─ tcpdump to capture packets
│
└─ Intermittent issues
├─ Check nf_conntrack
├─ Check DNS server load
└─ Continuously monitor and record failures
Summary
The core of DNS troubleshooting is understanding the resolution chain. The full chain: application → glibc → nsswitch → hosts/cache/DNS → resolv.conf → nameserver. Each link can break, so troubleshoot by eliminating each segment along the chain.
A few practical takeaways:
Comparing dig and nslookup results is the first line of defense: If dig succeeds but nslookup fails (or vice versa), you immediately know whether the problem is the local resolver or the DNS server
systemd-resolved is a double-edged sword: It provides caching and flexible configuration, but adds a middleman during troubleshooting. In production, if you don’t need its features, consider using traditional resolv.conf mode—simple, blunt, but easy to debug
The order of nameservers in resolv.conf matters: If the first one is unreachable, it slows down all queries. Add
options timeout:2to reduce timeout from 5 to 2 seconds, addrotateto distribute loadK8s ndots:5 is a massive trap: For Pods with frequent external requests, customize dnsConfig to lower ndots, or every request triggers 3-4 unnecessary DNS queries
When in doubt, capture packets: tcpdump is the ultimate tool. See whether DNS packets are actually going out, to whom, and whether responses come back. Many bizarre issues become clear with a packet capture
Account for caching on DNS changes: After switching DNS servers or modifying domain records, remember to clear local cache. Use
resolvectl flush-cachesfor systemd-resolved,nscd -i hostsfor nscd
DNS seems simple, but the troubleshooting chain is long when things go wrong. Memorize this toolset and workflow, and the next time you’re woken up by alerts at 2 AM, you won’t be scrambling.
References & Acknowledgments
This article referenced the following materials during writing. Thanks to the original authors for their contributions:
- Linux DNS 故障排查与解析命令完整指南 — CSDN, detailed guide on dig/nslookup/host command usage
- Linux 域名解析缓慢的定位流程 — php.cn, analysis of nslookup vs dig response time differences and systemd-resolved troubleshooting
- Linux查看DNS配置教程 resolv.conf文件详解 — php.cn, detailed explanation of resolv.conf parameters and dynamic management mechanisms
- Linux怎么诊断DNS解析失败 — php.cn, systematic approach to diagnosing DNS resolution failures
- Linux如何排查DNS解析异常 — php.cn, dig and nslookup diagnostic tutorial