Linux Time Synchronization and NTP: chrony Configuration, Troubleshooting, and Production Practices

Overview

One production incident made me take time synchronization seriously. In the middle of the night, the business team reported mass HTTPS request failures with SSL_ERROR_BAD_CERTIFICATE. We spent two hours troubleshooting—combed through network topology, firewall rules, certificate configs—nothing. Finally, someone happened to glance at the system clock: it was 12 seconds ahead of real time. Those 12 seconds made the server think client certificates hadn’t reached their notBefore time yet, so it rejected every handshake.

This isn’t an isolated case. Time desync is an invisible landmine in distributed systems: Kerberos authentication tolerates only 5 minutes of clock skew, beyond which it refuses logins outright. Kubernetes nodes with excessive clock drift get their kubelet heartbeats marked stale and Pods get evicted. MySQL replication relies on timestamps; large drift corrupts binlog ordering. Distributed two-phase commits fail their timeout checks when nodes’ clocks disagree.

Time sync sounds simple—install an NTP client, point it at a time server, done. But production is full of pitfalls: VM clock drift, no NTP daemon in containers, multi-DC source selection, chrony/ntpd conflicts, cert renewal failures from time jumps. This article tackles all of them, with production-ready configs and troubleshooting approaches.

From ntpd to chrony: Why the Switch

ntpd is the veteran time sync service, running for over twenty years. But starting with RHEL 8, CentOS 8, and Ubuntu 20.04, mainstream distros default to chrony. It’s not that ntpd is bad—it’s that chrony performs better in several key scenarios:

The key difference is initial sync speed. ntpd, by design philosophy of “never let time jump,” gradually approaches the target time after startup, potentially taking over ten minutes to stabilize. chrony’s iburst (initial burst) option sends 4 probe packets at startup, completing initial sync in seconds. This is critical for VMs and containers that start and stop frequently—you can’t wait 10 minutes for a container’s clock to be correct.

When you still need ntpd: Legacy system compatibility, broadcast/multicast mode requirements, special hardware timestamp (PTP) needs. For new deployments, chrony is the recommendation.

chrony Client Configuration

Basic Setup

# Install (pre-installed on most distros)
# RHEL/CentOS/Fedora
sudo dnf install chrony -y
# Debian/Ubuntu
sudo apt install chrony -y

# Start and enable on boot
sudo systemctl enable --now chronyd

Core config file /etc/chrony.conf, production-recommended configuration:

# /etc/chrony.conf

# === Time Sources ===
# Domestic recommendation: Alibaba Cloud NTP (low latency, stable), multi-source redundancy
server ntp.aliyun.com iburst minpoll 4 maxpoll 6
server ntp1.aliyun.com iburst minpoll 4 maxpoll 6
server time.google.com iburst minpoll 4 maxpoll 6
# pool as fallback (NTP Pool Project global nodes)
pool cn.pool.ntp.org iburst

# === Clock Adjustment Strategy ===
# Jump (instead of slewing) when offset > 1 second, max 3 jumps at startup
# Critical for VMs—snapshot restores can leave time way off
makestep 1.0 3

# Allow negative makestep (adjust time backwards)
# Default only allows forward adjustment; -1 allows backward, preventing "fake fast"
makestep 1.0 -1

# Periodically write system time to hardware clock (RTC), preventing post-reboot regression
rtcsync

# === Drift Recording ===
# Records local clock drift rate, allows estimation offline
driftfile /var/lib/chrony/drift

# === Logging ===
logdir /var/log/chrony
# Log when clock offset exceeds 0.5 seconds, for traceability
logchange 0.5
# Disable client logging (only needed when acting as server)
noclientlog

# === Security Restrictions ===
# Only allow local chronyc queries
bindcmdaddress 127.0.0.1
bindcmdaddress ::1

Parameters that are easy to get wrong:

• makestep 1.0 3: Often missed. By default, chrony “slews” (slowly adjusts) after startup, correcting a little each second. If the clock is 30 seconds off, it takes half an hour to fix. With makestep 1.0 3, offsets over 1 second trigger an immediate jump, max 3 jumps at startup. This is a lifesaver after VM snapshot restores.
• makestep 1.0 -1: Negative jumps. By default, only forward time adjustments are allowed. But if your clock is ahead of real time, you need to go backward, hence the -1. Without it, a fast clock has to “wait for real time to catch up,” which can take a long time.
• rtcsync: Periodically writes system time back to the hardware clock. Without this, post-reboot time may regress to an old value.
• iburst: Accelerates initial sync. Without it, first sync is slow.

Verifying Configuration

After changing config, reload the service (use reload, not restart):

sudo systemctl reload chronyd

Then verify:

# 1. Check service status
systemctl status chronyd

# 2. View time source status (critical)
chronyc sources -v

The chronyc sources -v output looks like this:

.-- Mode( ^ = server, = = peer, # = local clock)
/ .- Source state (* = current synced, + = combined, - = not combined)
| /   .- Source
| |   |                                                       .- Stratum(a)
| |   |                                                       |  .- Poll(pp)
| |   |                                                       |  |   .- Reach(oct)
| |   |                                                       |  |  |   .- LastRx
| |   |                                                       |  |  |   |  ==> .- Last sample
| |   |                                                       |  |  |   |     /   .- Sample milliseconds
MS Name/IP address         Stratum Poll Reach LastRx LastSample
=======================================================================
^* 203.107.6.88 (ntp.aliyun)   2   6   377    16   +1235us[+1235us] +/-   10ms
^+ time.google.com             2   6   377    17   +2356us[+2356us] +/-   20ms
^- 192.168.1.100               2   6   377    18   -3456us[-3456us] +/-   15ms

What to look for: Each source has a marker in front. Only sources marked * are actually being used for time correction. If you see only - or +, chrony considers all sources unreliable—commonly caused by firewall blocking UDP 123, DNS resolution failure, or the pool domain returning private IPs.

# 3. View sync quality
chronyc tracking

Output:

Reference ID    : CB6B0658 (203.107.6.88:123)
Stratum         : 3
Ref time (UTC)   : Fri Jul 18 16:30:00 2026
System time     : 0.000123456 seconds slow of NTP time
Last offset     : +0.000098765 seconds
RMS offset      : 0.000156789 seconds
Frequency       : 5.678 ppm slow
Residual freq   : 0.001 ppm
Skew            : 0.234 ppm
Root delay      : 0.012345 seconds
Root dispersion  : 0.000567 seconds
Update interval : 64.2 seconds
Leap status     : Normal

What to look for:

• System time: Current offset between system clock and NTP standard time. Under ±10ms is normal; financial scenarios require ±1ms.
• Last offset: Offset after the last correction.
• Leap status: Normal: Time is normal (not in leap second state).
• Stratum: How many hops you are from a stratum-1 source. 3 means 2 layers of relay—already close to a primary source.

Don’t just check Leap status. Many people only look at Leap status: Normal and assume everything’s fine, but this value shows Normal as long as chrony can reach any source. What you should actually check is whether the System time offset has converged to an acceptable range.

chrony as an Internal Network Time Server

In large clusters, having every machine connect to external NTP is impractical—wastes egress bandwidth, high latency, complex firewall rules. The standard approach is to set up 1-2 internal time servers that connect to authoritative sources externally and serve time internally.

# /etc/chrony.conf (Internal time server config)

# Upstream time sources (external)
server ntp.aliyun.com iburst
server time.google.com iburst
pool cn.pool.ntp.org iburst

# Allow internal networks to sync
allow 192.168.0.0/16
allow 10.0.0.0/8

# Local clock as fallback (when all upstream sources are unreachable)
# stratum 10 means this local clock has low confidence, only used as last resort
local stratum 10

# Server-side needs client access logging (for troubleshooting)
# noclientlog  # Turn off to save logs when client count is high
clientloglimit 52428800  # Per-client log cap 50MB

# Other params same as client config
makestep 1.0 -1
rtcsync
driftfile /var/lib/chrony/drift
bindcmdaddress 127.0.0.1

Key point local stratum 10: This line lets the time server use its local clock to serve internal clients even when all external sources are down—not accurate, but at least all internal machines stay consistent. Consistency is more important than absolute accuracy. This is the iron law of distributed systems.

Internal clients just point to the internal time server:

# /etc/chrony.conf (Internal client)
server 192.168.1.100 iburst prefer   # Primary time server
server 192.168.1.101 iburst           # Backup time server
makestep 1.0 -1
rtcsync
driftfile /var/lib/chrony/drift

Firewall: The time server needs UDP port 123 open:

# firewalld
sudo firewall-cmd --permanent --add-service=ntp
sudo firewall-cmd --reload

# iptables
sudo iptables -A INPUT -p udp --dport 123 -s 192.168.0.0/16 -j ACCEPT

Time Sync for VMs and Containers

VM Clock Drift

VM time sync is a disaster zone. A VM’s “clock” is virtualized by the host. When host load is high, the VM’s perceived time jitters. VMware/QEMU/KVM all have varying virtual clock precision, but all share this problem.

Solutions:

1. Install time sync on the host (VMware Tools, QEMU guest agent, or chrony)
2. Install chrony in the VM too, as a second layer of defense
3. Enable time sync options in the VM config (VMware Tools time sync)

Special chrony config for VMs:

# /etc/chrony.conf (Virtual Machine)

# Shorten polling interval, detect drift faster
server ntp.aliyun.com iburst minpoll 2 maxpoll 8

# VM clock jitter is high, allow wider makestep range
makestep 0.1 -1

# Record drift rate more frequently
driftfile /var/lib/chrony/drift

# rtcsync is less meaningful for VMs (virtual RTC), but harmless
rtcsync

# Disable hardware timestamps (VMs don't have them)
# hwtimestamp *  ← comment out this line

Container Scenarios

Container time sync is a special problem. Containers share the host’s kernel clock by default—if the host time is correct, the container time is correct. But there are two pitfalls:

Pitfall 1: Installing chrony in a container is useless. Containers don’t have an independent kernel clock. chrony inside a container is actually adjusting the host’s clock. Moreover, containers don’t have CAP_SYS_TIME capability by default—they can’t even execute the settimeofday syscall. chrony will error out with a permission denied. The correct approach is to install chrony on the host; containers don’t manage time themselves.

Pitfall 2: Container time looks correct, but application behavior is abnormal. Common with Java apps—JVM reads the system clock once at startup and relies on that cached baseline for scheduled tasks. If the host time jumps after JVM startup, the JVM’s internal time perception diverges from the system clock. Fix: add -XX:+UseSystemTimeSource to JVM startup params.

When containers must manage their own time: Privileged containers, containers using independent namespaces, certain compliance requirements. In these cases, add the CAP_SYS_TIME capability:

# docker-compose.yml
services:
  chrony:
    image: chrony:latest
    cap_add:
      - SYS_TIME
    network_mode: host   # NTP uses UDP 123, host network is easiest
    restart: unless-stopped

# Kubernetes Pod
apiVersion: v1
kind: Pod
metadata:
  name: chrony-sidecar
spec:
  containers:
    - name: chrony
      image: chrony:latest
      securityContext:
        capabilities:
          add: ["SYS_TIME"]
      volumeMounts:
        - name: etc-chrony
          mountPath: /etc/chrony.conf
          subPath: chrony.conf
  volumes:
    - name: etc-chrony
      configMap:
        name: chrony-config

Warning: Only use this approach when absolutely necessary. Normally, container time comes from the host. Running chrony inside containers is an anti-pattern—multiple containers adjusting the host clock simultaneously will fight each other.

Monitoring Time Sync Status

Time sync status must be monitored. Configuring it isn’t enough; you need to continuously confirm “time is actually synced.”

Collecting with node_exporter

node_exporter itself doesn’t collect time sync status, but you can use the textfile collector with a script to collect it. Write a script to parse chronyc tracking output and convert it to Prometheus metrics:

#!/usr/bin/env python3
"""chrony_status_exporter.py
Parses chronyc tracking output, exposes time sync metrics to Prometheus.
"""
import re
import subprocess
from prometheus_client import CollectorRegistry, Gauge, write_to_textfile

registry = CollectorRegistry()

# Offset between system time and NTP standard time (seconds)
clock_offset = Gauge(
    'chrony_clock_offset_seconds',
    'System clock offset from NTP time in seconds',
    registry=registry
)

# Sync status (0=not synced, 1=normal, 2=leap warning)
leap_status = Gauge(
    'chrony_leap_status',
    'Leap status: 0=not synced, 1=normal, 2=leap warning',
    registry=registry
)

# Root delay to reference source
root_delay = Gauge(
    'chrony_root_delay_seconds',
    'Root delay to reference source in seconds',
    registry=registry
)

# Stratum level
stratum = Gauge(
    'chrony_stratum',
    'Stratum of this clock relative to reference',
    registry=registry
)

try:
    output = subprocess.check_output(['chronyc', 'tracking'], text=True)
    
    # Parse System time
    m = re.search(r'System time\s+:\s+([\d.]+)\s+(\w+)', output)
    if m:
        value = float(m.group(1))
        if m.group(2) == 'slow':
            value = -value
        clock_offset.set(value)
    
    # Parse Leap status
    m = re.search(r'Leap status\s+:\s+(\w+)', output)
    if m:
        status_map = {'Normal': 1, 'Insert second': 2, 'Delete second': 2, 'Not synchronised': 0}
        leap_status.set(status_map.get(m.group(1), 0))
    
    # Parse Root delay
    m = re.search(r'Root delay\s+:\s+([\d.]+)\s+seconds', output)
    if m:
        root_delay.set(float(m.group(1)))
    
    # Parse Stratum
    m = re.search(r'Strum\s+:\s+(\d+)', output)
    if m:
        stratum.set(int(m.group(1)))

except (subprocess.CalledProcessError, FileNotFoundError):
    # chrony unavailable, set all metrics to 0
    leap_status.set(0)

write_to_textfile('/var/lib/node_exporter/textfile/chrony_status.prom', registry)

Drop it into crontab to run every minute:

# /etc/cron.d/chrony-exporter
* * * * * root /usr/local/bin/chrony_status_exporter.py

Prometheus Alert Rules

groups:
  - name: time_sync
    interval: 30s
    rules:
      # P1: Clock offset exceeds 100ms
      - alert: ClockOffsetHigh
        expr: abs(chrony_clock_offset_seconds) > 0.1
        for: 2m
        labels:
          severity: P1
        annotations:
          summary: "{{ $labels.instance }} clock offset too high"
          description: "System time offset {{ $value }} seconds from NTP, exceeds 100ms threshold"

      # P0: Clock offset exceeds 1 second (may cause cert failures, auth failures)
      - alert: ClockOffsetCritical
        expr: abs(chrony_clock_offset_seconds) > 1
        for: 1m
        labels:
          severity: P0
        annotations:
          summary: "{{ $labels.instance }} critical clock offset"
          description: "System time offset {{ $value }} seconds, may cause TLS cert validation failure, Kerberos auth failure"

      # P1: chrony not synced
      - alert: ChronyNotSynced
        expr: chrony_leap_status == 0
        for: 5m
        labels:
          severity: P1
        annotations:
          summary: "{{ $labels.instance }} chrony not synced"
          description: "chrony has not synced with any time source for 5 minutes"

      # P2: Stratum too high (too far from time source)
      - alert: ChronyStratumHigh
        expr: chrony_stratum > 10
        for: 10m
        labels:
          severity: P2
        annotations:
          summary: "{{ $labels.instance }} stratum too high"
          description: "Current stratum {{ $value }}, too far from primary time source, poor precision"

Common Troubleshooting

Issue 1: chrony started but can’t sync

# View time source status
chronyc sources -v
# If all show ^- (dash), sources are unavailable

Troubleshooting steps:

# 1. Test network connectivity (NTP uses UDP 123)
nc -uvz ntp.aliyun.com 123
# Or test with ntpdate (note: ntpdate is deprecated, for testing only)
ntpdate -q ntp.aliyun.com

# 2. Check firewall
sudo firewall-cmd --list-services
sudo iptables -L -n | grep 123

# 3. Check DNS resolution
nslookup ntp.aliyun.com
dig ntp.aliyun.com

# 4. Check chrony logs
journalctl -u chronyd --since "10 minutes ago"

Common causes: cloud security groups not allowing UDP 123, DNS resolution failure, NTP source server down. Switching to domestic sources usually resolves network issues.

Issue 2: Offset too large to correct

chrony doesn’t correct offsets over 1000 seconds by default. This is a safeguard—prevents a misconfigured time source from dragging the system clock wildly off.

# Force sync manually (jump)
sudo chronyc makestep

# If offset is still large, set approximate time first, then let chrony fine-tune
sudo date -s "2026-07-19 00:30:00"
sudo chronyc makestep

Prerequisite: config has makestep 1.0 -1, otherwise manual makestep will be rejected too.

Issue 3: Time regresses after reboot

Typical symptom: time was correct before reboot, but after reboot it went back to days ago.

Cause: Hardware clock (RTC) not synced. System time is maintained by the kernel at runtime, but reads initial time from RTC on boot. If RTC doesn’t have the correct time written back, time regresses after reboot.

# Check RTC time
sudo hwclock --show

# If RTC time is wrong, sync manually
sudo hwclock --systohc  # Write system time to RTC

# Confirm chrony config has rtcsync
grep rtcsync /etc/chrony.conf
# If not, add it and restart chronyd

Issue 4: Dual-boot (Windows + Linux) time conflict

Common on dual-boot machines: Windows shows correct time, but after rebooting into Linux, time is 8 hours ahead.

Cause: Windows treats the hardware clock as local time by default, while Linux treats it as UTC. The two disagree on what RTC means.

# Make Linux treat RTC as local time
sudo timedatectl set-local-rtc 1
# But this violates Linux convention; better to change Windows instead

The better approach is to change the Windows registry so Windows treats RTC as UTC:

# Windows Registry
[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation]
"RealTimeIsUniversal"=dword:00000001

Issue 5: ntpd and chronyd conflict

A machine can’t run ntpd and chronyd simultaneously—they fight over UDP port 123 and interfere with each other.

# Check if ntpd is running
systemctl status ntpd 2>/dev/null
systemctl status ntp 2>/dev/null

# If so, stop and disable
sudo systemctl disable --now ntpd
sudo systemctl disable --now ntp

# Also check systemd-timesyncd
systemctl status systemd-timesyncd
# If chronyd is running, timesyncd should be inactive

Rule: One machine, one time sync service. Pick one of chronyd, ntpd, or systemd-timesyncd. They cannot coexist.

Time Sync and the TLS Certificate Hazard

The failure case mentioned earlier—12-second offset causing mass HTTPS failures—stems from TLS certificate validity checking.

X.509 certificates have two time fields: notBefore (effective time) and notAfter (expiry time). OpenSSL verifies that the current system time falls within [notBefore, notAfter] during handshake. If system time is earlier than notBefore (“cert not yet valid”) or later than notAfter (“cert expired”), the handshake fails.

Most dangerous scenario: Let’s Encrypt cert auto-renewal tasks rely on system time to determine “the cert is about to expire, time to renew.” If system time is behind real time, the renewal task thinks “cert is still valid, no rush,” but in real time the cert has already expired. When users hit the service and get errors, you discover the cert has expired—but the auto-renewal task never triggered because of the incorrect time.

Prevention:

1. Monitor cert remaining validity (independent of local time, use openssl s_client to connect remotely and read the cert)
2. Add fault tolerance to renewal tasks—even if time-based logic says “no need to renew,” periodically force a check
3. Alert on time offset over 30 seconds, don’t wait for incidents to find out

# Cert expiry check independent of local time
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates
# notBefore=Jul 19 00:00:00 2026 GMT
# notAfter=Oct 17 00:00:00 2026 GMT

Turn this into a monitoring script that probes cert validity externally, independent of the monitored machine’s local time.

Production Practice Recommendations

1. Tiered time source architecture. In large clusters, don’t have every machine connect to external NTP. Set up 1-2 internal time servers that connect to authoritative sources externally and serve internally. If internal sources go down, local stratum kicks in as fallback, keeping internal time consistent. Consistency beats absolute accuracy.

2. Manage VMs and physical machines separately. Physical machine clocks are stable; chrony default config works. VM clocks jitter heavily—shorten polling intervals, increase makestep tolerance. Containers rely on the host; don’t run chrony inside containers.

3. Monitoring isn’t just service status. systemctl status chronyd showing active doesn’t mean time sync is working. Monitoring must read the actual offset from chronyc tracking and alert when offset exceeds thresholds.

4. Beware time jumps. makestep causes time jumps that impact certain apps (databases, scheduled tasks, log ordering). For financial trading systems, disable makestep and use slew mode for gradual adjustment. But for initial deployment or large offsets, jumping is more practical than slewing.

5. Leap seconds are a trap. When a leap second occurs (usually end of June or December), 23:59:59 is followed by 23:59:60. Many systems can’t handle this extra second and crash or error. chrony supports the leapsectz directive for handling leap seconds, but test before the leap second hits. Google’s approach is “smearing” the leap second—spreading the one-second offset across 24 hours to avoid an instant jump.

Summary

Time sync is infrastructure for distributed systems, but it’s so “low-level” that nobody pays attention until something blows up. I’ve seen payment systems grind to a halt for two hours due to clock offset, and K8s clusters suffer mass node evictions from clock desync. The common thread: time is never the first suspect during troubleshooting—because “how can time be wrong?”

chrony replacing ntpd is the trend. Configuration isn’t complex, but parameter tuning matters. makestep, rtcsync, driftfile are three must-haves; multi-source redundancy and local stratum fallback are production standards. VMs and containers each have their pitfalls. Remember one core principle: containers rely on the host, VMs rely on host + chrony as a dual safeguard.

Monitor, and monitor the actual clock offset value—not just whether the service process is alive. Time offset over 1 second warrants a P0 alert. This threshold isn’t fear-mongering; it’s because 1 second of offset is enough to break TLS cert validation, Kerberos auth, and distributed locks.

One last piece of experience: when troubleshooting time issues, first check if system time is correct, then look at service config. Many people dive straight into chrony config and logs, only to find the system time was manually changed or RTC wasn’t synced. Start from the simplest thing.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. chrony如何配置多个NTP服务器实现高可用时间同步? — CSDN Q&A, chrony source selection algorithm and multi-source HA configuration patterns
2. 5分钟搞定CentOS7时间同步:chronyd配置全攻略 — CSDN Column, chronyd core configuration and ntpq command deep dive
3. 如何在Linux中配置系统的本地时钟同步 — PHP.cn, chrony service checking and configuration verification in practice
4. Linux系统时间同步配置_ntp与chrony实践 — PHP.cn, chrony vs ntpd comparison and troubleshooting essentials
5. 主_旁路由时间不同步引发HTTPS证书批量失效:chrony高精度同步配置规范 — CSDN Column, root cause analysis of TLS cert failures from time desync and PTP hardware timestamp configuration
6. 北京时间服务器IP如何精准同步系统时间? — CSDN Q&A, multi-dimensional root cause analysis matrix for time sync anomalies
7. 2026年容器安全NTP服务配置实践指南 — Renrendoc, NTP service challenges and configuration practices in container environments