Scheduled Task Management: cron vs systemd timer

Overview

Scheduled tasks are a foundational component of operations automation — log rotation, data backups, certificate renewal, health checks, report generation — nearly every ops scenario relies on scheduled execution. Most people’s understanding of scheduled tasks stops at crontab -e plus a line like 0 2 * * * /path/to/script.sh, but this is far from sufficient in production: who gets notified when a task fails? Who handles execution timeouts? How do you coordinate tasks across multiple machines? This article systematically covers scheduled task management practices from cron to systemd timer to distributed scheduling.

References: cron Wikipedia, systemd.timer Official Documentation

I. cron Syntax and Limitations

1.1 cron Expressions

A cron expression consists of 5 fields:

┌──────── Minute (0-59)
│ ┌────── Hour (0-23)
│ │ ┌──── Day (1-31)
│ │ │ ┌── Month (1-12)
│ │ │ │ ┌ Day of week (0-7, both 0 and 7 are Sunday)
│ │ │ │ │
* * * * * command

Expression	Meaning
`0 2 * * *`	Every day at 2:00 AM
`/15 * * *`	Every 15 minutes
`0 /6 * *`	Every 6 hours
`0 0 * * 0`	Every Sunday at 0:00
`0 0 1 * *`	1st of every month at 0:00
`30 3-5 * * *`	3:30, 4:30, 5:30
`0 0 1 1,4,7,10 *`	1st of January, April, July, October
`@reboot`	At system startup
`@daily` / `@midnight`	Every day at 0:00
`@weekly`	Every Sunday at 0:00
`@monthly`	1st of every month at 0:00
`@yearly` / `@annually`	January 1st at 0:00

1.2 crontab Management

# Edit current user's crontab
crontab -e

# View current user's crontab
crontab -l

# Delete all scheduled tasks for current user
crontab -r

# Import crontab from file
crontab cron_jobs.txt

# Edit a specific user's crontab (requires root)
sudo crontab -u deploy -e

# System-level scheduled tasks (visible to all users)
sudo vim /etc/crontab
# Format includes an extra user field:
# m h dom mon dow user command
0 2 * * * root /opt/scripts/backup.sh

# Directory-based scheduled tasks (execute all scripts in the directory at preset frequency)
/etc/cron.hourly/    # Hourly
/etc/cron.daily/     # Daily
/etc/cron.weekly/    # Weekly
/etc/cron.monthly/   # Monthly

1.3 Limitations of cron

Limitation	Description	Impact
No error retry	Tasks don’t automatically retry after failure	Transient failures cause missed executions
No execution timeout	Tasks can run indefinitely	Slow tasks consume resources, block subsequent tasks
No concurrency control	Same task may run concurrently	Data races, resource exhaustion
Unfriendly logging	Output sent via email or lost	Difficult troubleshooting
No dependency management	Cannot define dependencies between tasks	Manual execution order enforcement needed
Minute-only precision	No sub-minute scheduling	Cannot meet high-frequency task needs
Restricted environment	Default PATH is short, no user env vars	Scripts can’t find commands
Single-machine only	Can only run on one machine	No cross-node coordination

# Typical cron problem scenarios

# Problem 1: Missing environment variables
0 2 * * * /opt/scripts/backup.sh
# backup.sh uses python3, but cron's PATH doesn't include /usr/local/bin
# Solution: Explicitly set PATH at the top of the script, or define it in crontab

PATH=/usr/local/bin:/usr/bin:/bin
0 2 * * * /opt/scripts/backup.sh

# Problem 2: Concurrent execution
*/1 * * * * /opt/scripts/process_data.sh
# If process_data.sh takes more than 1 minute, the next run starts concurrently
# Solution: Use flock for file locking
*/1 * * * * /usr/bin/flock -n /tmp/process_data.lock /opt/scripts/process_data.sh

# Problem 3: Lost output
0 2 * * * /opt/scripts/backup.sh
# If there's stderr output, cron tries to send email (lost if no mail configured)
# Solution: Redirect to a log file
0 2 * * * /opt/scripts/backup.sh >> /var/log/backup.log 2>&1

# Problem 4: No awareness of failures
0 2 * * * /opt/scripts/backup.sh >> /var/log/backup.log 2>&1
# Even if backup.sh exits non-zero, cron doesn't care
# Solution: Implement alerting within the script, or use systemd timer

II. systemd timer

2.1 Basic Concepts

A systemd timer consists of two files:

Service file: Defines the task to execute (same as a regular systemd service)
Timer file: Defines the trigger schedule

/etc/systemd/system/
├── backup.service      # Task definition
└── backup.timer        # Scheduled trigger

2.2 Creating a Timer

# /etc/systemd/system/backup.service
[Unit]
Description=Database Backup Service
Documentation=file:///opt/scripts/backup.sh
After=network.target postgresql.service

[Service]
Type=oneshot
ExecStart=/opt/scripts/backup.sh
ExecStartPost=/opt/scripts/notify-backup.sh
User=backup
Group=backup
# Resource limits
MemoryMax=512M
CPUQuota=50%
# Timeout
TimeoutStartSec=3600
# Working directory
WorkingDirectory=/opt/scripts
# Environment variables
Environment="PATH=/usr/local/bin:/usr/bin:/bin"
Environment="BACKUP_DIR=/data/backups"
EnvironmentFile=-/etc/sysconfig/backup

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/backup.timer
[Unit]
Description=Run Database Backup Daily
Requires=backup.service

[Timer]
# Trigger method 1: Calendar time (similar to cron)
OnCalendar=*-*-* 02:00:00

# Trigger method 2: Relative time
# OnBootSec=5min          # 5 minutes after boot
# OnUnitActiveSec=1h      # 1 hour after last execution
# OnUnitInactiveSec=30min # 30 minutes after last completion

# Random delay (prevent all tasks from starting simultaneously)
RandomizedDelaySec=300

# Persistent (missed tasks are run after startup)
Persistent=true

# Accuracy (reduce wakeup frequency, save power)
AccuracySec=1min

[Install]
WantedBy=timers.target

# Enable the timer
sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer

# View timer status
systemctl status backup.timer
systemctl list-timers backup.timer

# View all timers
systemctl list-timers --all

# Manual trigger
sudo systemctl start backup.service

# View execution logs
journalctl -u backup.service
journalctl -u backup.service --since today
journalctl -u backup.service -f

# View next trigger time
systemctl list-timers backup.timer

2.3 OnCalendar Syntax

systemd’s calendar time expressions are more powerful than cron:

# Basic format: DayOfWeek Year-Month-Day Hour:Minute:Second

OnCalendar=*-*-* 02:00:00              # Every day at 2:00
OnCalendar=*-*-* 02,14:00:00          # Every day at 2:00 and 14:00
OnCalendar=Mon..Fri 09:00:00          # Weekdays at 9:00
OnCalendar=Sat,Sun 02:00:00           # Weekends at 2:00
OnCalendar=*-*-01 00:00:00            # 1st of every month
OnCalendar=*-01,04,07,10-01 00:00:00  # 1st of quarter-start months
OnCalendar=Mon *-*-* 09:00:00         # Every Monday at 9:00
OnCalendar=*:0/15                     # Every 15 minutes (second-level precision)
OnCalendar=hourly                     # Hourly
OnCalendar=daily                      # Daily at 00:00
OnCalendar=weekly                     # Every Monday at 00:00
OnCalendar=monthly                    # 1st of every month at 00:00
OnCalendar=yearly                     # January 1st at 00:00

# Validate calendar expressions
systemd-analyze calendar "*-*-* 02:00:00"
systemd-analyze calendar "Mon..Fri 09:00:00"

2.4 cron vs systemd timer Comparison

Dimension	cron	systemd timer
Precision	Minute	Second
Error retry	None	Via `OnFailure=`
Execution timeout	None	`TimeoutStartSec=`
Concurrency control	Requires flock	`RefuseManualStart=` + `ConditionPathExists=`
Logging	Email or manual redirect	Native journalctl integration
Dependency management	None	`After=`, `Requires=`
Environment variables	Limited	`Environment=`, `EnvironmentFile=`
Resource limits	None	`MemoryMax=`, `CPUQuota=`
User isolation	crontab -u	`systemctl --user`
Missed tasks	Lost	`Persistent=true` catches up
Random delay	None	`RandomizedDelaySec=`
Status query	crontab -l	`systemctl list-timers`
Learning curve	Low	Medium

Conclusion: Prefer systemd timer for new projects. Existing cron setups can be migrated gradually. systemd timer’s advantages in error handling, logging, and resource control represent a qualitative improvement.

III. Task Design Principles

3.1 Idempotency

Scheduled tasks must be idempotent — running them any number of times produces the same result:

#!/usr/bin/env bash
# ❌ Non-idempotent: appends data every time
echo "backup data" >> /data/backup.log

# ✅ Idempotent: use a fixed filename or date-based filename
echo "backup data" > "/data/backup-$(date +%Y%m%d).log"

# ✅ Idempotent: check before executing
if [ -f /data/backup-done.flag ]; then
    echo "Already backed up today, skipping"
    exit 0
fi
# Execute backup...
touch /data/backup-done.flag

3.2 Concurrency Control

#!/usr/bin/env bash
set -euo pipefail

# Method 1: flock file lock
LOCK_FILE="/var/lock/$(basename "$0").lock"
exec 200>"$LOCK_FILE"
flock -n 200 || {
    echo "Another instance is running, exiting"
    exit 0
}

# Method 2: PID file lock
PID_FILE="/var/run/$(basename "$0" .sh).pid"
if [ -f "$PID_FILE" ]; then
    PID=$(cat "$PID_FILE")
    if kill -0 "$PID" 2>/dev/null; then
        echo "Process $PID is running, exiting"
        exit 0
    fi
fi
echo $$ > "$PID_FILE"
trap "rm -f $PID_FILE" EXIT

# systemd approach: more elegant concurrency control
# /etc/systemd/system/data-sync.service
[Service]
Type=oneshot
ExecStart=/opt/scripts/sync-data.sh
# Prevent manual duplicate starts
RefuseManualStart=no
# If already running, no new instance is started (default behavior)

3.3 Timeout Control

#!/usr/bin/env bash
set -euo pipefail

TIMEOUT=300  # 5 minutes

# Method 1: timeout command
timeout "$TIMEOUT" /opt/scripts/long_task.sh
exit_code=$?

if [ $exit_code -eq 124 ]; then
    echo "Task timed out (${TIMEOUT}s)"
    # Send alert
    curl -sf -X POST "$WEBHOOK_URL" \
        -d "{\"text\":\"Task timed out: $(basename $0)\"}"
    exit 1
fi

exit $exit_code

# Method 2: Background execution + timeout check
run_with_timeout() {
    local timeout=$1; shift
    local cmd="$*"

    $cmd &
    local pid=$!

    local start=$(date +%s)
    while kill -0 $pid 2>/dev/null; do
        local elapsed=$(( $(date +%s) - start ))
        if [ $elapsed -gt $timeout ]; then
            echo "Timed out, killing process $pid"
            kill -TERM $pid
            sleep 2
            kill -9 $pid 2>/dev/null || true
            return 124
        fi
        sleep 5
    done

    wait $pid
    return $?
}

# systemd approach
[Service]
TimeoutStartSec=300
# After timeout, systemd sends SIGTERM, then SIGKILL

3.4 Environment Isolation

#!/usr/bin/env bash
# Set up a complete environment at the top of the script

set -euo pipefail

# Explicitly set PATH
export PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"

# Load environment variables
if [ -f /etc/profile.d/app.sh ]; then
    source /etc/profile.d/app.sh
fi

# Set working directory
cd /opt/app || { echo "Cannot switch to working directory"; exit 1; }

# Set umask
umask 022

# Locale settings
export LANG=en_US.UTF-8
export LC_ALL=en_US.UTF-8

IV. Failure Retry and Alerting

4.1 systemd Failure Retry

# /etc/systemd/system/critical-task.service
[Unit]
Description=Critical Task
OnFailure=critical-task-failed@%n.service

[Service]
Type=oneshot
ExecStart=/opt/scripts/critical-task.sh

# Auto restart (not applicable to oneshot, but can be paired with timer)
# Restart=on-failure
# RestartSec=30

# Resource limits
MemoryMax=1G
CPUQuota=100%

# Timeout
TimeoutStartSec=600

# /etc/systemd/system/critical-task-failed@.service
# Template service: triggered when a task fails
[Unit]
Description=Handle Failure of %i

[Service]
Type=oneshot
ExecStart=/opt/scripts/alert-failure.sh %i

#!/usr/bin/env bash
# /opt/scripts/alert-failure.sh
# Task failure alerting script

FAILED_UNIT="$1"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
HOSTNAME=$(hostname)

# Get failure logs
LOGS=$(journalctl -u "$FAILED_UNIT" --since "10 min ago" --no-pager -n 20 2>/dev/null)

# Send alert
curl -sf -X POST "${WEBHOOK_URL}" \
    -H 'Content-Type: application/json' \
    -d "{
        \"text\": \"Scheduled task failure alert\nService: ${FAILED_UNIT}\nHost: ${HOSTNAME}\nTime: ${TIMESTAMP}\nLogs:\n${LOGS}\"
    }"

# Attempt retry
RETRY_FILE="/var/lib/task-retry/${FAILED_UNIT}.count"
mkdir -p /var/lib/task-retry
RETRY_COUNT=0
if [ -f "$RETRY_FILE" ]; then
    RETRY_COUNT=$(cat "$RETRY_FILE")
fi

if [ "$RETRY_COUNT" -lt 3 ]; then
    RETRY_COUNT=$((RETRY_COUNT + 1))
    echo "$RETRY_COUNT" > "$RETRY_FILE"
    echo "Retry attempt ${RETRY_COUNT}..."
    sleep $((RETRY_COUNT * 30))  # Incremental delay
    systemctl start "$FAILED_UNIT"
else
    echo "Max retries reached (3), manual intervention required"
    # Escalate alert
    curl -sf -X POST "${ESCALATION_URL}" \
        -d "{\"text\":\"Task ${FAILED_UNIT} failed after 3 retries, manual intervention required\"}"
fi

4.2 cron Task Alerting Wrapper

#!/usr/bin/env bash
# /opt/scripts/cron-wrapper.sh
# cron task wrapper: unified handling of logging, timeout, and alerting

set -euo pipefail

# Argument parsing
TASK_NAME="$1"; shift
TIMEOUT="${TIMEOUT:-300}"
WEBHOOK_URL="${WEBHOOK_URL:-}"
LOG_DIR="/var/log/cron"

mkdir -p "$LOG_DIR"

LOG_FILE="${LOG_DIR}/${TASK_NAME}-$(date +%Y%m%d).log"
START_TIME=$(date +%s)

echo "========================================" | tee -a "$LOG_FILE"
echo "Task: $TASK_NAME" | tee -a "$LOG_FILE"
echo "Time: $(date '+%Y-%m-%d %H:%M:%S')" | tee -a "$LOG_FILE"
echo "Command: $*" | tee -a "$LOG_FILE"
echo "========================================" | tee -a "$LOG_FILE"

# Execute task (with timeout)
set +e
timeout "$TIMEOUT" "$@" 2>&1 | tee -a "$LOG_FILE"
EXIT_CODE=${PIPESTATUS[0]}
set -e

END_TIME=$(date +%s)
DURATION=$((END_TIME - START_TIME))

echo "========================================" | tee -a "$LOG_FILE"
echo "Exit code: $EXIT_CODE" | tee -a "$LOG_FILE"
echo "Duration: ${DURATION}s" | tee -a "$LOG_FILE"
echo "========================================" | tee -a "$LOG_FILE"

# Handle result
if [ $EXIT_CODE -eq 0 ]; then
    echo "✓ Task succeeded" | tee -a "$LOG_FILE"
    # Update success marker
    touch "/var/lib/cron-status/${TASK_NAME}.success"
    rm -f "/var/lib/cron-status/${TASK_NAME}.failed"
else
    echo "✗ Task failed" | tee -a "$LOG_FILE"
    # Record failure
    mkdir -p /var/lib/cron-status
    echo "$(date +%s)" > "/var/lib/cron-status/${TASK_NAME}.failed"

    # Send alert
    if [ -n "$WEBHOOK_URL" ]; then
        FAILURE_LOG=$(tail -20 "$LOG_FILE")
        curl -sf -X POST "$WEBHOOK_URL" \
            -H 'Content-Type: application/json' \
            -d "{
                \"text\": \"⚠ Cron task failed\nTask: ${TASK_NAME}\nHost: $(hostname)\nExit code: ${EXIT_CODE}\nDuration: ${DURATION}s\nLogs:\n${FAILURE_LOG}\"
            }" 2>/dev/null || true
    fi
fi

# Log rotation
find "$LOG_DIR" -name "${TASK_NAME}-*.log" -mtime +30 -delete

exit $EXIT_CODE

# Use the wrapper in crontab
0 2 * * * /opt/scripts/cron-wrapper.sh backup 300 /opt/scripts/backup.sh
*/15 * * * * /opt/scripts/cron-wrapper.sh health-check 60 /opt/scripts/check.sh
0 * * * * /opt/scripts/cron-wrapper.sh sync-data 600 /opt/scripts/sync.sh

V. Log Integration

5.1 systemd Journal Logging

# View timer execution history
journalctl -u backup.service --since today
journalctl -u backup.service --since "2026-07-10" --until "2026-07-10 12:00"

# Errors only
journalctl -u backup.service -p err

# Follow live logs
journalctl -u backup.service -f

# View execution duration
journalctl -u backup.service -o json | jq '. | select(.MESSAGE | contains("Duration"))'

# View execution records for all timers
journalctl -u '*.timer' --since today

# Export logs
journalctl -u backup.service --since "2026-07-01" > backup-logs.txt

5.2 Structured Logging

#!/usr/bin/env bash
# Output structured logs from within scripts

# Method 1: logger command (writes to syslog/journal)
logger -t backup "Backup started: target=${BACKUP_TARGET}"
# ... perform backup ...
logger -t backup "Backup completed: size=${BACKUP_SIZE}, duration=${DURATION}s"

# Method 2: systemd-cat (supports priority levels)
echo "Backup started" | systemd-cat -t backup -p info
echo "Backup warning: low disk space" | systemd-cat -t backup -p warning
echo "Backup failed" | systemd-cat -t backup -p err

# Method 3: JSON format logging
log_json() {
    local level="$1"; shift
    local message="$*"
    echo "{\"timestamp\":\"$(date -u +%FT%TZ)\",\"level\":\"${level}\",\"task\":\"backup\",\"message\":\"${message}\"}" | systemd-cat -t backup
}

log_json info "Backup started"
log_json info "Backup completed: size=${BACKUP_SIZE}"
log_json error "Backup failed: ${ERROR_MSG}"

5.3 Log Aggregation and Querying

# Query execution status of all scheduled tasks
systemctl list-timers --all --output=json | \
    jq '.[] | {unit: .unit, next: .next, last: .last, result: .result}'

# Count today's task success rate
SUCCESS=$(journalctl --since today -u '*.service' --grep '✓ Task succeeded' | wc -l)
FAILED=$(journalctl --since today -u '*.service' --grep '✗ Task failed' | wc -l)
echo "Today's scheduled tasks: succeeded=${SUCCESS} failed=${FAILED}"

# Generate scheduled task daily report
generate_timer_report() {
    echo "=== Scheduled Task Daily Report $(date +%Y-%m-%d) ==="
    echo ""
    systemctl list-timers --all | head -n -1 | while read -r line; do
        unit=$(echo "$line" | awk '{print $1}')
        if [[ "$unit" == *.timer ]]; then
            service="${unit%.timer}.service"
            last_run=$(journalctl -u "$service" --since today -o short-iso | tail -1)
            result="N/A"
            if journalctl -u "$service" --since today | grep -q "Succeeded"; then
                result="✓"
            elif journalctl -u "$service" --since today | grep -q "Failed"; then
                result="✗"
            fi
            echo "  $result $unit"
        fi
    done
}

VI. Distributed Scheduled Tasks

6.1 Distributed Challenges with Single-Machine cron

When tasks need to be coordinated across multiple machines, single-machine cron falls short:

The same task should only run on one machine (requires distributed locking)
Cross-machine task dependencies (machine A’s task must complete before triggering machine B’s task)
Task orchestration needed (DAG dependencies)
Visual execution history and manual triggering needed

6.2 Solution Comparison

Solution	Use Case	Advantages	Disadvantages
cron + distributed lock	Simple scenarios	Low migration cost	Complex lock failure handling
systemd timer	Single-machine/per-machine independent	Native integration, feature-rich	No cross-machine coordination
XXL-Job	Small-to-medium distributed	Lightweight, good UI	Java ecosystem, limited features
Airflow	Data pipelines / complex DAGs	Powerful DAG orchestration, rich Operators	Heavyweight, steep learning curve
K8s CronJob	K8s environments	Native integration, declarative	Simple features, no DAG
Temporal	Long-running tasks/workflows	Code-as-workflow, strong consistency	Newer, learning curve

6.3 K8s CronJob

# k8s-cronjob-backup.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: database-backup
  namespace: production
spec:
  schedule: "0 2 * * *"
  # Number of successful jobs to retain
  successfulJobsHistoryLimit: 3
  # Number of failed jobs to retain
  failedJobsHistoryLimit: 5
  # Concurrency policy
  # Allow: Allow concurrency (default)
  # Forbid: Forbid concurrency
  # Replace: Replace the running old job
  concurrencyPolicy: Forbid
  # Job start deadline
  startingDeadlineSeconds: 200
  # Suspend
  # suspend: false
  jobTemplate:
    spec:
      # Retry count
      backoffLimit: 3
      # Job timeout
      activeDeadlineSeconds: 3600
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: backup
            image: backup-tool:1.0
            command:
            - /bin/bash
            - -c
            - |
              set -euo pipefail
              echo "Starting backup..."
              pg_dump "$DATABASE_URL" > /backup/db-$(date +%Y%m%d).sql
              echo "Backup completed"              
            env:
            - name: DATABASE_URL
              valueFrom:
                secretKeyRef:
                  name: db-secret
                  key: url
            resources:
              requests:
                memory: 256Mi
                cpu: 200m
              limits:
                memory: 512Mi
                cpu: 500m
            volumeMounts:
            - name: backup-storage
              mountPath: /backup
          volumes:
          - name: backup-storage
            persistentVolumeClaim:
              claimName: backup-pvc

# Manage K8s CronJob
kubectl create -f k8s-cronjob-backup.yaml

# View CronJob
kubectl get cronjob -n production

# View execution history
kubectl get jobs -n production

# View Pod logs
kubectl logs -n production job/database-backup-XXXXX

# Manual trigger
kubectl create job --from=cronjob/database-backup manual-backup -n production

# Suspend
kubectl patch cronjob database-backup -n production -p '{"spec":{"suspend":true}}'

# Resume
kubectl patch cronjob database-backup -n production -p '{"spec":{"suspend":false}}'

6.4 XXL-Job

// XXL-Job task example (Java)
@XxlJob("databaseBackupHandler")
public void databaseBackup() {
    String param = XxlJobHelper.getJobParam();
    XxlJobHelper.log("Starting database backup, param: {}", param);

    try {
        // Execute backup logic
        String result = backupService.backup(param);
        XxlJobHelper.log("Backup completed: {}", result);
        XxlJobHelper.handleSuccess(result);
    } catch (Exception e) {
        XxlJobHelper.log("Backup failed: {}", e.getMessage());
        XxlJobHelper.handleFail(e.getMessage());
    }
}

Core advantages of XXL-Job:

Visual scheduling center: Web UI to manage all tasks, supporting manual triggering, pausing, and viewing execution logs
Failure alerting: Automatic email/DingTalk alerts on task failure
Routing strategies: First, round-robin, random, sharded broadcast, etc.
Task sharding: Large-data tasks sharded across multiple executors for parallel processing
Dynamic parameters: Support modifying task parameters dynamically in the UI

6.5 Airflow DAG

# airflow_dags/etl_pipeline.py
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.operators.python import PythonOperator
from airflow.providers.ssh.operators.ssh import SSHOperator
from datetime import datetime, timedelta

default_args = {
    'owner': 'data-team',
    'depends_on_past': False,
    'email': ['data-alerts@example.com'],
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
    'retry_exponential_backoff': True,
    'max_retry_delay': timedelta(minutes=30),
}

dag = DAG(
    'etl_pipeline',
    default_args=default_args,
    description='Daily ETL data pipeline',
    schedule_interval='0 2 * * *',
    start_date=datetime(2026, 1, 1),
    catchup=False,  # Don't backfill historical tasks
    max_active_runs=1,  # No concurrency
    tags=['etl', 'production'],
)

# Task 1: Check data source availability
check_source = BashOperator(
    task_id='check_source',
    bash_command='curl -sf http://source-api/health || exit 1',
    dag=dag,
)

# Task 2: Extract data
extract_data = SSHOperator(
    task_id='extract_data',
    ssh_conn_id='etl_server',
    command='cd /opt/etl && python3 extract.py --date {{ ds }}',
    dag=dag,
)

# Task 3: Transform data
transform_data = SSHOperator(
    task_id='transform_data',
    ssh_conn_id='etl_server',
    command='cd /opt/etl && python3 transform.py --date {{ ds }}',
    dag=dag,
)

# Task 4: Load data
load_data = SSHOperator(
    task_id='load_data',
    ssh_conn_id='etl_server',
    command='cd /opt/etl && python3 load.py --date {{ ds }}',
    dag=dag,
)

# Task 5: Data quality check
def check_quality(**context):
    import requests
    ds = context['ds']
    resp = requests.get(f'http://quality-service/check?date={ds}')
    if resp.status_code != 200 or resp.json()['score'] < 0.95:
        raise ValueError(f"Data quality check failed: {resp.json()}")
    return resp.json()

quality_check = PythonOperator(
    task_id='quality_check',
    python_callable=check_quality,
    dag=dag,
)

# Task 6: Send notification
notify = BashOperator(
    task_id='notify',
    bash_command='curl -sf -X POST $WEBHOOK_URL -d "ETL pipeline completed: {{ ds }}"',
    dag=dag,
)

# Define dependencies (DAG)
check_source >> extract_data >> transform_data >> load_data >> quality_check >> notify

Core advantages of Airflow DAG:

DAG orchestration: Define task dependencies in Python code, supporting complex workflows
Rich Operators: BashOperator, PythonOperator, SSHOperator, KubernetesPodOperator, etc.
Jinja2 templates: {{ ds }}, {{ prev_ds }} and other template variables auto-inject execution dates
Backfill: Can re-run tasks for historical dates
UI visualization: DAG graph display, execution history, manual triggering, log viewing

VII. Production Practices

7.1 Scheduled Task Inventory Management

# /opt/scheduler/tasks.yaml - Scheduled task inventory (version-controlled)
tasks:
  - name: database-backup
    description: "Daily database backup"
    schedule: "0 2 * * *"
    command: /opt/scripts/backup.sh
    timeout: 3600
    retries: 3
    alert_on_failure: true
    owner: dba-team
    enabled: true

  - name: log-cleanup
    description: "Log cleanup (retain 30 days)"
    schedule: "0 4 * * *"
    command: /opt/scripts/clean-logs.sh
    timeout: 600
    retries: 1
    alert_on_failure: false
    owner: ops-team
    enabled: true

  - name: ssl-check
    description: "SSL certificate expiry check"
    schedule: "0 9 * * *"
    command: /opt/scripts/check-ssl.py
    timeout: 120
    retries: 2
    alert_on_failure: true
    owner: ops-team
    enabled: true

  - name: metrics-report
    description: "Generate daily monitoring report"
    schedule: "0 8 * * *"
    command: /opt/scripts/generate-report.py
    timeout: 300
    retries: 2
    alert_on_failure: true
    owner: sre-team
    enabled: true

7.2 Generating systemd timers from Inventory

#!/usr/bin/env python3
"""
Generate systemd timer and service files from a YAML inventory
"""

import yaml
import os
from pathlib import Path

SYSTEMD_DIR = "/etc/systemd/system"

def generate_service(task: dict) -> str:
    """Generate service file"""
    return f"""[Unit]
Description={task['description']}
After=network.target

[Service]
Type=oneshot
ExecStart={task['command']}
User=root
TimeoutStartSec={task.get('timeout', 300)}
Environment="PATH=/usr/local/bin:/usr/bin:/bin"
"""

def generate_timer(task: dict) -> str:
    """Generate timer file"""
    persistent = "true" if task.get('persistent', True) else "false"
    return f"""[Unit]
Description=Timer for {task['name']}
Requires={task['name']}.service

[Timer]
OnCalendar={cron_to_systemd(task['schedule'])}
Persistent={persistent}
RandomizedDelaySec=60

[Install]
WantedBy=timers.target
"""

def cron_to_systemd(cron_expr: str) -> str:
    """Convert a cron expression to systemd OnCalendar"""
    parts = cron_expr.split()
    minute, hour, day, month, weekday = parts

    # Simplified conversion (handles common patterns)
    if cron_expr == "@daily":
        return "daily"
    elif cron_expr == "@weekly":
        return "weekly"
    elif cron_expr == "@monthly":
        return "monthly"

    result = []
    if weekday != "*":
        # Convert 0-7 to Mon-Sun
        days = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
        if "/" in weekday:
            result.append(days[int(weekday.split("/")[0]) - 1])
        else:
            result.append(days[int(weekday) % 7])

    date_parts = []
    if month != "*":
        date_parts.append(month)
    else:
        date_parts.append("*")
    if day != "*":
        date_parts.append(day)
    else:
        date_parts.append("*")
    date_parts.append("*")

    result.append("-".join(date_parts))

    time_parts = []
    if hour != "*":
        time_parts.append(hour)
    else:
        time_parts.append("*")
    if minute != "*":
        time_parts.append(f"{minute}:00")
    else:
        time_parts.append("*:00")

    result.append(":".join(time_parts))
    return " ".join(result)

def main():
    with open("/opt/scheduler/tasks.yaml") as f:
        config = yaml.safe_load(f)

    for task in config['tasks']:
        if not task.get('enabled', True):
            continue

        name = task['name']
        service_content = generate_service(task)
        timer_content = generate_timer(task)

        service_path = Path(SYSTEMD_DIR) / f"{name}.service"
        timer_path = Path(SYSTEMD_DIR) / f"{name}.timer"

        with open(service_path, 'w') as f:
            f.write(service_content)
        with open(timer_path, 'w') as f:
            f.write(timer_content)

        print(f"Generated: {service_path} + {timer_path}")

    print("\nRun the following commands to enable:")
    print("  systemctl daemon-reload")
    for task in config['tasks']:
        if task.get('enabled', True):
            print(f"  systemctl enable --now {task['name']}.timer")

if __name__ == '__main__':
    main()

7.3 Scheduled Task Inspection

#!/usr/bin/env bash
#
# check_timers.sh - Scheduled task inspection script
#
set -euo pipefail

echo "============================================"
echo "Scheduled Task Inspection Report"
echo "Time: $(date '+%Y-%m-%d %H:%M:%S')"
echo "============================================"

# 1. All timer statuses
echo -e "\n=== Timer List ==="
systemctl list-timers --all --no-pager | head -30

# 2. Failed tasks
echo -e "\n=== Failed Tasks ==="
FAILED=$(systemctl list-units --type=service --state=failed --no-pager | grep -E '\.service' || true)
if [ -n "$FAILED" ]; then
    echo "$FAILED"
else
    echo "No failed tasks"
fi

# 3. Today's execution summary
echo -e "\n=== Today's Execution Summary ==="
TOTAL=0
SUCCESS=0
FAILED_COUNT=0

for timer in $(systemctl list-timers --all --no-pager | grep '\.timer' | awk '{print $1}'); do
    service="${timer%.timer}"
    TOTAL=$((TOTAL + 1))

    if journalctl -u "$service" --since today --no-pager 2>/dev/null | grep -q "Succeeded\|Deactivated successfully"; then
        SUCCESS=$((SUCCESS + 1))
    elif journalctl -u "$service" --since today --no-pager 2>/dev/null | grep -q "Failed\|failed"; then
        FAILED_COUNT=$((FAILED_COUNT + 1))
        echo "  ✗ $service"
    fi
done

echo "  Total: $TOTAL  Succeeded: $SUCCESS  Failed: $FAILED_COUNT"

# 4. Check cron tasks
echo -e "\n=== Cron Tasks ==="
if command -v crontab &>/dev/null; then
    for user in root $(cut -d: -f1 /etc/passwd); do
        CRON_JOBS=$(crontab -u "$user" -l 2>/dev/null | grep -v '^#' | grep -v '^$' || true)
        if [ -n "$CRON_JOBS" ]; then
            echo "  User $user:"
            echo "$CRON_JOBS" | while read -r line; do
                echo "    $line"
            done
        fi
    done
fi

# 5. Check for missed executions
echo -e "\n=== Potentially Missed Tasks ==="
for timer in $(systemctl list-timers --all --no-pager | grep '\.timer' | awk '{print $1}'); do
    service="${timer%.timer}"
    LAST_RUN=$(systemctl show "$service" -p ActiveEnterTimestamp --value 2>/dev/null || echo "")
    if [ -z "$LAST_RUN" ] || [ "$LAST_RUN" = "n/a" ]; then
        echo "  ⚠ $service has never executed"
    fi
done

echo -e "\n============================================"

7.4 Migrating cron to systemd timer

#!/usr/bin/env bash
#
# migrate_cron_to_systemd.sh - Migrate cron tasks to systemd timer
#
set -euo pipefail

SYSTEMD_DIR="/etc/systemd/system"

# Read crontab
CRON_FILE="${1:-/etc/crontab}"

if [ ! -f "$CRON_FILE" ]; then
    echo "File not found: $CRON_FILE"
    exit 1
fi

# Parse cron lines (skip comments and empty lines)
while IFS= read -r line; do
    # Skip comments and empty lines
    [[ "$line" =~ ^[[:space:]]*# ]] && continue
    [[ -z "${line// }" ]] && continue

    # Parse fields: m h dom mon dow user command
    read -r minute hour day month weekday user command <<< "$line"

    # Extract task name from command
    task_name=$(echo "$command" | awk '{print $1}' | xargs basename | sed 's/\.sh$//')

    echo "Migrating: $task_name ($minute $hour $day $month $weekday)"

    # Generate service file
    cat > "${SYSTEMD_DIR}/${task_name}.service" << EOF
[Unit]
Description=Migrated from cron: ${task_name}
After=network.target

[Service]
Type=oneshot
ExecStart=${command}
User=${user}
TimeoutStartSec=3600
Environment="PATH=/usr/local/bin:/usr/bin:/bin"
EOF

    # Generate timer file
    cat > "${SYSTEMD_DIR}/${task_name}.timer" << EOF
[Unit]
Description=Timer for ${task_name}
Requires=${task_name}.service

[Timer]
OnCalendar=${minute}:${hour} *-*-${day:-*}-${month:-*} ${weekday:-*}
Persistent=true
RandomizedDelaySec=60

[Install]
WantedBy=timers.target
EOF

    echo "  -> ${SYSTEMD_DIR}/${task_name}.service"
    echo "  -> ${SYSTEMD_DIR}/${task_name}.timer"

done < "$CRON_FILE"

echo ""
echo "Migration complete. Run the following commands to enable:"
echo "  systemctl daemon-reload"
echo "  # Enable one by one (recommended to test before enabling)"
echo "  systemctl enable --now <task-name>.timer"

Summary

Scheduled tasks may seem simple — just running a script on a timer, right? — but managing them in production is a systems engineering challenge. From cron to systemd timer to distributed scheduling, each layer addresses the limitations of the previous one. Key takeaways:

cron works but isn’t good enough: cron’s syntax is simple and ubiquitous, but it lacks error retry, timeout control, concurrency protection, and log integration. cron is fine for ad-hoc tasks, but don’t rely on it alone for critical tasks
systemd timer is the better single-machine solution: journal log integration, resource limits, persistent catch-up, random delay for avalanche prevention — these features transform scheduled tasks from “runs” to “reliable.” Choose systemd timer for new projects
Four task design principles: Idempotency (safe re-runs), concurrency control (flock or systemd), timeout protection (timeout or TimeoutStartSec), environment isolation (explicit PATH and env vars)
Failures must trigger alerts: cron task failures are silent by default — this is the biggest hidden danger. Use a wrapper script for unified logging and alerting, or use systemd’s OnFailure= mechanism
Distributed scenarios need specialized tools: K8s CronJob for K8s-native scenarios, XXL-Job for small-to-medium Java teams, Airflow for complex data pipeline DAGs. The choice depends on your team’s tech stack and task complexity
Task inventory should be version-controlled: Maintain all scheduled task configurations (name, schedule, command, timeout, owner) in a version-controlled YAML file, with scripts to auto-generate systemd configs. This way task changes have reviewable diffs, and anyone on the team can see “what scheduled tasks are running on the system”
Regular inspection is essential: Scheduled tasks need monitoring too. Inspection scripts check “which tasks have never executed,” “which tasks failed today,” “which tasks have abnormally long execution times” — ensuring the scheduled task system itself is healthy

Scheduled tasks are the infrastructure of automated operations. Their reliability directly impacts business stability. When you bring every scheduled task under version control, equip it with alerting, and inspect it regularly, the “woken up at 3 AM because some cron script died” scenario becomes increasingly rare.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

cron Wikipedia — En, referenced for cron Wikipedia
systemd.timer Official Documentation — freedesktop.org, referenced for systemd.timer Official Documentation

Overview#

I. cron Syntax and Limitations#

1.1 cron Expressions#

1.2 crontab Management#

1.3 Limitations of cron#

II. systemd timer#

2.1 Basic Concepts#

2.2 Creating a Timer#

2.3 OnCalendar Syntax#

2.4 cron vs systemd timer Comparison#

III. Task Design Principles#

3.1 Idempotency#

3.2 Concurrency Control#

3.3 Timeout Control#

3.4 Environment Isolation#

IV. Failure Retry and Alerting#

4.1 systemd Failure Retry#

4.2 cron Task Alerting Wrapper#

V. Log Integration#

5.1 systemd Journal Logging#

5.2 Structured Logging#

5.3 Log Aggregation and Querying#

VI. Distributed Scheduled Tasks#

6.1 Distributed Challenges with Single-Machine cron#

6.2 Solution Comparison#

6.3 K8s CronJob#

6.4 XXL-Job#

6.5 Airflow DAG#

VII. Production Practices#

7.1 Scheduled Task Inventory Management#

7.2 Generating systemd timers from Inventory#

7.3 Scheduled Task Inspection#

7.4 Migrating cron to systemd timer#

Summary#

References & Acknowledgments#