Production-Grade Bash Scripting Guide

Overview

Bash is the most widely used scripting language in the ops world — nearly every Linux server ships with the interpreter, requiring no runtime installation. But Bash is also the language most prone to producing scripts that “work but can’t be trusted”: no type checking, errors are silent by default, variables get word-split without quotes, and pipeline failures get swallowed. The root cause of many production incidents is a Bash script that didn’t handle errors properly. This article systematically covers production-grade Bash scripting practices, so your scripts are not just “working” but “trustworthy.”

References: Google Shell Style Guide, ShellCheck

I. Script Skeleton: set -euo pipefail

1.1 The Three-Line Header

Every production-grade Bash script should start with:

#!/usr/bin/env bash
set -euo pipefail

What these three lines do:

Option	Effect	Consequence Without It
`-e` (`errexit`)	Exit immediately on command failure	Errors are ignored, script continues, potentially catastrophic
`-u` (`nounset`)	Error on undefined variable usage	Misspelled variable names silently return empty strings
`-o pipefail`	Pipeline fails if any command fails	Earlier pipeline failures are ignored; only the last command’s exit code matters

A classic anti-pattern:

#!/bin/bash
# No safety settings at all

cd /nonexistent/directory     # Fails, but script continues
rm -rf *                      # Executes in the current directory! Disaster

With set -euo pipefail:

#!/usr/bin/env bash
set -euo pipefail

cd /nonexistent/directory     # Fails, script exits immediately
rm -rf *                      # Never reaches this point

1.2 Complete Script Template

#!/usr/bin/env bash
#
# script_name.sh - One-line description of what the script does
#
# Usage: script_name.sh [options] <arguments>
#
# Author: Xu Baojin
# Created: 2026-07-10
#

set -euo pipefail
IFS=$'\n\t'

# === Global Variables ===
SCRIPT_NAME="$(basename "$0")"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
VERSION="1.0.0"
VERBOSE=0
DRY_RUN=false

# === Color Definitions ===
if [[ -t 1 ]]; then
    RED='\033[0;31m'
    GREEN='\033[0;32m'
    YELLOW='\033[0;33m'
    BLUE='\033[0;34m'
    NC='\033[0m' # No Color
else
    RED='' GREEN='' YELLOW='' BLUE='' NC=''
fi

# === Logging Functions ===
log_info()  { echo -e "${GREEN}[INFO]${NC}  $(date '+%Y-%m-%d %H:%M:%S') $*"; }
log_warn()  { echo -e "${YELLOW}[WARN]${NC}  $(date '+%Y-%m-%d %H:%M:%S') $*" >&2; }
log_error() { echo -e "${RED}[ERROR]${NC} $(date '+%Y-%m-%d %H:%M:%S') $*" >&2; }
log_debug() { [[ "${VERBOSE}" -ge 1 ]] && echo -e "${BLUE}[DEBUG]${NC} $(date '+%Y-%m-%d %H:%M:%S') $*" >&2; }

# === Cleanup Function ===
cleanup() {
    local exit_code=$?
    log_debug "Cleaning up temporary files..."
    rm -f "${TMP_FILE:-}" 2>/dev/null || true
    exit "$exit_code"
}
trap cleanup EXIT

# === Usage ===
usage() {
    cat <<EOF
Usage: ${SCRIPT_NAME} [options] <arguments>

Options:
    -h, --help          Show help information
    -v, --verbose       Verbose output
    -n, --dry-run       Print without executing
    -V, --version       Show version number
    -c, --config FILE   Specify config file

Examples:
    ${SCRIPT_NAME} -v deploy production
    ${SCRIPT_NAME} --config /etc/app.conf deploy staging
EOF
    exit "${1:-0}"
}

# === Argument Parsing ===
TEMP_FILE="$(mktemp)"
log_info "Script started: ${SCRIPT_NAME} v${VERSION}"

# Main logic...

II. Error Handling and Logging

2.1 Error Handler Function

#!/usr/bin/env bash
set -euo pipefail

# Error handler: prints failure location
error_handler() {
    local exit_code=$?
    local line_no=$1
    local command=$2

    echo ""
    echo "========================================" >&2
    echo " Script Execution Failed" >&2
    echo "========================================" >&2
    echo " Exit code: ${exit_code}" >&2
    echo " Line:      ${line_no}" >&2
    echo " Command:   ${command}" >&2
    echo " Time:      $(date '+%Y-%m-%d %H:%M:%S')" >&2
    echo "========================================" >&2

    # Send alert
    if command -v curl &>/dev/null && [[ -n "${WEBHOOK_URL:-}" ]]; then
        curl -sf -X POST "${WEBHOOK_URL}" \
            -H 'Content-Type: application/json' \
            -d "{\"text\":\"Script failed: ${SCRIPT_NAME} line ${line_no} command: ${command}\"}" \
            2>/dev/null || true
    fi

    exit "$exit_code"
}

trap 'error_handler ${LINENO} "${BASH_COMMAND}"' ERR

2.2 Safe Execution Wrapper

#!/usr/bin/env bash
set -euo pipefail

# run_cmd: command executor with logging and error handling
run_cmd() {
    local desc="$1"; shift
    local cmd=("$@")

    log_info "Executing: ${desc}"
    log_debug "Command: ${cmd[*]}"

    if [[ "${DRY_RUN}" == "true" ]]; then
        log_warn "[DRY-RUN] ${cmd[*]}"
        return 0
    fi

    local start_time end_time duration exit_code
    start_time=$(date +%s)

    if "${cmd[@]}"; then
        exit_code=0
    else
        exit_code=$?
    fi

    end_time=$(date +%s)
    duration=$((end_time - start_time))

    if [[ ${exit_code} -eq 0 ]]; then
        log_info "Done: ${desc} (${duration}s)"
    else
        log_error "Failed: ${desc} (exit_code=${exit_code}, ${duration}s)"
        return ${exit_code}
    fi
}

# safe_eval: execute a string command, capture output
safe_eval() {
    local cmd="$1"
    local output exit_code

    output=$(eval "$cmd" 2>&1) && exit_code=0 || exit_code=$?

    echo "$output"
    return ${exit_code}
}

# retry: command execution with retries
retry() {
    local max_attempts=$1
    local delay=$2
    shift 2
    local attempt=1
    local exit_code=0

    while [[ ${attempt} -le ${max_attempts} ]]; do
        log_debug "Attempt ${attempt}/${max_attempts}: $*"
        if "$@"; then
            return 0
        fi
        exit_code=$?
        attempt=$((attempt + 1))
        if [[ ${attempt} -le ${max_attempts} ]]; then
            log_warn "Failed (exit_code=${exit_code}), retrying in ${delay}s..."
            sleep "${delay}"
        fi
    done

    log_error "Failed after ${max_attempts} retries"
    return ${exit_code}
}

# Usage examples
run_cmd "Install dependencies" apt-get install -y nginx
retry 3 5 curl -sf http://example.com/health

2.3 Logging System

#!/usr/bin/env bash
set -euo pipefail

# === Log Configuration ===
LOG_FILE="/var/log/${SCRIPT_NAME:-script}.log"
LOG_LEVEL="INFO"  # DEBUG, INFO, WARN, ERROR
LOG_TO_FILE=true
LOG_TO_STDOUT=true
LOG_MAX_SIZE="10M"
LOG_KEEP=5

# Numeric mapping for log levels
_log_level_num() {
    case "$1" in
        DEBUG) echo 0 ;;
        INFO)  echo 1 ;;
        WARN)  echo 2 ;;
        ERROR) echo 3 ;;
        *)     echo 1 ;;
    esac
}

# Log output
_log() {
    local level="$1"; shift
    local msg="$*"
    local timestamp
    timestamp=$(date '+%Y-%m-%d %H:%M:%S')

    local level_num msg_level_num
    msg_level_num=$(_log_level_num "${level}")
    level_num=$(_log_level_num "${LOG_LEVEL}")

    # Skip messages below configured level
    [[ ${msg_level_num} -lt ${level_num} ]] && return 0

    local line="[${timestamp}] [${level}] ${msg}"

    if [[ "${LOG_TO_STDOUT}" == "true" ]]; then
        case "${level}" in
            ERROR) echo -e "${RED}${line}${NC}" >&2 ;;
            WARN)  echo -e "${YELLOW}${line}${NC}" >&2 ;;
            INFO)  echo -e "${GREEN}${line}${NC}" ;;
            DEBUG) echo -e "${BLUE}${line}${NC}" ;;
        esac
    fi

    if [[ "${LOG_TO_FILE}" == "true" ]]; then
        echo "${line}" >> "${LOG_FILE}"
    fi
}

log_debug() { _log DEBUG "$@"; }
log_info()  { _log INFO "$@"; }
log_warn()  { _log WARN "$@"; }
log_error() { _log ERROR "$@"; }

# Log rotation
log_rotate() {
    local file="$1"
    local max_size="$2"
    local keep="$3"

    [[ ! -f "${file}" ]] && return 0

    local size
    size=$(stat -c %s "${file}" 2>/dev/null || echo 0)
    local max_bytes
    max_bytes=$(numfmt --from=iec "${max_size}" 2>/dev/null || echo 10485760)

    if [[ ${size} -gt ${max_bytes} ]]; then
        for ((i = keep; i >= 1; i--)); do
            [[ -f "${file}.$((i-1))" ]] && mv "${file}.$((i-1))" "${file}.${i}"
        done
        mv "${file}" "${file}.0"
        gzip "${file}.0" 2>/dev/null || true
        touch "${file}"
        log_info "Log rotation complete: ${file}"
    fi
}

III. Signal Handling

3.1 Catching Signals

#!/usr/bin/env bash
set -euo pipefail

# Global state
RUNNING=true
CHILD_PIDS=()

# Signal handler function
handle_interrupt() {
    echo ""
    log_warn "Interrupt signal received (SIGINT/SIGTERM), cleaning up..."

    RUNNING=false

    # Terminate all child processes
    for pid in "${CHILD_PIDS[@]}"; do
        if kill -0 "$pid" 2>/dev/null; then
            log_debug "Terminating child process: $pid"
            kill -TERM "$pid" 2>/dev/null || true
        fi
    done

    # Wait for child processes to exit
    wait 2>/dev/null

    log_info "Cleanup complete, exiting"
    exit 130
}

handle_exit() {
    local exit_code=$?
    # Clean up temporary files
    rm -f "${TMP_DIR:-}"/* 2>/dev/null || true
    rm -rf "${TMP_DIR:-}" 2>/dev/null || true
    exit "$exit_code"
}

# Register signal handlers
trap handle_interrupt SIGINT SIGTERM
trap handle_exit EXIT

3.2 Graceful Shutdown Pattern

#!/usr/bin/env bash
set -euo pipefail

# Graceful shutdown: finish current task before exiting
GRACEFUL_SHUTDOWN=false
CURRENT_TASK=""

handle_sigterm() {
    log_warn "SIGTERM received, initiating graceful shutdown..."
    GRACEFUL_SHUTDOWN=true
}

trap handle_sigterm SIGTERM

process_items() {
    local items=("$@")
    local total=${#items[@]}
    local current=0

    for item in "${items[@]}"; do
        # Check if graceful shutdown is needed
        if [[ "${GRACEFUL_SHUTDOWN}" == "true" ]]; then
            log_warn "Graceful shutdown: processed ${current}/${total}"
            break
        fi

        CURRENT_TASK="Processing ${item}"
        log_info "${CURRENT_TASK} (${current}/${total})"

        # Simulate processing
        sleep 2

        current=$((current + 1))
    done

    log_info "Processing complete: ${current}/${total}"
}

process_items "task1" "task2" "task3" "task4" "task5"

IV. Parallel Execution

4.1 Background Task Management

#!/usr/bin/env bash
set -euo pipefail

# Parallel execution function
parallel_run() {
    local max_jobs=$1
    shift

    local pids=()
    local jobs_running=0

    for cmd in "$@"; do
        # Wait for a slot
        while [[ ${jobs_running} -ge ${max_jobs} ]]; do
            wait -n 2>/dev/null || {
                # Fallback when wait -n is unavailable
                for pid in "${pids[@]}"; do
                    if ! kill -0 "$pid" 2>/dev/null; then
                        pids=("${pids[@]/$pid}")
                        jobs_running=$((jobs_running - 1))
                        break
                    fi
                done
                sleep 0.1
            }
            jobs_running=$(jobs -r | wc -l)
        done

        # Launch background task
        eval "$cmd" &
        pids+=($!)
        jobs_running=$((jobs_running + 1))
        log_debug "Started task: $cmd (PID: $!)"
    done

    # Wait for all tasks to complete
    local fail_count=0
    for pid in "${pids[@]}"; do
        if ! wait "$pid"; then
            fail_count=$((fail_count + 1))
            log_error "Task failed: PID $pid"
        fi
    done

    return ${fail_count}
}

# Parallel execution with xargs
parallel_xargs() {
    local max_jobs=$1
    shift
    local cmd="$1"
    shift

    printf '%s\n' "$@" | xargs -P "${max_jobs}" -I {} bash -c "${cmd} {}"
}

4.2 Batch Parallel SSH Execution

#!/usr/bin/env bash
set -euo pipefail

# Configuration
HOSTS_FILE="hosts.txt"
MAX_PARALLEL=10
SSH_TIMEOUT=30
SSH_USER="deploy"
COMMAND="$*"

if [[ -z "${COMMAND}" ]]; then
    echo "Usage: $0 <command>"
    exit 1
fi

# Result directory
RESULT_DIR=$(mktemp -d)
trap "rm -rf ${RESULT_DIR}" EXIT

# Parallel SSH execution
run_on_host() {
    local host=$1
    local result_file="${RESULT_DIR}/${host}.result"

    ssh -o ConnectTimeout="${SSH_TIMEOUT}" \
        -o StrictHostKeyChecking=no \
        -o BatchMode=yes \
        "${SSH_USER}@${host}" "${COMMAND}" \
        > "${result_file}.out" 2> "${result_file}.err"

    local exit_code=$?
    echo "${exit_code}" > "${result_file}.code"

    if [[ ${exit_code} -eq 0 ]]; then
        log_info "[OK] ${host}"
    else
        log_error "[FAIL] ${host} (exit=${exit_code})"
    fi
}

export -f run_on_host
export SSH_TIMEOUT SSH_USER COMMAND RESULT_DIR
export -f log_info log_error

# Parallel execution with xargs
cat "${HOSTS_FILE}" | xargs -P "${MAX_PARALLEL}" -I {} bash -c 'run_on_host "{}'

# Summarize results
echo ""
echo "=== Execution Results Summary ==="
total=0
success=0
failed=0

for host in $(cat "${HOSTS_FILE}"); do
    total=$((total + 1))
    code_file="${RESULT_DIR}/${host}.result.code"
    if [[ -f "${code_file}" ]] && [[ "$(cat "${code_file}")" == "0" ]]; then
        success=$((success + 1))
    else
        failed=$((failed + 1))
        echo "  [FAIL] ${host}"
        [[ -f "${RESULT_DIR}/${host}.result.err" ]] && \
            head -5 "${RESULT_DIR}/${host}.result.err" | sed 's/^/    /'
    fi
done

echo ""
echo "Total: ${total}  Success: ${success}  Failed: ${failed}"

V. Configuration Management

5.1 Config File Loading

#!/usr/bin/env bash
set -euo pipefail

# Default config
DEFAULT_CONFIG="/etc/myapp/config.conf"
USER_CONFIG="${HOME}/.config/myapp/config.conf"
LOCAL_CONFIG="./config.local"

# Load config files (later files override earlier ones)
load_config() {
    local config_files=("$@")

    for config_file in "${config_files[@]}"; do
        if [[ -f "${config_file}" ]]; then
            log_debug "Loading config: ${config_file}"
            # Safe loading: only allow KEY=VALUE format
            while IFS='=' read -r key value; do
                # Skip comments and empty lines
                [[ "$key" =~ ^[[:space:]]*# ]] && continue
                [[ -z "$key" ]] && continue
                # Strip leading/trailing whitespace
                key="${key// /}"
                value="${value# }"
                value="${value% }"
                # Strip quotes
                value="${value#\"}"
                value="${value%\"}"
                value="${value#\'}"
                value="${value%\'}"
                # Export as environment variable
                export "${key}=${value}"
            done < "${config_file}"
        fi
    done
}

# Load config (priority from low to high)
load_config "${DEFAULT_CONFIG}" "${USER_CONFIG}" "${LOCAL_CONFIG}"

5.2 Environment Variable Override

#!/usr/bin/env bash
set -euo pipefail

# Default values (can be overridden by environment variables)
: "${APP_PORT:=8080}"
: "${APP_HOST:=0.0.0.0}"
: "${APP_ENV:=development}"
: "${DB_HOST:=localhost}"
: "${DB_PORT:=5432}"
: "${DB_NAME:=myapp}"
: "${LOG_LEVEL:=info}"
: "${MAX_WORKERS:=4}"

# Config validation
validate_config() {
    local errors=0

    # Port range check
    if ! [[ "${APP_PORT}" =~ ^[0-9]+$ ]] || \
       [[ "${APP_PORT}" -lt 1 || "${APP_PORT}" -gt 65535 ]]; then
        log_error "Invalid APP_PORT: ${APP_PORT}"
        errors=$((errors + 1))
    fi

    # Environment check
    if ! [[ "${APP_ENV}" =~ ^(development|staging|production)$ ]]; then
        log_error "Invalid APP_ENV: ${APP_ENV} (allowed: development|staging|production)"
        errors=$((errors + 1))
    fi

    # Additional checks for production
    if [[ "${APP_ENV}" == "production" ]]; then
        if [[ "${LOG_LEVEL}" == "debug" ]]; then
            log_warn "Debug log level is not recommended in production"
        fi
        if [[ "${MAX_WORKERS}" -lt 2 ]]; then
            log_error "MAX_WORKERS must be at least 2 in production"
            errors=$((errors + 1))
        fi
    fi

    if [[ ${errors} -gt 0 ]]; then
        log_error "Config validation failed, ${errors} error(s) found"
        exit 1
    fi

    log_info "Config validation passed"
}

validate_config

VI. Argument Parsing

6.1 Short Options with getopts

#!/usr/bin/env bash
set -euo pipefail

# Default values
ENVIRONMENT="development"
VERBOSE=0
DRY_RUN=false
CONFIG_FILE=""
OUTPUT_DIR="."

# Parse short options with getopts
while getopts ":e:vc:dno:h" opt; do
    case ${opt} in
        e ) ENVIRONMENT="$OPTARG" ;;
        v ) VERBOSE=$((VERBOSE + 1)) ;;
        c ) CONFIG_FILE="$OPTARG" ;;
        d ) DRY_RUN=true ;;
        n ) DRY_RUN=true ;;
        o ) OUTPUT_DIR="$OPTARG" ;;
        h ) usage; exit 0 ;;
        \? ) echo "Invalid option: -$OPTARG" >&2; usage; exit 1 ;;
        : ) echo "Option -$OPTARG requires an argument" >&2; usage; exit 1 ;;
    esac
done
shift $((OPTIND - 1))

# Remaining arguments
ACTION="${1:-}"
[[ -z "${ACTION}" ]] && { echo "Missing action argument"; usage; exit 1; }

6.2 Long Option Parsing

#!/usr/bin/env bash
set -euo pipefail

# Long option parsing function
parse_args() {
    while [[ $# -gt 0 ]]; do
        case "$1" in
            --help|-h)
                usage; exit 0 ;;
            --version|-V)
                echo "${VERSION}"; exit 0 ;;
            --verbose|-v)
                VERBOSE=$((VERBOSE + 1)); shift ;;
            --dry-run|-n)
                DRY_RUN=true; shift ;;
            --env|-e)
                ENVIRONMENT="${2:-}"; shift 2 ;;
            --env=*)
                ENVIRONMENT="${1#*=}"; shift ;;
            --config|-c)
                CONFIG_FILE="${2:-}"; shift 2 ;;
            --config=*)
                CONFIG_FILE="${1#*=}"; shift ;;
            --output|-o)
                OUTPUT_DIR="${2:-}"; shift 2 ;;
            --output=*)
                OUTPUT_DIR="${1#*=}"; shift ;;
            --port)
                PORT="${2:-}"; shift 2 ;;
            --port=*)
                PORT="${1#*=}"; shift ;;
            --)
                shift; break ;;  # Arguments after -- are not parsed
            -*)
                echo "Unknown option: $1" >&2; usage; exit 1 ;;
            *)
                break ;;
        esac
    done

    # Remaining positional arguments
    POSITIONAL_ARGS=("$@")
}

parse_args "$@"

VII. Unit Testing with bats

7.1 Installing bats

# macOS
brew install bats-core

# Ubuntu/Debian
apt-get install bats

# Install from source (recommended, for latest version)
git clone https://github.com/bats-core/bats-core.git
cd bats-core
./install.sh "$HOME/.local"

7.2 Writing Tests

#!/usr/bin/env bats
# test_deploy.bats - Deploy script tests

# Load the script under test (only loads functions, doesn't execute main logic)
load 'test_helper'

setup() {
    # Setup before each test
    export TEST_DIR=$(mktemp -d)
    export APP_NAME="test-app"
    export DEPLOY_DIR="${TEST_DIR}/deploy"
    mkdir -p "${DEPLOY_DIR}"
}

teardown() {
    # Cleanup after each test
    rm -rf "${TEST_DIR}"
}

@test "create_release_dir should create version directory" {
    create_release_dir "1.0.0"
    [ -d "${DEPLOY_DIR}/releases/1.0.0" ]
}

@test "create_release_dir should fail when directory already exists" {
    create_release_dir "1.0.0"
    run create_release_dir "1.0.0"
    [ "$status" -ne 0 ]
}

@test "switch_symlink should correctly switch symlink" {
    mkdir -p "${DEPLOY_DIR}/releases/1.0.0"
    mkdir -p "${DEPLOY_DIR}/releases/2.0.0"
    ln -sfn "${DEPLOY_DIR}/releases/1.0.0" "${DEPLOY_DIR}/current"

    switch_symlink "2.0.0"

    [ "$(readlink ${DEPLOY_DIR}/current)" = "${DEPLOY_DIR}/releases/2.0.0" ]
}

@test "rollback should revert to previous version" {
    mkdir -p "${DEPLOY_DIR}/releases/1.0.0"
    mkdir -p "${DEPLOY_DIR}/releases/1.1.0"
    ln -sfn "${DEPLOY_DIR}/releases/1.1.0" "${DEPLOY_DIR}/current"

    run rollback
    [ "$status" -eq 0 ]
    [ "$(readlink ${DEPLOY_DIR}/current)" = "${DEPLOY_DIR}/releases/1.0.0" ]
}

@test "validate_config should fail with invalid port" {
    export APP_PORT="99999"
    run validate_config
    [ "$status" -ne 0 ]
    [[ "$output" == *"APP_PORT"* ]]
}

@test "health_check should return 0 when service is healthy" {
    # Mock a healthy service
    start_mock_server 8080 &
    local mock_pid=$!
    sleep 1

    run health_check "localhost:8080"
    [ "$status" -eq 0 ]

    kill "$mock_pid" 2>/dev/null
}

7.3 Running Tests

# Run all tests
bats test/

# Run a specific test file
bats test/test_deploy.bats

# Run only matching tests
bats --filter "symlink" test/

# Output TAP format (for CI integration)
bats --tap test/ | tee test-results.tap

# Output JUnit format
bats --formatter junit test/ > test-results.xml

7.4 GitHub Actions Integration

# .github/workflows/test.yml
name: Shell Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Install bats
        run: |
          git clone https://github.com/bats-core/bats-core.git
          cd bats-core && ./install.sh "$HOME/.local"          
      - name: Install bats-support and bats-assert
        run: |
          git clone https://github.com/bats-core/bats-support.git test/test_helper/bats-support
          git clone https://github.com/bats-core/bats-assert.git test/test_helper/bats-assert          
      - name: Run ShellCheck
        uses: ludeeus/action-shellcheck@master
        with:
          severity: warning
      - name: Run bats tests
        run: export PATH="$HOME/.local/bin:$PATH" && bats test/

VIII. ShellCheck

8.1 Installation and Usage

# Install
apt-get install shellcheck      # Debian/Ubuntu
brew install shellcheck          # macOS
# Or via Docker
docker run --rm -v "$PWD:/mnt" koalaman/shellcheck:stable check.sh

# Basic usage
shellcheck script.sh

# Specify severity level
shellcheck --severity=error script.sh    # Only errors
shellcheck --severity=warning script.sh  # error + warning
shellcheck --severity=info script.sh     # error + warning + info
shellcheck --severity=style script.sh    # All (default)

# Output format
shellcheck --format=gcc script.sh        # file:line:col: message (default)
shellcheck --format=json script.sh       # JSON format
shellcheck --format=checkstyle script.sh # CheckStyle XML (for CI integration)

8.2 Common Issues and Fixes

# === SC2086: Double-quote to prevent word splitting ===

# Wrong
cp $file $destination

# Correct
cp "$file" "$destination"

# === SC2046: Quote command substitutions ===

# Wrong
for f in $(ls *.txt); do

# Correct
for f in *.txt; do
    # Use glob directly

# === SC2004: $ is not needed in arithmetic expressions ===

# Wrong
if [[ $(( $a + $b )) -gt 10 ]]; then

# Correct
if (( a + b > 10 )); then

# === SC2181: Check command exit code directly ===

# Wrong
mkdir /opt/app
if [ $? -ne 0 ]; then
    echo "Failed"
fi

# Correct
if ! mkdir /opt/app; then
    echo "Failed"
fi

# Or use set -e to fail automatically
set -euo pipefail
mkdir /opt/app

# === SC2155: Declare and assign separately ===

# Wrong (command substitution exit code is masked)
local var=$(some_command)

# Correct
local var
var=$(some_command)

# === SC2230: Use command -v instead of which ===

# Wrong
if which docker > /dev/null; then

# Correct
if command -v docker > /dev/null; then

# === SC1090/SC1091: Don't source files from uncertain sources ===

# Dangerous
source /tmp/downloaded_script.sh

# Safe: validate before sourcing
if [[ -f /etc/app/config.sh ]] && [[ -O /etc/app/config.sh ]]; then
    source /etc/app/config.sh
fi

8.3 Inline Disabling

# Disable single-line check
echo "Line that doesn't need checking" # shellcheck disable=SC2086

# Disable for entire file
# shellcheck disable=SC1090
source "$CONFIG_FILE"

# Disable multiple rules
# shellcheck disable=SC1090,SC2154,SC2034

8.4 .shellcheckrc Configuration

# .shellcheckrc
disable=SC1090,SC1091,SC2154,SC2034
external-sources=true

IX. Secure Coding

9.1 Path Safety

#!/usr/bin/env bash
set -euo pipefail

# Safe path handling
safe_path() {
    local path="$1"
    local base_dir="$2"

    # Convert to absolute path
    local abs_path
    abs_path=$(realpath -m "$path" 2>/dev/null || readlink -f "$path" 2>/dev/null || echo "$path")

    # Check for path traversal
    if [[ "$path" == *".."* ]]; then
        log_error "Path contains .. : $path"
        return 1
    fi

    # Check if path is within allowed base directory
    case "$abs_path" in
        "$base_dir"/*)
            echo "$abs_path"
            return 0
            ;;
        *)
            log_error "Path out of bounds: $abs_path (base: $base_dir)"
            return 1
            ;;
    esac
}

# Safe deletion: prevent rm -rf / disaster
safe_rm() {
    local target="$1"

    # Never allow deleting root or home directory
    case "$target" in
        "/"|"/home"|"$HOME"|"/usr"|"/var"|"/etc"|"/bin"|"/sbin")
            log_error "Refusing to delete critical directory: $target"
            return 1
            ;;
    esac

    # Check that path starts with / but is not /
    if [[ "$target" == /* ]] && [[ "$target" != "/" ]]; then
        # Verify path exists
        if [[ ! -e "$target" ]]; then
            log_warn "Path does not exist: $target"
            return 0
        fi

        # Execute deletion
        log_warn "Deleting: $target"
        if [[ "${DRY_RUN:-false}" != "true" ]]; then
            rm -rf -- "$target"
        fi
    else
        log_error "Path must be absolute: $target"
        return 1
    fi
}

9.2 Input Validation

#!/usr/bin/env bash
set -euo pipefail

# Validate IP address
validate_ip() {
    local ip="$1"
    if [[ "$ip" =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}$ ]]; then
        local IFS='.'
        read -ra octets <<< "$ip"
        for octet in "${octets[@]}"; do
            if [[ "$octet" -gt 255 ]]; then
                return 1
            fi
        done
        return 0
    fi
    return 1
}

# Validate port number
validate_port() {
    local port="$1"
    if [[ "$port" =~ ^[0-9]+$ ]] && [[ "$port" -ge 1 ]] && [[ "$port" -le 65535 ]]; then
        return 0
    fi
    return 1
}

# Validate hostname
validate_hostname() {
    local host="$1"
    # RFC 1123 hostname rules
    if [[ "$host" =~ ^[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(\.[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$ ]]; then
        return 0
    fi
    return 1
}

# Safe SQL value escaping (prevent injection)
escape_sql() {
    local value="$1"
    # Escape single quotes
    echo "${value//\'/\'\'}"
}

# Validate filename safety
validate_filename() {
    local filename="$1"
    # Only allow alphanumeric, underscore, hyphen, dot
    if [[ "$filename" =~ ^[a-zA-Z0-9._-]+$ ]]; then
        return 0
    fi
    return 1
}

9.3 Temporary File Safety

#!/usr/bin/env bash
set -euo pipefail

# Safely create a temporary file
create_temp_file() {
    local prefix="${1:-tmp}"
    local tmp_file

    # mktemp creates a unique file with 600 permissions
    tmp_file=$(mktemp "/tmp/${prefix}.XXXXXX")
    echo "$tmp_file"
}

# Safely create a temporary directory
create_temp_dir() {
    local prefix="${1:-tmp}"
    local tmp_dir

    # mktemp -d creates a unique directory with 700 permissions
    tmp_dir=$(mktemp -d "/tmp/${prefix}.XXXXXX")
    echo "$tmp_dir"
}

# Usage example
main() {
    local tmp_file tmp_dir

    tmp_file=$(create_temp_file "deploy")
    tmp_dir=$(create_temp_dir "build")

    # Ensure cleanup on exit
    trap "rm -f '${tmp_file}'; rm -rf '${tmp_dir}'" EXIT

    # Use temporary file...
    echo "data" > "$tmp_file"

    # Temp file already has 600 permissions, but can be set explicitly
    chmod 600 "$tmp_file"
}

X. Complete Production-Grade Script Example

#!/usr/bin/env bash
#
# safe_deploy.sh - Safe deployment script
#
# Usage: safe_deploy.sh --env <env> --app <name> [--version <ver>] [--dry-run]
#
# Features:
#   - Complete error handling and logging
#   - Signal handling and graceful shutdown
#   - Argument validation and config management
#   - Rollback mechanism
#   - Health check
#

set -euo pipefail
IFS=$'\n\t'

# === Constants ===
readonly SCRIPT_NAME="$(basename "$0")"
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly VERSION="2.0.0"
readonly LOCK_FILE="/tmp/${SCRIPT_NAME}.lock"

# === Global Variables ===
ENVIRONMENT=""
APP_NAME=""
APP_VERSION=""
DRY_RUN=false
VERBOSE=0
BACKUP_DIR=""
RELEASE_DIR=""
CURRENT_LINK=""
KEEP_RELEASES=5

# === Colors ===
if [[ -t 1 ]]; then
    readonly RED=$'\033[0;31m'
    readonly GREEN=$'\033[0;32m'
    readonly YELLOW=$'\033[0;33m'
    readonly BLUE=$'\033[0;34m'
    readonly NC=$'\033[0m'
else
    readonly RED='' GREEN='' YELLOW='' BLUE='' NC=''
fi

# === Logging ===
log_info()  { echo "${GREEN}[INFO]${NC}  $(date '+%H:%M:%S') $*"; }
log_warn()  { echo "${YELLOW}[WARN]${NC}  $(date '+%H:%M:%S') $*" >&2; }
log_error() { echo "${RED}[ERROR]${NC} $(date '+%H:%M:%S') $*" >&2; }
log_debug() { [[ $VERBOSE -ge 1 ]] && echo "${BLUE}[DEBUG]${NC} $(date '+%H:%M:%S') $*" >&2; }

# === Error Handling ===
on_error() {
    local exit_code=$?
    local line=$1
    log_error "Script failed: line ${line}, exit code ${exit_code}"
    if [[ -n "${CURRENT_LINK:-}" ]] && [[ -n "${BACKUP_DIR:-}" ]]; then
        log_warn "Attempting automatic rollback..."
        if [[ -d "${BACKUP_DIR}" ]]; then
            ln -sfn "${BACKUP_DIR}" "${CURRENT_LINK}" 2>/dev/null || true
            log_info "Rolled back to previous version"
        fi
    fi
    release_lock
    exit ${exit_code}
}
trap 'on_error ${LINENO}' ERR

# === Signal Handling ===
on_interrupt() {
    log_warn "Interrupt signal received, exiting..."
    release_lock
    exit 130
}
trap on_interrupt SIGINT SIGTERM

# === File Lock ===
acquire_lock() {
    if [[ -f "${LOCK_FILE}" ]]; then
        local pid
        pid=$(cat "${LOCK_FILE}" 2>/dev/null || echo "")
        if [[ -n "$pid" ]] && kill -0 "$pid" 2>/dev/null; then
            log_error "Another instance is running (PID: $pid)"
            exit 1
        fi
        log_warn "Found stale lock file, removed"
        rm -f "${LOCK_FILE}"
    fi
    echo $$ > "${LOCK_FILE}"
}

release_lock() {
    rm -f "${LOCK_FILE}" 2>/dev/null || true
}
trap release_lock EXIT

# === Usage ===
usage() {
    cat <<EOF
Usage: ${SCRIPT_NAME} [options]

Required options:
    -e, --env ENV        Environment (staging|production)
    -a, --app NAME       Application name

Optional options:
    -v, --version VER    Deploy version (default: latest git tag)
    -n, --dry-run        Print without executing
    -V, --verbose        Verbose output
    -k, --keep N         Number of releases to keep (default: 5)
    -h, --help           Show help

Examples:
    ${SCRIPT_NAME} --env production --app myapp
    ${SCRIPT_NAME} -e staging -a api --version v1.2.3 --dry-run
EOF
    exit "${1:-0}"
}

# === Argument Parsing ===
parse_args() {
    while [[ $# -gt 0 ]]; do
        case "$1" in
            -e|--env)       ENVIRONMENT="$2"; shift 2 ;;
            --env=*)        ENVIRONMENT="${1#*=}"; shift ;;
            -a|--app)       APP_NAME="$2"; shift 2 ;;
            --app=*)        APP_NAME="${1#*=}"; shift ;;
            -v|--version)   APP_VERSION="$2"; shift 2 ;;
            --version=*)    APP_VERSION="${1#*=}"; shift ;;
            -n|--dry-run)   DRY_RUN=true; shift ;;
            -V|--verbose)   VERBOSE=$((VERBOSE+1)); shift ;;
            -k|--keep)      KEEP_RELEASES="$2"; shift 2 ;;
            -h|--help)      usage 0 ;;
            *)              log_error "Unknown option: $1"; usage 1 ;;
        esac
    done
}

# === Validation ===
validate() {
    local errors=0

    [[ -z "${ENVIRONMENT}" ]] && { log_error "Missing --env argument"; errors=$((errors+1)); }
    [[ -z "${APP_NAME}" ]] && { log_error "Missing --app argument"; errors=$((errors+1)); }

    if [[ "${ENVIRONMENT}" != "staging" && "${ENVIRONMENT}" != "production" ]]; then
        log_error "Invalid environment: ${ENVIRONMENT}"
        errors=$((errors+1))
    fi

    if ! [[ "${KEEP_RELEASES}" =~ ^[0-9]+$ ]] || [[ "${KEEP_RELEASES}" -lt 1 ]]; then
        log_error "Invalid keep count: ${KEEP_RELEASES}"
        errors=$((errors+1))
    fi

    [[ ${errors} -gt 0 ]] && exit 1

    # Set paths
    local base_dir="/opt/${APP_NAME}"
    RELEASE_DIR="${base_dir}/releases"
    CURRENT_LINK="${base_dir}/current"

    if [[ "${APP_VERSION}" == "" ]]; then
        APP_VERSION=$(git describe --tags --always 2>/dev/null || echo "unknown-$(date +%s)")
    fi

    log_info "Deploy configuration:"
    log_info "  Environment: ${ENVIRONMENT}"
    log_info "  App: ${APP_NAME}"
    log_info "  Version: ${APP_VERSION}"
    log_info "  Dry-run: ${DRY_RUN}"
}

# === Deploy Steps ===
pre_deploy_check() {
    log_info "Pre-deployment check..."

    # Check disk space
    local available
    available=$(df -m /opt | awk 'NR==2{print $4}')
    if [[ ${available} -lt 1024 ]]; then
        log_error "Insufficient disk space: ${available}MB (need >1024MB)"
        return 1
    fi
    log_debug "Disk space: ${available}MB"

    # Check if current version is already running
    if [[ -L "${CURRENT_LINK}" ]]; then
        local current_ver
        current_ver=$(basename "$(readlink "${CURRENT_LINK}")")
        if [[ "${current_ver}" == "${APP_VERSION}" ]]; then
            log_error "Version ${APP_VERSION} is already deployed"
            return 1
        fi
        BACKUP_DIR="$(readlink "${CURRENT_LINK}")"
        log_debug "Current version: ${current_ver}"
    fi
}

deploy() {
    log_info "Starting deployment of ${APP_NAME} ${APP_VERSION}"

    local target_dir="${RELEASE_DIR}/${APP_VERSION}"

    if [[ "${DRY_RUN}" == "true" ]]; then
        log_warn "[DRY-RUN] Create directory: ${target_dir}"
        log_warn "[DRY-RUN] Switch link: ${CURRENT_LINK} -> ${target_dir}"
        return 0
    fi

    # Create version directory
    mkdir -p "${target_dir}"
    log_debug "Created directory: ${target_dir}"

    # Copy files (simplified here; in practice, pull from artifact storage)
    log_info "Pulling artifact..."
    # cp -r build/* "${target_dir}/"

    # Switch symlink
    ln -sfn "${target_dir}" "${CURRENT_LINK}"
    log_info "Switched link: ${CURRENT_LINK} -> ${target_dir}"

    # Clean up old releases
    cleanup_old_releases
}

cleanup_old_releases() {
    log_info "Cleaning old releases (keeping ${KEEP_RELEASES})..."

    local releases=()
    while IFS= read -r dir; do
        releases+=("$dir")
    done < <(ls -1dt "${RELEASE_DIR}"/*/ 2>/dev/null | sed 's|/$||')

    local count=${#releases[@]}
    if [[ ${count} -le ${KEEP_RELEASES} ]]; then
        log_debug "Release count ${count} <= ${KEEP_RELEASES}, no cleanup needed"
        return 0
    fi

    local to_delete=$((count - KEEP_RELEASES))
    log_info "Cleaning ${to_delete} old release(s)"

    for ((i = KEEP_RELEASES; i < count; i++)); do
        local dir="${releases[$i]}"
        # Don't delete the currently active version
        if [[ "$(readlink "${CURRENT_LINK}")" != "${dir}" ]]; then
            log_debug "Deleting: ${dir}"
            [[ "${DRY_RUN}" != "true" ]] && rm -rf "${dir}"
        fi
    done
}

health_check() {
    log_info "Health check..."

    local max_retries=10
    local retry_interval=3

    for ((i = 1; i <= max_retries; i++)); do
        if curl -sf "http://localhost:8080/health" > /dev/null 2>&1; then
            log_info "Health check passed"
            return 0
        fi
        log_debug "Waiting for service to start... (${i}/${max_retries})"
        sleep "${retry_interval}"
    done

    log_error "Health check failed"
    return 1
}

rollback() {
    log_warn "Performing rollback..."

    if [[ -z "${BACKUP_DIR}" ]] || [[ ! -d "${BACKUP_DIR}" ]]; then
        log_error "No version to roll back to"
        return 1
    fi

    ln -sfn "${BACKUP_DIR}" "${CURRENT_LINK}"
    log_info "Rolled back to: ${BACKUP_DIR}"
}

# === Main Flow ===
main() {
    parse_args "$@"
    validate
    acquire_lock

    pre_deploy_check
    deploy
    health_check

    log_info "Deployment complete: ${APP_NAME} ${APP_VERSION}"
    release_lock
}

main "$@"

Summary

The “production-grade” quality of a Bash script isn’t about how complex its features are, but about whether it’s safe when things go wrong. A single set -e can prevent an rm -rf from executing in the wrong directory, a trap ensures temporary files get cleaned up, and a ShellCheck eliminates word-splitting and injection risks. Key takeaways from this article:

The three-line header is non-negotiable: set -euo pipefail + IFS=$'\n\t' is mandatory for every production script, no exceptions
Error handling is the core: trap ERR catches errors, run_cmd wrapper logs execution, retry mechanism handles transient failures — these three layers cover 90% of error scenarios
Signal handling ensures gracefulness: SIGTERM triggers graceful shutdown instead of a hard kill, allowing in-progress tasks to complete safely
Parallel execution must be controllable: xargs -P is the simplest parallel solution, wait -n is the Bash 4+ primitive — combining both gives you a bounded concurrency task pool
Argument parsing should be standardized: Use manual case parsing for long options, getopts for short options, and -- to separate options from positional arguments
Testing and checking are not optional: bats for unit testing, ShellCheck for static analysis — integrating both into CI keeps script quality under control
Secure coding is the red line: Path validation prevents traversal, input validation prevents injection, use mktemp for temporary files, and add confirmation for critical operations

Bash’s advantage is “it’s everywhere”; its disadvantage is “it’s easy to write insecure code.” Once you internalize these practices, your Bash scripts can be just as reliable as ops tools written in any other language — with zero deployment cost.

References & Acknowledgments

This article referenced the following materials during writing. We thank the original authors for their contributions:

Google Shell Style Guide — Google, referenced for Google Shell Style Guide
ShellCheck — Shellcheck, referenced for ShellCheck

Overview#

I. Script Skeleton: set -euo pipefail#

1.1 The Three-Line Header#

1.2 Complete Script Template#

II. Error Handling and Logging#

2.1 Error Handler Function#

2.2 Safe Execution Wrapper#

2.3 Logging System#

III. Signal Handling#

3.1 Catching Signals#

3.2 Graceful Shutdown Pattern#

IV. Parallel Execution#

4.1 Background Task Management#

4.2 Batch Parallel SSH Execution#

V. Configuration Management#

5.1 Config File Loading#

5.2 Environment Variable Override#

VI. Argument Parsing#

6.1 Short Options with getopts#

6.2 Long Option Parsing#

VII. Unit Testing with bats#

7.1 Installing bats#

7.2 Writing Tests#

7.3 Running Tests#

7.4 GitHub Actions Integration#

VIII. ShellCheck#

8.1 Installation and Usage#

8.2 Common Issues and Fixes#

8.3 Inline Disabling#

8.4 .shellcheckrc Configuration#

IX. Secure Coding#

9.1 Path Safety#

9.2 Input Validation#

9.3 Temporary File Safety#

X. Complete Production-Grade Script Example#

Summary#

References & Acknowledgments#