Stop Configuring Servers by Hand: A Complete Guide to Packer Golden Image Pipelines from Build to Security Scanning

Overview

2 AM, alert fires. Investigation reveals a server running an outdated OpenSSL version with known vulnerabilities. Security team asks: how many more machines are in the same state? You don’t have an answer.

That’s the cost of configuring servers manually. Each machine gets its own apt install, producing environments that drift apart. Someone forgot a patch, someone installed extra packages, someone changed a kernel parameter without documenting it. When things break, you realize—you have no idea how many “configuration drift” variants are running in production.

Packer solves exactly this problem. It turns “setting up a server” into code: you write a template, Packer launches a temporary instance, configures it according to the template, creates an image (AMI, VMDK, Docker image, qcow2, etc.), then destroys the temporary instance. Next time you need a new server? Just launch from the image—ready in seconds, configuration identical.

This isn’t a Hello World tutorial. I’m covering a production-grade golden image pipeline: from HCL2 template design, Ansible Provisioner integration, CI/CD auto-triggering, to Trivy security scanning, image version management, and canary rollout. Every step comes with real-world pitfalls.

What Is a “Golden Image”

Let’s clarify the concept first. A “Golden Image” is a standardized, security-hardened, tested machine template. Every server launched from it starts in the exact same state.

Think of it this way: configuring servers manually is like hand-folding dumplings—each one comes out slightly different. Using a golden image is like using a mold—every dumpling is identical in shape and size, quality-controlled.

Dimension	Manual Configuration	Golden Image
Init time	15-45 min	30-90 sec
Config consistency	Human-dependent, drifts	Image-locked, identical
Security baseline	Varies per machine	Unified at build time
Auditability	Hard to trace	Image version traceable
Rollback	Essentially none	Switch image version
Scale-out speed	Slow, SSH per machine	Fast, launch new instances

Netflix started using this pattern in 2013. They call it “Bake and Deploy”—bake code into an AMI first, then deploy with Spinnaker. A machine acting up? Destroy it, launch a new one from the same image. Much more reliable than SSH-ing in to debug.

Packer Core Concepts

Packer is an open-source tool by HashiCorp. It does one thing well: based on a template definition, it launches a temporary machine, runs configuration scripts, then creates an image. The machine is destroyed when done.

Three Core Components

A Packer template (HCL2 syntax) consists of three core parts:

1. Source: Tells Packer which base image to start from. For example, an Ubuntu AMI on AWS, or a local ISO file.

2. Provisioner: Operations executed on the temporary machine. Shell scripts, Ansible Playbooks, file uploads all work.

3. Post-Processors: Actions taken after the image is built. For example, pushing to a registry, tagging, triggering scans.

Source (base image) → Provisioner (configuration) → Post-Processors (post-processing)

Here’s the Packer workflow at a glance:

Step	Action	Time (reference)
1	Packer launches temporary instance	20-40s
2	SSH connect to instance	5-10s
3	Provisioner executes configuration	3-10min
4	Stop and snapshot	1-3min
5	Register as AMI/image	1-2min
6	Destroy temporary instance	10-20s
7	Post-Processor executes	30-60s

HCL2 vs JSON

Packer used JSON templates before version 1.7, then moved to HCL2 (HashiCorp Configuration Language). HCL2 supports variables, conditionals, and loops—much more flexible than JSON.

Stop using JSON templates. HashiCorp officially recommends HCL2. JSON templates are in maintenance mode; new features won’t be backported.

Project Structure Design

A production-grade Packer project shouldn’t be one giant file. Here’s the directory structure I recommend:

packer-golden-images/
├── variables.pkr.hcl          # Global variable definitions
├── sources.pkr.hcl            # Source definitions
├── build.pkr.hcl              # Build process definition
├── scripts/                   # Shell configuration scripts
│   ├── base-setup.sh          # Base system configuration
│   ├── security-hardening.sh  # Security hardening
│   └── cleanup.sh             # Clean up temp files
├── ansible/                   # Ansible Playbooks
│   ├── playbook.yml
│   ├── inventory.ini
│   └── roles/
│       ├── common/
│       ├── monitoring/
│       └── hardening/
├── policies/                  # Policy files
│   └── image-policy.json      # Image compliance policy
└── ci/
    └── build-image.sh         # CI build script

Why split into multiple files? Different roles need different parts. Security team only cares about security-hardening.sh, ops cares about ansible/roles/common/, platform team manages sources.pkr.hcl. Put it all in one file and every change requires reviewing the entire template.

Writing Packer Templates Step by Step

variables.pkr.hcl: Variable Definitions

# ====== Global Variables ======
variable "aws_region" {
  type    = string
  default = "ap-northeast-1"
  description = "AWS region for image building"
}

variable "source_ami_filter_name" {
  type    = string
  default = "ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-*"
  description = "Base AMI name filter, matches Ubuntu 22.04 LTS"
}

variable "instance_type" {
  type    = string
  default = "t3.medium"
  description = "Temporary instance type for building, t3.medium is sufficient"
}

variable "ssh_username" {
  type    = string
  default = "ubuntu"
}

variable "ami_name_prefix" {
  type    = string
  default = "golden-ubuntu-2204"
}

variable "app_version" {
  type    = string
  default = ""
  description = "App version, injected by CI, empty uses timestamp"
}

# Generate image name suffix using timestamp function
locals {
  ami_name = "${var.ami_name_prefix}-${var.app_version != "" ? var.app_version : formatdate("YYYYMMDDhhmmss", timestamp())}"
  common_tags = {
    Project   = "golden-images"
    ManagedBy = "packer"
    Os        = "ubuntu-22.04"
    CreatedAt = formatdate("YYYY-MM-DD", timestamp())
  }
}

Key points:

source_ami_filter_name uses wildcards instead of a fixed AMI ID. Fixed IDs break when Ubuntu updates the base image. Using a filter auto-selects the latest version—set and forget.
instance_type uses t3.medium not t2.micro. t2.micro has only 1GB RAM, frequently OOMs when running Ansible Playbooks. Don’t penny-pinch here—a build instance costs pennies.
locals block defines common tags. Tagging images makes it easy to see which ones were built by Packer in the AWS console.

sources.pkr.hcl: Source Definitions

# ====== AWS EBS Source ======
source "amazon-ebs" "ubuntu-base" {
  region   = var.aws_region
  ami_name = local.ami_name

  # Use filter to auto-select the latest Ubuntu 22.04 AMI
  source_ami_filter {
    filters = {
      virtualization-type = "hvm"
      name                = var.source_ami_filter_name
      root-device-type    = "ebs"
      architecture        = "x86_64"
    }
    most_recent = true
    owners      = ["099720109477"]  # Canonical official account
  }

  instance_type = var.instance_type
  ssh_username  = var.ssh_username

  # Temporary security group: only allow SSH from Packer build machine
  temporary_iam_instance_profile_policy_document {
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["ssm:GetParameters", "ssm:GetParameter"]
        Resource = "*"
      }
    ]
  }

  # EBS root volume configuration
  launch_block_device_mappings {
    device_name           = "/dev/sda1"
    volume_size           = 20
    volume_type           = "gp3"
    delete_on_termination = true
  }

  tags = merge(local.common_tags, {
    Name = "packer-builder-ubuntu"
    Purpose = "image-build"
  })
}

Here’s a trap many fall into. The owners field is mandatory. Anyone can publish public AMIs on AWS. If you don’t specify an owner, you might select a tampered AMI. Canonical’s official account is 099720109477, AWS official Linux is 137112412989.

Security reminder: Always specify the owners field. Selecting a malicious AMI means your image ships with a pre-installed backdoor.

build.pkr.hcl: Build Process

# ====== Build Process ======
build {
  sources = [
    "source.amazon-ebs.ubuntu-base"
  ]

  # ---------- Step 1: Shell base configuration ----------
  provisioner "shell" {
    script = "./scripts/base-setup.sh"
    execute_command = "sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
    environment_vars = [
      "DEBIAN_FRONTEND=noninteractive",
      "APP_VERSION=${var.app_version}"
    ]
  }

  # ---------- Step 2: Ansible configuration ----------
  provisioner "ansible" {
    playbook_file    = "./ansible/playbook.yml"
    inventory_directory = "./ansible/"
    user              = var.ssh_username
    use_proxy         = false
    ansible_env_vars = [
      "ANSIBLE_HOST_KEY_CHECKING=False",
      "ANSIBLE_PIPELINING=True",
      "ANSIBLE_PYTHON_INTERPRETER=/usr/bin/python3"
    ]
    extra_arguments = [
      "--extra-vars", "app_version=${var.app_version}",
      "--extra-vars", "image_build=true"
    ]
  }

  # ---------- Step 3: Security hardening script ----------
  provisioner "shell" {
    script = "./scripts/security-hardening.sh"
    execute_command = "sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
  }

  # ---------- Step 4: Cleanup ----------
  provisioner "shell" {
    script = "./scripts/cleanup.sh"
    execute_command = "sudo -S sh -c '{{ .Vars }} {{ .Path }}'"
  }

  # ---------- Post-processing: tagging ----------
  post-processors {
    post-processor "amazon-import" {
      only = ["amazon-ebs.ubuntu-base"]
    }
    post-processor "manifest" {
      output_file = "manifest.json"
      strip_path = true
      custom_data = {
        app_version = var.app_version
        build_date = formatdate("YYYY-MM-DD", timestamp())
      }
    }
  }
}

Note the Provisioner execution order. Shell does base configuration first (install packages, set timezone, tune kernel), then Ansible handles application-layer config (install monitoring agent, configure log collection), finally Shell does security hardening and cleanup.

Why not use Ansible for everything? Some operations are faster and simpler in Shell. For instance, apt update && apt upgrade is one line in Shell, but Ansible’s apt module makes you wait for it to parse the entire package tree.

Measured experience: Shell runs apt upgrade in about 2 minutes. Ansible’s apt module does the same in 4-5 minutes. The overhead comes from Ansible’s gather facts and module initialization.

Base Configuration Scripts

scripts/base-setup.sh

#!/bin/bash
set -euo pipefail

# ============================================================
# Base system configuration: package updates, timezone,
# kernel parameters, user management
# ============================================================

echo ">>> [1/5] Updating system packages"
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get upgrade -y -qq
apt-get install -y -qq \
  curl wget vim htop tree jq unzip \
  ca-certificates gnupg lsb-release \
  software-properties-common \
  python3 python3-pip \
  chrony

echo ">>> [2/5] Setting timezone"
timedatectl set-timezone Asia/Shanghai

echo ">>> [3/5] Configuring chrony time sync"
cat > /etc/chrony/chrony.conf << 'EOF'
# Use Alibaba Cloud NTP, more reliable in China
server ntp.aliyun.com iburst
server ntp1.aliyun.com iburst

# Allow large time step adjustments (new machines may have big drift)
makestep 1.0 3
rtcsync
logdir /var/log/chrony
EOF
systemctl enable chrony
systemctl restart chrony

echo ">>> [4/5] Kernel parameter tuning"
cat > /etc/sysctl.d/99-golden-image.conf << 'EOF'
# File descriptors
fs.file-max = 1048576
fs.nr_open = 1048576

# Network parameters
net.core.somaxconn = 4096
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.ip_local_port_range = 10000 65535

# Memory parameters
vm.swappiness = 10
vm.overcommit_memory = 1
vm.max_map_count = 262144

# BBR congestion control
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
EOF
sysctl -p /etc/sysctl.d/99-golden-image.conf

echo ">>> [5/5] Creating ops user"
if ! id -u ops >/dev/null 2>&1; then
  useradd -m -s /bin/bash -G sudo ops
  # Disable password login, key-only
  passwd -l ops
fi

echo ">>> Base configuration complete"

scripts/security-hardening.sh

#!/bin/bash
set -euo pipefail

# ============================================================
# Security hardening: SSH, firewall, audit, fail2ban
# ============================================================

echo ">>> [1/6] SSH hardening"
cp /etc/ssh/sshd_config /etc/ssh/sshd_config.bak
cat > /etc/ssh/sshd_config.d/99-hardening.conf << 'EOF'
# Disable root login
PermitRootLogin no

# Key-based authentication only
PasswordAuthentication no
PubkeyAuthentication yes

# No empty passwords
PermitEmptyPasswords no

# Limit max auth attempts
MaxAuthTries 3

# Login grace timeout 120 seconds
LoginGraceTime 120

# Disable X11 forwarding (servers don't need GUI)
X11Forwarding no

# Disable port forwarding
AllowTcpForwarding no

# Client alive check interval
ClientAliveInterval 300
ClientAliveCountMax 2

# Only allow ops user SSH
AllowUsers ops
EOF
systemctl restart sshd

echo ">>> [2/6] Configuring UFW firewall"
apt-get install -y -qq ufw
ufw --force reset
ufw default deny incoming
ufw default allow outgoing
ufw allow 22/tcp
ufw --force enable

echo ">>> [3/6] Installing fail2ban"
apt-get install -y -qq fail2ban
cat > /etc/fail2ban/jail.local << 'EOF'
[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 3
backend = systemd

[sshd]
enabled = true
port = 22
EOF
systemctl enable fail2ban
systemctl restart fail2ban

echo ">>> [4/6] Configuring audit logging"
apt-get install -y -qq auditd
cat > /etc/audit/rules.d/hardening.rules << 'EOF'
# Monitor critical file changes
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k sudo
-w /etc/ssh/sshd_config -p wa -k ssh

# Monitor system calls
-a always,exit -F arch=b64 -S chmod -S chown -S umask -k perm
EOF
systemctl enable auditd
systemctl restart auditd

echo ">>> [5/6] Disabling unnecessary services"
SERVICES_TO_DISABLE=(
  "cups"          # Printing
  "bluetooth"     # Bluetooth
  "avahi-daemon"  # mDNS discovery
  "speech-dispatcher"  # Speech
)
for svc in "${SERVICES_TO_DISABLE[@]}"; do
  if systemctl list-unit-files | grep -q "$svc"; then
    systemctl disable "$svc" 2>/dev/null || true
    systemctl mask "$svc" 2>/dev/null || true
  fi
done

echo ">>> [6/6] Automatic security updates"
cat > /etc/apt/apt.conf.d/50unattended-upgrades << 'EOF'
Unattended-Upgrade::Allowed-Origins {
  "${distro_id}:${distro_codename}-security";
};
Unattended-Upgrade::Automatic-Reboot "false";
Unattended-Upgrade::Automatic-Reboot-Time "04:00";
EOF
apt-get install -y -qq unattended-upgrades
dpkg-reconfigure -f noninteractive unattended-upgrades

echo ">>> Security hardening complete"

scripts/cleanup.sh

#!/bin/bash
set -euo pipefail

# ============================================================
# Cleanup: reduce image size, remove sensitive info
# ============================================================

echo ">>> [1/5] Cleaning apt cache"
apt-get autoremove -y
apt-get autoclean -y
apt-get clean
rm -rf /var/lib/apt/lists/*
rm -rf /var/cache/apt/archives/*.deb

echo ">>> [2/5] Cleaning temporary files"
rm -rf /tmp/*
rm -rf /var/tmp/*
rm -rf /var/log/*.gz
rm -rf /var/log/apt/*
rm -rf /root/.bash_history
rm -rf /home/*/.bash_history
rm -rf /var/lib/cloud/instances/*

echo ">>> [3/5] Removing SSH host keys"
# Image should not carry SSH host keys; auto-generate on boot
rm -f /etc/ssh/ssh_host_*
cat > /etc/rc.local << 'EOF'
#!/bin/bash
if [ ! -f /etc/ssh/ssh_host_rsa_key ]; then
  ssh-keygen -A
fi
systemctl restart sshd
EOF
chmod +x /etc/rc.local

echo ">>> [4/5] Clearing machine ID (systemd unique identifier)"
# Without this, all instances from this image share the same ID
echo "" > /etc/machine-id

echo ">>> [5/5] Zero-filling disk (reduces snapshot size)"
# Meaningful for EBS snapshots: zeroed blocks are skipped
dd if=/dev/zero of=/zerofile bs=1M 2>/dev/null || true
rm -f /zerofile

echo ">>> Cleanup complete"

Regarding removing SSH host keys in the cleanup script: this step is mandatory. Without it, all instances launched from this image share the same host keys, rendering SSH security meaningless. Removing them and using rc.local to auto-regenerate on first boot solves this.

Ansible Provisioner Integration

Packer has a built-in Ansible Provisioner that runs Ansible Playbooks directly on the temporary build machine.

ansible/playbook.yml

---
- name: Configure application-layer components for golden image
  hosts: all
  become: true
  gather_facts: true

  pre_tasks:
    - name: Verify Python3 is available
      raw: test -f /usr/bin/python3
      changed_when: false

    - name: Set hostname (temporary for build)
      hostname:
        name: "packer-build-{{ ansible_distribution_version | default('unknown') }}"

  roles:
    - role: common
      tags: [common]
    - role: monitoring
      tags: [monitoring]
      when: not image_build | default(false) | bool or true
    - role: hardening
      tags: [hardening]

  post_tasks:
    - name: Verify critical services installed
      command: "which {{ item }}"
      loop:
        - curl
        - jq
        - python3
        - chronyc
      changed_when: false

    - name: Verify Node Exporter port can listen
      wait_for:
        port: 9100
        state: started
        timeout: 30
      when: "'monitoring' in group_names or true"

ansible/roles/common/tasks/main.yml

---
- name: Create application directory structure
  file:
    path: "{{ item }}"
    state: directory
    owner: ops
    group: sudo
    mode: "0755"
  loop:
    - /opt/app
    - /opt/app/bin
    - /opt/app/conf
    - /opt/app/logs
    - /opt/app/data

- name: Install runtime dependencies
  apt:
    name:
      - libssl-dev
      - libcurl4-openssl-dev
      - libxml2-dev
      - zlib1g-dev
    state: present
    update_cache: true

- name: Configure log rotation
  template:
    src: app-logrotate.j2
    dest: /etc/logrotate.d/app
    mode: "0644"

- name: Configure file descriptor limits in PAM
  pam_limits:
    domain: ops
    limit_type: "-"
    limit_item: nofile
    value: "65536"

ansible/roles/monitoring/tasks/main.yml

---
- name: Download Node Exporter
  unarchive:
    src: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/node_exporter-{{ node_exporter_version }}.linux-amd64.tar.gz"
    dest: /tmp/
    remote_src: true
    creates: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/node_exporter"

- name: Install Node Exporter binary
  copy:
    src: "/tmp/node_exporter-{{ node_exporter_version }}.linux-amd64/node_exporter"
    dest: /usr/local/bin/node_exporter
    mode: "0755"
    remote_src: true

- name: Create systemd service
  template:
    src: node_exporter.service.j2
    dest: /etc/systemd/system/node_exporter.service
    mode: "0644"
  notify: enable node_exporter

- name: Ensure not started during build phase (only after image boot)
  systemd:
    name: node_exporter
    enabled: true
    state: stopped
  when: image_build | default(false) | bool

Key design here: state: stopped + enabled: true. Don’t start services during image build (the temporary machine gets destroyed after), but set them to auto-start on boot—so instances launched from the image automatically bring up services.

This was the deepest pit I hit. Initially I forgot state: stopped, and Node Exporter started during the build phase. Packer took the snapshot while the service was running. The resulting image had a confused systemd state after boot—service showed “running” but no actual process existed. Adding one line of state: stopped fixed it.

ansible/roles/monitoring/templates/node_exporter.service.j2

[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=ops
Group=sudo
Type=simple
ExecStart=/usr/local/bin/node_exporter \
  --web.listen-address=:9100 \
  --collector.systemd \
  --collector.processes \
  --no-collector.infiniband \
  --no-collector.mdadm
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Security Scanning: Trivy Integration

Building an image doesn’t mean it’s secure. You need to scan it for known vulnerabilities. I recommend Trivy—fast, low false-positive rate, free.

ci/build-and-scan.sh

#!/bin/bash
set -euo pipefail

# ============================================================
# CI build + scan script
# Usage: ./ci/build-and-scan.sh <app_version>
# ============================================================

APP_VERSION="${1:-}"
if [ -z "$APP_VERSION" ]; then
  APP_VERSION=$(date '+%Y%m%d%H%M%S')
  echo "No version specified, using timestamp: $APP_VERSION"
fi

echo "========== [1/5] Packer init =="
packer init .
echo "Packer plugins initialized"

echo "========== [2/5] Packer format check =="
packer fmt -check -diff .
if [ $? -ne 0 ]; then
  echo "Format check failed, run: packer fmt ."
  exit 1
fi
echo "HCL format check passed"

echo "========== [3/5] Packer build =="
PACKER_LOG=1 packer build \
  -var "app_version=${APP_VERSION}" \
  -var-file=variables.prod.pkr.hcl \
  . 2>&1 | tee build.log

# Extract AMI ID from manifest
AMI_ID=$(jq -r '.builds[0].artifact_id' manifest.json | head -1)
echo "Build complete, AMI: ${AMI_ID}"

echo "========== [4/5] Trivy vulnerability scan =="
# Scan AMI with Trivy (requires AWS permissions)
trivy server --listen 0.0.0.0:4954 &
TRIVY_PID=$!
sleep 3

# Scan via file system manifest (through AWS Systems Manager)
trivy fs --server http://localhost:4954 \
  --severity HIGH,CRITICAL \
  --ignore-unfixed \
  --format json \
  --output scan-result.json \
  --exit-code 1 \
  /tmp/scan-target/

SCAN_EXIT=$?
kill $TRIVY_PID 2>/dev/null || true

if [ $SCAN_EXIT -ne 0 ]; then
  echo "!!! Security scan found HIGH/CRITICAL vulnerabilities, build failed"
  echo "Scan results:"
  jq '.Results[] | select(.Vulnerabilities != null) | .Vulnerabilities[] | {ID: .VulnerabilityID, Pkg: .PkgName, Severity: .Severity, Fix: .FixedVersion}' scan-result.json
  exit 1
fi
echo "Security scan passed, no HIGH/CRITICAL vulnerabilities"

echo "========== [5/5] Update image version registry =="
# Record to version file for deployment reference
echo "{\"ami_id\": \"${AMI_ID}\", \"version\": \"${APP_VERSION}\", \"date\": \"$(date -Iseconds)\", \"scan\": \"passed\"}" >> docs/image-versions.jsonl
echo "Version registry updated"
echo ""
echo "Build successful!"
echo "  AMI ID: ${AMI_ID}"
echo "  Version: ${APP_VERSION}"
echo "  Scan: PASSED"

OPA Policy Validation

Scanning alone isn’t enough. You may need to ensure images meet specific compliance requirements—like “telnet must not be installed” or “fail2ban must be present.” Use OPA (Open Policy Agent) for policy checks:

# policies/image_policy.rego
package image.compliance

# Rule 1: fail2ban must be installed
deny[msg] {
  not input.installed_packages["fail2ban"]
  msg := "Compliance failed: fail2ban not installed"
}

# Rule 2: telnet must not be installed
deny[msg] {
  input.installed_packages["telnet"]
  msg := "Compliance failed: telnet detected, security risk"
}

# Rule 3: SSH must disable password auth
deny[msg] {
  input.ssh_config.password_authentication == "yes"
  msg := "Compliance failed: SSH still allows password login"
}

# Rule 4: Automatic security updates must be configured
deny[msg] {
  not input.unattended_upgrades_enabled
  msg := "Compliance failed: automatic security updates not configured"
}

CI/CD Integration

GitLab CI Configuration

# .gitlab-ci.yml
stages:
  - validate
  - build
  - scan
  - publish

variables:
  PACKER_VERSION: "1.11"
  TF_IN_AUTOMATION: "true"

validate_template:
  stage: validate
  image: hashicorp/packer:${PACKER_VERSION}
  script:
    - packer init .
    - packer fmt -check -diff .
    - packer validate -var "app_version=ci-test" .
  rules:
    - changes:
        - "*.pkr.hcl"
        - scripts/**/*
        - ansible/**/*

build_image:
  stage: build
  image: hashicorp/packer:${PACKER_VERSION}
  script:
    - apk add --no-cache jq
    - export APP_VERSION=${CI_COMMIT_SHORT_SHA}
    - packer build -var "app_version=${APP_VERSION}" .
    - AMI_ID=$(jq -r '.builds[0].artifact_id' manifest.json)
    - echo "AMI_ID=${AMI_ID}" > build.env
  artifacts:
    reports:
      dotenv: build.env
    paths:
      - manifest.json
      - scan-result.json
    expire_in: 30 days
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      changes:
        - "*.pkr.hcl"
        - scripts/**/*
        - ansible/**/*
  needs:
    - validate_template

scan_image:
  stage: scan
  image: aquasec/trivy:latest
  script:
    - trivy fs --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 --format json --output scan-result.json /tmp/
    - trivy fs --severity HIGH,CRITICAL --ignore-unfixed --format table .
  needs:
    - build_image
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

publish_image:
  stage: publish
  image: amazon/aws-cli:latest
  script:
    - aws ec2 create-tags --resources ${AMI_ID} --tags Key=Status,Value=approved Key=Version,Value=${CI_COMMIT_SHORT_SHA}
    - aws ssm put-parameter --name "/golden-images/latest" --value ${AMI_ID} --type String --overwrite
    - |
      # Keep latest 5 versions, tag old ones as deprecated
      aws ec2 describe-images --owners self --filters "Name=tag:Project,Values=golden-images" --query 'sort_by(Images,&CreationDate)[::-1]' > images.json
      DEPRECATED_COUNT=$(echo $(echo $(cat images.json | jq length)) - 5 | bc)
      if [ $DEPRECATED_COUNT -gt 0 ]; then
        for i in $(seq 0 $((DEPRECATED_COUNT - 1))); do
          OLD_AMI=$(jq -r ".[$((${i} + 5))].ImageId" images.json)
          aws ec2 create-tags --resources ${OLD_AMI} --tags Key=Status,Value=deprecated
          echo "Deprecated: ${OLD_AMI}"
        done
      fi      
  needs:
    - scan_image
  rules:
    - if: $CI_COMMIT_BRANCH == "main"

This CI config does several things:

validate: Auto-checks HCL format and syntax when templates change
build: Builds images when main branch templates change
scan: Auto-runs Trivy scan after build
publish: Updates the “latest image ID” in SSM Parameter Store, auto-tags old versions as deprecated

Key design: The latest image ID lives in AWS Systems Manager Parameter Store (/golden-images/latest), not hardcoded in deployment scripts. Terraform reads it dynamically via data "aws_ssm_parameter" "latest_ami". When the image updates, the next Terraform apply picks it up automatically—no code changes needed.

GitHub Actions Configuration

If you’re using GitHub Actions:

name: Build Golden Image

on:
  push:
    branches: [main]
    paths:
      - 'packer/**'
      - 'scripts/**'
      - 'ansible/**'
  workflow_dispatch:
    inputs:
      app_version:
        description: 'Image version tag'
        required: false

jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    permissions:
      id-token: write
      contents: read

    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS Credentials (OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-packer
          aws-region: ap-northeast-1

      - name: Setup Packer
        uses: hashicorp/packer-github-action@v4
        with:
          version: latest

      - name: Init and Validate
        working-directory: packer
        run: |
          packer init .
          packer fmt -check -diff .
          packer validate -var "app_version=test" .          

      - name: Build Image
        id: build
        working-directory: packer
        run: |
          VERSION="${{ github.event.inputs.app_version || github.sha }}"
          packer build -var "app_version=${VERSION}" .
          AMI_ID=$(jq -r '.builds[0].artifact_id' manifest.json)
          echo "ami_id=${AMI_ID}" >> $GITHUB_OUTPUT
          echo "version=${VERSION}" >> $GITHUB_OUTPUT          

      - name: Trivy Scan
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '/tmp/scan-target'
          severity: 'HIGH,CRITICAL'
          ignore-unfixed: true
          format: 'sarif'
          output: 'trivy-results.sarif'
          exit-code: '1'

      - name: Upload SARIF
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: trivy-results.sarif

      - name: Publish to SSM
        run: |
          aws ssm put-parameter \
            --name "/golden-images/latest" \
            --value "${{ steps.build.outputs.ami_id }}" \
            --type String --overwrite

Production Pitfalls Journal

Pitfall 1: SSH Timeout Causing Build Failure

Symptom: Packer fails mid-build with Error: timeout while waiting for SSH.

Cause: The temporary instance’s security group didn’t open the SSH port, or VPC Network ACLs blocked it.

Fix: Explicitly specify security_group_id in the source block, or use temporary_security_group_source_ip to specify the allowed source IP. If your build runs in a CI Runner with a dynamic IP, use temporary_security_group_source_ip = "0.0.0.0/0" with an extended ssh_timeout.

source "amazon-ebs" "ubuntu-base" {
  # ... other config ...
  ssh_timeout = "10m"  # Default 5 min, not enough on poor networks
  temporary_security_group_source_ip = "0.0.0.0/0"
}

Measured: For cross-region CI runners (e.g., GitHub Actions US runner building AP-region AMIs), SSH connection establishment can take 2-3 minutes. The default 5-minute timeout seems sufficient, but during AWS regional network jitter, 5 minutes isn’t always enough. Setting it to 10 minutes eliminated the problem entirely.

Pitfall 2: Ansible Playbook Can’t Find Python

Symptom: Packer fails at the Ansible Provisioner stage with module python3 not found.

Cause: Ubuntu 22.04 installs Python at /usr/bin/python3, but Ansible’s gather_facts looks for /usr/bin/python by default. Without a symlink, it errors out.

Fix: Add a line in the Shell Provisioner to install python3 and create the symlink. Or use ansible_env_vars to specify the interpreter path (I’ve already added ANSIBLE_PYTHON_INTERPRETER=/usr/bin/python3 in the build.pkr.hcl above).

Pitfall 3: Image Size Bloat

Symptom: Built AMI snapshot is 15GB, but the actual system only uses 6GB.

Cause: apt update downloaded package caches weren’t cleaned, log files accumulated, temporary files remained.

Fix: That’s what cleanup.sh is for. Key operations:

Cleanup Item	Space Saved	Notes
apt cache	1-3 GB	`/var/cache/apt/archives/`
Log files	200-500 MB	`/var/log/*.gz` etc.
Temp files	100-200 MB	`/tmp/` `/var/tmp/`
cloud-init residue	50-100 MB	`/var/lib/cloud/instances/`
Zero-fill disk	0 (smaller after compression)	dd if=/dev/zero then rm

Zero-filling looks odd but matters. EBS snapshots are incremental—deleted files still exist at the snapshot level (just marked as overwritable). dd if=/dev/zero fills unused blocks with zeros, allowing the snapshot to recognize these blocks as not needing storage, resulting in smaller compressed size.

Pitfall 4: Multi-Region Image Replication

Symptom: AMI built in ap-northeast-1 can’t be used in us-east-1.

Cause: AMIs are regional resources; they aren’t automatically available across regions.

Fix: Packer supports multi-region builds, but the more common approach is to use a post-processor for replication:

build {
  sources = ["source.amazon-ebs.ubuntu-base"]

  # ... provisioners ...

  post-processor "amazon-ami-copy" {
    regions = ["us-east-1", "eu-west-1", "ap-southeast-1"]
    ami_name = local.ami_name
    tags = merge(local.common_tags, {
      CopiedFrom = var.aws_region
    })
  }
}

Note: Cross-region replication incurs extra costs. AWS charges cross-region data transfer per GB. A 10GB AMI replicated to 3 regions costs about 3 × 10GB × $0.02/GB = $0.6 per build. With high build frequency, monthly costs add up. I recommend only replicating to all regions when releasing to production; test images stay in the build region.

Pitfall 5: Packer Build Residual Temporary Resources

Symptom: AWS bill shows unknown EC2 instances and security groups, all prefixed with packer-.

Cause: Packer build was interrupted (CI timeout, script error), leaving temporary instances and security groups uncleaned.

Fix: Two approaches:

Option 1: Packer has a built-in cleanup mechanism. On build failure, it attempts to destroy temporary instances. But when CI timeout kills the Packer process, nobody cleans up.

Option 2: Write a cleanup script, run it periodically:

#!/bin/bash
# cleanup-packer-residuals.sh
# Clean up Packer residual resources created 2+ hours ago

REGIONS=("ap-northeast-1" "us-east-1")

for REGION in "${REGIONS[@]}"; do
  echo ">>> Cleaning region: ${REGION}"

  # Clean up residual EC2 instances
  INSTANCES=$(aws ec2 describe-instances \
    --region "$REGION" \
    --filters "Name=tag:ManagedBy,Values=packer" "Name=instance-state-name,Values=running,pending" \
    --query 'Reservations[].Instances[].InstanceId' \
    --output text)

  for INSTANCE_ID in $INSTANCES; do
    LAUNCH_TIME=$(aws ec2 describe-instances \
      --region "$REGION" \
      --instance-ids "$INSTANCE_ID" \
      --query 'Reservations[0].Instances[0].LaunchTime' \
      --output text)

    # Only clean up instances launched 2+ hours ago
    if [ -n "$LAUNCH_TIME" ]; then
      AGE_HOURS=$(echo $(($(date +%s) - $(date -d "$LAUNCH_TIME" +%s))) / 3600)
      if [ "$AGE_HOURS" -ge 2 ]; then
        echo "  Terminating residual instance: ${INSTANCE_ID} (running ${AGE_HOURS}h)"
        aws ec2 terminate-instances --region "$REGION" --instance-ids "$INSTANCE_ID"
      fi
    fi
  done

  # Clean up residual security groups
  SGs=$(aws ec2 describe-security-groups \
    --region "$REGION" \
    --filters "Name=group-name,Values=packer-*" \
    --query 'SecurityGroups[].GroupId' \
    --output text 2>/dev/null || true)

  for SG_ID in $SGs; do
    echo "  Deleting residual security group: ${SG_ID}"
    aws ec2 delete-security-group --region "$REGION" --group-id "$SG_ID" 2>/dev/null || true
  done
done

Add this to cron, run daily. Saves you from discovering ghost instances when reconciling the monthly bill.

Image Lifecycle Management

Building an image is just the first step. You need to manage its entire lifecycle:

Build → Test → Approve → Canary → Full rollout → Retire old version

Version Management Strategy

Stage	Image Tag	Who Can Use	Automation Level
Just built	`Status=pending`	Nobody	CI auto
Scan passed	`Status=scanned`	Test env	CI auto
Manual approval	`Status=approved`	Staging	Manual trigger
Canary	`Status=canary`	10% production	Ops trigger
Full release	`Status=production`	100% production	Ops trigger
Retired	`Status=deprecated`	None (pending deletion)	Auto after 30 days

The core philosophy of this tagging system: images don’t go straight to production after building. They pass through scan, approval, and canary gates. CI handles the first two (build + scan), while canary and full release are triggered manually by ops. Full automation isn’t always good—with a human in the loop, at least someone catches issues before they hit production.

HCP Packer vs Self-Hosted

HashiCorp Cloud Platform (HCP) Packer offers a managed image version management service that tracks “which image version is running on which machine.” Convenient for small teams.

But if you don’t want SaaS, self-hosted approach:

# Write to version registry after build
post-processor "manifest" {
  output_file = "manifest.json"
  strip_path = true
  custom_data = {
    version      = var.app_version
    build_date   = formatdate("YYYY-MM-DD'T'HH:mm:ssZ", timestamp())
    git_commit   = var.git_commit_sha
    scan_status  = "pending"
    release_stage = "pending"
  }
}

Then write a simple script to push manifest.json to S3 or a database. Query the version registry when needed: “Which image version is production running?"—clear answer.

Terraform Integration

Image built—now what? Terraform reads the latest image dynamically during deployment:

# main.tf
data "aws_ssm_parameter" "latest_golden_ami" {
  name = "/golden-images/latest"
}

# Production ASG uses latest golden image
resource "aws_launch_template" "app" {
  name          = "app-launch-template"
  image_id      = data.aws_ssm_parameter.latest_golden_ami.value
  instance_type = "t3.large"
  key_name      = "ops-key"

  user_data = base64encode(<<-EOF
    #!/bin/bash
    # Image already has all base components installed
    # user_data only handles runtime configuration
    echo "Instance started with golden image: $(cat /etc/golden-image-version)"
    systemctl start node_exporter
  EOF
  )

  monitoring {
    enabled = true
  }

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name    = "app-server"
      Managed = "terraform"
      Image   = "golden"
    }
  }
}

resource "aws_autoscaling_group" "app" {
  name                = "app-asg"
  vpc_zone_identifier = module.vpc.private_subnets
  target_group_arns   = [aws_lb_target_group.app.arn]

  launch_template {
    id      = aws_launch_template.app.id
    version = "$Latest"
  }

  min_size         = 3
  max_size         = 10
  desired_capacity = 3

  # Rolling update strategy
  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 50
      instance_warmup        = 120
    }
    triggers = ["tag"]
  }
}

instance_refresh is key. When the Launch Template updates (because the image ID changed), the ASG automatically rolls out replacement of old instances. Each batch replaces a portion, ensuring 50% healthy instances. This is how immutable infrastructure deploys—not SSH-ing in to apt upgrade, but swapping the entire image.

Real-world result: After an image update, ASG rolling replacement of 10 instances takes about 15-20 minutes. Each new instance is ready within 90 seconds (because the image has everything pre-installed). Compared to 40 minutes of manual SSH apt upgrade per machine, both speed and reliability are dramatically better.

Cost Optimization

Packer builds aren’t free. Here’s where to save:

Optimization	Est. Savings	Approach
Downsize instance type	40-60%	Use `c5.large` for CPU-intensive builds, `t3.medium` for I/O; don’t use `m5.large`
Build frequency	30-50%	Don’t build on every commit. Only trigger on main branch merge
EBS snapshot cleanup	20-30%	Periodically delete deprecated image snapshots
Spot instances	60-80%	Use Spot Runners for build jobs; rerun if interrupted
Cache Ansible roles	15-20%	Use `ansible-galaxy install` cache directory

# Periodic old image cleanup script
#!/bin/bash
# Clean up AMIs marked deprecated 30+ days ago
DEPRECATED_AMIS=$(aws ec2 describe-images \
  --owners self \
  --filters "Name=tag:Status,Values=deprecated" \
  --query "Images[?CreationDate<='$(date -d '30 days ago' -Iseconds)'].ImageId" \
  --output text)

for AMI_ID in $DEPRECATED_AMIS; do
  # Deregister snapshots before AMI
  SNAPSHOTS=$(aws ec2 describe-images \
    --image-ids "$AMI_ID" \
    --query 'BlockDeviceMappings[?Ebs.SnapshotId].Ebs.SnapshotId' \
    --output text)

  aws ec2 deregister-image --image-id "$AMI_ID"
  for SNAP_ID in $SNAPSHOTS; do
    aws ec2 delete-snapshot --snapshot-id "$SNAP_ID"
  done
  echo "Cleaned up: ${AMI_ID}"
done

Summary

Packer isn’t a new concept—Netflix has used it for over a decade. But in the domestic ops community, the habit of manually configuring servers remains widespread. The reason is simple: learning curve. Writing HCL templates, configuring Ansible Provisioners, setting up CI/CD integration—all of this is more work than apt install.

But long-term, it doesn’t pay off. The “convenience” of manual configuration is false economy: you save template-writing time, spend it on troubleshooting; you save automation build costs, pay the price of configuration drift. The core value of the golden image pattern isn’t “fast”—it’s “consistent.” With 100 servers in production, you can be certain their environments are completely identical because they all come from the same image.

Implementation advice: don’t attempt a full migration all at once. Start with one service: write the Packer template, get CI building, verify in test environment. Once it’s stable, gradually expand. I’ve seen too many teams try to do everything at once, only to have the template abandoned halfway—too many Provisioners, unclear role boundaries, CI config complexity explosion. Small steps, get one service’s image pipeline running first, then replicate horizontally.

References & Acknowledgments

This article referenced the following materials during writing. Thanks to the original authors for their contributions:

Build Images in CI/CD | Packer — HashiCorp, official guide for building images in CI/CD with Packer, covering integration of Packer Builders with different CI platforms
Packer HCL2 Configuration Support — Volcengine, reference examples and source block configuration for Packer HCL2 syntax
How to Build Immutable Infrastructure with Packer and CircleCI Workflows — CircleCI, practical case of building immutable infrastructure with Packer in CI
Using Packer and Ansible to Build Immutable Infrastructure in CodeShip — CodeShip, engineering practices for integrating Packer with Ansible Provisioner
Operations Automation in Practice: From Manual Struggle to Elegant Control — CSDN, comparison of comprehensive ops automation tool stacks, including Terraform, Ansible, and CI/CD division of responsibilities
Linux Automated Deployment Pipeline: How to Integrate with Ansible — php.cn, explanation of Ansible’s role as the “environment execution layer” in automated deployment pipelines

Overview#

What Is a “Golden Image”#

Packer Core Concepts#

Three Core Components#

HCL2 vs JSON#

Project Structure Design#

Writing Packer Templates Step by Step#

variables.pkr.hcl: Variable Definitions#

sources.pkr.hcl: Source Definitions#

build.pkr.hcl: Build Process#

Base Configuration Scripts#

scripts/base-setup.sh#

scripts/security-hardening.sh#

scripts/cleanup.sh#

Ansible Provisioner Integration#

ansible/playbook.yml#

ansible/roles/common/tasks/main.yml#

ansible/roles/monitoring/tasks/main.yml#

ansible/roles/monitoring/templates/node_exporter.service.j2#

Security Scanning: Trivy Integration#

ci/build-and-scan.sh#

OPA Policy Validation#

CI/CD Integration#

GitLab CI Configuration#

GitHub Actions Configuration#

Production Pitfalls Journal#

Pitfall 1: SSH Timeout Causing Build Failure#

Pitfall 2: Ansible Playbook Can’t Find Python#

Pitfall 3: Image Size Bloat#

Pitfall 4: Multi-Region Image Replication#

Pitfall 5: Packer Build Residual Temporary Resources#

Image Lifecycle Management#

Version Management Strategy#

HCP Packer vs Self-Hosted#

Terraform Integration#

Cost Optimization#

Summary#

References & Acknowledgments#