Pushing the Same Image 3 Times to 3 Environments? CI/CD Artifact Promotion Pipeline Layered Design and 4 Production Decisions

Overview

You’ve probably seen this release workflow: a developer builds a Docker image, tags it as dev, and pushes it to the dev registry. After testing passes, they re-tag it as staging and push it again. Finally, for production, they tag it as prod and push a third time. Same image, built once, pushed three times with three different tags.

This seems fine — until one day you discover that the image validated in staging and the image running in prod are not the same artifact at all.

Artifact Promotion solves exactly this problem. The core idea is straightforward: build the image once, identify it by digest (content hash), and when promoting across environments, copy the same immutable artifact rather than re-tagging it. This sounds like common sense, but after implementing it across multiple teams, I found that fewer than 30% actually get this pipeline working end-to-end. The reason isn’t technical complexity — it’s that the pipeline involves registry architecture, permission models, security gates, and rollback strategies, each with its own pitfalls.

This article breaks down the layered design of an artifact promotion pipeline, drawing from the promotion pipeline I built during a 120+ microservice K8s migration at a new-energy logistics platform. We’ll cover 4 key production decisions and the pitfalls I hit along the way.

What Artifact Promotion Actually Is

Let’s clarify the concept first. Many teams conflate “artifact promotion” with “re-tagging,” which is the first misconception.

The re-tagging approach:

# Build once
docker build -t registry.company.com/app:dev-1.0.0 .
docker push registry.company.com/app:dev-1.0.0

# "Promote" to staging — re-tag and push
docker tag registry.company.com/app:dev-1.0.0 registry.company.com/app:staging-1.0.0
docker push registry.company.com/app:staging-1.0.0

# "Promote" to prod — tag and push again
docker tag registry.company.com/app:staging-1.0.0 registry.company.com/app:prod-1.0.0
docker push registry.company.com/app:prod-1.0.0

What’s wrong with this? On the surface, all three pushes reference the same image ID. But docker tag only creates a new label reference in the registry. Tags are mutable — someone can overwrite staging-1.0.0 to point to a different image without your knowledge. The image you validated in staging and the image prod pulls may look the same by tag, but they could be two different artifacts.

The artifact promotion approach:

# Build once, push only to dev registry
docker build -t registry-dev.company.com/app:1.0.0 .
docker push registry-dev.company.com/app:1.0.0

# Get the digest (content hash of the image, immutable)
DIGEST=$(skopeo inspect docker://registry-dev.company.com/app:1.0.0 --format '{{.Digest}}')
# Output example: sha256:a1b2c3d4e5f6...

# Promote to staging — copy by digest, no rebuild
skopeo copy docker://registry-dev.company.com/app:1.0.0 \
  docker://registry-staging.company.com/app:1.0.0

# Promote to prod — same copy by digest
skopeo copy docker://registry-staging.company.com/app:1.0.0 \
  docker://registry-prod.company.com/app:1.0.0

The key difference: promotion is not “re-tag and push again,” but copying the same immutable artifact across registries. The image’s digest (SHA256 content hash) never changes throughout its lifecycle. What dev validates is what prod runs.

There’s a concept worth explaining here. A Docker image has two identifiers: tag and digest. A tag is a human-readable version label, like 1.0.0 — it’s mutable; the same tag can be re-pointed to different images. A digest is the SHA256 content hash of the image, like sha256:a1b2c3d4... — it’s immutable; if the image content changes, the digest changes. Locking images by digest in production eliminates the risk of “what you tested isn’t what’s running” caused by tag overwrites.

Layered Architecture: Three Registries + Four Gates

When implementing artifact promotion at the new-energy logistics platform, we designed a three-tier registry + four-gate architecture. All 120+ microservice images flow through this pipeline.

Three-Tier Registry Isolation

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  Dev Registry   │────▶│  Staging Registry │────▶│  Prod Registry  │
│                 │     │                   │     │                 │
│ • Dev writable   │     │ • CI writable     │     │ • Promotion only │
│ • All readable   │     │ • QA readable      │     │ • Prod readable  │
│ • No scanning    │     │ • Must pass scan   │     │ • Must verify sig│
│ • 7-day retention│     │ • 30-day retention │     │ • Permanent      │
└─────────────────┘     └──────────────────┘     └─────────────────┘

The three registries are physically isolated — not three projects in one Harbor instance, but three independent Harbor instances deployed in separate network zones. I’ll explain why later.

Four Quality Gates

Each gate corresponds to a pipeline Stage. If any Stage fails, the entire promotion pipeline halts. This is fundamentally different from the traditional “build and push straight to prod” — where security scanning is often done post-deployment, meaning a vulnerable image is already running in production by the time you catch it.

Decision 1: Promote by Digest or by Tag

This is the most fundamental decision in the entire design, and also the most overlooked.

The Pitfall of Tag Promotion

I took over a team’s pipeline refactoring where they used tag promotion: dev-1.0.0 → staging-1.0.0 → prod-1.0.0. One night, production blew up at 3 AM, and they executed a rollback:

# Rollback to previous version
kubectl set image deployment/api-gateway \
  api-gateway=registry.company.com/app:prod-0.9.9

The image pulled, the container started, but the behavior was wrong. Investigation revealed that prod-0.9.9 had been “conveniently” overwritten after the previous release — it no longer pointed to the stable version. Tags are mutable; docker tag can re-point a tag to any image at any time, with no audit trail and no alert.

This is the fundamental problem with tag promotion: tags are mutable references that don’t guarantee content consistency.

The Correct Approach: Digest Promotion

A digest is the SHA256 hash of the image content. The same digest always corresponds to the same image content — it cannot be overwritten. If the content changes, the digest changes. Promoting by digest means you’re promoting “a specific, determined image content,” not “a tag reference.”

# Get image digest
DIGEST=$(skopeo inspect \
  --tls-verify=false \
  docker://registry-dev.company.com/app:1.0.0 \
  --format '{{.Digest}}')

echo "Promoting image digest: $DIGEST"
# sha256:a1b2c3d4e5f67890...

# Copy to staging by digest
skopeo copy \
  --src-tls-verify=false \
  --dest-tls-verify=false \
  docker://registry-dev.company.com/app@${DIGEST} \
  docker://registry-staging.company.com/app:1.0.0

# Record digest in audit log
echo "$(date -Iseconds) | app:1.0.0 | $DIGEST | dev→staging" >> /data/promotion-log.csv

Note that skopeo copy uses app@${DIGEST} as the source, not app:1.0.0. Even if the dev registry’s 1.0.0 tag gets overwritten later, you’re still copying the exact content from when you fetched the digest.

My Recommendation

Use digest for all environment promotions; tags serve only as human-readable辅助 identifiers. In production deployment manifests, use image: registry-prod.company.com/app@sha256:a1b2c3d4... instead of image: registry-prod.company.com/app:1.0.0.

Some people find digest-based deployments uninformative — you can’t see the version number at a glance. My approach: include both tag and digest in the deployment manifest. K8s pulls by digest; the tag exists only for readability:

# K8s deployment manifest — tag for humans, digest for correctness
spec:
  containers:
    - name: api-gateway
      image: registry-prod.company.com/app:1.0.0@sha256:a1b2c3d4e5f6...

This syntax is supported by both containerd and Docker 1.13+. The tag exists purely for readability; the actual pull resolves to the digest’s image layers.

Decision 2: Skopeo or Docker CLI for Cross-Registry Copy

Cross-registry image copying is the core operation of promotion. Two approaches: the docker pull + docker tag + docker push three-step dance, or skopeo copy in one shot.

Problems with Docker CLI

The traditional three-step chain looks like this:

Remote Registry → Local Docker Daemon → Local Disk Cache → Local Docker Daemon → Target Registry

Problems abound:

1. Must pull the entire image locally first. A 500MB image takes 500MB of disk on pull and another 500MB on push. CI machines can’t handle batch promotions.
2. Depends on Docker Daemon. CI Runners need Docker-in-Docker or host socket mounting — significant security risk.
3. Multi-arch images are awkward. Manifest lists need extra handling; the three-step approach can’t deal with them properly.
4. Slow. Pull + tag + push means three sets of network round-trips and disk I/O.

Skopeo’s Advantages

Skopeo is a container image operations tool whose core capability is copying images directly between registries — no Docker Daemon, no local disk.

Source Registry ──────────────────────────────▶ Target Registry
              (Skopeo copies directly between registries)

# Direct cross-registry copy, no local disk involved
skopeo copy \
  --src-creds "dev-bot:${DEV_TOKEN}" \
  --dest-creds "staging-bot:${STAGING_TOKEN}" \
  --src-tls-verify=false \
  --dest-tls-verify=false \
  docker://registry-dev.company.com/app:1.0.0 \
  docker://registry-staging.company.com/app:1.0.0

Key advantages:

Where does the 90s vs 30s gap come from? Docker’s three-step approach requires two complete network transfers (one pull, one push) plus local disk read/write. Skopeo does a single streamed transfer from source to target. When batch-promoting 20+ microservice images, the gap widens to minutes.

Pitfall: Skopeo’s TLS Certificates

Skopeo verifies TLS certificates by default. If your internal Harbor uses self-signed certificates (very common), Skopeo will refuse to connect. Two options:

# Option 1: Skip TLS verification (not recommended for production)
skopeo copy --src-tls-verify=false --dest-tls-verify=false ...

# Option 2: Add the self-signed CA to the trust chain (recommended)
# Copy Harbor's CA certificate to system trust directory
cp harbor-ca.crt /etc/ssl/certs/harbor-ca.crt
update-ca-certificates
# Skopeo now verifies successfully without --tls-verify=false

I strongly push for Option 2 in production. Skipping TLS verification means the promotion chain is vulnerable to man-in-the-middle attacks — an image could be swapped during copy without your knowledge. Internal networks shouldn’t be exempt.

Decision 3: Physically Isolate Three Harbors or Logically Isolate Three Projects

Implementing three-tier registries has two approaches: one Harbor instance with three Projects, or three independent Harbor instances.

Logical Isolation: One Harbor, Three Projects

Harbor (single instance)
├── dev/app          ← Development project
├── staging/app      ← Staging project
└── prod/app         ← Production project

Simple, resource-efficient — one Harbor instance does it all. But the problem is: all three Projects share the same Harbor process, the same database, the same authentication system. If the Harbor instance itself has issues (database crash, disk full), all three environments go down simultaneously. Worse, the security boundary is blurred — if someone’s Harbor admin credentials leak, they can access all dev and prod images at once.

Physical Isolation: Three Harbor Instances

Harbor-Dev (dev network zone)      Harbor-Staging (staging network zone)     Harbor-Prod (prod network zone)
├── Independent database            ├── Independent database                   ├── Independent database
├── Independent storage backend      ├── Independent storage backend           ├── Independent storage backend
├── Independent auth                ├── Independent auth                       ├── Independent auth
└── Robot account dev-bot           └── Robot account staging-bot             └── Robot account prod-bot

Three Harbor instances deployed in three network zones, copying images between them via Skopeo. Security boundaries are clear: dev CI Runners can’t directly access the prod registry; the prod registry only accepts promotion copies from staging.

My Recommendation: Choose by Scale

We chose physical isolation for 120+ microservices. Initially, we tried logical isolation — three Projects in one Harbor with RBAC. After two months, two issues emerged:

1. Garbage collection strategy was hard to configure. Dev project retains 7 days, prod retains permanently, but one Harbor instance’s GC is global — you can’t configure GC frequency per Project. Dev’s garbage images piled up, filled the disk, and prod image pushes started failing too.

2. Audit logs were mixed together. All three Projects’ operation logs were in one audit table. Troubleshooting promotion chain issues required filtering through massive logs — very inefficient.

After physical isolation, each Harbor instance has independent storage and GC, separate audit logs, and troubleshooting is much faster.

For Harbor-specific configuration and management, I previously wrote Related: Pushed an Image and Forgot About It? Harbor Artifact Repository: Permission Layering, Vulnerability Scanning, and 200GB Storage Governance — check it out for registry-level governance.

Decision 4: How to Design Security Gates That Don’t Over-Block

Adding security scanning gates to the promotion pipeline is standard practice, but many teams fall into the trap: scan gates have such high false positive rates that teams either let vulnerabilities through to production, or block so aggressively that people bypass the gates entirely.

Four-Layer Security Gate Design

We designed a four-layer security gate. The core principle is layered interception, layered de-escalation:

Artifact Build → [L1: Base Image Scan] → [L2: App Layer Scan] → [L3: License Check] → [L4: Signature Verify] → Promotion Complete

Pitfall: Trivy CVE False Positives

Trivy scans all known CVEs by default, but many CVEs don’t actually affect your image. For example, a Python image might report 50 CVEs, 30 of which are C library vulnerabilities that the Python app never touches. If you block all of them, the promotion pipeline goes red every day, and developers come to ops asking for “exceptions.”

Our approach: introduce CVE exception rules (natively supported by Harbor). For confirmed non-impacting CVEs, mark them in Harbor’s CVE exception list with a reason and approver. Exceptions have expiration — default 30 days. If the CVE is still present after expiration, re-evaluate.

# Trivy scan script, distinguishing block vs alert
trivy image --severity CRITICAL --exit-code 1 \
  registry-staging.company.com/app:1.0.0

# Exit code 1 = Critical CVE found, block promotion
# Exit code 0 = No Critical, allow promotion

But Trivy’s --exit-code only does a binary block/pass. We needed finer-grained policy — some Critical CVEs don’t pose actual threats under specific conditions. We added an OPA policy engine between Trivy and the promotion pipeline:

# OPA policy: allow specific CVEs to be waived under conditions
package image.promotion

default allow = false

allow {
  count(blocked_cves) == 0
}

blocked_cves[cve] {
  cve := input.scan_results.vulnerabilities[_]
  cve.severity == "CRITICAL"
  not cve_ignored(cve.id)
}

# Approved CVE waiver list
cve_ignored(id) {
  id == "CVE-2024-1234"  # Reason: Python doesn't use affected library
}

For the complete CI/CD security gate design, including the process of reducing false positive rates from 70% to 5%, see Related: False Positive Rate from 70% to 5%: A Four-Layer Security Gate Design and False Positive Management in CI/CD.

Signature Verification: The Last Gate

Signature verification is the final layer. Images are signed with cosign after build, and verified before promotion to the prod registry. If an image is tampered with during the promotion pipeline (even a single bit change in digest), signature verification fails immediately.

# Sign after build
cosign sign --key cosign.key \
  registry-dev.company.com/app:1.0.0

# Verify signature before prod promotion
cosign verify --key cosign.pub \
  registry-staging.company.com/app:1.0.0

The signing private key is only available in the build pipeline; the promotion pipeline only has the public key. Even if the promotion pipeline is compromised, attackers can’t forge signed images.

Complete Promotion Pipeline Implementation

Combining all the decisions above, the complete artifact promotion pipeline looks like this:

#!/bin/bash
# promote.sh — Artifact promotion script
# Usage: ./promote.sh <image-name> <tag> <from-env> <to-env>

set -euo pipefail

IMAGE_NAME="${1}"
TAG="${2}"
FROM_ENV="${3}"  # dev | staging
TO_ENV="${4}"    # staging | prod

REGISTRY_DEV="registry-dev.company.com"
REGISTRY_STAGING="registry-staging.company.com"
REGISTRY_PROD="registry-prod.company.com"

# Select source and target registries by environment
case "${FROM_ENV}" in
  dev)     SRC="${REGISTRY_DEV}" ;;
  staging) SRC="${REGISTRY_STAGING}" ;;
  *) echo "Unsupported source env: ${FROM_ENV}"; exit 1 ;;
esac

case "${TO_ENV}" in
  staging) DST="${REGISTRY_STAGING}" ;;
  prod)    DST="${REGISTRY_PROD}" ;;
  *) echo "Unsupported target env: ${TO_ENV}"; exit 1 ;;
esac

# Credentials from environment variables, not hardcoded
SRC_USER="promotion-bot"
SRC_PASS_VAR="${FROM_ENV^^}_REGISTRY_TOKEN"  # DEV_REGISTRY_TOKEN or STAGING_REGISTRY_TOKEN
SRC_PASS="${!SRC_PASS_VAR}"

DST_USER="promotion-bot"
DST_PASS_VAR="${TO_ENV^^}_REGISTRY_TOKEN"
DST_PASS="${!DST_PASS_VAR}"

echo "=========================================="
echo "Artifact promotion: ${IMAGE_NAME}:${TAG}"
echo "  ${FROM_ENV} → ${TO_ENV}"
echo "=========================================="

# Step 1: Get source image digest
echo "[1/5] Getting source image digest..."
DIGEST=$(skopeo inspect \
  --creds "${SRC_USER}:${SRC_PASS}" \
  docker://${SRC}/${IMAGE_NAME}:${TAG} \
  --format '{{.Digest}}')
echo "  digest: ${DIGEST}"

# Step 2: Security scan (only for dev→staging)
if [[ "${TO_ENV}" == "staging" ]]; then
  echo "[2/5] Running security scan..."
  trivy image --severity CRITICAL --exit-code 1 \
    --ignore-unfixed \
    docker://${SRC}/${IMAGE_NAME}@${DIGEST}
  if [[ $? -ne 0 ]]; then
    echo "  Security scan failed, blocking promotion"
    exit 1
  fi
  echo "  Security scan passed"
else
  echo "[2/5] Skipping security scan (staging→prod already scanned in staging)"
fi

# Step 3: Signature verification (only for staging→prod)
if [[ "${TO_ENV}" == "prod" ]]; then
  echo "[3/5] Verifying image signature..."
  cosign verify --key /etc/cosign/cosign.pub \
    ${SRC}/${IMAGE_NAME}:${TAG}
  if [[ $? -ne 0 ]]; then
    echo "  Signature verification failed, blocking promotion"
    exit 1
  fi
  echo "  Signature verification passed"
else
  echo "[3/5] Skipping signature verification (dev→staging stage)"
fi

# Step 4: Cross-registry copy by digest
echo "[4/5] Executing cross-registry copy..."
skopeo copy \
  --src-creds "${SRC_USER}:${SRC_PASS}" \
  --dest-creds "${DST_USER}:${DST_PASS}" \
  docker://${SRC}/${IMAGE_NAME}@${DIGEST} \
  docker://${DST}/${IMAGE_NAME}:${TAG}
echo "  Copy complete"

# Step 5: Record audit log
echo "[5/5] Recording audit log..."
echo "$(date -Iseconds)|${IMAGE_NAME}:${TAG}|${DIGEST}|${FROM_ENV}→${TO_ENV}|$(whoami)" \
  >> /data/promotion-audit.log
echo "  Audit record written"

echo "=========================================="
echo "Promotion complete: ${DST}/${IMAGE_NAME}:${TAG}"
echo "  digest: ${DIGEST}"
echo "=========================================="

Key design decisions in this script:

1. Digest throughout the entire flow. The digest obtained in Step 1 is used for copy, audit, and deployment — no dependency on tag stability.
2. Credentials from environment variables. Not hardcoded; injected via CI/CD platform Secrets.
3. Layered gates. dev→staging does security scanning; staging→prod does signature verification — separated responsibilities.
4. Audit log. Every promotion records timestamp, image, digest, operator — traceable when issues arise.

Production Pitfalls

Pitfall 1: Harbor Replication Policy Conflicting with Skopeo

Harbor has a built-in Replication feature for automatic image sync. Initially, we used Harbor’s replication policy for dev→staging sync while also running Skopeo manual copies in the CI pipeline. The two replication mechanisms clashed — the same image was copied twice, resulting in duplicate manifests in the staging registry with the same digest but confused tag references.

After investigation, we found that Harbor replication and Skopeo manual copy shouldn’t run on the same promotion path. The final decision: promotion pipeline uses Skopeo only; Harbor replication is used only for cross-region disaster recovery sync (e.g., the prod registry’s异地容灾副本). The two don’t overlap.

Pitfall 2: Over-Privileged Robot Accounts

Initially, the promotion pipeline’s robot account had Developer permissions — push, pull, and delete. Once, a CI script had a bug: skopeo copy’s target address was missing a path segment, and the image was pushed to the registry root instead of the project directory. Cleanup revealed that the Developer permissions were enough to delete images within the project, but couldn’t delete the “orphan image” in the root directory — a Harbor admin had to manually clean it up.

Fix: configure robot accounts with minimal permissions — write access to the target project and read access to the source project only, no delete permissions. Harbor’s robot accounts support Project-level permission control. Combined with three physically isolated Harbors, the permission boundary is very clear.

# Robot account permission matrix
# dev-bot:     push to dev project + pull from all projects
# staging-bot: push to staging project + pull from dev project
# prod-bot:    push to prod project + pull from staging project

Note: prod-bot must not have pull access to the dev project — prod should never pull images directly from the dev registry. Promotion must go through staging. This is called “unidirectional promotion path,” preventing skipping staging and promoting directly from dev to prod.

Pitfall 3: GC Timing Causing Disk Bloat

Harbor’s garbage collection (GC) requires marking images as “immutable” first, then running GC to clean up unreferenced blobs. But in the promotion pipeline, an image might be marked for deletion in the dev registry (7-day retention expired) while the staging registry is still copying it from dev.

What happened? The staging registry’s copy failed with “source image not found” — because GC had already deleted the blob. The fix: adjust the GC strategy — clean up tag references first, delay blob cleanup by 24 hours. This gives the promotion pipeline a window to complete cross-registry copies.

# Harbor GC strategy adjustment
# Original: mark deletion → immediately GC blob
# New: mark deletion → 24h buffer → GC blob
# This gives the promotion pipeline a 24-hour window for cross-registry copy

Pitfall 4: Multi-Arch Image Promotion

Initially, all our images were single-arch (amd64), and the promotion pipeline worked fine. After ARM servers joined the cluster, we started building multi-arch images (amd64 + arm64). Skopeo copy by default only copies the single-arch manifest — the multi-arch manifest list wasn’t copied. The staging registry only had the amd64 image; arm64 nodes couldn’t pull images.

Fix: add --all to skopeo copy to copy the complete manifest list:

# --all copies all architectures in the manifest list
skopeo copy --all \
  docker://${SRC}/${IMAGE_NAME}@${DIGEST} \
  docker://${DST}/${IMAGE_NAME}:${TAG}

This pitfall is sneaky — skopeo copy without --all doesn’t error out. The copy “succeeds,” but ARM nodes discover the manifest list is incomplete only when pulling. If you have multi-arch images, your promotion script must include --all.

Troubleshooting this took a while. Initially, ARM nodes reported ImagePullBackOff, which we assumed was a network issue. crictl inspect showed the image manifest had only an amd64 entry — the arm64 one was never copied. Digging through Skopeo docs revealed that --all is opt-in, not default. I recommend adding --all to your promotion script by default — even if you only have single-arch images now, the flag has no side effects and saves you from the same pitfall when adding ARM nodes.

If you want to make manifest troubleshooting easier, use skopeo inspect --raw to view the complete manifest list structure and confirm all architecture digests are present. I added a validation step in the CI pipeline: after promotion, automatically inspect the target image and assert the manifest list contains the expected architectures. If mismatched, trigger an alert. This validation uses jq to parse Skopeo’s JSON output, matching "platform.architecture" fields — 5 lines of script that save a lot of post-hoc troubleshooting time.

Performance Comparison: Promotion Pipeline vs Traditional Three-Step

We ran a real-world comparison under the 120+ microservice batch release scenario. One release involves 20 microservice image promotions:

The 12 vs 38 minute gap comes from two sources: Skopeo’s single streamed transfer vs Docker’s double network + disk I/O, and automated security scanning in the promotion pipeline eliminating manual triggering.

This is consistent with my experience building a Go DAG scheduler engine — deployment time dropped from 1.5 hours to 5 minutes, with the artifact promotion pipeline optimization contributing roughly 30% of the speedup. For more detailed CI/CD optimization, see Related: CI/CD Deployment Speed Optimization in Practice: From 90 Minutes to 5 Minutes.

Another easily overlooked benefit is reduced audit cost. The traditional three-step approach has no structured audit log; every compliance check requires piecing together records from the Docker registry’s API logs. The promotion pipeline’s audit log is structured CSV — one line per promotion: timestamp, image name, tag, digest, source env, target env, operator. Compliance audits export the CSV to the audit team — 5 minutes done, no manual log digging.

Monitoring and Alerting for the Promotion Pipeline

The promotion pipeline itself needs monitoring. It’s not “set and forget” — if the pipeline breaks, the entire release flow stops. We instrumented three key metrics:

Three Monitoring Metrics

A spike in promotion_duration_seconds usually means a network issue — Skopeo’s cross-registry copy uses HTTPS, and if bandwidth between dev and staging registries is limited (e.g., cross-AZ), copy time jumps from 30 seconds to several minutes. We hit this during cross-datacenter deployment: staging registry in Zone A, prod registry in Zone B, connected via a 100Mbps dedicated line. Batch promotion of 20 images queued for 8 minutes.

The fix: add concurrency control — Skopeo copy supports copying multiple images simultaneously, but limit concurrency to avoid saturating bandwidth:

# Use GNU parallel to control concurrency, max 3 concurrent copies
ls images-to-promote.txt | parallel -j 3 \
  './promote.sh $(cat) dev staging'

Alert-ifying Audit Logs

Audit logs aren’t just for archival — they can drive real-time alerts. We use Vector to collect promotion audit logs from files into Loki, then configured three alert rules:

# Alert 1: Non-pipeline image pushes to prod (bypassing the pipeline)
# Audit log entries where the operator is not promotion-bot writing to prod
sum(rate(promotion_audit_log{env="prod",user!="promotion-bot",action="push"}[5m])) > 0

# Alert 2: Abnormal promotion interval (same image promoted multiple times in 1 hour)
count by (image, tag) (
  promotion_audit_log{action="promote"}[1h]
) > 2

# Alert 3: Cross-environment jump promotion (dev directly to prod, skipping staging)
sum(promotion_audit_log{from="dev",to="prod"}) > 0

The third alert is especially important. Someone takes a shortcut and promotes directly from dev to prod, skipping staging — this violates the unidirectional promotion path principle. The alert triggers an automatic notification to the SRE channel, requiring investigation into who bypassed the pipeline.

Promotion Pipeline Availability

The promotion pipeline runs on CI Runners — if the Runner is down, the pipeline stops. We built two layers of redundancy:

1. Multi-node CI Runners. Promotion tasks aren’t bound to a specific Runner; any idle Runner can execute them.
2. Degradation strategy. If all Runners are unavailable, manual execution of promote.sh is allowed — the script itself doesn’t depend on the CI platform; it just needs Skopeo and registry credentials.

This degradation strategy was used twice in practice. Once when GitLab CI platform upgrade took all Runners offline, once when the K8s cluster nodes hosting Runners went NotReady. Both times, manual script execution completed the promotion without missing the release window. When running manually, note one thing: pass credentials via environment variables, never write tokens in cleartext on the command line. Use export STAGING_REGISTRY_TOKEN=xxx before running the script, then unset after.

FAQ

Q: We already have one Harbor instance; migrating to three is too costly

Start with three Projects in a single Harbor, use RBAC + immutable tag rules for logical isolation, and split when image volume grows. The key is to use digest promotion and Skopeo copy from the start — the registry architecture can be changed later; the promotion pipeline’s core logic stays the same.

When migrating to three Harbor instances, there’s an order: deploy Staging and Prod instances first, configure replication policies and robot accounts, then route new builds through the new pipeline. Leave old images in the original Harbor to expire naturally. Don’t migrate all images at once — it’s not worth it, and the network overhead of batch skopeo copy on many images is significant. We migrated 200+ images in 4 hours, bottlenecked by cross-AZ bandwidth.

Q: How to integrate Skopeo with Jenkins / GitLab CI

Package Skopeo into a CI Runner base image. Jenkins uses a Shared Library to wrap the promotion step; GitLab CI uses an include template. The key is to never hardcode registry credentials in CI scripts — use the CI platform’s Secret management for injection.

# GitLab CI example
stages:
  - build
  - promote

promote-to-staging:
  stage: promote
  image: 
    name: quay.io/skopeo/stable:latest
    entrypoint: [""]
  variables:
    SRC_REGISTRY: $DEV_REGISTRY
    DST_REGISTRY: $STAGING_REGISTRY
  script:
    - DIGEST=$(skopeo inspect --creds "dev-bot:$DEV_TOKEN" docker://$SRC_REGISTRY/app:$CI_COMMIT_TAG --format '{{.Digest}}')
    - skopeo copy --all --src-creds "dev-bot:$DEV_TOKEN" --dest-creds "staging-bot:$STAGING_TOKEN" docker://$SRC_REGISTRY/app@$DIGEST docker://$DST_REGISTRY/app:$CI_COMMIT_TAG
  only:
    - tags

Q: What happens if promotion fails and I need to roll back

Promotion failure usually means one of two things: security scan didn’t pass, or Skopeo copy timed out on the network. A scan failure is a normal block — you don’t need to “roll back” because the image never entered the target registry. A network timeout — just retry. Skopeo copy is idempotent; running it again won’t create duplicate images.

The real rollback scenario is “image was already promoted to prod but runtime issues were discovered.” In this case, you’re not rolling back the promotion pipeline — you’re rolling back the deployment version by deploying the previous digest. This is exactly why the audit log must record every promotion’s digest.

# Find the previous version's digest from the audit log
grep "app:1.0.0" /data/promotion-audit.log | tail -2
# Output:
# 2026-09-18T14:30:00+08:00|app:1.0.0|sha256:aaa111...|staging→prod|jenkins
# 2026-09-18T10:00:00+08:00|app:0.9.9|sha256:bbb222...|staging→prod|jenkins

# Roll back using the previous digest
kubectl set image deployment/api-gateway \
  api-gateway=registry-prod.company.com/app@sha256:bbb222...

Summary

Artifact promotion isn’t a new concept, but many teams stop at “re-tag and push three times.” The 4 decisions in this article boil down to a few sentences:

1. Promote by digest, not by tag. Digests are immutable; tags are mutable. Lock images by digest in production for reliable rollbacks.
2. Use Skopeo for cross-registry copy. No Docker Daemon dependency, no local disk, add --all for multi-arch images.
3. Choose registry isolation by scale. Physical isolation for 50+ microservices; logical isolation with strict RBAC for fewer. Minimal robot account permissions, unidirectional promotion path.
4. Layered security gates with CVE exception rules. Block Critical CVEs, alert on High, time-bound approved exceptions. Signature verification as the last gate.

After implementing this pipeline, our deployment flow went from “build and push straight to prod” to a segmented “build → scan → promote → verify → deploy” pipeline. Every stage is auditable, blockable, and reversible. Batch releases of 120+ microservices dropped from 38 minutes to 12 minutes, and change failure rate dropped from 17% to under 3%.

The core philosophy of artifact promotion is “immutability” — the same image from build to production, unchanged; what changes is only its location. The concept isn’t complex, but it requires coordination across registry architecture, promotion scripts, security gates, and audit logs. Once it’s running, you’ll never worry about “the version tested in staging isn’t the version running in prod” again.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. Skopeo Image Operations Tool — containers organization, cross-registry image copy and inspection tool
2. Harbor Official Documentation - Administration — Harbor team, registry management, replication policies, and project quota configuration
3. Harbor 1.8 Security and Replication New Features — Harbor team, cross-registry replication and OIDC integration
4. Docker Image Version Management Best Practices — SimCompile, from tag chaos to semantic versioning in image governance
5. Docker Image Migration and Acceleration in Domestic Network Environments — mToLa4mVs, using Skopeo instead of docker pull/tag/push
6. Docker Image Management Complete Guide — weixin_28419039, image layering, digest and tag mechanisms
7. Enterprise CI/CD Platform Construction: From Architecture Design to Best Practices — weixin_33391446, four-layer CI/CD architecture design and quality gates