Container Image Signing and Verification: Using cosign to Guard the Last Gate of Supply Chain Security

Overview

Let me start with a story that still haunts me.

Last year, our internal image registry had a permissions misconfiguration that was loose for several days. Post-incident review found no evidence of tampering, but those few days had me on edge — what if someone had pushed an image with the same tag, replacing app:latest with their own version? Our deployment pipeline would have pulled and run it just the same. Who would have noticed?

The digest would have changed, sure. But back then, nobody in our deployment process was checking digests. An image, by default, follows the logic of “trust whoever can push.” There’s an app:v1.2.3 sitting in the registry — how do you know it was built and pushed by your CI, not by an impersonator?

The answer is uncomfortable: looking at it, you can’t tell. You need signatures.

This article covers how to use cosign (the core tool of the Sigstore ecosystem) to sign and verify container images, solving the trust problem across the chain from image build to runtime. The core approach: CI signs the image after building, deployment verifies the signature before running, and images that fail verification never enter production.

Supply Chain Security Threats

Before discussing solutions, we need to understand what we’re defending against.

Attack Paths

The hallmark of container image supply chain attacks is that attackers don’t need to directly breach your production servers — they just need to interfere at any point before the image reaches production.

Typical attack paths:

Attacker
  │
  ├─ Path 1: Upload malicious base image to Docker Hub
  │           → Developer builds app based on that image
  │           → Malicious code enters CI/CD pipeline
  │           → Deployed to production
  │
  ├─ Path 2: Compromise CI system, tamper with build artifacts
  │           → Push same-tag image with modifications
  │           → Deployment system pulls "latest" image
  │           → Backdoor enters production
  │
  ├─ Path 3: Man-in-the-middle attack, tamper with image transfer
  │           → Intercept docker pull request
  │           → Return tampered image layers
  │
  └─ Path 4: Insider error or malicious action
            → Push unverified image directly to production registry

Paths 1 and 2 are the most common. In 2024, security researchers found over 10,000 images on Docker Hub leaking production system credentials. Sysdig’s container security report shows that over 65% of container images contain known high-severity vulnerabilities — often from base images.

Why Traditional Solutions Fall Short

Digest pinning — writing the image SHA256 digest into deployment configs instead of tags — is the most common approach. It does prevent the tag re-pointing problem. But it doesn’t solve the trust question of “who built this digest’s image.” An attacker could push a new image, get a new digest, and find a way to update your deployment config to use that new digest.

Signing solves exactly this trust problem: it proves “this image was indeed built and signed by a specified build system (like your GitHub Actions workflow) and hasn’t been tampered with since.”

Sigstore Ecosystem: Making Signing Less Painful

Why Nobody Bothered with the Old Way

Signing has existed for twenty years — the GPG approach. The problem is that key management is too painful.

You generate a key pair. The private key must be stored securely — where? In CI secrets? If CI is compromised, the key is leaked. Private keys expire, need rotation, and must be revoked when people leave. Then there’s the question of how to distribute the public key to verifiers and establish trust. The overhead is so heavy that most teams sign a few times and give up — “it’s internal, should be fine, right?” This mentality is exactly where security protection starts to fail.

Sigstore’s Core Innovation: Keyless Signing

Sigstore is an open-source project under the Linux Foundation. Its core innovation is called keyless signing. It’s not that there are no keys — it’s that keys are generated ephemerally and discarded after use.

In plain terms: traditional signing is like carrying a personal seal with you everywhere, using it for every stamp — if the seal is lost or stolen, you’re done. Keyless signing is like carving a fresh seal each time, destroying it after use, while the fact that “you did stamp this” is guaranteed by an authoritative notary (Sigstore’s certificate authority, Fulcio) and recorded in a public ledger that everyone can check (the transparency log, Rekor).

The full flow:

When signing:
  1. cosign uses OIDC to let your CI prove identity
     → "I am the GitHub Actions workflow for repo xxx"
  2. Fulcio (certificate authority) issues a short-lived certificate
     → Valid for just a few minutes, then void
  3. Sign with the ephemeral private key corresponding to that certificate
  4. Discard the private key — no need to store it
  5. Signature record + certificate written to Rekor (transparency log)
     → Immutable, publicly queryable

When verifying:
  1. Look up signature record and certificate from Rekor
  2. Verify the certificate was indeed issued by Fulcio
  3. Verify the identity info in the certificate (OIDC issuer + identity)
  4. Verify the signature matches the image digest
  5. All pass → trust this image

No long-term private key to store means no key leakage or rotation problems. This is Sigstore’s smartest design.

The Sigstore Trio

They work together: cosign initiates signing → Fulcio issues short-lived certificate based on OIDC identity → signature record written to Rekor. Verification reverses: look up record from Rekor → verify Fulcio certificate → verify identity and signature.

cosign in Practice

Installation

# Linux
curl -O -L "https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64"
mv cosign-linux-amd64 /usr/local/bin/cosign
chmod +x /usr/local/bin/cosign

# macOS
brew install cosign

# Verify installation
cosign version

Approach 1: Keyless Signing (Recommended)

Keyless signing is Sigstore’s core capability and my recommended approach for production.

# Sign the digest, not the tag
# A tag is a mutable pointer — today it points here, tomorrow it can be re-pointed
# A digest is a content hash — change one byte and it changes, signing it is meaningful
cosign sign registry.example.com/app@sha256:abc123def456...

# When running locally, it opens a browser for OIDC login
# In CI, OIDC token is usually configured for automatic authentication

Why must you sign the digest, not the tag? A tag is just a mutable pointer — today it points to your image, tomorrow someone can re-point it to another. Signing a tag guarantees nothing — when verification passes, the tag might already be pointing to a different image. The digest is the SHA256 hash of the content — change one byte and it changes. Signing the digest is what actually matters.

Approach 2: Key Pair Signing

Some scenarios don’t support keyless (e.g., air-gapped environments can’t reach Fulcio). In those cases, use the traditional key pair approach:

# Generate key pair
# Will prompt for a password to encrypt the private key
cosign generate-key-pair
# Generates two files:
#   cosign.key - private key (must be kept secret)
#   cosign.pub - public key (distribute to verifiers)

# Sign with private key
cosign sign --key cosign.key \
  registry.example.com/app@sha256:abc123def456...

# Verify with public key
cosign verify --key cosign.pub \
  registry.example.com/app@sha256:abc123def456...

The key pair approach has the drawbacks mentioned earlier: private key storage, rotation, and revocation are all burdens. But if your environment can’t access the internet (can’t use Fulcio’s OIDC), this is the only option.

Private key storage advice: Don’t put it in CI secrets — if CI is compromised, the key is leaked. Use a key management system like HashiCorp Vault, AWS KMS, or Azure Key Vault. cosign natively supports these backends.

Verification: Signing Without Verifying Is Useless

Signing is just the first step. What actually blocks tampering is verification — and it must be at the admission gate, so images that fail verification never reach production.

# Keyless verification - must specify identity and issuer
# These two parameters are critical, most commonly missed by beginners
cosign verify \
  --certificate-identity "https://github.com/myorg/myapp/.github/workflows/release.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  registry.example.com/app@sha256:abc123def456...

Verification can’t just check “this image has a signature” — an attacker can sign their own image too. You need to verify “this signature comes from a build system I trust.”

• --certificate-identity: specifies the build identity you trust (e.g., your GitHub Actions workflow URL)
• --certificate-oidc-issuer: specifies the OIDC issuer (for GitHub Actions, it’s https://token.actions.githubusercontent.com)

Together, these parameters form the whitelist of “whose builds I trust.” Only signatures matching both identity and issuer pass.

CI/CD Integration: GitHub Actions

Embed signing into the CI pipeline so every build’s image is automatically signed.

Configuring OIDC Permissions

GitHub Actions keyless signing requires OIDC permissions. Add this to the workflow file:

# .github/workflows/build-and-sign.yml
name: Build and Sign

on:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write   # Critical: enable OIDC token write permission
  packages: write

jobs:
  build-sign:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Login to registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build image
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:latest
          # Critical: also output digest, needed for signing
          outputs: type=oci,dest=/tmp/image.tar

      - name: Install cosign
        uses: sigstore/cosign-installer@v3

      - name: Get image digest
        id: digest
        run: |
          DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ghcr.io/${{ github.repository }}:latest | cut -d'@' -f2)
          echo "digest=$DIGEST" >> $GITHUB_OUTPUT
          echo "ref=ghcr.io/${{ github.repository }}@${DIGEST}" >> $GITHUB_OUTPUT          

      - name: Sign image (keyless)
        run: |
          cosign sign --yes \
            ghcr.io/${{ github.repository }}@${{ steps.digest.outputs.digest }}          
        # --yes skips interactive confirmation, required in CI

      - name: Verify signature
        run: |
          cosign verify \
            --certificate-identity "https://github.com/${{ github.repository }}/.github/workflows/build-and-sign.yml@refs/heads/main" \
            --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
            ghcr.io/${{ github.repository }}@${{ steps.digest.outputs.digest }}          
        # Verify immediately after signing to confirm the signature is valid

      - name: Attach SBOM (optional but recommended)
        run: |
          # Generate SBOM and attach to image
          # SBOM = Software Bill of Materials
          syft ghcr.io/${{ github.repository }}@${{ steps.digest.outputs.digest }} -o spdx-json > /tmp/sbom.spdx.json
          cosign attach sbom --sbom /tmp/sbom.spdx.json \
            ghcr.io/${{ github.repository }}@${{ steps.digest.outputs.digest }}

          # Sign the SBOM to prevent tampering
          cosign sign --yes \
            --attachment sbom \
            ghcr.io/${{ github.repository }}@${{ steps.digest.outputs.digest }}          

Security Checkpoints in a Complete Pipeline

A complete supply chain security pipeline has more than just signing — it’s a series of checkpoints:

Signing is one checkpoint in this chain. Don’t do only signing without vulnerability scanning — a signed vulnerable image is still a ticking bomb.

Kubernetes Admission Verification

Putting signing in CI isn’t enough — what if someone bypasses CI and pushes images directly to the registry, then deploys manually? The real defense line must be at the Kubernetes cluster entrance, so images without valid signatures can’t be deployed at all.

Option 1: Kyverno Policy

Kyverno is a Kubernetes policy engine that can write policies to verify image signatures. This is my recommended approach — simple configuration, policy as code.

# kyverno-verify-images.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce  # Enforce=reject, Audit=log only
  background: false
  webhookTimeoutSeconds: 30
  rules:
    - name: verify-ghcr-images
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/*"  # Only verify our own registry images
          mutateDigest: true      # Auto-replace tag with digest
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/myorg/*/.github/workflows/*.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

What this policy does:

1. Matches all ghcr.io/myorg/* images
2. mutateDigest: true auto-replaces tag with digest (prevents tag re-pointing)
3. Verifies signature using keyless method, specifying trusted subject and issuer
4. Looks up signature record from Rekor transparency log
5. In Enforce mode, rejects deployment if verification fails

After deploying this policy, any attempt to deploy a ghcr.io/myorg/* image without a valid signature will be rejected by Kubernetes.

Option 2: cosign webhook

If you don’t want to install Kyverno, you can use cosign’s built-in admission webhook. Configuration is slightly more complex but sufficient:

# cosign admission controller config
apiVersion: v1
kind: ConfigMap
metadata:
  name: cosign-webhook-config
  namespace: cosign-system
data:
  config.yaml: |
    # Specify allowed images and their verification methods
    allow:
      - pattern: "ghcr.io/myorg/*"
        verify_config:
          keyless:
            subject: "https://github.com/myorg/*/.github/workflows/*.yml@refs/heads/main"
            issuer: "https://token.actions.githubusercontent.com"
      - pattern: "docker.io/library/*"
        # Official images don't need signature verification
        skip_verify: true    

Exemption Policies

Not all images need signature verification. System components like pause and kube-proxy, and official docker.io/library/* images should be whitelisted:

# Kyverno policy with whitelist
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
spec:
  validationFailureAction: Enforce
  background: false
  webhookTimeoutSeconds: 30
  rules:
    - name: exclude-system-namespaces
      match:
        any:
          - resources:
              kinds:
                - Pod
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - kube-public
                - kube-node-lease
                - cosign-system
                - kyverno
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/*"
          # ... verification config

Advanced: Attestations

Signing only proves “I built this image.” But supply chain security needs to answer more questions: what base image was used? What’s the dependency list at build time? Was it scanned for vulnerabilities? What tests passed?

This information is recorded with “attestations.” An attestation is structured data attached to an image, itself signed.

Generating SBOM Attestation

# Generate SBOM with syft
syft ghcr.io/myorg/app@sha256:abc123... -o spdx-json > sbom.spdx.json

# Attach SBOM as attestation (auto-signed)
cosign attest --yes \
  --predicate sbom.spdx.json \
  --type spdxjson \
  ghcr.io/myorg/app@sha256:abc123...

# Verify SBOM attestation
cosign verify-attestation \
  --certificate-identity "https://github.com/myorg/app/.github/workflows/build.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  --type spdxjson \
  ghcr.io/myorg/app@sha256:abc123...

Vulnerability Scan Attestation

# Scan vulnerabilities with grype
grype ghcr.io/myorg/app@sha256:abc123... -o json > vuln-scan.json

# Attach scan result as attestation
cosign attest --yes \
  --predicate vuln-scan.json \
  --type vuln \
  ghcr.io/myorg/app@sha256:abc123...

Multi-Layer Verification Policy

With multiple attestations, you can write fine-grained policies in Kyverno: only allow images with SBOM attestations and no Critical vulnerabilities.

# Kyverno multi-layer verification policy
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-sbom-and-scan
spec:
  validationFailureAction: Enforce
  background: false
  webhookTimeoutSeconds: 30
  rules:
    - name: check-signature-and-sbom
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/*"
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/myorg/*/.github/workflows/*.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
          attestations:
            - type: spdxjson
              attestors:
                - count: 1
                  entries:
                    - keyless:
                        subject: "https://github.com/myorg/*/.github/workflows/*.yml@refs/heads/main"
                        issuer: "https://token.actions.githubusercontent.com"
            # Can also add vulnerability scan attestation verification
            - type: vuln
              attestors:
                - count: 1
                  entries:
                    - keyless:
                        subject: "https://github.com/myorg/*/.github/workflows/*.yml@refs/heads/main"
                        issuer: "https://token.actions.githubusercontent.com"

Practical Pitfalls Summary

Several pitfalls encountered while implementing cosign:

Pitfall 1: Signing tags instead of digests. Initially took the easy route: cosign sign ghcr.io/myorg/app:latest. Later found the tag had been re-pointed to another image, and verification still passed — because signature verification checks the digest the tag points to. When the tag changed, the digest changed, but the old signature was still in the registry. Correct approach: always sign digests, enable mutateDigest: true in Kyverno policy to force tag-to-digest conversion.

Pitfall 2: Verifying only “has signature” without checking identity. Running cosign verify without --certificate-identity and --certificate-oidc-issuer only checks “image has a signature,” not “signature is from whom.” An attacker who signs their own image would pass. These two parameters are mandatory.

Pitfall 3: Webhook timeout. Keyless verification queries the Rekor transparency log, which can timeout on poor networks. Kubernetes default webhook timeout is 10 seconds — recommend increasing to 30 seconds. If timeouts persist, consider narrowing the query scope with --certificate-identity or deploying a local Rekor instance.

Pitfall 4: Public network dependency. Keyless signing and verification both depend on Sigstore’s public services (Fulcio, Rekor). Air-gapped environments can’t use them. In this case, use key pair signing or self-host a Sigstore instance (not cheap — recommend only for large teams).

Pitfall 5: Private key management with key pair signing. With key pair signing, where to store the private key is always a problem. Tried CI secrets, K8s Secrets, employee laptops. Finally settled on HashiCorp Vault. cosign natively supports the Vault backend — configure it and use cosign sign --key hashivault://, and the private key never leaves Vault.

Summary

Container image signing solves the supply chain trust problem: proving “this image was indeed built by me and hasn’t been tampered with since.” Without signing, your deployment system can’t distinguish “trusted images” from “tampered images.”

Recommended implementation in three steps:

1. Step 1: CI signing. Add cosign sign at the end of the build pipeline, using keyless mode for zero key management overhead. This is the lightest starting point, requiring no deployment-side changes.

2. Step 2: Deployment verification. Install Kyverno, configure verifyImages policy, and enforce signature verification at the Kubernetes admission layer. Images failing verification are rejected. This is the defense line that actually matters.

3. Step 3: Multi-layer attestations. Add SBOM attestations, vulnerability scan attestations, and require both signatures and SBOMs in policies. Supply chain security upgrades from “has signature” to “has signature + has bill of materials + vulnerabilities scanned.”

Several key decisions:

• Choose keyless over key pair signing — unless air-gapped, keyless eliminates all key management burden
• Sign digests, not tags — tags are mutable pointers, signing them guarantees nothing
• Verification must check identity — --certificate-identity and --certificate-oidc-issuer cannot be omitted
• Defense at admission layer, not just CI — CI can always be bypassed; Kubernetes admission is the last gate

Supply chain security isn’t “sign and done.” Signing is one link in the trust chain — combined with vulnerability scanning, SBOM, and policy engines, it forms complete protection.

References & Acknowledgments

The following resources were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. 用 Sigstore 给镜像签名，下游能验来源不怕被掉包 — Baijiahao blogger, Sigstore keyless signing principles and cosign command practices, the primary reference for this article’s core approach
2. Docker 容器安全深度实战：从镜像构建到运行时防护的生产级安全体系 — Programmer Qiezi, container supply chain attack path analysis and multi-stage build security practices
3. Red Hat Trusted Application Pipeline Documentation — Red Hat, enterprise-grade image signature verification and policy compliance framework
4. 2026 DevSecOps Engineer Interview: 50 Practical Questions on CI/CD Security + Container Escape Defense + Supply Chain Poisoning Detection — CSDN blogger, DevSecOps supply chain security checkpoint landscape and shift-left security practices
5. Docker Registry Security Compliance and Xinchuang Adaptation Practice Guide (2026) — Blog Park blogger, image vulnerability scanning, SBOM generation, and compliance practices