3,000 Lines of HCL Nobody Dares to Touch? 5 Production Decisions and Migration Pitfalls of Redoing IaC in a Real Programming Language with Pulumi

Overview

A team has been running Terraform for three years. Their main module has grown to 3,000 lines of HCL, with deeply nested variable definitions, multiple providers, and for_each loops that make code review a nightmare. Every change is like defusing a bomb—the pull request gets reviewed five times, tested in three environments, and the final plan output scrolls for 200+ lines before anyone clicks apply. The last guy who understood the module structure left two months ago, and the handover doc is an abandoned Notion page.

Then someone discovers Pulumi. No more HCL; you write TypeScript, Python, Go, or .NET. You get real type checking, IDE auto-complete, and unit test frameworks that actually work. The pitch is intoxicating. But migration is another story entirely—here is where the real engineering decisions begin.

This article focuses on the five decisions you must make before, during, and after migrating from Terraform to Pulumi, using a concrete AWS production scenario as the running example. Each decision includes a “pitfall journal” drawn from real-world incidents, so you don’t have to pay the same tuition twice.

Why Terraform’s HCL Becomes Unmanageable at Scale

HCL is a domain-specific language, not a general-purpose one. When your infrastructure was 20 resources, HCL was clean and efficient—declarative syntax let you define resources concisely. But when it grows to 200+ resources, HCL starts to expose fundamental flaws:

1. No Real Type System

HCL’s variable blocks are essentially strings, lists, and maps with no structural validation. You define a variable as type = map(string), but at the plan stage it explodes because a required key is missing. There’s no compiler telling you beforehand—errors only surface at runtime.

variable "subnet_configs" {
  type = map(object({
    cidr_block     = string
    availability_zone = string
    map_public_ip_address = bool
  }))
}

The 30-line type definition above doesn’t tell you which keys are required until you run terraform plan and it fails. In TypeScript, you’d have compiler-level checking the moment you type it.

2. No Abstraction Mechanism

HCL’s module system is essentially “copy code and parameterize.” You can’t define interfaces, inheritance, or composition. When you need to create slightly different resource groups for dev, staging, and prod, the options are: duplicate three copies of 300 lines each, or write for_each loops so complex that nobody understands them. Real programming languages have classes, functions, and design patterns to handle this elegantly.

3. for_each Is Not a Loop

Technically, HCL’s for_each and for are iteration constructs, but they’re expressions, not statements. You can’t break out of them, can’t accumulate state, can’t conditionally skip. In practice, you end up writing expressions like this:

{ for k, v in local.configs : k => merge(v, {
  tags = merge(local.default_tags, v.tags)
  }) if v.enabled
}

When this expression grows to 50+ lines, you’ve lost track of what it does. The equivalent in Python or TypeScript is a plain for loop with an if condition—10 lines, clear at a glance.

4. State File Fragility

Terraform’s state file (.tfstate) is the single source of truth for infrastructure. It’s a JSON file storing every resource’s ID, attributes, and dependencies. But this file is fragile: concurrent runs corrupt it, team members’ local modifications desync it, and terraform refresh sometimes turns into a horror show. Pulumi also has state management, but because you can write real code to handle state logic, it’s more controllable.

These four pain points are why teams growing past a certain scale consider Pulumi. But the decision to migrate is just the beginning—let’s get into the real engineering decisions.

Decision 1: Migration Strategy—Full Rewrite vs. Gradual Coexistence

The Core Conflict

The first decision when starting a Pulumi migration is: do you rewrite all existing Terraform resources in Pulumi, or migrate gradually? This is not a technical choice—it’s a trade-off between risk, team capability, and business continuity.

Option A: Full Rewrite (High Risk, High Reward)

You rebuild all resources from scratch in Pulumi, then switch the entire infrastructure to Pulumi management in one go.

Applicable conditions:

• Infrastructure scale is moderate (< 200 resources)
• Team has bandwidth to focus on the migration
• Business can tolerate a full cutover window (maintenance period acceptable)

Risks:

• During the rewrite, Terraform infrastructure continues to evolve; your Pulumi code is always a step behind
• Full cutover means you might miss resources that only exist in Terraform state
• Rollback difficulty—if something goes wrong, you need to switch back to Terraform state

Option B: Gradual Coexistence (Low Risk, Long Cycle)

Terraform manages some resources, Pulumi manages others, importing from Terraform state incrementally.

Applicable conditions:

• Large infrastructure scale (> 200 resources), can’t afford full rewrite
• Business cannot tolerate full cutover
• Team is learning Pulumi and needs time to ramp up

Risks:

• Two IaC tools running long-term increases operational complexity
• Resource dependencies cross tool boundaries—Terraform resource A depends on Pulumi resource B—creating coupling
• State synchronization issues—Terraform’s state doesn’t know about Pulumi changes, and vice versa

My Recommendation: Start Gradual, Then Consolidate

In practice, the most reliable strategy is: start with gradual coexistence, pick the most independent resource group (e.g., monitoring configuration, IAM policies), migrate it to Pulumi first, validate the workflow and CI/CD pipeline, then gradually expand scope. When 80% of resources are on Pulumi, do a final full cutover for the remaining 20%.

Pitfall Journal: Cross-Tool Dependencies Turned into a Maze

Scenario: A team migrated IAM policies to Pulumi but kept VPC and subnets in Terraform. Then they added a new IAM role in Pulumi that needed to reference Terraform’s VPC ID.

Root cause: Cross-tool dependency. Pulumi needed the VPC ID as an input, but the VPC was managed by Terraform. Options: hardcode the VPC ID (bad—rebuilds need manual updates), or read it via aws_ec2_describe_vpcs at runtime (coupling—VPC changes break Pulumi runs).

Resolution: Created a “boundary resource” pattern—export Terraform VPC attributes to a parameter store (AWS Systems Manager Parameter Store), and have Pulumi read from there. Changes to either side only affect the parameter store, not direct coupling.

Lesson: When doing gradual coexistence, define clear boundaries for cross-tool resource dependencies from day one, and use an intermediary store (Parameter Store, Vault, etc.) for decoupling. Don’t let Terraform resources directly reference Pulumi outputs or vice versa.

Decision 2: Language Selection—TypeScript vs. Python vs. Go

The Core Trade-Off

Pulumi supports multiple languages, and this is its biggest advantage over Terraform. But language choice is not just about personal preference—each has different ecosystem maturity, type system strength, and team adoption cost.

TypeScript

Pros:

• Pulumi’s first-class citizen—newest features land in TypeScript first
• Type system is powerful, IDE support is excellent (VS Code auto-complete, type checking)
• npm ecosystem is vast, with countless utility libraries
• Async/await model is clear for resource dependency handling

Cons:

• Node.js runtime environment has some quirks (memory management, event loop)
• For teams with non-frontend backgrounds, the learning curve is real
• Large projects’ node_modules is notoriously messy

Best for: Teams with frontend/Node.js background, or projects prioritizing DX (developer experience)

Python

Pros:

• Syntax is concise and readable, very friendly for DevOps engineers
• Rich ecosystem (boto3, requests, etc.) integrates well with AWS operations
• Dynamic typing is flexible—you can write code fast
• No compilation step, iteration is quick

Cons:

• Dynamic typing means Pulumi’s type hints are just hints, not compile-time checks
• Virtualenv management adds complexity
• Performance is relatively lower, though it rarely matters for IaC

Best for: Teams with Python/DevOps background, projects prioritizing fast iteration

Go

Pros:

• Strong type system, compile-time error checking
• Single-binary deployment, no runtime dependencies
• Goroutine concurrency model is natural for parallel resource creation
• Excellent fit with cloud-native ecosystems (Kubernetes, Terraform providers)

Cons:

• Pulumi Go SDK is relatively newer, with fewer community examples
• Go’s error handling boilerplate is tedious for large codebases
• Less mature IDE support compared to TypeScript

Best for: Teams with Go/cloud-native background, projects prioritizing performance and type safety

My Recommendation

If your team is Go-native (like many SRE teams), choose Go. If you’re frontend-leaning, TypeScript. If you value speed and simplicity, Python. Don’t pick a language just because “Pulumi recommends TypeScript”—team familiarity matters more than ecosystem maturity.

Pitfall Journal: Python Dynamic Typing Caused a Production Incident

Scenario: A team used Python to write Pulumi code. A resource’s property was renamed from vpc_id to vpcID in a Pulumi provider upgrade. Python didn’t raise any error at “compile time” (there’s no compile step), and the error only surfaced at runtime—pulumi up failed halfway, leaving resources in a half-created state.

Root cause: Python’s dynamic typing meant Pulumi’s type hints were just IDE suggestions, not compile-time enforcement. The property rename was a breaking change, but Python didn’t catch it until runtime.

Resolution: Switched the project to Go (which has compile-time type checking). A similar provider upgrade later caused a compile error that was caught immediately, never reaching runtime.

Lesson: If your infrastructure is critical and you want maximum type safety, choose Go or TypeScript, not Python. Python’s flexibility is a double-edged sword—fast to write, but the type system can’t save you from breaking changes.

Decision 3: State Management—Pulumi Cloud vs. Self-Hosted Backend

The Core Trade-Off

Pulumi’s state management is similar to Terraform’s, but the options differ. Terraform has remote backends (S3, Consul, Terraform Cloud), while Pulumi has two main options: Pulumi Cloud (SaaS) or self-hosted backend (AWS S3 + DynamoDB locking).

Pulumi Cloud (SaaS)

Pros:

• Zero configuration—sign up, log in, start using
• Built-in CI/CD integration, policy packs, and audit logs
• Web UI for state visualization and resource graphs
• Free tier supports up to 200 resources, sufficient for small teams

Cons:

• State data is stored on Pulumi’s servers—security and compliance concerns
• Network latency (Pulumi Cloud is in the US), CI/CD runs may be slower
• Vendor lock-in—if you leave, exporting state isn’t trivial
• Enterprise pricing is high—$75/user/month for the Team tier

Self-Hosted Backend (S3 + DynamoDB)

Pros:

• State data stays in your AWS account, fully under your control
• No network latency for AWS resources
• No per-user fees—cost is just S3 storage (a few dollars/month)
• Meet strict compliance requirements (data residency, audit)

Cons:

• You manage the backend infrastructure (S3 bucket, DynamoDB lock table, IAM policies)
• No web UI—state visualization requires CLI or custom tooling
• No built-in policy packs—you need to set up OPA or custom policies
• Team members need IAM access to the backend

My Recommendation

For teams already on AWS with compliance requirements, self-hosted backend (S3 + DynamoDB) is the better choice. You keep full control of state data, avoid per-user fees, and can meet audit requirements. For small teams or POCs, Pulumi Cloud’s free tier is fine—start fast, migrate later.

Pitfall Journal: S3 Backend Lock Failure Caused State Corruption

Scenario: A team set up a self-hosted backend using S3 + DynamoDB. Two CI/CD pipelines ran pulumi up simultaneously—DynamoDB locking should have prevented this. But a misconfigured IAM policy prevented the lock from being acquired, both runs proceeded, and the state file was corrupted.

Root cause: The IAM role for CI/CD was missing dynamodb:PutItem permission on the lock table. Pulumi’s locking mechanism silently failed—no error, just no lock acquired. Both pipelines read the same state, made changes, and one overwrote the other.

Resolution: Added the correct IAM permissions and implemented a pre-check script that verifies the lock is acquired before proceeding. Also added a state file version on S3 (S3 versioning) so corrupted states can be rolled back.

Lesson: Self-hosted backend isn’t “set and forget.” You must verify the locking mechanism actually works—don’t assume it does. Add S3 versioning for state file rollback, and implement a pre-flight check that the lock is acquired before any pulumi up.

Decision 4: CI/CD Pipeline—From terraform plan to pulumi preview

The Core Trade-Off

Terraform’s CI/CD workflow is well-established: terraform init → terraform plan → manual approval → terraform apply. Pulumi’s workflow is similar: pulumi install → pulumi preview → manual approval → pulumi up. But the devil is in the details—Pulumi’s CI/CD has some unique pitfalls.

Key Differences

CI/CD Pipeline Design

A robust Pulumi CI/CD pipeline should include:

1. Dependency installation: npm ci (TypeScript) / pip install -r requirements.txt (Python) / go mod download (Go)
2. Pulumi login: pulumi login <backend-url> (self-hosted) or pulumi login (Pulumi Cloud)
3. Secrets decryption: pulumi config set --secret values are decrypted from the backend
4. Preview: pulumi preview --diff to show what will change
5. Manual approval: PR review or Slack notification with approve/reject buttons
6. Apply: pulumi up --yes to execute changes
7. State verification: pulumi stack export to verify state consistency

Pitfall Journal: pulumi preview and pulumi up Show Different Results

Scenario: A team’s CI/CD pipeline ran pulumi preview in CI, showed no changes, got approved. Then pulumi up in the deploy stage created 15 new resources. The preview was a lie.

Root cause: pulumi preview runs in “dry run” mode—it calls the provider’s Check method but doesn’t actually create resources. If the provider has a bug in its Check method (e.g., not properly computing defaults), preview can show “no changes” while pulumi up creates resources. This is a known issue with some community providers.

Resolution: Upgraded the community provider to a newer version that fixed the Check method bug. Also added a “preview with refresh” step (pulumi refresh before pulumi preview) to ensure state is current. Additionally, added a resource count check—if preview shows 0 changes but the stack has < N resources, fail the pipeline.

Lesson: pulumi preview is not 100% reliable. Always run pulumi refresh before preview to ensure state is current. If you’re using community providers, check their issue tracker for known Check method bugs. Add sanity checks—resource count, diff size—to catch preview/apply discrepancies.

Decision 5: Drift Detection and Remediation—Who Watches the Watcher?

The Core Trade-Off

Infrastructure drift—resources are modified outside of IaC (manual console changes, AWS CLI, other tools)—is a problem for both Terraform and Pulumi. But Pulumi has an advantage: because you write real code, you can implement more sophisticated drift detection logic.

Drift Detection Methods

Method 1: pulumi refresh

The simplest method—run pulumi refresh to sync state with actual cloud resources. It reads the current state from the cloud and updates the Pulumi state file. Any differences show up as “drift.”

pulumi refresh --yes
pulumi preview --diff

Pros: Simple, built-in. Cons: Only detects drift, doesn’t prevent it. Also, refresh can be slow for large stacks.

Method 2: Scheduled Drift Detection with Notifications

Use Pulumi’s automation API (available in TypeScript, Python, Go) to write a scheduled drift detection script:

// Drift detection example (Go)
func checkDrift(ctx context.Context, stackName string) ([]DriftResult, error) {
    // Use Pulumi automation API to run refresh
    stack, err := auto.SelectStack(ctx, stackName, opts...)
    if err != nil {
        return nil, err
    }
    
    // Refresh state
    _, err = stack.Refresh(ctx)
    if err != nil {
        return nil, err
    }
    
    // Preview to get drift
    previewResult, err := stack.Preview(ctx)
    if err != nil {
        return nil, err
    }
    
    // Parse preview result for changes
    var drifts []DriftResult
    for _, change := range previewResult.ChangeSummary {
        if change.Action != "same" {
            drifts = append(drifts, DriftResult{
                Resource: change.URN,
                Action:   change.Action,
            })
        }
    }
    return drifts, nil
}

Run this script on a schedule (cron, EventBridge) and send alerts to Slack/PagerDuty when drift is detected.

Pros: Automated, continuous monitoring. Cons: Requires custom development; automation API has a learning curve.

Method 3: Cloud-Native Configuration Guardrails

Use AWS Config rules or equivalent to detect resource changes at the cloud level, independent of Pulumi. This catches drift regardless of the IaC tool:

• AWS Config: config-managed-rules for resource configuration compliance
• CloudTrail: log all API calls, alert on changes outside of CI/CD
• GuardDuty: detect anomalous API activity

Pros: Cloud-native, catches all changes. Cons: AWS-specific; doesn’t know about Pulumi state, only cloud state.

My Recommendation

Combine methods 2 and 3. Use Pulumi’s automation API for scheduled drift detection (catches Pulumi state drift), and use AWS Config/CloudTrail for cloud-level change detection (catches all changes regardless of tool). This gives you two layers of protection.

Pitfall Journal: Manual Console Changes Went Undetected for a Week

Scenario: A developer manually changed an RDS instance type via the AWS console (from db.t3.micro to db.t3.medium for a load test). Nobody noticed for a week, until the next pulumi up tried to change it back, causing an unexpected instance type downgrade during business hours.

Root cause: No scheduled drift detection. The team relied on pulumi preview in CI/CD, which only ran when a PR was created. Since no PR was created that week, the drift went unnoticed. When the next PR was created, pulumi preview showed the instance type change, but the developer didn’t realize it was drift—they thought it was the PR’s intended change.

Resolution: Implemented a daily scheduled drift detection script using Pulumi’s automation API. When drift is detected, it sends a Slack alert with the resource URN and the detected change. Also added a “drift remediation” mode: the script can optionally run pulumi up --yes to automatically remediate drift for non-critical resources.

Lesson: Don’t rely on PR-triggered preview for drift detection. Set up a scheduled job (daily or hourly) that runs pulumi refresh + pulumi preview and alerts on drift. For critical resources (databases, production instances), require manual review; for non-critical resources (tags, descriptions), auto-remediate.

Architecture Trade-Off Analysis

Summary

Migrating from Terraform to Pulumi is not just a syntax change—it’s a paradigm shift. HCL is a declarative DSL; Pulumi is real programming. This brings real benefits (type safety, testability, abstraction) but also real challenges (migration strategy, state management, CI/CD rework).

The five decisions in this article are not independent—they interact:

• Language choice affects state management (Go’s compiled binaries are easier to deploy in CI/CD)
• State management affects drift detection (self-hosted backend gives you more control)
• CI/CD design affects migration strategy (gradual coexistence requires more pipeline complexity)

If you’re just starting a Pulumi evaluation, my advice: pick one independent resource group, migrate it as a POC, validate the full workflow (write → preview → apply → drift detection), then decide whether to expand. Don’t commit to a full migration until you’ve run Pulumi in production for at least one month.

The 3,000 lines of HCL is a problem, but rewriting it in Pulumi won’t automatically solve it. The real solution is understanding your infrastructure, defining clear abstractions, and establishing robust CI/CD and drift detection processes. Pulumi is a tool that makes these easier—but it doesn’t do them for you.

References & Acknowledgments

The following materials were referenced during the writing of this article. Thanks to the original authors for their contributions:

1. Pulumi vs. Terraform — Pulumi official documentation, detailed feature comparison, real-world case studies, and migration path guidance
2. How Pulumi IaC Works — Pulumi official documentation, introduces the architecture model of language host, deployment engine, and resource provider
3. Token Efficiency vs Cognitive Efficiency: Choosing IaC for AI Agents — Pulumi Blog, provides deployability comparison data for AI code generation between HCL and Pulumi (4/5 vs 0/5)
4. Top 10 IaC Tools for DevOps in 2026 — DEV Community, comprehensive 2026 IaC tools comparison including Terraform, Pulumi, and OpenTofu positioning analysis
5. Drift detection and remediation — Pulumi official documentation, covers scheduled drift detection and automated remediation configuration
6. Pulumi Deep Dive — Programmer Eggplant, engineering perspective on Pulumi engine internals and Go/TypeScript/Python implementations
7. Terraform and Pulumi Division of Labor in DigitalOcean Scenarios — CSDN, analyzes Terraform and Pulumi collaboration strategies from a delivery lifecycle perspective
8. Infrastructure as Code: Terraform, Pulumi, and GitOps — TuFaLianGang, provides analysis of fundamental differences between Pulumi’s execution model and Terraform’s, including Plan non-determinism and code review difficulty
9. Troubleshooting Pulumi in CI/CD — Pulumi official documentation, common failure categories and troubleshooting methods for Pulumi pipelines in CI/CD

Thanks to the Pulumi community and all the engineers who shared their migration experiences on GitHub Discussions and blog posts. This article’s pitfall journals are condensed from real incidents—your openness made this article possible.