Terraform with Kubernetes: A Production-Ready Guide

Most guides about Terraform with Kubernetes start from the wrong assumption: one Terraform pipeline should provision the cluster and deploy every workload inside it. That approach looks tidy in a diagram, but it creates an ownership conflict in production. Terraform records and applies a snapshot of infrastructure, while Kubernetes continuously reconciles resources through controllers, operators, rollouts, and autoscalers.

The practical answer is a boundary, not a bigger module. Terraform should own long-lived infrastructure and carefully selected bootstrap components. Kubernetes-native controllers and GitOps should own resources that change frequently. That division gives platform teams repeatable provisioning without turning every application rollout into an infrastructure state event.

Why Terraform With Kubernetes Is a Boundary Problem

Terraform and Kubernetes solve different operational problems. Terraform is effective when a team needs deterministic ordering, an auditable plan, and controlled changes to VPCs, IAM, node pools, cluster configuration, and cloud services. Kubernetes is effective when controllers must continuously bring live resources back to a declared state.

The conflict appears when Terraform owns fast-moving workloads. A Deployment can change because of a rollout, an operator, a horizontal scaler, or an emergency intervention. Terraform may then see differences between its state and the cluster, and a later plan can propose changes that are technically consistent with Terraform but wrong for the workload's current lifecycle.

Kubernetes had already reached broad operational adoption when its formal integration path with Terraform matured. The official HashiCorp Kubernetes tutorial describes Terraform as a way to provision and manage clusters across AWS, Azure, and GCP, while Kubernetes guidance identified Terraform's plan and apply lifecycle as a promising automation pattern. CNCF reported that 58% of survey respondents were running Kubernetes in production in 2018, while 66% of potential or actual consumers were using it in production by 2023 in its annual survey reporting. The infrastructure layer Terraform helps create is no longer experimental.

A boundary that survives incidents

A useful ownership rule is simple:

• Terraform owns: networks, clusters, node pools, cloud IAM, cluster access, storage integrations, and slow-changing add-ons.
• GitOps owns: Deployments, StatefulSets, Services, Ingress objects, application ConfigMaps, Helm values, and namespace-level application policy.
• Controllers own: resources they mutate or reconcile, such as certificates, external secrets, autoscaling decisions, and operator-managed custom resources.

The boundary fails when two tools write the same resource. It also fails when Terraform manages a dependency before the cluster API or admission webhooks are available. A platform team should ask four questions before assigning ownership:

1. Does this resource exist before the cluster is usable?
2. Does a controller change it after creation?
3. Does it need continuous reconciliation?
4. Would a change require infrastructure-level approval?

If the answer is yes to the first or fourth question, Terraform is usually appropriate. If the answer is yes to the second or third, Kubernetes-native reconciliation is usually safer. Teams designing an internal developer platform should make these rules visible in their internal developer platform guidance, rather than hiding them inside undocumented pipeline behavior.

Practical rule: One resource should have one authoritative writer. When ownership must be shared at the field level — for example, GitOps owns a Deployment's container spec while an HPA controller owns spec.replicas — the GitOps manifests should omit or explicitly ignore the controller-managed fields to avoid fighting the autoscaler.

Provisioning EKS, GKE, and AKS Clusters

Cluster provisioning is where Terraform earns its place. The cloud provider modules give teams a consistent interface for networks, control planes, node pools, identity, and access, while still exposing provider-specific behavior that operators need to understand.

EKS with managed identity and access

For Amazon EKS, the terraform-aws-modules/eks/aws module is a practical foundation. Configure managed node groups, enable the cluster OIDC provider for IAM Roles for Service Accounts, and use EKS access entries where your access model supports them. Pin the Kubernetes version instead of inheriting an implicit default, then plan upgrades around the EKS support window.

EKS changes can take time because the control plane, node groups, networking components, and add-ons follow separate update paths. Keep cluster creation and Kubernetes API resources in separate apply stages. The Kubernetes provider needs a reachable API during planning, so a single apply that both creates an unavailable cluster and populates it with manifests is brittle.

GKE and workload identity

The terraform-google-modules/kubernetes-engine/google module provides a structured starting point for Google Kubernetes Engine. Use a dedicated VPC and subnet design, choose private nodes when the network model requires it, and create Workload Identity bindings as part of the platform layer.

Autopilot and Standard clusters expose different infrastructure surfaces. Autopilot reduces node management but changes which settings platform engineers control. Standard clusters provide more node-level control and therefore create more Terraform responsibilities. Decide this before writing shared modules, because RBAC, add-ons, scheduling assumptions, and identity bindings can differ substantially.

AKS and Entra integration

For Azure Kubernetes Service, use the azurerm_kubernetes_cluster resource with Microsoft Entra integration and a deliberate network plugin choice. New platform designs should evaluate Entra workload identity rather than carrying forward older managed service identity patterns without review. Azure CNI and kubenet also impose different networking and address-management considerations, so the module should expose that choice explicitly.

Install cert-manager, cloud load balancer controllers, and similar cluster add-ons through Helm or the platform's GitOps bootstrap after the API is available. Terraform can create the identity and permissions those controllers need, but it shouldn't become the long-term owner of controller-mutated objects.

The managed Kubernetes services overview is useful when comparing the operational responsibilities that remain after the cloud provider manages the control plane.

Export kubeconfig data carefully, and avoid placing credentials in logs or broadly accessible CI outputs. Remote state should live in the provider environment with appropriate locking and access controls, but the state layer itself must remain independent from workload deployment state.

Managing Kubernetes Resources and Helm Charts

The Kubernetes and Helm providers are useful at the edge of the boundary. They can install a small set of stable platform components after the cluster exists, but they aren't a substitute for a continuously reconciling deployment system.

The Helm provider represents a chart as a Terraform resource. A release normally specifies a repository, chart, version, and values supplied through files, variables, or set blocks. That makes chart upgrades visible in a Terraform plan and gives infrastructure reviewers an audit trail for platform components such as ingress controllers or observability agents.

A controlled provider pattern

Pin provider and chart versions. Put environment-specific values in a clear variable structure instead of scattering overrides across long set blocks. A release that depends on a namespace should express that dependency with depends_on, or Terraform can attempt operations in an order that the API rejects.

The Kubernetes provider can manage typed resources and manifests. Use it for namespaces, service accounts, stable RBAC objects, and foundational resources that rarely change. Where a controller owns selected fields, use lifecycle settings such as ignore_changes carefully. Ignoring an entire resource can hide a meaningful configuration error, so the ignored field should correspond to a documented controller responsibility.

For monitoring foundations, a pinned Prometheus chart can be installed through the same pattern, as shown in this Prometheus Helm chart example. The important decision isn't whether Terraform can install the chart. It can. The decision is whether Terraform should own every subsequent value change and reconciliation event.

Where the model stops working

Terraform can't watch a CRD in the same way a Kubernetes operator can. It doesn't continuously react to admission webhook health, controller status, or application-level rollout conditions. A chart that creates CRDs and immediately depends on them can also expose ordering problems during initial installation and upgrades.

Use Terraform for a stable bootstrap layer when the release has a clear platform owner and a controlled change rate. Move application charts, environment overlays, and frequently updated manifests to ArgoCD or Flux. That keeps Terraform plans focused and lets Kubernetes controllers manage the lifecycle they were designed to manage.

State Management, Secrets, and Drift Control

Terraform state is not bookkeeping. It is an operational control plane containing resource identities, dependencies, and sometimes sensitive values. A local state file may be acceptable for a personal experiment, but shared cluster infrastructure needs a remote backend, locking, encryption at rest, versioning, and tightly scoped access.

Use the backend native to the cloud environment where possible. S3 with DynamoDB locking, Google Cloud Storage, and Azure Storage with blob leases are common patterns. Separate state by ownership and blast radius. A cluster foundation state should not contain every application object in the fleet, and a multi-cluster setup should make it possible to recover one environment without rewriting unrelated environments.

Workspaces can help when the configuration is the same and only inputs differ. Directory or stack separation is often easier to reason about when regions, cloud accounts, trust boundaries, or lifecycle policies diverge. Large inline Kubernetes manifests can make state and plans harder to inspect, which is another reason to keep rapidly changing workloads outside Terraform.

Secrets require an explicit decision

Secrets can enter Terraform through variables, data sources, provider responses, or resource arguments. Even when a secret isn't printed in a plan, it may still be present in state. Choose a secret management approach based on who needs to retrieve the secret and where rotation belongs. These tools complement your Terraform remote state backend — they handle secret lifecycle, not state storage.

Use terraform plan on a schedule to detect infrastructure drift, then compare the result with kubectl diff or the GitOps controller's status for in-cluster resources. A reported difference isn't automatically a Terraform problem. First identify the authoritative writer, then either import the legitimate change, update the declared configuration, or remove the unauthorized mutation.

If state is corrupted or stolen, stop applies, revoke credentials that could access the backend, preserve the version history, and inspect audit logs. Restore a known-good state version only after confirming the configuration and real infrastructure align. A versioned bucket policy, MFA delete where supported, and a rebuild-from-import procedure provide recovery options when restoration isn't trustworthy. The Terraform challenges discussion reinforces why remote state, locking, and concurrent-write protection belong in the initial design.

Terragrunt and OpenTofu for Multi-Cluster Setups

One cluster per cloud rarely needs an orchestration wrapper. A well-structured Terraform repository can provide clear modules, provider configuration, backend settings, and environment variables without adding another interpretation layer. The complexity changes when the same platform pattern spans regions, accounts, subscriptions, projects, and several cluster environments.

Terragrunt reduces repetition through shared configuration and include blocks. A parent configuration can propagate backend settings, provider assumptions, IAM conventions, and module source definitions into regional or environment-specific stacks. That works well for shared VPC modules and repeated EKS, GKE, or AKS foundations.

The cost is indirection. Engineers troubleshooting a failed plan may need to read generated inputs, inherited settings, and dependency declarations before they understand the effective Terraform configuration. Hidden ordering bugs can also appear when dependencies are represented in wrapper configuration rather than in the module graph.

Choosing the execution model

OpenTofu is a separate implementation shaped by licensing and governance considerations. It remains compatible with many Terraform configurations and commonly used providers, including Kubernetes and Helm providers, but compatibility should be tested against the exact provider and module versions in your repository. Migration from Terraform 1.5.x should be treated as a controlled state operation, with backups, a test workspace or environment, and a review of provider lock behavior before production adoption. For a straightforward binary cutover with the same backend, tofu init is sufficient. Reserve tofu init -migrate-state for when you are deliberately changing the backend configuration at the same time — using it unnecessarily can trigger an unintended state migration.

A practical recommendation is to start with vanilla Terraform, introduce Terragrunt when repeated environment scaffolding becomes a maintenance burden, and select OpenTofu when its governance or supply-chain properties are a deciding requirement. None of these tools solves workload reconciliation. They only change how the infrastructure layer is organized and executed.

GitOps and CI/CD Integration Patterns

The cleanest handoff starts before the cluster exists. Terraform creates the cluster, node pools, IAM or workload identity prerequisites, access controls, and the minimum bootstrap needed to install a GitOps controller. ArgoCD or Flux then becomes the owner of application manifests, Kustomize overlays, Helm values, and continuous reconciliation.

A production pipeline can follow this sequence:

1. A pull request changes Terraform modules or environment inputs.
2. CI runs formatting, validation, static analysis, policy checks, and terraform plan against the remote backend.
3. The pipeline posts a sanitized plan summary to the pull request. Raw plan artifacts (binary or JSON) can still contain sensitive values, so they must remain in encrypted, access-controlled storage with limited retention — never uploaded to the PR directly.
4. Reviewers approve the proposed infrastructure change.
5. A protected branch triggers terraform apply against the saved plan artifact (-out from step 2), ensuring reviewers approved exactly what gets applied. If state or configuration changed between approval and apply, the saved plan is invalidated and the pipeline must re-plan and re-approve.
6. Terraform hands application delivery to the GitOps repository or controller.

The bootstrap question needs an explicit answer. A small Terraform-managed Helm release can install ArgoCD or Flux, or a separate trusted bootstrap job can do it once. After that handoff, the controller should manage its own applications and avoid competing with Terraform over the same manifests.

Sequencing identity and operators

Identity prerequisites must exist before dependent operators deploy. EKS workloads may require the OIDC provider and IRSA role first. GKE workloads may require Workload Identity bindings. AKS deployments may need Entra workload identity configuration. Model these dependencies in Terraform, then let GitOps deploy the operator after the cluster-side prerequisites are available.

ArgoCD ApplicationSets are generally a better fit than Terraform loops for fleet-wide application rollout. ApplicationSets can generate applications from cluster labels, repositories, or environment metadata while preserving the GitOps controller's reconciliation model. Terraform loops are more suitable for stable infrastructure objects whose lifecycle belongs to the platform state.

Keep infrastructure and applications in separate repository folders or repositories. Run OPA or Conftest policy gates before merge, and make ownership visible in code review. A short handoff document should state which team can change each layer, how an emergency override is recorded, and how control returns to the declared source of truth.

The embedded walkthrough below illustrates the handoff concept in a practical format.

Production Checklist and Common Failure Modes

A healthy Terraform-managed Kubernetes platform makes ordinary changes boring. Engineers can explain who owns each resource, preview changes before applying them, recover state without improvising, and see whether a rollout failed because of infrastructure, identity, or application reconciliation.

Start with mandatory pre-merge controls:

• Format and syntax: Run terraform fmt and terraform validate on every relevant module.
• Static analysis: Use tflint, tfsec, or Checkov to catch provider and security problems before apply.
• Policy gates: Apply OPA or Conftest rules for encryption, network exposure, identity scope, and approved registries.
• Plan review: Require two engineers to review meaningful infrastructure diffs, especially replacements and access changes.
• Ownership checks: Reject a change that gives Terraform control over a resource already reconciled by ArgoCD, Flux, or an operator.

The failure modes are predictable. Privilege escalation often begins with a broadly scoped IAM role or Kubernetes ClusterRole, so review bindings and monitor audit events. Unencrypted or overexposed state can disclose infrastructure relationships and secret values, so enforce backend policies and restrict state readers. Provider version drift can change behavior between runners, so commit lock files and pin module and provider versions.

Helm value drift deserves separate treatment. If Terraform installs a release while ArgoCD manages its values, the same chart has two desired-state systems. Choose one owner and remove the other configuration. Kubeconfig leakage requires redacted CI logs, short-lived credentials, and controlled artifact retention. Ephemeral clusters need an explicit destroy path, owner, expiration signal, and budget guardrail, otherwise abandoned infrastructure becomes a silent operational liability.

DORA metrics become useful when tied to these workflows. Deployment frequency reflects how often GitOps can safely reconcile application changes. Lead time includes the time from a reviewed Terraform or manifest change to an applied result. Change failure rate should distinguish infrastructure replacement failures from application rollout failures, while mean time to recovery should measure the actual path from alert to rollback, state repair, or controller reconciliation.

The objective isn't maximal automation. It's repeatable change with a small blast radius, clear ownership, and a recovery path that works under pressure.

CloudCops GmbH helps teams design Terraform-managed Kubernetes platforms across AWS, Azure, and Google Cloud, including remote state, split ownership, GitOps with ArgoCD or FluxCD, and policy-as-code controls. If your cluster pipeline is mixing infrastructure and workload responsibilities, visit CloudCops GmbH to discuss an architecture review or hands-on platform engineering engagement.

Continue Reading

Read Terraform Best Practices for Multi-Cloud Teams in 2026

Sep 11, 2026

Terraform Best Practices for Multi-Cloud Teams in 2026

Practical terraform best practices for AWS, Azure, and GCP teams covering state, modules, CI/CD, policy-as-code, drift, and cost controls.

terraform best practices

+4

C

CloudCops

Read Terraform State Files: Your 2026 Management Guide

Jun 1, 2026

Terraform State Files: Your 2026 Management Guide

Master Terraform state files. Learn remote backends, locking, security, CI/CD, and managing large state files with our 2026 enterprise guide.

terraform state files

+4

C

CloudCops

Read 10 Infrastructure as Code Best Practices for 2026

Apr 6, 2026

10 Infrastructure as Code Best Practices for 2026

Master infrastructure as code best practices for 2026. This guide covers IaC testing, GitOps, security, cost control, and more with expert tips and examples.

infrastructure as code best practices

+4

C

CloudCops