Infrastructure Lifecycle and Operations · Topic 5 of 5

Troubleshooting Deployment Issues

Terraform Professional 002: classify errors, enable TF_LOG, fix for_each and cycle errors, handle timeouts, and recover state with state pull and push.

  • 5 min
In a nutshell
  • Four error types: language, state, core and provider.
  • terraform validate needs init but does not contact remote state or provider APIs.
  • TF_LOG_CORE and TF_LOG_PROVIDER separate the two log sources.

What it is

Troubleshooting means working out which layer failed, then using the tool made for that layer. Terraform errors fall into four groups: language, state, core and provider.

Exam questions usually show an error or a symptom and ask for the next step. The right answer is the least invasive step that addresses the actual cause.

SymptomLanguageterraform fmt, terraform validateStaterefresh only planCore or providerTF_LOG traceFix and plan again

How it works

The four error types

TypeTypical causeFirst toolWhere to report a bug
LanguageInvalid HCL, bad referencesterraform fmt, terraform validateFix it yourself
StateState no longer matches realityRefresh-only plan, import, replaceFix it yourself
CoreTerraform's own logic, possibly a bugTF_LOG_CORE=TRACETerraform core repository
ProviderAuthentication, API calls, resource mappingTF_LOG_PROVIDER=TRACEThe provider's repository

terraform validate needs an initialized directory, because it checks against provider schemas. It does not contact remote state or provider APIs and does not check variable values, so it can pass on a configuration that later fails during plan or apply.

Logging

export TF_LOG_PROVIDER=TRACE
export TF_LOG_PATH=provider.log
terraform apply
VariableEffect
TF_LOGLogs to stderr at TRACE, DEBUG, INFO, WARN or ERROR (most to least verbose)
TF_LOG=JSONTRACE level logs as JSON; not a stable interface
TF_LOG_CORETerraform core only
TF_LOG_PROVIDERProvider plugins only
TF_LOG_PATHAppends logs to a file; a level must also be set

TF_LOG_PATH alone enables nothing, which is a popular trap. Trace logs are very detailed, so review them for sensitive data before sharing them in a bug report.

Tracing a broken configuration

resource "aws_instance" "web" {
  for_each      = toset(var.names)
  ami           = var.ami
  instance_type = "t3.micro"
}

output "ids" {
  value = aws_instance.web.id
}

terraform validate reports a missing resource instance key. With for_each, aws_instance.web is a map of instances, so it has no single id.

The fix is [for i in aws_instance.web : i.id], or aws_instance.web["a"].id for one instance.

Errors worth recognizing

Error or symptomCauseFix
Cycle: a, bTwo resources refer to each otherBreak the loop, such as with separate security group rule resources
Splat used in for_eachfor_each needs a map or set, and a splat returns a listUse a map, or convert with toset
each.id is invalideach has only each.key and each.valueUse each.value
Invalid for_each argumentNot a map or set of strings, or keys not known until applyUse a map or toset, and build keys from known values
does not match configured version constraintLock conflicts with a new constraintterraform init -upgrade
Error acquiring the state lockAnother run holds the lock, or one crashedWait, set -lock-timeout, or force-unlock your own stale lock
Newer state version errorState uses a format the older CLI does not supportUpgrade the older Terraform

Timeouts

Slow operations can fail because a resource hit its time limit, not because anything is wrong with the configuration. Some resource types accept a timeouts block, and the provider docs list which operations it covers.

resource "aws_db_instance" "main" {
  # other arguments unchanged
  timeouts {
    create = "60m"
    delete = "2h"
  }
}

Repairing state

Prefer tools that show a plan first: -refresh-only, moved, removed and import blocks. Manual repair is the last resort.

terraform state pull > backup.tfstate
terraform state push backup.tfstate

state pull writes the raw state to stdout, which is useful for a backup or for reading with jq. state push overwrites the destination state, after two safety checks.

CheckRefuses the push when
LineageThe lineage differs, suggesting a different state entirely
SerialThe destination serial is higher, suggesting it has newer changes

-force disables both checks and is not recommended. Reaching for it usually means you are about to lose someone's changes.

How the exam asks about this

  • "Terraform reports an authentication failure from the API, and you need more detail." Set TF_LOG_PROVIDER=TRACE with TF_LOG_PATH.
  • "TF_LOG_PATH is set, but no log file appears." Set TF_LOG, or another level variable, because the path alone enables nothing.
  • "A plan shows drift from a console change, and the change should stay." Run terraform apply -refresh-only, then update the configuration to match.

What to remember for the exam

  • Four error types: language, state, core and provider.
  • terraform validate needs init but does not contact remote state or provider APIs.
  • TF_LOG_CORE and TF_LOG_PROVIDER separate the two log sources.
  • each exposes only each.key and each.value.
  • for_each needs a map or set of strings with keys known at plan.
  • state push checks lineage and serial unless -force is used.
Watch out

Do not edit state as the first response to an error. A failed plan or apply usually points to configuration, credentials or an API problem, and a hand-edited state adds a second problem.

Common mistakes

Treating a passing validate as proof. Validate never calls provider APIs, so permission errors and quota limits appear only at plan or apply.

Running the deprecated refresh command for drift. terraform apply -refresh-only does the same work and lets you review it first.

Force-unlocking on sight. A lock error often means a run is active right now. Confirm that it is yours and that it has stopped before unlocking.

Check yourself

  • Why does aws_instance.web.id fail when the resource uses for_each?

    The resource is a map of instances, so it needs a key or a for expression to reach an instance's id.

  • Which variable enables logging for provider plugins only?

    TF_LOG_PROVIDER, set to a level such as TRACE. TF_LOG_PATH can then send those logs to a file.

  • What stops terraform state push from overwriting newer remote state by default?

    The serial check refuses the push when the destination serial is higher. The lineage check also refuses a state that looks entirely different.

Topic 5 of 5 in Infrastructure Lifecycle and Operations

Test yourself on Infrastructure Lifecycle and Operations

The full HashiCorp Terraform Infrastructure Engineer Professional question bank has 600 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by HashiCorp. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.