Infrastructure Lifecycle and Operations · Topic 5 of 5
Troubleshooting Deployment Issues
Terraform Professional 002: classify errors, enable TF_LOG, fix for_each and cycle errors, handle timeouts, and recover state with state pull and push.
- 5 min
- Four error types: language, state, core and provider.
terraform validateneeds init but does not contact remote state or provider APIs.TF_LOG_COREandTF_LOG_PROVIDERseparate the two log sources.
What it is
Troubleshooting means working out which layer failed, then using the tool made for that layer. Terraform errors fall into four groups: language, state, core and provider.
Exam questions usually show an error or a symptom and ask for the next step. The right answer is the least invasive step that addresses the actual cause.
How it works
The four error types
| Type | Typical cause | First tool | Where to report a bug |
|---|---|---|---|
| Language | Invalid HCL, bad references | terraform fmt, terraform validate | Fix it yourself |
| State | State no longer matches reality | Refresh-only plan, import, replace | Fix it yourself |
| Core | Terraform's own logic, possibly a bug | TF_LOG_CORE=TRACE | Terraform core repository |
| Provider | Authentication, API calls, resource mapping | TF_LOG_PROVIDER=TRACE | The provider's repository |
terraform validate needs an initialized directory, because it checks against provider schemas. It does not contact remote state or provider APIs and does not check variable values, so it can pass on a configuration that later fails during plan or apply.
Logging
export TF_LOG_PROVIDER=TRACE
export TF_LOG_PATH=provider.log
terraform apply
| Variable | Effect |
|---|---|
TF_LOG | Logs to stderr at TRACE, DEBUG, INFO, WARN or ERROR (most to least verbose) |
TF_LOG=JSON | TRACE level logs as JSON; not a stable interface |
TF_LOG_CORE | Terraform core only |
TF_LOG_PROVIDER | Provider plugins only |
TF_LOG_PATH | Appends logs to a file; a level must also be set |
TF_LOG_PATH alone enables nothing, which is a popular trap. Trace logs are very detailed, so review them for sensitive data before sharing them in a bug report.
Tracing a broken configuration
resource "aws_instance" "web" {
for_each = toset(var.names)
ami = var.ami
instance_type = "t3.micro"
}
output "ids" {
value = aws_instance.web.id
}
terraform validate reports a missing resource instance key. With for_each, aws_instance.web is a map of instances, so it has no single id.
The fix is [for i in aws_instance.web : i.id], or aws_instance.web["a"].id for one instance.
Errors worth recognizing
| Error or symptom | Cause | Fix |
|---|---|---|
Cycle: a, b | Two resources refer to each other | Break the loop, such as with separate security group rule resources |
Splat used in for_each | for_each needs a map or set, and a splat returns a list | Use a map, or convert with toset |
each.id is invalid | each has only each.key and each.value | Use each.value |
Invalid for_each argument | Not a map or set of strings, or keys not known until apply | Use a map or toset, and build keys from known values |
does not match configured version constraint | Lock conflicts with a new constraint | terraform init -upgrade |
Error acquiring the state lock | Another run holds the lock, or one crashed | Wait, set -lock-timeout, or force-unlock your own stale lock |
| Newer state version error | State uses a format the older CLI does not support | Upgrade the older Terraform |
Timeouts
Slow operations can fail because a resource hit its time limit, not because anything is wrong with the configuration. Some resource types accept a timeouts block, and the provider docs list which operations it covers.
resource "aws_db_instance" "main" {
# other arguments unchanged
timeouts {
create = "60m"
delete = "2h"
}
}
Repairing state
Prefer tools that show a plan first: -refresh-only, moved, removed and import blocks. Manual repair is the last resort.
terraform state pull > backup.tfstate
terraform state push backup.tfstate
state pull writes the raw state to stdout, which is useful for a backup or for reading with jq. state push overwrites the destination state, after two safety checks.
| Check | Refuses the push when |
|---|---|
| Lineage | The lineage differs, suggesting a different state entirely |
| Serial | The destination serial is higher, suggesting it has newer changes |
-force disables both checks and is not recommended. Reaching for it usually means you are about to lose someone's changes.
How the exam asks about this
- "Terraform reports an authentication failure from the API, and you need more detail." Set TF_LOG_PROVIDER=TRACE with
TF_LOG_PATH. - "TF_LOG_PATH is set, but no log file appears." Set TF_LOG, or another level variable, because the path alone enables nothing.
- "A plan shows drift from a console change, and the change should stay." Run terraform apply -refresh-only, then update the configuration to match.
What to remember for the exam
- Four error types: language, state, core and provider.
terraform validateneeds init but does not contact remote state or provider APIs.TF_LOG_COREandTF_LOG_PROVIDERseparate the two log sources.eachexposes onlyeach.keyandeach.value.for_eachneeds a map or set of strings with keys known at plan.state pushchecks lineage and serial unless-forceis used.
Do not edit state as the first response to an error. A failed plan or apply usually points to configuration, credentials or an API problem, and a hand-edited state adds a second problem.
Common mistakes
Treating a passing validate as proof. Validate never calls provider APIs, so permission errors and quota limits appear only at plan or apply.
Running the deprecated refresh command for drift. terraform apply -refresh-only does the same work and lets you review it first.
Force-unlocking on sight. A lock error often means a run is active right now. Confirm that it is yours and that it has stopped before unlocking.
Check yourself
Why does aws_instance.web.id fail when the resource uses for_each?
The resource is a map of instances, so it needs a key or a
forexpression to reach an instance'sid.Which variable enables logging for provider plugins only?
TF_LOG_PROVIDER, set to a level such asTRACE.TF_LOG_PATHcan then send those logs to a file.What stops terraform state push from overwriting newer remote state by default?
The serial check refuses the push when the destination serial is higher. The lineage check also refuses a state that looks entirely different.
Topic 5 of 5 in Infrastructure Lifecycle and Operations
Test yourself on Infrastructure Lifecycle and Operations
The full HashiCorp Terraform Infrastructure Engineer Professional question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by HashiCorp. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.