Usage Guide¶

This guide covers common operations for managing Terraform state files in the PostgreSQL backend.

Getting the Connection String¶

Retrieve the database password from Kubernetes:

kubectl get secret terraform-state-db-credentials -n terraform-state \
  -o jsonpath='{.data.password}' | base64 -d

Build the full connection string:

postgres://terraform_backend:PASSWORD@terraform-state-db-rw.terraform-state.svc.cluster.local:5432/terraform_state?sslmode=disable

Configuring Terraform Backend¶

Method 1: Environment Variable (Recommended for CI/CD)¶

In your main.tf:

terraform {
  backend "pg" {}
}

Initialize with backend config:

terraform init \
  -backend-config="conn_str=$TF_STATE_CONN_STR"

Method 2: Gitea Actions¶

Create .gitea/workflows/terraform.yml:

name: Terraform

on:
  push:
    branches: [main]

jobs:
  terraform:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2

      - name: Terraform Init
        run: terraform init
        env:
          TF_CLI_ARGS_init: "-backend-config=conn_str=${{ secrets.TF_STATE_CONN_STR }}"

      - name: Terraform Plan
        run: terraform plan

      - name: Terraform Apply
        if: github.ref == 'refs/heads/main'
        run: terraform apply -auto-approve

Creating a New State File¶

Terraform automatically creates state entries when you run terraform apply.

Example: New Terraform Project¶

Create main.tf:

terraform {
  backend "pg" {}
}

provider "hcloud" {
  token = var.hcloud_token
}

resource "hcloud_server" "web" {
  name        = "web-1"
  image       = "ubuntu-22.04"
  server_type = "cx11"
}

Initialize and apply:

export TF_STATE_CONN_STR="postgres://terraform_backend:PASSWORD@..."
terraform init -backend-config="conn_str=$TF_STATE_CONN_STR"
terraform apply

Verify state was created:

kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -c "SELECT name FROM states;"

Output:

   name
----------
 default

Using Workspaces¶

Each workspace creates a separate state entry:

# Create workspace
terraform workspace new production

# Apply configuration
terraform apply

# List all workspaces
terraform workspace list

View all states:

kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -c "SELECT name FROM states;"

Output:

    name
------------
 default
 production

Viewing State Files¶

List All States¶

kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -c \
  "SELECT id, name, length(data) as size_bytes FROM states;"

Example output:

 id |    name    | size_bytes
----+------------+------------
  1 | default    |      12543
  2 | production |      15782

View State Content¶

kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -t -c \
  "SELECT data FROM states WHERE name='default';" | jq .

Export State to File¶

kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -t -c \
  "SELECT data FROM states WHERE name='production';" \
  > terraform.tfstate

# Pretty print
jq . terraform.tfstate

Deleting State Files¶

Data Loss Warning

Deleting a state file removes Terraform's knowledge of managed resources. Resources will still exist in the cloud but Terraform won't manage them anymore. Only delete when absolutely sure!

Method 1: Using Terraform (Recommended)¶

Always destroy resources first:

# Destroy all resources
terraform destroy

# Switch to default workspace
terraform workspace select default

# Delete the workspace (removes state)
terraform workspace delete production

Method 2: Direct Database Deletion¶

For emergency cleanup or orphaned states:

# List all states first
kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -c "SELECT name FROM states;"

# Delete specific state
kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -c \
  "DELETE FROM states WHERE name='old-workspace';"

# Verify deletion
kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -c "SELECT name FROM states;"

Method 3: Delete All States (Nuclear Option)¶

kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -c "TRUNCATE TABLE states;"

Unlocking State Files¶

State locking prevents concurrent modifications. Locks can get stuck when:

Terraform crashes mid-operation
Network interruption during apply
Manual termination of Terraform process

Symptoms of a Locked State¶

Error message when running Terraform:

Error: Error acquiring the state lock

Error message: pq: could not serialize access due to concurrent update
Lock Info:
  ID:        abc123-def456
  Path:      production
  Operation: OperationTypeApply
  Who:       user@hostname
  Version:   1.5.0
  Created:   2024-01-24 10:30:00 UTC

Unlock Using Terraform (Recommended)¶

terraform force-unlock <LOCK_ID>

Example:

terraform force-unlock abc123-def456

Terraform will ask for confirmation:

Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.

  Enter a value: yes

Automatic Unlock¶

Locks have TTLs and expire automatically:

Apply locks: ~30 minutes
Plan locks: ~15 minutes

Wait and try again later.

Prevent Lock Issues¶

Set Lock Timeout¶

terraform apply -lock-timeout=10m

This makes Terraform wait up to 10 minutes for a lock to be released.

In Gitea Actions¶

- name: Terraform Apply
  run: terraform apply -auto-approve -lock-timeout=15m
  timeout-minutes: 30  # Prevents indefinite execution

State Migration¶

Migrating FROM Local State¶

# Current setup uses local backend
terraform init

# Add PostgreSQL backend to main.tf
cat >> main.tf <<EOF
terraform {
  backend "pg" {}
}
EOF

# Migrate state to PostgreSQL
terraform init \
  -backend-config="conn_str=$TF_STATE_CONN_STR" \
  -migrate-state

Terraform prompts:

Do you want to copy existing state to the new backend?

  Enter a value: yes

Migrating TO Local State¶

# Remove backend block from main.tf

# Migrate back to local
terraform init -migrate-state

# State downloaded to terraform.tfstate

Backup and Restore¶

Manual Backup¶

Single State¶

kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -t -c \
  "SELECT data FROM states WHERE name='production';" \
  > backup-production-$(date +%Y%m%d).json

All States¶

kubectl exec -n terraform-state terraform-state-db-1 -- \
  pg_dump -U postgres -d terraform_state -t states --data-only \
  > backup-all-states-$(date +%Y%m%d).sql

Restore State¶

Single State¶

STATE_DATA=$(cat backup-production-20240124.json)

kubectl exec -i -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state <<EOF
INSERT INTO states (name, data)
VALUES ('production', '$STATE_DATA')
ON CONFLICT (name) DO UPDATE SET data = EXCLUDED.data;
EOF

All States¶

kubectl exec -i -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state < backup-all-states-20240124.sql

Monitoring State Usage¶

Database Size¶

kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -c "
    SELECT
      pg_size_pretty(pg_database_size('terraform_state')) as db_size,
      pg_size_pretty(pg_total_relation_size('states')) as table_size;
  "

Count States¶

kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -c \
  "SELECT COUNT(*) as total_states FROM states;"

Find Large States¶

kubectl exec -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state -c "
    SELECT
      name,
      pg_size_pretty(length(data)) as size
    FROM states
    ORDER BY length(data) DESC
    LIMIT 10;
  "

Troubleshooting¶

"no pg_hba.conf entry" Error¶

Check password:

kubectl get secret terraform-state-db-credentials -n terraform-state \
  -o jsonpath='{.data.password}' | base64 -d

"connection refused" Error¶

Test connectivity:

kubectl run -it --rm debug --image=postgres:15 --restart=Never -- \
  pg_isready -h terraform-state-db-rw.terraform-state.svc.cluster.local -p 5432

State Corruption¶

Restore from backup:

kubectl exec -i -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state <<EOF
DELETE FROM states WHERE name='corrupted-state';
INSERT INTO states (name, data) VALUES ('corrupted-state', '$(cat backup.json)');
EOF

Best Practices¶

1. Use Workspaces for Environments¶

terraform workspace new development
terraform workspace new staging
terraform workspace new production

2. Never Hardcode Connection Strings¶

Always use secrets in CI/CD:

env:
  TF_CLI_ARGS_init: "-backend-config=conn_str=${{ secrets.TF_STATE_CONN_STR }}"

3. Regular Backups¶

Back up before major changes:

kubectl exec -n terraform-state terraform-state-db-1 -- \
  pg_dump -U postgres -d terraform_state > backup-$(date +%Y%m%d).sql

4. Descriptive Workspace Names¶

terraform workspace new hetzner-prod-cluster
terraform workspace new hetzner-dev-cluster

5. Set Timeouts¶

terraform apply -lock-timeout=15m

6. Clean Up Old Workspaces¶

terraform workspace list
terraform workspace delete unused-workspace

Quick Reference¶

Database Connection¶

# Interactive psql
kubectl exec -it -n terraform-state terraform-state-db-1 -- \
  psql -U postgres -d terraform_state

# List states
SELECT * FROM states;

# Database size
SELECT pg_size_pretty(pg_database_size('terraform_state'));

Common Queries¶

-- List all states
SELECT name FROM states;

-- Count states
SELECT COUNT(*) FROM states;

-- Find large states
SELECT name, pg_size_pretty(length(data)) as size
FROM states
ORDER BY length(data) DESC;

-- Delete old state
DELETE FROM states WHERE name='old-workspace';