Terraform Remote State Files

Link CloudTruth external parameter values to Terraform Remote State Files within S3.

One practical use of CloudTruth external values is the ability to automatically reference Terraform state outputs. These outputs are normally generated and stored by Terraform in a file named terraform.tfstate.HashiCorp provides the ability for Terraform state files to be stored remotely. AWS S3 is a common choice for backend remote storage, and is often considered a best practice, especially when working with multiple users that may depend upon Terraform state outputs for CI/CD.

Once CloudTruth external parameters referencing state files have been created, any parameter access will fetch the the value directly from the latest Terraform state on-demand! This includes parameter references from within the Template mechanism that enables you to generate configuration files per-environment as needed.

Remote State with Terraform

You may already have remote state stored in S3 which can be used directly with the CloudTruth AWS S3 Integration to assign state output as external values.

This walkthrough shows you how to setup a basic S3 bucket and deploy an EC2 instance to us-west-2 with a Terraform remote state file backend in S3 and reference the output as an external CloudTruth parameter value.

Deploy an S3 bucket with Terraform

You can use an existing S3 bucket for your remote backend or deploy a new S3 bucket with a unique name.

Create an s3 directory, change into the directory and create a file to define the S3 infrastructure.

mkdir s3
cd s3
touch main.tf

Open main.tf , paste in the configuration below, save the file, and run terraform init and terraform apply.

provider "aws" {
  profile = "default"
  region  = "us-west-2"
}

resource "random_pet" "petname" {
  length    = 1
  separator = "-"
}

resource "aws_s3_bucket" "demo" {
  bucket = "cloudtruth-s3-${random_pet.petname.id}"
  acl    = "private"
  force_destroy = true
}

output "YOUR_UNIQUE_BUCKET_NAME" {
  value = aws_s3_bucket.demo.bucket
}

Note the output name of your newly created S3 bucket to use in the terraform s3 backend configuration.

Deploy configuration to a Terraform S3 backend

Create a remote-state directory, change into the directory and create a file to define the infrastructure.

mkdir remote-state
cd remote-state
touch main.tf

The following Terraform configuration specifies S3 as the backend for your state file. Open main.tf and paste in the configuration below. Provide the bucket with YOUR_UNIQUE_BUCKET_NAME, save the file, then run terraform init and terraform apply.

The EC2 instance will be created in us-west-2. Terraform will place the state file in the specified S3 bucket path demo/instance/terraform.tfstate and provide outputs for the instance id and public ip.

terraform {
  backend "s3" {
    bucket = "YOUR_UNIQUE_BUCKET_NAME"
    key    = "demo/instance/terraform.tfstate"
    region = "us-west-2"
  }
  }

provider "aws" {
  profile = "default"
  region  = "us-west-2"
}

resource "aws_instance" "app_server" {
  ami           = "ami-830c94e3"
  instance_type = "t2.micro"

  tags = {
    Name = "RemoteStateInstance"
  }
}

output "instance_id" {
  description = "ID of the EC2 instance"
  value       = aws_instance.app_server.id
}

output "instance_public_ip" {
  description = "Public IP address of the EC2 instance"
  value       = aws_instance.app_server.public_ip
}

Now that you have a remote state file in S3 and have configured the AWS S3 CloudTruth Integration, you can set your terraform.tfstate outputs as CloudTruth External Values.

Set state as an External Value

As an example, while editing the External value of a parameter navigate to the S3 bucket destination path demo/instance/terraform.tfstate in the S3 integration and use the JMESPATH selector outputs.instance_id.value. Hit save to set the value to the instance id output from the remote state file.

Don't forget to cleanup your deploy infrastructure by executing a terraform destroy in the S3 and remote-state folders.

Remote State with Terragrunt

Terragrunt provides a quick start on creating a DRY remote_state configuration for the backend. You can use Terragrunt generated remote state with the CloudTruth AWS S3 Integration to assign state output as external values.

Following the Terragrunt quick start guide as an example we created the following folder structure which will configure an S3 bucket and deploy an instance with remote state using the deployed bucket.

# terragrunt-remote-state
├── stage
│   ├── instance
│   │   └── main.tf
│   │   └── terragrunt.hcl
│   └── terragrunt.hcl

The backend configuration is defined once in the root stage terragrunt.hcl file. It will create a dynamodb lock table called my-lock-table and an S3 backend. You can update the bucket config with a unique name to follow along.

# stage/terragrunt.hcl
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket = "YOUR_UNIQUE_BUCKET_NAME"

    key = "${path_relative_to_include()}/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "my-lock-table"
  }
}

The instance folder contains a terragrunt.hcl that contains the Terragrunt helper find_in_parent_folders() which will inherit the remote_state configuration from the root terragrunt.hcl file in the directory tree.

# stage/instance/terragrunt.hcl
include {
  path = find_in_parent_folders()
}

The instance folder also contains a main.tf that will configure an EC2 instance in us-east-1 with outputs for the instance_id and public ip.

# stage/instance/main.tf
provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

resource "aws_instance" "app_server" {
  ami           = "ami-0aeeebd8d2ab47354"
  instance_type = "t2.micro"

  tags = {
    Name = "RemoteStateInstance"
  }
}

output "instance_id" {
  description = "ID of the EC2 instance"
  value       = aws_instance.app_server.id
}

output "instance_public_ip" {
  description = "Public IP address of the EC2 instance"
  value       = aws_instance.app_server.public_ip
}

Now you can change directory to the instance folder and run terragrunt apply.

cd stage/instance
terragrunt apply -auto-approve --terragrunt-non-interactive

Terragrunt creates an S3 bucket with your unique name and is a destination for the instance configuration terraform.tfstate file!

Now that you have a remote state file in S3 and have configured the AWS S3 CloudTruth Integration, you can set your terraform.tfstate outputs as CloudTruth External Values.

Set state as an External Value

while editing the external value of a parameter navigate to the Terragrunt created S3 bucket destination path instance/terraform.tfstate in the S3 integration and use the JMESPATH selector outputs.instance_id.value. Hit save to set the value to the instance id output from the remote state file.

Don't forget to cleanup your deploy infrastructure by executing a terragrunt destroy in the instance folder. You will need to manually delete the Terragrunt created S3 bucket as the wrapper does not provide a way to delete generated backends.