terragrunt VS terraformer

name: Configure on: push: branches: - main pull_request: branches: - main workflow_dispatch: inputs: destroy: description: 'Run Terragrunt destroy command' required: true default: 'false' type: choice options: - true - false jobs: apply: if: ${{ !inputs.destroy || inputs.destroy == 'false' }} runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v2 - name: Setup SSH uses: webfactory/[email protected] with: ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }} - name: Setup Terraform uses: hashicorp/setup-terraform@v2 with: terraform_version: 1.5.5 terraform_wrapper: false - name: Setup Terragrunt run: | curl -LO "https://github.com/gruntwork-io/terragrunt/releases/download/v0.48.1/terragrunt_linux_amd64" chmod +x terragrunt_linux_amd64 sudo mv terragrunt_linux_amd64 /usr/local/bin/terragrunt terragrunt -v - name: Apply Terraform changes run: | cd dev terragrunt run-all apply -auto-approve --terragrunt-non-interactive -var AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -var AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -var AWS_REGION=$AWS_DEFAULT_REGION env: AWS_ACCESS_KEY_ID: ${{ vars.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_DEFAULT_REGION: ${{ vars.AWS_DEFAULT_REGION }} destroy: if: ${{ inputs.destroy == 'true' }} runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v2 - name: Setup SSH uses: webfactory/[email protected] with: ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }} - name: Setup Terraform uses: hashicorp/setup-terraform@v2 with: terraform_version: 1.5.5 terraform_wrapper: false - name: Setup Terragrunt run: | curl -LO "https://github.com/gruntwork-io/terragrunt/releases/download/v0.48.1/terragrunt_linux_amd64" chmod +x terragrunt_linux_amd64 sudo mv terragrunt_linux_amd64 /usr/local/bin/terragrunt terragrunt -v - name: Destroy Terraform changes run: | cd dev terragrunt run-all destroy -auto-approve --terragrunt-non-interactive -var AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -var AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -var AWS_REGION=$AWS_DEFAULT_REGION env: AWS_ACCESS_KEY_ID: ${{ vars.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_DEFAULT_REGION: ${{ vars.AWS_DEFAULT_REGION }}

‍Terragrunt is a thin wrapper that provides extra tools for keeping your Terraform configurations DRY (Don't Repeat Yourself), working with multiple Terraform modules, and managing remote state. It's particularly useful in managing large-scale infrastructure deployments with Terraform.

... #************************** Terraform ************************************* ARG TERRAFORM_VERSION=1.7.3 RUN set -ex \ && curl -O https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /usr/local/bin/ RUN set -ex \ && mkdir -p $HOME/.terraform.d/plugin-cache && echo 'plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"' > ~/.terraformrc #************************* Terragrunt ************************************* ARG TERRAGRUNT_VERSION=0.55.1 RUN set -ex \ && wget https://github.com/gruntwork-io/terragrunt/releases/download/v${TERRAGRUNT_VERSION}/terragrunt_linux_amd64 -q \ && mv terragrunt_linux_amd64 /usr/local/bin/terragrunt \ && chmod +x /usr/local/bin/terragrunt #*********************** Terramate **************************************** ARG TERRAMATE_VERSION=0.4.5 RUN set -ex \ && wget https://github.com/mineiros-io/terramate/releases/download/v${TERRAMATE_VERSION}/terramate_${TERRAMATE_VERSION}_linux_x86_64.tar.gz \ && tar -xzf terramate_${TERRAMATE_VERSION}_linux_x86_64.tar.gz \ && mv terramate /usr/local/bin/terramate \ && chmod +x /usr/local/bin/terramate #*********************** tfsec ******************************************** ARG TFSEC_VERSION=1.28.5 RUN set -ex \ && wget https://github.com/aquasecurity/tfsec/releases/download/v${TFSEC_VERSION}/tfsec-linux-amd64 \ && mv tfsec-linux-amd64 /usr/local/bin/tfsec \ && chmod +x /usr/local/bin/tfsec \ && terragrunt --version #**********************Terraform docs ************************************ ARG TERRRAFORM_DOCS_VERSION=0.17.0 RUN set -ex \ && curl -sSLo ./terraform-docs.tar.gz https://terraform-docs.io/dl/v${TERRRAFORM_DOCS_VERSION}/terraform-docs-v${TERRRAFORM_DOCS_VERSION}-$(uname)-amd64.tar.gz \ && tar -xzf terraform-docs.tar.gz \ && chmod +x terraform-docs \ && mv terraform-docs /usr/local/bin/terraform-docs #********************* ShellCheck ***************************************** ARG SHELLCHECK_VERSION="stable" RUN set -ex \ && wget -qO- "https://github.com/koalaman/shellcheck/releases/download/${SHELLCHECK_VERSION?}/shellcheck-${SHELLCHECK_VERSION?}.linux.x86_64.tar.xz" | tar -xJv \ && cp "shellcheck-${SHELLCHECK_VERSION}/shellcheck" /usr/bin/ \ && shellcheck --version ...

Our first attempt was to introduce other engineering teams to Terraform - the Platform team was already using it extensively with Terragrunt, and using Atlantis to automate plan and apply operations in a Git flow to ensure infrastructure was consistent. We'd written modules, with documentation, and an engineer would simply need to raise a PR to use the module and provide the right values, and Atlantis (once the PR was approved by Platform) would go ahead and set it up for them.

Terragrunt: An extension of Terraform, Terragrunt assists in managing complex infrastructure with less duplication and more efficiency. Its power lies in its ability to manage dependencies and its dry configuration approach.

Copy the link and download on your terminal using the wget command. Example: wget https://github.com/gruntwork-io/terragrunt/releases/download/v0.54.19/terragrunt_linux_amd64

name: Configure on: push: branches: - main pull_request: branches: - main jobs: terraform: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v2 - name: Setup SSH uses: webfactory/[email protected] with: ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }} - name: Setup Terraform uses: hashicorp/setup-terraform@v2 with: terraform_version: 1.5.5 terraform_wrapper: false - name: Setup Terragrunt run: | curl -LO "https://github.com/gruntwork-io/terragrunt/releases/download/v0.48.1/terragrunt_linux_amd64" chmod +x terragrunt_linux_amd64 sudo mv terragrunt_linux_amd64 /usr/local/bin/terragrunt terragrunt -v - name: Apply Terraform changes run: | cd dev terragrunt run-all apply -auto-approve --terragrunt-non-interactive -var AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -var AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -var AWS_REGION=$AWS_DEFAULT_REGION cd apache-server/ec2-web-server public_ip=$(terragrunt output instance_public_ip) echo "$public_ip" > public_ip.txt cat public_ip.txt env: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} - name: Upload artifact uses: actions/upload-artifact@v4 with: name: ip-artifact path: dev/apache-server/ec2-web-server/public_ip.txt ansible: runs-on: ubuntu-latest needs: terraform steps: - name: Download artifact uses: actions/download-artifact@v4 with: name: ip-artifact - name: Configure Ansible run: | sudo apt update sudo pipx inject ansible-core jmespath ansible-playbook --version sudo echo "[web]" >> ansible_hosts sudo cat public_ip.txt >> ansible_hosts mv ansible_hosts $HOME sudo cat $HOME/ansible_hosts - name: Configure playbook run: | cd $HOME cat > deploy.yml < Test Page This is a test page EOF cat $HOME/deploy.yml - name: Run playbook uses: dawidd6/action-ansible-playbook@v2 with: playbook: deploy.yml directory: /home/runner key: ${{secrets.SSH_PRIVATE_KEY}} options: | --inventory ansible_hosts --verbose

Created and maintained by Gruntwork, Terragrunt is a tool designed to enhance Terraform’s capabilities. It acts as a thin wrapper around Terraform, offering additional features to streamline and optimise Terraform usage. Key functions of Terragrunt include helping users keep their Terraform configurations DRY (Don’t Repeat Yourself), efficiently managing multiple Terraform modules, and handling remote state management. By reducing repetition in Terraform code and simplifying the management of complex module dependencies and remote state, Terragrunt makes working with Terraform more efficient, especially for larger or more complex infrastructure deployments.

locals { # Automatically load region-level variables region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl")) # Automatically load environment-level variables` environment_vars = read_terragrunt_config(find_in_parent_folders("env.hcl")) # Extract the variables we need for easy access account_name = local.environment_vars.locals.account_name account_id = local.environment_vars.locals.aws_account_id aws_region = local.region_vars.locals.aws_region # This is the S3 bucket where the Terraform State Files will be stored remote_state_bucket = "devops-bucket" # This is the DynamoDB table where Terraform will add the locking status dynamodb_table = "terraform-state-lock" # IAM Role for Terraform backend to assume terraform_backend_role = "arn:aws:iam::{shared-services_account_id}:role/terraform-backend-role" environment_path = replace(path_relative_to_include(), "environments/", "") # https://github.com/hashicorp/terraform/releases terraform_version = "latest" # https://github.com/gruntwork-io/terragrunt/releases terragrunt_version = "latest" } # Generate an AWS provider block generate "provider" { path = "provider.tf" if_exists = "overwrite_terragrunt" contents = <

If your team works with Terraform, you should definitely try Terragrunt (and obviously, its CLI tool!). It's an Infrastructure as Code (IaC) tool that acts as a wrapper for Terraform and simplifies dealing with multiple Terraform modules in different environments.

‍Terraformer is a CLI tool developed by Google that generates Terraform files from existing infrastructure (reverse Terraform), simplifying the process of adopting Terraform in existing environments and speeding up the initial setup process. Terraformer supports multiple cloud providers, including AWS, Google Cloud, Azure, and others.

Looking into efficient ways to import existing infrastructure. Using Terraformer to get the everything into Terraform and then refactoring into modules, for_each, etc. using moved blocks seems like it would be a good approach. Refactoring them to use existing modules from providers will take a little work and likely going back and forth with terraform plan, and assuming many things won't translate straight across because of what resources a module might be creating and how they are creating them.

We had thought about the ability to convert HCL to Go, but not much thought in querying existing infra to generate HCL (like terraformer). If you want to experiment building on top of Lingon then please go ahead! I’d be happy to help provide the context from Lingon.

If "ClickOps" is your starting point, as you mention, you could try creating a new scratch cloud provider account, do all your "ClickOps", then use a TF export tool (e.g., Terraformer) to see the exported TF resources to find all the references to other TF resources in the cloud resource dependency chain.

I haven't worked with any of the ones that purport to work with AWS, but a quick Google shows terraformer and Terracognita as options - maybe look into trying those out?

This maybe a bit more complicated if you're not into coding/terraform but I'd use terraformer to convert all of your infrastructure from implementation to code while in AWS, then switch providers and with a bit of jiggly of the code implement your infra into gcs as IaC and keep it that way if/when you switch again.

Maybe terraformer could help. I never tried it personally . https://github.com/GoogleCloudPlatform/terraformer

I've been wondering if this tool is any good. https://github.com/GoogleCloudPlatform/terraformer/blob/master/docs/aws.md

If you've to write the code (I assumed you just needed to reorganize it without TG) maybe consider https://github.com/GoogleCloudPlatform/terraformer