dhall-kubernetes VS terragrunt

I hate yaml with a passion. It marginally better than xml for reading (wins huge on comment syntax) and worse for everything else. It makes zero sense we somehow ended up with it as standard configuration serialization format.

Note yaml is not a DSL. It's a tree serialization format! Everything interesting is happening after it is parsed. Extreme examples point to e.g. github actions conditions.

Anyway, back on topic - maybe not prolog for CDK, but still quite interesting: Dhall-kubernetes - https://github.com/dhall-lang/dhall-kubernetes

At this point, if it’s painful enough, why isn’t compiling-to-yml tools more popular?

Example: https://github.com/dhall-lang/dhall-kubernetes

Haven’t used dhall myself but I’d definitely prefer a DSL on top of yaml.

Dhall provides https://github.com/dhall-lang/dhall-kubernetes which is exactly this: statically type-checked kubernetes config generation.

Dhall is my favorite configuration language that I never get around to using.

I manage DNS in Terraform, and since every Terraform provider uses different objects definitions, and every object definition is rather verbose, Dhall would be a way to specify my own DRY types and leave the provider-specific details in one place. Adding new DNS entries and moving several domains between providers would be a matter of changing fewer lines.

Dhall also has Kubernetes bindings:

https://github.com/dhall-lang/dhall-kubernetes

Although I'm tempted to just stick to Helm here, even though it's less type-safe.

Not Helm directly, but does something like Dhall fit your question? https://github.com/dhall-lang/dhall-kubernetes

Have you heard about Dhall? It’s a programming language used for generating configuration files for a variety of purposes. One of them is to replace old and limited formats such as JSON and YAML. It is DRYable, secure, and even suitable for creating K8s manifests. The latter option isn’t something for anyone: you have to learn a new language and deal with its peculiarities, but it might be really helpful when you have tons of YAML configs. I’ve recently made a short intro to Dhall for K8s in this review.

Best thing is Dhall that I am aware of. Same situation, working as a consultant, forced to use broken things.

https://github.com/dhall-lang/dhall-kubernetes

Lets look at a specific example. Take Kubernetes: everything is yaml, with complete schemas, all the way down. From your perspective this is configuration utopia, right? Meanwhile back in reality k8s is the poster child of "yaml hell". From the day it was released, people took one look at it, gave it a giant NOPE and instantly spawned half a dozen templating languages. The most popular of these is helm, which has a terrible, no good, very bad design: full of potential injection attacks from purely textual string substitution, manually specified indentation to embed parameterized blocks, virtually no intermediate validation, no way to validate unused features, etc etc

Compare to dhall which publishes a complete set of dhall-k8s schema mappings which enables you to factor out any design you want down to as few configuration variables as you like, while validating the configuration generators themselves at design time. https://github.com/dhall-lang/dhall-kubernetes#more-modular-...

The solution I like is Dhall. They even have a Kubernetes solution that will catch a lot of issues at compile-time, before you try to apply it to Kubernetes. At earthly we aren't actually using it though. Our Kubernetes guru found it to be a bit slow but I am hopeful it or something like it will be the future.

name: Configure on: push: branches: - main pull_request: branches: - main workflow_dispatch: inputs: destroy: description: 'Run Terragrunt destroy command' required: true default: 'false' type: choice options: - true - false jobs: apply: if: ${{ !inputs.destroy || inputs.destroy == 'false' }} runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v2 - name: Setup SSH uses: webfactory/[email protected] with: ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }} - name: Setup Terraform uses: hashicorp/setup-terraform@v2 with: terraform_version: 1.5.5 terraform_wrapper: false - name: Setup Terragrunt run: | curl -LO "https://github.com/gruntwork-io/terragrunt/releases/download/v0.48.1/terragrunt_linux_amd64" chmod +x terragrunt_linux_amd64 sudo mv terragrunt_linux_amd64 /usr/local/bin/terragrunt terragrunt -v - name: Apply Terraform changes run: | cd dev terragrunt run-all apply -auto-approve --terragrunt-non-interactive -var AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -var AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -var AWS_REGION=$AWS_DEFAULT_REGION env: AWS_ACCESS_KEY_ID: ${{ vars.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_DEFAULT_REGION: ${{ vars.AWS_DEFAULT_REGION }} destroy: if: ${{ inputs.destroy == 'true' }} runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v2 - name: Setup SSH uses: webfactory/[email protected] with: ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }} - name: Setup Terraform uses: hashicorp/setup-terraform@v2 with: terraform_version: 1.5.5 terraform_wrapper: false - name: Setup Terragrunt run: | curl -LO "https://github.com/gruntwork-io/terragrunt/releases/download/v0.48.1/terragrunt_linux_amd64" chmod +x terragrunt_linux_amd64 sudo mv terragrunt_linux_amd64 /usr/local/bin/terragrunt terragrunt -v - name: Destroy Terraform changes run: | cd dev terragrunt run-all destroy -auto-approve --terragrunt-non-interactive -var AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -var AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -var AWS_REGION=$AWS_DEFAULT_REGION env: AWS_ACCESS_KEY_ID: ${{ vars.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_DEFAULT_REGION: ${{ vars.AWS_DEFAULT_REGION }}

‍Terragrunt is a thin wrapper that provides extra tools for keeping your Terraform configurations DRY (Don't Repeat Yourself), working with multiple Terraform modules, and managing remote state. It's particularly useful in managing large-scale infrastructure deployments with Terraform.

... #************************** Terraform ************************************* ARG TERRAFORM_VERSION=1.7.3 RUN set -ex \ && curl -O https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /usr/local/bin/ RUN set -ex \ && mkdir -p $HOME/.terraform.d/plugin-cache && echo 'plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"' > ~/.terraformrc #************************* Terragrunt ************************************* ARG TERRAGRUNT_VERSION=0.55.1 RUN set -ex \ && wget https://github.com/gruntwork-io/terragrunt/releases/download/v${TERRAGRUNT_VERSION}/terragrunt_linux_amd64 -q \ && mv terragrunt_linux_amd64 /usr/local/bin/terragrunt \ && chmod +x /usr/local/bin/terragrunt #*********************** Terramate **************************************** ARG TERRAMATE_VERSION=0.4.5 RUN set -ex \ && wget https://github.com/mineiros-io/terramate/releases/download/v${TERRAMATE_VERSION}/terramate_${TERRAMATE_VERSION}_linux_x86_64.tar.gz \ && tar -xzf terramate_${TERRAMATE_VERSION}_linux_x86_64.tar.gz \ && mv terramate /usr/local/bin/terramate \ && chmod +x /usr/local/bin/terramate #*********************** tfsec ******************************************** ARG TFSEC_VERSION=1.28.5 RUN set -ex \ && wget https://github.com/aquasecurity/tfsec/releases/download/v${TFSEC_VERSION}/tfsec-linux-amd64 \ && mv tfsec-linux-amd64 /usr/local/bin/tfsec \ && chmod +x /usr/local/bin/tfsec \ && terragrunt --version #**********************Terraform docs ************************************ ARG TERRRAFORM_DOCS_VERSION=0.17.0 RUN set -ex \ && curl -sSLo ./terraform-docs.tar.gz https://terraform-docs.io/dl/v${TERRRAFORM_DOCS_VERSION}/terraform-docs-v${TERRRAFORM_DOCS_VERSION}-$(uname)-amd64.tar.gz \ && tar -xzf terraform-docs.tar.gz \ && chmod +x terraform-docs \ && mv terraform-docs /usr/local/bin/terraform-docs #********************* ShellCheck ***************************************** ARG SHELLCHECK_VERSION="stable" RUN set -ex \ && wget -qO- "https://github.com/koalaman/shellcheck/releases/download/${SHELLCHECK_VERSION?}/shellcheck-${SHELLCHECK_VERSION?}.linux.x86_64.tar.xz" | tar -xJv \ && cp "shellcheck-${SHELLCHECK_VERSION}/shellcheck" /usr/bin/ \ && shellcheck --version ...

Our first attempt was to introduce other engineering teams to Terraform - the Platform team was already using it extensively with Terragrunt, and using Atlantis to automate plan and apply operations in a Git flow to ensure infrastructure was consistent. We'd written modules, with documentation, and an engineer would simply need to raise a PR to use the module and provide the right values, and Atlantis (once the PR was approved by Platform) would go ahead and set it up for them.

Terragrunt: An extension of Terraform, Terragrunt assists in managing complex infrastructure with less duplication and more efficiency. Its power lies in its ability to manage dependencies and its dry configuration approach.

Copy the link and download on your terminal using the wget command. Example: wget https://github.com/gruntwork-io/terragrunt/releases/download/v0.54.19/terragrunt_linux_amd64

name: Configure on: push: branches: - main pull_request: branches: - main jobs: terraform: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v2 - name: Setup SSH uses: webfactory/[email protected] with: ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }} - name: Setup Terraform uses: hashicorp/setup-terraform@v2 with: terraform_version: 1.5.5 terraform_wrapper: false - name: Setup Terragrunt run: | curl -LO "https://github.com/gruntwork-io/terragrunt/releases/download/v0.48.1/terragrunt_linux_amd64" chmod +x terragrunt_linux_amd64 sudo mv terragrunt_linux_amd64 /usr/local/bin/terragrunt terragrunt -v - name: Apply Terraform changes run: | cd dev terragrunt run-all apply -auto-approve --terragrunt-non-interactive -var AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID -var AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY -var AWS_REGION=$AWS_DEFAULT_REGION cd apache-server/ec2-web-server public_ip=$(terragrunt output instance_public_ip) echo "$public_ip" > public_ip.txt cat public_ip.txt env: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} AWS_DEFAULT_REGION: ${{ secrets.AWS_DEFAULT_REGION }} - name: Upload artifact uses: actions/upload-artifact@v4 with: name: ip-artifact path: dev/apache-server/ec2-web-server/public_ip.txt ansible: runs-on: ubuntu-latest needs: terraform steps: - name: Download artifact uses: actions/download-artifact@v4 with: name: ip-artifact - name: Configure Ansible run: | sudo apt update sudo pipx inject ansible-core jmespath ansible-playbook --version sudo echo "[web]" >> ansible_hosts sudo cat public_ip.txt >> ansible_hosts mv ansible_hosts $HOME sudo cat $HOME/ansible_hosts - name: Configure playbook run: | cd $HOME cat > deploy.yml < Test Page This is a test page EOF cat $HOME/deploy.yml - name: Run playbook uses: dawidd6/action-ansible-playbook@v2 with: playbook: deploy.yml directory: /home/runner key: ${{secrets.SSH_PRIVATE_KEY}} options: | --inventory ansible_hosts --verbose

Created and maintained by Gruntwork, Terragrunt is a tool designed to enhance Terraform’s capabilities. It acts as a thin wrapper around Terraform, offering additional features to streamline and optimise Terraform usage. Key functions of Terragrunt include helping users keep their Terraform configurations DRY (Don’t Repeat Yourself), efficiently managing multiple Terraform modules, and handling remote state management. By reducing repetition in Terraform code and simplifying the management of complex module dependencies and remote state, Terragrunt makes working with Terraform more efficient, especially for larger or more complex infrastructure deployments.

locals { # Automatically load region-level variables region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl")) # Automatically load environment-level variables` environment_vars = read_terragrunt_config(find_in_parent_folders("env.hcl")) # Extract the variables we need for easy access account_name = local.environment_vars.locals.account_name account_id = local.environment_vars.locals.aws_account_id aws_region = local.region_vars.locals.aws_region # This is the S3 bucket where the Terraform State Files will be stored remote_state_bucket = "devops-bucket" # This is the DynamoDB table where Terraform will add the locking status dynamodb_table = "terraform-state-lock" # IAM Role for Terraform backend to assume terraform_backend_role = "arn:aws:iam::{shared-services_account_id}:role/terraform-backend-role" environment_path = replace(path_relative_to_include(), "environments/", "") # https://github.com/hashicorp/terraform/releases terraform_version = "latest" # https://github.com/gruntwork-io/terragrunt/releases terragrunt_version = "latest" } # Generate an AWS provider block generate "provider" { path = "provider.tf" if_exists = "overwrite_terragrunt" contents = <

If your team works with Terraform, you should definitely try Terragrunt (and obviously, its CLI tool!). It's an Infrastructure as Code (IaC) tool that acts as a wrapper for Terraform and simplifies dealing with multiple Terraform modules in different environments.