Our great sponsors
-
terraform
Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
-
Pulumi
Pulumi - Infrastructure as Code in any programming language. Build infrastructure intuitively on any cloud using familiar languages 🚀
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
terragrunt
Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules.
-
terraform-cdk
Define infrastructure resources using programming constructs and provision them using HashiCorp Terraform
-
mazzle
run server for building large and complicated consistent environments http://devops-pipeline.com
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
aws-cloudformation-resource-providers-rds
The CloudFormation Resource Provider Package For Amazon Relational Database Service
-
-
-
pulumi-provider-boilerplate-openapi
Boilerplate showing how to create a native Pulumi provider using an Open API specification
-
pulumi-terraform-bridge
A library allowing providers built with the Terraform Plugin SDK to be bridged into Pulumi.
-
-
coc.nvim
Nodejs extension host for vim & neovim, load extensions like VSCode and host language servers.
-
-
-
-
porter
Porter enables you to package your application artifact, client tools, configuration and deployment logic together as an installer that you can distribute, and install with a single command. (by getporter)
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Yes, however this will work (typically) if the cluster already exists (a previous run), but typically not if you creating the cluster, and kubernetes provider, as part of the same run.
IIRC you'll end up with a kubernetes provider without auth (typically pointing at your local machine), which is 1, not helpful, and 2) can be actively bad.
I believe the core issue here is that providers don't have the ability to specify a `depends_on` relation: https://github.com/hashicorp/terraform/issues/2430
You may be interested in Pulumi: https://www.pulumi.com/
Basically it's Terraform but instead of declaring your resources in HCL, you declare them in a real programming language. You're still producing a declarative config that the engine then diffs, applies etc. In fact, it's compatible with existing terraform providers, so it has a surprisingly large selection of things you can use it for.
Note their docs will try to guide you towards using their hosted service which basically does nothing except host the state file, but you can use an S3 or GCS bucket instead and it works fine.
It's definitely not without its own problems, but I'd say it's overall an improvement.
Have you tried Terragrunt [0]? It helps a lot with managing a set of related stacks. Still feels like a bandaid on a broken model, but it is what we have.
[0] https://terragrunt.gruntwork.io/
Regarding performance, last time I looked, Hashicorp's documentation implied there was no limit to the size of a Terraform stack. I think they meant theoretically in a science fiction universe where humanity had converted all of the sun's output to performing terraform plan and apply...
People mention pulumi but hashicorp are creating something similar with https://github.com/hashicorp/terraform-cdk. But all the existing terraform providers work with it afaik.
I use a tool I wrote to layer my infrastructure with layers called components and I configure it with a Graphviz file.
My tool, called mazzle (previously devops-pipeline) would run parts of the graph that can run in parallel in parallel. It can also run parts of the build on SSH workers. You bring up the workers at the beginning of the build.
Here's an example of a graph generated from graphviz file: https://github.com/samsquire/mazzle/blob/master/docs/archite...
This graph brings up a hashicorp vault server, Java application, bastion proxy etc.
here's the graphviz file: https://github.com/samsquire/mazzle-starter/blob/master/arch...
It describes the ordering of the infrastructure, the invocation of Ansible, packer, shell scripts to set up vault etc.
The idea is to be able to bring up a new environment by changing one parameter. There's a React GUI too.
https://devops-pipeline.com
I use a tool I wrote to layer my infrastructure with layers called components and I configure it with a Graphviz file.
My tool, called mazzle (previously devops-pipeline) would run parts of the graph that can run in parallel in parallel. It can also run parts of the build on SSH workers. You bring up the workers at the beginning of the build.
Here's an example of a graph generated from graphviz file: https://github.com/samsquire/mazzle/blob/master/docs/archite...
This graph brings up a hashicorp vault server, Java application, bastion proxy etc.
here's the graphviz file: https://github.com/samsquire/mazzle-starter/blob/master/arch...
It describes the ordering of the infrastructure, the invocation of Ansible, packer, shell scripts to set up vault etc.
The idea is to be able to bring up a new environment by changing one parameter. There's a React GUI too.
https://devops-pipeline.com
We have a big amount of available resources inside of our Spacelift provider[0] weights ~20 MB.
It'll probably mostly depend on the libraries you use.
[0]:https://github.com/spacelift-io/terraform-provider-spacelift...
This would be great. Perhaps it could be based on https://dhall-lang.org/
Merely as the technical answer to your question, not as advocacy: CFN has custom providers [0] and they've started publishing quite a few implementations on GH (but I haven't tried them to know if they're for real): e.g. https://github.com/aws-cloudformation/aws-cloudformation-res...
As far as I know, it is possible to bridge terraform providers into a CFN stack using that mechanism, similar to how Pulumi works
0: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
Merely as the technical answer to your question, not as advocacy: CFN has custom providers [0] and they've started publishing quite a few implementations on GH (but I haven't tried them to know if they're for real): e.g. https://github.com/aws-cloudformation/aws-cloudformation-res...
As far as I know, it is possible to bridge terraform providers into a CFN stack using that mechanism, similar to how Pulumi works
0: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGui...
Terraform is such an underappreciated tool. It seems like so much of the hate surrounds HCL1 (back in Terraform before 0.12) and doesn't reflect modern Terraform.
For example, after introducing `for_each` and dynamic blocks, it's possible to nearly entirely ditch variables files and local modules, and just add more infrastructure by editing a local YAML file. The only variables your Terraform code should have should be credentials / other secrets that are not loaded from environment variables by providers. A great public example of this usage pattern is supplied by https://github.com/concourse/governance to manage their GitHub repositories.
> If Pulumi didn't bless it, it doesn't exist in Pulumi's world.
That has not been my experience. I have personally ported a Sentry TF provider into Pulumi, and I will grant you that their docs and examples are bordering on active user hatred for exercising the process, but it does work:
https://github.com/pulumi/pulumi-terraform-bridge#adapting-a...
https://github.com/pulumi/pulumi-tf-provider-boilerplate#rea...
What mystifies me about that situation is that I do actually appreciate the amount of silliness that is required to avoid using Pulumi cloud: they are not financially incentivized to make that easy, but I'd guess a lot more folks would nope right out if they didn't make it possible
However, I would think they'd want to make ingesting a TF provider into Pulumi as smooth and reliable as possible, so they don't have people close their browser tab when they don't find a supported provider for Pulumi but it exists in TF
> If Pulumi didn't bless it, it doesn't exist in Pulumi's world.
That has not been my experience. I have personally ported a Sentry TF provider into Pulumi, and I will grant you that their docs and examples are bordering on active user hatred for exercising the process, but it does work:
https://github.com/pulumi/pulumi-terraform-bridge#adapting-a...
https://github.com/pulumi/pulumi-tf-provider-boilerplate#rea...
What mystifies me about that situation is that I do actually appreciate the amount of silliness that is required to avoid using Pulumi cloud: they are not financially incentivized to make that easy, but I'd guess a lot more folks would nope right out if they didn't make it possible
However, I would think they'd want to make ingesting a TF provider into Pulumi as smooth and reliable as possible, so they don't have people close their browser tab when they don't find a supported provider for Pulumi but it exists in TF
https://github.com/hashicorp/terraform-ls
These are both fairly limited but you can see every editor (except intellij) uses this under the hood.
I've used them at companies with 300+ terraform repos and have never had much of an issue navigating/understanding TF through Vim.
For GCP, both ansible modules and terraform modules are actually generated from https://github.com/GoogleCloudPlatform/magic-modules, so their "production readiness" are the same.
I understand that mitchellh himself personally created a bunch of cloud modules for terraform at the beginning, and those were likely of higher quality than whatever created by some internal developers assigned by Google/Microsoft, and might be slightly better than the AWS modules maintained by community.
Anyway, when it comes to ansible versus terraform, we shall move the discourse to states management instead. With ansible, you don't have to deal with states, but will need to clean up the cloud resources separately. With terraform, you can use the tool to clean up the cloud resources easily, but then you also have the headache of managing states. Plus, whenever you change something, there is always the nagging feeling that it will do a destroy/recreate instead of an in-place update.
I'm closely tracking an effort by Microsoft that aims to do a lot of what you're describing since I find myself bridging between these tools and deploying stacks that span tools and roles. [CNAB](https://cnab.io/) and the front-running implementation, [Porter](https://porter.sh/), enable one-step infra deployments, packaged as a single OCI-compatible container, with any number of steps, using the best tools for each of those steps. Think of using aws-cli for some initialization step (create or verify presence of a state bucket), applying some terraform to create infra, and finishing with a helm chart to complete deployment of app components. Each stage in a bundle packages not only the code to run it but also the execution binary of the tool that runs it. The spec and porter are still a moving target but it's a promising space and a nice adjacent evolution of the current state of tooling.
I'm closely tracking an effort by Microsoft that aims to do a lot of what you're describing since I find myself bridging between these tools and deploying stacks that span tools and roles. [CNAB](https://cnab.io/) and the front-running implementation, [Porter](https://porter.sh/), enable one-step infra deployments, packaged as a single OCI-compatible container, with any number of steps, using the best tools for each of those steps. Think of using aws-cli for some initialization step (create or verify presence of a state bucket), applying some terraform to create infra, and finishing with a helm chart to complete deployment of app components. Each stage in a bundle packages not only the code to run it but also the execution binary of the tool that runs it. The spec and porter are still a moving target but it's a promising space and a nice adjacent evolution of the current state of tooling.
Best thing is Dhall that I am aware of. Same situation, working as a consultant, forced to use broken things.
https://github.com/dhall-lang/dhall-kubernetes