mazzle-starter VS magic-modules

I take advantage of I/O being parallel in Python in my mazzle continuous integration pipeline tool. I'm not sharing mutable state.

I spin up a graph of python Threads and each joins others in a graph. This way we can run graphs in parallel. See this graph - the parts that look like this:

dependency -> {parallel1; parallel2; parallel3} -> postparallel

parallel1, parallel2, parallel3 can run in parallel in a separate python thread because the IO is parallel.

postparallel joins parallel1, parallel2, parallel3 and waits for them all to complete.

Where parallel1-3 is things like ansible, packer (slow), AMI builds, chef runs etc.

https://github.com/samsquire/mazzle-starter/blob/master/arch...

I use a tool I wrote to layer my infrastructure with layers called components and I configure it with a Graphviz file.

My tool, called mazzle (previously devops-pipeline) would run parts of the graph that can run in parallel in parallel. It can also run parts of the build on SSH workers. You bring up the workers at the beginning of the build.

Here's an example of a graph generated from graphviz file: https://github.com/samsquire/mazzle/blob/master/docs/archite...

This graph brings up a hashicorp vault server, Java application, bastion proxy etc.

here's the graphviz file: https://github.com/samsquire/mazzle-starter/blob/master/arch...

It describes the ordering of the infrastructure, the invocation of Ansible, packer, shell scripts to set up vault etc.

The idea is to be able to bring up a new environment by changing one parameter. There's a React GUI too.

https://devops-pipeline.com

Given: https://registry.terraform.io/providers/hashicorp/google/5.3...

how would any reasonable person know what https://registry.terraform.io/providers/hashicorp/google/5.3... to enable without (a) trying it and squinting at the error message (b) clicking on the <> then realizing it, also, does not mention run.googleapis.com, click on "supported service endpoints" <https://cloud.google.com/run/docs/reference/rest#rest_endpoi...> and only then learning about https://cloud.google.com/run/docs/reference/rest#service:-ru...

Repeat for https://registry.terraform.io/providers/hashicorp/google/5.3... although in both cases I guess the astute reader may have spotted the run.googleapis.com in the forbidden service labels and cloudidentity.googleapis.com in the example

Since, to the best of my knowledge those bindings are auto generated <https://github.com/GoogleCloudPlatform/magic-modules#magic-m...>, I would hypothesize it is not insurmountable drop in the seemingly existing declaration of APIs required: https://github.com/GoogleCloudPlatform/magic-modules/blob/7d... https://github.com/GoogleCloudPlatform/magic-modules/blob/7d...

In the case of the Google Cloud Platform provider, folks at Google built magic modules with the explicit goal of being able to generate schemas and behaviors for a Terraform provider and for other systems with similar needs. Since the vendor was explicitly aiming to support Terraform, this was the most ideal case where the schema could be designed to contain all of the information needed to generate a functional, usable provider.

I think the "Developing the provider" instructions in this provider's repository are rather stale, because they still talk about GOPATH even though that's been obsolete for several Go versions now. Note also that much of the code in that repository is auto-generated from an upstream repository googleCloudPlatform/magic-modules, and so for some changes it may be better to contribute there once you've tested the modifications more directly inside the provider repository.

For GCP, both ansible modules and terraform modules are actually generated from https://github.com/GoogleCloudPlatform/magic-modules, so their "production readiness" are the same.

I understand that mitchellh himself personally created a bunch of cloud modules for terraform at the beginning, and those were likely of higher quality than whatever created by some internal developers assigned by Google/Microsoft, and might be slightly better than the AWS modules maintained by community.

Anyway, when it comes to ansible versus terraform, we shall move the discourse to states management instead. With ansible, you don't have to deal with states, but will need to clean up the cloud resources separately. With terraform, you can use the tool to clean up the cloud resources easily, but then you also have the headache of managing states. Plus, whenever you change something, there is always the nagging feeling that it will do a destroy/recreate instead of an in-place update.

The Terraform provider for Google Cloud uses partial autogeneration, here is the repo that does the autogeneration for multiple automation tools:

https://github.com/GoogleCloudPlatform/magic-modules