magic-modules VS dhall-lang

Given: https://registry.terraform.io/providers/hashicorp/google/5.3...

how would any reasonable person know what https://registry.terraform.io/providers/hashicorp/google/5.3... to enable without (a) trying it and squinting at the error message (b) clicking on the <> then realizing it, also, does not mention run.googleapis.com, click on "supported service endpoints" <https://cloud.google.com/run/docs/reference/rest#rest_endpoi...> and only then learning about https://cloud.google.com/run/docs/reference/rest#service:-ru...

Repeat for https://registry.terraform.io/providers/hashicorp/google/5.3... although in both cases I guess the astute reader may have spotted the run.googleapis.com in the forbidden service labels and cloudidentity.googleapis.com in the example

Since, to the best of my knowledge those bindings are auto generated <https://github.com/GoogleCloudPlatform/magic-modules#magic-m...>, I would hypothesize it is not insurmountable drop in the seemingly existing declaration of APIs required: https://github.com/GoogleCloudPlatform/magic-modules/blob/7d... https://github.com/GoogleCloudPlatform/magic-modules/blob/7d...

In the case of the Google Cloud Platform provider, folks at Google built magic modules with the explicit goal of being able to generate schemas and behaviors for a Terraform provider and for other systems with similar needs. Since the vendor was explicitly aiming to support Terraform, this was the most ideal case where the schema could be designed to contain all of the information needed to generate a functional, usable provider.

I think the "Developing the provider" instructions in this provider's repository are rather stale, because they still talk about GOPATH even though that's been obsolete for several Go versions now. Note also that much of the code in that repository is auto-generated from an upstream repository googleCloudPlatform/magic-modules, and so for some changes it may be better to contribute there once you've tested the modifications more directly inside the provider repository.

For GCP, both ansible modules and terraform modules are actually generated from https://github.com/GoogleCloudPlatform/magic-modules, so their "production readiness" are the same.

I understand that mitchellh himself personally created a bunch of cloud modules for terraform at the beginning, and those were likely of higher quality than whatever created by some internal developers assigned by Google/Microsoft, and might be slightly better than the AWS modules maintained by community.

Anyway, when it comes to ansible versus terraform, we shall move the discourse to states management instead. With ansible, you don't have to deal with states, but will need to clean up the cloud resources separately. With terraform, you can use the tool to clean up the cloud resources easily, but then you also have the headache of managing states. Plus, whenever you change something, there is always the nagging feeling that it will do a destroy/recreate instead of an in-place update.

The Terraform provider for Google Cloud uses partial autogeneration, here is the repo that does the autogeneration for multiple automation tools:

https://github.com/GoogleCloudPlatform/magic-modules

Fail to see how this is any different than Dhall (https://dhall-lang.org/) other than it produces plists too.

Kubernetes config is a decent example. I had ChatGPT generate a representative silly example -- the content doesn't matter so much as the structure:

https://gist.github.com/cstrahan/528b00cd5c3a22e3d8f057bb1a7...

Now consider 100s (if not 1000s) of such files.

I haven't given Pkl an in depth look yet, but I can say that the Industry Standard™ of "simple YAML" + string substitution (with delicate, error prone indentation -- since YAML is indentation sensitive) is easily beat by any of:

- https://jsonnet.org/

- https://nickel-lang.org/

- https://nixos.org/manual/nix/stable/language/index.html

- https://dhall-lang.org/

- (insert many more here, probably including Pkl)

There are underpowered languages / tools, that can only solve a problem for which they are intended poorly. But not all limited tools are like that.

Say, eBPF is prominently not Turing-complete, which allows to guarantee that a eBPF program terminates, and even how soon. Still eBPF is hugely useful in its area.

Or, say, regular expressions are limited to regular languages; in particular, they famously [1] cannot process recursive structures, like trees. Still tools like grep / ag / rg are mightily useful.

Yes, I agree that YAML is underpowered for proper k8s configuration! But it's also too powerful for its own good in other aspects [2]. I wish Google used Dhall [3] or their own purely functional config language (FCL? I already forgot the name) instead of YAML; sadly, they did not.

[1]: https://stackoverflow.com/a/1732454/223424

[2]: https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-fr...

[3]: https://dhall-lang.org/

Dhall: Dhall is a programmable configuration language that combines features like JSON, functions, types, and import capabilities. Its style leans towards functional programming, so if you're familiar with functional-style languages such as Haskell, you might find Dhall to be quite intuitive.

I've been thinking along these lines but more 'strongly validated' than statically typed in the sense that you'd be better off being able to load the entire config and then produce a list of problems (and should be able to offer good editor support if done correctly).

Though https://dhall-lang.org/ demonstrates that you can statically type quite a lot of configuration to great advantage, which appears to be programmatically embeddable in multiple languages per https://docs.dhall-lang.org/howtos/How-to-integrate-Dhall.ht...

> Where practical is in the sense of an engineer (or in their terms, a CS practitioner),

Configuration processing. E.g. I'd like my yamls to be decidable, though I'd settle for guaranteed to halt[1].

[1] https://dhall-lang.org/

Maybe you'd like jsonnet: https://jsonnet.org/

I find it particularly useful for configurations that often have repeated boilerplate, like ansible playbooks or deploying a bunch of "similar-but" services to kubernetes (with https://tanka.dev).

Dhall is also quite interesting, with some tradeoffs: https://dhall-lang.org/

A few years ago I did a small comparison by re-implementing one of my simpler ansible playbooks: https://github.com/retzkek/ansible-dhall-jsonnet

And to checkout Dhall: https://dhall-lang.org/