Our great sponsors
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
You are on the right track. I have exactly the same story, data engineer, working on automation a lot.
YAML, JSON and HCL should be purged from IaC asap. Luckily Dhall can be integrated everywhere we need YAML otherwise we would go nuts. The bigges failure of k8s is to understand that YAML does not scale to teams and beyond 100 lines, especially not good to nest it (hello ArgoCD).
A type safe not accidentally turning complete configuration language or for that matter any langue (hello Pulumi) is much better than YAML ever will be.
Anyways, if I can give you an advise, drop k8s like its hot. I is much more trouble than worth. We have phased it out everywhere we could. The companies still using it are in 1 year long implementation cycles with it with 10+ consultants. It is going very well. For consultants, obviously.
Heroku came up with the idea of buildpacks [01] the idea has later been picked up by Cloud Foundry [02]. Using buildpacks (as an alternative to dockerfiles/images) mitigates this problem. A lot of best practices are build into the buildpacks.
Of course this does not work for everything. But for most cases this is just good enough.