Our great sponsors
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
drone
Discontinued Gitness is an Open Source developer platform with Source Control management, Continuous Integration and Continuous Delivery. [Moved to: https://github.com/harness/gitness]
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
What I would really like to see is a CI system that lets me write a script in a language of my choice instead of defining a pipeline config file. That way I can run the pipeline locally, put breakpoints in, etc.
Nuke [1] gets close but there are still a lot of tasks that don't have C# bindings, such as publishing build artifacts and uploading test results.
While I'm dreaming about my perfect CI, I'd also like the ability to download benchmark results from previous commits so that I can generate trend graphs and publish them in the build results.
To do this right the CI system would have to have an API using REST, GraphQL, gRPC or some such API format that generates clients in many languages. That way they don't have to maintain bindings in every language.
[1] https://nuke.build/
To me it reminds me of troposphere[1]. It's similarly using an imperative language (also python) to generate a declarative file (CloudFormation).
[1] https://github.com/cloudtools/troposphere
Check out tekton CI, it's a Kubernetes operator to run a CI pipeline defined as commands that run inside any container. Yeah you need a k8s cluster, but even a simple kind dev cluster that you spin up in 30 seconds with one command on your laptop will work. https://tekton.dev/
I like it a lot because it enforces very little structure on you and doesn't reinvent everything. Stuff like storage (either ephermeral or existing volumes), secrets, configuration, etc. are already modeled and supported by Kubernetes and tekton can use all of that natively.
If you're really averse to k8s though, check out drone. It has a local execution mode that is similar and just runs whatever pipeline commands you want in docker containers. https://github.com/drone/drone
Check out tekton CI, it's a Kubernetes operator to run a CI pipeline defined as commands that run inside any container. Yeah you need a k8s cluster, but even a simple kind dev cluster that you spin up in 30 seconds with one command on your laptop will work. https://tekton.dev/
I like it a lot because it enforces very little structure on you and doesn't reinvent everything. Stuff like storage (either ephermeral or existing volumes), secrets, configuration, etc. are already modeled and supported by Kubernetes and tekton can use all of that natively.
If you're really averse to k8s though, check out drone. It has a local execution mode that is similar and just runs whatever pipeline commands you want in docker containers. https://github.com/drone/drone
My main problem is running the job/step locally and not wait to CI steps to validate some change.
My solution is: https://github.com/rosineygp/mkdkr
I can write somethings like this:
```Makefile
I think the only exit to this cycle is something like dhall, which adds features like functions and imports, but is "total" / forbids side-effects. https://dhall-lang.org/
I came across this great blog post: https://www.objectif-libre.com/en/blog/2021/02/23/a-new-era-...
Documentation: https://docs.gitlab.com/ee/ci/parent_child_pipelines.html
GitLab 13.10 also added support for parallel matrix job execution in child pipelines, speeding up the execution once more.
https://about.gitlab.com/releases/2021/03/22/gitlab-13-10-re...
There's more to dynamic pipeline creation, collecting blog post ideas in https://gitlab.com/gitlab-com/www-gitlab-com/-/issues/11122#... :)
Slightly relevant: I made a hack to write Ansible playbooks in Python [1]. It doesn't fix everything, for example runtime control flow still has to be described as attributes of tasks, but it does help with parametrising. It's a question whether it's worth it to add such a hack or if I should just make do with YAML.
[0]: https://gitlab.com/-/snippets/2112382
Lets look at a specific example. Take Kubernetes: everything is yaml, with complete schemas, all the way down. From your perspective this is configuration utopia, right? Meanwhile back in reality k8s is the poster child of "yaml hell". From the day it was released, people took one look at it, gave it a giant NOPE and instantly spawned half a dozen templating languages. The most popular of these is helm, which has a terrible, no good, very bad design: full of potential injection attacks from purely textual string substitution, manually specified indentation to embed parameterized blocks, virtually no intermediate validation, no way to validate unused features, etc etc
Compare to dhall which publishes a complete set of dhall-k8s schema mappings which enables you to factor out any design you want down to as few configuration variables as you like, while validating the configuration generators themselves at design time. https://github.com/dhall-lang/dhall-kubernetes#more-modular-...