woodpecker
argo
Our great sponsors
woodpecker | argo | |
---|---|---|
54 | 43 | |
3,588 | 14,182 | |
5.1% | 1.4% | |
9.9 | 9.8 | |
5 days ago | 2 days ago | |
Go | Go | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
woodpecker
- The worst thing about Jenkins is that it works
-
GitHub: “Human eyes” will never see the contents of your private repositories
> I wish it had some sort of CI like github actions or bitbucket pipeline
I use Gitea with Drone CI and it works pretty well: https://www.drone.io/
Some might also prefer the Woodpecker CI fork due to the license: https://woodpecker-ci.org/
I setup Drone as a part of my migration away from GitLab Omnibus and have no complaints so far: https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-...
-
What self-hosted Git server ?
https://woodpecker-ci.org/ Open source clone of drone.io
-
GitHub actions top alternatives
https://www.drone.io/ or the more open fork https://woodpecker-ci.org/
-
Codeberg – Fast Open Source Alternative to GitHub
I’m trying to migrate of my personal repos from GitHub to Codeberg. The biggest problem is to find a replacement for GitHub Actions (the free offering is so generous), and my current solution for that is to self-host an instance of Woodpecker CI [1].
I’d like to see even more diversity in Git hosting beyond “let’s all migrate from X to Y”, and for that to happen, Forgejo (a soft fork of Gitea) has already began implementing federation [2].
-
JSON vs XML
The open source version of drone is https://woodpecker-ci.org/
-
Can any Hetzner user, please explain there workflow on Hetzner?
I use Hetzner, Contabo, Time4VPS and other platforms in pretty much the same way (as IaaS VPS providers on top of which I run software, as opposed to SaaS/PaaS), but here's a quick glance at how I do things.
> deploy from source repo? Terraform?
Personally, I use Gitea for my repos and Drone CI for CI/CD.
Gitea: https://gitea.io/en-us/
Drone CI: https://www.drone.io/
Some might prefer Woodpecker due to licensing: https://woodpecker-ci.org/ but honestly most solutions out there are okay, even Jenkins.
Then I have some sort of a container cluster on the servers, so I can easily deploy things: I still like Docker Swarm (projects like CapRover might be nice to look at as well), though many might enjoy the likes of K3s or K0s more (lightweight Kubernetes clusters).
Docker Swarm: https://docs.docker.com/engine/swarm/ (uses the Compose spec for manifests)
K3s: https://k3s.io/
K0s: https://k0sproject.io/ though MicroK8s and others are also okay.
I also like having something like Portainer to have a GUI to manage the clusters: https://www.portainer.io/ for Kubernetes Rancher might offer more features, but will have a higher footprint
It even supports webhooks, so I can do a POST request at the end of a CI run and the cluster will automatically pull and launch the latest tagged version of my apps: https://docs.portainer.io/user/docker/services/webhooks
> keep software up to date? ex: Postgres, OS
I build my own base container images and rebuild them (with recent package versions) on a regular basis, which is automatically scheduled: https://blog.kronis.dev/articles/using-ubuntu-as-the-base-fo...
Drone CI makes this easy to have happen in the background, as long as I don't update across major versions, or Maven decides to release a new version and remove their old version .tar.gz archives from the downloads site for some reason, breaking my builds and making me update the URL: https://docs.drone.io/cron/
Some images like databases etc. I just proxy to my Nexus instance, version upgrades are relatively painless most of the time, at least as long as I've set up the persistent data directories correctly.
> do load balancing? built-in load balancer?
This is a bit more tricky. I use Apache2 with mod_md to get Let's Encrypt certificates and Docker Swarm networking for directing the incoming traffic across the services: https://blog.kronis.dev/tutorials/how-and-why-to-use-apache-...
Some might prefer Caddy, which is another great web server with automatic HTTPS: https://caddyserver.com/ but the Apache modules do pretty much everything I need and the performance has never actually been too bad for my needs. Up until now, applications themselves have always been the bottleneck, actually working on a blog post about comparing some web servers in real world circumstances.
However, making things a bit more failure resilient might involve just paying Hetzner (in this case) to give you a load balancer: https://www.hetzner.com/cloud/load-balancer which will make everything less painless once you need to scale.
Why? Because doing round robin DNS with the ACME certificate directory accessible and synchronized across multiple servers is a nuisance, although servers like Caddy attempt to get this working: https://caddyserver.com/docs/automatic-https#storage You could also get DNS-01 challenges working, but that needs even more work and integration with setting up TXT records. Even if you have multiple servers for resiliency, not all clients would try all of the IP addresses if one of the servers is down, although browsers should: https://webmasters.stackexchange.com/a/12704
So if you care about HTTPS certificates and want to do it yourself with multiple servers having the same hostname, you'll either need to get DNS-01 working, do some messing around with shared directories (which may or may not actually work), or will just need to get a regular commercial cert that you'd manually propagate to all of the web servers.
From there on out it should be a regular reverse proxy setup, in my case Docker Swarm takes care of the service discovery (hostnames that I can access).
> handle scaling? Terraform?
None, I manually provision how many nodes I need, mostly because I'm too broke to hand over my wallet to automation.
They have an API that you or someone else could probably hook up: https://docs.hetzner.cloud/
> automate backups? ex: databases, storage. Do you use provided backups and snapshots?
I use bind mounts for all of my containers for persistent storage, so the data is accessible on the host directly.
Then I use something like BackupPC to connect to those servers (SSH/rsync) and pull data to my own backup node, which then compresses and deduplicates the data: https://backuppc.github.io/backuppc/
It was a pain to setup, but it works really well and has saved my hide dozens of times. Some might enjoy Bacula more: https://www.bacula.org/
> maintain security? built-in firewall and DDoS protection?
I personally use Apache2 with ModSecurity and the OWASP ruleset, to act as a lightweight WAF: https://owasp.org/www-project-modsecurity-core-rule-set/
You might want to just cave in and go with Cloudflare for the most part, though: https://www.cloudflare.com/waf/
- Gitea 1.19.0
argo
-
StackStorm – IFTTT for Ops
Like Argo Workflows?
-
Creators of Argo CD Release New OSS Project Kargo for Next Gen Gitops
Dagger looks more comparable to Argo Workflows: https://argoproj.github.io/argo-workflows/ That's the first of the Argo projects, which can run multi-step workflows within containers on Kubernetes.
For what it's worth, my colleagues and I have had great luck with Argo Workflows and wrote up a blog post about some of its advantages a few years ago: https://www.interline.io/blog/scaling-openstreetmap-data-wor...
-
Practical Tips for Refactoring Release CI using GitHub Actions
Despite other alternatives like Circle CI, Travis CI, GitLab CI or even self-hosted options using open-source projects like Tekton or Argo Workflow, the reason for choosing GitHub Actions was straightforward: GitHub Actions, in conjunction with the GitHub ecosystem, offers a user-friendly experience and access to a rich software marketplace.
-
(Not) to Write a Pipeline
author seems to be describing the kind of patterns you might make with https://argoproj.github.io/argo-workflows/ . or see for example https://github.com/couler-proj/couler , which is an sdk for describing tasks that may be submitted to different workflow engines on the backend.
it's a little confusing to me that the author seems to object to "pipelines" and then equate them with messaging-queues. for me at least, "pipeline" vs "workflow-engine" vs "scheduler" are all basically synonyms in this context. those things may or may not be implemented with a message-queue for persistence, but the persistence layer itself is usually below the level of abstraction that $current_problem is really concerned with. like the author says, eventually you have to track state/timestamps/logs, but you get that from the beginning if you start with a workflow engine.
i agree with author that message-queues should not be a knee-jerk response to most problems because the LoE for edge-cases/observability/monitoring is huge. (maybe reach for a queue only if you may actually overwhelm whatever the "scheduler" can handle.) but don't build the scheduler from scratch either.. use argowf, kubeflow, or a more opinionated framework like airflow, mlflow, databricks, aws lamda or step-functions. all/any of these should have config or api that's robust enough to express rate-limit/retry stuff. almost any of these choices has better observability out-of-the-box than you can easily get from a queue. but most importantly.. they provide idioms for handling failure that data-science folks and junior devs can work with. the right way to structure code is just much more clear and things like structuring messages/events, subclassing workers, repeating/retrying tasks, is just harder to mess up.
-
what technologies are people using for job scheduling in/with k8s?
Argo Workflows + Argo Events
-
What are some good self-hosted CI/CD tools where pipeline steps run in docker containers?
Drone, or Tekton, Argo Workflows if you’re on k8s
-
job scheduling for scientific computing on k8s?
Check out Argo Workflows.
-
Which build system do you use?
go-git has a lot of bugs and is not actively maintained. The bug even affects Argo Workflow, which caused our data pipeline to fail unexpectedly (reference: https://github.com/argoproj/argo-workflows/issues/10091)
-
Cron alternative that can run a job every two weeks without convoluted tricks
Yea... I personally like the jobber format which is similar to argo workflows ( https://github.com/argoproj/argo-workflows/blob/master/examples/coinflip.yaml ) which can get triggered by an event which can look like cron ( https://argoproj.github.io/argo-workflows/cron-workflows/ ).
-
Why can't I label Argo Workflows workflow-controller-metrics service for Prometheus to scrape? Everything works in some cases but fails in most
--filename https://github.com/argoproj/argo-workflows/releases/download/v3.4.4/namespace-install.yaml \
What are some alternatives?
drone - Gitness is an Open Source developer platform with Source Control management, Continuous Integration and Continuous Delivery. [Moved to: https://github.com/harness/gitness]
temporal - Temporal service
keda - KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
Airflow - Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
flyte - Scalable and flexible workflow orchestration platform that seamlessly unifies data, ML and analytics stacks.
StackStorm - StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html
n8n - Free and source-available fair-code licensed workflow automation tool. Easily automate tasks across different services.
lens - Lens - The way the world runs Kubernetes
Jenkins - Jenkins automation server
gitlab-runner
github-act-runner - act as self-hosted runner
Concourse - Concourse is a container-based continuous thing-doer written in Go.