ci-cd VS ctop

Thanks for your feedback. GitLab team member here.

> We(as in everyone) are in a serious need of a new git server product. Just do git serving, and do it well. Preferably in a way multiple nodes can be run active-active for scaling and reliability. No need for cicd (Jenkins is fine for that, thank you very much).

You can integrate Jenkins into GitLab. https://docs.gitlab.com/ee/integration/jenkins.html Suggest considering a migration to GitLab CI/CD in your migration planning, following the updated documentation: https://docs.gitlab.com/ee/ci/migration/jenkins.html and more automated imports in https://about.gitlab.com/blog/2023/09/26/atlassian-server-en...

> Web hooks sending and receiving. For example launch a merge request webhook(to lambda via aws api gateway, or to Jenkins). Receive a webhook as merge request approval when some Jenkins job finishes.

(FYI) https://docs.gitlab.com/ee/user/project/integrations/webhook... and https://docs.gitlab.com/ee/user/project/integrations/webhook...

You can trigger a pipeline from external webhooks using a trigger token, and execute an action against the GitLab REST API. The example for triggering pipelines in https://docs.gitlab.com/ee/ci/triggers/#trigger-a-pipeline can be expanded into more actions, i.e. using the API to create MR approvals or comments.

For Python, I'd recommend looking into python-gitlab and this tutorial blog post: https://about.gitlab.com/blog/2023/02/01/efficient-devsecops...

> if you create cicd jobs/pipelines there is no way to giving someone an ability to run that pipeline without giving that person ability to push to the repository and submit merge requests. Yes, you can then set it so approval is needed before merge, protecting said pipeline, but why? There has been a ticket on gitlabs own issues page about it for years and it is still not resolved.

Please share the URL :)

When creating a new GitLab project, the default branch is protected by default, and only maintainer roles can push to the default branch. https://docs.gitlab.com/ee/user/project/protected_branches.h...

A developer role can create non-protected Git branches, merge requests, and as such trigger a pipeline from a merge request. You've mentioned approval rules as a safeguard already - CODEOWNERS can be an additional way to ensure that review workflows are followed. https://docs.gitlab.com/ee/user/project/codeowners/

You can also use branch protection rules to allow `No one` for push actions, i.e. any branch that matches the pattern, except for `main` or git tag patterns. https://docs.gitlab.com/ee/user/project/protected_branches.h...

> Gitlab enterprise has no mode of working that let's you have more than one active server at a time so goodbye horizontal scaling.

Suggest reviewing the reference architectures documentation in https://docs.gitlab.com/ee/administration/reference_architec... to decide whether horizontal scaling is needed for your environment.

For distributed environments, suggest looking into Geo: https://docs.gitlab.com/ee/administration/geo/index.html

> You want to scale your cicd worker nodes? They want you to use docker mashine(a deprecated product) instead of writing a Plugin like ec2-fleet for Jenkins.

The current GitLab CI/CD runner architecture involves docker-machine, based on a fork maintained by GitLab. This fork receives security and bug fixes to ensure users and customers can rely on auto-scaling in production. https://gitlab.com/gitlab-org/ci-cd/docker-machine#%EF%B8%8F... Please review the support statement in https://gitlab.com/groups/gitlab-org/-/epics/2502 to learn more for how long the fork remains supported.

The new auto-scaling architecture provides a task scheduler, and so-called fleeting plugins. You can review the architecture blueprint in https://docs.gitlab.com/ee/architecture/blueprints/runner_sc... and follow the documentation in https://docs.gitlab.com/runner/runner_autoscale/

If you prefer a timeline, please follow the Docker Machine Replacement Project Plan in https://gitlab.com/groups/gitlab-org/-/epics/6995 For example, the AWS EC2 Fleeting plugin is available in Beta since GitLab 16.5 and scheduled for 16.7 GA, see the epic https://gitlab.com/groups/gitlab-org/-/epics/8856

When using Kubernetes, you can take advantage of the Kubernetes executor to auto-scale pods. https://docs.gitlab.com/runner/executors/kubernetes.html

To optimize the CI/CD infrastructure next to auto-scaling, these tips might be handy, too: https://docs.gitlab.com/ee/ci/pipelines/pipeline_efficiency....

> into saas.

Next to GitLab self-managed and SaaS, you can also use GitLab Dedicated, where you get access your own isolated cloud instance. https://about.gitlab.com/blog/2023/08/03/building-gitlab-wit...

The Fargate driver doesn’t support ECS Exec yet. For more info: https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/fargate/-/issues/49

Their fork is, as one might imagine, open source. You can see what it does against a normal Ubuntu system, for example: https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/blob/v0.16.2-gitlab.13/libmachine/provision/ubuntu_systemd.go

Gitlab maintains a fork of docker-machine with critical bug fixes that I've been using recently since Docker deprecated it: https://gitlab.com/gitlab-org/ci-cd/docker-machine

Just look at the project https://gitlab.com/gitlab-org/ci-cd/codequality/container_registry

This does remind me of ctop as well: https://github.com/bcicen/ctop

It also let's you look at containers, resource usage graphs, their logs and even do some actions through a TUI.

ssh, nnn, micro and ctop is all I need on my dockerhosts

My problem with ctop is, that it seems to show wrong memory usage data: https://github.com/bcicen/ctop/issues/314

When talk about interface and cli... I am a huge fan of ctop

You can use CTOP. It's like a lite portainer on CLI. You can check logs, stats, restart containers.

In the terminal, there are also a few useful projects:

  - for Docker, there is ctop: https://github.com/bcicen/ctop

> I tried portainer, awful UX experience and all good features are inside paid version.

This is interesting to me, because it doesn't quite match my experience - I've been using Portainer for around 3 years at this point and it's been pretty decent.

The worst issues that I've gotten is networking issues in some hybrid configurations with Docker Swarm (e.g. Portainer cannot reach the manager node of the cluster for a bit), or troubles configuring Traefik ingresses when managing Kubernetes (though I think the recent patch notes talked about improving the ingress section, so maybe the experience will get better with non-Nginx ingresses).

Other than that, it's been great for onboarding new people, illustrating the cluster state at a glance, easily operating with stacks and scaling/restarting services as needed, including pulling new images, viewing the logs or even connecting to containers through a web UI if need be. The webhook functionality in particular is really nice - you can just do a curl request against a given URL and that will pull the new container versions for the given image and do a redeploy, which works nicely with a variety of CI solutions.

When I last tried, initializing Nomad clusters with networking encryption was a bit less of a smooth experience (needing to essentially manage your own PKI) and the web UI felt more like a dashboard, instead of something that you could click around in, if you're a proponent of that workflow.

Rancher is probably better than both of those options, though there's a certain overhead in regards to running both that software and a full Kubernetes cluster. If Kubernetes feels like a good fit for a particular project and resources aren't an issue, definitely check it out! You can, of course, also have some success with lightweight clusters, like K3s: https://k3s.io/

I'll definitely agree that Lazydocker is a nice tool, but I wouldn't call it superior, just different (TUI vs GUI), their demo video is nice though: https://youtu.be/NICqQPxwJWw

It actually reminds me of ctop, which you might also want to check out, though it's not something that you'd manage clusters in, merely the individual containers on a node (which won't always be enough, same as Docker Compose isn't): https://github.com/bcicen/ctop

Regardless, for Kubernetes, I'm inclined to say that you'd enjoy k9s a bunch then, it has a similar TUI approach: https://k9scli.io/