ci-cd VS Docker Compose

Thanks for your feedback. GitLab team member here.

> We(as in everyone) are in a serious need of a new git server product. Just do git serving, and do it well. Preferably in a way multiple nodes can be run active-active for scaling and reliability. No need for cicd (Jenkins is fine for that, thank you very much).

You can integrate Jenkins into GitLab. https://docs.gitlab.com/ee/integration/jenkins.html Suggest considering a migration to GitLab CI/CD in your migration planning, following the updated documentation: https://docs.gitlab.com/ee/ci/migration/jenkins.html and more automated imports in https://about.gitlab.com/blog/2023/09/26/atlassian-server-en...

> Web hooks sending and receiving. For example launch a merge request webhook(to lambda via aws api gateway, or to Jenkins). Receive a webhook as merge request approval when some Jenkins job finishes.

(FYI) https://docs.gitlab.com/ee/user/project/integrations/webhook... and https://docs.gitlab.com/ee/user/project/integrations/webhook...

You can trigger a pipeline from external webhooks using a trigger token, and execute an action against the GitLab REST API. The example for triggering pipelines in https://docs.gitlab.com/ee/ci/triggers/#trigger-a-pipeline can be expanded into more actions, i.e. using the API to create MR approvals or comments.

For Python, I'd recommend looking into python-gitlab and this tutorial blog post: https://about.gitlab.com/blog/2023/02/01/efficient-devsecops...

> if you create cicd jobs/pipelines there is no way to giving someone an ability to run that pipeline without giving that person ability to push to the repository and submit merge requests. Yes, you can then set it so approval is needed before merge, protecting said pipeline, but why? There has been a ticket on gitlabs own issues page about it for years and it is still not resolved.

Please share the URL :)

When creating a new GitLab project, the default branch is protected by default, and only maintainer roles can push to the default branch. https://docs.gitlab.com/ee/user/project/protected_branches.h...

A developer role can create non-protected Git branches, merge requests, and as such trigger a pipeline from a merge request. You've mentioned approval rules as a safeguard already - CODEOWNERS can be an additional way to ensure that review workflows are followed. https://docs.gitlab.com/ee/user/project/codeowners/

You can also use branch protection rules to allow `No one` for push actions, i.e. any branch that matches the pattern, except for `main` or git tag patterns. https://docs.gitlab.com/ee/user/project/protected_branches.h...

> Gitlab enterprise has no mode of working that let's you have more than one active server at a time so goodbye horizontal scaling.

Suggest reviewing the reference architectures documentation in https://docs.gitlab.com/ee/administration/reference_architec... to decide whether horizontal scaling is needed for your environment.

For distributed environments, suggest looking into Geo: https://docs.gitlab.com/ee/administration/geo/index.html

> You want to scale your cicd worker nodes? They want you to use docker mashine(a deprecated product) instead of writing a Plugin like ec2-fleet for Jenkins.

The current GitLab CI/CD runner architecture involves docker-machine, based on a fork maintained by GitLab. This fork receives security and bug fixes to ensure users and customers can rely on auto-scaling in production. https://gitlab.com/gitlab-org/ci-cd/docker-machine#%EF%B8%8F... Please review the support statement in https://gitlab.com/groups/gitlab-org/-/epics/2502 to learn more for how long the fork remains supported.

The new auto-scaling architecture provides a task scheduler, and so-called fleeting plugins. You can review the architecture blueprint in https://docs.gitlab.com/ee/architecture/blueprints/runner_sc... and follow the documentation in https://docs.gitlab.com/runner/runner_autoscale/

If you prefer a timeline, please follow the Docker Machine Replacement Project Plan in https://gitlab.com/groups/gitlab-org/-/epics/6995 For example, the AWS EC2 Fleeting plugin is available in Beta since GitLab 16.5 and scheduled for 16.7 GA, see the epic https://gitlab.com/groups/gitlab-org/-/epics/8856

When using Kubernetes, you can take advantage of the Kubernetes executor to auto-scale pods. https://docs.gitlab.com/runner/executors/kubernetes.html

To optimize the CI/CD infrastructure next to auto-scaling, these tips might be handy, too: https://docs.gitlab.com/ee/ci/pipelines/pipeline_efficiency....

> into saas.

Next to GitLab self-managed and SaaS, you can also use GitLab Dedicated, where you get access your own isolated cloud instance. https://about.gitlab.com/blog/2023/08/03/building-gitlab-wit...

The Fargate driver doesn’t support ECS Exec yet. For more info: https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/fargate/-/issues/49

Their fork is, as one might imagine, open source. You can see what it does against a normal Ubuntu system, for example: https://gitlab.com/gitlab-org/ci-cd/docker-machine/-/blob/v0.16.2-gitlab.13/libmachine/provision/ubuntu_systemd.go

Gitlab maintains a fork of docker-machine with critical bug fixes that I've been using recently since Docker deprecated it: https://gitlab.com/gitlab-org/ci-cd/docker-machine

Just look at the project https://gitlab.com/gitlab-org/ci-cd/codequality/container_registry

sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose

Solutions are many, and could include Docker Compose, VS Code dev containers, Telepresence, Localstack or setting up temporary AWS accounts as a development environment for serverless applications.

Using Docker Compose to run Postgres and any third-party services locally provides a streamlined and consistent development environment. Developers can define services in a docker-compose.yml file, which enables them to configure and launch an entire stack with a single command. In this case, Postgres is encapsulated within a container with predefined configurations. Docker Compose also facilitates easy scaling, updates, and isolation of services, enhancing development efficiency and reducing the setup time for new team members or transitioning between projects. It encapsulates complex configurations, such as Postgres' performance monitoring and logging settings, in a manageable, version-controlled file, simplifying and replicating the service setup across different environments.

“Compose simplifies the control of your entire application stack, making it easy to manage services, networks, and volumes in a single, comprehensible YAML configuration file. Then, with a single command, you create and start all the services from your configuration file.” - Docker documentation

curl -fsSL https://get.docker.com -o get-docker.sh sudo sh get-docker.sh # Install docker compose sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose # Apply executable permissions to the binary sudo chmod +x /usr/local/bin/docker-compose # Run Project docker-compose up -d

Docker,

This foundation now opens the door to even more powerful concepts. You can explore more advanced concepts such as container networking, streamlining the management of complex applications with Docker Compose, and how to make your application data persistent using volumes.

In many projects that are containerized, especially in cases where development is also done locally with docker-compose, teams often have two Dockerfiles, 1 for Development, the other for Production. If you happen to have multiple environments like pre-prod, staging and so on, some teams could have different Dockerfiles for these environments.