gl-infra VS multi-env-deploy

GitLab team member here. Thanks for asking.

Incidents can have different types, i.e. when an application bug or performance regression is discovered, this can involve reverting MRs and rolling back releases. The Platform, Delivery group has a top-level responsibility for ensuring continuous delivery of the GitLab application software to GitLab SaaS, https://about.gitlab.com/handbook/engineering/infrastructure...

Other incidents may involve hardware or infrastructure failures, or a combination of both, infrastructure failure that renders GitLab application services unavailable. This requires cross-functional collaboration from infrastructure, product, engineering, etc. teams in the incident.

To get a better understanding here, it is helpful to review the incident management handbook https://about.gitlab.com/handbook/engineering/infrastructure...

Additional helpful information:

- The GitLab.com SaaS production architecture is documented in https://about.gitlab.com/handbook/engineering/infrastructure...

- The Monitoring of GitLab.com handbook provides insights into monitoring workflows, incident management, SLAs, etc. https://about.gitlab.com/handbook/engineering/monitoring/

- Runbooks https://about.gitlab.com/handbook/engineering/infrastructure...

For the current incident discussed in this HN thread, the review issue can be followed in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1... to learn more.

Lol. They let a certificate expire: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/14422

I have used CodePipeline/CodeBuild, CodeDeploy, and CodeCommit quite a bit. You can see an example of it all working together with Terraform here: https://github.com/cogini/multi-env-deploy

Parameter Store is a good place to store things. ECS can read from it and set variables. This is a complete example of using Terraform to manage infrastructure with EC2 or ECS: https://github.com/cogini/multi-env-deploy Here is an app that runs in ECS: https://github.com/cogini/phoenix_container_example This task file sets env vars based on parameter store: https://github.com/cogini/phoenix_container_example/blob/master/ecs/taskdef.json

AWS has a fine stack for deploying "cloud native" apps on top of EC2 instances.

Build a base AMI using Packer and launch it to an Auto Scaling Group behind a load balancer. Deploy code to the ASG using CodeDeploy. Use RDS for the database.

This is a good match for languages that have good concurrency like Elixir. They benefit from deploying to big machines that have a lot of CPU cores, and keeping a common in-memory cache on the EC2 instance is more efficient than using an external cache like Elasticache. It also works well for resource-hungry systems with poor concurrency like Ruby on Rails. Putting these kinds of apps into big containers is just a waste of money.

Here is a complete example of that architecture using Terraform: https://github.com/cogini/multi-env-deploy

Similarly, bare metal can be really cost-effective. For $115/month, I can get a dedicated server with 24 VCPU cores (2x Intel Hexa-Core Xeon E5-2620 CPU), 64 GB RAM, 4x8 TB SATA, 30 TB traffic (see https://www.leaseweb.com/dedicated-servers#NL). That would be an order of magnitude more expensive on AWS with containers.

Here is how I did it with Terraform: https://github.com/cogini/multi-env-deploy/blob/master/terraform/modules/iam-codepipeline-app/main.tf

The AWS components are managed via Terraform: https://github.com/cogini/multi-env-deploy

A fully featured infrastructure using terraform with terragrunt can be found in this repo: https://github.com/cogini/multi-env-deploy/tree/master/terraform

You might like this full-featured example of using Terraform to set up the infrastructure for an application using EC2 instances in an autoscaling group or ECS containers. https://github.com/cogini/multi-env-deploy

Here is a set of hand-coded modules I wrote that handle deploying real world complex apps to AWS: https://github.com/cogini/multi-env-deploy