copilot-cli VS amazon-ecs-agent

AWS Copilot CLI is an open source command line interface for running containers on AWS App Runner, Amazon Elastic Container Service (ECS), and AWS Fargate.

sudo curl -Lo /usr/local/bin/copilot https://github.com/aws/copilot-cli/releases/latest/download/copilot-linux && sudo chmod +x /usr/local/bin/copilot

Check out AWS Copilot CLI: https://aws.github.io/copilot-cli/

This is by far the best way to deploy compute into AWS in containerized workloads.

The abstraction you want is Jobs: ttps://aws.github.io/copilot-cli/docs/concepts/jobs/

Building this any other way on AWS would require provisioning multiple artifacts. The Copilot Jobs abstraction basically encapsulates the provisioning of those artifacts into one repeatable pattern.

AWS Copilot CLI is a tool that lets you deploy production-ready, scalable services on AWS from a Dockerfile in one command.

and here is the copilot page if you are interested https://aws.github.io/copilot-cli/

For larger high availability required things, AWS ECS with RDS, ElastiCache, CloudFront, S3, etc.. Really like Copilot for deployment/env/secret/sidecar management (probably needs a rename now): https://aws.github.io/copilot-cli/

We are spending about 60% less. Workload has actually lessened since AWS is so much more stable. Getting to a similar DX as Heroku was quite the lift, but once it's done, it's done. These days we generally only have outages when we screw something up ourselves. I recommend https://github.com/aws/copilot-cli for starting out on ECS.

I'd recommend checking out AWS Copilot (https://aws.github.io/copilot-cli/)

[4] https://github.com/aws/amazon-ecs-agent/issues/3129

Exactly. The ECS agent (https://github.com/aws/amazon-ecs-agent/) inside of the EC2 make requests to the AWS. To it work, it is necessary internet access (though Internet Gateway or a NAT Gateway) or the VPC endpoints (https://docs.aws.amazon.com/vpc/latest/privatelink/concepts.html#concepts-service-providers).

The approach I am going to take is to create an Apache Airflow workflow (DAG) and leverage an Apache Airflow operator, ECSOperator, which allows us to launch container based images. The container based images we launch will contain our ETL code, and this will be parameterised so that we can re-use this multiple times, changing the behaviour by providing parameters during launch (for example, different SQL queries). Finally, we will use ECS Anywhere, which uses the open source amazon-ecs-agent to simplify how we can run our containers anywhere - in the Cloud, on premises or on other Clouds.

... ... Ping ECS Agent registered successfully! Container instance arn: "arn:aws:ecs:eu-west-1:704533066374:container-instance/ecs-anywhere-ec2-mydcecsclusterBB109425-r7l2mKClssuV/8dfb8700d9a1460dad403a321db6b5b9" You can check your ECS cluster here https://console.aws.amazon.com/ecs/home?region=eu-west-1#/clusters/ecs-anywhere-ec2-mydcecsclusterBB109425-r7l2mKClssuV # ok ########################## ########################## This script installed three open source packages that all use Apache License 2.0. You can view their license information here: - ECS Agent https://github.com/aws/amazon-ecs-agent/blob/master/LICENSE - SSM Agent https://github.com/aws/amazon-ssm-agent/blob/master/LICENSE - Docker engine https://github.com/moby/moby/blob/master/LICENSE ##########################

It looks like they have added something for this in the latest ECS agent: https://github.com/aws/amazon-ecs-agent/releases/tag/v1.50.0