data_engineering_project_template
Docker Compose
data_engineering_project_template | Docker Compose | |
---|---|---|
3 | 388 | |
176 | 32,367 | |
- | 0.7% | |
2.2 | 9.6 | |
10 months ago | 7 days ago | |
HCL | Go | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
data_engineering_project_template
-
Cloud services for small projects?
SCPs your code into EC2
-
Data engineering projects with template: Airflow, dbt, Docker, Terraform (IAC), Github actions (CI/CD) & more
Docker is used to containerize your application. For e.g. this Dockerfile is used to create a container and it specifies what OS it is, etc. You can run docker on any machine and you can think of it as running a separate os (not exactly, but close enough) on the machine. What Docker provides is the ability to replicate OS & its packages (e.g. python modules) across machines so that you don't run into "hey that worked on my computer" type issues.
Docker Compose
- Docker Compose: `version` is obsolete
-
12 Factor: 13 years later
Solutions are many, and could include Docker Compose, VS Code dev containers, Telepresence, Localstack or setting up temporary AWS accounts as a development environment for serverless applications.
-
Let's write a simple microservice in Clojure
Using Docker Compose to run Postgres and any third-party services locally provides a streamlined and consistent development environment. Developers can define services in a docker-compose.yml file, which enables them to configure and launch an entire stack with a single command. In this case, Postgres is encapsulated within a container with predefined configurations. Docker Compose also facilitates easy scaling, updates, and isolation of services, enhancing development efficiency and reducing the setup time for new team members or transitioning between projects. It encapsulates complex configurations, such as Postgres' performance monitoring and logging settings, in a manageable, version-controlled file, simplifying and replicating the service setup across different environments.
- Live reload em Go com docker e compile daemon
-
Docker compose, orchestrating and automating services
“Compose simplifies the control of your entire application stack, making it easy to manage services, networks, and volumes in a single, comprehensible YAML configuration file. Then, with a single command, you create and start all the services from your configuration file.” - Docker documentation
-
Hosting a simple docker-compose app with Nginx and generate a SSL with certbot on digitalocean droplet
curl -fsSL https://get.docker.com -o get-docker.sh sudo sh get-docker.sh # Install docker compose sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose # Apply executable permissions to the binary sudo chmod +x /usr/local/bin/docker-compose # Run Project docker-compose up -d
-
One Minute: Compose
Docker,
-
How to Set Up a Docker Container
This foundation now opens the door to even more powerful concepts. You can explore more advanced concepts such as container networking, streamlining the management of complex applications with Docker Compose, and how to make your application data persistent using volumes.
-
Use same Dockerfile for Dev & Production
In many projects that are containerized, especially in cases where development is also done locally with docker-compose, teams often have two Dockerfiles, 1 for Development, the other for Production. If you happen to have multiple environments like pre-prod, staging and so on, some teams could have different Dockerfiles for these environments.
-
How to Dockerise a NodeJS - TypeScript API || A Comprehensive Guide from Environment Setup to Deployment with a CI/CD Pipeline
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
What are some alternatives?
public-api-lists - A collective list of free APIs for use in software and web development 🚀
supervisor - Supervisor process control system for Unix (supervisord)
Flake8 - flake8 is a python tool that glues together pycodestyle, pyflakes, mccabe, and third-party plugins to check the style and quality of some python code.
LibreNMS-docker - LibreNMS Docker image
black - The uncompromising Python code formatter
terraform - Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
isort - A Python utility / library to sort imports.
Portainer - Making Docker and Kubernetes management easy.
Cloud-Init - unofficial mirror of Ubuntu's cloud-init
k3s - Lightweight Kubernetes
docker-cloudflared - Cloudflared proxy-dns Docker image