beginner_de_project
sec-airflow-ingester
| beginner_de_project | sec-airflow-ingester | |
|---|---|---|
| 1 | 2 | |
| 389 | 12 | |
| - | - | |
| 2.8 | 0.0 | |
| about 1 month ago | about 2 years ago | |
| HCL | HCL | |
| MIT License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
beginner_de_project
-
Data Engineering project for beginners V2
Repo: https://github.com/josephmachado/beginner_de_project
sec-airflow-ingester
-
Amazon Managed Workflows for Apache Airflow
I've been using MWAA for a couple months now and so far, it has been decent for my use case. I'm pulling in remote security feeds and pushing data into s3 so it's a pretty simple workflow. In the event we outgrow the managed service or find some other reasons to migrate, we can do that pretty easily at that point. I also created a Terraform project to manage MWAA environments https://github.com/alias454/sec-airflow-ingester. Overall, using MWAA will cost a bit more but I haven't had to touch it other than trying to figure out what works and what doesn't when standing it up.
-
Sharing a new Terraform project for deploying Airflow in AWS using MWAA
I have been working with Airflow for the last month or two and created a Terraform deployment project for it. Sharing it here https://github.com/alias454/sec-airflow-ingester
What are some alternatives?
docker-airflow - Docker Apache Airflow
terraform-aws-eks - Terraform module to create AWS Elastic Kubernetes (EKS) resources 🇺🇦
AWS Data Wrangler - pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
terraform-best-practices - Terraform Best Practices for AWS users
Skytrax-Data-Warehouse - A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.
terraform-ecs - AWS ECS terraform module
dbd - dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.
terraform-aws-secure-baseline - Terraform module to set up your AWS account with the secure baseline configuration based on CIS Amazon Web Services Foundations and AWS Foundational Security Best Practices.
terraform-aws-mwaa - Terraform module for Amazon MWAA(Apache Airflow)