sec-airflow-ingester
beginner_de_project
sec-airflow-ingester | beginner_de_project | |
---|---|---|
2 | 1 | |
12 | 401 | |
- | - | |
0.0 | 2.8 | |
about 2 years ago | about 2 months ago | |
HCL | HCL | |
- | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
sec-airflow-ingester
-
Amazon Managed Workflows for Apache Airflow
I've been using MWAA for a couple months now and so far, it has been decent for my use case. I'm pulling in remote security feeds and pushing data into s3 so it's a pretty simple workflow. In the event we outgrow the managed service or find some other reasons to migrate, we can do that pretty easily at that point. I also created a Terraform project to manage MWAA environments https://github.com/alias454/sec-airflow-ingester. Overall, using MWAA will cost a bit more but I haven't had to touch it other than trying to figure out what works and what doesn't when standing it up.
-
Sharing a new Terraform project for deploying Airflow in AWS using MWAA
I have been working with Airflow for the last month or two and created a Terraform deployment project for it. Sharing it here https://github.com/alias454/sec-airflow-ingester
beginner_de_project
-
Data Engineering project for beginners V2
Repo: https://github.com/josephmachado/beginner_de_project
What are some alternatives?
terraform-aws-eks - Terraform module to create AWS Elastic Kubernetes (EKS) resources πΊπ¦
docker-airflow - Docker Apache Airflow
terraform-best-practices - Terraform Best Practices for AWS users
AWS Data Wrangler - pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
terraform-ecs - AWS ECS terraform module
Skytrax-Data-Warehouse - A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.
terraform-aws-secure-baseline - Terraform module to set up your AWS account with the secure baseline configuration based on CIS Amazon Web Services Foundations and AWS Foundational Security Best Practices.
dbd - dbd is a database prototyping tool that enables data analysts and engineers to quickly load and transform data in SQL databases.
terraform-aws-mwaa - Terraform module for Amazon MWAA(Apache Airflow)