dagster-example-pipeline
aws-data-wrangler
| dagster-example-pipeline | aws-data-wrangler | |
|---|---|---|
| 1 | 1 | |
| 64 | 3,559 | |
| - | - | |
| 0.0 | 10.0 | |
| about 2 years ago | 9 months ago | |
| Python | Python | |
| Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
dagster-example-pipeline
-
Developing in Dagster
The associated code repo can be found here
aws-data-wrangler
-
Interacting with Amazon S3 using AWS Data Wrangler (awswrangler) SDK for Pandas: A Comprehensive Guide
AWS Data Wrangler GitHub Repository: https://github.com/awslabs/aws-data-wrangler
What are some alternatives?
mlrun - MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.
boto3 - AWS SDK for Python
Apache Superset - Apache Superset is a Data Visualization and Data Exploration Platform [Moved to: https://github.com/apache/superset]
Trapheus - This tool automates restoration of RDS database instances from snapshots into any dev, staging or production environments. It supports individual RDS Snapshot as well as cluster snapshot restore operations.
AWS Data Wrangler - pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Pandas - Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
Prefect - The easiest way to build, run, and monitor data pipelines at scale.
ray_snowflake - Ray Data Connector for Snowflake
canarypy - CanaryPy - A light and powerful canary release for Data Pipelines
demo-code - Bits of code I use during live demos
portable-data-stack-dagster - A portable Datamart and Business Intelligence suite built with Docker, Dagster, dbt, DuckDB, PostgreSQL and Superset
aws-simple-websocket - Using AWS's API Gateway + Lambda to run a simple websocket application. For learning/testing.