quilt
demo-code
| quilt | demo-code | |
|---|---|---|
| 2 | 2 | |
| 1,314 | 28 | |
| 0.2% | - | |
| 9.5 | 0.0 | |
| about 17 hours ago | 3 months ago | |
| Jupyter Notebook | Jupyter Notebook | |
| Apache License 2.0 | Creative Commons Zero v1.0 Universal |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
quilt
-
Unstructured Data Governance for ML
Quilt: https://quiltdata.com/
-
Lessons From Building a Highly Reproducible Data Warehouse
We do a lot of SQL queries over (CSVs, Parquet, JSON) in S3 using AWS Athena. I work on an OSS project called Quilt that snapshots immutable datasets in S3 and supports reproducibility, idempotency, and functional data engineering.
demo-code
-
Difference between databricks and EMR +jupyper?
Sounds good - the setup steps are still a bit manual since it’s in preview, but CloudFormation support was just released and I’ll be posting a template in my demo-code repo next week.
-
Lessons From Building a Highly Reproducible Data Warehouse
Pretty nice article! I've been experimenting lately with using AWS CDK to deploy a big data stack including EMR on EKS and Airflow. The code is available on my GitHub - dacort/demo-code and I also just recorded a video stepping through how it's all put together.
What are some alternatives?
dvc - 🦉 ML Experiments and Data Management with Git
pyspark-on-aws-emr - The goal of this project is to offer an AWS EMR template using Spot Fleet and On-Demand Instances that you can use quickly. Just focus on writing pyspark code.
lakeFS - lakeFS - Data version control for your data lake | Git for data
aws-data-wrangler - pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL). [Moved to: https://github.com/aws/aws-sdk-pandas]
data - Data and code behind the articles and graphics at FiveThirtyEight
amazon-emr-with-delta-lake - Amazon EMR Notebook to show how to read from and write to Delta tables with Amazon EMR
data-engineering-nd - Projects of the Udacity Data Engineering Nanodegree Program.
AWS Data Wrangler - pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
data-engineering-book - Accumulated knowledge and experience in the field of Data Engineering
cc-notebooks - Various Jupyter notebooks about Common Crawl data
datasets - 🎁 5,400,000+ Unsplash images made available for research and machine learning
reactivesearch - Search UI components for React and Vue