data-engineering-nd
quilt
| data-engineering-nd | quilt | |
|---|---|---|
| 7 | 2 | |
| 8 | 1,314 | |
| - | 0.3% | |
| 0.0 | 9.5 | |
| about 2 years ago | 3 days ago | |
| Jupyter Notebook | Jupyter Notebook | |
| - | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
data-engineering-nd
-
Data Pipelines explained with Airflow
In the following lines I am doing a write-up about everything I learned about data pipelines at the Udacity online class. It gives a general overview about data pipelines and provides also the core concepts of Airflow and some links to code examples on github.
-
Run Spark locally with Docker
You can find the code also here.
-
Spark for beginners - and you
Coding examples here.
-
Cloud computing quickstart
IaC - Infrastructure as Code Example with boto3
-
Relational data models
If you need a higher resolution please use this page
-
Get started with data engineering
In addition you can find the according exercises on my github account.
-
Structured Query Language
If the resolution here is too low - in case you really want to read it - you can find a higher resolution here.
quilt
-
Unstructured Data Governance for ML
Quilt: https://quiltdata.com/
-
Lessons From Building a Highly Reproducible Data Warehouse
We do a lot of SQL queries over (CSVs, Parquet, JSON) in S3 using AWS Athena. I work on an OSS project called Quilt that snapshots immutable datasets in S3 and supports reproducibility, idempotency, and functional data engineering.
What are some alternatives?
udacimak - Udacity Nanodegree and Course Downloader
dvc - 🦉 ML Experiments and Data Management with Git
Data-Engineering-Projects - Personal Data Engineering Projects
lakeFS - lakeFS - Data version control for your data lake | Git for data
migrate - Database migrations. CLI and Golang library.
data - Data and code behind the articles and graphics at FiveThirtyEight
Apache Hadoop - Apache Hadoop
demo-code - Bits of code I use during live demos
data-engineering-book - Accumulated knowledge and experience in the field of Data Engineering
datasets - 🎁 5,400,000+ Unsplash images made available for research and machine learning
rihal-challenges - This repository is used to house Rihal's challenges for hiring.
nessie - Nessie: Transactional Catalog for Data Lakes with Git-like semantics