practical-data-engineering
open-data-stack
| practical-data-engineering | open-data-stack | |
|---|---|---|
| 4 | 3 | |
| 453 | 64 | |
| 6.2% | - | |
| 7.7 | 0.0 | |
| about 2 months ago | 11 months ago | |
| Jupyter Notebook | Python | |
| - | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
practical-data-engineering
- Show HN: Hands-On Data Engineering with a Real-Estate Project Guide
-
What's your favorite end-to-end tech stack?
Or in a side project: I combined an extensive amount of tools for web-scraping real-estates, uploading them to S3 with MinIO, Spark, and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualizing dashboards with Superset and managing everything with Dagster (blog, github)
-
✨ 5 Open Source Data Engineering Projects 🔥
2️⃣ Building a Data Engineering Project in 20 Minutes
-
Building a Data Engineering Project in 20 Minutes
The source-code you can find on practical-data-engineering for the data pipeline or in data-engineering-devops with all it’s details to set things up. Although not all is finished, you can observe the current status of the project on real-estate-project.
open-data-stack
-
What's your favorite end-to-end tech stack?
I distilled my open data stack into four core tools: airbyte, dbt, metabase and dagster (blog, github, example)
-
The Open Data Stack Distilled into Four Core Tools
This has nothing to do with marketing, as a data engineer I tried all the tools and offer multiple open-source projects and real estate scraper to get started yourself. At some point everyone has to make a choice, these are mine. Happy to hear your favorites too :). There is also great value (IMO) in a distilled list for people who do not spend all their time monitoring all the new tools out there.
- Show HN: Open Data Stack: Examples of End-to-End DE Projects
What are some alternatives?
faros-community-edition - BI, API and Automation layer for your Engineering Operations data
go-hnsw
ngods-stocks - New Generation Opensource Data Stack Demo
PANDAS-TUTORIAL - Jupyter Notebooks and Data Sets for Pandas Library
data-engineering-devops - Full stack data engineering tools and infrastructure set-up
Data-Engineering-Projects - Personal Data Engineering Projects
HashtagCashtag - My Insight Data Engineering Fellowship project. I implemented a big data processing pipeline based on lambda architecture, that aggregates Twitter and US stock market data for user sentiment analysis using open source tools - Apache Kafka for data ingestions, Apache Spark & Spark Streaming for batch & real-time processing, Apache Cassandra f or storage, Flask, Bootstrap and HighCharts f or frontend.
WebCrawlerForOnlineInflation - Price Crawler - Tracking Price Inflation
Hugo - The world’s fastest framework for building websites.