practical-data-engineering
ngods-stocks
practical-data-engineering | ngods-stocks | |
---|---|---|
4 | 3 | |
455 | 373 | |
6.6% | - | |
7.7 | 0.0 | |
about 2 months ago | over 1 year ago | |
Jupyter Notebook | Jupyter Notebook | |
- | BSD 3-clause "New" or "Revised" License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
practical-data-engineering
- Show HN: Hands-On Data Engineering with a Real-Estate Project Guide
-
What's your favorite end-to-end tech stack?
Or in a side project: I combined an extensive amount of tools for web-scraping real-estates, uploading them to S3 with MinIO, Spark, and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualizing dashboards with Superset and managing everything with Dagster (blog, github)
-
✨ 5 Open Source Data Engineering Projects 🔥
2️⃣ Building a Data Engineering Project in 20 Minutes
-
Building a Data Engineering Project in 20 Minutes
The source-code you can find on practical-data-engineering for the data pipeline or in data-engineering-devops with all it’s details to set things up. Although not all is finished, you can observe the current status of the project on real-estate-project.
ngods-stocks
-
I'm way over my head
I've worked for 3-4 years in positions where I helped structure ETLs, DWs and alike. However, I'm now on the cusp of being hired to help structure the area in a big investment fund here, helping the research area have an easier time focusing on their models. My previous experience led me to grasp DBT, SQL, and most of my experience came from using a Microsoft stack with SSIS, Analysis Services and the like. I'm feeling wayyyy over my head to start building this, and the multitude of possible stacks make me very afraid that I might overengineer this, and I will initially be alone in the area. What do I do? Fake it till I make it? I never lied in my resume, so it's not like they expect a senior with plenty of experience but still... I read this: https://github.com/zsvoboda/ngods-stocks And it seems like a good starter, albeit overly complex for our use case. I could use suggestions, people to talk to, etc. Please help
-
Apache Iceberg-based opensource analytics stack demo
Hi, I've created an opensource demo of a Docker-based local analytics stack that includes Apache Iceberg, Trino, Spark, Dagster (orchestration), Cube.dev (analytics model), Metabase (reports and dashboards), and Jupyter (data science notebook). I think that this is a pretty good starting point for Iceberg projects. Feel free to check it out at GitHub.
-
Iceberg + Spark + Trino + Dagster: modern, open-source data stack installation
I’m guessing that you use the Spark JDBC dataframes. Trino is in my opinion easier to use. You get SQL access to all pgsql tables with this simple config file. No need to write a piece of code for each table. The config above just maps the pgsql schema to a Trino schema. Then you configure Iceberg with another config file and you can do cross-schema SQL queries like create table pgsql.xyz from select * from iceberg.abc. Or you can use dbt that is based on SQL.
What are some alternatives?
faros-community-edition - BI, API and Automation layer for your Engineering Operations data
amazon-emr-with-delta-lake - Amazon EMR Notebook to show how to read from and write to Delta tables with Amazon EMR
open-data-stack - Open Data Stack Projects: Examples of End to End Data Engineering Projects
synapse-azure-data-explorer-101 - Getting started with Azure Synapse and Azure Data Explorer
PANDAS-TUTORIAL - Jupyter Notebooks and Data Sets for Pandas Library
dbt-metabase - dbt + Metabase integration
data-engineering-devops - Full stack data engineering tools and infrastructure set-up
udacity_bike_share_datalake_project - Azure Data Lake
Data-Engineering-Projects - Personal Data Engineering Projects
data-engineering-zoomcamp - Free Data Engineering course!
HashtagCashtag - My Insight Data Engineering Fellowship project. I implemented a big data processing pipeline based on lambda architecture, that aggregates Twitter and US stock market data for user sentiment analysis using open source tools - Apache Kafka for data ingestions, Apache Spark & Spark Streaming for batch & real-time processing, Apache Cassandra f or storage, Flask, Bootstrap and HighCharts f or frontend.
H2O - H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.