ngods-stocks
synapse-azure-data-explorer-101
| ngods-stocks | synapse-azure-data-explorer-101 | |
|---|---|---|
| 3 | 1 | |
| 373 | 4 | |
| - | - | |
| 0.0 | 0.0 | |
| over 1 year ago | almost 3 years ago | |
| Jupyter Notebook | Jupyter Notebook | |
| BSD 3-clause "New" or "Revised" License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
ngods-stocks
-
I'm way over my head
I've worked for 3-4 years in positions where I helped structure ETLs, DWs and alike. However, I'm now on the cusp of being hired to help structure the area in a big investment fund here, helping the research area have an easier time focusing on their models. My previous experience led me to grasp DBT, SQL, and most of my experience came from using a Microsoft stack with SSIS, Analysis Services and the like. I'm feeling wayyyy over my head to start building this, and the multitude of possible stacks make me very afraid that I might overengineer this, and I will initially be alone in the area. What do I do? Fake it till I make it? I never lied in my resume, so it's not like they expect a senior with plenty of experience but still... I read this: https://github.com/zsvoboda/ngods-stocks And it seems like a good starter, albeit overly complex for our use case. I could use suggestions, people to talk to, etc. Please help
-
Apache Iceberg-based opensource analytics stack demo
Hi, I've created an opensource demo of a Docker-based local analytics stack that includes Apache Iceberg, Trino, Spark, Dagster (orchestration), Cube.dev (analytics model), Metabase (reports and dashboards), and Jupyter (data science notebook). I think that this is a pretty good starting point for Iceberg projects. Feel free to check it out at GitHub.
-
Iceberg + Spark + Trino + Dagster: modern, open-source data stack installation
I’m guessing that you use the Spark JDBC dataframes. Trino is in my opinion easier to use. You get SQL access to all pgsql tables with this simple config file. No need to write a piece of code for each table. The config above just maps the pgsql schema to a Trino schema. Then you configure Iceberg with another config file and you can do cross-schema SQL queries like create table pgsql.xyz from select * from iceberg.abc. Or you can use dbt that is based on SQL.
synapse-azure-data-explorer-101
-
Getting started with Azure Data Explorer and Azure Synapse Analytics for Big Data processing
Notebooks are available in this GitHub repo — https://github.com/abhirockzz/synapse-azure-data-explorer-101
What are some alternatives?
practical-data-engineering - Practical Data Engineering: A Hands-On Real-Estate Project Guide
azure-kusto-spark - Apache Spark Connector for Azure Kusto
amazon-emr-with-delta-lake - Amazon EMR Notebook to show how to read from and write to Delta tables with Amazon EMR
project - Predict how many points an European football team will end the season with, according to the characteristics of its players. Project for the Big Data Computing course at Sapienza University of Rome (2021-22)
dbt-metabase - dbt + Metabase integration
dracula - a brief analysis to the most common words in Dracula, by Bram Stoker
udacity_bike_share_datalake_project - Azure Data Lake
workshop-introduction-to-machine-learning - Come ready to discover the goals and approaches of machine learning, and how to build effective algorithms and solutions!
data-engineering-zoomcamp - Free Data Engineering course!
project-atlas-sao-paulo - A project for the development of rich geospatial data from the city of São Paulo for use in Machine Learning models.
H2O - H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
pyspark_nlp_workshop - Instructions and code for the workshop "From Big Data to NLP Insights: Unlocking the Power of PySpark and Spark NLP"