synapse-azure-data-explorer-101
ngods-stocks
Our great sponsors
synapse-azure-data-explorer-101 | ngods-stocks | |
---|---|---|
1 | 3 | |
4 | 354 | |
- | - | |
0.0 | 0.0 | |
almost 3 years ago | about 1 year ago | |
Jupyter Notebook | Jupyter Notebook | |
- | BSD 3-clause "New" or "Revised" License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
synapse-azure-data-explorer-101
-
Getting started with Azure Data Explorer and Azure Synapse Analytics for Big Data processing
Notebooks are available in this GitHub repo — https://github.com/abhirockzz/synapse-azure-data-explorer-101
ngods-stocks
-
I'm way over my head
I've worked for 3-4 years in positions where I helped structure ETLs, DWs and alike. However, I'm now on the cusp of being hired to help structure the area in a big investment fund here, helping the research area have an easier time focusing on their models. My previous experience led me to grasp DBT, SQL, and most of my experience came from using a Microsoft stack with SSIS, Analysis Services and the like. I'm feeling wayyyy over my head to start building this, and the multitude of possible stacks make me very afraid that I might overengineer this, and I will initially be alone in the area. What do I do? Fake it till I make it? I never lied in my resume, so it's not like they expect a senior with plenty of experience but still... I read this: https://github.com/zsvoboda/ngods-stocks And it seems like a good starter, albeit overly complex for our use case. I could use suggestions, people to talk to, etc. Please help
-
Apache Iceberg-based opensource analytics stack demo
Hi, I've created an opensource demo of a Docker-based local analytics stack that includes Apache Iceberg, Trino, Spark, Dagster (orchestration), Cube.dev (analytics model), Metabase (reports and dashboards), and Jupyter (data science notebook). I think that this is a pretty good starting point for Iceberg projects. Feel free to check it out at GitHub.
-
Iceberg + Spark + Trino + Dagster: modern, open-source data stack installation
I’m guessing that you use the Spark JDBC dataframes. Trino is in my opinion easier to use. You get SQL access to all pgsql tables with this simple config file. No need to write a piece of code for each table. The config above just maps the pgsql schema to a Trino schema. Then you configure Iceberg with another config file and you can do cross-schema SQL queries like create table pgsql.xyz from select * from iceberg.abc. Or you can use dbt that is based on SQL.
What are some alternatives?
azure-kusto-spark - Apache Spark Connector for Azure Kusto
practical-data-engineering - Practical Data Engineering: A Hands-On Real-Estate Project Guide
project - Predict how many points an European football team will end the season with, according to the characteristics of its players. Project for the Big Data Computing course at Sapienza University of Rome (2021-22)
amazon-emr-with-delta-lake - Amazon EMR Notebook to show how to read from and write to Delta tables with Amazon EMR
dracula - a brief analysis to the most common words in Dracula, by Bram Stoker
dbt-metabase - dbt + Metabase integration
workshop-introduction-to-machine-learning - Come ready to discover the goals and approaches of machine learning, and how to build effective algorithms and solutions!
udacity_bike_share_datalake_project - Azure Data Lake