Stop vibe-debugging. Every exception, every backtrace, grouped so you see patterns, not noise. Learn more →
Top 23 Python Spark Projects
-
data-science-ipython-notebooks
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
-
AppSignal
AppSignal knows why the f*#k it crashed. Stop vibe-debugging. Every exception, every backtrace, grouped so you see patterns, not noise.
-
Redash
Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
Project mention: I ran my code analyzer on 9 codebases it had never seen. It found 3 real bugs, and embarrassed me first. | dev.to | 2026-07-16redash ships a client method calling GET /api/dashboards/recent with no backend route serves it. (Latent: the method has no callers today, which the issue says plainly. #7769)
-
ChuanhuChatGPT
GUI for ChatGPT API and many LLMs. Supports agents, file-based QA, GPT finetuning and query with web search. All with a neat UI.
-
We should also declare an interest. Interlace's entire intermediate representation is sqlglot — the parser that came with the Tobiko acquisition. Every model we compile is parsed, canonicalised and transpiled by a library Fivetran now stewards. We are not neutral observers of this consolidation; we are downstream of it.
-
Mage
🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai
-
dev-setup
macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
-
spark-operator
Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.
-
splink
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
Project mention: Building an investing knowledge graph, part 4: building the resolver | dev.to | 2026-08-31Splink is the open source probabilistic record linkage library the model is built on. It uses DuckDB as the computation backend for the pair scoring, which handles the candidate generation and blocking step — filtering the space of possible pairs before running the full model.
-
fugue
A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
-
Optimus
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark (by ironmussa)
-
-
listenbrainz-server
Server for the ListenBrainz project, including the front-end (javascript/react) code that it serves and all of the data processing components that LB uses.
Project mention: Why media fans want to escape algorithms with CDs, DVDs and vinyl | news.ycombinator.com | 2026-10-02> I do wish I could figure out some sort of recommendation system to find new magic, though. Obviously not to steam new/unowned songs, just to point me in a direction next time I go to the store.
Navidrome can scrobble to https://listenbrainz.org/ which has a recommendations mechanism
Or set up "soulsync" and just use the library scanning, scrobbling, and recommendations / artist watching features if you don't want to pirate the music as well.
-
streamify
A data engineering project with Kafka, Spark Streaming, dbt, Docker, Airflow, Terraform, GCP and much more!
-
-
enterprise_gateway
A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
-
-
-
flytekit
Extensible Python SDK for developing Flyte tasks and workflows. Simple to get started and learn and highly extensible.
-
-
-
cape-dataframes
Privacy transformations on Spark and Pandas dataframes backed by a simple policy language.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python Spark discussion
Python Spark related posts
-
How to Build an Open Lakehouse on Your Laptop
-
The same pipeline, minus the cluster
-
Building an investing knowledge graph, part 4: building the resolver
-
Building an investing knowledge graph, part 3: same company, seven names
-
Loop: life DWH, logical and physical models. Part 1
-
PyTorch Library for Running LLM on Intel CPU and GPU
-
Splink: Fast, accurate, scalable probabilistic data linkage
-
A note from our sponsor - AppSignal
www.appsignal.com | 7 Oct 2026
Index
What are some of the best open-source Spark projects in Python? This list will help you:
| # | Project | Stars |
|---|---|---|
| 1 | data-science-ipython-notebooks | 29,349 |
| 2 | Redash | 28,831 |
| 3 | ChuanhuChatGPT | 15,275 |
| 4 | sqlglot | 9,650 |
| 5 | Mage | 8,829 |
| 6 | dev-setup | 6,265 |
| 7 | TensorFlowOnSpark | 3,845 |
| 8 | koalas | 3,375 |
| 9 | spark-operator | 3,153 |
| 10 | splink | 2,452 |
| 11 | fugue | 2,169 |
| 12 | Optimus | 1,536 |
| 13 | sparkmagic | 1,366 |
| 14 | listenbrainz-server | 1,026 |
| 15 | streamify | 889 |
| 16 | dingo | 757 |
| 17 | enterprise_gateway | 667 |
| 18 | datacompy | 658 |
| 19 | popmon | 511 |
| 20 | flytekit | 320 |
| 21 | visions | 223 |
| 22 | geopyspark | 179 |
| 23 | cape-dataframes | 175 |
Sponsored
AppSignal knows why the f*#k it crashed.
Stop vibe-debugging. Every exception, every backtrace, grouped so you see patterns, not noise.
www.appsignal.com