Top 22 Python Spark Projects
-
data-science-ipython-notebooks
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
-
Redash
Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
Its a bad idea use Redash as ETL and Data warehouse? Im not a data engineer just looking for a low/mid scale solution for experiment.
-
SonarQube
Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.
-
Project mention: [D] PyTorch Distributed Training Libraries: What are the current options? | reddit.com/r/MachineLearning | 2021-12-07
Check out Horovod - https://github.com/horovod/horovod
-
dev-setup
macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.
donnemartin - dev setup
-
Project mention: [D] Plug or Integrate a GNN Pytorch code base into Spark Cluster | reddit.com/r/MachineLearning | 2022-01-03
https://github.com/yahoo/TensorFlowOnSpark : check out if this project is useful for you.
-
If you like excessive use of square brackets.. I mean pandas, you might wanna check out Koalas. Koalas suppose to provide pandas datafrafe API implementation atop of Spark.
-
-
Scout APM
Less time debugging, more time building. Scout APM allows you to find and fix performance issues with no hassle. Now with error monitoring and external services monitoring, Scout is a developer's best friend when it comes to application development.
-
Project mention: [P] Announcing Feast 0.10: The simplest way to serve features in production | reddit.com/r/MachineLearning | 2021-04-15
Github: https://github.com/feast-dev/feast
-
Optimus
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark (by ironmussa)
-
Things get a bit more complicated on interactive sessions. We've created Sparkmagic compatible REST API so that Sparkmagic kernel could communicate with Lighter the same way as it does with Apache Livy. When a user creates an interactive session Lighter server submits a custom PySpark application which contains an infinite loop which constantly checks for new commands to be executed. Each Sparkmagic command is saved on Java collection, retrieved by the PySpark application through Py4J Gateway and executed.
-
Broad Institute of MIT and Harvard | Cambridge, MA | Associate Software Engineer | Onsite
We are seeking an associate software engineer interested in contributing to an open-source data visualization library for analyzing the biological impact human genetic variation. You will contribute to projects like gnomAD (https://gnomad.broadinstitute.org), the world's largest catalogue of human genetic variation used by hundreds of thousands of researchers and help us scale towards millions of genomes in the coming years. We are also developing next-generation tools for enabling genetic analyses of large biobanks across richly phenotyped individuals (https://genebass.org). In this role you will gain experience developing data-intensive web applications with Typescript, React, Python, Terraform, Google Cloud Platform, and will make use of the scalable data analysis library Hail (https://hail.is). Key to our success is growing a strong team with a diverse membership who foster a culture of continual learning, and who support the growth and success of one another. Towards this end, we are committed to seeking applications from women and from underrepresented groups. We know that many excellent candidates choose not to apply despite their capabilities; please allow us to enthusiastically counter this tendency.
Please provide a CV and links previous work or projects, ideally with contributions visible on Github.
email: [email protected]
-
listenbrainz-server
Server for the ListenBrainz project, including the front-end (javascript/react) code that it serves and all of the data processing components that LB uses.
-
fugue
A unified interface for distributed computing. Fugue executes SQL, Python, and Pandas code on Spark and Dask without any rewrites.
There's dask-sql, but I think it is being abandoned for fugue-project. I'm actually excited for this project as it is trying to provide a backend agnostic solution, which would seem like a difficult, lofty goal. I wish them luck.
-
Project mention: Monitor the stability of a pandas or spark dataframe | news.ycombinator.com | 2021-09-15
-
cape-python
Collaborate on privacy-preserving policy for data science projects in Pandas and Apache Spark
Well, many of the features in this project are simply wrappers around other libraries like this one. Therefore, the value proposition of this project would either have to be the automation aspect or the idea that you can shield the user from the details of how the implemented techniques work. I think both approaches are risky in this setting.
-
flytekit
Extensible Python SDK for developing Flyte tasks and workflows. Simple to get started and learn and highly extensible.
Based on references on setup Python project, package structure, and a production-level Python package, I refactor the package as shown below:
-
prosto
Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby
Most of the self-service or no-code BI, ETL, data wrangling tools are am aware of (like airtable, fieldbook, rowshare, Power BI etc.) were thought of as a replacement for Excel: working with tables should be as easily as working with spreadsheets. This problem can be solved when defining columns within one table: ``ColumnA=ColumnB+ColumnC, ColumnD=ColumnAColumnE`` we get a graph of column computations* similar to the graph of cell dependencies in spreadsheets.
Yet, the main problem is in working multiple tables: how can we define a column in one table in terms of columns in other tables? For example: ``Table1::ColumnA=FUNCTION(Table2::ColumnB, Table3::ColumnC)`` Different systems provided different answers to this question but all of them are highly specific and rather limited.
Why it is difficult to define new columns in terms of other columns in other tables? Short answer is that working with columns is not the relational approach. The relational model is working with sets (rows of tables) and not with columns.
One generic approach to working with columns in multiple tables is provided in the concept-oriented model of data which treats mathematical functions as first-class elements of the model. Previously it was implemented in a data wrangling tool called Data Commander. But them I decided to implement this model in the *Prosto* data processing toolkit which is an alternative to map-reduce and SQL:
https://github.com/asavinov/prosto
It defines data transformations as operations with columns in multiple tables. Since we use mathematical functions, no joins and no groupby operations are needed and this significantly simplifies and makes more natural the task of data transformations.
Moreover, now it provides *Column-SQL* which makes it even easier to define new columns in terms of other columns:
https://github.com/asavinov/prosto/blob/master/notebooks/col...
-
learningOrchestra
learningOrchestra is a distributed Machine Learning processing tool that facilitates and streamlines iterative processes in a Data Science project.
Project mention: Someone with a good experience in python can rate my code? | reddit.com/r/learnpython | 2021-02-16https://github.com/learningOrchestra/learningOrchestra/blob/552994056f2415ee54c24f2e1101fcca7fbd694b/microservices/code_executor_image/utils.py#L93-L94
-
openverse-catalog
Identifies and collects data on cc-licensed content across web crawl data and public apis.
Issue, Pull Request, Blog Post
-
-
Project mention: fastdbfs - An interactive command line client for Databricks DBFS | reddit.com/r/dataengineering | 2021-05-07
fastdbfs is an interactive command line client for accessing Databricks DBFS. It aims to be much more friendly and faster than the official CLI tool and also feature rich.
-
etl-markup-toolkit
ETL Markup Toolkit is a spark-native tool for expressing ETL transformations as configuration
Project mention: How do you serialize and save "transformations" in your pipeline? | reddit.com/r/dataengineering | 2021-08-31I have a side project (https://github.com/leozqin/etl-markup-toolkit, if you're interested) that takes transformations as yaml files and outputs step-level logs about each step of the transformation. I've always felt that both artifacts could made searchable using an ELK stack or something... Do you have similar artifacts? Or perhaps there's a way to turn SQL into a structured or semi-structured form to aid in searchability
Python Spark related posts
- How can you do efficient text preprocessing?
- [D] Plug or Integrate a GNN Pytorch code base into Spark Cluster
- Pyspark now provides a native Pandas API
- No-Code Self-Service BI/Data Analytics Tool
- FugueSQL: SQL-ish for pandas, dask, spark
- How do you serialize and save "transformations" in your pipeline?
- Alternative tools to DBT / SQL and Python for writing business logic? Trying to prevent creating a mountain of undocumented spaghetti
Index
What are some of the best open-source Spark projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | data-science-ipython-notebooks | 22,261 |
| 2 | Redash | 20,290 |
| 3 | horovod | 12,024 |
| 4 | dev-setup | 5,590 |
| 5 | TensorFlowOnSpark | 3,748 |
| 6 | koalas | 3,054 |
| 7 | dpark | 2,668 |
| 8 | feast | 2,656 |
| 9 | Optimus | 1,159 |
| 10 | sparkmagic | 1,076 |
| 11 | Hail | 772 |
| 12 | listenbrainz-server | 464 |
| 13 | fugue | 462 |
| 14 | popmon | 218 |
| 15 | cape-python | 144 |
| 16 | flytekit | 67 |
| 17 | prosto | 52 |
| 18 | learningOrchestra | 49 |
| 19 | openverse-catalog | 22 |
| 20 | ds2ai | 8 |
| 21 | fastdbfs | 4 |
| 22 | etl-markup-toolkit | 3 |
Are you hiring? Post a new remote job listing for free.
