Top 23 Spark Open-Source Projects
-
Have a look at Apache Spark
-
data-science-ipython-notebooks
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
data science ipython notebooks
-
Scout APM
Scout APM: A developer's best friend. Try free for 14-days. Scout APM uses tracing logic that ties bottlenecks to source code so you know the exact line of code causing performance issues and can get back to building a great product faster.
-
Redash
Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.
Project mention: How often do you use SQL query tool or service in your daily work? | reddit.com/r/SQL | 2021-11-21Regarding the subqueries: try https://tablum.io or https://redash.io, they materialize queried data so you can do a subquery multiple times.
-
Deeplearning4j
Model import deployment framework for retraining models (pytorch, tensorflow,keras) deploying in JVM Micro service environments, mobile devices, iot, and Apache Spark
Project mention: Does Java has similar project like this one in C#? (ml, data) | reddit.com/r/java | 2021-05-23Also, the website is now redirected to: https://deeplearning4j.konduit.ai/
-
If you just want to run tensorflow or pytorch for a Jupyter notebook, setting the environment shouldn't be difficult. I know that AWS has a marketplace of preconfigured images. However, you can go as advanced as setting up a cluster of gpu-equipped nodes to setup Horovod (https://github.com/horovod/horovod) to do distributed machine learning. Yes, there's a learning curve, but you cannot acquire this skillet any other way.
-
I'll be using a hosted Cube deployment on Cube Cloud to get aggregated data from an e-commerce dataset and Retool as the visualization tool to generate a metrics dashboard.
-
Thingsboard
Open-source IoT Platform - Device management, data collection, processing and visualization.
Project mention: (esp32+adxl325+sdcard)Storing sensor data in sd card(in txt file) and then sending it to cloud. | reddit.com/r/IOT | 2021-10-16Thingsboard and thingspeak are some more user friendly alternatives that MIGHT help accomplish your goal, I've not used them before.
-
-
H2O
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
-
dev-setup
macOS development environment setup: Easy-to-understand instructions with automated setup scripts for developer tools like Vim, Sublime Text, Bash, iTerm, Python data analysis, Spark, Hadoop MapReduce, AWS, Heroku, JavaScript web development, Android development, common data stores, and dev-based OS X defaults.
donnemartin - dev setup
-
Zeppelin
Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
[1] Apache Zeppelin http://zeppelin.apache.org/ [2] Zeppelin notebooks website http://zeppelin-notebook.com/. [3] Zeppelin notebooks git repo https://github.com/zjffdu/zeppelin-notebook
-
Alluxio (formerly Tachyon)
Alluxio, data orchestration for analytics and machine learning in the cloud
-
delta
An open-source storage layer that brings scalable, ACID transactions to Apache Spark™ and big data workloads. (by delta-io)
Project mention: Snowflake’s response to Databricks’ TPC-DS post | news.ycombinator.com | 2021-11-12>the delta also still keeps partitioning information in the hive metastore, while iceberg keeps it in storage, making it a far superior design.
Check out https://github.com/delta-io/delta/blob/3ffb30d86c6acda9b59b9... when you get a chance. You don't need hive metastore to query delta tables since all metadata for a Delta table is stored alongside the data
>they did not include features like optimizing small files
-
Intel BigDL for Spark which again is for Spark.
-
-
Project mention: Home lab with security monitoring tools? | reddit.com/r/netsecstudents | 2021-09-23
HELK can help for the SIEM and detection part
-
If you like excessive use of square brackets.. I mean pandas, you might wanna check out Koalas. Koalas suppose to provide pandas datafrafe API implementation atop of Spark.
-
Project mention: Microsoft AI Open-Sources ‘SynapseML’ For Developing Scalable Machine Learning Pipelines | reddit.com/r/artificial | 2021-11-18
Microsoft has announced the release of SynapseML, an open-source library that simplifies and speeds up the creation of machine learning (ML) pipelines. SynapseML can be used for building scalable and intelligent systems to solve various types of challenges, including anomaly detection, computer vision, deep learning, form and face recognition, Gradient boosting, microservice orchestration, model interpretability, reinforcement learning, and personalization, search and retrieval, speech processing, text analytics, and translation.
-
-
Project mention: [P] Announcing Feast 0.10: The simplest way to serve features in production | reddit.com/r/MachineLearning | 2021-04-15
Github: https://github.com/feast-dev/feast
-
Project mention: November 2021 workshops -- please comment about your preferences | reddit.com/r/Clojure | 2021-10-15
-
might be able to use one of the roaring bitmap implementations https://roaringbitmap.org/ via ffi, or port one to CL. been using them from clojure via java implementation, great lib.
-
deequ
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Project mention: PySpark - How to get Corrupted Records after Casting | reddit.com/r/dataengineering | 2021-09-28Deequ (this is the Scala version but they have PyDeequ also)
-
I know that libraries like Scodec and shapeless were rewritten practically from scratch for Scala 3, taking advantage of the next syntax and internals, as well as protoquill - a Scala 3 implementation of Quill.
Spark related posts
- Exception in thread "main" org.apache.spark.sql.AnalysisException: Cannot modify the value of a Spark config: spark.executor.memory;
- Microsoft AI Open-Sources ‘SynapseML’ For Developing Scalable Machine Learning Pipelines
- Kamu: A Git-like tool for building decentralized data pipelines
- Kamu: New take on Git for Data (Project Update)
- We turned Spark and Flink into Git for data
- No-Code Self-Service BI/Data Analytics Tool
- Snowflake’s response to Databricks’ TPC-DS post
Index
What are some of the best open-source Spark projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | Apache Spark | 31,417 |
| 2 | data-science-ipython-notebooks | 21,842 |
| 3 | Redash | 19,987 |
| 4 | Deeplearning4j | 12,250 |
| 5 | horovod | 11,881 |
| 6 | cube.js | 11,815 |
| 7 | Thingsboard | 10,406 |
| 8 | H2O | 5,631 |
| 9 | dev-setup | 5,566 |
| 10 | Zeppelin | 5,486 |
| 11 | Alluxio (formerly Tachyon) | 5,335 |
| 12 | delta | 3,840 |
| 13 | BigDL | 3,802 |
| 14 | Spark Notebook | 3,073 |
| 15 | HELK | 3,071 |
| 16 | koalas | 3,027 |
| 17 | SynapseML | 2,757 |
| 18 | dpark | 2,664 |
| 19 | feast | 2,490 |
| 20 | spark-nlp | 2,475 |
| 21 | RoaringBitmap | 2,427 |
| 22 | deequ | 1,988 |
| 23 | Quill | 1,982 |
Are you hiring? Post a new remote job listing for free.
