sparktorch
openfl
| sparktorch | openfl | |
|---|---|---|
| 1 | 4 | |
| 334 | 660 | |
| - | 1.4% | |
| 2.5 | 8.7 | |
| 12 months ago | 2 days ago | |
| Python | Jupyter Notebook | |
| MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
sparktorch
-
Spark2 + pytorch on GPU
Was reading the documentation of sparktorch (https://github.com/dmmiller612/sparktorch) which says you need spark >= 2.4.4. But to the best of my knowledge spark2 doesn't have gpu compute capabilities. Does that mean it can only use cpu compute? Am I missing something?
openfl
-
Dectralized AI solutions/building blocks?
We have petals.ml for distributing running a single model large model for inference. We have [flower](flower.dev),tensor flow federated,openfl, etc for building federated learning.
- Launch HN: Flower (YC W23) – Train AI models on distributed or sensitive data
-
[R][P] OpenFL: An open-source framework for Federated Learning
Code for https://arxiv.org/abs/2105.06413 found: https://github.com/intel/openfl
What are some alternatives?
pytorch-lightning - Build high-performance AI models with PyTorch Lightning (organized PyTorch). Deploy models with Lightning Apps (organized Python to build end-to-end ML systems). [Moved to: https://github.com/Lightning-AI/lightning]
PySyft - Perform data science on data that remains in someone else's server
fastT5 - ⚡ boost inference speed of T5 models by 5x & reduce the model size by 3x.
FATE - An Industrial Grade Federated Learning Framework
torch2trt - An easy to use PyTorch to TensorRT converter
fedjax - FedJAX is a JAX-based open source library for Federated Learning simulations that emphasizes ease-of-use in research.
BERT-NER - Pytorch-Named-Entity-Recognition-with-BERT
tf-encrypted - A Framework for Encrypted Machine Learning in TensorFlow
polyaxon - MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle
FindTheMag - A tool to determine optimal projects for Gridcoin crunchers. Maximize your magnitude!
pytorch-lightning - Pretrain, finetune and deploy AI models on multiple GPUs, TPUs with zero code changes.
FedML - FEDML - The unified and scalable ML library for large-scale distributed training, model serving, and federated learning. FEDML Launch, a cross-cloud scheduler, further enables running any AI jobs on any GPU cloud or on-premise cluster. Built on this library, FEDML Nexus AI (https://fedml.ai) is your generative AI platform at scale.