webdataset
PySyft
Our great sponsors
| webdataset | PySyft | |
|---|---|---|
| 7 | 7 | |
| 1,962 | 9,253 | |
| 7.4% | 1.1% | |
| 8.8 | 10.0 | |
| 17 days ago | 6 days ago | |
| Python | Python | |
| BSD 3-clause "New" or "Revised" License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
webdataset
-
How to use data stored in a (private) S3 Bucket for training?
As an alternative, I've looked into using WebDataset, but couldn't figure out how to access data that is stored in a private bucket.
- [D] Title: Best tools and frameworks for working with million-billion image datasets?
-
[D] Training networks on extremely large datasets (10+TB)?
You can try webdataset (https://github.com/webdataset/webdataset).
- Question: TIFF image dataset - size in RAM.
-
How to upload large amounts of data to a server?
compress it to .tar format and then load it as a webdataset
-
Does mit 6.824 help for distributed deep learning?
Would guess not but there should be some good niche resources: check out the introductory videos here https://github.com/webdataset/webdataset
-
How to effectively load a large text dataset with PyTorch?
I found a pretty good solution that is similar to the TFRecord from Tensorflow. You just need to load the data, tokenized it, and save the arrays in shards with webdataset package.
PySyft
-
A Better Mastodon Client
https://github.com/OpenMined/PySyft - Federated Learning data science
Incentives are much harder but smart contracts can handle the tech part.
Going this route eventually you quickly have "quantum AI app store" and your system of government is a 12GB download. Can't even say if it's a good idea compared to e.g. anarcho-primitivism.
-
Just about conspirancy theories... Can you say this guy isn't rigth?
Something that maybe can help keeping sensor specs secret while still getting critical information out: https://github.com/OpenMined/PySyft
-
I made a YT video showing how to host your own super accurate (microsecond) network time (NTP) server using the PPS output of a $12 GPS module
Love this kind of project. To me this is just like https://github.com/open-quantum-safe/oqs-demos/ or https://github.com/OpenMined/PySyft or even k3s so often mentioned in this sub in the sense that I personally don't have a need for it. Yet I find it amazing that us, random curious geeks, have access to this kind of mind blowing technologies for basically free.
-
Help on creating a Federated Recommender System
Or do I have to actually simulate the whole client server thing because thats how these frameworks do it - Flower and Pysyft .
-
Integration test: Complexity of privacy-preserving bird call bio-sensor for distributed ecological monitoring?
Some of the technologies which could be integrated include differential privacy, distributed online machine learning, misinformation resilience and multi-party computation, all within the context of smart contracts and bioinformatics.
-
Google Strikes Deal With Hospital Chain to Develop Healthcare Algorithms
I think this is how it will be done. Look up PySift for how we can extract high-level insights from private datasets while preserving granular privacy.
-
Is it even possible to have a service as "intelligent" as Google while still being privacy respecting?
What you are talking about is privacy-focused fed ML. Google FLOC is actually trying to achieve something similar. If you are interested in building something for yourself, check this out. https://github.com/OpenMined/PySyft
What are some alternatives?
Practical_RL - A course in reinforcement learning in the wild
fastai - The fastai deep learning library
NYU-DLSP20 - NYU Deep Learning Spring 2020
openfl - The Open Flash Library for creative expression on the web, desktop, mobile and consoles.
Made-With-ML - Learn how to design, develop, deploy and iterate on production-grade ML applications.
AIDungeon - Infinite adventures await!
ffcv - FFCV: Fast Forward Computer Vision (and other ML workloads!)
99-ML-Learning-Projects - A list of 99 machine learning projects for anyone interested to learn from coding and building projects
openfl - An open framework for Federated Learning.
ModelNet40-C - Repo for "Benchmarking Robustness of 3D Point Cloud Recognition against Common Corruptions" https://arxiv.org/abs/2201.12296
Watermark-Removal-Pytorch - 🔥 CNN for Watermark Removal using Deep Image Prior with Pytorch 🔥.