A Distributed File System in Go Cut Average Metadata Memory Usage to 100 Bytes

• examples

  23 21,727 7.7 Python

  A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.

  

For “cloud-native” apps, JuiceFS is not needed.

S3 is not designed for intensive metadata operations, like listing, renaming etc. For these operations, you will need a somewhat POSIX-complaint system. For example, if you want to train on ImageNet dataset, the “canonical” way [1] is to extract the images and organize them into folders, class by class. The whole dataset is discovered by directory listing. This where JuiceFS shines.

Of course, if the dataset is really massive, you will mostly end-up with in-house solutions.

[1]: https://github.com/pytorch/examples/blob/main/imagenet/extra...

• minio

  99 44,220 9.9 Go

  The Object Store for AI Data Infrastructure

  

Looks like minio added this in 2022:

https://github.com/minio/minio/pull/15433

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project