How do you manage large training datasets?

Our great sponsors

WorkOS - The modern identity platform for B2B SaaS

InfluxDB - Power Real-Time Data Analytics at Scale

SaaSHub - Software Alternatives and Reviews

Our great sponsors

goofys

16 5,031 0.0 Go

a high-performance, POSIX-ish Amazon S3 file system written in Go

So, we just need to change the dataloader function a bit to make this work then. Did you try just mounting S3 using https://github.com/kahing/goofys. In this case, we need not even change the dataloader code. Not sure of the performance though.

WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project