Our great sponsors
-
vowpal_wabbit
Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
It sounds like this may not need to even be distributed, but hard to tell. This may be a case where an out-of-core design is sufficient to get the job done. Usually this involves some multi-threaded programming with on set of threads doing the IO work, reading/writing to disk, while another set of threads doing the compute - and having an input/output queue. Out-of-core is in many ways an extension of the blocking/tiling approach, extended all the way to disk as the slowest level of memory access - with some extra programming differences due to it being disk instead of RAM. A few tools have built-in support for this like VW.