-
lithops
A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
At work for non-ML/non-GPU stuff I've been using Lithops for running code on dynamically-provisioned cloud resources (serverless or VM). It pickles your code & runtime variables, sends them to cloud storage, runs the code & downloads the results, all relatively transparently. You're just calling Python functions with Python objects on your local computer and not having to worry about deploying your code, packaging your data, etc. Better still, you can scale up for things like hyperparameter sweeps by just dispatching more calls in parallel, and it will provision more resources.
Related posts
-
Developed Python CLI to easily download Dutch point cloud data
-
PHP Doesn't Suck Anymore
-
Unlocking Profit Potential: Building an Arbitrage Betting Client with Hexagonal Architecture in Golang
-
Bountysource Stole at Least $17,000 from Open Source Developers
-
Fnug runs all your lints, tests and commands at once, in the terminal