Top 4 Python offloading Projects

FlexGen

39 8,999 3.0 Python

Running large language models on a single GPU for throughput-oriented scenarios.

Project mention: Run 70B LLM Inference on a Single 4GB GPU with This New Technique | news.ycombinator.com | 2023-12-03

mixtral-offloading

3 2,230 8.7 Python

Run Mixtral-8x7B models in Colab or consumer desktops

Project mention: DBRX: A New Open LLM | news.ycombinator.com | 2024-03-27

Waiting for Mixed Quantization with MQQ and MoE Offloading [1]. With that I was able to run Mistral 8x7B on my 10 GB VRAM rtx3080... This should work for DBRX and should shave off a ton of VRAM requirement.
1. https://github.com/dvmazur/mixtral-offloading?tab=readme-ov-...

WorkOS

workos.com sponsored

The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
unfaasener

1 25 7.7 Python

A lightweight framework that enables serverless users to reduce their bills by harvesting non-serverless compute resources such as their VMs, on-premise servers, or personal computers.

Project mention: UnFaaSener -- a lightweight framework for offloading serverless functions to non-serverless hosts | /r/serverless | 2023-10-18

GitHub Repo: https://github.com/ubc-cirrus-lab/unfaasener

monero-agent

4 24 0.0 Python

Monero hardware wallet protocol implementation for Trezor, agent

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).