Can we discuss MLOps, Deployment, Optimizations, and Speed?

• accelerate

  19 7,789 9.7 Python

  🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

  

  accelerate is a best-in-class lib for deploying models, especially across multi-gpu and multi-node.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• transformers

  188 132,909 10.0 Python

  🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

  

  transformers uses accelerate if you call it with device_map='auto'

• unsloth

  17 16,378 9.7 Python

  Finetune Llama 3.2, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

  

  The unsloth project offers some low-level optimizations for Llama et al, and as of today some prelim Mistral work (which I heard is the llama architecture?)

• llama.cpp

  823 65,845 10.0 C++

  LLM inference in C/C++

  llama.cpp is a great resource for running Quants, and even though it's called llama, it's the goto backend for basically all LLMs right now (ctransformers is dead)

• DeepSpeed

  51 34,987 9.7 Python

  DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

  

  DeepSpeed can handle parallelism concerns, and even offload data/model to RAM, or even NVMe (!?) . I'm surprised I don't see this project used more.

• ollama

  301 92,524 9.9 C++

  Get up and running with Llama 3.2, Mistral, Gemma 2, and other large language models.

  

• uniteai

  17 268 4.8 Python

  Your AI Stack in Your Editor

  I recently went through the same with UniteAI, and had to swap ctransformers back out for llama.cpp

Suggest a related project