Can we discuss MLOps, Deployment, Optimizations, and Speed?

• accelerate

  18 7,154 9.7 Python

  🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

  

accelerate is a best-in-class lib for deploying models, especially across multi-gpu and multi-node.

• transformers

  181 126,915 10.0 Python

  🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

  

transformers uses accelerate if you call it with device_map='auto'

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

• unsloth

  15 10,755 9.4 Python

  Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

  

The unsloth project offers some low-level optimizations for Llama et al, and as of today some prelim Mistral work (which I heard is the llama architecture?)

• llama.cpp

  789 59,389 10.0 C++

  LLM inference in C/C++

llama.cpp is a great resource for running Quants, and even though it's called llama, it's the goto backend for basically all LLMs right now (ctransformers is dead)

• DeepSpeed

  51 33,205 9.8 Python

  DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

  

DeepSpeed can handle parallelism concerns, and even offload data/model to RAM, or even NVMe (!?) . I'm surprised I don't see this project used more.

• ollama

  221 69,806 9.9 Go

  Get up and running with Llama 3, Mistral, Gemma, and other large language models.

  

• uniteai

  17 224 8.2 Python

  Your AI Stack in Your Editor

I recently went through the same with UniteAI, and had to swap ctransformers back out for llama.cpp

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project