Can I Run AI locally?

1. llama.cpp

   1 1,073 123,686 10.0 C++

   LLM inference in C/C++

   

   https://github.com/ggml-org/llama.cpp/releases - has mac binaries

   https://unsloth.ai/docs/models/qwen3.5 - running locally guide.

2. SaaSHub

   www.saashub.com sponsored

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. llmfit

   2 9 31,419 9.8 Rust

   Hundreds of models & providers. One command to find what runs on your hardware.

   

   Is this just llmfit but a web version of it?

   https://github.com/AlexsJones/llmfit

4. anubis-oss

   3 3 198 9.3 Swift

   Local LLM Testing & Benchmarking for Apple Silicon

   

   I made and opensourced a comprehensive benchmarking app for Apple Silicon to fill the void here on performance testing without having to download huge models first, and the data can be submitted to a public repository with one click, so we can gather info on a large scale. The releases are signed with an apple dev cert for safety. If the repo gets a few more stars it qualifies for homebrew distribution. Feel free to integrate the data feed anywhere.

   https://github.com/uncSoft/anubis-oss

   https://devpadapp.com/leaderboard.html

   https://devpadapp.com/explorer.html

   https://imgur.com/a/sBj2xWR

5. unfirehose-nextjs-logger

   4 1 4 9.8 TypeScript

   unfirehose next.js turbo to unfirehose dot org

   

   if you do, would you still want to collect data in a single pane of glass? see my open source repo for aggregating harness data from multiple machine learning model harnesses & models into a single place to discover what you are working on & spending time & money. there is plans for a scrobble feature like last.fm but for agent research & code development & execution.

   https://github.com/russellballestrini/unfirehose-nextjs-logg...

   thanks, I'll check for comments, feel free to fork but if you want to contribute you'll have to find me off of github, I develop privately on my own self hosted gitlab server. good luck & God bless.

6. Speech

   5 33 18,116 9.7 Python

   A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

   

   For ASR/STT on a budget, you want https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3 - it works great on CPU.

   I haven't tried on a raspberry pi, but on Intel it uses a little less than 1s of CPU time per second of audio. Using https://github.com/NVIDIA-NeMo/NeMo/blob/main/examples/asr/a... for chunked streaming inference, it takes 6 cores to process audio ~5x faster than realtime. I expect with all cores on a Pi 4 or 5, you'd probably be able to at least keep up with realtime.

   (Batch inference, where you give it the whole audio file up front, is slightly more efficient, since chunked streaming inference is basically running batch inference on overlapping windows of audio.)

7. llamafile

   6 73 25,585 8.5 C++

   Distribute and run LLMs with a single file.

   

   Personally I'd start with llamafile [0] then move to compiling your own llama.cpp.

   It's not as bad as you might think to compile llama.cpp for your target architecture and spin up an OpenAI compatible API endpoint. It even downloads the models for you.

   [0]: https://github.com/mozilla-ai/llamafile

8. Olares

   7 5 5,223 9.9 Go

   Open-Source Personal Cloud OS for Always-On Agents

   

   The 'spent 100 hours configuring' part hits home. That fragmentation is exactly why we started building Olares (https://github.com/beclab/Olares).

   It’s basically an open-source OS layer that standardizes the local AI stack—Kubernetes (K3s) for orchestration, standardized model serving, and GPU scheduling. The goal is to stop fiddling with Python environments/drivers and just treat local agents like standardized containers. It runs on Mac Minis or dedicated hardware.

9. LlamaEdge

   8 3 1,651 6.8 Rust

   The easiest & fastest way to run customized and fine-tuned LLMs locally or on the edge

   

   a lighter-weight alternative of docker and python is the Rust+Wasm stack https://github.com/LlamaEdge/LlamaEdge

Suggest a related project