turbopilot VS web-llm

fyi https://github.com/ravenscroftj/turbopilot

TurboPilot: self-hosted copilot clone which uses the library behind llama.cpp to run the 6 Billion Parameter Salesforce Codegen model in 4GiB of RAM. (https://github.com/ravenscroftj/turbopilot)

This repo has a potential

Right now https://github.com/ggerganov/llama.cpp is the dominant back-end for querying models, but forks and alternatives like https://github.com/ravenscroftj/turbopilot keep popping up. Increasingly, models submitted to huggingface explicitly note in their READMEs that the model is not compatible with llama.cpp, and that a different back-end must be used.

There's turbopilot. I haven't tried it yet, but it looks promising.

Turbopilot | open source LLM code completion engine and Copilot alternative

This 6B parameters tool (based on README) could be runned with 4 Gb of RAM. https://github.com/ravenscroftj/turbopilot

there is an interesting link https://github.com/ravenscroftj/turbopilot/wiki/Converting-and-Quantizing-The-Models , just wondering if anyone have done this with 16b and put the weights somewhere

So I was playing with MLC webllm locally. I got my mistral 7B model installed and quantised. Converted it using mlc lib to metal package for Apple chips. Now it takes only 3.5GB of memory

Maybe they're talking about https://github.com/mlc-ai/mlc-llm which is used for web-llm (https://github.com/mlc-ai/web-llm)? Seems to be using TVM.

Link to github for those who want to know about MLC straight from them. Web demo is cool but takes a long time to load first time. https://github.com/mlc-ai/web-llm

web-llm: Bringing large-language models and chat to web browsers. (https://github.com/mlc-ai/web-llm)

WebLLM - https://mlc.ai/web-llm/

which makes this works in the browser

https://mlc.ai/web-llm/#chat-demo