uniteai VS llama.cpp

I recently went through the same with UniteAI, and had to swap ctransformers back out for llama.cpp

I maintain the uniteai project, and have implemented a custom backend for serving transformers-compatible LLMs. (That file's actually a great ultra-light-weight server if transformers satisfies your needs; one clean file).

UniteAI brings together speech recognition and document / code search. The major difference is your UI is your preferred text editor.

https://github.com/freckletonj/uniteai

In answer to the same question I built UniteAI https://github.com/freckletonj/uniteai

It's local first, and ties many different AIs into one text editor, any arbitrary text editor in fact.

It does speech recognition, which isn't useful for writing code, but is useful for generating natural language LLM prompts and comments.

It does CodeLlama (and any HuggingFace-based language model)

It does ChatGPT

It does Retrieval Augmented Gen, which is where you have a query that searches through eg PDFs, Youtube transcripts, code bases, HTML, local or online files, Arxiv papers, etc. It then surfaces passages relevant to your query, that you can then further use in conjunction with an LLM.

I don't know how mainstream LLM-powered software looks, but for devs, I love this format of tying in the best models as they're released into one central repo where they can all play off each others' strengths.

I'm working on UniteAI, a project to tie different AI capabilities into the editor, and it has a clean LSP Server.

*TL;DR*: chat with AI, code with AI, speak to AI (voice-to-text + vice versa), have AI search huge corpora or websites for you, all via an interface of collaborating on a text doc together in the editor you use now.

*Motivation*

I find the last year of AI incredibly heartening. Researchers are still regularly releasing SoTA models in disparate domains. Meta is releasing powerful Llama under generous provisions (As is the UAE with Falcon?!). And the Open Source community has shown a tidal wave of interest and effort into building things out of these tools (112k repos on GH mentioning ML!).

Facing this deluge of valuable things that communities are shepherding into the world, I wanted to incorporate them into my workflows, which as a software engineer, means my text editor.

*UniteAI*

So I started *UniteAI* https://github.com/freckletonj/uniteai, an Apache-2.0 licensed tool.

Check out the screencasts: https://github.com/freckletonj/uniteai#some-core-features

This project:

* Ties in to *any editor* via Language Server Protocol. Like collaborating in G-Docs, you collab with whatever AI directly in the document, all of you writing alongside each other concurrently.

* Like Copilot / Cursor, it can write code/text right in your doc.

* It supports *any Locally runnable model* (Llama family, Falcon, Finetunes, the 21k available models on HF, etc.)

* It supports *OpenAI/ChatGPT* via API key.

* *Speech-to-Text*, useful for writing prompts to your LLM

* You can do *Semantic Search* (Retrieval Augmented Generation) on many sources: local files, Arxiv, youtube transcripts, Project Gutenberg books, any online HTML, basically if you give it a URI, it can probably use it.

* You can trigger features easily via [key combos](https://github.com/freckletonj/uniteai#keycombos).

* Written in Python, so, much more generic than writing a bespoke `some_specific_editor` plugin.

*Caveat*

Since it always comes up, *AI is not perfect*. AI is a tool to augment your time, not replace it. It hallucinates, it lies, it bullshits, it writes bad code, it gives dangerous advice.

But can still do many useful things, and for me it is a *huge force multiplier.*

*You need a Human In The Loop*, which is why it's nice to work together iteratively on a text document, per, this project. You keep it on track.

*Why is this interesting*

These tools play well when used together:

* *Code example:* I can Voice-to-Text a function comment then send that to an LLM to write the function.

* *Code example 2:* I can chit chat about project architecture plans, and strategies, and libraries I should consider.

* *Documentation example:* I can retrieve relevant sections of my city's building code with a natural language query, then send that to an LLM to expound upon.

* *Authorship example*: I can have my story arcs and character dossiers in some markdown file, and use that guidance to contextualize an AI as it works with me for writing a story.

* *Entertainment example*: I told my AI it was a Dungeon Master, then over breakfast with friends, used Voice-to-Text and Text-to-Wizened-Wizard-Voice, and played a hillarious game. I still had to drive all this via a text doc, and handy key combos.

*RFC*

Installation instructions are on the repo: https://github.com/freckletonj/uniteai#quickstart-installing...

This is still nascent, and I welcome all feedback, positive or critical.

We have a community linked on the repo which you're invited to join.

I'd love love to chat with people who like this idea, use it, want to see other features, want to contribute their effort, want to file bug reports, etc.

A big part of my motivation in this is to socialize with like-minds, and build something cool.

*Thanks for checking this out!*

[UniteAI](https://github.com/freckletonj/uniteai) I think fits the bill for you.

This is my project, where the goal is to Unite your AI-stack inside your editor (so, Speech-to-text, Local LLMs, Chat GPT, Retrieval Augmented Gen, etc).

It's built atop a Language Server, so, while no one has made an IntelliJ client yet, it's simple to. I'll help you do it if you make a GH Issue!

UniteAI (github)

> Doing everything on-device would result in a horrible user experience. They might as well not participate in this generative AI rush at all if they hoped to keep it on-device.

On the contrary, I'm shocked over the last few months how "on device" on a Macbook Pro or Mac Studio competes plausibly with last year's early GPT-4, leveraging Llama 3 70b or Qwen2 72b.

There are surprisingly few things you "need" 128GB of so-called "unified RAM" for, but with M-series processors and the memory bandwidth, this is a use case that shines.

From this thread covering performance of llama.cpp on Apple Silicon M-series …

https://github.com/ggerganov/llama.cpp/discussions/4167

… "Buy as much memory as you can afford would be my bottom line!"

I'd love to use local models, but seems like most of the easy to use software out there (LM Studio, Backyard AI, koboldcpp) doesn't really play all that nicely with my Intel Arc GPU and it's painfully slow on my Ryzen 5 4500. Even my M1 MacBook isn't that fast at generating text with even 7B models.

I wonder if llama.cpp with SYCL could help, will have to try it out: https://github.com/ggerganov/llama.cpp/blob/master/README-sy...

But even if that worked, I'd still have the problem that IDEs and whatever else I have open already eats most of the 32 GB of RAM my desktop PC has. Whereas if I ran a small code model on the MacBook and connected to it through my PC, it'd still probably be too slow for autocomplete, when compared to GitHub Copilot and less accurate than ChatGPT or Phind for most stuff.

You're correct if you're focused exclusively on the work surrounding building foundation models to begin with. But if you take a broader view, having open models that we can legally fine tune and hack with locally has created a large and ever-growing community of builders and innovators that could not exist without these open models. Just take a look at projects like InvokeAI [0] in the image space or especially llama.cpp [1] in the text generation space. These projects are large, have lots of contributors, move very fast, and drive a lot of innovation and collaboration in applying AI to various domains in a way that simply wouldn't be possible without the open models.

[0] https://github.com/invoke-ai/InvokeAI

[1] https://github.com/ggerganov/llama.cpp

The first example will build an Embeddings database backed by llama.cpp vectorization.

and at some point (https://github.com/ggerganov/llama.cpp/issues/7444)

git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp LLAMA_CUDA=1 make -j

I focused on supporting Ollama, GPT4All, and LMStudio, all of which run smoothly on a Mac computer. Many of these tools are user-friendly wrappers around Llama.cpp, allowing easy model downloads and providing a REST interface to query the available models. Last week, I also added "👋🏼 Jan" support because HuggingFace has endorsed this provider out-of-the-box.

The output of the LLM is not just one token, but a statistical distribution across all possible output tokens. The tool you use to generate output will sample from this distribution with various techniques, and you can put constraints on it like not being too repetitive. Some of them support getting very specific about the allowed output format, e.g. https://github.com/ggerganov/llama.cpp/blob/master/grammars/... So even if the LLM says that an invalid token is the most likely next token, the tool will never select it for output. It will only sample from valid tokens.