llama.cpp VS Open-Assistant

KodiBot is a desktop app that enables users to run their own AI chat assistants locally and offline on Windows, Mac, and Linux operating systems. KodiBot is a standalone app and does not require an internet connection or additional dependencies to run local chat assistants. It supports both Llama.cpp compatible models and OpenAI API.

There are already some implementations out there which attempt to accomplish this!

Here's an example: https://github.com/silphendio/sliced_llama

A gist pertaining to said example: https://gist.github.com/silphendio/535cd9c1821aa1290aa10d587...

Here's a discussion about integrating this capability with ExLlama: https://github.com/turboderp/exllamav2/pull/275

And same as above but for llama.cpp: https://github.com/ggerganov/llama.cpp/issues/4718#issuecomm...

For those who might not be aware of this, there is also an open source project on GitHub called "Twinny" which is an offline Visual Studio Code plugin equivalent to Copilot: https://github.com/rjmacarthy/twinny

It can be used with a number of local model services. Currently for my setup on a NVIDIA 4090, I'm running both the base and instruct model for deepseek-coder 6.7b using 5_K_M Quantization GGUF files (for performance) through llama.cpp "server" where the base model is for completions and the instruct model for chat interactions.

llama.cpp: https://github.com/ggerganov/llama.cpp/

deepseek-coder 6.7b base GGUF files: https://huggingface.co/TheBloke/deepseek-coder-6.7B-base-GGU...

deepseek-coder 6.7b instruct GGUF files: https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct...

If I'm reading this correctly, they had to discard Llama 2 answers and only use GPT-3.5 given answers to test the hypothesis.

GPT-3.5 answering questions through the OAI API alone is not an acceptable method of testing problem solving ability across a range of temperatures. OpenAI does some blackbox wizardry on their end.

There are many complex and clever sampling techniques for which temperature is just one (possibly dynamic) component

One example from the llama.cpp codebase is dynamic temperature sampling

https://github.com/ggerganov/llama.cpp/pull/4972/files

Not sure what you mean by whole model state given that there are tens of thousands of possible tokens and the models have billions of parameters in XX,XXX-dimensional space. How many queries across how many sampling methods might you need? Err..how much time? :)

git clone https://github.com/ggerganov/llama.cpp.git

Even history rewrites would be visible with Githu's new Activity tab, e.g., see the two force-pushes in llama.cpp https://github.com/ggerganov/llama.cpp/activity So, while, yes, git history can be rewritten, commits pushed to Github can effectively never be deleted. Personally, I find this a downside. Think, personal information, etc. But, in this case, it is helpful. Of course, the repository is suspended right now, so the Activity cannot be checked.

https://github.com/ggerganov/llama.cpp/issues/91

Justine was kicked out of llama.cpp for introducing changes before the rest of the maintainers approved them.

Much drama has been had since then and I've lost interest in both projects.

It's just been exhausting wanting to build good software without ego in this space. Everyone is trying to get rich quick before the inevitable AI ice age starts and all these skill are again useless.

Here are links to the most recent pull requests sent

    https://github.com/ggerganov/llama.cpp/pull/6414

It fascinating to me that coming up on a year since Sapphire Rapids has been available in the public cloud, developers are still targeting AVX512 when they should be targeting VNNI and AMX.

https://github.com/ggerganov/llama.cpp/issues/2555

it seems that BERT can be run on the llama.cpp platform https://github.com/ggerganov/llama.cpp/pull/5423

so presumably those models could benefit from the speed ups described in OP article when running on CPU

For open assistant, the code: https://github.com/LAION-AI/Open-Assistant/tree/main/inference

https://open-assistant.io seems to have the same guardrails, as ChatGPT. Tried it on several prompts and it wouldn't comply.

I also suggest Open Assistant (https://open-assistant.io) which is not even filtered (whenever you trigger a moralist filter, it's because of its training data, and it's not on purpose).

Open-Assistant [https://open-assistant.io/]

ok ima just do this instead: https://www.getdevkit.com/devgpt https://stunning.so/dashboard https://promptchainer.io/dashboard/settings https://www.kaggle.com/ https://leetcode.com https://app.sttabot.io/bot/ai_chatbot_agent_mem_debug_auditer/ https://app.questmate.com/home https://app.gitwit.dev/projects# https://agentgpt.reworkd.ai/ https://socra.com/journey https://dashboard.cohere.ai/playground/embed https://studio.ai21.com/playground/complete?modelId=j2-jumbo https://dust.tt https://chat.forefront.ai/ https://github.com/jina-ai/dev-gpt https://whattheai.tech/ https://www.chatbotui.com/ https://cebra.ai/ https://thesamur.ai/?bl=en https://chat.getgptapi.com/#/ https://www.patience.ai/ https://aiagent.app/ https://unlimitedgpt.co/ https://www.futuredesk.io/welcome https://stunning.so/ https://www.wellknown.ai/ https://huggingface.co/spaces/ysharma/ChatGPT4 https://realpython.com/jupyter-notebook-introduction/ https://github.com/miurla/babyagi-ui https://chat.lmsys.org/ https://www.loom.com/use-case/marketing https://thesamur.ai/mygpt https://open-assistant.io/dashboard https://www.cognosys.ai/ https://github.com/LAION-AI/Open-Assistant https://github.com/muellerberndt/micro-gpt https://github.com/oobabooga/text-generation-webui https://huggingface.co/databricks/dolly-v2-12b https://github.com/nomic-ai/gpt4all https://dashboard.cohere.ai/playground/generate https://www.theindieai.com/ https://leetcode.com/accounts/login/?next= https://neetcode.io/

Open Assistant by LAION AI on GitHub

https://open-assistant.io - oasst-sft-6-llama-30b