ggllm.cpp VS qdrant

I'd like to try https://huggingface.co/TheBloke/WizardLM-Uncensored-Falcon-40B-GGML with SillyTavern (running on Apple Silicon). The only way I've found to run Falcon 40B quantized on Apple Silicon is with https://github.com/cmp-nct/ggllm.cpp but I haven't figured out any way to get SillyTavern to use that as a local model. Does anyone know of a way to get this working?

It doesn't support Falcon right now, but there's a fork that does (https://github.com/cmp-nct/ggllm.cpp/).

To load falcon models with the new file format ggcc wich is a new file format similar to ggml, I'm using this tool: https://github.com/cmp-nct/ggllm.cpp Wich is a fork from : https://github.com/ggerganov/llama.cpp

The GGLLM fork seems to be the leading falcon winner for now [1]

It comes with its own variant of the GGML sub format "ggcv1" but there's quants available on HF [2]

Although if you have a GPU I'd go with the newly released AWQ quantization instead [3] the performance is better.

(I may or may not have a mild local LLM addiction - and video cards cost more then drugs)

[1] https://github.com/cmp-nct/ggllm.cpp

[2] https://huggingface.co/TheBloke/falcon-7b-instruct-GGML

[3] https://huggingface.co/abhinavkulkarni/tiiuae-falcon-7b-inst...

For base tooling, things like:

https://huggingface.co/ (finding models and downloading them)

https://github.com/ggerganov/llama.cpp (llama)

https://github.com/cmp-nct/ggllm.cpp (falcon)

For interactive work (art/chat/research/playing around), things like:

https://github.com/oobabooga/text-generation-webui/blob/main... (llama) (Also - they just added a decent chat server built into llama.cpp the project)

https://github.com/invoke-ai/InvokeAI (stable-diffusion)

Plus a bunch of hacked together scripts.

Some example models (I'm linking to quantized versions that someone else has made, but the tooling is in the above repos to create them from the published fp16 models)

https://huggingface.co/TheBloke/llama-65B-GGML

https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ

https://huggingface.co/TheBloke/Wizard-Vicuna-30B-Uncensored...

etc. Hugging face has quite a number, although some require filling out forms for the base models for tuning/training.

I did another test running LLM model (gpt4all-falcon) quantized to Q5_0 and Q5_1 to AMD GPU (https://huggingface.co/nomic-ai/gpt4all-falcon). I used this awesome project (forked from https://github.com/ggerganov/llama.cpp to https://github.com/cmp-nct/ggllm.cpp). I hipified the CUDA file into HIP code. and made some modifications on it (PR: https://github.com/cmp-nct/ggllm.cpp/pull/3). Checkout https://huggingface.co/nomic-ai/gpt4all-falcon

Great. Now that we have the embeddings, we need to store them in a vector database. We will be using Qdrant for this purpose. Qdrant is an open-source vector database that allows you to store and query high-dimensional vectors. The easiest way to get started with the Qdrant database is using the docker.

I took Qdrant for this project. The reason was that Qdrant stands for high-performance vector search, the best choice against use cases like finding similar function calls based on semantic similarity. Qdrant is not only powerful but also scalable to support a variety of advanced search features that are greatly useful to nuanced caching mechanisms like ours.

I'm currently looking to implement locally, using QDrant [1] for instance.

I'm just playing around, but it makes sense to have a runnable example for our users at work too :) [2].

[1]. https://qdrant.tech/

Also compare with qdrant's Rust implementation; they tout their performance. https://github.com/qdrant/qdrant/tree/master/lib/segment/src...

There are much better known examples, such as https://qdrant.tech/ and https://github.com/lancedb/lancedb

For more information, see our release notes. Qdrant is an open source project. We welcome your contributions; raise issues, or contribute via pull requests!

Initialize the Qdrant Client with in-memory storage. The collection name will be “imagebind_data” and we will be using cosine distance.

Qdrant is an open-source vector search engine optimized for performance and flexibility. It supports both exact and approximate nearest neighbor search, providing a balance between accuracy and speed for various AI and ML applications.