llama-cpp-python VS basaran

There's a Python binding for llama.cpp which is actively maintained and has worked well for me: https://github.com/abetlen/llama-cpp-python

I'll share the core bit that took a while to figure out the right format, my main script is a hot mess using embeddings with SentenceTransformer, so I won't share that yet. E.g: last night I did a PR for llama-cpp-python that shows how Phi might be used with JSON only for the author to write almost exactly the same code at pretty much the same time. https://github.com/abetlen/llama-cpp-python/pull/1184

Python Bindings for llama.cpp

If the model could be made to work with llama.cpp, then https://github.com/abetlen/llama-cpp-python might be more compact. llama.cpp only supports a limited list of model types though.

https://github.com/abetlen/llama-cpp-python has a web server mode that replicates openai's API iirc and the readme shows it has docker builds already.

LocalAI https://localai.io/ and LMStudio https://lmstudio.ai/ both have fairly complete OpenAI compatibility layers. llama-cpp-python has a FastAPI server as well: https://github.com/abetlen/llama-cpp-python/blob/main/llama_... (as of this moment it hasn't merged GGUF update yet though)

I went with Python, llama-cpp-python, since my goal is just to get a small project up and running locally.

What are the current best "no reinventing the wheel" approaches to have Langchain use an LLM through a locally hosted REST API, the likes of Oobabooga or hyperonym/basaran with streaming support for 4-bit GPTQ?

Disclaimer: I am curating LLM-tools on github [1]

A few thoughts:

* allow for custom endpoint URLs, this way people can use open source LLMs with a fake openAI API backend like basaran[2] or llama-api-server[3]

* look into better embedding methods for info-retrieval like InstructorEmbeddings or Document Summary Index

* Don't use a single embedding per content item, use multiple to increase retrieval quality

1 https://github.com/underlines/awesome-marketing-datascience/...

2 https://github.com/hyperonym/basaran

3 https://github.com/iaalm/llama-api-server

open-source alternative to the OpenAI text completion API (https://github.com/hyperonym/basaran)

Guanaco-65B[0] using Basaran[1] for your OpenAI compatible API. You can use any ChatGPT front-end which lets you change the OpenAI endpoint URL.

[0] An fp4 finetune of LLaMA-30B by Tim Dettmers

[1] https://github.com/hyperonym/basaran

For lm-eval, I think you'd either need to take GPTQ's inference script and shim it into a model: https://github.com/EleutherAI/lm-evaluation-harness/tree/master/lm_eval/models or you might be able to use a project like https://github.com/hyperonym/basaran and then you could use the gpt3 model...

There are also: - Basaran repo: "Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models". "...Compatibility with OpenAI API and client libraries..."; - llama-cpp-python repo: "Simple Python bindings for @ggerganov's llama.cpp library...". "...OpenAI-like API...".

I want to start with a totally freely available model, so again, that excludes things like LLaMA where the weights are only available through a wait list. The two models that most get my attention and (I think, and hope) fit my criteria of open availability are Cerebras-GPT (13b) and Bloomz (7b). The tools to process and fine-tune that seem most feasible to me, from my limit knowledge, are xturing and basaran.