Langchain and self hosted LLaMA hosted API

• basaran

  22 1,281 10.0 Python

  Discontinued Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.

  

What are the current best "no reinventing the wheel" approaches to have Langchain use an LLM through a locally hosted REST API, the likes of Oobabooga or hyperonym/basaran with streaming support for 4-bit GPTQ?

• text-generation-inference

  29 7,881 9.6 Python

  Large Language Model Text Generation Inference

  

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• ctransformers

  4 1,702 8.6 C

  Python bindings for the Transformer models implemented in C/C++ using GGML library.

For ggml https://github.com/marella/ctransformers/ and https://github.com/abetlen/llama-cpp-python has a decent server. https://github.com/go-skynet/LocalAI is very active too.

• llama-cpp-python

  55 6,475 9.8 Python

  Python bindings for llama.cpp

For ggml https://github.com/marella/ctransformers/ and https://github.com/abetlen/llama-cpp-python has a decent server. https://github.com/go-skynet/LocalAI is very active too.

• LocalAI

  82 19,593 9.9 C++

  :robot: The free, Open Source OpenAI alternative. Self-hosted, community-driven and local-first. Drop-in replacement for OpenAI running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. It allows to generate Text, Audio, Video, Images. Also with voice cloning capabilities.

  

For ggml https://github.com/marella/ctransformers/ and https://github.com/abetlen/llama-cpp-python has a decent server. https://github.com/go-skynet/LocalAI is very active too.

• LangChain_PDFChat_Oobabooga

  4 65 3.0 Python

  oobaboga -text-generation-webui implementation of wafflecomposite - langchain-ask-pdf-local

Here you can find a way to use Oobabooga API with langchain. https://github.com/sebaxzero/LangChain_PDFChat_Oobabooga

Suggest a related project