ialacol VS llama.cpp

# This is the configuration file for DevSpace # # devspace use namespace private-ai # suggest to use a namespace instead of the default name space # devspace deploy # deploy the skeleton of the app and the dependencies (ialacol) # devspace dev # start syncing files to the container # devspace purge # to clean up version: v2beta1 deployments: # This are the manifest our private app deployment # The app will be in "sleep mode" after `devspace deploy`, and start when we start # syncing files to the container by `devspace dev` private-ai-app: helm: chart: # We are deploying the so-called Component Chart: https://devspace.sh/component-chart/docs name: component-chart repo: https://charts.devspace.sh values: containers: - image: ghcr.io/loft-sh/devspace-containers/python:3-alpine command: - "sleep" args: - "99999" service: ports: - port: 8000 labels: app.kubernetes.io/name: private-ai-app ialacol: helm: # the backend for the AI app, we are using ialacol https://github.com/chenhunghan/ialacol/ chart: name: ialacol repo: https://chenhunghan.github.io/ialacol # overriding values.yaml of ialacol helm chart values: replicas: 1 deployment: image: quay.io/chenhunghan/ialacol:latest env: # We are using MPT-30B, which is the most sophisticated model at the moment # If you want to start with some small but mightym try orca-mini # DEFAULT_MODEL_HG_REPO_ID: TheBloke/orca_mini_3B-GGML # DEFAULT_MODEL_FILE: orca-mini-3b.ggmlv3.q4_0.bin # MPT-30B DEFAULT_MODEL_HG_REPO_ID: TheBloke/mpt-30B-GGML DEFAULT_MODEL_FILE: mpt-30b.ggmlv0.q4_1.bin DEFAULT_MODEL_META: "" # Request more resource if needed resources: {} # pvc for storing the cache cache: persistence: size: 5Gi accessModes: - ReadWriteOnce storageClass: ~ cacheMountPath: /app/cache # pvc for storing the models model: persistence: size: 20Gi accessModes: - ReadWriteOnce storageClass: ~ modelMountPath: /app/models service: type: ClusterIP port: 8000 annotations: {} # You might want to use the following to select a node with more CPU and memory # for MPT-30B, we need at least 32GB of memory nodeSelector: {} tolerations: [] affinity: {}

You might be wondering why running Kubernetes is necessary for this project. This article was actually created during the development of a testing CI for the OSS project ialacol. The goal was to have a basic smoke test that verifies the Helm charts and ensures the endpoint returns a 200 status code. You can find the full source of the testing CI YAML here.

We are deploying a Helm release orca-mini-3b using Helm chart ialacol

well https://github.com/ggerganov/llama.cpp/issues/6849

Have just done this recently for local chat with pdf feature in https://recurse.chat. (It's a macOS app that has built-in llama.cpp server and local vector database)

Running an embedding server locally is pretty straightforward:

- Get llama.cpp release binary: https://github.com/ggerganov/llama.cpp/releases

Ah, thanks for this! I can't edit my parent comment that you replied to any longer unfortunately.

As I said, I only compared the contributors graphs [0] and checked for overlaps. But those apparently only go back about year and only list at most 100 contributors ranked by number of commits.

[0]: https://github.com/ollama/ollama/graphs/contributors and https://github.com/ggerganov/llama.cpp/graphs/contributors

KodiBot is a desktop app that enables users to run their own AI chat assistants locally and offline on Windows, Mac, and Linux operating systems. KodiBot is a standalone app and does not require an internet connection or additional dependencies to run local chat assistants. It supports both Llama.cpp compatible models and OpenAI API.

There are already some implementations out there which attempt to accomplish this!

Here's an example: https://github.com/silphendio/sliced_llama

A gist pertaining to said example: https://gist.github.com/silphendio/535cd9c1821aa1290aa10d587...

Here's a discussion about integrating this capability with ExLlama: https://github.com/turboderp/exllamav2/pull/275

And same as above but for llama.cpp: https://github.com/ggerganov/llama.cpp/issues/4718#issuecomm...

For those who might not be aware of this, there is also an open source project on GitHub called "Twinny" which is an offline Visual Studio Code plugin equivalent to Copilot: https://github.com/rjmacarthy/twinny

It can be used with a number of local model services. Currently for my setup on a NVIDIA 4090, I'm running both the base and instruct model for deepseek-coder 6.7b using 5_K_M Quantization GGUF files (for performance) through llama.cpp "server" where the base model is for completions and the instruct model for chat interactions.

llama.cpp: https://github.com/ggerganov/llama.cpp/

deepseek-coder 6.7b base GGUF files: https://huggingface.co/TheBloke/deepseek-coder-6.7B-base-GGU...

deepseek-coder 6.7b instruct GGUF files: https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct...