cortex
jan
cortex | jan | |
---|---|---|
8 | 20 | |
1,698 | 19,848 | |
5.8% | 11.1% | |
9.8 | 10.0 | |
5 days ago | 8 days ago | |
C++ | TypeScript | |
GNU Affero General Public License v3.0 | GNU Affero General Public License v3.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
cortex
-
Introducing Jan
Jan incorporates a lightweight, built-in inference server called Nitro. Nitro supports both llama.cpp and NVIDIA's TensorRT-LLM engines. This means many open LLMs in the GGUF format are supported. Jan's Model Hub is designed for easy installation of pre-configured models but it also allows you to install virtually any model from Hugging Face or even your own.
-
Ollama Python and JavaScript Libraries
I'd like to see a comparison to nitro https://github.com/janhq/nitro which has been fantastic for running a local LLM.
- FLaNK Weekly 08 Jan 2024
-
Nitro: A fast, lightweight 3MB inference server with OpenAI-Compatible API
Look... I appreciate a cool project, but this is probably not a good idea.
> Built on top of the cutting-edge inference library llama.cpp, modified to be production ready.
It's not. It's literally just llama.cpp -> https://github.com/janhq/nitro/blob/main/.gitmodules
Llama.cpp makes no pretense at being a robust safe network ready library; it's a high performance library.
You've made no changes to llama.cpp here; you're just calling the llama.cpp API directly from your drogon app.
Hm.
...
Look... that's interesting, but, honestly, I know there's this wave of "C++ is back!" stuff going on, but building network applications in C++ is very tricky to do right, and while this is cool, I'm not sure 'llama.cpp is in c++ because it needs to be fast' is a good reason to go 'so lets build a network server in c++ too!'.
I mean, I guess you could argue that since llama.cpp is a C++ application, it's fair for them to offer their own server example with an openai compatible API (which you can read about here: https://github.com/ggerganov/llama.cpp/issues/4216, https://github.com/ggerganov/llama.cpp/blob/master/examples/...).
...but a production ready application?
I wrote a rust binding to llama.cpp and my conclusion was that llama.cpp is pretty bleeding edge software, and bluntly, you should process isolate it from anything you really care about, if you want to avoid undefined behavior after long running inference sequences; because it updates very often, and often breaks. Those breaks are usually UB. It does not have a 'stable' version.
Further more, when you run large models and run out of memory, C++ applications are notoriously unreliable in their 'handle OOM' behaviour.
Soo.... I know there's something fun here, but really... unless you had a really really compelling reason to need to write your server software in c++ (and I see no compelling reason here), I'm curious why you would?
It seems enormously risky.
The quality of this code is 'fun', not 'production ready'.
- Apple Silicon Llama 7B running in docker?
- Is there any LLM that can be installed with out python
jan
- Jan β Turn your computer into an AI computer
-
Devoxx Genie Plugin : an Update
I focused on supporting Ollama, GPT4All, and LMStudio, all of which run smoothly on a Mac computer. Many of these tools are user-friendly wrappers around Llama.cpp, allowing easy model downloads and providing a REST interface to query the available models. Last week, I also added "ππΌ Jan" support because HuggingFace has endorsed this provider out-of-the-box.
-
Ask HN: Which LLMs can run locally on most consumer computers
seconded - IMHO Jan has the cleanest UI and most straightforward setup out of all LLM frontends available now.
https://jan.ai/
https://github.com/janhq/jan
-
Introducing Jan
As we continue this blog series, let's explore a fully open-source alternative to LM Studio - Jan, a project from Southeast Asia.
-
AI enthusiasm - episode #2π
Jan.ai is a 100% local alternative to ChatGPT: you can download LLMs and run them directly from within the application, or even prompting them and retrieving their response via API.
- Ask HN: What is the current (Apr. 2024) gold standard of running an LLM locally?
-
Show HN: I made an app to use local AI as daily driver
It would be cool to have the option to use the OpenAI API as well in the same interface. http://jan.ai does this, so that's what I'm using at the moment.
- Jan β Bringing AI to Your Desktop
- FLaNK 15 Jan 2024
-
Why the M2 is more advanced that it seemed
Was it this? I havenβt tried it yet but it does look nice.
https://jan.ai/
What are some alternatives?
ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.
unstructured - Open source libraries and APIs to build custom preprocessing pipelines for labeling, training, or production machine learning pipelines.
bionic-gpt - BionicGPT is an on-premise replacement for ChatGPT, offering the advantages of Generative AI while maintaining strict data confidentiality
chainlit - Build Conversational AI in minutes β‘οΈ
csvlens - Command line csv viewer
obsidian-local-llm - Obsidian Local LLM is a plugin for Obsidian that provides access to a powerful neural network, allowing users to generate text in a wide range of styles and formats using a local LLM.
nnl - a low-latency and high-performance inference engine for large models on low-memory GPU platform.
Tribuo - Tribuo - A Java machine learning library
FLaNK-VectorDB - NiFi and Vector Databases
hyperfine - A command-line benchmarking tool
modelfusion-llamacpp-nextjs-starter - Starter examples for using Next.js and the Vercel AI SDK with Llama.cpp and ModelFusion.