cortex
nitro
cortex | nitro | |
---|---|---|
8 | 21 | |
1,661 | 5,238 | |
3.7% | 3.1% | |
9.8 | 9.7 | |
6 days ago | 7 days ago | |
C++ | TypeScript | |
GNU Affero General Public License v3.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
cortex
-
Introducing Jan
Jan incorporates a lightweight, built-in inference server called Nitro. Nitro supports both llama.cpp and NVIDIA's TensorRT-LLM engines. This means many open LLMs in the GGUF format are supported. Jan's Model Hub is designed for easy installation of pre-configured models but it also allows you to install virtually any model from Hugging Face or even your own.
-
Ollama Python and JavaScript Libraries
I'd like to see a comparison to nitro https://github.com/janhq/nitro which has been fantastic for running a local LLM.
- FLaNK Weekly 08 Jan 2024
-
Nitro: A fast, lightweight 3MB inference server with OpenAI-Compatible API
Look... I appreciate a cool project, but this is probably not a good idea.
> Built on top of the cutting-edge inference library llama.cpp, modified to be production ready.
It's not. It's literally just llama.cpp -> https://github.com/janhq/nitro/blob/main/.gitmodules
Llama.cpp makes no pretense at being a robust safe network ready library; it's a high performance library.
You've made no changes to llama.cpp here; you're just calling the llama.cpp API directly from your drogon app.
Hm.
...
Look... that's interesting, but, honestly, I know there's this wave of "C++ is back!" stuff going on, but building network applications in C++ is very tricky to do right, and while this is cool, I'm not sure 'llama.cpp is in c++ because it needs to be fast' is a good reason to go 'so lets build a network server in c++ too!'.
I mean, I guess you could argue that since llama.cpp is a C++ application, it's fair for them to offer their own server example with an openai compatible API (which you can read about here: https://github.com/ggerganov/llama.cpp/issues/4216, https://github.com/ggerganov/llama.cpp/blob/master/examples/...).
...but a production ready application?
I wrote a rust binding to llama.cpp and my conclusion was that llama.cpp is pretty bleeding edge software, and bluntly, you should process isolate it from anything you really care about, if you want to avoid undefined behavior after long running inference sequences; because it updates very often, and often breaks. Those breaks are usually UB. It does not have a 'stable' version.
Further more, when you run large models and run out of memory, C++ applications are notoriously unreliable in their 'handle OOM' behaviour.
Soo.... I know there's something fun here, but really... unless you had a really really compelling reason to need to write your server software in c++ (and I see no compelling reason here), I'm curious why you would?
It seems enormously risky.
The quality of this code is 'fun', not 'production ready'.
- Apple Silicon Llama 7B running in docker?
- Is there any LLM that can be installed with out python
nitro
-
Why I keep an eye on the Vue ecosystem and you should too
Nitro is a nice https webserver that you can deploy everywhere. Comparing it to express, it doesn't need weird middlewares for json, it has a simple way to support caching, a file system router, tasks and scheduled tasks that avoid quite a few shell scripts, db:migrations etc, plugins, KV storages, SQL connectors, websockets...
-
What is Vinxi, and how does it compare to Vike?
Vinxi is really a kind of Meta-Router / Router Manager (built on the dev-server and bundler-toolkit Vite and the http-server Nitro). Vinxi uses various routers of your choosing as a core primitive, and allows you to compose them in a centralized config so that they work together. Be it server or client routers.
-
The State of Angular SSR Deployment in 2024
Analog allows you to deploy anywhere, literally. It uses Nitro toolkit which has options for anything while working with Vite.
- FLaNK Stack Weekly 22 January 2024
-
Introducing Nitro by @unjs: Simplifying API Development in Nodejs
For additional reading, please explore the Nitro docs at: https://nitro.unjs.io
-
Nitro: A fast, lightweight 3MB inference server with OpenAI-Compatible API
Not to be confused with https://nitro.unjs.io the server tech behind Nuxt and SolidStart
-
Web scraper in Nuxt 3 - part I - Introduction and setting up
Nuxt is powered by an internal server called Nitro. You can see its manifestation in the terminal console right after you start your local dev server:
-
Ask HN: Is Express still "de-facto" for building Node back ends?
Until it support websocket, I think it is simply a no.
https://github.com/unjs/nitro/issues/678
It's 2023 and there is a web framework that "can't" handle websocket at all. (Not even just proxying and doing nothing else.) Feels like a joke to me.
- Update regarding the recent loading/caching issues
-
How to edit & replace NPM package dependency
Last week, my first actual contribution into a public open source project has been merged. Its actually only a small changes into Unstorage library, which is used by Nitros server engine, which in turn used by the popular NuxtJs framework. At the time, I was stuck in one of my side project which used Nuxt because Unstorage is lacking some functionalities that I need. However because Unstorage is 2 layers deep in the dependency tree, its not trivial for me to change the code there.
What are some alternatives?
ollama - Get up and running with Llama 3, Mistral, Gemma, and other large language models.
hono - Web Framework built on Web Standards
bionic-gpt - BionicGPT is an on-premise replacement for ChatGPT, offering the advantages of Generative AI while maintaining strict data confidentiality
vite - Next generation frontend tooling. It's fast!
csvlens - Command line csv viewer
Express - Fast, unopinionated, minimalist web framework for node.
nnl - a low-latency and high-performance inference engine for large models on low-memory GPU platform.
esbuild - An extremely fast bundler for the web
Tribuo - Tribuo - A Java machine learning library
ai - Build AI-powered applications with React, Svelte, Vue, and Solid
hyperfine - A command-line benchmarking tool
unstorage - 💾 Unstorage provides an async Key-Value storage API with conventional features like multi driver mounting, watching and working with metadata, dozens of built-in drivers and a tiny core.