cortex VS ultralytics

Jan incorporates a lightweight, built-in inference server called Nitro. Nitro supports both llama.cpp and NVIDIA's TensorRT-LLM engines. This means many open LLMs in the GGUF format are supported. Jan's Model Hub is designed for easy installation of pre-configured models but it also allows you to install virtually any model from Hugging Face or even your own.

I'd like to see a comparison to nitro https://github.com/janhq/nitro which has been fantastic for running a local LLM.

Look... I appreciate a cool project, but this is probably not a good idea.

> Built on top of the cutting-edge inference library llama.cpp, modified to be production ready.

It's not. It's literally just llama.cpp -> https://github.com/janhq/nitro/blob/main/.gitmodules

Llama.cpp makes no pretense at being a robust safe network ready library; it's a high performance library.

You've made no changes to llama.cpp here; you're just calling the llama.cpp API directly from your drogon app.

Hm.

...

Look... that's interesting, but, honestly, I know there's this wave of "C++ is back!" stuff going on, but building network applications in C++ is very tricky to do right, and while this is cool, I'm not sure 'llama.cpp is in c++ because it needs to be fast' is a good reason to go 'so lets build a network server in c++ too!'.

I mean, I guess you could argue that since llama.cpp is a C++ application, it's fair for them to offer their own server example with an openai compatible API (which you can read about here: https://github.com/ggerganov/llama.cpp/issues/4216, https://github.com/ggerganov/llama.cpp/blob/master/examples/...).

...but a production ready application?

I wrote a rust binding to llama.cpp and my conclusion was that llama.cpp is pretty bleeding edge software, and bluntly, you should process isolate it from anything you really care about, if you want to avoid undefined behavior after long running inference sequences; because it updates very often, and often breaks. Those breaks are usually UB. It does not have a 'stable' version.

Further more, when you run large models and run out of memory, C++ applications are notoriously unreliable in their 'handle OOM' behaviour.

Soo.... I know there's something fun here, but really... unless you had a really really compelling reason to need to write your server software in c++ (and I see no compelling reason here), I'm curious why you would?

It seems enormously risky.

The quality of this code is 'fun', not 'production ready'.

YOLO is the most advanced vision detection model. It is maintained by Ultralytics, a leading computer vision team. The model is easy to train, evaluate, and deploy. Plus, its size is compact enough to run in a browser or on a smartphone.

# Clone ultralytics repo git clone https://github.com/ultralytics/ultralytics # cd to local directory cd ultralytics # Install dependencies pip install -r requirements.txt

Yep, I noticed this a while ago. It posts easily identifiable ChatGPT responses. It also posts garbage wrong answers which makes it worse than useless. Totally disrespectful to the userbase.

https://github.com/ultralytics/ultralytics/issues/5748#issue...

There are obvious places it is being used that I have noticed organically. For instance, check out the answers in this repo:

https://github.com/ultralytics/ultralytics/issues/5748#issue...

If you read the answers there, the style of answering is always to repeat the question in a very specific way. Once you see it you can’t in-see it.

When browsing the state-of-the-art in object detection on Papers with Code, I found the YOLO model to be one of the most popular, accurate, and fastest. That being said, I would recommend having a look at Ultralytics, which provides the tools to evaluate, predict, and export the latest versions of YOLO models with only a few lines of code.

YOLOv8 (You Only Look Once) is an open-source Computer Vision AI model released on January 10th, 2023. It’s called YOLO because it detects everything inside an image in a single pass. The new version can perform image detection, classification, instance segmentation, tracking, and pose estimation tasks.

Next, I needed to figure out, how can I access the stream, recognize an animal, then let Max know? There are tons of examples of recognizing an object via camera frames, but I ultimately found this python library called ultralytics that supports RTSP streams and classifying objects in the video frames using pre-built models. The docs looked like it would be pretty low effort, so after some experimentation, I was successful in having the ultralytics library recognize objects from my cheap camera!