container2wasm VS cortex

The existence of vscode.dev always makes me wonder why Microsoft never released an iOS version of VSCode to get more users into its ecosystem. Sure, it's almost as locked down as the web environment, but there's a lot of space in that "almost" - you could do all sorts of things like let users run their code, or complex extensions, in containers in a web view using https://github.com/ktock/container2wasm or similar.

Unfortunately cosmopolitan wouldn't work for dockerc. Cosmopolitan works as long as you only use it but container runtimes require additional features. Also containers contain arbitrary executables so not sure how that would work either...

As for WASM, this is already possible using container2wasm[0] and wasmer[1]'s ability to generate static binaries.

[0]: https://github.com/ktock/container2wasm

[1]: https://wasmer.io/

Really impressed by the depth and breadth of this project, well done!

A particularly interesting part is the socket layer inside the browser. Other people solving this problem have previously used a proxy to a server that does the real socket implementation. This means you can't have a "browser-only" solution.

The author has solved this (for HTTP/S only) by proxying HTTP requests and then re-creating them as fetch requests (details here: https://github.com/ktock/container2wasm/tree/main/examples/n...). I'm very interested in using this approach for my own project Runno (https://runno.dev).

Jan incorporates a lightweight, built-in inference server called Nitro. Nitro supports both llama.cpp and NVIDIA's TensorRT-LLM engines. This means many open LLMs in the GGUF format are supported. Jan's Model Hub is designed for easy installation of pre-configured models but it also allows you to install virtually any model from Hugging Face or even your own.

I'd like to see a comparison to nitro https://github.com/janhq/nitro which has been fantastic for running a local LLM.

Look... I appreciate a cool project, but this is probably not a good idea.

> Built on top of the cutting-edge inference library llama.cpp, modified to be production ready.

It's not. It's literally just llama.cpp -> https://github.com/janhq/nitro/blob/main/.gitmodules

Llama.cpp makes no pretense at being a robust safe network ready library; it's a high performance library.

You've made no changes to llama.cpp here; you're just calling the llama.cpp API directly from your drogon app.

Hm.

...

Look... that's interesting, but, honestly, I know there's this wave of "C++ is back!" stuff going on, but building network applications in C++ is very tricky to do right, and while this is cool, I'm not sure 'llama.cpp is in c++ because it needs to be fast' is a good reason to go 'so lets build a network server in c++ too!'.

I mean, I guess you could argue that since llama.cpp is a C++ application, it's fair for them to offer their own server example with an openai compatible API (which you can read about here: https://github.com/ggerganov/llama.cpp/issues/4216, https://github.com/ggerganov/llama.cpp/blob/master/examples/...).

...but a production ready application?

I wrote a rust binding to llama.cpp and my conclusion was that llama.cpp is pretty bleeding edge software, and bluntly, you should process isolate it from anything you really care about, if you want to avoid undefined behavior after long running inference sequences; because it updates very often, and often breaks. Those breaks are usually UB. It does not have a 'stable' version.

Further more, when you run large models and run out of memory, C++ applications are notoriously unreliable in their 'handle OOM' behaviour.

Soo.... I know there's something fun here, but really... unless you had a really really compelling reason to need to write your server software in c++ (and I see no compelling reason here), I'm curious why you would?

It seems enormously risky.

The quality of this code is 'fun', not 'production ready'.