TensorRT-LLM VS htmx

Yes, we are also looking at integrating MLX [1] which is optimized for Apple Silicon and built by an incredible team of individuals, a few of which were behind the original Torch [2] project. There's also TensorRT-LLM [3] by Nvidia optimized for their recent hardware.

All of this of course acknowledging that llama.cpp is an incredible project with competitive performance and support for almost any platform.

[1] https://github.com/ml-explore/mlx

[2] https://en.wikipedia.org/wiki/Torch_(machine_learning)

[3] https://github.com/NVIDIA/TensorRT-LLM

NVIDIA GPU LLM https://github.com/NVIDIA/TensorRT-LLM

https://github.com/NVIDIA/TensorRT-LLM

It's quite a thin wrapper around putting both projects into %LocalAppData%, along with a miniconda environment with the correct dependnancies installed. Also for some reason the LLaMA 13b (24.5GB) and Ministral 7b (13.6GB) but only installed Ministral?

Ministral 7b runs about as accurate as I remeber, but responses are faster than I can read. This seems at the cost of context and variance/temperature - although it's a chat interface the implementation doesn't seem to take into account previous questions or answers. Asking it the same question also gives the same answer.

The RAG (llamaindex) is okay, but a little suspect. The installation comes with a default folder dataset, containing text files of nvidia marketing materials. When I tried asking questions about the files, it often cites the wrong file even if it gave the right answer.

Yeah, seems a bit odd because the TensorRT-LLM repo lists Turing as supported architecture.

https://github.com/NVIDIA/TensorRT-LLM?tab=readme-ov-file#pr...

I support any progress to erode the Nvidia monopoly.

That said from what I'm seeing here the free and open source (less other aspects of the CUDA stack, of course) TensorRT-LLM[0] almost certainly bests this implementation using the Nvidia hardware they reference for comparison.

I don't have an A6000 but as an example with the tensorrt_llm backend for Nvidia Triton Inference Server (also free and open source) I get roughly 30 req/s with Mistral 7B on my RTX 4090 with significantly lower latency. Comparison benchmarks are tough, especially when published benchmarks like these are fairly scant on the real details.

TensorRT-LLM has only been public for a few months and if you peruse the docs, PRs, etc you'll see they have many more optimizations in the works.

In typical Nvidia fashion TensorRT-LLM runs on any Nvidia card (from laptop to datacenter) going back to Turing (five year old cards) assuming you have the VRAM.

You can download and run this today, free and "open source" for these implementations at least. I'm extremely skeptical of the claim "MK1 Flywheel has the Best Throughput and Latency for LLM Inference on NVIDIA". You'll note they compare to vLLM, which is an excellent and incredible project but if you look at vLLM vs Triton w/ TensorRT-LLM the performance improvements are dramatic.

Of course it's the latest and greatest ($$$$$$ and unobtanium) but one look at H100/H200 performance[3] and you can see what happens when the vendor has a robust software ecosystem to help sell their hardware. Pay the Nvidia tax on the frontend for the hardware, get it back as a dividend on the software.

I feel like MK1 must be aware of TensorRT-LLM but of course those comparison benchmarks won't help sell their startup.

[0] - https://github.com/NVIDIA/TensorRT-LLM

[1] - https://github.com/triton-inference-server/tensorrtllm_backe...

[2] - https://mkone.ai/blog/mk1-flywheel-race-tuned-and-track-read...

[3] - https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source...

I have followed the instructions on https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/llama to convert the float16 Llama2 13B to FP8 and build a tensorRT-LLM engine.

Hi there! I want to show off a little feature I made using hanami, htmx and a little bit of redis + sidekiq.

Recently, I just rewrite one of my application Stashbin from Next.js to GO. Though my main motivation of this migration was to learn GO and experimenting with HTMX. I also aiming to reduce the resource usage of my application and simplify the deployment process. Initially, Stashbin codebase are split into two seperate repository, one for the frontend that uses Next.js and another for the backend that already uses GO. The backend repository is just a REST API responsible for storing and retreiving data from the database.

HTMX is another library that gained popularity due to its server-first approach to rendering data, although seeking a much simpler way of appealing to developers.

When I work with HTMX I need isolated component that can be reusable a form. So I create a PHP Function that generate the Input Datalist.

I was inspired by htmz (which was in turn inspired by htmx) and how the author got pretty close to a basic htmx-like experience just using an iframe. I wanted to push it a little further so whipped this demo together. My submission demonstrates progressive enhancement for the form - with js enabled the request targets an iframe that is inserted into the dom, meaning the page doesn't actually navigate (similar to event.preventDefault()). The iframe receives the html response from the request and on load triggers a function to swap out it's contents into the main page.

As described on htmx.org: "htmx gives you access to AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, using attributes, so you can build modern user interfaces with the simplicity and power of hypertext"

Apart from the source code itself, the repo's README also gives a bit more details about the various packages I used.

1. htmx: https://htmx.org/

Also, there’s some response header juggling you have to do when submitting forms that have a validation step before redirecting: https://github.com/bigskysoftware/htmx/issues/369

I’ve tried to iron out any footguns or server requirements I’ve bumped into while using HTMX & Hotwire in my projects.

✨ In recent months, I have been developing web projects using GOTTHA stack: Go + Templ + Tailwind CSS + htmx + Alpine.js. As soon as I'm ready to talk about all the subtleties and pitfalls, I'll post it on my social networks.