TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines. (by NVIDIA)
Add to my DEV experience Suggest topics
Source Code
nvidia.github.io
TensorRT-LLM Reviews
Suggest alternative
Edit details
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured

TensorRT-LLM Alternatives

Similar projects and alternatives to TensorRT-LLM

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better TensorRT-LLM alternative or higher similarity.
Suggest an alternative to TensorRT-LLM

TensorRT-LLM reviews and mentions

Posts with mentions or reviews of TensorRT-LLM. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2024-04-28.
  • Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
    11 projects | news.ycombinator.com | 28 Apr 2024
    Yes, we are also looking at integrating MLX [1] which is optimized for Apple Silicon and built by an incredible team of individuals, a few of which were behind the original Torch [2] project. There's also TensorRT-LLM [3] by Nvidia optimized for their recent hardware.

    All of this of course acknowledging that llama.cpp is an incredible project with competitive performance and support for almost any platform.

    [1] https://github.com/ml-explore/mlx

    [2] https://en.wikipedia.org/wiki/Torch_(machine_learning)

    [3] https://github.com/NVIDIA/TensorRT-LLM

  • FLaNK AI for 11 March 2024
    46 projects | dev.to | 11 Mar 2024
  • FLaNK Stack 26 February 2024
    50 projects | dev.to | 26 Feb 2024
    NVIDIA GPU LLM https://github.com/NVIDIA/TensorRT-LLM
  • FLaNK Stack Weekly 19 Feb 2024
    50 projects | dev.to | 19 Feb 2024
  • Nvidia Chat with RTX
    2 projects | news.ycombinator.com | 13 Feb 2024
    https://github.com/NVIDIA/TensorRT-LLM

    It's quite a thin wrapper around putting both projects into %LocalAppData%, along with a miniconda environment with the correct dependnancies installed. Also for some reason the LLaMA 13b (24.5GB) and Ministral 7b (13.6GB) but only installed Ministral?

    Ministral 7b runs about as accurate as I remeber, but responses are faster than I can read. This seems at the cost of context and variance/temperature - although it's a chat interface the implementation doesn't seem to take into account previous questions or answers. Asking it the same question also gives the same answer.

    The RAG (llamaindex) is okay, but a little suspect. The installation comes with a default folder dataset, containing text files of nvidia marketing materials. When I tried asking questions about the files, it often cites the wrong file even if it gave the right answer.

  • Nvidia's Chat with RTX is a promising AI chatbot that runs locally on your PC
    7 projects | news.ycombinator.com | 13 Feb 2024
    Yeah, seems a bit odd because the TensorRT-LLM repo lists Turing as supported architecture.

    https://github.com/NVIDIA/TensorRT-LLM?tab=readme-ov-file#pr...

  • MK1 Flywheel Unlocks the Full Potential of AMD Instinct for LLM Inference
    3 projects | news.ycombinator.com | 8 Jan 2024
    I support any progress to erode the Nvidia monopoly.

    That said from what I'm seeing here the free and open source (less other aspects of the CUDA stack, of course) TensorRT-LLM[0] almost certainly bests this implementation using the Nvidia hardware they reference for comparison.

    I don't have an A6000 but as an example with the tensorrt_llm backend for Nvidia Triton Inference Server (also free and open source) I get roughly 30 req/s with Mistral 7B on my RTX 4090 with significantly lower latency. Comparison benchmarks are tough, especially when published benchmarks like these are fairly scant on the real details.

    TensorRT-LLM has only been public for a few months and if you peruse the docs, PRs, etc you'll see they have many more optimizations in the works.

    In typical Nvidia fashion TensorRT-LLM runs on any Nvidia card (from laptop to datacenter) going back to Turing (five year old cards) assuming you have the VRAM.

    You can download and run this today, free and "open source" for these implementations at least. I'm extremely skeptical of the claim "MK1 Flywheel has the Best Throughput and Latency for LLM Inference on NVIDIA". You'll note they compare to vLLM, which is an excellent and incredible project but if you look at vLLM vs Triton w/ TensorRT-LLM the performance improvements are dramatic.

    Of course it's the latest and greatest ($$$$$$ and unobtanium) but one look at H100/H200 performance[3] and you can see what happens when the vendor has a robust software ecosystem to help sell their hardware. Pay the Nvidia tax on the frontend for the hardware, get it back as a dividend on the software.

    I feel like MK1 must be aware of TensorRT-LLM but of course those comparison benchmarks won't help sell their startup.

    [0] - https://github.com/NVIDIA/TensorRT-LLM

    [1] - https://github.com/triton-inference-server/tensorrtllm_backe...

    [2] - https://mkone.ai/blog/mk1-flywheel-race-tuned-and-track-read...

    [3] - https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source...

  • FP8 quantized results are bad compared to int8 results
    1 project | /r/LocalLLaMA | 7 Dec 2023
    I have followed the instructions on https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/llama to convert the float16 Llama2 13B to FP8 and build a tensorRT-LLM engine.
  • Optimum-NVIDIA - 28x faster inference in just 1 line of code !?
    4 projects | /r/LocalLLaMA | 6 Dec 2023
  • Incoming: TensorRT-LLM version 0.6 with support for MoE, new models and more quantization
    1 project | /r/LocalLLaMA | 5 Dec 2023
  • A note from our sponsor - InfluxDB
    www.influxdata.com | 1 May 2024
    Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Stats

Basic TensorRT-LLM repo stats
14
6,556
8.4
8 days ago

NVIDIA/TensorRT-LLM is an open source project licensed under Apache License 2.0 which is an OSI approved license.

The primary programming language of TensorRT-LLM is C++.

Popular Comparisons

  1. TensorRT-LLM VS trt-llm-rag-windows
  2. TensorRT-LLM VS gpt-fast
  3. TensorRT-LLM VS optimum-nvidia
  4. TensorRT-LLM VS stable-fast
  5. TensorRT-LLM VS tensorrtllm_backe
  6. TensorRT-LLM VS daytona
  7. TensorRT-LLM VS gpt-neox
  8. TensorRT-LLM VS prql
  9. TensorRT-LLM VS FLaNK-python-watsonx-processor
  10. TensorRT-LLM VS OpenPDF

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com