Considering getting a Jetson AGX Orin.. anyone have experience with it?

This page summarizes the projects mentioned and recommended in the original post on /r/LocalLLaMA

InfluxDB
Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • bitsandbytes

    Accessible large language models via k-bit quantization for PyTorch.

    • Do you by chance have any details on how to run oobagooba on the Orin? I keep running into this issue seemingly related to bitsandbytes.

  • bitsandbytes_jetsonX

    8-bit CUDA functions for PyTorch, modified to build on Jetson Xavier

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • llama.cpp

    LLM inference in C/C++

    • How did you compile llama.cpp? Did you just do the patches from this thread: https://github.com/ggerganov/llama.cpp/issues/1455

  • text-generation-webui

    A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

    • I'm not sure what to expect with this. Does the LLaMA part auto-start on boot? Or do I need to start each individually? Should this be as easy as once I get it figured out I can go to http://agx.fqdn:7860 and it'll look like a web-browser with a chat window (like what the github shows)?

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts