Why is 4bit llama slower on a 32GB RAM 3090 windows machine vs. a M1 Pro 32GB ram with llama.cpp?

This page summarizes the projects mentioned and recommended in the original post on /r/LocalLLaMA
Wsl2 GUI Windows Flutter Wsl

InfluxDB
Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
Our great sponsors

  • text-generation-webui

    A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

    • There's a mistake in that doc, as version 2 is supported on Windows 10 version 21H2 or later (right click start, system to confirm) and you'll definitely want that. It basically gives you a GPU accelerated Ubuntu virtual machine inside of Windows. Once you get it set up, then you can just follow Linux instructions to set Oobabooga up (https://github.com/oobabooga/text-generation-webui).

  • wsl2-distro-manager

    A GUI to quickly manage your WSL2 instances

    • https://github.com/bostrot/wsl2-distro-manager/ (scroll down for the link to the releases or the MS Store page)

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts