Ask HN: How to get started with local language models?

• ollama

  192 58,943 9.9 Go

  Get up and running with Llama 3, Mistral, Gemma, and other large language models.

  

You might want to start playing with with Ollama or LmStudio to get you started. The model's weights aren't normally inside your .exe . Typically you can address 128TB of virtual memory on Windows.

https://ollama.com/

https://lmstudio.ai/

• text-generation-webui

  876 36,293 9.9 Python

  A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

You can use webui https://github.com/oobabooga/text-generation-webui

Once you get a version up and running I make a copy before I update it as several times updates have broken my working version and caused headaches.

a decent explanation of parameters outside of reading archive papers: https://github.com/oobabooga/text-generation-webui/wiki/03-%...

a news ai website:

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• Local-LLM-User-Guideline

  1 152 8.3

• Cgml

  21 37 8.6 C++

  GPU-targeted vendor-agnostic AI library for Windows, and Mistral model implementation.

If you just want to run Mistral on Windows, you could try my port: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral...

The setup is relatively easy: install .NET runtime, download 4.5 GB model file from BitTorrent, unpack a small ZIP file and run the EXE.

• femtoGPT

  15 713 8.8 Rust

  Pure Rust implementation of a minimal Generative Pretrained Transformer

If you want to get deep in LLMs: https://github.com/keyvank/femtoGPT

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project