Python llm-training

Open-source Python projects categorized as llm-training

Top 12 Python llm-training Projects

llm-training
  1. ludwig

    Low-code framework for building custom LLMs, neural networks, and other AI models

    Project mention: Show HN: Toolkit for LLM Fine-Tuning, Ablating and Testing | news.ycombinator.com | 2024-04-07

    This is a great project, little bit similar to https://github.com/ludwig-ai/ludwig, but it includes testing capabilities and ablation.

    questions regarding the LLM testing aspect: How extensive is the test coverage for LLM use cases, and what is the current state of this project area? Do you offer any guarantees, or is it considered an open-ended problem?

    Would love to see more progress toward this area!

  2. CodeRabbit

    CodeRabbit: AI Code Reviews for Developers. Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.

    CodeRabbit logo
  3. skypilot

    SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

    Project mention: Today's Top 30+ items of Github - Dec 20, 2024 | dev.to | 2024-12-20
  4. xtuner

    An efficient, flexible and full-featured toolkit for fine-tuning LLM (InternLM2, Llama3, Phi3, Qwen, Mistral, ...)

    Project mention: PaliGemma: Open-Source Multimodal Model by Google | news.ycombinator.com | 2024-05-15
  5. h2o-llmstudio

    H2O LLM Studio - a framework and no-code GUI for fine-tuning LLMs. Documentation: https://docs.h2o.ai/h2o-llmstudio/

  6. dbrx

    Code examples and resources for DBRX, a large language model developed by Databricks

    Project mention: Hello OLMo: A Open LLM | news.ycombinator.com | 2024-04-08

    One thing I wanted to add and call attention to is the importance of licensing in open models. This is often overlooked when we blindly accept the vague branding of models as “open”, but I am noticing that many open weight models are actually using encumbered proprietary licenses rather than standard open source licenses that are OSI approved (https://opensource.org/licenses). As an example, Databricks’s DBRX model has a proprietary license that forces adherence to their highly restrictive Acceptable Use Policy by referencing a live website hosting their AUP (https://github.com/databricks/dbrx/blob/main/LICENSE), which means as they change their AUP, you may be further restricted in the future. Meta’s Llama is similar (https://github.com/meta-llama/llama/blob/main/LICENSE ). I’m not sure who can depend on these models given this flaw.

  7. dstack

    dstack is a lightweight, open-source alternative to Kubernetes & Slurm, simplifying AI container orchestration with multi-cloud & on-prem support. It natively supports NVIDIA, AMD, TPU, and Intel accelerators.

    Project mention: Dstack: An alternative to K8 for AI/ML tasks | news.ycombinator.com | 2024-11-05
  8. dlrover

    DLRover: An Automatic Distributed Deep Learning System

    Project mention: DLRover: A Large-scale Intelligent Distributed Training System | dev.to | 2024-08-21

    Star our project on GitHub: https://github.com/intelligent-machine-learning/dlrover

  9. Nutrient

    Nutrient – The #1 PDF SDK Library, trusted by 10K+ developers. Other PDF SDKs promise a lot - then break. Laggy scrolling, poor mobile UX, tons of bugs, and lack of support cost you endless frustrations. Nutrient’s SDK handles billion-page workloads - so you don’t have to debug PDFs. Used by ~1 billion end users in more than 150 different countries.

    Nutrient logo
  10. LLM-VM

    irresponsible innovation. Try now at https://chat.dev/

  11. Finetune_LLMs

    Repo for fine-tuning Casual LLMs

  12. LLMtuner

    FineTune LLMs in few lines of code (Text2Text, Text2Speech, Speech2Text)

  13. Auto-Data

    Auto Data is a library designed for quick and effortless creation of datasets tailored for fine-tuning Large Language Models (LLMs).

    Project mention: Show HN: Fine-Tuning Data Generator Written Purely in Python | news.ycombinator.com | 2024-04-10
  14. discus

    A data-centric AI package for ML/AI. Get the best high-quality data for the best results. Discord: https://discord.gg/t6ADqBKrdZ

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python llm-training discussion

Log in or Post with

Index

What are some of the best open-source llm-training projects in Python? This list will help you:

# Project Stars
1 ludwig 11,304
2 skypilot 7,131
3 xtuner 4,191
4 h2o-llmstudio 4,159
5 dbrx 2,531
6 dstack 1,662
7 dlrover 1,325
8 LLM-VM 486
9 Finetune_LLMs 454
10 LLMtuner 232
11 Auto-Data 96
12 discus 63

Sponsored
CodeRabbit: AI Code Reviews for Developers
Revolutionize your code reviews with AI. CodeRabbit offers PR summaries, code walkthroughs, 1-click suggestions, and AST-based analysis. Boost productivity and code quality across all major languages with each PR.
coderabbit.ai

Did you know that Python is
the 2nd most popular programming language
based on number of references?