Top 7 data-centric Open-Source Projects

ludwig

3 10,845 9.5 Python

Low-code framework for building custom LLMs, neural networks, and other AI models

Project mention: Show HN: Toolkit for LLM Fine-Tuning, Ablating and Testing | news.ycombinator.com | 2024-04-07

This is a great project, little bit similar to https://github.com/ludwig-ai/ludwig, but it includes testing capabilities and ablation.
questions regarding the LLM testing aspect: How extensive is the test coverage for LLM use cases, and what is the current state of this project area? Do you offer any guarantees, or is it considered an open-ended problem?
Would love to see more progress toward this area!

lance

10 3,296 9.8 Rust

Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..

Project mention: The Nimble File Format by Meta | news.ycombinator.com | 2024-04-25

InfluxDB

www.influxdata.com featured

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
data-centric-AI

4 990 4.5

A curated, but incomplete, list of data-centric AI resources. (by daochenzha)
Encord Active

6 420 8.8 Python

Open source active learning toolkit to find failure modes in your computer vision models, prioritize data to label next, and drive data curation to improve model performance.

Project mention: Launch HN: Encord (YC W21) – Unit testing for computer vision models | news.ycombinator.com | 2024-01-31

We base our pricing on your user and consumption scale and would be happy to discuss this with you directly. Please feel free to explore the OS version of Active at https://github.com/encord-team/encord-active. Note that some features, such as natural language search using GPU accelerated APIs, are not included in the cloud version.

DataCLUE

2 145 0.0 Python

DataCLUE: 数据为中心的NLP基准和工具包
vuejs-form

6 41 0.0 JavaScript

Vue Form with Laravel Inspired Validation and Simply Enjoyable Error Messages Api. (Form Api, Validator Api, Rules Api, Error Messages Api)
pypely

3 16 2.9 Python

From local functions to cloud deployed pipelines
SaaSHub

www.saashub.com featured

SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

data-centric related posts

Python Open Closed Design Pattern (Python SOLID Principles)

1 project | dev.to | 5 Feb 2023
The Proficient, Efficient, and Consistent Solution (Skillful, Minimal, & Disciplined Practitioners)

1 project | dev.to | 23 Jul 2022
[R] DataCLUE: A Benchmark Suite for Data-centric NLP

2 projects | /r/MachineLearning | 17 Nov 2021