Top 18 Python retrieval-augmented-generation Projects
-
txtai
đź’ˇ All-in-one open-source embeddings database for semantic search, LLM orchestration and language model workflows
-
ragflow
RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
GenerativeAIExamples
Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
raptor
The official implementation of RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (by parthsarthi03)
-
cognita
Cognita by TrueFoundry - Framework for building modular, open source RAG applications for production.
-
-
-
repochat
Chatbot assistant enabling GitHub repository interaction using LLMs with Retrieval Augmented Generation
-
tonic_validate
Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.
-
SimplyRetrieve
Lightweight chat AI platform featuring custom knowledge, open-source LLMs, prompt-engineering, retrieval analysis. Highly customizable. For Retrieval-Centric & Retrieval-Augmented Generation.
-
tvallogging
A tool for evaluating and tracking your RAG experiments. This repo contains the Python SDK for logging to Tonic Validate.
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: What contributing to Open-source is, and what it isn't | news.ycombinator.com | 2024-04-27I tend to agree with this sentiment. Many junior devs and/or those in college want to contribute. Then they feel entitled to merge a PR that they worked hard on often without guidance. I'm all for working with people but projects have standards and not all ideas make sense. In many cases, especially with commercial open source, the project is the base of a companies identity. So it's not just for drive-by ideas to pad a resume or finish a school project.
For those who do want to do this, I'd recommend writing an issue and/or reaching out to the developers to engage in a dialogue. This takes work but it will increase the likelihood of a PR being merged.
Disclaimer: I'm the primary developer of txtai (https://github.com/neuml/txtai), an open-source vector database + RAG framework
Project mention: RAGFlow is an open-source RAG engine based on deep document understanding | news.ycombinator.com | 2024-04-01Just link them to https://github.com/infiniflow/ragflow/blob/main/rag/llm/chat... :)
Project mention: More Agents Is All You Need: LLMs performance scales with the number of agents | news.ycombinator.com | 2024-04-06I couldn't agree more. You should check out LLMWare's SLIM agents (https://github.com/llmware-ai/llmware/tree/main/examples/SLI...). It's focusing on pretty much exactly this and chaining multiple local LLMs together.
A really good topic that ties in with this is the need for deterministic sampling (I may have the terminology a bit incorrect) depending on what the model is indended for. The LLMWare team did a good 2 part video on this here as well (https://www.youtube.com/watch?v=7oMTGhSKuNY)
I think dedicated miniture LLMs are the way forward.
Disclaimer - Not affiliated with them in any way, just think it's a really cool project.
Project mention: Show HN: R2R – Open-source framework for production-grade RAG | news.ycombinator.com | 2024-02-26
Project mention: FastLLM by Qdrant – lightweight LLM tailored For RAG | news.ycombinator.com | 2024-04-01
Project mention: Ask HN: Has Anyone Trained a personal LLM using their personal notes? | news.ycombinator.com | 2024-04-03hadn't seen your repo yet [1] - adding it to my list right now.
Your blog post is really neat on top - thanks for sharing
https://github.com/eugeneyan/obsidian-copilot
Project mention: Raptor: Recursive Abstractive Processing for Tree-Organized Retrieval | news.ycombinator.com | 2024-03-06
Project mention: Show HN: Cognita – open-source RAG framework for modular applications | news.ycombinator.com | 2024-04-27
Project mention: Launch HN: Relari (YC W24) – Identify the root cause of problems in LLM apps | news.ycombinator.com | 2024-03-08Hi HN, we are the founders of Relari, the company behind continuous-eval (https://github.com/relari-ai/continuous-eval), an evaluation framework that lets you test your GenAI systems at the component level, pinpointing issues where they originate.
We experienced the need for this when we were building a copilot for bankers. Our RAG pipeline blew up in complexity as we added components: a query classifier (to triage user intent), multiple retrievers (to grab information from different sources), filtering LLM (to rerank / compress context), a calculator agent (to call financial functions) and finally the synthesizer LLM that gives the answer. Ensuring reliability became more difficult with each of these we added.
When a bad response was detected by our answer evaluator, we had to backtrack multiple steps to understand which component(s) made a mistake. But this quickly became unscalable beyond a few samples.
I did my Ph.D. in fault detection for autonomous vehicles, and I see a strong parallel between the complexity of autonomous driving software and today's LLM pipelines. In self-driving systems, sensors, perception, prediction, planning, and control modules are all chained together. To ensure system-level safety, we use granular metrics to measure the performance of each module individually. When the vehicle makes an unexpected decision, we use these metrics to pinpoint the problem to a specific component. Only then we can make targeted improvements, systematically.
Based on this thinking, we developed the first version of continuous-eval for ourselves. Since then we’ve made it more flexible to fit various types of GenAI pipelines. Continuous-eval allows you to describe (programmatically) your pipeline and modules, and select metrics for each module. We developed 30+ metrics to cover retrieval, text generation, code generation, classification, agent tool use, etc. We now have a number of companies using us to test complex pipelines like finance copilots, enterprise search, coding agents, etc.
As an example, one customer was trying to understand why their RAG system did poorly on trend analysis queries. Through continuous-eval, they realized that the “retriever” component was retrieving 80%+ of all relevant chunks, but the “reranker” component, that filters out “irrelevant” context, was dropping that to below 50%. This enabled them to fix the problem, in their case by skipping the reranker for certain queries.
We’ve also built ensemble metrics that do a surprisingly good job of predicting user feedback. Users often rate LLM-generated answers by giving a thumbs up/down about how good the answer was. We train our custom metrics on this user data, and then use those metrics to generate thumbs up/down ratings on future LLM answers. The results turn out to be 90% aligned with what the users say. This gives developers a feedback loop from production data to offline testing and development. Some customers have found this to be our most unique advantage.
Lastly, to make the most out of evaluation, you should use a diverse dataset—ideally with ground truth labels for comprehensive and consistent assessment. Because ground truth labels are costly and time-consuming to curate manually, we also have a synthetic data generation pipeline that allows you to get started quickly. Try it here (https://www.relari.ai/#synthetic_data_demo)
What’s been your experience testing and iterating LLM apps? Please let us know your thoughts and feedback on our approaches (modular framework, leveraging user feedback, testing with synthetic data).
Project mention: Validating the RAG Performance of Amazon Titan vs. Cohere Using Amazon Bedrock | news.ycombinator.com | 2024-02-09I tried out Amazon Bedrock, and used Tonic Validate to do a head to head comparison of very simple RAG system's built using embedding and text models available in Amazon Bedrock. I compared Amazon Titan's embedding and text models to Cohere's embedding and text models in RAG systems that employ Amazon Bedrock Knowledge Bases as the vector db and retrieval components of the system.
The code for the comparison is in this jupyter notebook https://github.com/TonicAI/tonic_validate/blob/main/examples...
Let me know what you think, And your experiences building RAG with Amazon Bedrock!
Project mention: Show HN: Open-Source Chat AI Platform with Custom Knowledge | news.ycombinator.com | 2023-08-22
Project mention: Show HN: Tonic Validate Logging – an open-sourced SDK and convenient UI | news.ycombinator.com | 2023-10-31
Python retrieval-augmented-generation related posts
Index
What are some of the best open-source retrieval-augmented-generation projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | txtai | 6,990 |
| 2 | ragflow | 5,516 |
| 3 | TaskingAI | 4,233 |
| 4 | llmware | 3,127 |
| 5 | GenerativeAIExamples | 1,502 |
| 6 | R2R | 1,180 |
| 7 | autollm | 908 |
| 8 | fastembed | 781 |
| 9 | obsidian-copilot | 427 |
| 10 | raptor | 424 |
| 11 | cognita | 349 |
| 12 | AnglE | 341 |
| 13 | continuous-eval | 302 |
| 14 | txtchat | 221 |
| 15 | repochat | 207 |
| 16 | tonic_validate | 199 |
| 17 | SimplyRetrieve | 187 |
| 18 | tvallogging | 6 |
Sponsored