ragas
chameleon-llm
ragas | chameleon-llm | |
---|---|---|
10 | 3 | |
4,874 | 1,020 | |
17.7% | - | |
9.6 | 6.2 | |
3 days ago | 5 months ago | |
Python | Jupyter Notebook | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
ragas
-
Show HN: Ragas – the de facto open-source standard for evaluating RAG pipelines
congrats on launching! i think my continuing struggle with looking at Ragas as a company rather than an oss library is that the core of it is like 8 metrics (https://github.com/explodinggradients/ragas/tree/main/src/ra...) that are each 1-200 LOC. i can inline that easily in my app and retain full control, or model that in langchain or haystack or whatever.
why is Ragas a library and a company, rather than an overall "standard" or philosophy (eg like Heroku's 12 Factor Apps) that could maybe be more robust?
(just giving an opp to pitch some underappreciated benefits of using this library)
- FLaNK 04 March 2024
- FLaNK Stack 05 Feb 2024
-
SuperDuperDB - how to use it to talk to your documents locally using llama 7B or Mistral 7B?
Also, at some point you'll need to get serious about evaluation (trust me, you will). You may be interested in https://github.com/explodinggradients/ragas
- Ragas – Framework for RAG Evaluation
- Ragas: Open-source Evaluation framework for RAG pipelines
-
Building a customer support chatbot using GPT-3.5 and lLamaIndex🚀
The problem becomes worse if you want to inspect outputs from not just one, but several different queries. Luckily, there are several free open source packages such as ragas and DeepEval that can help evaluate your chatbot so you don't have to manually do it 😌
-
Patterns for Building LLM-Based Systems and Products
We have build RAGAS framework for this https://github.com/explodinggradients/ragas
-
[R] All about evaluating Large language models
Hi u/thecuteturtle, I am building open-source projects for evaluating LLM-based applications. Check it out https://github.com/explodinggradients/ragas and if you like to collaborate let me know :)
chameleon-llm
-
Giving GPT “Infinite” Knowledge
> Do you know any active research in this area? I briefly considered playing with this, but my back-of-the-envelope semi-educated feeling for now is that it won't scale.
I am aware of a couple of potentially promising research directions. One formally academic called Chameleon [0], and one that's more like a grassroots organic effort that aims to build an actually functional Auto-GPT-like, called Agent-LLM [1]. I have read the Chameleon paper, and I must say I'm quite impressed with their architecture. It added a few bits and pieces that most of the early GPT-based agents didn't have, and I have a strong intuition that these will contribute to these things actually working.
Auto-GPT is another, relatively famous piece of work in this area. However, at least as of v0.2.2, I found it relatively underwhelming. For any online knowledge retrieval+synthesis and retrieval+usage tasks, it seemed to get stuck, but it did sort-of-kind-of OK on plain online knowledge retrieval. After having a look at the Auto-GPT source code, my intuition (yes, I know - "fuzzy feelings without a solid basis" - but I believe that this is simply due to not having an AI background to explain this with crystal-clear wording) is that the poor performance of the current version of Auto-GPT is insufficient skill in prompt-chain architecture and the surprisingly low quality and at times buggy code.
I think Auto-GPT has some potential. I think the implementation lets down the concept, but that's just a question of refactoring the prompts and the overall code - which it seems like the upstream Github repo has been quite busy with, so I might give it another go in a couple of weeks to see how far it's moved forward.
> Specifically, as task complexity grows, the amount of results to combine will quickly exceed the context window size of the "combiner" GPT-4. Sure, you can stuff another layer on top, turning it into a tree/DAG, but eventually, I think the partial result itself will be larger than 8k, or even 32k tokens - and I feel this "eventually" will be hit rather quickly. But maybe my feelings are wrong and there is some mileage in this approach.
Auto-GPT uses an approach based on summarisation and something I'd term 'micro-agents'. For example, when Auto-GPT is searching for an answer to a particular question online, for each search result it finds, it spins up a sub-chain that gets asked a question 'What does this page say about X?' or 'Based on the contents of this page, how can you do Y?'. Ultimately, intelligence is about lossy compression, and this is a starkly exposed when it comes to LLMs because you have no choice but to lose some information.
> I think the partial result itself will be larger than 8k, or even 32k tokens - and I feel this "eventually" will be hit rather quickly. But maybe my feelings are wrong and there is some mileage in this approach.
The solution to that would be to synthesize output section by section, or even as an "output stream" that can be captured and/or edited outside the LLM in whole or in chunks. IMO, I do think there's some mileage to be exploited in a recursive "store, summarise, synthesise" approach, but the problem will be that of signal loss. Every time you pass a subtask to a sub-agent, or summarise the outcome of that sub-agent into your current knowledge base, some noise is introduced. It might be that the signal to noise ratio will dissipate as higher and higher order LLM chains are used - analogously to how terrible it was to use electricity or radio waves before any amplification technology became available.
One possible avenue to explore to crack down on decreasing SNR (based on my own original research, but I can also see some people disclosing online that they are exploring the same path), is to have a second LLM in the loop, double-checking the result of the first one. This has some limitations, but I have successfully used this approach to verify that, for example, the LLM does not outright refuse to carry out a task. This is currently cost-prohibitive to do in a way that would make me personally satisfied and confident enough in the output to make it run full-auto, but I expect that increasing ability to run AI locally will make people more willing to experiment with massive layering of cooperating LLM chains that check each others' work, cooperate, and/or even repeat work using different prompts to pick the best output a la redundant avionics computers.
[0]: https://github.com/lupantech/chameleon-llm
- chameleon-llm: Codes for "Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models".
What are some alternatives?
deepeval - The LLM Evaluation Framework
Azure-Cognitive-Search-Azure-OpenAI-Accelerator - Virtual Assistant - GPT Smart Search Engine - Bot Framework + Azure OpenAI + Azure AI Search + Azure SQL + Bing API + Azure Document Intelligence + LangChain + CosmosDB
Local-LLM-Langchain - Load local LLMs effortlessly in a Jupyter notebook for testing purposes alongside Langchain or other agents. Contains Oobagooga and KoboldAI versions of the langchain notebooks with examples.
vault-ai - OP Vault ChatGPT: Give ChatGPT long-term memory using the OP Stack (OpenAI + Pinecone Vector Database). Upload your own custom knowledge base files (PDF, txt, epub, etc) using a simple React frontend.
FastLoRAChat - Instruct-tune LLaMA on consumer hardware with shareGPT data
AgentBench - A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
agenta - The all-in-one LLM developer platform: prompt management, evaluation, human feedback, and deployment all in one place.
AGiXT - AGiXT is a dynamic AI Agent Automation Platform that seamlessly orchestrates instruction management and complex task execution across diverse AI providers. Combining adaptive memory, smart features, and a versatile plugin system, AGiXT delivers efficient and comprehensive AI solutions.
text-generation-webui-colab - A colab gradio web UI for running Large Language Models
LLM-Training-Puzzles - What would you do with 1000 H100s...
ReAct - [ICLR 2023] ReAct: Synergizing Reasoning and Acting in Language Models
llmflows - LLMFlows - Simple, Explicit and Transparent LLM Apps