RefChecker VS selfcheckgpt

Compare RefChecker vs selfcheckgpt and see what are their differences.

RefChecker

RefChecker provides automatic checking pipeline and benchmark dataset for detecting fine-grained hallucinations generated by Large Language Models. (by amazon-science)

selfcheckgpt

SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (by potsawee)
Scout Monitoring - Free Django app performance insights with Scout Monitoring
Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
www.scoutapm.com
featured
InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
RefChecker selfcheckgpt
1 1
213 383
2.8% -
7.6 6.9
12 days ago 16 days ago
Python Python
Apache License 2.0 MIT License
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.

RefChecker

Posts with mentions or reviews of RefChecker. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2024-05-03.
  • How to Detect AI Hallucinations
    5 projects | dev.to | 3 May 2024
    RefChecker operates through a 3-stage pipeline: 1. Triplets Extraction: Utilizes LLMs to break down text into knowledge triplets for detailed analysis. 2. Checker Stage: Predicts hallucination labels on the extracted triplets using LLM-based or NLI-based checkers. 3. Aggregation: Combines individual triplet-level results to determine the overall hallucination label for the input text based on predefined rules. Additionally, RefChecker includes a human labeling tool, a search engine for Zero Context settings, and a localization model to map knowledge triplets back to reference snippets for comprehensive analysis. Triplets in the context of RefChecker refer to knowledge units extracted from text using Large Language Models (LLMs). These triplets consist of three elements that capture essential information from the text. The extraction of triplets helps in finer-grained detection and evaluation of claims by breaking down the original text into structured components for analysis. The triplets play a crucial role in detecting hallucinations and assessing the factual accuracy of claims made by language models. RefChecker includes support for various Large Language Models (LLMs) that can be used locally for processing and analysis. Some of the popular LLMs supported by RefChecker include GPT4, GPT-3.5-Turbo, InstructGPT, Falcon, Alpaca, LLaMA2, and Claude 2. These models can be utilized within the RefChecker framework for tasks such as response generation, claim extraction, and hallucination detection without the need for external connections to cloud-based services. I did not use it as it requires integration with several other providers or a large GPU for Mistral model. But this looks very promising and In future I will come back to this one (depends on how much I want to spend on GPU for my open source project)

selfcheckgpt

Posts with mentions or reviews of selfcheckgpt. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2024-05-03.
  • How to Detect AI Hallucinations
    5 projects | dev.to | 3 May 2024
    SelfCheckGPT is a method designed for detecting hallucinations in Large Language Models (LLMs) without the need for external resources. It evaluates the consistency of responses generated by LLMs to determine if the information provided is factual or not. SelfCheckGPT outperforms other methods and serves as a strong baseline for assessing the reliability of LLM-generated text. It works by comparing multiple responses generated by a Large Language Model (LLM) in response to a query. It measures the consistency between these responses to determine if the information provided is factual or hallucinated. By sampling multiple responses, SelfCheckGPT can identify inconsistencies and contradictions, indicating potential hallucinations in the generated text. This method does not require external databases and can be used for black-box models, making it a versatile tool for detecting unreliable information generated by LLMs. One of the main features of SelfCheckGPT is MQAG which stands for Multiple-choice Question Answering and Generation. It evaluates information consistency between source and summary using multiple-choice questions. Consists of question generation stage, statistical distance analysis, and answerability threshold setting. Uses total variation as the main statistical distance for comparison. Provides a novel approach to assess information content in summaries through question answering. Comparing Multiple Responses: SelfCheckGPT compares multiple responses generated by an LLM to measure consistency and identify potential hallucinations. Sampling Responses: By sampling multiple responses, SelfCheckGPT can detect inconsistencies and contradictions in the generated text. Utilizing Question Answering: SelfCheckGPT incorporates question answering to assess the consistency of information by generating multiple-choice questions and evaluating the answers. Entropy-based Metrics: It uses entropy-based metrics to analyze the probability distribution of words in the generated text, providing insights into the reliability of the information. Zero-resource Approach: SelfCheckGPT is a zero-resource approach that does not rely on external databases, making it applicable to black-box LLMs ( and that is the exact reason I like it )
Scout Monitoring - Free Django app performance insights with Scout Monitoring
Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
www.scoutapm.com
featured
InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured