This repository contains the source code for HaloCheck, a component discussed in our research paper titled "HaLo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Models."
To install HaloCheck, use the following pip command:
pip install git+https://github.com/EngSalem/HaLo.gitHaloCheck is a tool designed to estimate the severity of hallucinations in Large Language Model (LLM) responses. It operates on the principles of cross entailment and is built upon similar concepts as selfcheckGPT, albeit with some key distinctions.
HaloCheck primarily focuses on assessing hallucination severity at a sentence-level granularity. It evaluates whether an LLM generates consistent information across its various response samples, thereby providing a finer-grained estimation of the severity of hallucinations.
HaloCheck better correlates both (pearson $\rho$ and kendal tau $\tau$ ) with human annotation of consistency and avergae factuality of the answers compared to selfcheckGPT.
*Note that we developed HaLoCheck around the same time selfcheckGPT added NLI (entailment wasn't part of the original selfcheckGPT). *We take into account all samples and use SummaC which facilitates computing of entailment between two pieces of text. *HaloCheck range is easily interpreted [-1,1] where -1 indicated infactual, 1 factual. The higher the score the more consistent your samples are.
HaloCheck is also faster than all selfcheckGPT measures, and doesn't need LLM calls. Although it's knowledge free, therefore its incapable of identifying consistent inaccuracies.
To use HaloCheck in your Python code, follow these steps:
importHaloCheckascheckerinconsistent_samples= [
'The 1958 NBA Finals was played between the St. Louis Hawks and Boston Celtics. The Hawks won the series 4 games to 2 in the best of 7 playoff.',
'The 1958 NBA Finals was played between the Minneapolis Lakers and Boston Celtics and was won by the Lakers 4 games to 3.',
'The 1958 NBA Finals was played on April 17, 1958, between the Boston Celtics and the St. Louis Hawks.',
'The 1958 NBA Finals was played between the Boston Celtics and Minneapolis Lakers. The Celtics won the series 4 games to 2 for their 5th championship.',
'The 1958 NBA Finals was played between the Boston Celtics and Minneapolis Lakers. The MVP of the 1958 NBA Finals was Bill Russell.'
]
scorer=checker.HaloCheck(device='cpu', granularity='sentence', nli_model='mnli') # Change to 'cuda' if a GPU is availableprint(scorer.score(inconsistent_samples))
# Expected score: -0.417 (indicating inconsistency)NBAQA.csv
Feel free to modify and adapt the code to suit your specific use case and requirements.
- Please cite the following work if you are using any of our ideas or code
@article{elaraby2023halo,
title={Halo: Estimation and Reduction of Hallucinations in Open-Source Weak Large Language Models},
author={Elaraby, Mohamed and Lu, Mengyin and Dunn, Jacob and Zhang, Xueying and Wang, Yu and Liu, Shizhu},
journal={arXiv preprint arXiv:2308.11764},
year={2023}
}
Note: This repository is part of ongoing research, and the tools provided here are subject to further updates and improvements.