GHOST: Getting to the Bottom of Hallucinations with a Multi-round Consistency Benchmark [WACV 2026]

[Project Page] [arXiv] [PDF] [Supplemental] [Slides] [BibTeX]

Installation

pip install -r requirements.txt

For local VLM models, install VLMEvalKit:

git clone https://github.com/open-compass/VLMEvalKit.git
cd VLMEvalKit && pip install -e .

Quick Start

Evaluate Predictions

python evaluate_ghost.py --pred-path predictions.json

Generate Predictions

Local VLM:

python run_predictions.py \
--data-path dataset/ghost_full_merged.json \
--image-dir dataset/images/ \
--model-name llava_v1.5_7b \
--model-type vlm \
--output-path predictions/llava_predictions.json

API Model:

python run_predictions.py \
--data-path dataset/ghost_full_merged.json \
--image-dir dataset/images/ \
--model-name gpt-4o \
--model-type api \
--output-path predictions/gpt4o_predictions.json \
--api-key YOUR_API_KEY

Checkpoint/Resume:

• Predictions are automatically saved every 10 questions
• If interrupted, rerun the same command to resume
• Use --no-resume to start from scratch
• Use --checkpoint-every N to change checkpoint frequency

Dataset Format

JSON format with question keys: {image_id}_{object_id}_{question_type}_{pos/neg}

{
"2406158_obj3_1pos": "A wheels is present in the image.",
"2406158_obj3_attr1_1pos": "The color of the wheels present in the image is white.",
"2406158_obj3_rel1_1pos": "The spatial relation between the wheels and man is that the wheels is to the left of the man."
}

Question Types:

• Object: 1pos, 1neg, 2neg, ...
• Attribute: attr1_1pos, attr1_1neg, ...
• Relation: rel1_1pos, rel1_2neg, ...

GhostConsistencyScore Metric

Categories:

• Objects GCS: Consistency on object presence questions
• Attributes GCS: Consistency on object attribute questions
• Relations GCS: Consistency on spatial relation questions

Output Format

Predictions are saved as JSON:

[
{
"question_id": "2406158_obj3_1pos",
"object_id": "2406158_obj3",
"image": "2406158.jpg",
"text": "A wheels is present in the image.",
"label": "yes",
"model_name": "llava_v1.5_7b",
"prediction": "true"
}
]

Repository Structure

ghost-evaluation/
├── dataset/
│ ├── ghost_full_merged.json
│ └── images/
├── ghost_consistency_score.py
├── utils.py
├── evaluate_ghost.py
├── run_predictions.py
├── requirements.txt
├── .gitignore
└── README.md

Library Usage

fromevaluate_ghostimportevaluateresults=evaluate('predictions.json')
print(f"Objects GCS: {results['objects_gcs']:.2f}%")
print(f"Attributes GCS: {results['attributes_gcs']:.2f}%")
print(f"Relations GCS: {results['relations_gcs']:.2f}%")

API Implementation

To use API models, implement the get_api_prediction() function in run_predictions.py:

defget_api_prediction(model_name: str, image_path: str, prompt: str, api_key: str=None) ->str:
ifmodel_name=='gpt-4o':
# Implement OpenAI API callpasselifmodel_name=='gemini-pro':
# Implement Google Gemini API callpass# Add more models as needed

Citation

@article{ghost2024,
title={GHOST: Getting to the Bottom of Hallucinations with a Multi-round Consistency Benchmark},
author={[Authors]},
journal={[Journal/Conference]},
year={2024}
}

License

[license]