Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare

Hanwei Zhu^1*, Haoning Wu^2*, Yixuan Li¹, Zicheng Zhang³, Baoliang Chen¹, Lingyu Zhu¹,

Yuming Fang⁴, Guangtao Zhai³, Weisi Lin², Shiqi Wang^1#

¹City University of Hong Kong ²Nanyang Technological University, ³Shanghai Jiao Tong University, ⁴Jiangxi University of Finance and Economics

^*Equal contribution. ^#Corresponding author.

Compare2Score (HF Model) | Arxiv | Homepage

Motivation

Training & Inference

Structure

Quicker Start with Hugging Face AutoModel

No need to install this GitHub repo.

importrequestsimporttorchfromtransformersimportAutoModelForCausalLMmodel=AutoModelForCausalLM.from_pretrained("q-future/Compare2Score", trust_remote_code=True, torch_dtype=torch.float16, device_map="auto")
fromPILimportImageimage_path_url="https://raw.githubusercontent.com/Q-Future/Q-Align/main/fig/singapore_flyer.jpg"print("The quality score of this image is {}".format(model.score(image_path_url)) 

Installation

Evaluation:

git clone https://github.com/Q-Future/Compare2Score.git
cd Compare2Score
pip install -e .

Training:

pip install -e ".[train]"
pip install flash_attn --no-build-isolation

Visual Quality Scorer

fromq_alignimportCompare2ScorerfromPILimportImagescorer=Compare2Scorer()
image_path="figs/i04_03_4.bmp"print("The quality score of this image is {}.".format(scorer(image_path)))

Training & Evaluation

Get Datasets

Download all IQA datasets and training JSONs

importos, globfromhuggingface_hubimportsnapshot_downloadsnapshot_download("VQA-CityU/IQA_data", repo_type="dataset", local_dir="./playground/data", local_dir_use_symlinks=False)
gz_files=glob.glob("playground/data/*.zip")
forgz_fileingz_files:
print(gz_file)
os.system("unzip ./playground/data/".format(gz_file))

Evaluation

After preparing the datasets, you can evaluate pre-trained Compare2Score as follows:

python q_align/evaluate/IQA_dataset_eval.py --model-path q-future/Compare2Score --device cuda:0

Training from Scratch

sh scripts/train_bid_csiq_clive_kadid_koniq_live_compare.sh

Citation

@article{zhu2024adaptive,
title={Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare},
author={Zhu, Hanwei and Wu, Haoning and Li, Yixuan and Zhang, Zicheng and Chen, Baoliang and Zhu, Lingyu and Fang, Yuming and Zhai, Guangtao and Lin, Weisi and Wang, Shiqi},
journal={arXiv preprint arXiv:2405.19298},
year={2024},
}