VLM for Image Classification

—— Finetune SmolVLM-Instruct for Image classification in specific fields

中文介绍

---

• Introduction
  • Potential of Small-Parameter VLMs in Specific Image Domains
  • Low-Latency Deployment with Consumer-Grade GPUs
• Repository Structure
  • 1. download_dataset.py
  • 2. generate_train_json.py
  • 3. fine_tune.py
  • 4. vlm_benchmark_test_dataset.py
  • 5. vlm_test.py
• Usage Guide
• Training Results Example

---

Introduction

This code repository provides the full process code from dataset creation, model fine-tuning, and evaluation workflows. While specifically designed for SmolVLM-256M-Instruct, the framework supports other Vision Language Models (VLMs) and is tailored for the Solar panel clean and faulty images dataset on Kaggle, with potential for domain adaptation.

Potential of Small-Parameter VLMs in Specific Image Domains

Fine-tuning improves classification accuracy from 0.2 to over 0.98+ on this dataset.

Low-Latency Deployment with Consumer-Grade GPUs

Small-parameter VLMs demonstrate exceptional efficiency:

• Minimum VRAM consumption: 4.5GB during fine-tuning (with batch_size=1)
• Scalable via per_device_train_batch_size and per_device_eval_batch_size adjustments in fine_tune.py

The models show remarkable deployment efficiency:

• SmolVLM-256M-Instruct requires only 1.2GB GPU memory at bfloat16 precision
• Quantized to INT4 format: just 0.6GB VRAM

---

Repository Structure

 1. download_dataset.py

Downloads the Solar panel clean and faulty images dataset from Kaggle directly to your project root directory.

 2. generate_train_json.py

Generates training JSON files from local data:

1. Scans subfolders in dataset root as class labels

2. Iterates through image files

3. Combines question templates with class descriptions to generate Q&A pairs, including direct classification prompts

4. Splits data (default: 80% train/20% test)

5. Saves JSON files in script directory

 3. fine_tune.py

Main fine-tuning script that produces trained models and training logs.

Key Parameters:

# Path configurationslocal_model_path="./SmolVLM-256M-Instruct"# Local model directorytrain_json_path="solar_panel_train_dataset.json"# Training JSONtest_json_path="solar_panel_test_dataset.json"# Test JSONoutput_dir="./SmolVLM-256M-Instruct-finetuned"# Output directory# Training configurationequivalent_epochs_to_train=1per_device_train_batch_size=4per_device_eval_batch_size=4gradient_accumulation_steps=4warmup_steps=50learning_rate=3e-4weight_decay=0.01logging_steps=25

 4. vlm_benchmark_test_dataset.py

Evaluates VLM performance on the test dataset, reporting final accuracy metrics.

 5. vlm_test.py

Full dataset evaluation script. Defaults to using ./SmolVLM-256M-Instruct - modify model_name variable to evaluate fine-tuned models.

Directory Structure:

├── download_dataset.py
├── generate_train_json.py
├── fine_tune.py
├── vlm_benchmark_test_dataset.py
├── vlm_test.py
├── solar_panel_test_dataset.json
├── solar_panel_train_dataset.json
├── README.md
├── requirements.txt
├── SmolVLM-256M-Instruct/
│ └── model_files
├── SmolVLM-256M-Instruct-finetuned/
│ └── finetuned_model_files
└── Faulty_solar_panel/ # Dataset root
├── Category/
└── Images.jpg

---

Usage Guide

Environment Setup

1. Create virtual environment:

conda create -n vlm python=3.10

2. Activate environment:

conda activate vlm

Project Installation

1. Clone repository:

git clone https://github.com/stlin256/VLM4Classification.git

2. Open project directory:

cd VLM4Classification

3. Install dependencies:

python -m pip install --upgrade pip

Download Pytorch from pytoch.org, example：

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

Install other dependencies

pip install -r requirements.txt

Data Preparation

1. Download SmolVLM-256M-Instruct to ./SmolVLM-256M-Instruct
2. Download dataset:

python download_dataset.py

Dataset Processing

python generate_train_json.py

Training

Verify batch size settings (per_device_train_batch_size, per_device_eval_batch_size) before execution

# ~4.5GB VRAMper_device_train_batch_size=1per_device_eval_batch_size=1# ~14GB VRAMper_device_train_batch_size=4per_device_eval_batch_size=4

Run training:

python fine_tune.py

Evaluation

1. Base model full evaluation:

python vlm_test.py

2. Fine-tuned model test set evaluation:

python vlm_benchmark_test_dataset.py

3. Full evaluation of fine-tuned model:

# Modify model_name variable first
python vlm_test.py

Training Results Example

The following parameters were used for training:

#generate_train_json.pytrain_set_ratio=0.8#finetune.py#SFTConfigequivalent_epochs_to_train=1per_device_train_batch_size=4per_device_eval_batch_size=4gradient_accumulation_steps=4warmup_steps=50learning_rate=3e-4weight_decay=0.01

Full evaluation before training:

# vlm_test.py---CategoryAccuracyResults---type:Snow-Covered,correct__rate:0.8373983739837398type:Electrical-damage,correct__rate:0.34951456310679613type:Bird-drop,correct__rate:0.02617801047120419type:Clean,correct__rate:0.09326424870466321type:Dusty,correct__rate:0.08994708994708994type:Physical-Damage,correct__rate:0.028985507246376812---OverallAccuracy---TotalCorrect: 181/868, Accuracy: 0.2085

Full evaluation after training:

# vlm_test.py ---CategoryAccuracyResults---type:Snow-Covered,correct__rate:1.0type:Electrical-damage,correct__rate:0.9320388349514563type:Bird-drop,correct__rate:0.9895287958115183type:Clean,correct__rate:0.9948186528497409type:Dusty,correct__rate:0.9788359788359788type:Physical-Damage,correct__rate:0.9855072463768116---OverallAccuracy---TotalCorrect: 853/868, Accuracy: 0.9827

Evaluation on test dataset after training:

# vlm_benchmark_test_dataset.py---CategoryAccuracyResults---type:Electrical-damage,correct_rate:0.8571 (18/21)
type:Snow-Covered,correct_rate:1.0000 (27/27)
type:Dusty,correct_rate:0.9574 (45/47)
type:Bird-drop,correct_rate:1.0000 (46/46)
type:Clean,correct_rate:1.0000 (31/31)
type:Physical-Damage,correct_rate:1.0000 (15/15)
---OverallAccuracy---TotalCorrect: 182/187, Accuracy: 0.9733

---

---

VLM for Image Classification

—— 微调SomlVLM模型用于图像分类问题

---

• 介绍
  • 展现了小参数量VLM在特定图像领域分类的潜力
  • 小参数量VLM延迟低，性能要求低，在消费级GPU上轻松微调
• 仓库文件介绍
  • 1. download_dataset.py
  • 2. generate_train_json.py
  • 3. fine_tune.py
  • 4. vlm_benchmark_test_dataset.py
  • 5. vlm_test.py
• 使用方式
• 训练结果示例

---

介绍

这个代码仓库提供了从创建数据集、微调、评估的全流程代码。适用于SmolVLM-256M-Instruct以及其它VLM，适用于Kaggle上的Solar panel clean and faulty images数据集，并可迁移使用。

展现了小参数量VLM在特定图像领域分类的潜力：

通过微调，模型在这个数据集上的分类精确度从 0.2 提升到了大于 0.98！

小参数量VLM延迟低，性能要求低，在消费级GPU上轻松微调：

小参数量VLM微调时对显存大小要求低，本仓库微调脚本最低仅消耗4.5G显存。(当batch_size被设置为1的时候)

你可以通过调整fine_tune.py中的per_device_train_batch_size和per_device_eval_batch_size来优化显存占用。

小参数量VLM使用时对算力和显存大小要求低，对端侧部署十分友好，SmolVLM-256M-Instruct在以bfloat16精度下推理时只需1.2G显存，使用INT4量化时只需0.6G显存。

---

仓库文件介绍：

 1. download_dataset.py

用于从Kaggle下载Solar panel clean and faulty images数据集。

直接运行即可，数据集文件夹会被下载到项目文件夹根目录

 2. generate_train_json.py

用于从本地文件创建用于训练的json文件，工作原理如下：

1.从数据集根文件夹开始，搜寻子文件夹，并以子文件夹名为类名

2.遍历子文件夹中的文件。

3.将问题列表与各类别细节描述进行组合，生成QA对，同时固定生成一个仅包含类别名称回答的QA对。

4.打乱顺序，分割训练集和测试集(默认80%与20%)

5.将json文件保存到脚本同级目录中。

 3. fine_tune.py

用于进行模型微调

此脚本将会执行模型的微调工作，并给出微调好的模型文件和记录到的日志。

参数说明：

#路径信息local_model_path="./SmolVLM-256M-Instruct"#基座模型目录(为了加载方便，脚本从本地目录加载)train_json_path="solar_panel_train_dataset.json"#训练集jsontest_json_path="solar_panel_test_dataset.json"#测试集jsonoutput_dir="./SmolVLM-256M-Instruct-finetuned"#微调输出目录#SFTConfig(主要部分)equivalent_epochs_to_train=1#等效epoch数per_device_train_batch_size=4#训练批量大小per_device_eval_batch_size=4#评估批量大小gradient_accumulation_steps=4#梯度累加warmup_steps=50#热身步数learning_rate=3e-4#初始学习率weight_decay=0.01#权重衰减logging_steps=25#日志记录间隔

 4. vlm_benchmark_test_dataset.py

此脚本用于从测试集中评估vlm模型的性能，最终给出模型的正确率。

 5. vlm_test.py

此脚本用于使用整个数据集对模型进行评估，最终给出模型的正确率。

此脚本默认使用的是./SmolVLM-256M-Instruct中的模型，如果你需要对微调好的模型进行评估，请你修改model_name变量。

完整的文件树：

├── download_dataset.py
├── generate_train_json.py
├── fine_tune.py
├── vlm_benchmark_test_dataset.py
├── vlm_test.py
├── solar_panel_test_dataset.json
├── solar_panel_train_dataset.json
├── README.md
├── requirements.txt
├── SmolVLM-256M-Instruct
│ └── model_files
├── SmolVLM-256M-Instruct-finetuned
│ └── finetuned model_files
└── Faulty_solar_panel #dataset root dir
├── Category
└─── Images.jpg

---

使用方式：

配置虚拟环境

1.创建虚拟环境

conda create -n vlm python=3.10

2.激活虚拟环境

conda activate vlm

2.克隆本项目并安装依赖

1.从Github克隆本项目

git clone https://github.com/stlin256/VLM4Classification.git

2.打开项目文件夹

cd VLM4Classification

3.安装依赖

建议先升级pip

python -m pip install --upgrade pip

从Pytorch官网复制适合你的环境的安装命令并运行，如：

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128

安装其它依赖

pip install -r requirements.txt

3.下载所需的文件

1.从SmolVLM-256M-Instruct下载模型文件，保存到./SmolVLM-256M-Instruct中。

2.下载数据集

运行脚本

python download_dataset.py

如果报404错误，则请检查互联网连接或者代理设置。

4.创建训练所用的json文件

运行脚本

python generate_train_json.py

5.进行训练

在运行前请检查批量大小设置，即per_device_train_batch_size和per_device_eval_batch_size变量。

#占用约4.5G显存per_device_train_batch_size=1per_device_eval_batch_size=1#占用约14G显存per_device_train_batch_size=4per_device_eval_batch_size=4

运行脚本

python fine_tune.py

在运行过程中，脚本会输出训练进度、剩余时间、损失、梯度、学习率等信息

各变量的作用请参照前文

6.评估（基于准确率）

1)对原始模型进行完整评估

python vlm_test.py

2)使用测试集对微调模型进行评估

python vlm_benchmark_test_dataset.py

3)对微调模型进行完整评估

将model_name变量的值修改为"./SmolVLM-256M-Instruct-finetuned"

python vlm_test.py

---

训练结果示例

使用以下参数进行训练：

#generate_train_json.pytrain_set_ratio=0.8#finetune.py#SFTConfigequivalent_epochs_to_train=1per_device_train_batch_size=4per_device_eval_batch_size=4gradient_accumulation_steps=4warmup_steps=50learning_rate=3e-4weight_decay=0.01

训练前全量评估

#vlm_test.py---CategoryAccuracyResults---type:Snow-Covered,correct__rate:0.8373983739837398type:Electrical-damage,correct__rate:0.34951456310679613type:Bird-drop,correct__rate:0.02617801047120419type:Clean,correct__rate:0.09326424870466321type:Dusty,correct__rate:0.08994708994708994type:Physical-Damage,correct__rate:0.028985507246376812---OverallAccuracy---TotalCorrect: 181/868, Accuracy: 0.2085

训练后全量评估

#vlm_test.py---CategoryAccuracyResults---type:Snow-Covered,correct__rate:1.0type:Electrical-damage,correct__rate:0.9320388349514563type:Bird-drop,correct__rate:0.9895287958115183type:Clean,correct__rate:0.9948186528497409type:Dusty,correct__rate:0.9788359788359788type:Physical-Damage,correct__rate:0.9855072463768116---OverallAccuracy---TotalCorrect: 853/868, Accuracy: 0.9827

训练后在测试集上的评估

#vlm_benchmark_test_dataset.py---CategoryAccuracyResults---type:Electrical-damage,correct_rate:0.8571 (18/21)
type:Snow-Covered,correct_rate:1.0000 (27/27)
type:Dusty,correct_rate:0.9574 (45/47)
type:Bird-drop,correct_rate:1.0000 (46/46)
type:Clean,correct_rate:1.0000 (31/31)
type:Physical-Damage,correct_rate:1.0000 (15/15)
---OverallAccuracy---TotalCorrect: 182/187, Accuracy: 0.9733

---