GRAID: Generating Reasoning questions from Analysis of Images via Discriminative artificial intelligence

🚀 Quick Start

Installation

0. Install uv (optional if you already have it): curl -LsSf https://astral.sh/uv/install.sh | sh (or see uv installation guide)
1. Create a virtual environment: uv venv
2. Activate it: source .venv/bin/activate (or use direnv with the provided .envrc)
3. Install dependencies: uv sync
4. Install all backends: uv run install_all

🤗 HuggingFace Dataset Generation

Generate high-quality VQA datasets for modern ML workflows:

# Interactive mode with step-by-step guidance
graid generate-dataset

Key Features:

• 🎯 Object Filtering: Smart allowable sets for focused object detection
• 🔬 Multi-Model Ensemble: Weighted Boxes Fusion (WBF) for improved accuracy
• ⚙️ Flexible Configuration: JSON configs for reproducible experiments
• 🌐 HuggingFace Hub Integration: Direct upload to share datasets
• 🖼️ PIL Image Support: Ready for modern vision-language models
• 📊 Rich Metadata: Comprehensive dataset documentation

Quick Examples:

# Generate with specific object types (autonomous driving focus)
uv run graid generate-dataset --allowable-set "person,car,truck,bicycle,traffic light"# Multi-model ensemble for enhanced accuracy
uv run graid generate-dataset --config examples/wbf_ensemble.json
# Upload directly to HuggingFace Hub
uv run graid generate-dataset --upload-to-hub --hub-repo-id "your-org/dataset-name"# List all valid COCO objects
uv run graid generate-dataset --list-objects

🎛️ Configuration-Driven Workflows

Create reusable configurations for systematic experiments:

Basic Configuration:

{
"dataset_name": "bdd",
"split": "val", "models": [
{
"backend": "detectron",
"model_name": "faster_rcnn_R_50_FPN_3x",
"confidence_threshold": 0.7
},
{
"backend": "mmdetection", "model_name": "co_detr",
"confidence_threshold": 0.6
}
],
"use_wbf": true,
"wbf_config": {
"iou_threshold": 0.6,
"model_weights": [1.0, 1.2]
},
"allowable_set": ["person", "car", "truck", "bus", "motorcycle", "bicycle"],
"confidence_threshold": 0.5,
"batch_size": 4
}

Advanced Configuration with Custom Questions and Transforms:

{
"dataset_name": "bdd",
"split": "val",
"models": [
{
"backend": "ultralytics",
"model_name": "yolov8x.pt",
"confidence_threshold": 0.6
}
],
"use_wbf": false,
"allowable_set": ["person", "car", "bicycle", "motorcycle", "traffic light"],
"confidence_threshold": 0.5,
"batch_size": 2,
"questions": [
{
"name": "HowMany",
"params": {}
},
{
"name": "Quadrants", "params": {
"N": 3,
"M": 3
}
},
{
"name": "WidthVsHeight",
"params": {
"threshold": 0.4
}
},
{
"name": "LargestAppearance",
"params": {
"threshold": 0.35
}
},
{
"name": "MostClusteredObjects",
"params": {
"threshold": 80
}
}
],
"transforms": {
"type": "yolo_bdd",
"new_shape": [640, 640]
},
"save_path": "./datasets/custom_bdd_vqa",
"upload_to_hub": true,
"hub_repo_id": "your-org/bdd-reasoning-dataset",
"hub_private": false
}

Custom Model Configuration:

{
"dataset_name": "custom",
"split": "train",
"models": [
{
"backend": "detectron",
"model_name": "custom_retinanet",
"custom_config": {
"config": "path/to/config.yaml", "weights": "path/to/model.pth"
}
},
{
"backend": "ultralytics",
"model_name": "custom_yolo",
"custom_config": {
"model_path": "path/to/custom_yolo.pt"
}
}
],
"transforms": {
"type": "yolo_bdd",
"new_shape": [832, 832]
},
"questions": [
{
"name": "IsObjectCentered",
"params": {}
},
{
"name": "LeftOf", "params": {}
},
{
"name": "RightOf",
"params": {}
}
]
}

📦 Custom Dataset Support

Bring Your Own Data: GRAID supports any PyTorch-compatible dataset:

fromgraid.data.generate_datasetimportgenerate_datasetfromtorch.utils.dataimportDatasetclassCustomDataset(Dataset):
"""Your custom dataset implementation"""def__getitem__(self, idx):
# Return: (image_tensor, optional_annotations, metadata)# Annotations are only needed for mAP/mAR evaluation# For VQA generation, only images are requiredpass# Generate HuggingFace dataset from your datadataset=generate_dataset(
dataset_name="custom",
split="train",
models=your_models,
allowable_set=["person", "vehicle"], save_path="./datasets/custom_vqa"
)

Key Point: Custom datasets only require images for VQA generation. Annotations are optional and only needed if you want to evaluate model performance with mAP/mAR metrics.

🔧 Advanced Features

Multi-Model Ensemble with WBF

Combine predictions from multiple models using Weighted Boxes Fusion for enhanced detection accuracy:

• Improved precision through model consensus
• Configurable fusion parameters and model weights
• Supports mixed backends (Detectron2 + MMDetection + Ultralytics)

Intelligent Object Filtering

Focus datasets on specific object categories:

• Common presets: Autonomous driving, indoor scenes, animals
• Interactive selection: Visual picker from 80 COCO categories
• Manual specification: Comma-separated object lists
• Validation: Automatic checking against COCO standard

Production-Ready Outputs

Generated datasets include:

• PIL Images: Direct compatibility with vision-language models
• Rich Annotations: Bounding boxes, confidence scores, object classes
• Structured QA Pairs: Question templates with precise answers
• Comprehensive Metadata: Model info, generation parameters, statistics

📊 Supported Models & Datasets

Backends

	Detectron2	MMDetection	Ultralytics
Object Detection	✅	✅	✅
Instance Segmentation	✅	✅	✅
WBF Ensemble	✅	✅	✅

Built-in Datasets

	BDD100K	NuImages	Waymo
Object Detection	✅	✅	✅
Instance Segmentation	✅	✅	✅
HuggingFace Export	✅	✅	✅

Example Models

Detectron2:faster_rcnn_R_50_FPN_3x, retinanet_R_101_FPN_3x
MMDetection:co_detr, dino, rtmdet
Ultralytics:yolov8x, yolov10x, yolo11x, rtdetr-x

🎯 Research Applications

This framework enables systematic evaluation of:

• Vision-Language Models: Generate targeted VQA benchmarks
• Object Detection Methods: Compare model performance on specific object types
• Reasoning Capabilities: Create challenging spatial and counting questions
• Domain Adaptation: Generate domain-specific evaluation sets
• Ensemble Methods: Evaluate fusion strategies across detection models

📈 Quality Assurance

Generated datasets undergo comprehensive validation:

• Model Verification: Automatic testing of model loading and inference
• Annotation Quality: Confidence score filtering and duplicate removal
• Metadata Integrity: Complete provenance tracking for reproducibility
• Format Compliance: COCO-standard annotations with HuggingFace compatibility

🔍 Example commands

Interactive CLI: User-friendly prompts for dataset and model selection

uv run graid generate

Available Commands:

uv run graid --help # Show help
uv run graid list-models # List available models 
uv run graid list-questions # List available question types with parameters
uv run graid info # Show project information
uv run graid generate-dataset # Modern HuggingFace generation# Interactive features
uv run graid generate-dataset --interactive-questions # Select questions interactively
uv run graid generate-dataset --list-questions # Show available questions

📄 License

GRAID is open source software licensed under the Apache License 2.0. This applies to both the GRAID framework code and any datasets generated using GRAID.

Important: When using GRAID with source datasets (BDD100K, Waymo, nuImages, etc.), you must also comply with the original source dataset license terms.