Road Scene Semantic Segmentation

A PyTorch-based implementation of various deep learning architectures for semantic segmentation of unstructured road scenes using the Indian Driving Dataset (IDD).

Table of Contents

• Overview
• Dataset
• Model Architectures
  • FCN
  • U-Net
  • LinkNet
  • DeepLabV3+
• Architecture-Comparision
• Results
• Installation
• Usage
• Project Structure

Overview

This project implements and compares five popular deep learning architectures for semantic segmentation:

• FCN (Fully Convolutional Network)
• U-Net
• PSPNet
• LinkNet
• DeepLabV3+

Dataset

The models are trained on the IDD-Lite dataset, which contains road scene images from Indian cities, annotated with 8 classes:

• Drivable area
• Non-drivable area
• Living things
• Vehicles
• Roadside objects
• Far objects
• Sky
• Miscellaneous

Model Architectures

This project implements five deep learning architectures, each with its unique strengths for semantic segmentation:

1. Fully Convolutional Network (FCN)

graph LR
subgraph VGG16_Backbone
I[Input] --> C1[Conv Block 1]
C1 --> C2[Conv Block 2]
C2 --> C3[Conv Block 3]
C3 --> C4[Conv Block 4]
C4 --> C5[Conv Block 5]
end
subgraph FCN_Head
C5 --> FC6[Conv 7x7]
FC6 --> FC7[Conv 1x1]
FC7 --> S1[Score]
end
subgraph Skip_Connections
C4 --> S2[Score Pool4]
C3 --> S3[Score Pool3]
S1 --> U1[Upsample 2x]
U1 --> F1[Fuse]
S2 --> F1
F1 --> U2[Upsample 2x]
U2 --> F2[Fuse]
S3 --> F2
F2 --> U3[Upsample 8x]
U3 --> O[Output]
end
style I fill:#f9f,stroke:#333
style O fill:#9ff,stroke:#333

Loading

The FCN architecture transforms traditional classification networks into fully convolutional networks for semantic segmentation. Key features:

• Based on VGG16 backbone
• Replaces fully connected layers with 1x1 convolutions
• Uses skip connections from earlier layers for fine-grained prediction
• Multi-scale prediction fusion for better segmentation details

2. U-Net Architecture

graph TD
subgraph Encoder
I[Input Image] --> C1[Conv Block 1]
C1 --> P1[MaxPool]
P1 --> C2[Conv Block 2]
C2 --> P2[MaxPool]
P2 --> C3[Conv Block 3]
C3 --> P3[MaxPool]
P3 --> C4[Conv Block 4]
end
subgraph Bottleneck
C4 --> B[Bottleneck]
end
subgraph Decoder
B --> U1[UpConv 1]
U1 --> D1[Conv Block 5]
D1 --> U2[UpConv 2]
U2 --> D2[Conv Block 6]
D2 --> U3[UpConv 3]
U3 --> D3[Conv Block 7]
D3 --> O[Output]
end
%% Skip Connections
C1 -.-> D3
C2 -.-> D2
C3 -.-> D1
style I fill:#f9f,stroke:#333
style O fill:#9ff,stroke:#333
style B fill:#ff9,stroke:#333

Loading

The U-Net architecture features a symmetric encoder-decoder structure that's particularly effective for detailed segmentation:

• Contracting path (encoder) captures context
• Expanding path (decoder) enables precise localization
• Skip connections transfer detailed features from encoder to decoder
• Particularly effective at preserving fine structural details

3. LinkNet

graph TD
subgraph Encoder
I[Input] --> E1[Encoder Block 1]
E1 --> E2[Encoder Block 2]
E2 --> E3[Encoder Block 3]
E3 --> E4[Encoder Block 4]
end
subgraph Decoder
E4 --> D4[Decoder Block 4]
D4 --> D3[Decoder Block 3]
D3 --> D2[Decoder Block 2]
D2 --> D1[Decoder Block 1]
end
%% Skip Connections
E1 -.-> D1
E2 -.-> D2
E3 -.-> D3
E4 -.-> D4
D1 --> F[Final Conv]
F --> O[Output]
style I fill:#f9f,stroke:#333
style O fill:#9ff,stroke:#333

Loading

LinkNet is designed for efficient semantic segmentation:

• Memory-efficient architecture with strong performance
• Direct connections between encoder and decoder blocks
• Residual connections for better gradient flow
• Lighter computational footprint compared to U-Net
• Ideal for real-time applications

4. DeepLabV3+

graph TD
subgraph Encoder
I[Input] --> B[Backbone]
B --> ASPP{ASPP Module}
end
subgraph ASPP_Module
ASPP --> A1[1x1 Conv]
ASPP --> A2[3x3 Rate 6]
ASPP --> A3[3x3 Rate 12]
ASPP --> A4[3x3 Rate 18]
ASPP --> A5[Global Pool]
end
subgraph Decoder
A1 & A2 & A3 & A4 & A5 --> C[Concat]
C --> C1[Conv 1x1]
B --> LF[Low-level Features]
LF --> C2[Conv 1x1]
C1 --> U1[Upsample 4x]
U1 --> M[Merge]
C2 --> M
M --> U2[Upsample 4x]
U2 --> O[Output]
end
style I fill:#f9f,stroke:#333
style O fill:#9ff,stroke:#333

Loading

DeepLabV3+ represents the state-of-the-art in semantic segmentation:

• Atrous Spatial Pyramid Pooling (ASPP) for multi-scale processing
• Multiple dilation rates (6, 12, 18) for broader receptive fields
• Encoder-decoder structure with ASPP module
• Fusion of low-level and high-level features
• Superior performance on boundary regions

Architecture Comparison

Architecture	Strengths	Best Use Cases	Memory Usage	Inference Speed
FCN	Simple, effective baseline	General segmentation	Medium	Fast
U-Net	Fine detail preservation	Medical imaging, detailed segmentation	High	Medium
LinkNet	Efficiency, good performance	Real-time applications	Low	Fast
DeepLabV3+	State-of-the-art accuracy	High-accuracy requirements	High	Slow

Results

Model performance comparison on IDD-Lite dataset:

Architecture	Training Set	Testing Set	Mean F1 Score
FCN	0.9032	0.9034	0.687
UNET	0.8784	0.7406	0.586
LINKNET	0.9231	0.7579	0.750
DEEPLABV3+	0.8040	0.7712	0.787

Installation

# Clone the repository
git clone https://github.com/your-username/road-scene-segmentation.git
cd road-scene-segmentation
# Install dependencies
pip install -e .

Requirements

• Python 3.7+
• PyTorch >= 1.9.0
• torchvision >= 0.10.0
• albumentations >= 1.0.3
• OpenCV
• NumPy
• Matplotlib
• tqdm

Dataset Setup

The project uses IDD-Lite dataset (~50MB). To set up the dataset:

python setup_data.py

This will download and organize the IDD-Lite dataset in the correct directory structure.

Usage

Training

To train a model:

python train.py --config config.yaml

Configure training parameters in config.yaml:

MODEL_TYPE: 'unet'# Options: 'fcn', 'unet', 'linknet', 'deeplabv3'BACKBONE: 'resnet34'NUM_CLASSES: 8BATCH_SIZE: 16EPOCHS: 100LEARNING_RATE: 0.001

Evaluation

To evaluate a trained model:

python evaluate.py --config config.yaml --model-path checkpoints/final_model.pth

Inference

For inference on a single image:

fromsegmentationimportSegmentationConfig, UNet, Visualizerimportcv2# Initialize model and load weightsconfig=SegmentationConfig(MODEL_TYPE='unet')
model=UNet(config)
model.load_checkpoint('checkpoints/final_model.pth')
# Run inferenceimage=cv2.imread('path/to/image.jpg')
prediction=model.predict(image)

Project Structure

├── segmentation/
│ ├── models/
│ │ ├── fcn.py
│ │ ├── unet.py
│ │ ├── linknet.py
│ │ └── deeplabv3.py
│ ├── config.py
│ ├── dataset.py
│ └── utils/
├── train.py
├── evaluate.py
├── setup_data.py
└── config.yaml