Smart Building Energy Management with Multi-Agent Reinforcement Learning

Multi-Agent Reinforcement Learning for Smart Building Energy Management using CityLearn

Overview

This project applies and compares five control strategies for smart building demand response:

Algorithm	Type	Description
RBC	Baseline	Rule-Based Controller (charge at night, discharge at peak)
PPO	Single-Agent RL	Proximal Policy Optimization with flattened observations
SAC	Single-Agent RL	Soft Actor-Critic with flattened observations
I-PPO	Multi-Agent RL	Independent PPO (one agent per building)
MAPPO	Multi-Agent RL	Centralized Training, Decentralized Execution

Project Structure

project/
├── requirements.txt # Python dependencies
├── README.md # This file
├── setup.bat / setup.sh # Environment setup scripts
├── src/
│ ├── __init__.py
│ ├── environment_wrapper.py # CityLearn Gymnasium wrappers
│ ├── baseline_rbc.py # Rule-Based Controller
│ ├── single_agent.py # PPO and SAC training
│ ├── multi_agent.py # I-PPO and MAPPO training
│ ├── utils.py # Shared utilities and KPI computation
│ └── evaluation.py # Unified evaluation and comparison
├── train.py # Main training script
├── evaluate.py # Model evaluation script
├── plot_results.py # Figure generation script
├── report.md # Detailed project report
├── figures/ # Generated figures
├── models/ # Saved trained models
└── results/ # JSON result files

Quick Start

1. Install Dependencies

# Option A: Using setup script# Windows:
setup.bat
# Linux/Mac:
chmod +x setup.sh && ./setup.sh
# Option B: Manual installation
pip install -r requirements.txt

2. Run All Experiments

python train.py

This will sequentially run:

1. RBC baseline evaluation
2. PPO single-agent training (200K timesteps)
3. SAC single-agent training (200K timesteps)
4. I-PPO multi-agent training (200K timesteps)
5. MAPPO multi-agent training (200K timesteps)

3. Run Individual Experiments

# Run only the baseline
python train.py --algorithm rbc
# Run only PPO
python train.py --algorithm ppo
# Run only SAC
python train.py --algorithm sac
# Run single-agent algorithms (PPO + SAC)
python train.py --algorithm single
# Run multi-agent algorithms (I-PPO + MAPPO)
python train.py --algorithm multi
# Customise training steps
python train.py --algorithm ppo --timesteps 50000
# Set random seed
python train.py --seed 123

4. Evaluate Saved Models

python evaluate.py # Evaluate all saved models
python evaluate.py --algorithm ppo # Evaluate specific model

5. Generate Plots

python plot_results.py

This creates:

• figures/rbc_load_curve.png — RBC district load curve (summer week)
• figures/single_agent_learning_curves.png — PPO vs SAC learning curves
• figures/multi_agent_learning_curves.png — I-PPO vs MAPPO learning curves
• figures/best_agent_load_curve.png — Best algorithm's load curve
• figures/comparison_bar_chart.png — Normalised score comparison

Environment

• CityLearn schema: citylearn_challenge_2022_phase_1
• Buildings: 5 residential buildings
• Episode length: 8,760 timesteps (1 year, hourly resolution)
• Dataset: Built-in (no external data required)

Key Performance Indicators (KPIs)

KPI	Description	Goal
Ramping	Load curve smoothness	Lower is better
1 − Load Factor	Load utilisation efficiency	Lower is better
Average Daily Peak	Mean daily peak demand	Lower is better
Peak Demand	Maximum annual demand	Lower is better
Net Electricity Consumption	Total annual electricity	Lower is better

All values are normalised so that the no-control baseline = 1.0.

Requirements

• Python ≥ 3.8
• See requirements.txt for full package list

Report

See report.md for a comprehensive analysis including:

• MDP formulation (28 observation features, continuous actions)
• Algorithm descriptions and hyperparameters
• Results, analysis, and discussion
• References to relevant literature

Reproducibility

All experiments use seed 42 by default. Override with --seed <value>.

# Full reproducible run
python train.py --seed 42
python plot_results.py

License

This project is for educational purposes only.