RL Pipelines — Bank Marketing Dataset

Two end-to-end machine learning pipelines on the Bank Marketing dataset (bank-full.csv, 45 211 rows, 17 columns):

• Pipeline 1 — Missing Data Simulation + Q-Learning Imputation
• Pipeline 2 — Imbalanced Learning + RL Reward-Shaping

---

Project Structure

RL/
├── main.py # Entry point — runs both pipelines
├── config.py # Paths, column lists, all hyperparameters
│
├── data/
│ └── loader.py # load_and_preprocess, check_edge_cases
│
├── ground_truth/
│ └── generator.py # Step 0: XGBoost + Logistic Regression baseline
│
├── pipeline1/
│ ├── missingness.py # inject_mcar, inject_mar
│ ├── baseline_imputer.py # Mean / Median / KNN / Random Forest imputers
│ ├── rl_imputer.py # RLImputer — Q-Learning class
│ ├── evaluator.py # evaluate_imputation, MAE/RMSE table printer
│ └── pipeline.py # run_pipeline1 orchestrator
│
├── pipeline2/
│ ├── analysis.py # analyze_imbalance
│ ├── rl_reward_shaper.py # RLRewardShaper — vectorised weight update loop
│ ├── evaluator.py # imbalance_metrics, F1/MCC/G-Mean table printer
│ └── pipeline.py # run_pipeline2 orchestrator
│
└── outputs/ # All generated CSVs and JSON metrics (never modified by hand)

---

Dataset

Property	Value
File	bank-full.csv (semicolon-delimited)
Rows	45 211
Features	16 (7 numeric, 9 categorical)
Target y	Binary — no (39 922) / yes (5 289)
Imbalance Ratio	7.55 : 1
Missing values	None (injected synthetically in Pipeline 1)

Numeric columns: age, balance, day, duration, campaign, pdays, previous

---

Quickstart

pip install xgboost scikit-learn pandas numpy
python main.py

All outputs are written to outputs/. Nothing else is modified.

---

Pipeline 1 — Missing Data + RL Imputation

Flow

Clean dataset
│
├── Inject MCAR (20 % per column, random)
└── Inject MAR (conditioned on age, campaign, pdays)
│
├── Baseline imputers: Mean · Median · KNN · Random Forest
└── RL Imputer (Q-Learning)
│
└── evaluate_imputation → MAE, RMSE per method

Missingness Rules

Type	Rule
MCAR	20 % of values blanked uniformly at random in each numeric column
MAR	age > median(age) → balance missing (80 % of qualifying rows)
MAR	campaign > 2 → duration missing (75 %)
MAR	pdays == min → previous missing (70 %)

RL Imputer — Q-Learning Design

Component	Definition
State	Z-score bin of the current candidate value (12 bins, range −3σ to +3σ)
Actions	0 = increase by δ, 1 = decrease by δ (δ = 8 % of column std)
Reward	prev_error − new_error — positive when moving toward the true value
Initial guess	KNN estimate (k = 5) from neighbouring rows in feature space
Training	400 episodes on observed rows with simulated masking; ε-greedy with decay 0.40 → 0.05
Inference	Greedy argmax policy applied until the state bin stops changing

Using KNN as the starting point (rather than the column mean) ensures each missing value receives a distinct initial estimate, so the RL policy produces dynamic — not constant — imputations.

Imputation Results

Method	MCAR MAE	MCAR RMSE	MAR MAE	MAR RMSE
Mean	233.06	359.81	557.62	882.37
Median	195.11	382.65	496.50	947.06
KNN	251.65	393.66	631.23	954.90
Random Forest	248.09	378.01	635.75	938.83
RL	245.96	371.82	546.29	830.94

RL achieves the best RMSE in both scenarios — it penalises large-error outliers less than the other methods.

Output Files

File	Description
missing_dataset_MCAR_20pct.csv	Dataset with MCAR missingness injected
missing_dataset_MAR.csv	Dataset with MAR missingness injected
rl_imputed_MCAR_20pct.csv	RL-imputed output (MCAR)
rl_imputed_MAR.csv	RL-imputed output (MAR)
baseline_imputed_RF_MCAR_20pct.csv	RF baseline imputed (MCAR)
baseline_imputed_RF_MAR.csv	RF baseline imputed (MAR)
imputation_metrics.json	MAE + RMSE for all methods, both scenarios

---

Pipeline 2 — Imbalanced Learning + RL Reward Shaping

Flow

Clean dataset (IR = 7.55 : 1)
│
├── Baseline: XGBoost (no weighting)
├── Baseline: XGBoost + scale_pos_weight
├── Baseline: Logistic Regression (class_weight='balanced')
└── RL Reward Shaper
│ 60 iterations: train XGBoost → assign rewards → update weights
└── Final XGBoost trained on RL-learned sample weights
│
└── evaluate → F1, MCC, G-Mean, Recall, Precision

RL Reward-Shaping Design

Prediction outcome	Reward
Correct minority (yes)	+5
Wrong minority (yes)	−5
Correct majority (no)	+1
Wrong majority (no)	−1

Weight update per iteration (vectorised):

misclassified : w_i ×= (1 + α · |reward|) # boost hard samples
correct : w_i ×= max(floor, 1 − α · decay) # gently reduce easy samples

Parameters: α = 0.04, decay = 0.05, weight_floor = 0.05, n_iterations = 60

Classification Results

Model	F1	MCC	G-Mean	Recall
XGBoost Baseline	0.5589	0.5131	0.6923	0.4981
XGBoost scale_pos_weight	0.5999	0.5535	0.8352	0.7836
LogReg Balanced	0.4962	0.4473	0.8049	0.7977
XGBoost RL RewardShaping	0.4609	0.3883	0.6580	0.4679

Output Files

File	Description
baseline_predictions.csv	XGBoost baseline predictions on test set
rl_predictions.csv	RL reward-shaping model predictions
ground_truth_predictions.csv	Step 0 ground truth (XGBoost + LogReg)
imbalance_metrics.json	F1, MCC, G-Mean, Recall, Precision for all models

---

Configuration

All tuneable values live in config.py — no other file needs editing to change hyperparameters.

MCAR_RATE=0.20# fraction of values blanked per columnRL_IMP=dict(
n_states=12, # Q-table z-score binsn_episodes=400, # training episodes per columnalpha=0.15, # Q-learning rategamma=0.90, # discount factorepsilon_start=0.40, # initial exploration rateepsilon_end=0.05, # final exploration rateknn_k=5, # neighbours for initial guess
)
RL_SHAPER=dict(
n_iterations=60, # reward-shaping iterationsalpha=0.04, # weight update step sizedecay=0.05, # correct-sample weight decayweight_floor=0.05, # minimum sample weight
)

---

Edge Case Handling

Scenario	Handling
Column > 40 % missing	Warning printed; RL still runs, KNN falls back to column mean
Entire column missing	Replaced with global column mean from training data
Non-numeric columns	Label-encoded before any pipeline step
Extreme outliers	Winsorised at 1st–99th percentile on load
Extreme class imbalance	scale_pos_weight baseline + RL reward asymmetry (+5/−5 vs +1/−1)
Minority absent in batch	Stratified train/test split ensures minority always present