spark-optimization

⚡ Intelligent Data Skew Detection & Mitigation in Apache Spark — ML Severity Classifier + Automated Salting/Repartitioning

Demonstrates PySpark broadcast joins vs regular joins in Databricks, with execution plan comparison for performance optimization.

For the GitHub description field: "Hands-on PySpark performance optimization repo — broadcast joins, partitioning, caching, bucketing, AQE, and skew handling, each with runnable before/after benchmarks and correctness tests.

This project demonstrates key PySpark performance optimization techniques using a synthetic banking transactions dataset (~5,000 records). Built using Databricks and Delta Lake.