Elegant mathematical transformations for skewed data using spline-based precision
SkewNormalizer is a cutting-edge Python library that transforms skewed data into normal distributions with mathematical precision and perfect reversibility. Unlike traditional methods (Box-Cox, Yeo-Johnson), it uses advanced spline interpolation to create elegant, exact transformations.
- 🧮 Mathematical Precision: Spline-based transformations instead of approximations
- ⚡ Intelligent Subsampling: Handles datasets of any size efficiently
- 🔄 Perfect Reversibility: Error typically < 1e-10
- 🤖 Auto-optimization: Detects optimal parameters automatically
- 📈 Production Ready: Serialization, batch processing, comprehensive metrics
# Install via pip (coming soon)
pip install skewnormalizer
# Install from source
git clone https://github.com/yourrepo/skewnormalizer.git
cd skewnormalizer
pip install -e .- Core:
numpy,scipy - Optional:
matplotlib(for visualizations),pandas(DataFrame support)
importnumpyasnpfromskewnormalizerimportSkewNormalizer# Generate skewed datanp.random.seed(42)
skewed_data=np.concatenate([
np.random.normal(100, 20, 7000), # Main componentnp.random.normal(60, 15, 3000) # Skewing component
])
# Transform to normal distributionnormalizer=SkewNormalizer()
normalized_data=normalizer.fit_transform(skewed_data)
# Perfect reversibilityrecovered_data=normalizer.inverse_transform(normalized_data)
print(f"Original skewness: {normalizer.transformation_metrics['original_skewness']:.3f}")
print(f"Normalized skewness: {normalizer.transformation_metrics['transformed_skewness']:.3f}")
print(f"Reversibility error: {normalizer.transformation_metrics['reversibility_error']:.2e}")Output:
📊 Using subsampling: 5,000 samples from 10,000 (50.0%) - Method: stratified
Original skewness: -0.892
Normalized skewness: -0.023
Reversibility error: 3.45e-11
# For large datasets (>10k points), automatic optimization kicks inlarge_data=np.random.exponential(2, 100_000)
normalizer=SkewNormalizer(
enable_subsampling=True, # Auto-enabled for large datasetssubsample_ratio=0.05, # Use 5% for trainingstratified_sampling=True# Preserve distribution shape
)
# Lightning fast fittingtransformed=normalizer.fit_transform(large_data) # ~0.8s instead of ~15s# Get performance insightsprint(normalizer.summary())| Dataset Size | Without Subsampling | With Subsampling (5%) | Speed Improvement |
|---|---|---|---|
| 10k points | 0.2s | 0.2s | 1x (no change) |
| 100k points | 15s | 0.8s | 19x faster |
| 1M points | 180s | 3.2s | 56x faster |
# Get detailed transformation insightsnormalizer.plot_transformation() # Requires matplotlib# Extract mathematical detailsspline_info=normalizer.get_spline_equation()
print("CDF Spline knots:", spline_info['cdf_spline']['knots'][:5])
# Performance recommendationsrecommendations=normalizer.get_performance_recommendations()# Save trained modelnormalizer.save_model("my_normalizer.pkl")
# Load and useloaded_normalizer=SkewNormalizer.load_model("my_normalizer.pkl")
result=loaded_normalizer.transform(new_data)# Handle extremely large datasets efficientlyhuge_dataset=np.random.gamma(2, 2, 5_000_000) # 5M points# Process in memory-efficient batchestransformed=normalizer.transform_batches(huge_dataset, batch_size=50_000)
recovered=normalizer.inverse_transform_batches(transformed, batch_size=50_000)The visualization includes:
- 📈 Original vs Transformed Distributions
- 📊 Q-Q Plots for normality verification
- 🔵 Spline Functions (CDF and inverse)
- 📋 Quality Metrics summary
- Empirical CDF Estimation:
F(x) = rank(x) / (n+1) - Spline Interpolation: Smooth function fitting with optimal parameters
- Normal Quantile Mapping:
normalized = Φ⁻¹(F(x)) - Inverse Transformation:
original = F⁻¹(Φ(normalized))
# Intelligent parameter selectionSkewNormalizer(
smoothing_method='auto', # GCV, MSE, or manualspline_degree=3, # 1-5, cubic optimal for most datasubsample_threshold=10000, # When to activate subsamplingstratified_sampling=True# Preserve distribution characteristics
)| Method | Reversibility | Speed | Precision | Automation |
|---|---|---|---|---|
| SkewNormalizer | ✅ Perfect (1e-10) | ⚡ Fast | 🎯 High | 🤖 Full |
| Box-Cox | ❌ Parametric only | 🐌 Medium | 📊 Medium | 🔧 Manual |
| Yeo-Johnson | ❌ Parametric only | 🐌 Medium | 📊 Medium | 🔧 Manual |
| QuantileTransformer | ⚡ Fast | 📊 Medium | 🤖 Partial |
# Handles edge cases gracefullytry:
normalizer.fit(problematic_data)
exceptValueErrorase:
print(f"Validation caught: {e}")
# Provides clear guidance for data cleaningValidates against:
- 🚫 NaN and Inf values
- 📏 Insufficient data points
- ⚖️ Zero variance (constant data)
- 🔢 Inappropriate spline degrees
# Run comprehensive test suiteif__name__=="__main__":
# Automatic testing with multiple scenarios# - Small datasets (traditional approach)# - Large datasets (subsampling optimization) # - Quality vs performance trade-offs# - Serialization and batch processing# - Input validation and edge casesWe welcome contributions! Areas of interest:
- 🔬 New smoothing algorithms
- 📊 Additional distribution families
- ⚡ Performance optimizations
- 📚 Documentation improvements
- 🧪 Test coverage expansion
git clone https://github.com/yourrepo/skewnormalizer.git
cd skewnormalizer
pip install -e ".[dev]"
pytest tests/classSkewNormalizer:
def__init__(self, smoothing_method='auto', spline_degree=3, ...)
deffit(self, data, smoothing_factor=None, analyze_full_dataset=False)
deftransform(self, data) ->np.ndarraydefinverse_transform(self, normalized_data) ->np.ndarraydeffit_transform(self, data, ...) ->np.ndarraydefplot_transformation(self, figsize=(15, 10))
defget_spline_equation(self, precision=6) ->Dictdefget_transformation_function(self) ->Tuple[Callable, Callable]
defsummary(self) ->strdefget_performance_recommendations(self) ->Dictdefsave_model(self, filepath)
@classmethoddefload_model(cls, filepath) ->'SkewNormalizer'deftransform_batches(self, data, batch_size=10000) ->np.ndarraydefinverse_transform_batches(self, data, batch_size=10000) ->np.ndarrayThis project is licensed under the MIT License - see the LICENSE file for details.
Developed by David Ochoa with assistance from AI tools during development and optimization.
- SciPy Team: For robust spline interpolation foundations
- NumPy Community: For efficient numerical computing
- Statistics Community: For normalization theory and best practices
- 🐛 Bug Reports: GitHub Issues
- 💬 Discussions: GitHub Discussions
- 📧 Email: your.email@domain.com
- 📖 Documentation: Read the Docs (coming soon)
Made with ❤️ for the Data Science Community
⭐ Star us on GitHub if this helped your project! ⭐