Fast, scalable, and scikit-learn compatible optimization for machine learning
ReHLine-Python is the official Python implementation of ReHLine, a powerful solver for large-scale empirical risk minimization (ERM) problems with convex piecewise linear-quadratic (PLQ) loss functions and linear constraints. Built with high-performance C++ core and seamless Python integration, ReHLine delivers exceptional speed while maintaining ease of use.
See more details in the ReHLine documentation.
- 🚀 Blazing Fast: Linear computational complexity per iteration, scales to millions of samples
- 🎯 Versatile: Supports any convex PLQ loss (hinge, check, Huber, and more)
- 🔒 Constrained Optimization: Handle linear equality and inequality constraints
- 📊 Scikit-Learn Compatible: Drop-in replacement with
GridSearchCV,Pipelinesupport - 🐍 Pythonic API: Both low-level and high-level interfaces for flexibility
pip install rehlineFor contributors and developers:
git clone https://github.com/softmin/ReHLine-python.git
cd ReHLine-python
pip install -e ".[dev]"To run tests:
pytest tests/
ReHLine provides plq_Ridge_Classifier and plq_Ridge_Regressor that work seamlessly with scikit-learn:
fromrehlineimportplq_Ridge_Classifierfromsklearn.datasetsimportmake_classificationfromsklearn.model_selectionimporttrain_test_split, GridSearchCVfromsklearn.pipelineimportPipelinefromsklearn.preprocessingimportStandardScaler# Generate datasetX, y=make_classification(n_samples=1000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test=train_test_split(X, y, test_size=0.2)
# Simple usageclf=plq_Ridge_Classifier(loss={'name': 'svm'}, C=1.0)
clf.fit(X_train, y_train)
print(f"Accuracy: {clf.score(X_test, y_test):.3f}")
# Use in Pipelinepipeline=Pipeline([
('scaler', StandardScaler()),
('classifier', plq_Ridge_Classifier(loss={'name': 'svm'}))
])
pipeline.fit(X_train, y_train)
# Hyperparameter tuning with GridSearchCVparam_grid= {
'C': [0.1, 1.0, 10.0],
'loss': [{'name': 'svm'}, {'name': 'sSVM'}]
}
grid_search=GridSearchCV(plq_Ridge_Classifier(loss={"name": "svm"}), param_grid, cv=5)
grid_search.fit(X_train, y_train)
print(f"Best params: {grid_search.best_params_}")See more details in ReHLine with Scikit-Learn.
fromrehlineimportReHLineimportnumpyasnp# Generate sample datanp.random.seed(42)
X=np.random.randn(100, 5)
y=np.random.choice([-1, 1], size=100)
n, d=X.shapeC=1.0# Define custom PLQ loss parametersclf=ReHLine()
# Set custom U, V matrices for ReLU loss# and S, T, tau for ReHU loss## Uclf._U=-(C*y).reshape(1,-1)
## Vclf._V= (C*np.ones(n)).reshape(1,-1)
# Set custom linear constraints A*beta + b >= 0X_sen=X[:,0]
tol_sen=0.1clf._A=np.repeat([X_sen @ X], repeats=[2], axis=0) /nclf._A[1] =-clf._A[1]
clf.fit(X)See more detailed in Manual ReHLine Formulation.
ReHLine excels at solving a wide range of machine learning problems:
| Problem | Description | Key Benefits |
|---|---|---|
| Support Vector Machines | Binary and multi-class classification | 100-400× faster than CVXPY solvers |
| Fair Machine Learning | Classification with fairness constraints | Handles demographic parity efficiently |
| Quantile Regression | Robust conditional quantile estimation | 2800× faster than general solvers |
| Huber Regression | Outlier-resistant regression | Superior to specialized solvers |
| Sparse Learning | Feature selection with L1 regularization | Scales to high dimensions |
| Custom Optimization | Any PLQ loss with linear constraints | Flexible framework for research |
ReHLine delivers exceptional speed compared to state-of-the-art solvers. Here are speed-up factors on real-world datasets:
| Task | vs. ECOS | vs. MOSEK | vs. SCS | vs. Specialized Solvers |
|---|---|---|---|---|
| SVM | 415× faster | ∞ (failed) | 340× faster | 4.5× vs. LIBLINEAR |
| Fair SVM | 273× faster | 100× faster | 252× faster | ∞ vs. DCCP (failed) |
| Quantile Regression | 2843× faster | ∞ (failed) | ∞ (failed) | — |
| Huber Regression | ∞ (failed) | 452× faster | ∞ (failed) | 2.4× vs. hqreg |
| Smoothed SVM | — | — | — | 1.6-2.3× vs. SAGA/SAG/SDCA/SVRG |
Note: "∞" indicates the competing solver failed to produce a valid solution or exceeded time limits. Results from NeurIPS 2023 paper.
All benchmarks are reproducible via benchopt at our ReHLine-benchmark repository.
| Problem | Benchmark Code | Interactive Results |
|---|---|---|
| SVM | Code | 📊 View |
| Smoothed SVM | Code | 📊 View |
| Fair SVM | Code | 📊 View |
| Quantile Regression | Code | 📊 View |
| Huber Regression | Code | 📊 View |
We welcome contributions! Whether it's bug reports, feature requests, or code contributions:
- 🐛 Open an issue
- 💬 Start a discussion
- 🔀 Submit a pull request
If you use ReHLine in your research, please cite our NeurIPS 2023 paper:
@inproceedings{dai2023rehline,
title={ReHLine: Regularized Composite ReLU-ReHU Loss Minimization with Linear Computation and Linear Convergence},
author={Dai, Ben and Qiu, Yixuan},
booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
year={2023}
}
|
|