💡 Pearsonify

Probabilistic Classification with Conformalized Intervals

Pearsonify is a lightweight 🐍 Python package for generating classification intervals around predicted probabilities in binary classification tasks.

It uses Pearson residuals and principles of conformal prediction to quantify uncertainty without making strong distributional assumptions.

🚀 Why Pearsonify?

• 📊 Intuitive Classification Intervals: Get reliable intervals for binary classification predictions.
• 🧠 Statistically Grounded: Uses Pearson residuals, a well-established metric from classical statistics.
• ⚡ Model-Agnostic: Works with any model that provides probability estimates.
• 🛠️ Lightweight: Minimal dependencies, easy to integrate into existing projects.

📦 How to install?

Use pip to install the package from GitHub:

pip install pearsonify
# or from GitHub:
pip install git+https://github.com/xRiskLab/pearsonify.git

💻 How to use?

importnumpyasnpfrompearsonifyimportPearsonifyfromsklearn.svmimportSVCfromsklearn.datasetsimportmake_classificationfromsklearn.model_selectionimporttrain_test_split# Generate synthetic classification datanp.random.seed(42)
X, y=make_classification(
n_samples=1000, n_features=20, n_informative=10, n_classes=2, random_state=42
)
# Split data into train, calibration, and test setsX_train, X_temp, y_train, y_temp=train_test_split(X, y, test_size=0.4, random_state=42)
X_cal, X_test, y_cal, y_test=train_test_split(X_temp, y_temp, test_size=0.5, random_state=42)
# Initialize Pearsonify with an SVC modelclf=SVC(probability=True, random_state=42)
model=Pearsonify(estimator=clf, alpha=0.05)
# Fit the model on training and calibration setsmodel.fit(X_train, y_train, X_cal, y_cal)
# Generate prediction intervals for test sety_test_pred_proba, lower_bounds, upper_bounds=model.predict_intervals(X_test)
# Calculate coveragecoverage=model.evaluate_coverage(y_test, lower_bounds, upper_bounds)
print(f"Coverage: {coverage:.2%}")
# Plot the intervalsmodel.plot_intervals(y_test_pred_proba, lower_bounds, upper_bounds)

Running example.py will generate the following plot:

This plot shows predicted probabilities with 95% confidence intervals, sorted by prediction score.

📖 References

Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied Logistic Regression. John Wiley & Sons.

Tibshirani, R. (2023). Conformal Prediction. Advanced Topics in Statistical Learning, Spring 2023.

📝 License

This project is licensed under the MIT License - see the LICENSE file for details.