Bstrap: A Python Package for confidence intervals and hypothesis testing using bootstrapping.

You are an amazing machine learning researcher.

You invented a new super cool method.

You are not sure that it is significantly better than your baseline.

You don't have 3000 GPUs to rerun your experiment and check it out.

Then, what you want to do is bootstrap your results!

The bstrap package allows you to compare two methods and claim that one is better than the other.

Installation

pip install bstrap

That's all you need, really.

Maybe tough, you can still read the instructions and check out the examples to make sure you get it right...

Features

Bootstrapping is a simple method to compute statistics over your custom metrics, using only one run of the method for each sample in your evaluation set. It has the advantage of being very versatile, and can be used with any metric really.

• Bootstrapping for computation of confidence interval
• Bootstrapping for hypothesis testing (claim that one method is better than another for a given metric)
• Supports metrics that can be computed sample-wise and metrics that cannot.

Keep in mind: non-overlapping confidence intervals means that there is a significant statistical difference. Overlapping confidence intervals does not mean that there is no significant statistical difference. To verify this further, you will need to compute the bootstrap hypothesis testing and check the p-value.

Instructions

You will need to implement your metric and provide the data sample-wise as a single Pandas dataframe for each method. That's about it. Your metric is more complex than simply averaging results for each sample? For example, you cannot compute sample-wise, maybe like AUC or mAP? Then just give your predictions and ground truths sample-wise, which also works with Boostrap.

To use this code, you need to:

1. Implement your own metric: should take the one pandas dataframe of data as input and return a scalar value.
2. Load your data.
3. Reformat data to a single pandas dataframe per method with standardized column names, and one sample per row.
4. Check that your estimates (confidence interval and p-value) are stable over several runs of the bootstrapping method. If the estimates are not stable, increase nbr_runs

Enjoy!

Usage

You can find example dataframes under src/bstrap/example_dataframes.

Example 1: Mean metric

importpandasaspdimportnumpyasnpfrombstrapimportbootstrap, boostrapping_CI# 1. implement metricmetric=np.mean# 2. load datadf=pd.read_csv("example_dataframes/example_dataframe_mean.csv")
# 3. reformat data to a single pandas dataframe per methoddata_method1=df["loss_method_1"]
data_method2=df["loss_method_2"]
# 4. compare method 1 and 2stats_method1, stats_method2, p_value=bootstrap(metric, data_method1, data_method2, nbr_runs=1000)
print(stats_method1)
print(stats_method2)
print(p_value)
# compute CI and mean for each method separatelystats_method1=boostrapping_CI(metric, data_method1, nbr_runs=1000)
stats_method2=boostrapping_CI(metric, data_method2, nbr_runs=1000)
print(stats_method1)
print(stats_method2)

Example 2: F1 score

importpandasaspdimportsysfrombstrapimportbootstrap, boostrapping_CI# 1. implement metricdefcompute_f1(data):
val_target=data["gt"].astype('bool')
val_predict=data["predictions"].astype('bool')
tp=np.count_nonzero(val_target*val_predict)
fp=np.count_nonzero(~val_target*val_predict)
fn=np.count_nonzero(val_target*~val_predict)
returntp*1./ (tp+0.5* (fp+fn) +sys.float_info.epsilon)
metric=compute_f1# 2. load datadf=pd.read_csv("example_dataframes/example_dataframe_f1.csv")
# 3. reformat data to a single pandas dataframe per method with standardized column namesdata_method1=df[["gt", "method1"]]
data_method1=data_method1.rename(columns={"method1": "predictions"})
data_method2=df[["gt", "method2"]]
data_method2=data_method2.rename(columns={"method2": "predictions"})
# 4. compare method 1 and 2 (same code as example 1)stats_method1, stats_method2, p_value=bootstrap(metric, data_method1, data_method2, nbr_runs=1000)
print(stats_method1)
print(stats_method2)
print(p_value)
# compute CI and mean for each method separately (same code as example 1)stats_method1=boostrapping_CI(metric, data_method1, nbr_runs=1000)
stats_method2=boostrapping_CI(metric, data_method2, nbr_runs=1000)
print(stats_method1)
print(stats_method2)

Example 3: AUC

importpandasaspdfromsklearn.metricsimportauc, roc_curvefrombstrapimportbootstrap, boostrapping_CI# 1. implement metricdefcompute_auc(data):
fpr, tpr, thresholds=roc_curve(data["gt"], data["predictions"], pos_label=1)
returnauc(fpr, tpr)
metric=compute_auc# 2. load datadf=pd.read_csv("example_dataframes/example_dataframe_auc.csv")
# 3. reformat data to a single pandas dataframe per method with standardized column namesdata_method1=df[["gt", "method1"]]
data_method1=data_method1.rename(columns={"method1": "predictions"})
data_method2=df[["gt", "method2"]]
data_method2=data_method2.rename(columns={"method2": "predictions"})
# 4. compare method 1 and 2 (same code as example 1)stats_method1, stats_method2, p_value=bootstrap(metric, data_method1, data_method2, nbr_runs=1000)
print(stats_method1)
print(stats_method2)
print(p_value)
# compute CI and mean for each method separately (same code as example 1)stats_method1=boostrapping_CI(metric, data_method1, nbr_runs=1000)
stats_method2=boostrapping_CI(metric, data_method2, nbr_runs=1000)
print(stats_method1)
print(stats_method2)

Example 4: Multiclass: mean Average Precision (mAP)

importpandasaspdfromsklearn.metricsimportaverage_precision_scorefrombstrapimportbootstrap, boostrapping_CI# 1. implement metricdefcompute_mAP(data):
gt=data[[columnforcolumnindata.columnsif'gt'incolumn]]
predictions=data[[columnforcolumnindata.columnsif'pred'incolumn]]
returnaverage_precision_score(gt, predictions, average='weighted')
metric=compute_mAP# 2. load datagt=pd.read_csv("example_dataframes/example_dataframe_mAP_gt.csv")
predictions_method1=pd.read_csv("example_dataframes/example_dataframe_mAP_predictions_method1.csv")
predictions_method2=pd.read_csv("example_dataframes/example_dataframe_mAP_predictions_method2.csv")
# 3. reformat data to a single pandas dataframe per method with standardized column namesgt=gt.rename(columns=dict([(column, 'gt_'+column) forcolumningt.columns]))
predictions_method1=predictions_method1.rename(
columns=dict([(column, 'pred_'+column) forcolumninpredictions_method1.columns]))
predictions_method2=predictions_method2.rename(
columns=dict([(column, 'pred_'+column) forcolumninpredictions_method2.columns]))
data_method1=pd.concat([gt, predictions_method1], axis=1)
data_method2=pd.concat([gt, predictions_method2], axis=1)
# 4. compare method 1 and 2 (same code as example 1)stats_method1, stats_method2, p_value=bootstrap(metric, data_method1, data_method2, nbr_runs=100)
print(stats_method1)
print(stats_method2)
print(p_value)
# compute CI and mean for each method separately (same code as example 1)stats_method1=boostrapping_CI(metric, data_method1, nbr_runs=100)
stats_method2=boostrapping_CI(metric, data_method2, nbr_runs=100)
print(stats_method1)
print(stats_method2)

Reference:
Efron, B. and Tibshirani, R.J., 1994. An introduction to the bootstrap. CRC press.