ColdSnap: Freeze ML models and their training/testing data

The ColdSnap framework allows for training/testing data as well as machine learning models to be "frozen" aka serialized to disk.

Machine learning projects often require careful tracking and storage of not only model architectures and parameters but also the datasets they were trained on. Having a robust mechanism for storing both models and their associated data snapshots is essential for reproducibility, version control, and long-term evaluation of model performance. ColdSnap was created to address these needs by providing a unified framework where machine learning models and their corresponding datasets can be seamlessly stored, serialized, and evaluated. By preserving both the model and data as a single unit, ColdSnap enables consistent evaluation across iterations, aids in model comparisons, and ensures that all aspects of a model’s creation—data transformations, training splits, and performance metrics—are easily retrievable, facilitating high-quality machine learning workflows.

Installation

How to use ColdSnap

The code below can be found in ./docs/example, in a nutshell you create a Data object, which contains your training and testing data, that data is added to a model along with the classifier to use and that can be serialized to disk. This model, along with the data, can be loaded again from another script/notebook. The create_overview function can summarize a list of models.

Creating Snapshot of Data and Models

The code below shows how to create and store Data and Models.

fromcoldsnapimportData, Modelfromsklearnimportdatasetsfromsklearn.ensembleimportRandomForestClassifierimportpandasaspdimportosiris=datasets.load_iris(as_frame=True)
iris_df=pd.merge(
iris.data, iris.target, how="inner", left_index=True, right_index=True
)
if__name__=="__main__":
try:
os.mkdir("./tmp/")
exceptFileExistsError:
passcs_data=Data.from_df(
iris_df, "target", random_state=1910, description="Iris Dataset"
)
cs_data.to_pickle("./tmp/iris_data.pkl.gz")
# Create random forest classifierclf=RandomForestClassifier(random_state=1910)
cs_model=Model(
data=cs_data,
clf=clf,
description="RandomForestClassifier, default params on Iris dataset",
)
cs_model.fit()
cs_model.to_pickle("./tmp/iris_model.pkl.gz")

Using Transformers with ColdSnap

ColdSnap also supports sklearn transformers like StandardScaler, PCA, etc. This is useful for saving fitted preprocessing pipelines along with your data. The example below shows how to fit a StandardScaler and save it as a snapshot.

fromcoldsnapimportData, Modelfromsklearnimportdatasetsfromsklearn.preprocessingimportStandardScalerimportpandasaspdimportosiris=datasets.load_iris(as_frame=True)
iris_df=pd.merge(
iris.data, iris.target, how="inner", left_index=True, right_index=True
)
if__name__=="__main__":
try:
os.mkdir("./tmp/")
exceptFileExistsError:
pass# Create data objectcs_data=Data.from_df(
iris_df, "target", random_state=1910, description="Iris Dataset"
)
# Create a StandardScaler transformerscaler=StandardScaler()
# Create a Model with the transformer using the 'estimator' parametercs_scaler_model=Model(
data=cs_data,
estimator=scaler,
description="StandardScaler for Iris dataset",
)
# Fit the scalercs_scaler_model.fit()
# Transform the training dataX_train_scaled=cs_scaler_model.transform(cs_data.X_train)
print("Original data (first 3 samples):")
print(cs_data.X_train.head(3))
print("\nScaled data (first 3 samples):")
print(X_train_scaled.head(3)) # DataFrame structure is preserved automatically# Save the fitted transformercs_scaler_model.to_pickle("./tmp/iris_scaler.pkl.gz")
# Later, you can load and use it on new dataloaded_scaler=Model.from_pickle("./tmp/iris_scaler.pkl.gz")
X_test_scaled=loaded_scaler.transform(cs_data.X_test)
print("\nTransformer successfully saved and loaded!")

Using Regressors with ColdSnap

ColdSnap supports sklearn regression models like LinearRegression, Ridge, RandomForestRegressor, etc. Regressors work similarly to classifiers but return regression-specific evaluation metrics (RMSE, MAE, R2, MSE).

fromcoldsnapimportData, Modelfromsklearnimportdatasetsfromsklearn.linear_modelimportLinearRegressionimportpandasaspdimportos# Load the iris dataset for regressioniris=datasets.load_iris(as_frame=True)
iris_df=pd.merge(
iris.data, iris.target, how="inner", left_index=True, right_index=True
)
# Drop the target column as we'll predict petal width from other featuresiris_df=iris_df.drop(columns=["target"])
if__name__=="__main__":
try:
os.mkdir("./tmp/")
exceptFileExistsError:
pass# Create data object for regression - predict petal width from other measurementscs_data=Data.from_df(
iris_df, "petal width (cm)", random_state=1910, description="Iris Petal Width Regression"
)
# Create a LinearRegression modelregressor=LinearRegression()
cs_model=Model(
data=cs_data,
estimator=regressor,
description="LinearRegression predicting petal width on Iris dataset",
)
# Fit the modelcs_model.fit()
# Evaluate with regression metricsmetrics=cs_model.evaluate()
print("Regression Metrics:")
print(f" RMSE: {metrics['rmse']:.4f}")
print(f" MAE: {metrics['mae']:.4f}")
print(f" R2 Score: {metrics['r2']:.4f}")
print(f" MSE: {metrics['mse']:.4f}")
# Make predictionspredictions=cs_model.predict(cs_data.X_test)
# Save the modelcs_model.to_pickle("./tmp/iris_regressor.pkl.gz")
print("\nRegressor successfully trained, evaluated, and saved!")

Loading a Model

Once a model has been stored, it can easily be loaded using Model.from_pickle(path). Once loaded, details on the model and its performance can be retrieved using .summary.

fromcoldsnapimportModelif__name__=="__main__":
try:
cs_model=Model.from_pickle("./tmp/iris_model.pkl.gz")
exceptOSError:
print("Model not found, run the script to create models first !")
quit()
print(cs_model.summary())

Creating an Overview of Your Models

To quickly compare a number of models the function create_overview can be used as shown below.

fromcoldsnap.utilsimportcreate_overviewif__name__=="__main__":
paths= [
"./tmp/iris_model.pkl.gz",
"./tmp/iris_model_svc.pkl.gz",
"./tmp/iris_model_dt.pkl.gz",
]
overview_df=create_overview(paths)
print(overview_df.to_markdown())

The table below shows the output, you get for each model in the input list the summary and evaluation criteria.

	path	model_code	model_description	model_hash	data_code	data_description	data_hash	num_features	features	num_classes	classes	accuracy	precision	recall	f1	roc_auc
0	./tmp/iris_model.pkl.gz	RF01	RandomForestClassifier, default params on Iris dataset	b3f8665bce0ee979b51c9729019ae76d7ed3b83522024b9fb3375e1b96a3dc11	IrD	Iris Dataset	975cdbb5f836a810ad019751a998b18683437093f372f4545fd00be5335d5e4b	4	sepal length (cm), sepal width (cm), petal length (cm), petal width (cm)	3	0, 1, 2	0.973684	0.975564	0.973684	0.973545	0.997973
1	./tmp/iris_model_svc.pkl.gz	SVC01	SVC (with probabilities) on Iris dataset	280f5c4ca76b77144bbe7e9768bfc663b45fdafe61be3bbdc793458597f75e07	IrD	Iris Dataset	975cdbb5f836a810ad019751a998b18683437093f372f4545fd00be5335d5e4b	4	sepal length (cm), sepal width (cm), petal length (cm), petal width (cm)	3	0, 1, 2	0.973684	0.975564	0.973684	0.973545	0.997973
2	./tmp/iris_model_dt.pkl.gz	DT01	DecisionTreeClassifier (max_depth=2) on Iris dataset	3814de3d290288de03f1b2388897c964b967c7f8ffa44303c61c41575da5d856	IrD	Iris Dataset	975cdbb5f836a810ad019751a998b18683437093f372f4545fd00be5335d5e4b	4	sepal length (cm), sepal width (cm), petal length (cm), petal width (cm)	3	0, 1, 2	0.947368	0.947368	0.947368	0.947368	0.975673

Evaluating Model performance

There are a few common metrics built into ColdSnap. See the example below (which assumes a model is loaded in cs_model).

print(cs_model.evaluate())
# Confusion matrixprint(cs_model.confusion_matrix())
fig, ax=plt.subplots()
disp=cs_model.display_confusion_matrix(ax=ax, cmap="Blues")
plt.show()
# ROC curvefig, ax=plt.subplots()
roc_disp=cs_model.display_roc_curve(ax=ax)
plt.show()
# SHAP beeswarmcs_model.display_shap_beeswarm()

Contributing

Any contributions you make are greatly appreciated.

• Found a bug or have some suggestions? Open an issue.
• Pull requests are welcome! Though open an issue first to discuss which features/changes you wish to implement.

Contact

ColdSnap was developed by Sebastian Proost at the RaesLab. ColdSnap is available under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license.

For commercial access inquiries, please contact Jeroen Raes.